Development

• Development Setup
  • Install conda / mamba
  • Fork and Clone the PUDL Repository
  • Create the PUDL Dev Environment
  • Updating the PUDL Development Environment
  • Set Up Code Linting
  • Creating a Workspace
• Running the ETL Pipeline
  • Database initialization
  • Database schema migration
  • Dagster
  • Running the ETL via the Dagster UI
  • Running the ETL with CLI Commands
  • Settings Files
  • The Fast ETL
  • The Full ETL
  • Custom ETL
  • Processing EPA CEMS Separately
  • Additional Notes
  • Foreign Keys
• Developing with Dagster
  • Reloading code locations
  • Viewing Logs
  • Assets getting out of sync
  • Configuring resources
• Project Management
  • Issues and Project Tracking
  • Our GitHub Workflow
  • Pull Requests
  • Releases
  • User Support
• Testing PUDL
  • Software Tests
  • Running tests with Make
  • Selecting Input Data for Integration Tests
  • Data Validation
  • Running pytest Directly
• Building the Documentation
• Working with the Datastore
  • Adding a new Dataset to the Datastore
• Cloning the FERC Form 1 DB
• Annual Updates
  • 1. Obtain Fresh Data
  • 2. Map the Structure of the New Data
  • 3. Test Data Extraction
  • 4. Update Table & Column Transformations
  • 5. Update the PUDL DB Schema
  • 6. Connect Datasets
  • 7. Run the ETL
  • 8. Update the Output Routines and Run Full Tests
  • 9. Run and Update Data Validations
  • 10. Update the Documentation
• PUDL ID Mapping
  • The Status of FERC 1 and EIA IDs
  • PUDL IDs
  • FERC1 Utility IDs
  • Checking for Unmapped Records
  • Assigning PUDL IDs to Unmapped Records
  • Testing Newly Mapped Records
  • Integrating Newly Mapped Records into PUDL
• Naming Conventions
  • Glossary of Abbreviations
  • Data Extraction Functions
  • Output Functions
  • Table Names
  • Columns and Field Names
• Data and ETL Design Guidelines
  • Input vs. Output Data
  • Minimize Data Alteration
  • Make Tidy Data
  • Use Consistent Units
  • Silo the ETL Process
  • Separate Data from Glue
  • Partition Big Data
  • Naming Conventions
  • Complete, Continuous Time Series
• Nightly Data Builds
  • Breaking the Builds
  • Debugging a Broken Build
  • The GitHub Action
  • Google Compute Engine
  • Docker
  • How to access the nightly build outputs from AWS
  • How to access the nightly build outputs and logs (for the Catalyst team only)
• Infrastructure as Code
  • Overview
  • Setup
  • Development