pudl.etl
#
Dagster definitions for the PUDL ETL and Output tables.
Submodules#
Package Contents#
Classes#
Main settings validation class. |
Functions#
|
IO Manager that writes EPA CEMS partitions to individual parquet files. |
|
Create a SQLiteManager dagster resource for the ferc1 dbf database. |
|
Create a SQLiteManager dagster resource for the ferc1 dbf database. |
|
Create a SQLiteManager dagster resource for the pudl database. |
|
Dagster resource for parameterizing PUDL ETL assets. |
|
Dagster resource to interact with Zenodo archives. |
Dagster resource for parameterizing the |
|
|
Create a selection of assets excluding CEMS and all downstream assets. |
|
Load dataset settings from a settings file in pudl.package_data.settings. |
Attributes#
A collection of dagster assets, resources, IO managers, and jobs for the PUDL ETL. |
- pudl.etl.epacems_io_manager(init_context: dagster.InitResourceContext) PandasParquetIOManager [source]#
IO Manager that writes EPA CEMS partitions to individual parquet files.
- pudl.etl.ferc1_dbf_sqlite_io_manager(init_context) FercDBFSQLiteIOManager [source]#
Create a SQLiteManager dagster resource for the ferc1 dbf database.
- pudl.etl.ferc1_xbrl_sqlite_io_manager(init_context) FercXBRLSQLiteIOManager [source]#
Create a SQLiteManager dagster resource for the ferc1 dbf database.
- pudl.etl.pudl_sqlite_io_manager(init_context) PudlSQLiteIOManager [source]#
Create a SQLiteManager dagster resource for the pudl database.
- pudl.etl.dataset_settings(init_context) pudl.settings.DatasetsSettings [source]#
Dagster resource for parameterizing PUDL ETL assets.
This resource allows us to specify the years we want to process for each datasource in the Dagit UI.
- pudl.etl.datastore(init_context) pudl.workspace.datastore.Datastore [source]#
Dagster resource to interact with Zenodo archives.
- pudl.etl.ferc_to_sqlite_settings(init_context) pudl.settings.FercToSqliteSettings [source]#
Dagster resource for parameterizing the
ferc_to_sqlite
graph.This resource allows us to specify the years we want to process for each datasource in the Dagit UI.
- class pudl.etl.EtlSettings[source]#
Bases:
pydantic.BaseSettings
Main settings validation class.
- ferc_to_sqlite_settings: FercToSqliteSettings#
- datasets: DatasetsSettings#
- classmethod from_yaml(path: str) EtlSettings [source]#
Create an EtlSettings instance from a yaml_file path.
- Parameters:
path – path to a yaml file; this could be remote.
- Returns:
An ETL settings object.
- pudl.etl.create_non_cems_selection(all_assets: list[dagster.AssetsDefinition]) dagster.AssetSelection [source]#
Create a selection of assets excluding CEMS and all downstream assets.
- Parameters:
all_assets – A list of asset definitions to remove CEMS assets from.
- Returns:
An asset selection with all_assets assets excluding CEMS assets.