pudl.settings

Module for validating pudl etl settings.

Module Contents

Classes

XbrlFormNumber	Contains full list of supported FERC XBRL forms.
BaseModel	BaseModel with global configuration.
GenericDatasetSettings	An abstract pydantic model for generic datasets.
Ferc1Settings	An immutable pydantic model to validate Ferc1Settings.
Ferc714Settings	An immutable pydantic model to validate Ferc714Settings.
EpaCemsSettings	An immutable pydantic model to validate EPA CEMS settings.
Eia923Settings	An immutable pydantic model to validate EIA 923 settings.
Eia861Settings	An immutable pydantic model to validate EIA 861 settings.
Eia860Settings	An immutable pydantic model to validate EIA 860 settings.
GlueSettings	An immutable pydantic model to validate Glue settings.
EiaSettings	An immutable pydantic model to validate EIA datasets settings.
DatasetsSettings	An immutable pydantic model to validate PUDL Dataset settings.
Ferc1DbfToSqliteSettings	An immutable Pydantic model to validate FERC 1 to SQLite settings.
FercGenericXbrlToSqliteSettings	An immutable pydantic model to validate Ferc1 to SQLite settings.
Ferc1XbrlToSqliteSettings	An immutable pydantic model to validate Ferc1 to SQLite settings.
Ferc2XbrlToSqliteSettings	An immutable pydantic model to validate FERC from 2 XBRL to SQLite settings.
Ferc2DbfToSqliteSettings	An immutable Pydantic model to validate FERC 2 to SQLite settings.
Ferc6DbfToSqliteSettings	An immutable Pydantic model to validate FERC 6 to SQLite settings.
Ferc6XbrlToSqliteSettings	An immutable pydantic model to validate FERC from 6 XBRL to SQLite settings.
Ferc60DbfToSqliteSettings	An immutable Pydantic model to validate FERC 60 to SQLite settings.
Ferc60XbrlToSqliteSettings	An immutable pydantic model to validate FERC from 60 XBRL to SQLite settings.
Ferc714XbrlToSqliteSettings	An immutable pydantic model to validate FERC from 714 XBRL to SQLite settings.
FercToSqliteSettings	An immutable pydantic model to validate FERC XBRL to SQLite settings.
EtlSettings	Main settings validation class.

Functions

_convert_settings_to_dagster_config(→ None)	Convert dictionary of dataset settings to dagster config.
create_dagster_config(→ dict)	Create a dictionary of dagster config for the DatasetsSettings Class.
_make_doi_clickable(link)	Make a clickable DOI.

class pudl.settings.XbrlFormNumber(*args, **kwds)

Bases: enum.Enum

Contains full list of supported FERC XBRL forms.

FORM1 = 1

FORM2 = 2

FORM6 = 6

FORM60 = 60

FORM714 = 714

class pudl.settings.BaseModel

Bases: pydantic.BaseModel

BaseModel with global configuration.

class Config

Pydantic config.

allow_mutation = False

extra = 'forbid'

class pudl.settings.GenericDatasetSettings

Bases: BaseModel

An abstract pydantic model for generic datasets.

Each dataset must specify working partitions. A dataset can have an arbitrary number of partitions.

Parameters:

disabled – if true, skip processing this dataset.

property partitions: list[None | dict[str, str]]

Return list of dictionaries representing individual partitions.

Convert a list of partitions into a list of dictionaries of partitions. This is intended to be used to store partitions in a format that is easy to use with pd.json_normalize.

disabled: bool = False

validate_partitions(partitions)

Validate the requested data partitions.

Check that all the partitions defined in the working_partitions of the associated data_source (e.g. years or states) have been assigned in the definition of the class, and that the requested values are a subset of the allowable values defined by the data_source.

class pudl.settings.Ferc1Settings

Bases: GenericDatasetSettings

An immutable pydantic model to validate Ferc1Settings.

Parameters:

• data_source – DataSource metadata object

• years – list of years to validate.

property dbf_years

Return validated years for which DBF data is available.

property xbrl_years

Return validated years for which DBF data is available.

data_source: ClassVar[pudl.metadata.classes.DataSource]

years: list[int]

class pudl.settings.Ferc714Settings

Bases: GenericDatasetSettings

An immutable pydantic model to validate Ferc714Settings.

Parameters:

data_source – DataSource metadata object

data_source: ClassVar[pudl.metadata.classes.DataSource]

years: list[int]

class pudl.settings.EpaCemsSettings

Bases: GenericDatasetSettings

An immutable pydantic model to validate EPA CEMS settings.

Parameters:

• data_source – DataSource metadata object

• years – list of years to validate.

• states – list of states to validate.

• partition – Whether to output year-state partitioned Parquet files. If True, all available threads / CPUs will be used in parallel.

data_source: ClassVar[pudl.metadata.classes.DataSource]

years: list[int]

states: list[str]

allow_all_keyword(states)

Allow users to specify [‘all’] to get all states.

class pudl.settings.Eia923Settings

Bases: GenericDatasetSettings

An immutable pydantic model to validate EIA 923 settings.

Parameters:

• data_source – DataSource metadata object

• years – list of years to validate.

data_source: ClassVar[pudl.metadata.classes.DataSource]

years: list[int]

class pudl.settings.Eia861Settings

Bases: GenericDatasetSettings

An immutable pydantic model to validate EIA 861 settings.

Parameters:

• data_source – DataSource metadata object

• years – list of years to validate.

• transform_functions – list of transform functions to be applied to eia861

data_source: ClassVar[pudl.metadata.classes.DataSource]

years: list[int]

class pudl.settings.Eia860Settings

Bases: GenericDatasetSettings

An immutable pydantic model to validate EIA 860 settings.

This model also check 860m settings.

Parameters:

• data_source – DataSource metadata object

• years – list of years to validate.

• ClassVar[str] (eia860m_date) – The 860m year to date.

data_source: ClassVar[pudl.metadata.classes.DataSource]

eia860m_data_source: ClassVar[pudl.metadata.classes.DataSource]

eia860m_date: ClassVar[str]

years: list[int]

eia860m: bool = True

check_eia860m_date(eia860m: bool) → bool

Check 860m date-year is exactly one year after most recent working 860 year.

Parameters:

eia860m – True if 860m is requested.

Returns:

True if 860m is requested.

Return type:

eia860m

Raises:

ValueError – the 860m date is within 860 working years.

class pudl.settings.GlueSettings

Bases: BaseModel

An immutable pydantic model to validate Glue settings.

Parameters:

• eia – Include eia in glue settings.

• ferc1 – Include ferc1 in glue settings.

eia: bool = True

ferc1: bool = True

class pudl.settings.EiaSettings

Bases: BaseModel

An immutable pydantic model to validate EIA datasets settings.

Parameters:

• eia860 – Immutable pydantic model to validate eia860 settings.

• eia923 – Immutable pydantic model to validate eia923 settings.

eia860: Eia860Settings

eia861: Eia861Settings

eia923: Eia923Settings

default_load_all(values)

If no datasets are specified default to all.

Parameters:

values (Dict[str, BaseModel]) – dataset settings.

Returns:

dataset settings.

Return type:

values (Dict[str, BaseModel])

check_eia_dependencies(values)

Make sure the dependencies between the eia datasets are satisfied.

Dependencies: * eia923 requires eia860 for harvesting purposes.

Parameters:

values (Dict[str, BaseModel]) – dataset settings.

Returns:

dataset settings.

Return type:

values (Dict[str, BaseModel])

class pudl.settings.DatasetsSettings

Bases: BaseModel

An immutable pydantic model to validate PUDL Dataset settings.

Parameters:

• ferc1 – Immutable pydantic model to validate ferc1 settings.

• eia – Immutable pydantic model to validate eia(860, 923) settings.

• glue – Immutable pydantic model to validate glue settings.

• epacems – Immutable pydantic model to validate epacems settings.

eia: EiaSettings

epacems: EpaCemsSettings

ferc1: Ferc1Settings

ferc714: Ferc714Settings

glue: GlueSettings

default_load_all(values)

If no datasets are specified default to all.

Parameters:

values (Dict[str, BaseModel]) – dataset settings.

Returns:

dataset settings.

Return type:

values (Dict[str, BaseModel])

add_glue_settings(values)

Add glue settings if ferc1 and eia data are both requested.

Parameters:

values (Dict[str, BaseModel]) – dataset settings.

Returns:

dataset settings.

Return type:

values (Dict[str, BaseModel])

get_datasets()

Gets dictionary of dataset settings.

make_datasources_table(ds: pudl.workspace.datastore.Datastore) → pandas.DataFrame

Compile a table of dataset information.

There are three places we can look for information about a dataset: * the datastore (for DOIs, working partitions, etc) * the ETL settings (for partitions that are used in the ETL) * the DataSource info (which is stored within the ETL settings)

The ETL settings and the datastore have different levels of nesting - and therefore names for datasets. The nesting happens particularly with the EI data. There are three EIA datasets right now eia923, eia860 and eia860m. eia860m is a monthly update of a few tables in the larger eia860 dataset.

Parameters:

ds – An initalized PUDL Datastore from which the DOI’s for each raw input dataset can be obtained.

Returns:

a dataframe describing the partitions and DOI’s of each of the datasets in this settings object.

class pudl.settings.Ferc1DbfToSqliteSettings

Bases: GenericDatasetSettings

An immutable Pydantic model to validate FERC 1 to SQLite settings.

Parameters:

years – List of years to validate.

data_source: ClassVar[pudl.metadata.classes.DataSource]

years: list[int]

refyear: ClassVar[int]

class pudl.settings.FercGenericXbrlToSqliteSettings

Bases: pydantic.BaseSettings

An immutable pydantic model to validate Ferc1 to SQLite settings.

Parameters:

• taxonomy – URL of XBRL taxonomy used to create structure of SQLite DB.

• years – list of years to validate.

• disabled – if True, skip processing this dataset.

taxonomy: pydantic.AnyHttpUrl

years: list[int]

disabled: bool = False

class pudl.settings.Ferc1XbrlToSqliteSettings

Bases: FercGenericXbrlToSqliteSettings

An immutable pydantic model to validate Ferc1 to SQLite settings.

Parameters:

• taxonomy – URL of taxonomy used to .

• years – list of years to validate.

data_source: ClassVar[pudl.metadata.classes.DataSource]

years: list[int]

taxonomy: pydantic.AnyHttpUrl = 'https://eCollection.ferc.gov/taxonomy/form1/2022-01-01/form/form1/form-1_2022-01-01.xsd'

class pudl.settings.Ferc2XbrlToSqliteSettings

Bases: FercGenericXbrlToSqliteSettings

An immutable pydantic model to validate FERC from 2 XBRL to SQLite settings.

Parameters:

years – List of years to validate.

data_source: ClassVar[pudl.metadata.classes.DataSource]

years: list[int]

taxonomy: pydantic.AnyHttpUrl = 'https://eCollection.ferc.gov/taxonomy/form2/2022-01-01/form/form2/form-2_2022-01-01.xsd'

class pudl.settings.Ferc2DbfToSqliteSettings

Bases: GenericDatasetSettings

An immutable Pydantic model to validate FERC 2 to SQLite settings.

Parameters:

• years – List of years to validate.

• disabled – if True, skip processing this dataset.

data_source: ClassVar[pudl.metadata.classes.DataSource]

years: list[int]

refyear: ClassVar[int]

class pudl.settings.Ferc6DbfToSqliteSettings

Bases: GenericDatasetSettings

An immutable Pydantic model to validate FERC 6 to SQLite settings.

Parameters:

• years – List of years to validate.

• disabled – if True, skip processing this dataset.

data_source: ClassVar[pudl.metadata.classes.DataSource]

years: list[int]

disabled: bool = False

refyear: ClassVar[int]

class pudl.settings.Ferc6XbrlToSqliteSettings

Bases: FercGenericXbrlToSqliteSettings

An immutable pydantic model to validate FERC from 6 XBRL to SQLite settings.

Parameters:

years – List of years to validate.

data_source: ClassVar[pudl.metadata.classes.DataSource]

years: list[int]

taxonomy: pydantic.AnyHttpUrl = 'https://eCollection.ferc.gov/taxonomy/form6/2022-01-01/form/form6/form-6_2022-01-01.xsd'

class pudl.settings.Ferc60DbfToSqliteSettings

Bases: GenericDatasetSettings

An immutable Pydantic model to validate FERC 60 to SQLite settings.

Parameters:

• years – List of years to validate.

• disabled – if True, skip processing this dataset.

data_source: ClassVar[pudl.metadata.classes.DataSource]

years: list[int]

disabled: bool = False

refyear: ClassVar[int]

class pudl.settings.Ferc60XbrlToSqliteSettings

Bases: FercGenericXbrlToSqliteSettings

An immutable pydantic model to validate FERC from 60 XBRL to SQLite settings.

Parameters:

years – List of years to validate.

data_source: ClassVar[pudl.metadata.classes.DataSource]

years: list[int]

taxonomy: pydantic.AnyHttpUrl = 'https://eCollection.ferc.gov/taxonomy/form60/2022-01-01/form/form60/form-60_2022-01-01.xsd'

class pudl.settings.Ferc714XbrlToSqliteSettings

Bases: FercGenericXbrlToSqliteSettings

An immutable pydantic model to validate FERC from 714 XBRL to SQLite settings.

Parameters:

years – List of years to validate.

data_source: ClassVar[pudl.metadata.classes.DataSource]

years: list[int] = [2021]

taxonomy: pydantic.AnyHttpUrl = 'https://eCollection.ferc.gov/taxonomy/form714/2022-01-01/form/form714/form-714_2022-01-01.xsd'

class pudl.settings.FercToSqliteSettings

Bases: pydantic.BaseSettings

An immutable pydantic model to validate FERC XBRL to SQLite settings.

Parameters:

• ferc1_dbf_to_sqlite_settings – Settings for converting FERC 1 DBF data to SQLite.

• ferc1_xbrl_to_sqlite_settings – Settings for converting FERC 1 XBRL data to SQLite.

• other_xbrl_forms – List of non-FERC1 forms to convert from XBRL to SQLite.

ferc1_dbf_to_sqlite_settings: Ferc1DbfToSqliteSettings

ferc1_xbrl_to_sqlite_settings: Ferc1XbrlToSqliteSettings

ferc2_dbf_to_sqlite_settings: Ferc2DbfToSqliteSettings

ferc2_xbrl_to_sqlite_settings: Ferc2XbrlToSqliteSettings

ferc6_dbf_to_sqlite_settings: Ferc6DbfToSqliteSettings

ferc6_xbrl_to_sqlite_settings: Ferc6XbrlToSqliteSettings

ferc60_dbf_to_sqlite_settings: Ferc60DbfToSqliteSettings

ferc60_xbrl_to_sqlite_settings: Ferc60XbrlToSqliteSettings

ferc714_xbrl_to_sqlite_settings: Ferc714XbrlToSqliteSettings

default_load_all(values)

If no datasets are specified default to all.

Parameters:

values (Dict[str, BaseModel]) – dataset settings.

Returns:

dataset settings.

Return type:

values (Dict[str, BaseModel])

get_xbrl_dataset_settings(form_number: XbrlFormNumber) → FercGenericXbrlToSqliteSettings

Return a list with all requested FERC XBRL to SQLite datasets.

Parameters:

form_number – Get settings by FERC form number.

class pudl.settings.EtlSettings

Bases: pydantic.BaseSettings

Main settings validation class.

ferc_to_sqlite_settings: FercToSqliteSettings

datasets: DatasetsSettings

name: str

title: str

description: str

version: str

publish_destinations: list[str] = []

classmethod from_yaml(path: str) → EtlSettings

Create an EtlSettings instance from a yaml_file path.

Parameters:

path – path to a yaml file; this could be remote.

Returns:

An ETL settings object.

pudl.settings._convert_settings_to_dagster_config(d: dict) → None

Convert dictionary of dataset settings to dagster config.

For each partition parameter in a GenericDatasetSettings subclass, create a Noneable Dagster field with a default value of None. The GenericDatasetSettings subclasses will default to include all working paritions if the partition value is None. Get the value type so dagster can do some basic type checking in the UI.

Parameters:

d – dictionary of datasources and their parameters.

pudl.settings.create_dagster_config(settings: BaseModel) → dict

Create a dictionary of dagster config for the DatasetsSettings Class.

Returns:

A dictionary of dagster configuration.

pudl.settings._make_doi_clickable(link)

Make a clickable DOI.