pudl.settings#

Module for validating pudl etl settings.

Module Contents#

Classes#

XbrlFormNumber

Contains full list of supported FERC XBRL forms.

BaseModel

BaseModel with global configuration.

GenericDatasetSettings

An abstract pydantic model for generic datasets.

Ferc1Settings

An immutable pydantic model to validate Ferc1Settings.

Ferc714Settings

An immutable pydantic model to validate Ferc714Settings.

EpaCemsSettings

An immutable pydantic model to validate EPA CEMS settings.

Eia923Settings

An immutable pydantic model to validate EIA 923 settings.

Eia861Settings

An immutable pydantic model to validate EIA 861 settings.

Eia860Settings

An immutable pydantic model to validate EIA 860 settings.

GlueSettings

An immutable pydantic model to validate Glue settings.

EiaSettings

An immutable pydantic model to validate EIA datasets settings.

DatasetsSettings

An immutable pydantic model to validate PUDL Dataset settings.

Ferc1DbfToSqliteSettings

An immutable Pydantic model to validate FERC 1 to SQLite settings.

FercGenericXbrlToSqliteSettings

An immutable pydantic model to validate Ferc1 to SQLite settings.

Ferc1XbrlToSqliteSettings

An immutable pydantic model to validate Ferc1 to SQLite settings.

Ferc2XbrlToSqliteSettings

An immutable pydantic model to validate FERC from 2 XBRL to SQLite settings.

Ferc6XbrlToSqliteSettings

An immutable pydantic model to validate FERC from 6 XBRL to SQLite settings.

Ferc60XbrlToSqliteSettings

An immutable pydantic model to validate FERC from 60 XBRL to SQLite settings.

Ferc714XbrlToSqliteSettings

An immutable pydantic model to validate FERC from 714 XBRL to SQLite settings.

FercToSqliteSettings

An immutable pydantic model to validate FERC XBRL to SQLite settings.

EtlSettings

Main settings validation class.
class pudl.settings.XbrlFormNumber[source]#

Bases: enum.Enum

Contains full list of supported FERC XBRL forms.

FORM1 = 1[source]#
FORM2 = 2[source]#
FORM6 = 6[source]#
FORM60 = 60[source]#
FORM714 = 714[source]#
class pudl.settings.BaseModel[source]#

Bases: pydantic.BaseModel

BaseModel with global configuration.

class Config[source]#

Pydantic config.

allow_mutation = False[source]#
extra = 'forbid'[source]#
class pudl.settings.GenericDatasetSettings[source]#

Bases: BaseModel

An abstract pydantic model for generic datasets.

Each dataset must specify working tables and partitions. A dataset can have an arbitrary number of partitions.

tables: list[str][source]#
validate_partitions(partitions)[source]#

Validate the requested data partitions.

Check that all the partitions defined in the working_partitions of the associated data_source (e.g. years or states) have been assigned in the definition of the class, and that the requested values are a subset of the allowable values defined by the data_source.

validate_tables(tables)[source]#

Validate tables are available.

class pudl.settings.Ferc1Settings[source]#

Bases: GenericDatasetSettings

An immutable pydantic model to validate Ferc1Settings.

Parameters:

  • data_source – DataSource metadata object

  • years – list of years to validate.

  • tables – list of tables to validate.

property dbf_years[source]#

Return validated years for which DBF data is available.

property xbrl_years[source]#

Return validated years for which DBF data is available.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#
years: list[int][source]#
tables: list[str][source]#
validate_tables(tables)[source]#

Validate tables are available.

class pudl.settings.Ferc714Settings[source]#

Bases: GenericDatasetSettings

An immutable pydantic model to validate Ferc714Settings.

Parameters:

  • data_source – DataSource metadata object

  • tables – list of tables to validate.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#
tables: list[str][source]#
years: list[int][source]#
class pudl.settings.EpaCemsSettings[source]#

Bases: GenericDatasetSettings

An immutable pydantic model to validate EPA CEMS settings.

Parameters:

  • data_source – DataSource metadata object

  • years – list of years to validate.

  • states – list of states to validate.

  • tables – list of tables to validate.

  • partition – Whether to output year-state partitioned Parquet files. If True, all available threads / CPUs will be used in parallel.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#
years: list[int][source]#
states: list[str][source]#
tables: list[str][source]#
partition: bool = False[source]#
allow_all_keyword(states)[source]#

Allow users to specify [‘all’] to get all states.

class pudl.settings.Eia923Settings[source]#

Bases: GenericDatasetSettings

An immutable pydantic model to validate EIA 923 settings.

Parameters:

  • data_source – DataSource metadata object

  • years – list of years to validate.

  • tables – list of tables to validate.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#
years: list[int][source]#
tables: list[str][source]#
class pudl.settings.Eia861Settings[source]#

Bases: GenericDatasetSettings

An immutable pydantic model to validate EIA 861 settings.

Parameters:

  • data_source – DataSource metadata object

  • years – list of years to validate.

  • tables – list of tables to validate.

  • transform_functions – list of transform functions to be applied to eia861

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#
years: list[int][source]#
tables: list[str][source]#
transform_functions: list[str][source]#
generate_transform_functions(values)[source]#

Map tables to transform functions.

Parameters:

values – eia861 settings.

Returns:

eia861 settings.

Return type:

values

class pudl.settings.Eia860Settings[source]#

Bases: GenericDatasetSettings

An immutable pydantic model to validate EIA 860 settings.

This model also check 860m settings.

Parameters:

  • data_source – DataSource metadata object

  • years – list of years to validate.

  • tables – list of tables to validate.

  • ClassVar[str] (eia860m_date) – The 860m year to date.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#
eia860m_data_source: ClassVar[pudl.metadata.classes.DataSource][source]#
eia860m_date: ClassVar[str][source]#
years: list[int][source]#
tables: list[str][source]#
eia860m: bool = True[source]#
check_eia860m_date(eia860m: bool) → bool[source]#

Check 860m date-year is exactly one year after most recent working 860 year.

Parameters:

eia860m – True if 860m is requested.

Returns:

True if 860m is requested.

Return type:

eia860m

Raises:

ValueError – the 860m date is within 860 working years.

class pudl.settings.GlueSettings[source]#

Bases: BaseModel

An immutable pydantic model to validate Glue settings.

Parameters:

  • eia – Include eia in glue settings.

  • ferc1 – Include ferc1 in glue settings.

eia: bool = True[source]#
ferc1: bool = True[source]#
class pudl.settings.EiaSettings[source]#

Bases: BaseModel

An immutable pydantic model to validate EIA datasets settings.

Parameters:

  • eia860 – Immutable pydantic model to validate eia860 settings.

  • eia923 – Immutable pydantic model to validate eia923 settings.

eia860: Eia860Settings[source]#
eia923: Eia923Settings[source]#
default_load_all(values)[source]#

If no datasets are specified default to all.

Parameters:

values (Dict[str, BaseModel]) – dataset settings.

Returns:

dataset settings.

Return type:

values (Dict[str, BaseModel])

check_eia_dependencies(values)[source]#

Make sure the dependencies between the eia datasets are satisfied.

Dependencies: * eia860 requires eia923.boiler_fuel_eia923 and eia923.generation_eia923. * eia923 requires eia860 for harvesting purposes.

Parameters:

values (Dict[str, BaseModel]) – dataset settings.

Returns:

dataset settings.

Return type:

values (Dict[str, BaseModel])

class pudl.settings.DatasetsSettings[source]#

Bases: BaseModel

An immutable pydantic model to validate PUDL Dataset settings.

Parameters:

  • ferc1 – Immutable pydantic model to validate ferc1 settings.

  • eia – Immutable pydantic model to validate eia(860, 923) settings.

  • glue – Immutable pydantic model to validate glue settings.

  • epacems – Immutable pydantic model to validate epacems settings.

ferc1: Ferc1Settings[source]#
eia: EiaSettings[source]#
glue: GlueSettings[source]#
epacems: EpaCemsSettings[source]#
default_load_all(values)[source]#

If no datasets are specified default to all.

Parameters:

values (Dict[str, BaseModel]) – dataset settings.

Returns:

dataset settings.

Return type:

values (Dict[str, BaseModel])

add_glue_settings(values)[source]#

Add glue settings if ferc1 and eia data are both requested.

Parameters:

values (Dict[str, BaseModel]) – dataset settings.

Returns:

dataset settings.

Return type:

values (Dict[str, BaseModel])

get_datasets()[source]#

Gets dictionary of dataset settings.

class pudl.settings.Ferc1DbfToSqliteSettings[source]#

Bases: GenericDatasetSettings

An immutable Pydantic model to validate FERC 1 to SQLite settings.

Parameters:

  • tables – List of tables to validate.

  • years – List of years to validate.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#
years: list[int][source]#
tables: list[str][source]#
refyear: ClassVar[int][source]#
validate_tables(tables)[source]#

Validate tables.

class pudl.settings.FercGenericXbrlToSqliteSettings[source]#

Bases: pydantic.BaseSettings

An immutable pydantic model to validate Ferc1 to SQLite settings.

Parameters:

  • taxonomy – URL of XBRL taxonomy used to create structure of SQLite DB.

  • tables – list of tables to validate.

  • years – list of years to validate.

taxonomy: pydantic.AnyHttpUrl[source]#
tables: list[int] | None[source]#
years: list[int][source]#
class pudl.settings.Ferc1XbrlToSqliteSettings[source]#

Bases: FercGenericXbrlToSqliteSettings

An immutable pydantic model to validate Ferc1 to SQLite settings.

Parameters:

  • taxonomy – URL of taxonomy used to .

  • years – list of years to validate.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#
years: list[int][source]#
taxonomy: pydantic.AnyHttpUrl = 'https://eCollection.ferc.gov/taxonomy/form1/2022-01-01/form/form1/form-1_2022-01-01.xsd'[source]#
tables: list[str][source]#
validate_tables(tables)[source]#

Validate tables.

class pudl.settings.Ferc2XbrlToSqliteSettings[source]#

Bases: FercGenericXbrlToSqliteSettings

An immutable pydantic model to validate FERC from 2 XBRL to SQLite settings.

Parameters:

years – List of years to validate.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#
years: list[int][source]#
taxonomy: pydantic.AnyHttpUrl = 'https://eCollection.ferc.gov/taxonomy/form2/2022-01-01/form/form2/form-2_2022-01-01.xsd'[source]#
class pudl.settings.Ferc6XbrlToSqliteSettings[source]#

Bases: FercGenericXbrlToSqliteSettings

An immutable pydantic model to validate FERC from 6 XBRL to SQLite settings.

Parameters:

years – List of years to validate.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#
years: list[int][source]#
taxonomy: pydantic.AnyHttpUrl = 'https://eCollection.ferc.gov/taxonomy/form6/2022-01-01/form/form6/form-6_2022-01-01.xsd'[source]#
class pudl.settings.Ferc60XbrlToSqliteSettings[source]#

Bases: FercGenericXbrlToSqliteSettings

An immutable pydantic model to validate FERC from 60 XBRL to SQLite settings.

Parameters:

years – List of years to validate.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#
years: list[int][source]#
taxonomy: pydantic.AnyHttpUrl = 'https://eCollection.ferc.gov/taxonomy/form60/2022-01-01/form/form60/form-60_2022-01-01.xsd'[source]#
class pudl.settings.Ferc714XbrlToSqliteSettings[source]#

Bases: FercGenericXbrlToSqliteSettings

An immutable pydantic model to validate FERC from 714 XBRL to SQLite settings.

Parameters:

years – List of years to validate.

data_source: ClassVar[pudl.metadata.classes.DataSource][source]#
years: list[int] = [2021][source]#
taxonomy: pydantic.AnyHttpUrl = 'https://eCollection.ferc.gov/taxonomy/form714/2022-01-01/form/form714/form-714_2022-01-01.xsd'[source]#
class pudl.settings.FercToSqliteSettings[source]#

Bases: pydantic.BaseSettings

An immutable pydantic model to validate FERC XBRL to SQLite settings.

Parameters:

  • ferc1_dbf_to_sqlite_settings – Settings for converting FERC 1 DBF data to SQLite.

  • ferc1_xbrl_to_sqlite_settings – Settings for converting FERC 1 XBRL data to SQLite.

  • other_xbrl_forms – List of non-FERC1 forms to convert from XBRL to SQLite.

ferc1_dbf_to_sqlite_settings: Ferc1DbfToSqliteSettings[source]#
ferc1_xbrl_to_sqlite_settings: Ferc1XbrlToSqliteSettings[source]#
ferc2_xbrl_to_sqlite_settings: Ferc2XbrlToSqliteSettings[source]#
ferc6_xbrl_to_sqlite_settings: Ferc6XbrlToSqliteSettings[source]#
ferc60_xbrl_to_sqlite_settings: Ferc60XbrlToSqliteSettings[source]#
ferc714_xbrl_to_sqlite_settings: Ferc714XbrlToSqliteSettings[source]#
get_xbrl_dataset_settings(form_number: XbrlFormNumber) → FercGenericXbrlToSqliteSettings[source]#

Return a list with all requested FERC XBRL to SQLite datasets.

Parameters:

form_number – Get settings by FERC form number.

class pudl.settings.EtlSettings[source]#

Bases: pydantic.BaseSettings

Main settings validation class.

ferc_to_sqlite_settings: FercToSqliteSettings[source]#
datasets: DatasetsSettings[source]#
name: str[source]#
title: str[source]#
description: str[source]#
version: str[source]#
pudl_in: str[source]#
pudl_out: str[source]#
classmethod from_yaml(path: str) → EtlSettings[source]#

Create an EtlSettings instance from a yaml_file path.

Parameters:

path – path to a yaml file.

Returns:

An ETL settings object.