pudl.settings

Module for validating pudl etl settings.

Module Contents

Classes

BaseModel

BaseModel with global configuration.

GenericDatasetSettings

An abstract pydantic model for generic datasets.

Eia860Settings

An immutable pydantic model to validate EIA860 settings.

GlueSettings

An immutable pydantic model to validate Glue settings.

EpaCemsSettings

An immutable pydantic nodel to validate EPA CEMS settings.

EiaSettings

An immutable pydantic model to validate EIA datasets settings.

DatasetsSettings

An immutable pydantic model to validate PUDL Dataset settings.

Ferc1ToSqliteSettings

An immutable pydantic nodel to validate Ferc1 to SQLite settings.

EtlSettings

Main settings validation class.

Functions

create_dataset_settings(name: str) → GenericDatasetSettings

Create a pydantic settings class for a dataset.

Attributes

Ferc1Settings

Eia923Settings

Eia860BaseSettings

EpaCemsBaseSettings

class pudl.settings.BaseModel[source]

Bases: pydantic.BaseModel

BaseModel with global configuration.

class Config[source]

Pydantic config.

allow_mutation = False[source]
extra = forbid[source]
class pudl.settings.GenericDatasetSettings[source]

Bases: BaseModel, abc.ABC

An abstract pydantic model for generic datasets.

Each dataset must specify working tables are partitions. A dataset can have an arbitrary number of partitioins.

tables :List[source]
property working_partitions(cls) Dict[str, List[Any]][source]

Abstract working_partitions property.

property working_tables(cls) List[source]

Abstract working_tables property.

validate_partitions(cls, partitions)[source]

Validate partitions are available.

validate_tables(cls, tables)[source]

Validate tables are available.

pudl.settings.create_dataset_settings(name: str) GenericDatasetSettings[source]

Create a pydantic settings class for a dataset.

This function dynamically creates subclasses of GenericDatasetSettings using information from the constants WORKING_PARTITIONS and PUDL_TABLES. The returned class can be subclassed to add additional fields or validators.

Parameters

name – the name of the dataset.

Returns

Subclass of GenericDatasetSettings

Return type

dataset_model

pudl.settings.Ferc1Settings[source]
pudl.settings.Eia923Settings[source]
pudl.settings.Eia860BaseSettings[source]
pudl.settings.EpaCemsBaseSettings[source]
class pudl.settings.Eia860Settings[source]

Bases: Eia860BaseSettings

An immutable pydantic model to validate EIA860 settings.

This model also check 860m settings.

Parameters
  • years – List of years to validate.

  • ClassVar[Dict[str (working_partitions) – working paritions.

  • Any]] – working paritions.

  • ClassVar[str] (eia860m_date) – The 860m year to date.

eia860m :bool = False[source]
eia860m_date :ClassVar[str][source]
check_860m_date(cls, eia860m: bool) bool[source]

Check 860m date year is exactly one year later than most recent working 860 year.

Parameters

eia860m – True if 860m is requested.

Returns

True if 860m is requested.

Return type

eia860m

Raises

ValueError – the 860m date is within 860 working years.

class pudl.settings.GlueSettings[source]

Bases: BaseModel

An immutable pydantic model to validate Glue settings.

Parameters
  • eia – Include eia in glue settings.

  • ferc1 – Include ferc1 in glue settings.

eia :bool = True[source]
ferc1 :bool = True[source]
class pudl.settings.EpaCemsSettings[source]

Bases: EpaCemsBaseSettings

An immutable pydantic nodel to validate EPA CEMS settings.

Parameters
  • states – List of states to validate.

  • years – List of years to validate.

allow_all_keyword(cls, states)[source]

Allow users to specify [‘all’] to get all states.

class pudl.settings.EiaSettings[source]

Bases: BaseModel

An immutable pydantic model to validate EIA datasets settings.

Parameters
  • eia860 – Immutable pydantic model to validate eia860 settings.

  • eia923 – Immutable pydantic model to validate eia923 settings.

eia860 :Eia860Settings[source]
eia923 :Eia923Settings[source]
default_load_all(cls, values)[source]

If no datasets are specified default to all.

Parameters

values (Dict[str, BaseModel]) – dataset settings.

Returns

dataset settings.

Return type

values (Dict[str, BaseModel])

check_eia_dependencies(cls, values)[source]

Make sure the dependencies between the eia datasets are satisfied.

Dependencies: * eia860 requires eia923.boiler_fuel_eia923 and eia923.generation_eia923. * eia923 requires eia860 for harvesting purposes.

Parameters

values (Dict[str, BaseModel]) – dataset settings.

Returns

dataset settings.

Return type

values (Dict[str, BaseModel])

class pudl.settings.DatasetsSettings[source]

Bases: BaseModel

An immutable pydantic model to validate PUDL Dataset settings.

Parameters
  • ferc1 – Immutable pydantic model to validate ferc1 settings.

  • eia – Immutable pydantic model to validate eia(860, 923) settings.

  • glue – Immutable pydantic model to validate glue settings.

  • epacems – Immutable pydantic model to validate epacems settings.

ferc1 :Ferc1Settings[source]
eia :EiaSettings[source]
glue :GlueSettings[source]
epacems :EpaCemsSettings[source]
default_load_all(cls, values)[source]

If no datasets are specified default to all.

Parameters

values (Dict[str, BaseModel]) – dataset settings.

Returns

dataset settings.

Return type

values (Dict[str, BaseModel])

add_glue_settings(cls, values)[source]

Add glue settings if ferc1 and eia data are both requested.

Parameters

values (Dict[str, BaseModel]) – dataset settings.

Returns

dataset settings.

Return type

values (Dict[str, BaseModel])

get_datasets(cls)[source]

Gets dictionary of dataset settings.

class pudl.settings.Ferc1ToSqliteSettings[source]

Bases: GenericDatasetSettings

An immutable pydantic nodel to validate Ferc1 to SQLite settings.

Parameters
  • tables – List of tables to validate.

  • years – List of years to validate.

working_partitions :ClassVar[source]
working_tables :ClassVar[source]
years :List[int][source]
tables :List[str][source]
refyear :ClassVar[int][source]
bad_cols :tuple = [][source]
class pudl.settings.EtlSettings[source]

Bases: pydantic.BaseSettings

Main settings validation class.

ferc1_to_sqlite_settings :Ferc1ToSqliteSettings[source]
datasets :DatasetsSettings[source]
name :str[source]
title :str[source]
description :str[source]
version :str[source]
pudl_in :str[source]
pudl_out :str[source]
classmethod from_yaml(cls, path: str)[source]

Create an EtlSettings instance from a yaml_file path.

Parameters

path – path to a yaml file.

Returns

etl settings object.

Return type

EtlSettings