pudl.transform.eia860#

Module to perform data cleaning functions on EIA860 data tables.

Module Contents#

Functions#

_core_eia860__ownership(→ pandas.DataFrame)

Pull and transform the ownership table.

_core_eia860__generators(→ pandas.DataFrame)

Pull and transform the generators table.

_core_eia860__plants(→ pandas.DataFrame)

Pull and transform the plants table.

_core_eia860__boiler_generator_assn(→ pandas.DataFrame)

Pull and transform the boilder generator association table.

_core_eia860__utilities(→ pandas.DataFrame)

Pull and transform the utilities table.

_core_eia860__boilers(...)

Pull and transform the boilers table.

_core_eia860__emissions_control_equipment(...)

Pull and transform the emissions control equipment table.

_core_eia860__boiler_emissions_control_equipment_assn(...)

Pull and transform the emissions control <> boiler ID link tables.

_core_eia860__boiler_cooling(→ pandas.DataFrame)

Pull and transform the EIA 860 boiler to cooler ID table.

_core_eia860__boiler_stack_flue(→ pandas.DataFrame)

Pull and transform the EIA 860 boiler to stack flue ID table.

Attributes#

logger

pudl.transform.eia860.logger[source]#
pudl.transform.eia860._core_eia860__ownership(raw_eia860__ownership: pandas.DataFrame) → pandas.DataFrame[source]#

Pull and transform the ownership table.

Transformations include:

  • Replace . values with NA.

  • Convert pre-2012 ownership percentages to proportions to match post-2012 reporting.

Parameters:

raw_eia860__ownership – The raw raw_eia860__ownership dataframe.

Returns:

Cleaned _core_eia860__ownership dataframe ready for harvesting.

pudl.transform.eia860._core_eia860__generators(raw_eia860__generator_proposed: pandas.DataFrame, raw_eia860__generator_existing: pandas.DataFrame, raw_eia860__generator_retired: pandas.DataFrame, raw_eia860__generator: pandas.DataFrame) → pandas.DataFrame[source]#

Pull and transform the generators table.

There are three tabs that the generator records come from (proposed, existing, retired). Pre 2009, the existing and retired data are lumped together under a single generator file with one tab. We pull each tab into one dataframe and include an operational_status to indicate which tab the record came from. We use operational_status to parse the pre 2009 files as well.

Transformations include:

  • Replace . values with NA.

  • Update operational_status_code to reflect plant status as either proposed, existing or retired.

  • Drop values with NA for plant and generator id.

  • Replace 0 values with NA where appropriate.

  • Convert Y/N/X values to boolean True/False.

  • Convert U/Unknown values to NA.

  • Map full spelling onto code values.

  • Create a fuel_type_code_pudl field that organizes fuel types into clean, distinguishable categories.

Parameters:

  • raw_eia860__generator_proposed – The raw raw_eia860__generator_proposed dataframe.

  • raw_eia860__generator_existing – The raw raw_eia860__generator_existing dataframe.

  • raw_eia860__generator_retired – The raw raw_eia860__generator_retired dataframe.

  • raw_eia860__generator – The raw raw_eia860__generator dataframe.

Returns:

Cleaned _core_eia860__generators dataframe ready for harvesting.

pudl.transform.eia860._core_eia860__plants(raw_eia860__plant: pandas.DataFrame) → pandas.DataFrame[source]#

Pull and transform the plants table.

Much of the static plant information is reported repeatedly, and scattered across several different pages of EIA 923. The data frame which this function uses is assembled from those many different pages, and passed in via the same dictionary of dataframes that all the other ingest functions use for uniformity.

Transformations include:

  • Replace . values with NA.

  • Homogenize spelling of county names.

  • Convert Y/N/X values to boolean True/False.

Parameters:

raw_eia860__plant – The raw raw_eia860__plant dataframe.

Returns:

Cleaned _core_eia860__plants dataframe ready for harvesting.

pudl.transform.eia860._core_eia860__boiler_generator_assn(raw_eia860__boiler_generator_assn: pandas.DataFrame) → pandas.DataFrame[source]#

Pull and transform the boilder generator association table.

Transformations include:

  • Drop non-data rows with EIA notes.

  • Drop duplicate rows.

Parameters:

raw_eia860__boiler_generator_assn (df) – Each entry in this dictionary of DataFrame objects corresponds to a page from the EIA860 form, as reported in the Excel spreadsheets they distribute.

Returns:

Cleaned _core_eia860__boiler_generator_assn dataframe ready for harvesting.

pudl.transform.eia860._core_eia860__utilities(raw_eia860__utility: pandas.DataFrame) → pandas.DataFrame[source]#

Pull and transform the utilities table.

Transformations include:

  • Replace . values with NA.

  • Fix typos in state abbreviations, convert to uppercase.

  • Drop address_3 field (all NA).

  • Combine phone number columns into one field and set values that don’t mimic real US phone numbers to NA.

  • Convert Y/N/X values to boolean True/False.

  • Map full spelling onto code values.

Parameters:

raw_eia860__utility – The raw raw_eia860__utility dataframe.

Returns:

Cleaned _core_eia860__utilities dataframe ready for harvesting.

pudl.transform.eia860._core_eia860__boilers(raw_eia860__emission_control_strategies, raw_eia860__boiler_info)[source]#

Pull and transform the boilers table.

Transformations include:

  • Replace . values with NA.

  • Convert Y/N/NA values to boolean True/False.

  • Combine month and year columns into date columns.

  • Add boiler manufacturer name column.

  • Convert pre-2012 efficiency percentages to proportions to match post-2012 reporting.

Parameters:

  • raw_eia860__emission_control_strategies (pandas.DataFrame) – DataFrame extracted from EIA forms earlier in the ETL process.

  • raw_eia860__boiler_info (pandas.DataFrame) – DataFrame extracted from EIA forms earlier in the ETL process.

Returns:

the transformed boilers table

Return type:

pandas.DataFrame

pudl.transform.eia860._core_eia860__emissions_control_equipment(raw_eia860__emissions_control_equipment: pandas.DataFrame) → pandas.DataFrame[source]#

Pull and transform the emissions control equipment table.

pudl.transform.eia860._core_eia860__boiler_emissions_control_equipment_assn(raw_eia860__boiler_so2: pandas.DataFrame, raw_eia860__boiler_mercury: pandas.DataFrame, raw_eia860__boiler_nox: pandas.DataFrame, raw_eia860__boiler_particulate: pandas.DataFrame) → pandas.DataFrame[source]#

Pull and transform the emissions control <> boiler ID link tables.

Parameters:

  • raw_eia860__boiler_so2 – Raw EIA 860 boiler to SO2 emission control equipment association table.

  • raw_eia860__boiler_mercury – Raw EIA 860 boiler to mercury emission control equipment association table.

  • raw_eia860__boiler_nox – Raw EIA 860 boiler to nox emission control equipment association table.

  • raw_eia860__boiler_particulate – Raw EIA 860 boiler to particulate emission control equipment association table.

  • raw_eia860__boiler_cooling – Raw EIA 860 boiler to cooling equipment association table.

  • raw_eia860__boiler_stack_flue – Raw EIA 860 boiler to stack flue equipment association table.

Returns:

A combination of all the emission control equipment association

tables.

Return type:

pd.DataFrame

pudl.transform.eia860._core_eia860__boiler_cooling(raw_eia860__boiler_cooling: pandas.DataFrame) → pandas.DataFrame[source]#

Pull and transform the EIA 860 boiler to cooler ID table.

Parameters:

raw_eia860__boiler_cooling – Raw EIA 860 boiler to cooler ID association table.

Returns:

A cleaned and normalized version of the EIA boiler to cooler ID

table.

Return type:

pd.DataFrame

pudl.transform.eia860._core_eia860__boiler_stack_flue(raw_eia860__boiler_stack_flue: pandas.DataFrame) → pandas.DataFrame[source]#

Pull and transform the EIA 860 boiler to stack flue ID table.

Parameters:

raw_eia860__boiler_stack_flue – Raw EIA 860 boiler to stack flue ID association table.

Returns:

A cleaned and normalized version of the EIA boiler to stack flue

ID table.

Return type:

pd.DataFrame