pudl.transform.ferc1
#
Classes & functions to process FERC Form 1 data before loading into the PUDL DB.
Note that many of the classes/objects here inherit from/are instances of classes defined
in pudl.transform.classes
. Their design and relationships to each other are
documented in that module.
See pudl.transform.params.ferc1
for the values that parameterize many of these
transformations.
Module Contents#
Classes#
Enumeration of allowed FERC 1 raw data sources. |
|
Enumeration of the allowed FERC 1 table IDs. |
|
Dictionaries for renaming either XBRL or DBF derived FERC 1 columns. |
|
Parameters for converting a wide table to a tidy table with value types. |
|
Parameters for converting either or both XBRL and DBF table from wide to tidy. |
|
Parameters for merging in XBRL metadata. |
|
Parameter for dropping duplicate DBF rows. |
|
Parameters for aligning DBF row numbers with metadata from mannual maps. |
|
Parameters for |
|
Parameters for |
|
Parameters for |
|
A model defining what TransformParams are allowed for FERC Form 1. |
|
An abstract class defining methods common to many FERC Form 1 tables. |
|
A table transformer specific to the fuel_ferc1 table. |
|
Transformer class for the plants_steam_ferc1 table. |
|
A table transformer specific to the plants_hydro_ferc1 table. |
|
Transformer class for plants_pumped_storage_ferc1 table. |
|
Transformer class for purchased_power_ferc1 table. |
|
A transformer for the plant_in_service_ferc1 table. |
|
A table transformer specific to the plants_small_ferc1 table. |
|
A table transformer for the transmission_statistics_ferc1 table. |
|
Transformer class for electric_energy_sources_ferc1 table. |
|
Transformer class for electric_energy_dispositions_ferc1 table. |
|
Transformer class for utility_plant_summary_ferc1 table. |
|
Transformer class for balance_sheet_liabilities_ferc1 table. |
|
Transformer class for balance_sheet_assets_ferc1 table. |
|
Transformer class for the income_statement_ferc1 table. |
|
Transformer class for retained_earnings_ferc1 table. |
|
Transformer class for depreciation_amortization_summary_ferc1 table. |
|
Transformer class for electric_plant_depreciation_changes_ferc1 table. |
|
Transformer for electric_plant_depreciation_functional_ferc1 table. |
|
Transformer class for electric_operating_expenses_ferc1 table. |
|
Transformer class for electric_operating_revenues_ferc1 table. |
|
Transform class for cash_flow_ferc1 table. |
|
Transform class for electricity_sales_by_rate_schedule_ferc1 table. |
|
Transformer class for other_regulatory_liabilities_ferc1 table. |
Functions#
|
Reshape wide tables with FERC account columns to tidy format. |
|
Merge metadata based on params. |
|
Drop duplicate DBF rows if duplicates have indentical data or one row has nulls. |
|
Rename the xbrl_factoid column after |
|
Select DBF rows with values listed or found in XBRL in a categorical-like column. |
Turn start year end year rows into columns for each value type. |
|
|
Combine axis columns from squished XBRL tables into one column with no NAs. |
|
Identify DBF rows that need to be mapped to XBRL columns. |
|
Regenerate the FERC 1 DBF+XBRL glue while retaining existing mappings. |
|
Read the manually compiled DBF row to XBRL column mapping for a given table. |
|
Forward-fill missing years in the minimal, manually compiled DBF to XBRL mapping. |
|
Get a list of all XBRL data columns appearing in a given XBRL table. |
|
Create an asset that pulls in raw ferc Form 1 assets and applies transformations. |
Create a list of transformed FERC Form 1 assets. |
|
|
Create the clean plants_steam_ferc1 table. |
Attributes#
- class pudl.transform.ferc1.SourceFerc1(*args, **kwds)[source]#
Bases:
enum.Enum
Enumeration of allowed FERC 1 raw data sources.
- class pudl.transform.ferc1.TableIdFerc1(*args, **kwds)[source]#
Bases:
enum.Enum
Enumeration of the allowed FERC 1 table IDs.
Hard coding this doesn’t seem ideal. Somehow it should be either defined in the context of the Package, the Ferc1Settings, an etl_group, or DataSource. All of the table transformers associated with a given data source should have a table_id that’s from that data source’s subset of the database. Where should this really happen? Alternatively, the allowable values could be derived from the structure of the Package. But this works for now.
- class pudl.transform.ferc1.RenameColumnsFerc1[source]#
Bases:
pudl.transform.classes.TransformParams
Dictionaries for renaming either XBRL or DBF derived FERC 1 columns.
This is FERC 1 specific, because we need to store both DBF and XBRL rename dictionaires separately. Note that this parameter model does not have its own unique transform function. Like the generic
pudl.transform.classes.RenameColumns
it depends on the build inpd.rename()
method, which is called with the values DBF or XBRL parameters depending on the context.Potential parameters validations that could be implemented
Validate that all keys appear in the original dbf/xbrl sources. This has to be true, but right now we don’t have stored metadata enumerating all of the columns that exist in the raw data, so we don’t have anything to check against. Implement once when we have schemas defined for after the extract step.
Validate all values appear in PUDL tables, and all expected PUDL names are mapped. Actually we can’t require that the rename values appear in the PUDL tables, because there will be cases in which the original column gets dropped or modified, e.g. in the case of unit conversions with a column rename.
- duration_xbrl: pudl.transform.classes.RenameColumns[source]#
- instant_xbrl: pudl.transform.classes.RenameColumns[source]#
- class pudl.transform.ferc1.WideToTidy[source]#
Bases:
pudl.transform.classes.TransformParams
Parameters for converting a wide table to a tidy table with value types.
- value_types: list[str] | None[source]#
List of names of value types that will end up being the column names.
Some of the FERC tables have multiple data types spread across many different categories. In the input dataframe given to
wide_to_tidy()
, the value types must be the suffixes of the column names. If the table does not natively have the pattern of “{to-be stacked category}_{value_type}”, rename the columns using arename_columns.duration_xbrl
,rename_columns.instant_xbrl
orrename_columns.dbf
parameter which will be employed inprocess_duration_xbrl()
,process_instant_xbrl()
orprocess_dbf()
.
- expected_drop_cols: int = 0[source]#
The number of columns that are expected to be dropped.
wide_to_tidy_xbrl()
will generate a regex pattern assuming thevalue_types
are the column name’s suffixes. If a column does not conform to that pattern, it will be filtered out. This is helpful for us to not include a bunch of columns from the input dataframe incorrectly included in the stacking process. We could enumerate every column that we want to drop, but this could be tedious and potentially error prone. But this does mean that if a column is incorrectly named - or barely missing the pattern, it will be dropped. This parameter enables us to lock the number of expected columns. If the dropped columns are a different number, an error will be raised.
- class pudl.transform.ferc1.WideToTidySourceFerc1[source]#
Bases:
pudl.transform.classes.TransformParams
Parameters for converting either or both XBRL and DBF table from wide to tidy.
- xbrl: WideToTidy | list[WideToTidy][source]#
- dbf: WideToTidy | list[WideToTidy][source]#
- pudl.transform.ferc1.wide_to_tidy(df: pandas.DataFrame, params: WideToTidy) pandas.DataFrame [source]#
Reshape wide tables with FERC account columns to tidy format.
The XBRL table coming into this method could contain all the data from both the instant and duration tables in a wide format – with one column for every combination of value type (e.g. additions, ending_balance) and value category, which means ~500 columns for some tables.
We tidy this into a long table with one column for each of the value types in
params.value_types
and a new column namedxbrl_factoid
that contains categories that were previously the XBRL column name stems.This allows aggregations of multiple
xbrl_factoid
categories in a columnar fashion such as aggregation across groups of rows to total up various hierarchical accounting categories (hydraulic turbines -> hydraulic production plant -> all production plant -> all electric utility plant) though the categorical columns required for that aggregation are added later.For table that have a internal relationship between the values in the
params.value_types
, such as the plant_in_service_ferc1 table, this also enables aggregation across columns to calculate the ending balance based on the starting balance and all of the reported changes.
- class pudl.transform.ferc1.MergeXbrlMetadata[source]#
Bases:
pudl.transform.classes.TransformParams
Parameters for merging in XBRL metadata.
- rename_columns: dict[str, str][source]#
Dictionary to rename columns in the normalized metadata before merging.
This dictionary will be passed as
pd.DataFrame.rename()
columns
parameter.
- on: str | None[source]#
Column name to merge on in
merge_xbrl_metadata()
.
- pudl.transform.ferc1.merge_xbrl_metadata(df: pandas.DataFrame, xbrl_metadata: pandas.DataFrame, params: MergeXbrlMetadata) pandas.DataFrame [source]#
Merge metadata based on params.
- class pudl.transform.ferc1.DropDuplicateRowsDbf[source]#
Bases:
pudl.transform.classes.TransformParams
Parameter for dropping duplicate DBF rows.
- table_name: TableIdFerc1 | None[source]#
Name of table used to grab primary keys of PUDL table to check for duplicates.
- pudl.transform.ferc1.drop_duplicate_rows_dbf(df: pandas.DataFrame, params: DropDuplicateRowsDbf, return_dupes_w_unique_data: bool = False) pandas.DataFrame [source]#
Drop duplicate DBF rows if duplicates have indentical data or one row has nulls.
There are several instances of the DBF data reporting the same value on multiple rows. This function checks to see if all of the duplicate values that have the same primary keys have reported the same data or have records with null data in any of the data columns while the other record has complete data. If the duplicates have no unique data, the duplicates are dropped with
keep="first"
. If any duplicates do not contain the same data or half null data, an assertion will be raised.- Parameters:
df – DBF table containing PUDL primary key columns
params – an instance of
DropDuplicateRowsDbf
return_dupes_w_unique_data – Boolean flag used for debuging only which returns the duplicates which contain actually unique data instead of raising assertion. Default is False.
- class pudl.transform.ferc1.AlignRowNumbersDbf[source]#
Bases:
pudl.transform.classes.TransformParams
Parameters for aligning DBF row numbers with metadata from mannual maps.
- pudl.transform.ferc1.align_row_numbers_dbf(df: pandas.DataFrame, params: AlignRowNumbersDbf) pandas.DataFrame [source]#
Rename the xbrl_factoid column after
align_row_numbers_dbf()
.
- class pudl.transform.ferc1.SelectDbfRowsByCategory[source]#
Bases:
pudl.transform.classes.TransformParams
Parameters for
select_dbf_rows_by_category()
.- select_by_xbrl_categories: bool = False[source]#
Boolean flag to indicate whether or not to use the categories in the XBRL table.
If True,
select_dbf_rows_by_category()
will find the list of categories that exist in the passed inprocessed_xbrl
to select by.
- additional_categories: list[str] = [][source]#
List of additional categories to select by.
If
select_by_xbrl_categories
isTrue
, these categories will be added to the XBRL categories and both will be used to select rows from the DBF data. Ifselect_by_xbrl_categories
isFalse
, only the “additional” categories will be the used to select rows from the DBF data.
- len_expected_categories_to_drop: int = 0[source]#
Number of categories that are expected to be dropped from the DBF data.
This is here to ensure no unexpected manipulations to the categories have occured. A warning will be flagged if this number is different than the number of categories that are being dropped.
- pudl.transform.ferc1.select_dbf_rows_by_category(processed_dbf: pandas.DataFrame, processed_xbrl: pandas.DataFrame, params: SelectDbfRowsByCategory) pandas.DataFrame [source]#
Select DBF rows with values listed or found in XBRL in a categorical-like column.
The XBRL data often breaks out sub-sections of DBF tables into their own table. These breakout tables are often messy, unstructured portions of a particular schedule or page on the FERC1 PDF. We often want to preserve some of the ways the XBRL data is segmented so we need to be able to select only portions of the DBF table to be concatenated with the XBRL data.
In mapping DBF data to XBRL data for the tables that rely on their
row_number
we map each row to its correspondingxbrl_factoid
. The standard use of this transformer is to use thecolumn_name
that corresponds to thexbrl_factoid
that was merged into the DBF data viaalign_row_numbers_dbf()
and was converted into a column in the XBRL data viawide_to_tidy()
.Note: Often, the unstructured portion of the DBF table that (possibly) sums up into a single value in structured data has the same
xbrl_factoid
name in the XBRL tables. By convention, we are employing a pattern in thedbf_to_xbrl.csv
map that involves adding an_unstructed
suffix to the rows that correspond to the unstructured portion of the table. This enables a simple selection of the structured part of the table. When processing the unstructured table, you can either rename the XBRL data’s factoid name to include an_unstructed
suffix or you can specify the categories with_unstructed
suffixes using theadditional_categories
parameter.
- class pudl.transform.ferc1.UnstackBalancesToReportYearInstantXbrl[source]#
Bases:
pudl.transform.classes.TransformParams
Parameters for
unstack_balances_to_report_year_instant_xbrl()
.
- pudl.transform.ferc1.unstack_balances_to_report_year_instant_xbrl(df: pandas.DataFrame, params: UnstackBalancesToReportYearInstantXbrl, primary_key_cols: list[str]) pandas.DataFrame [source]#
Turn start year end year rows into columns for each value type.
Called in
Ferc1AbstractTableTransformer.process_instant_xbrl()
.Some instant tables report year-end data, with their datestamps in different years, but we want year-start and year-end data within a single report_year (which is equivalent) stored in two separate columns for compatibility with the DBF data.
This function unstacks that table and adds the suffixes
_starting_balance
and_ending_balance
to each of the columns. These can then be used asvalue_types
inwide_to_tidy()
to normalize the table.There are two checks in place:
First, it will make sure that there are not duplicate entries for a single year + other primary key fields. Ex: a row for 2020-12-31 and 2020-06-30 for entitiy_id X means that the data isn’t annually unique. We could just drop these mid-year values, but we might want to keep them or at least check that there is no funny business with the data.
We also check that there are no mid-year dates at all. If an entity reports a value from the middle of the year, we can’t identify it as a start/end of year value.
- Params:
- primary_key_cols: The columns that should be used to check for duplicated data,
and also for unstacking the balance – these are set to be the index before unstack is called. These are typically set by the wrapping method and generated automatically based on other class transformation parameters via
Ferc1AbstractTableTransformer.source_table_primary_key()
.
- class pudl.transform.ferc1.CombineAxisColumnsXbrl[source]#
Bases:
pudl.transform.classes.TransformParams
Parameters for
combine_axis_columns_xbrl()
.
- pudl.transform.ferc1.combine_axis_columns_xbrl(df: pandas.DataFrame, params: CombineAxisColumnsXbrl) pandas.DataFrame [source]#
Combine axis columns from squished XBRL tables into one column with no NAs.
Called in
Ferc1AbstractTableTransformer.process_xbrl()
.There are instances (ex: sales_by_rate_schedule_ferc1) where the DBF table is equal to several concatenated XBRL tables. These XBRL tables are extracted together with the function
extract_xbrl_concat()
. Once combined, we need to deal with their axis columns.We use the axis columns (the primary key for the raw XBRL tables) in the creation of
record_id``s for each of the rows. If each of the concatinated XBRL tables has the same axis column name then there's no need to fret. However, if the columns have slightly different names (ex: ``residential_sales_axis
vs.industrial_sales_axis
), we’ll need to combine them. We combine them to get rid of NA values which aren’t allowed in primary keys. Otherwise it would look like this:residential_sales_axis
industrial_sales_axis
value1
NA
value2
NA
NA
valueA
NA
valueB
vs. this:
sales_axis
value1
value2
valueA
valueB
- class pudl.transform.ferc1.Ferc1TableTransformParams[source]#
Bases:
pudl.transform.classes.TableTransformParams
A model defining what TransformParams are allowed for FERC Form 1.
This adds additional parameter models beyond the ones inherited from the
pudl.transform.classes.AbstractTableTransformer
class.- rename_columns_ferc1: RenameColumnsFerc1[source]#
- wide_to_tidy: WideToTidySourceFerc1[source]#
- merge_xbrl_metadata: MergeXbrlMetadata[source]#
- align_row_numbers_dbf: AlignRowNumbersDbf[source]#
- drop_duplicate_rows_dbf: DropDuplicateRowsDbf[source]#
- select_dbf_rows_by_category: SelectDbfRowsByCategory[source]#
- unstack_balances_to_report_year_instant_xbrl: UnstackBalancesToReportYearInstantXbrl[source]#
- combine_axis_columns_xbrl: CombineAxisColumnsXbrl[source]#
- pudl.transform.ferc1.get_ferc1_dbf_rows_to_map(ferc1_engine: sqlalchemy.engine.Engine) pandas.DataFrame [source]#
Identify DBF rows that need to be mapped to XBRL columns.
Select all records in the
f1_row_lit_tbl
where the row literal associated with a given combination of table and row number is different from the preceeding year. This is the smallest set of records which we can use to reproduce the whole table by expanding the time series to include all years, and forward filling the row literals.
- pudl.transform.ferc1.update_dbf_to_xbrl_map(ferc1_engine: sqlalchemy.engine.Engine) pandas.DataFrame [source]#
Regenerate the FERC 1 DBF+XBRL glue while retaining existing mappings.
Reads all rows that need to be mapped out of the
f1_row_lit_tbl
and appends columns containing any previously mapped values, returning the resulting dataframe.
- pudl.transform.ferc1.read_dbf_to_xbrl_map(dbf_table_names: list[str]) pandas.DataFrame [source]#
Read the manually compiled DBF row to XBRL column mapping for a given table.
- Parameters:
dbf_table_name – The original name of the table in the FERC Form 1 DBF database whose mapping to the XBRL data you want to extract. for example
f1_plant_in_srvce
.- Returns:
DataFrame with columns
[sched_table_name, report_year, row_number, row_type, xbrl_factoid]
- pudl.transform.ferc1.fill_dbf_to_xbrl_map(df: pandas.DataFrame, dbf_years: list[int] | None = None) pandas.DataFrame [source]#
Forward-fill missing years in the minimal, manually compiled DBF to XBRL mapping.
The relationship between a DBF row and XBRL column/fact/entity/whatever is mostly consistent from year to year. To minimize the amount of manual mapping work we have to do, we only map the years in which the relationship changes. In the end we do need a complete correspondence for all years though, and this function uses the minimal information we’ve compiled to fill in all the gaps, producing a complete mapping across all requested years.
One complication is that we need to explicitly indicate which DBF rows have headers in them (which don’t exist in XBRL), to differentiate them from null values in the exhaustive index we create below. We set a
HEADER_ROW
sentinel value so we can distinguish between two different reasons that we might find NULL values in thexbrl_factoid
field:It’s NULL because it’s between two valid mapped values (the NULL was created in our filling of the time series) and should thus be filled in, or
It’s NULL because it was a header row in the DBF data, which means it should NOT be filled in. Without the
HEADER_ROW
value, when a row number from year X becomes associated with a non-header row in year X+1 the ffill will keep right on filling, associating all of the new header rows with the value ofxbrl_factoid
that was associated with the old row number.
- Parameters:
df – A dataframe containing a DBF row to XBRL mapping for a single FERC 1 DBF table.
dbf_years – The list of years that should have their DBF row to XBRL mapping filled in. This defaults to all available years of DBF data for FERC 1. In general this parameter should only be set to a non-default value for testing purposes.
- Returns:
A complete mapping of DBF row number to XBRL columns for all years of data within a single FERC 1 DBF table. Has columns of
[report_year, row_number, xbrl_factoid]
- pudl.transform.ferc1.get_data_cols_raw_xbrl(raw_xbrl_instant: pandas.DataFrame, raw_xbrl_duration: pandas.DataFrame) list[str] [source]#
Get a list of all XBRL data columns appearing in a given XBRL table.
- Returns:
A list of all the data columns found in the original XBRL DB that correspond to the given PUDL table. Includes columns from both the instant and duration tables but excludes structural columns that appear in all XBRL tables.
- class pudl.transform.ferc1.Ferc1AbstractTableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
pudl.transform.classes.AbstractTableTransformer
An abstract class defining methods common to many FERC Form 1 tables.
This subclass remains abstract because it does not define transform_main(), which is always going to be table-specific.
Methods that only apply to XBRL data should end with _xbrl
Methods that only apply to DBF data should end with _dbf
- table_id: TableIdFerc1[source]#
- has_unique_record_ids: bool = True[source]#
True if each record in the transformed table corresponds to one input record.
For tables that have been transformed from wide-to-tidy format, or undergone other kinds of reshaping, there is not a simple one-to-one relationship between input and output records, and so we should not expect record IDs to be unique. In those cases they serve only a forensic purpose, telling us where to find the original source of the transformed data.
- xbrl_metadata: pandas.DataFrame[source]#
Dataframe combining XBRL metadata for both instant and duration table columns.
- transform_start(raw_dbf: pandas.DataFrame, raw_xbrl_instant: pandas.DataFrame, raw_xbrl_duration: pandas.DataFrame) pandas.DataFrame [source]#
Process the raw data until the XBRL and DBF inputs have been unified.
- transform_main(df: pandas.DataFrame) pandas.DataFrame [source]#
Generic FERC1 main table transformer.
- Params:
df: Pre-processed, concatenated XBRL and DBF data.
- Returns:
A single transformed table concatenating multiple years of cleaned data derived from the raw DBF and/or XBRL inputs.
- transform_end(df: pandas.DataFrame) pandas.DataFrame [source]#
Standardized final cleanup after the transformations are done.
Enforces dataframe schema. Checks for empty dataframes and null columns.
- select_dbf_rows_by_category(processed_dbf: pandas.DataFrame, processed_xbrl: pandas.DataFrame, params: SelectDbfRowsByCategory | None = None) pandas.DataFrame [source]#
Wrapper method for
select_dbf_rows_by_category()
.
- process_xbrl_metadata(xbrl_metadata_json) pandas.DataFrame [source]#
Normalize the XBRL JSON metadata, turning it into a dataframe.
This process concatenates and deduplicates the metadata which is associated with the instant and duration tables, since the metadata is only combined with the data after the instant and duration (and DBF) tables have been merged. This happens in
Ferc1AbstractTableTransformer.merge_xbrl_metadata()
.
- merge_xbrl_metadata(df: pandas.DataFrame, params: MergeXbrlMetadata | None = None) pandas.DataFrame [source]#
Combine XBRL-derived metadata with the data it pertains to.
While the metadata we’re using to annotate the data comes from the more recent XBRL data, it applies generally to all the historical DBF data as well! This method reads the normalized metadata out of an attribute.
- align_row_numbers_dbf(df: pandas.DataFrame, params: AlignRowNumbersDbf | None = None) pandas.DataFrame [source]#
Align historical FERC1 DBF row numbers with XBRL account IDs.
Additional Parameterization TBD with additional experience. See: https://github.com/catalyst-cooperative/pudl/issues/2012
- drop_duplicate_rows_dbf(df: pandas.DataFrame, params: DropDuplicateRowsDbf | None = None) pandas.DataFrame [source]#
Drop the DBF rows where the PKs and data columns are duplicated.
Wrapper function for
drop_duplicate_rows_dbf()
.
- process_dbf(raw_dbf: pandas.DataFrame) pandas.DataFrame [source]#
DBF-specific transformations that take place before concatenation.
- process_xbrl(raw_xbrl_instant: pandas.DataFrame, raw_xbrl_duration: pandas.DataFrame) pandas.DataFrame [source]#
XBRL-specific transformations that take place before concatenation.
- rename_columns(df: pandas.DataFrame, rename_stage: Literal['dbf', 'xbrl', 'xbrl_instant', 'xbrl_duration'] | None = None, params: RenameColumns | None = None)[source]#
Grab the params based on the rename stage and run default rename_columns.
- Parameters:
df – Table to be renamed.
rename_stage – Name of stage in the transform process. Used to get specific stage’s parameters if None have been passed.
params – Rename column parameters.
- select_annual_rows_dbf(df)[source]#
Select only annually reported DBF Rows.
There are some DBF tables that include a mix of reporting frequencies. For now, the default for PUDL tables is to have only the annual records.
- unstack_balances_to_report_year_instant_xbrl(df: pandas.DataFrame, params: UnstackBalancesToReportYearInstantXbrl | None = None) pandas.DataFrame [source]#
Turn start year end year rows into columns for each value type.
- wide_to_tidy(df: pandas.DataFrame, source_ferc1: SourceFerc1, params: WideToTidy | None = None) pandas.DataFrame [source]#
Reshape wide tables with FERC account columns to tidy format.
The XBRL table coming into this method contains all the data from both the instant and duration tables in a wide format – with one column for every combination of value type (e.g. additions, ending_balance) and accounting category, which means ~500 columns.
We tidy this into a long table with one column for each of the value types (6 in all), and a new column that contains the accounting categories. This allows aggregation across columns to calculate the ending balance based on the starting balance and all of the reported changes, and aggregation across groups of rows to total up various hierarchical accounting categories (hydraulic turbines -> hydraulic production plant -> all production plant -> all electric utility plant) though the categorical columns required for that aggregation are added later.
- combine_axis_columns_xbrl(df: pandas.DataFrame, params: CombineAxisColumnsXbrl | None = None) pandas.DataFrame [source]#
Combine axis columns from squished XBRL tables into one column with no NA.
- merge_instant_and_duration_tables_xbrl(raw_xbrl_instant: pandas.DataFrame, raw_xbrl_duration: pandas.DataFrame) pandas.DataFrame [source]#
Merge XBRL instant and duration tables, reshaping instant as needed.
FERC1 XBRL instant period signifies that it is true as of the reported date, while a duration fact pertains to the specified time period. The
date
column for an instant fact corresponds to theend_date
column of a duration fact.When merging the instant and duration tables, we need to preserve row order. For the small generators table, row order is how we label and extract information from header and note rows. Outer merging messes up the order, so we need to use a one-sided merge. So far, it seems like the duration df contains all the index values in the instant df. To be sure, there’s a check that makes sure there are no unique intant df index values. If that passes, we merge the instant table into the duration table, and the row order is preserved.
Note: This should always be applied before :meth:
rename_columns
- Parameters:
raw_xbrl_instant – table representing XBRL instant facts.
raw_xbrl_duration – table representing XBRL duration facts.
- Returns:
A unified table combining the XBRL duration and instant facts, if both types of facts were present. If either input dataframe is empty, the other dataframe is returned unchanged, except that several unused columns are dropped. If both input dataframes are empty, an empty dataframe is returned.
- process_instant_xbrl(df: pandas.DataFrame) pandas.DataFrame [source]#
Pre-processing required to make instant and duration tables compatible.
Column renaming is sometimes required because a few columns in the instant and duration tables do not have corresponding names that follow the naming conventions of ~95% of all the columns, which we rely on programmatically when reshaping and concatenating these tables together.
- process_duration_xbrl(df: pandas.DataFrame) pandas.DataFrame [source]#
Pre-processing required to make instant and duration tables compatible.
Column renaming is sometimes required because a few columns in the instant and duration tables do not have corresponding names that follow the naming conventions of ~95% of all the columns, which we rely on programmatically when reshaping and concatenating these tables together.
- select_current_year_annual_records_duration_xbrl(df)[source]#
Select for annual records within their report_year.
Select only records that have a start_date at begining of the report_year and have an end_date at the end of the report_year.
- drop_footnote_columns_dbf(df: pandas.DataFrame) pandas.DataFrame [source]#
Drop DBF footnote reference columns, which all end with _f.
- source_table_primary_key(source_ferc1: SourceFerc1) list[str] [source]#
Look up the pre-renaming source table primary key columns.
- renamed_table_primary_key(source_ferc1: SourceFerc1) list[str] [source]#
Look up the post-renaming primary key columns.
- drop_unused_original_columns_dbf(df: pandas.DataFrame) pandas.DataFrame [source]#
Remove residual DBF specific columns.
- assign_record_id(df: pandas.DataFrame, source_ferc1: SourceFerc1) pandas.DataFrame [source]#
Add a column identifying the original source record for each row.
It is often useful to be able to tell exactly which record in the FERC Form 1 database a given record within the PUDL database came from.
Within each FERC Form 1 DBF table, each record is supposed to be uniquely identified by the combination of: report_year, report_prd, utility_id_ferc1_dbf, spplmnt_num, row_number.
The FERC Form 1 XBRL tables do not have these supplement and row number columns, so we construct an id based on: report_year, utility_id_ferc1_xbrl, and the primary key columns of the XBRL table
- Parameters:
df – table to assign record_id to
table_name – name of table
source_ferc1 – data source of raw ferc1 database.
- Raises:
ValueError – If any of the primary key columns are missing from the DataFrame being processed.
ValueError – If there are any null values in the primary key columns.
ValueError – If the resulting record_id column is non-unique.
- assign_utility_id_ferc1(df: pandas.DataFrame, source_ferc1: SourceFerc1) pandas.DataFrame [source]#
Assign the PUDL-assigned utility_id_ferc1 based on the native utility ID.
We need to replace the natively reported utility ID from each of the two FERC1 sources with a PUDL-assigned utilty. The mapping between the native ID’s and these PUDL-assigned ID’s can be accessed in the database tables
utilities_dbf_ferc1
andutilities_xbrl_ferc1
.- Parameters:
df – the input table with the native utilty ID column.
source_ferc1 – the
- Returns:
an augemented version of the input
df
with a new column that replaces the natively reported utility ID with the PUDL-assigned utility ID.
- class pudl.transform.ferc1.FuelFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
A table transformer specific to the fuel_ferc1 table.
The fuel_ferc1 table reports data about fuel consumed by large thermal power plants in the plants_steam_ferc1 table. Each record in the steam table is typically associated with several records in the fuel table, with each fuel record reporting data for a particular type of fuel consumed by that plant over the course of a year. The fuel table presents several challenges.
The type of fuel, which is part of the primary key for the table, is a freeform string with hundreds of different nonstandard values. These strings are categorized manually and converted to
fuel_type_code_pudl
. Some values cannot be categorized and are set toother
. In other string categorizations we set the unidentifiable values to NA, but in this table the fuel type is part of the primary key and primary keys cannot contain NA values.This simplified categorization occasionally results in records with duplicate primary keys. In those cases the records are aggregated into a single record if they have the same apparent physical units. If the fuel units are different, only the first record is retained.
Several columns have unspecified, inconsistent, fuel-type specific units of measure associated with them. In order for records to be comparable and aggregatable, we have to infer and standardize these units.
In the raw FERC Form 1 data there is a
fuel_units
column which describes the units of fuel delivered or consumed. Most commonly this is short tons for solid fuels (coal), thousands of cubic feet (Mcf) for gaseous fuels, and barrels (bbl) for liquid fuels. However, thefuel_units
column is also a freeform string with hundreds of nonstandard values which we have to manually categorize, and many of the values do not map directly to the most commonly used units for fuel quantities. E.g. some solid fuel quantities are reported in pounds, or thousands of pounds, not tons; some liquid fuels are reported in gallons or thousands of gallons, not barrels; and some gaseous fuels are reported in cubic feet not thousands of cubic feet.Two additional columns report fuel price per unit of heat content and fuel heat content per physical unit of fuel. The units of those columns are not explicitly reported, vary by fuel, and are inconsistent within individual fuel types.
We adopt standardized units and attempt to convert all reported values in the fuel table into those units. For physical fuel units we adopt those that are used by the EIA: short tons (tons) for solid fuels, barrels (bbl) for liquid fuels, and thousands of cubic feet (mcf) for gaseous fuels. For heat content per (physical) unit of fuel, we use millions of British thermal units (mmbtu). All fuel prices are converted to US dollars, while many are reported in cents.
Because the reported fuel price and heat content units are implicit, we have to infer them based on observed values. This is only possible because these quantities are ratios with well defined ranges of valid values. The common units that we observe and attempt to standardize include:
coal: primarily BTU/pound, but also MMBTU/ton and MMBTU/pound.
oil: primarily BTU/gallon.
gas: reported in a mix of MMBTU/cubic foot, and MMBTU/thousand cubic feet.
- table_id: TableIdFerc1[source]#
- transform_main(df: pandas.DataFrame) pandas.DataFrame [source]#
Table specific transforms for fuel_ferc1.
- Parameters:
df – Pre-processed, concatenated XBRL and DBF data.
- Returns:
A single transformed table concatenating multiple years of cleaned data derived from the raw DBF and/or XBRL inputs.
- process_dbf(raw_dbf: pandas.DataFrame) pandas.DataFrame [source]#
Start with inherited method and do some fuel-specific processing.
We have to do most of the transformation before the DBF and XBRL data have been concatenated because the fuel type column is part of the primary key and it is extensively modified in the cleaning process.
- process_xbrl(raw_xbrl_instant: pandas.DataFrame, raw_xbrl_duration: pandas.DataFrame) pandas.DataFrame [source]#
Special pre-concat treatment of the fuel_ferc1 table.
We have to do most of the transformation before the DBF and XBRL data have been concatenated because the fuel type column is part of the primary key and it is extensively modified in the cleaning process. For the XBRL data, this means we can’t create a record ID until that fuel type value is clean. In addition, the categorization of fuel types results in a number of duplicate fuel records which need to be aggregated.
- Parameters:
raw_xbrl_instant – Freshly extracted XBRL instant fact table.
raw_xbrl_duration – Freshly extracted XBRL duration fact table.
- Returns:
Almost fully transformed XBRL data table, with instant and duration facts merged together.
- standardize_physical_fuel_units(df: pandas.DataFrame) pandas.DataFrame [source]#
Convert reported fuel quantities to standard units depending on fuel type.
Use the categorized fuel type and reported fuel units to convert all fuel quantities to the following standard units, depending on whether the fuel is a solid, liquid, or gas. When a single fuel reports its quantity in fundamentally different units, convert based on typical values. E.g. 19.85 MMBTU per ton of coal, 1.037 Mcf per MMBTU of natural gas, 7.46 barrels per ton of oil.
solid fuels (coal and waste): short tons [ton]
liquid fuels (oil): barrels [bbl]
gaseous fuels (gas): thousands of cubic feet [mcf]
Columns to which these physical units apply:
fuel_consumed_units (tons, bbl, mcf)
fuel_cost_per_unit_burned (usd/ton, usd/bbl, usd/mcf)
fuel_cost_per_unit_delivered (usd/ton, usd/bbl, usd/mcf)
One remaining challenge in this standardization is that nuclear fuel is reported in both mass of Uranium and fuel heat content, and it’s unclear if there’s any reasonable typical conversion between these units, since available heat content depends on the degree of U235 enrichement, the type of reactor, and whether the fuel is just Uranium, or a mix of Uranium and Plutonium from decommissioned nuclear weapons. See:
https://world-nuclear.org/information-library/facts-and-figures/heat-values-of-various-fuels.aspx
- aggregate_duplicate_fuel_types_xbrl(fuel_xbrl: pandas.DataFrame) pandas.DataFrame [source]#
Aggregate the fuel records having duplicate primary keys.
- drop_total_rows(df: pandas.DataFrame) pandas.DataFrame [source]#
Drop rows that represent plant totals rather than individual fuels.
This is an imperfect, heuristic process. The rows we identify as probably representing totals rather than individual fuels:
have zero or null values in all of their numerical data columns
have no identifiable fuel type
have no identifiable fuel units
DO report a value for MMBTU / MWh (heat rate)
In the case of the fuel_ferc1 table, we drop any row where all the data columns are null AND there’s a non-null value in the
fuel_mmbtu_per_mwh
column, as it typically indicates a “total” row for a plant. We also require a null value for the fuel_units and an “other” value for the fuel type.
- drop_invalid_rows(df: pandas.DataFrame, params: InvalidRows | None = None) pandas.DataFrame [source]#
Drop invalid rows from the fuel table.
This method both drops rows in which all required data columns are null (using the inherited parameterized method) and then also drops those rows we believe represent plant totals. See
FuelFerc1TableTransformer.drop_total_rows()
.
- class pudl.transform.ferc1.PlantsSteamFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
Transformer class for the plants_steam_ferc1 table.
- table_id: TableIdFerc1[source]#
- transform_main(df: pandas.DataFrame, transformed_fuel: pandas.DataFrame) pandas.DataFrame [source]#
Perform table transformations for the plants_steam_ferc1 table.
Note that this method has a non-standard call signature, since the plants_steam_ferc1 table depends on the fuel_ferc1 table.
- Parameters:
df – The pre-processed steam plants table.
transformed_fuel – The fully transformed fuel_ferc1 table. This is required because fuel consumption information is used to help link steam plant records together across years using
plants_steam_assign_plant_ids()
- transform(raw_dbf: pandas.DataFrame, raw_xbrl_instant: pandas.DataFrame, raw_xbrl_duration: pandas.DataFrame, transformed_fuel: pandas.DataFrame) pandas.DataFrame [source]#
Redfine the transform method to accommodate the use of transformed_fuel.
This is duplicating code from the parent class, but is necessary because the steam table needs the fuel table for its transform. Is there a better way to do this that doesn’t require cutting and pasting the whole method just to stick the extra dataframe input into transform_main()?
- class pudl.transform.ferc1.PlantsHydroFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
A table transformer specific to the plants_hydro_ferc1 table.
- table_id: TableIdFerc1[source]#
- targeted_drop_duplicates(df)[source]#
Targeted removal of known duplicate record.
There are two records in 2019 with a
utility_id_ferc1
of 200 and aplant_name_ferc1
of “marmet”. The records are nearly duplicates of eachother, except one have nulls in the capex columns. Surgically remove the record with the nulls.
- class pudl.transform.ferc1.PlantsPumpedStorageFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
Transformer class for plants_pumped_storage_ferc1 table.
- table_id: TableIdFerc1[source]#
- class pudl.transform.ferc1.PurchasedPowerFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
Transformer class for purchased_power_ferc1 table.
This table has data about inter-utility power purchases into the PUDL DB. This includes how much electricty was purchased, how much it cost, and who it was purchased from. Unfortunately the field describing which other utility the power was being bought from is poorly standardized, making it difficult to correlate with other data. It will need to be categorized by hand or with some fuzzy matching eventually.
- table_id: TableIdFerc1[source]#
- class pudl.transform.ferc1.PlantInServiceFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
A transformer for the plant_in_service_ferc1 table.
- table_id: TableIdFerc1[source]#
- process_xbrl_metadata(xbrl_metadata_json) pandas.DataFrame [source]#
Transform the metadata to reflect the transformed data.
The XBRL Taxonomy metadata as extracted pertains to the XBRL data as extracted. When we re-shape the data, we also need to adjust the metadata to be usable alongside the reshaped data. For the plant in service table, this means selecting metadata fields that pertain to the “stem” column name (not differentiating between starting/ending balance, retirements, additions, etc.)
We fill in some gaps in the metadata, e.g. for FERC accounts that have been split across multiple rows, or combined without being calculated. We also need to rename the XBRL metadata categories to conform to the same naming convention that we are using in the data itself (since FERC doesn’t quite follow their own naming conventions…). We use the same rename dictionary, but as an argument to
pd.Series.replace()
instead ofpd.DataFrame.rename()
.
- apply_sign_conventions(df) pandas.DataFrame [source]#
Adjust rows and column sign conventsion to enable aggregation by summing.
Columns have uniform sign conventions, which we have manually inferred from the original metadata. This can and probably should be done programmatically in the future. If not, we’ll probably want to store the column_weights as a parameter rather than hard-coding it in here.
- targeted_drop_duplicates_dbf(df: pandas.DataFrame) pandas.DataFrame [source]#
Drop bad duplicate records from a specific utility in 2018.
This is a very specific fix, meant to get rid of a particular observed set of duplicate records: FERC Respondent ID 187 in 2018 has two sets of plant in service records, one of which contains a bunch of null data.
This method is part of the DBF processing because we want to be able to hard-code a specific value of
utility_id_ferc1_dbf
and those IDs are no longer available later in the process. I think.
- process_dbf(raw_dbf: pandas.DataFrame) pandas.DataFrame [source]#
Drop targeted duplicates in the DBF data so we can use FERC respondent ID.
- transform_main(df: pandas.DataFrame) pandas.DataFrame [source]#
The main table-specific transformations, affecting contents not structure.
Annotates and alters data based on information from the XBRL taxonomy metadata.
- class pudl.transform.ferc1.PlantsSmallFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
A table transformer specific to the plants_small_ferc1 table.
- table_id: TableIdFerc1[source]#
- transform_main(df: pandas.DataFrame) pandas.DataFrame [source]#
Table specific transforms for plants_small_ferc1.
- Params:
df: Pre-processed, concatenated XBRL and DBF data.
- Returns:
A single transformed table concatenating multiple years of cleaned data derived from the raw DBF and/or XBRL inputs.
- extract_ferc1_license(df: pandas.DataFrame) pandas.DataFrame [source]#
Extract FERC license number from
plant_name_ferc1
.Many FERC license numbers are embedded in the
plant_name_ferc1
column, but not all numbers in theplant_name_ferc1
column are FERC licenses. Some are dates, dollar amounts, page numbers, or numbers of wind turbines. This function extracts valid FERC license numbers and puts them in a new column calledlicense_id_ferc1
.Potential FERC license numbers are valid when:
Two or more integers were found.
The found integers were accompanied by key phrases such as:
["license", "no.", "ferc", "project"]
.The accompanying name does not contain phrases such as:
["page", "pg", "$", "wind", "units"]
.The found integers don’t fall don’t fall within the range of a valid year, defined as: 1900-2050.
The plant record is categorized as
hydro
or not categorized via theplant_type
andfuel_type
columns.
This function also fills
other
fuel types withhydro
for all plants with valid FERC licenses because only hydro plants have FERC licenses.- Params:
df: Pre-processed, concatenated XBRL and DBF data.
- Returns:
The same input DataFrame but with a new column called
license_id_ferc1
that contains FERC 1 license infromation extracted fromplant_name_ferc1
.
- _find_possible_header_or_note_rows(df: pandas.DataFrame) pandas.DataFrame [source]#
Find and label rows that might be headers or notes.
Called by the coordinating function
label_row_types()
.This function creates a column called
possible_header_or_note
that is either True or False depending on whether a group of columns are all NA. Rows labeled as True will be further scrutinized in the_label_header_rows()
and_label_note_rows()
functions to determine whether they are actually headers or notes.- Params:
df: Pre-processed, concatenated XBRL and DBF data.
- Returns:
The same input DataFrame but with a new column called
possible_header_or_note
that flags rows that might contain useful header or note information.
- _find_note_clumps(group: pandas.core.groupby.DataFrameGroupBy) tuple[pandas.core.groupby.DataFrameGroupBy, pandas.DataFrame] [source]#
Find groups of rows likely to be notes.
Once the
_find_possible_header_or_note_rows()
function identifies rows that are either headers or notes, we must deterine which one they are. As described in the_label_note_rows()
function, notes rows are usually adjecent rows with no content.This function itentifies instances of two or more adjecent rows where
possible_header_or_note
= True. It takes individual utility-year groups as a parameter as opposed to the entire dataset because adjecent rows are only meaningful if they are from the same reporting entity in the same year. If we were to run this on the whole dataframe, we would see “note clumps” that are actually notes from the end of one utility’s report and headers from the beginning of another. For this reason, we run this function from within the_label_note_rows_group()
function.The output of this function is not a modified version of the original utility-year group, rather, it is a DataFrame containing information about the nature of the
possible_header_or_note
= True rows that is used to determine if that row is a note or not. It also returns the original utility-year-group as groupby objects seperated by each timepossible_header_or_note
changes from True to False or vice versa.If you pass in the following df:
plant_name_ferc1
possible_header_or_note
HYDRO:
True
rainbow falls (b)
False
cadyville (a)
False
keuka (c)
False
project #2738
True
project #2835
True
project #2852
True
You will get the following output (in addition to the groupby objects for each clump):
header_or_note
rows_per_clump
True
1
False
3
True
3
This shows each clump of adjecent records where
possible_header_or_note
is True or False and how many records are in each clump.- Params:
- group: A utility-year grouping of the concatenated FERC XBRL and DBF tables.
This table must have been run through the
_find_possible_header_or_note_rows()
function and contain the columnpossible_header_or_note
.
- Returns:
A tuple containing groupby objects for each of the note and non-note clumps and a DataFrame indicating the number of rows in each note or non-note clump.
- _label_header_rows(df: pandas.DataFrame) pandas.DataFrame [source]#
Label header rows by adding
header
torow_type
column.Called by the coordinating function
label_row_types()
.Once possible header or notes rows have been identified via the
_find_possible_header_or_note_rows()
function, this function sorts out which ones are headers. It does this by identifying a list of strings that, when found in theplant_name_ferc1
column, indicate that the row is or is not a header.Sometimes this function identifies a header that is acutally a note. For this reason, it’s important that the function be called before
_label_note_rows()
so that the bad header values get overridden by thenote
designation.- Params:
df: Pre-processed, concatenated XBRL and DBF data that has been run through the
_find_possible_header_or_note_rows()
function and contains the columnpossible_header_or_note
.
- Returns:
The same input DataFrame but with likely headers rows containing the string
header
in therow_type
column.
- _label_note_rows_group(util_year_group: pandas.core.groupby.DataFrameGroupBy) pandas.core.groupby.DataFrameGroupBy [source]#
Label note rows by adding
note
torow_type
column.Called within the wraper function
_label_note_rows()
This function breaks the data down by reporting unit (utility and year) and determines whether a
possible_header_note
= True row is a note based on two criteria:Clumps of 2 or more adjecent rows where
possible_header_or_note
is True.Instances where the last row in a utility-year group has
possible_header_or_note
as True.
There are a couple of important exceptions that this function also addresses. Utilities often have multiple headers in a single utility-year grouping. You might see something like:
pd.Series([header, plant1, plant2, note, header, plant3, plant4])
. In this case, a note clump is actually comprised of a note followed by a header. This function will not override the header as a note. Unfortunately, there is always the possability that a header row is followed by a plant that had no values reported. This would look like, and therefore be categorized as a note clump. I haven’t built a work around, but hopefully there aren’t very many of these.- Params:
util_year_group: A groupby object that contains a single year and utility.
- Returns:
The same input but with likely note rows containing the string
note
in therow_type
column.
- _label_note_rows(df: pandas.DataFrame) pandas.DataFrame [source]#
Wrapper for
_label_note_rows_group()
.The small plants table has lots of note rows that contain useful information. Unfortunately, the notes are in their own row rather than their own column! This means that useful information pertaining to plant rows is floating around as a junk row with no other information except the note in the
plant_name_ferc1
field. Luckily, the data are reported just like they would be on paper. I.e., The headers are at the top, and the notes are at the bottom. See the table inlabel_row_types()
for more detail. This function labels note rows.Note rows are determined by row location within a given report, so we must break the data into reporting units (utility and year) and then apply note-finding methodology defined in
_label_note_rows_group()
to each group.- Params:
df: Pre-processed, concatenated XBRL and DBF data that has been run through the
_find_possible_header_or_note_rows()
function and contains the columnpossible_header_or_note
.
- Returns:
The same input DataFrame but with likely note rows containing the string
note
in therow_type
column.
- _label_total_rows(df: pandas.DataFrame) pandas.DataFrame [source]#
Label total rows by adding
total
torow_type
column.Called within the wraper function
_label_note_rows()
For the most part, when
plant_name_ferc1
contains the stringtotal
, the values therein are duplicates of what is already reported, i.e.: a total value. However, there are some cases where that’s not true. For example, the phraseamounts are for the total
appears when chunks of plants (usually but not always wind) are reported together. It’s a total, but it’s not double counting which is the reason for thetotal
flag.Similar to
_label_header_rows()
, it’s important that this be called before_label_note_rows()
inlabel_row_types()
so that not clumps can override certain non-totals that are mistakenly labeled as such.- Params:
df: Pre-processed, concatenated XBRL and DBF data.
- Returns:
The same input DataFrame but with likely total rows containing the string
total
in therow_type
column.
- label_row_types(df: pandas.DataFrame) pandas.DataFrame [source]#
Coordinate labeling of
row_types
as headers, notes, or totals.The small plants table is more like a digitized PDF than an actual data table. The rows contain all sorts of information in addition to what the columns might suggest. For instance, there are header rows, note rows, and total rows that contain useful information, but cause confusion in their current state, mixed in with the rest of the data.
Here’s an example of what you might find in the small plants table:
plant_name_ferc1
plant_type
capacity_mw
HYDRO:
NA
NA
rainbow falls (b)
NA
30
cadyville (a)
NA
100
keuka (c)
NA
80
total plants
NA
310
project #2738
NA
NA
project #2835
NA
NA
project #2852
NA
NA
Notice how misleading it is to have all this infomration in one column. The goal of this function is to coordinate labeling functions so that we can identify which rows contain specific plant information and which rows are headers, notes, or totals.
Once labeled, other functions can either remove rows that might cause double counting, extract useful plant or fuel type information from headers, and extract useful context or license id information from notes.
Coordinates
_label_header_rows()
,_label_total_rows()
,_label_note_rows()
.- Params:
df: Pre-processed, concatenated XBRL and DBF data that has been run through the
_find_possible_header_or_note_rows()
function and contains the columnpossible_header_or_note
.
- Returns:
The same input DataFrame but with a column called
row_type
containg the stringsheader
,note
,total
, or NA to indicate what type of row it is.
- prep_header_fuel_and_plant_types(df: pandas.DataFrame, show_unmapped_headers=False) pandas.DataFrame [source]#
Forward fill header rows to prep for fuel and plant type extraction.
The headers we’ve identified in
_label_header_rows()
can be used to supplement the values in theplant_type
andfuel_type
columns.This function groups the data by utility, year, and header; extracts the header into a new column; and forward fills the headers so that each record in the header group is associated with that header. Because the headers map to different fuel types and plant types (ex:
solar pv
maps to fuel typesolar
and plant typephotovoltaic
), the new forward-filled header column is duplicated and calledfuel_type_from_header
andplant_type_from_header
. Inmap_header_fuel_and_plant_types()
, these columns will be mapped to their respective fuel and plant types, used to fill in blank values in theplant_type
andfuel_type
, and then eventually removed.Why separate the prep step from the map step?
We trust the values originally reported in the
fuel_type
andplant_type
columns more than the extracted and forward filled header values, so we only want to replacefuel_type
andplant_type
values that are labeled aspd.NA
orother
. The values reported to those columns are extremely messy and must be cleaned viapudl.transform.classes.categorize_strings()
in order for us to know which are truelypd.NA
orother
. Because we also usepudl.transform.classes.categorize_strings()
to map the headers to fuel and plant types, it makes sense to clean all four columns at once and then combine them.Here’s a look at what this function does. It starts with the following table:
plant_name_ferc1
plant_type
fuel_type
row_type
HYDRO:
NA
NA
header
rainbow falls (b)
NA
NA
NA
cadyville (a)
NA
NA
NA
keuka (c)
NA
NA
NA
Wind Turbines:
NA
NA
header
sunny grove
NA
NA
NA
green park wind
NA
wind
NA
And ends with this:
plant_name_ferc1
plant _type
fuel _type
plant_type _from_header
fuel_type _from_header
HYDRO:
NA
NA
HYDRO:
HYDRO:
rainbow falls (b)
NA
NA
HYDRO:
HYDRO:
cadyville (a)
NA
NA
HYDRO:
HYDRO:
keuka (c)
NA
NA
HYDRO:
HYDRO:
Wind Turbines:
NA
NA
Wind Turbines:
Wind Turbines:
sunny grove
NA
NA
Wind Turbines:
Wind Turbines:
green park wind
NA
wind
Wind Turbines:
Wind Turbines:
NOTE: If a utility’s
plant_name_ferc1
values look like this:["STEAM", "coal_plant1", "coal_plant2", "wind_turbine1"]
, then this algorythem will think that last wind turbine is a steam plant. Luckily, when a utility embeds headers in the data it usually includes them for all plant types:["STEAM", "coal_plant1", "coal_plant2", "WIND", "wind_turbine"]
.- Params:
df: Pre-processed, concatenated XBRL and DBF data that has been run through
_label_row_type()
and contains the columnsrow_type
.
- Returns:
The same input DataFrame but with new columns
plant_type_from_header
andfuel_type_from_header
that forward fill the values in the header rows by utility, year, and header group.
- map_header_fuel_and_plant_types(df: pandas.DataFrame) pandas.DataFrame [source]#
Fill
pd.NA
andother
plant and fuel types with cleaned headers.prep_header_fuel_and_plant_types()
extracted and forward filled the header values;pudl.transform.params.categorize_strings()
cleaned them according to both the fuel and plant type parameters. This function combines thefuel_type_from_header
withfuel_type
andplant_type_from_header
withplant_type
when the reported, cleaned values arepd.NA
orother
.To understand more about why these steps are necessary read the docstrings for
prep_header_fuel_and_plant_types()
.- Params:
df: Pre-processed, concatenated XBRL and DBF data that has been run through
prep_header_fuel_and_plant_types()
and contains the columnsfuel_type_from_header
andplant_type_from_header
.
- Returns:
The same input DataFrame but with rows with
pd.NA
orother
in thefuel_type
andplant_type
columns filled in with the respective values fromfuel_type_from_header
andplant_type_from_header
when available.fuel_type_from_header
andplant_type_from_header
columns removed.
- map_plant_name_fuel_types(df: pandas.DataFrame) pandas.DataFrame [source]#
Suppliment
fuel_type
with information inplant_name_ferc1
.Sometimes fuel type is embedded in a plant name (not just headers). In this case we can identify that what that fuel is from the name and fill in empty
fuel_type
values. Right now, this only works for hydro plants because the rest are complicated and have a slew of exceptions. This could probably be applied to theplant_type
column in the future too.- Params:
df: Pre-processed, concatenated XBRL and DBF data.
- Returns:
The same input DataFrame but with rows with
other
in thefuel_type
column filled in notable fuel types extracted from the theplant_name_ferc1
column.
- associate_notes_with_values(df: pandas.DataFrame) pandas.DataFrame [source]#
Use footnote indicators to map notes and FERC licenses to plant rows.
There are many utilities that report a bunch of mostly empty note rows at the bottom of their yearly entry. These notes often pertain to specific plant rows above. Sometimes the notes and their respective plant rows are linked by a common footnote indicator such as (a) or (1) etc.
This function takes this:
plant_name_ferc1
row_type
license_id_ferc1
HYDRO:
header
NA
rainbow falls (b)
NA
NA
cadyville (a)
NA
NA
keuka (c)
NA
NA
total plants
total
NA
project #2738
note
2738
project #2835
note
2738
project #2852
note
2738
Finds the note rows with footnote indicators, maps the content from the note row into a new note column that’s associated with the value row, and maps any FERC license extracted from this note column to the
license_id_ferc1
column in the value row.plant_name_ferc1
row_type
notes
license_id_ferc1
HYDRO:
header
NA
NA
rainbow falls (b)
NA
project #2835
2835
cadyville (a)
NA
project #2738
2738
keuka (c)
NA
project #2852
2752
total plants
total
NA
NA
project #2738
note
NA
2738
project #2835
note
NA
2835
project #2852
note
NA
2752
(Header and note rows are removed later).
NOTE: Note rows that don’t have a footnote indicator or note rows with a footnote indicator that don’t have a cooresponding plant row with the same indicator are not captured. They will ultimately get removed and their content will not be preserved.
- Params:
df: Pre-processed, concatenated XBRL and DBF data that has been run through
label_row_types()
and contains the columnrow_type
.
- Returns:
The same input DataFrame but with a column called
notes
that contains notes, reported below, in the same row as the plant values they pertain to. Also, any further additions to thelicense_id_ferc1
field as extracted from these newly associated notes.
- spot_fix_rows(df: pandas.DataFrame) pandas.DataFrame [source]#
Fix one-off row errors.
In 2004, utility_id_ferc1 251 reports clumps of units together. Each unit clump looks something like this:
intrepid wind farm (107 units @ 1.5 mw each)
and is followed by a row that looks like this:(amounts are for the total of all 107 units)
. For the most part, these rows are useless note rows. However, there is one instance where important values are reported in this note row rather than in the actual plant row above.There are probably plenty of other spot fixes one could add here.
- Params:
df: Pre-processed, concatenated XBRL and DBF data.
- Returns:
The same input DataFrame but with some spot fixes corrected.
- class pudl.transform.ferc1.TransmissionStatisticsFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
A table transformer for the transmission_statistics_ferc1 table.
- table_id: TableIdFerc1[source]#
- class pudl.transform.ferc1.ElectricEnergySourcesFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
Transformer class for electric_energy_sources_ferc1 table.
The raw DBF and XBRL table will be split up into two tables. This transformer generates the sources of electricity for utilities, dropping the information about dispositions. For XBRL, this is a duration-only table. Right now we are merging in the metadata but not actually keeping anything from it. We are also not yet doing anything with the sign.
- table_id: TableIdFerc1[source]#
- class pudl.transform.ferc1.ElectricEnergyDispositionsFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
Transformer class for electric_energy_dispositions_ferc1 table.
- table_id: TableIdFerc1[source]#
- class pudl.transform.ferc1.UtilityPlantSummaryFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
Transformer class for utility_plant_summary_ferc1 table.
- table_id: TableIdFerc1[source]#
- class pudl.transform.ferc1.BalanceSheetLiabilitiesFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
Transformer class for balance_sheet_liabilities_ferc1 table.
- table_id: TableIdFerc1[source]#
- class pudl.transform.ferc1.BalanceSheetAssetsFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
Transformer class for balance_sheet_assets_ferc1 table.
- table_id: TableIdFerc1[source]#
- class pudl.transform.ferc1.IncomeStatementFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
Transformer class for the income_statement_ferc1 table.
- table_id: TableIdFerc1[source]#
- process_dbf(raw_dbf)[source]#
Drop incorrect row numbers from f1_incm_stmnt_2 before standard processing.
In 2003, two rows were added to the
f1_income_stmnt
dbf table, which bumped the startingrow_number
off1_incm_stmnt_2
from 25 to 27. A small handfull of respondents seem to have not gotten the memo about this this in 2003 and have information on these row numbers that shouldn’t exist at all for this table.This step necessitates the ability to know which source table each record actually comes from, which required adding a column (
sched_table_name
) in the extract step before these two dbf input tables were concatenated.Right now we are just dropping these bad row numbers. Should we actually be bumping the whole respondent’s row numbers - assuming they reported incorrectly for the whole table? See: https://github.com/catalyst-cooperative/pudl/issues/471
- class pudl.transform.ferc1.RetainedEarningsFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
Transformer class for retained_earnings_ferc1 table.
- table_id: TableIdFerc1[source]#
- process_dbf(raw_dbf: pandas.DataFrame) pandas.DataFrame [source]#
Preform generic
process_dbf()
, plus deal with duplicates.Along with the standard processing in
Ferc1AbstractTableTransformer.process_dbf()
, this method runs: *targeted_drop_duplicates_dbf()
*condense_double_year_earnings_types_dbf()
- targeted_drop_duplicates_dbf(df: pandas.DataFrame) pandas.DataFrame [source]#
Drop duplicates with truly duplicate data.
There are instances of utilities that reported multiple values for several earnings types for a specific year (utility_id_ferc1 68 in 1998 & utility_id_ferc1 296 in 2015). We are taking the largest value reported and dropping the rest. There very well could be a better strategey here, but there are only 25 records that have this problem, so we’ve going with this.
- condense_double_year_earnings_types_dbf(df: pandas.DataFrame) pandas.DataFrame [source]#
Condense current and past year data reported in 1 report_year into 1 record.
The DBF table includes two different earnings types that have: “Begining of Period” and “End of Period” rows. But the table has both an amount column that corresponds to a balance and a starting balance column. For these two earnings types, this means that there is in effect two years of data in this table for each report year: a starting and ending balance for the pervious year and a starting and ending balance for the current year. The ending balance for the previous year should be the same as the starting balance for the current year.
We don’t actually want two years of data for each report year, so we want to check these assumptions, extract as much information from these two years of data, but end up with only one annual record for each of these two earnings types for each utility.
- Raises:
AssertionError – There are a very small number of instances in which the ending balance from the previous year does not match the starting balance from the current year. The % of these non-matching instances should be less than 2% of the records with these date duplicative earnings types.
- process_xbrl_metadata(xbrl_metadata_json) pandas.DataFrame [source]#
Transform the metadata to reflect the transformed data.
Run the generic
process_xbrl_metadata()
and then remove some suffixes that were removed duringwide_to_tidy()
.
- class pudl.transform.ferc1.DepreciationAmortizationSummaryFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
Transformer class for depreciation_amortization_summary_ferc1 table.
- table_id: TableIdFerc1[source]#
- merge_xbrl_metadata(df: pandas.DataFrame, params: MergeXbrlMetadata | None = None) pandas.DataFrame [source]#
Annotate data using manually compiled FERC accounts &
ferc_account_label
.The FERC accounts are not provided in the XBRL taxonomy like they are for other tables. However, there are only 4 of them, so a hand-compiled mapping is hardcoded here, and merged onto the data similar to how the XBRL taxonomy derived metadata is merged on for other tables.
- class pudl.transform.ferc1.ElectricPlantDepreciationChangesFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
Transformer class for electric_plant_depreciation_changes_ferc1 table.
- table_id: TableIdFerc1[source]#
- process_xbrl_metadata(xbrl_metadata_json) pandas.DataFrame [source]#
Transform the metadata to reflect the transformed data.
Replace the name of the balance column reported in the XBRL Instant table with starting_balance / ending_balance since we pull those two values into their own separate labeled rows, each of which should get the original metadata for the Instant column.
- process_dbf(df: pandas.DataFrame) pandas.DataFrame [source]#
Accumulated Depreciation table specific DBF cleaning operations.
The XBRL reports a utility_type which is always electric in this table, but which may be necessary for differentiating between different values when this data is combined with other tables. The DBF data doesn’t report this value so we are adding it here for consistency across the two data sources.
Also rename the
ending_balance_accounts
toending_balance
- process_instant_xbrl(df: pandas.DataFrame) pandas.DataFrame [source]#
Pre-processing required to make the instant and duration tables compatible.
This table has a rename that needs to take place in an unusual spot – after the starting / ending balances have been usntacked, but before the instant & duration tables are merged. This method just reversed the order in which these operations happen, comapared to the inherited method.
- class pudl.transform.ferc1.ElectricPlantDepreciationFunctionalFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
Transformer for electric_plant_depreciation_functional_ferc1 table.
- table_id: TableIdFerc1[source]#
- process_xbrl_metadata(xbrl_metadata_json) pandas.DataFrame [source]#
Transform the metadata to reflect the transformed data.
Transform the xbrl factoid values so that they match the final plant functional classification categories and can be merged with the output dataframe.
- process_dbf(df: pandas.DataFrame) pandas.DataFrame [source]#
Accumulated Depreciation table specific DBF cleaning operations.
The XBRL reports a utility_type which is always electric in this table, but which may be necessary for differentiating between different values when this data is combined with other tables. The DBF data doesn’t report this value so we are adding it here for consistency across the two data sources.
- process_instant_xbrl(df: pandas.DataFrame) pandas.DataFrame [source]#
Pre-processing required to make the instant and duration tables compatible.
This table has a rename that needs to take place in an unusual spot – after the starting / ending balances have been usntacked, but before the instant & duration tables are merged. This method reverses the order in which these operations happen comapared to the inherited method. We also want to strip the
accumulated_depreciation
that appears on every plant functional class.
- class pudl.transform.ferc1.ElectricOperatingExpensesFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
Transformer class for electric_operating_expenses_ferc1 table.
- table_id: TableIdFerc1[source]#
- targeted_drop_duplicates_dbf(raw_df: pandas.DataFrame) pandas.DataFrame [source]#
Drop incorrect duplicate from 2002.
In 2002, utility_id_ferc1_dbf 96 reported two values for administrative_and_general_operation_expense. I found the correct value by looking at the prev_yr_amt value in 2003. This removes the incorrect row.
- process_dbf(raw_dbf: pandas.DataFrame) pandas.DataFrame [source]#
Process DBF but drop a bad row that is flagged by drop_duplicates.
- class pudl.transform.ferc1.ElectricOperatingRevenuesFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
Transformer class for electric_operating_revenues_ferc1 table.
- table_id: TableIdFerc1[source]#
- class pudl.transform.ferc1.CashFlowFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
Transform class for cash_flow_ferc1 table.
- table_id: TableIdFerc1[source]#
- process_instant_xbrl(df: pandas.DataFrame) pandas.DataFrame [source]#
Pre-processing required to make the instant and duration tables compatible.
This table has a rename that needs to take place in an unusual spot – after the starting / ending balances have been usntacked, but before the instant & duration tables are merged. This method just reversed the order in which these operations happen, comapared to the inherited method.
- targeted_drop_duplicates(df)[source]#
Drop one duplicate record from 2020, utility_id_ferc1 2037.
Note: This step could be avoided if we employed a
drop_invalid_rows()
transform step withrequired_valid_cols = ["amount"]
- validate_start_end_balance(df)[source]#
Validate of start balance + net = end balance.
Add a quick check to ensure the vast majority of the ending balances are calculable from the net change + the starting balance = the ending balance.
- process_xbrl_metadata(xbrl_metadata_json) pandas.DataFrame [source]#
Transform the metadata to reflect the transformed data.
Replace the name of the balance column reported in the XBRL Instant table with starting_balance / ending_balance since we pull those two values into their own separate labeled rows, each of which should get the original metadata for the Instant column.
- class pudl.transform.ferc1.ElectricitySalesByRateScheduleFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
Transform class for electricity_sales_by_rate_schedule_ferc1 table.
- table_id: TableIdFerc1[source]#
- add_axis_to_total_table_rows(df: pandas.DataFrame)[source]#
Add total to the axis column for rows from the total table.
Because we’re adding the sales_of_electricity_by_rate_schedules_account_totals_304 table into the mix, we have a bunch of total values that get mixed in with all the _billed columns from the individual tables. If left alone, these totals aren’t labeled in any way becuse they don’t have the same _axis columns explaining what each of the values are. In order to distinguish them from the rest of the sub-total data we use this function to create an _axis value for them noting that they are totals.
It’s worth noting that there are also some total values in there already. Those would be hard to clean. The idea is that if you want the actual totals, don’t try and sum the sub-components, look at the actual labeled total rows.
This function relies on the
sched_table_name
column, so it must be called before that gets dropped.- Parameters:
df – The sales table with a
sched_table_name
column.
- process_xbrl(raw_xbrl_instant: pandas.DataFrame, raw_xbrl_duration: pandas.DataFrame) pandas.DataFrame [source]#
Rename columns before running wide_to_tidy.
- class pudl.transform.ferc1.OtherRegulatoryLiabilitiesFerc1TableTransformer(params: TableTransformParams | None = None, cache_dfs: bool = False, clear_cached_dfs: bool = True, xbrl_metadata_json: dict[Literal['instant', 'duration'], list[dict[str, Any]]] | None = None)[source]#
Bases:
Ferc1AbstractTableTransformer
Transformer class for other_regulatory_liabilities_ferc1 table.
- table_id: TableIdFerc1[source]#
- pudl.transform.ferc1.ferc1_transform_asset_factory(table_name: str, ferc1_tfr_classes: collections.abc.Mapping[str, type[Ferc1AbstractTableTransformer]], io_manager_key: str = 'pudl_sqlite_io_manager', convert_dtypes: bool = True, generic: bool = False) dagster.AssetsDefinition [source]#
Create an asset that pulls in raw ferc Form 1 assets and applies transformations.
This is a convenient way to create assets for tables that only depend on raw dbf, raw xbrl instant and duration tables and xbrl metadata. For tables with additional upstream dependencies, create a stand alone asset using an asset decorator. See the plants_steam_ferc1 asset.
- Parameters:
table_name – The name of the table to create an asset for.
ferc1_tfr_classes – A dictionary of table names to the corresponding transformer class.
io_manager_key – the dagster io_manager key to use. None defaults to the fs_io_manager.
convert_dtypes – convert dtypes of transformed dataframes.
generic – If using GenericPlantFerc1TableTransformer pass table_id to constructor.
- Returns:
An asset for the clean table.
- pudl.transform.ferc1.create_ferc1_transform_assets() list[dagster.AssetsDefinition] [source]#
Create a list of transformed FERC Form 1 assets.
- Returns:
A list of AssetsDefinitions where each asset is a clean ferc form 1 table.
- pudl.transform.ferc1.plants_steam_ferc1(xbrl_metadata_json: dict[str, dict[str, list[dict[str, Any]]]], f1_steam: pandas.DataFrame, steam_electric_generating_plant_statistics_large_plants_402_duration: pandas.DataFrame, steam_electric_generating_plant_statistics_large_plants_402_instant: pandas.DataFrame, fuel_ferc1: pandas.DataFrame) pandas.DataFrame [source]#
Create the clean plants_steam_ferc1 table.
- Parameters:
xbrl_metadata_json – XBRL metadata json for all tables.
f1_steam – Raw f1_steam table.
steam_electric_generating_plant_statistics_large_plants_402_duration – raw XBRL duration table.
steam_electric_generating_plant_statistics_large_plants_402_instant – raw XBRL instant table.
fuel_ferc1 – Transformed fuel_ferc1 table.
- Returns:
Clean plants_steam_ferc1 table.