pudl.ferc_to_sqlite.cli

A script using Dagster to convert FERC data fom DBF and XBRL to SQLite databases.

Attributes

logger

Functions

ferc_to_sqlite_job_factory(...)	Factory for parameterizing a reconstructable ferc_to_sqlite job.
main(etl_settings_yml, batch_size, workers, ...)	Use Dagster to convert FERC data fom DBF and XBRL to SQLite databases.

Module Contents

pudl.ferc_to_sqlite.cli.logger

pudl.ferc_to_sqlite.cli.ferc_to_sqlite_job_factory(logfile: str | None = None, loglevel: str = 'INFO', dataset_only: str | None = None) → collections.abc.Callable[[], dagster.JobDefinition]

Factory for parameterizing a reconstructable ferc_to_sqlite job.

Parameters:

• logfile – Path to a log file for the job’s execution.

• loglevel – The log level for the job’s execution.

Returns:

The job definition to be executed.

pudl.ferc_to_sqlite.cli.main(etl_settings_yml: pathlib.Path, batch_size: int, workers: int | None, dagster_workers: int, gcs_cache_path: str, logfile: pathlib.Path, loglevel: str, dataset_only: str)

Use Dagster to convert FERC data fom DBF and XBRL to SQLite databases.

Reads settings specifying which forms and years to convert from ETL_SETTINGS_YML.

Also produces JSON versions of XBRL taxonomies and datapackage descriptors which annotate the XBRL derived SQLite databases.