pudl.ferc_to_sqlite.cli#

A script using Dagster to convert FERC data fom DBF and XBRL to SQLite databases.

Module Contents#

Functions#

ferc_to_sqlite_job_factory(...)

Factory for parameterizing a reconstructable ferc_to_sqlite job.

main(etl_settings_yml, batch_size, workers, ...)

Use Dagster to convert FERC data fom DBF and XBRL to SQLite databases.

Attributes#

logger

pudl.ferc_to_sqlite.cli.logger[source]#
pudl.ferc_to_sqlite.cli.ferc_to_sqlite_job_factory(logfile: str | None = None, loglevel: str = 'INFO', dataset_only: str | None = None) → collections.abc.Callable[[], dagster.JobDefinition][source]#

Factory for parameterizing a reconstructable ferc_to_sqlite job.

Parameters:

  • logfile – Path to a log file for the job’s execution.

  • loglevel – The log level for the job’s execution.

Returns:

The job definition to be executed.

pudl.ferc_to_sqlite.cli.main(etl_settings_yml: pathlib.Path, batch_size: int, workers: int | None, dagster_workers: int, clobber: bool, gcs_cache_path: str, logfile: pathlib.Path, loglevel: str, dataset_only: str)[source]#

Use Dagster to convert FERC data fom DBF and XBRL to SQLite databases.

Reads settings specifying which forms and years to convert from ETL_SETTINGS_YML.

Also produces JSON versions of XBRL taxonomies and datapackage descriptors which annotate the XBRL derived SQLite databases.