pudl.ferc_to_sqlite#

Dagster definitions for the FERC to SQLite process.

Submodules#

Package Contents#

Classes#

RuntimeSettings

Encodes runtime settings for the ferc_to_sqlite graphs.

EtlSettings

Main settings validation class.

XbrlFormNumber

Contains full list of supported FERC XBRL forms.

Functions#

xbrl2sqlite_op_factory(→ collections.abc.Callable)

Generates xbrl2sqlite op for a given FERC form.

datastore(→ pudl.workspace.datastore.Datastore)

Dagster resource to interact with Zenodo archives.

ferc_to_sqlite_settings(...)

Dagster resource for parameterizing the ferc_to_sqlite graph.

ferc_to_sqlite()

Clone the FERC FoxPro databases and XBRL filings into SQLite.

Attributes#

ALL_DBF_EXTRACTORS

logger

default_resources_defs

ferc_to_sqlite_full

ferc_to_sqlite_fast_settings

ferc_to_sqlite_fast

defs

A collection of dagster assets, resources, IO managers, and jobs for the FERC to
pudl.ferc_to_sqlite.ALL_DBF_EXTRACTORS[source]#
pudl.ferc_to_sqlite.xbrl2sqlite_op_factory(form: pudl.settings.XbrlFormNumber) → collections.abc.Callable[source]#

Generates xbrl2sqlite op for a given FERC form.

class pudl.ferc_to_sqlite.RuntimeSettings[source]#

Bases: dagster.ConfigurableResource

Encodes runtime settings for the ferc_to_sqlite graphs.

clobber: bool = False#
xbrl_num_workers: None | int#
xbrl_batch_size: int = 50#
pudl.ferc_to_sqlite.datastore(init_context) → pudl.workspace.datastore.Datastore[source]#

Dagster resource to interact with Zenodo archives.

pudl.ferc_to_sqlite.ferc_to_sqlite_settings(init_context) → pudl.settings.FercToSqliteSettings[source]#

Dagster resource for parameterizing the ferc_to_sqlite graph.

This resource allows us to specify the years we want to process for each datasource in the Dagit UI.

class pudl.ferc_to_sqlite.EtlSettings(_case_sensitive: bool | None = None, _env_prefix: str | None = None, _env_file: pydantic_settings.sources.DotenvType | None = ENV_FILE_SENTINEL, _env_file_encoding: str | None = None, _env_ignore_empty: bool | None = None, _env_nested_delimiter: str | None = None, _env_parse_none_str: str | None = None, _secrets_dir: str | pathlib.Path | None = None, **values: Any)[source]#

Bases: pydantic_settings.BaseSettings

Main settings validation class.

ferc_to_sqlite_settings: FercToSqliteSettings | None#
datasets: DatasetsSettings | None#
name: str | None#
title: str | None#
description: str | None#
version: str | None#
publish_destinations: list[str] = []#
classmethod from_yaml(path: str) → EtlSettings[source]#

Create an EtlSettings instance from a yaml_file path.

Parameters:

path – path to a yaml file; this could be remote.

Returns:

An ETL settings object.

class pudl.ferc_to_sqlite.XbrlFormNumber(*args, **kwds)[source]#

Bases: enum.Enum

Contains full list of supported FERC XBRL forms.

FORM1 = 1#
FORM2 = 2#
FORM6 = 6#
FORM60 = 60#
FORM714 = 714#
pudl.ferc_to_sqlite.logger[source]#
pudl.ferc_to_sqlite.ferc_to_sqlite()[source]#

Clone the FERC FoxPro databases and XBRL filings into SQLite.

pudl.ferc_to_sqlite.default_resources_defs[source]#
pudl.ferc_to_sqlite.ferc_to_sqlite_full[source]#
pudl.ferc_to_sqlite.ferc_to_sqlite_fast_settings[source]#
pudl.ferc_to_sqlite.ferc_to_sqlite_fast[source]#
pudl.ferc_to_sqlite.defs: dagster.Definitions[source]#

A collection of dagster assets, resources, IO managers, and jobs for the FERC to SQLite ETL.