pudl.ferc_to_sqlite

Dagster definitions for the FERC to SQLite process.

Submodules

Package Contents

Classes

RuntimeSettings

Encodes runtime settings for the ferc_to_sqlite graphs.

EtlSettings

Main settings validation class.

XbrlFormNumber

Contains full list of supported FERC XBRL forms.

Functions

xbrl2sqlite_op_factory(→ collections.abc.Callable)

Generates xbrl2sqlite op for a given FERC form.

datastore(→ pudl.workspace.datastore.Datastore)

Dagster resource to interact with Zenodo archives.

ferc_to_sqlite_settings(...)

Dagster resource for parameterizing the ferc_to_sqlite graph.

ferc_to_sqlite()

Clone the FERC FoxPro databases and XBRL filings into SQLite.

Attributes

ALL_DBF_EXTRACTORS

logger

default_resources_defs

ferc_to_sqlite_full

ferc_to_sqlite_fast_settings

ferc_to_sqlite_fast

defs

A collection of dagster assets, resources, IO managers, and jobs for the FERC to
pudl.ferc_to_sqlite.ALL_DBF_EXTRACTORS[source]
pudl.ferc_to_sqlite.xbrl2sqlite_op_factory(form: pudl.settings.XbrlFormNumber) → collections.abc.Callable[source]

Generates xbrl2sqlite op for a given FERC form.

class pudl.ferc_to_sqlite.RuntimeSettings[source]

Bases: dagster.ConfigurableResource

Encodes runtime settings for the ferc_to_sqlite graphs.

xbrl_num_workers: None | int
xbrl_batch_size: int = 50
pudl.ferc_to_sqlite.datastore(init_context) → pudl.workspace.datastore.Datastore[source]

Dagster resource to interact with Zenodo archives.

pudl.ferc_to_sqlite.ferc_to_sqlite_settings(init_context) → pudl.settings.FercToSqliteSettings[source]

Dagster resource for parameterizing the ferc_to_sqlite graph.

This resource allows us to specify the years we want to process for each datasource in the Dagit UI.

class pudl.ferc_to_sqlite.EtlSettings(_case_sensitive: bool | None = None, _env_prefix: str | None = None, _env_file: pydantic_settings.sources.DotenvType | None = ENV_FILE_SENTINEL, _env_file_encoding: str | None = None, _env_ignore_empty: bool | None = None, _env_nested_delimiter: str | None = None, _env_parse_none_str: str | None = None, _secrets_dir: str | pathlib.Path | None = None, **values: Any)[source]

Bases: pydantic_settings.BaseSettings

Main settings validation class.

ferc_to_sqlite_settings: FercToSqliteSettings | None
datasets: DatasetsSettings | None
name: str | None
title: str | None
description: str | None
version: str | None
publish_destinations: list[str] = []
classmethod from_yaml(path: str) → EtlSettings[source]

Create an EtlSettings instance from a yaml_file path.

Parameters:

path – path to a yaml file; this could be remote.

Returns:

An ETL settings object.

class pudl.ferc_to_sqlite.XbrlFormNumber(*args, **kwds)[source]

Bases: enum.Enum

Contains full list of supported FERC XBRL forms.

FORM1 = 1
FORM2 = 2
FORM6 = 6
FORM60 = 60
FORM714 = 714
pudl.ferc_to_sqlite.logger[source]
pudl.ferc_to_sqlite.ferc_to_sqlite()[source]

Clone the FERC FoxPro databases and XBRL filings into SQLite.

pudl.ferc_to_sqlite.default_resources_defs[source]
pudl.ferc_to_sqlite.ferc_to_sqlite_full[source]
pudl.ferc_to_sqlite.ferc_to_sqlite_fast_settings[source]
pudl.ferc_to_sqlite.ferc_to_sqlite_fast[source]
pudl.ferc_to_sqlite.defs: dagster.Definitions[source]

A collection of dagster assets, resources, IO managers, and jobs for the FERC to SQLite ETL.