Data Access

We publish the PUDL pipeline outputs in several ways to serve different users and use cases. We’re always trying to increase accessibility of the PUDL data, so if you have suggestions or questions please open a GitHub issue or email us at pudl@catalyst.coop.

How Should You Access PUDL Data?

We provide four primary ways of interacting with PUDL data. Here’s how to find out which one is right for you and your use case.

Access Method

Types of User

Use Cases

Datasette

Curious Explorer, Spreadsheet Analyst, Web Developer

Explore the PUDL database interactively in a web browser. Select data to download as CSVs for local analysis in spreadsheets. Create sharable links to a particular selection of data. Access PUDL data via a REST API.

Zenodo Archives

Researcher, Database User, Notebook Analyst

Use a stable, citable, fully processed version of the PUDL on your own computer. Use PUDL in Jupyer Notebooks running in a stable, archived Docker container. Access the SQLite DB and Parquet files directly using any toolset.

JupyterHub

New Python User, Notebook Analyst

Work through the PUDL example notebooks without any downloads or setup. Perform your own notebook-based analyses using PUDL data and limited computational resources.

Development Environment

Python Developer, Data Wrangler

Run the PUDL data processing pipeline on your own computer. Edit the PUDL source code and run the software tests and data validations. Integrate a new data source or newly released data from one of existing sources.

Data Packages

Deprecated

For working with our published data prior to v0.4.0

Datasette

We provide web-based access to the PUDL data via a Datasette deployment at https://data.catalyst.coop.

Datasette is an open source tool that wraps SQLite databases in an interactive front-end. It allows users to browse database tables, select portions of them using dropdown menus, build their own SQL queries, and download data to CSVs. It also creates a REST API allowing the data in the database to be queried programmatically. All the query parameters are stored in the URL so you can also share links to the data you’ve selected.

Note that only data that has been fully integrated into the SQLite databases are available here. Currently this includes the core PUDL database and our concatenation of all historical FERC Form 1 databases.

Zenodo Archives

We use Zenodo to archive our fully processed data as a SQLite databasees and Parquet files. We also archive a Docker image that contains the software environment required to use PUDL within Jupyter Notebooks. You can find all our archived data products in the Catalyst Cooperative Community on Zenodo.

  • The current version of the archived data and Docker container can be downloaded from This Zenodo archive

  • Detailed instructions on how to access the archived PUDL data using a Docker container can be found in our PUDL Examples repository.

  • The SQLite databases and Parquet files containing the PUDL data, the complete FERC 1 database, and EPA CEMS hourly data are contained in that same archive, if you want to access them directly without using PUDL.

Note

If you’re already familiar with Docker, you can also pull the image we use to run Jupyter directly:

$ docker pull catalystcoop/pudl-jupyter:latest

JupyterHub

We’ve set up a JupyterHub in collaboration with 2i2c.org to provide access to all of the processed PUDL data and the software environment required to work with it. You don’t have to download or install anything to use it, but we do need to create an account for you.

  • Request an account by submitting this form.

  • Once we’ve created an account for you follow this link to log in and open up the first example notebook on the JupyterHub.

  • You can create your own notebooks and upload, save, and download modest amounts of data on the hub.

We can only offer a small amount of memory (4-6GB) and processing power (1 CPU) per user on the JupyterHub for free. If you need to work with lots of data or do computationally intensive analysis, you may want to look into using the Zenodo Archives option on your own computer. The JupyterHub uses exactly the same data and software environment as the Zenodo Archives. Eventually we also want to offer paid access to the JupyterHub with plenty of computing power.

Development Environment

If you want to run the PUDL data processing pipeline yourself from scratch, run the software tests, or make changes to the source code, you’ll need to set up our development environment. This is a bit involved, so it has its own separate documentation.

Most users shouldn’t need to do this, and will probably find working with the pre-processed data via one of the other access modes easier. But if you want to contribute to the project please give it a shot!

Data Packages

Prior to v0.4.0 of PUDL we only published processed data as tabular data packages. In PUDL v0.4.0 we switched to distributing the SQLite databases and Apache Parquet files derived from data packages.

As of PUDL v0.5.0 we will generate SQLite and Apache Parquet outputs directly, and will no longer be archiving tabular data packages as the format of record. You can track Github #1176 for the status of this work in development.