Data Access
We publish the PUDL pipeline outputs in several ways to serve different users and use cases. We’re always trying to increase accessibility of the PUDL data, so if you have suggestions or questions please open a GitHub issue or email us at pudl@catalyst.coop.
How Should You Access PUDL Data?
We provide four primary ways of interacting with PUDL data. Here’s how to find out which one is right for you and your use case.
Access Method |
Types of User |
Use Cases |
|---|---|---|
Curious Explorer, Spreadsheet Analyst, Web Developer |
Explore the PUDL database interactively in a web browser. Select data to download as CSVs for local analysis in spreadsheets. Create sharable links to a particular selection of data. Access PUDL data via a REST API. |
|
Researcher, Database User, Notebook Analyst |
Use a stable, citable, fully processed version of the PUDL on your own computer. Use PUDL in Jupyer Notebooks running in a stable, archived Docker container. Access the SQLite DB and Parquet files directly using any toolset. |
|
New Python User, Notebook Analyst |
Work through the PUDL example notebooks without any downloads or setup. Perform your own notebook-based analyses using PUDL data and limited computational resources. |
|
Python Developer, Data Wrangler |
Run the PUDL data processing pipeline on your own computer. Edit the PUDL source code and run the software tests and data validations. Integrate a new data source or newly released data from one of existing sources. |
|
Deprecated |
For working with our published data prior to v0.4.0 |
Datasette
We provide web-based access to the PUDL data via a Datasette deployment at https://data.catalyst.coop.
Datasette is an open source tool that wraps SQLite databases in an interactive front-end. It allows users to browse database tables, select portions of them using dropdown menus, build their own SQL queries, and download data to CSVs. It also creates a REST API allowing the data in the database to be queried programmatically. All the query parameters are stored in the URL so you can also share links to the data you’ve selected.
Note that only data that has been fully integrated into the SQLite databases are available here. Currently this includes the core PUDL database and our concatenation of all historical FERC Form 1 databases.
Zenodo Archives
We use Zenodo to archive our fully processed data as a SQLite databasees and Parquet files. We also archive a Docker image that contains the software environment required to use PUDL within Jupyter Notebooks. You can find all our archived data products in the Catalyst Cooperative Community on Zenodo.
The current version of the archived data and Docker container can be downloaded from This Zenodo archive
Detailed instructions on how to access the archived PUDL data using a Docker container can be found in our PUDL Examples repository.
The SQLite databases and Parquet files containing the PUDL data, the complete FERC 1 database, and EPA CEMS hourly data are contained in that same archive, if you want to access them directly without using PUDL.
Note
If you’re already familiar with Docker, you can also pull the image we use to run Jupyter directly:
$ docker pull catalystcoop/pudl-jupyter:latest
JupyterHub
We’ve set up a JupyterHub in collaboration with 2i2c.org to provide access to all of the processed PUDL data and the software environment required to work with it. You don’t have to download or install anything to use it, but we do need to create an account for you.
Request an account by submitting this form.
Once we’ve created an account for you follow this link to log in and open up the first example notebook on the JupyterHub.
You can create your own notebooks and upload, save, and download modest amounts of data on the hub.
We can only offer a small amount of memory (4-6GB) and processing power (1 CPU) per user on the JupyterHub for free. If you need to work with lots of data or do computationally intensive analysis, you may want to look into using the Zenodo Archives option on your own computer. The JupyterHub uses exactly the same data and software environment as the Zenodo Archives. Eventually we also want to offer paid access to the JupyterHub with plenty of computing power.
Development Environment
If you want to run the PUDL data processing pipeline yourself from scratch, run the software tests, or make changes to the source code, you’ll need to set up our development environment. This is a bit involved, so it has its own separate documentation.
Most users shouldn’t need to do this, and will probably find working with the pre-processed data via one of the other access modes easier. But if you want to contribute to the project please give it a shot!
Data Packages
Prior to v0.4.0 of PUDL we only published processed data as tabular data packages. In PUDL v0.4.0 we switched to distributing the SQLite databases and Apache Parquet files derived from data packages.
As of PUDL v0.5.0 we will generate SQLite and Apache Parquet outputs directly, and will no longer be archiving tabular data packages as the format of record. See #1176, #1211 for details on the change.