Repository files navigation

👉🏽 👉🏽 👉🏽Full writeup:Flat Data Project 👈🏽 👈🏽 👈🏽

Flat Editor is a VSCode extension that steps you through the process of creating aFlat Data Action, which makes it easy to fetch data and commit it to your repository.

Flat Data is a GitHub action which makes it easy to fetch data and commit it to your repository as flatfiles. The action is intended to be run on a schedule, retrieving data from any supported target and creating a commit if there is any change to the fetched data. Flat Data builds on the“git scraping” approach pioneered by Simon Willison to offer a simple pattern for bringing working datasets into your repositories and versioning them, because developing against local datasets is faster and easier than working with data over the wire.

To use Flat Editor, firstinstall the extension.

If you're starting from an empty repository, invoke the VSCode Command Palette via the shortcutCmd+Shift+P and selectInitialize Flat YML File

This will generate aflat.yml file in the.github/workflows directory, and will open a GUI through which you can configure your Flat action.

At any given time, you can view the raw content of the underlying YML file via theView the raw YAML button in the GUI, or via the following button at the top right of your VSCode workspace.

Changes toflat.yml are saved automatically when using the GUI, but feel free to save manually viaCmd+S if the habit is as deeply engrained for you as it is for us 😀

Currently, Flat supports the ingestion of data via the two following sources:

1. Publicly accessible HTTP endpoint
2. SQL Database (accessible via connection string)

Flat assumes that you'd like to run your workflow on a given schedule, and to that end exposes a GUI for specifying a CRON job as part of the action definition. We've selected a handful of default values, but feel free to enter any valid CRON string here. We'll even validate the CRON for you as you type!

To create an HTTP action, you'll be asked for the following inputs:

1. A result filename (the filename and extension that your results will be written to, e.g.,vaccination-data.json).
2. A publicly accessible URL (we'll try to render a helpful inline preview of the response from this endpoint if we can)
3. An optional path to apostprocessing script, if you wish to perform further transformation or work on the fetched date

To create a SQL action, you'll be asked for the following inputs:

1. A result filename (the filename and extension that your results will be written to, e.g.,vaccination-data.json).
2. A path to a SQL query
3. A database connection string*
4. An optional path to apostprocessing script, if you wish to perform further transformation or work on the fetched date

* Note that we will encrypt this value and create aGitHub secret in your repository for this connection string. No sensitive data will be committed to your repository. Keep in mind that your repository must have an upstream remote on github.com in order for us to create the secret.

After you've added the requisite steps to your Flat action, push your changes to your GitHub repository. Your workflow should run automatically. Additionally, under the hood, the extension lists your optional postprocessing and/or SQL query files as workflow triggers, meaning the workflow will run anytime these files change. You can run your workflows manually, too, thanks to theworkflow_dispatch: {} value that the extension adds to your Flat action.

workflow_dispatch:{}push:paths:    -.github/workflows/flat.yml    -./rearrange-vax-data.ts

Deploy a new version with:

First make sure you're a part of the githubocto marketplace teamhere.

1. Get a PAThere (first time)
2. vsce login githubocto (first time)
3. vsce publish [minor|major|patch] This will create a new version and update package.json accordingly.
4. git push the change topackage.json

If you run into any trouble or have questions, feel free toopen an issue. Sharing yourflat.yml with us in the issue will help us understand what might be happening.

❤️ GitHub OCTO

MIT