- Notifications
You must be signed in to change notification settings - Fork2
Automation that scrapes USEPA github and provides that metadata for code.gov
License
USEPA/code-json-generator
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
This application generates a machine-readable software inventory compliantwithCode.gov Metadata Schema version 2.0.0. To accomplish this goal,the application connects to a series of Git hosting API "endpoints",inspects the configured projects, and generates an inventory. This application has been enhanced to automate the generation of theAgency's code.json file through GitHub Actions.
The application configuration supports internal/external hosting platformswith or without authentication (for private repositories). It currently workswith GitHub and GitLab REST APIs but could easily be extended for other hostingplatform APIs as well.
For each project, the application will attempt to use a developer-provided"code.json" snippet which is a JSON file containing the "releases" arrayfor the project as defined by theCode.gov Metadata Schema version 2.0.0.
If a "code.json" snippet file is not provided by the project maintainer, theapplication will attempt to infer sufficient/required properties from thetarget project and generate the best possible metadata for inclusion in theagency inventory.
GitHub Actions is being used to run a scheduled event to create the Agency's code.json file daily.
As mentioned earlier, if a "code.json" snippet file is not provided by the project maintainer, theapplication will attempt to infer sufficient/required properties from thetarget project and generate the best possible metadata for inclusion in theagency inventory.
The example EPA Agency code.json for the github.com/USEPA organization can befound here.
The code.json file can be generated by any of the GitHub Actions:
Thedaily.yml generates the code.json file to this repository'soutput branch. This workflow is enabled.
Thedaily-deploy.yml generates the code.json file to this repository'smaster branch. This workflow is disabled.
Thego-push (replacing Build and Deploy) generates the code.json file to this repository'smaster branch on the git push command. This workflow is disabled.
- output - includes the code.json file generated by thedaily.yml.
- cloudgov-deployment - provides intial code-json-generator GitHub Action workflow and removes 4000+ commit history generating the code.json file and deploying to cloud.gov.
- including-history and history-cloud.gov-deployment - provides intial code-json-generator GitHub Action workflow and includes 4000+ commit history generating the code.json file and deploying to cloud.gov.
This section describes the process to acquire and run the code-json-generatorapplication locally, GitHub, and in cloud.gov.
Follow these prerequisites for running code.json generator locally, in GitHub, and for deploying to cloud.gov.
This application is a command line Node application. Node must be installedon the system prior to running this application.
On the command line:
$ npm install -g code-json-generatorThe application requires a configuration file to dictate which repositoriesto include in the generated inventory. Anexample configuration fileis provided to help get started.
- Update the configuration file with your Agency's respective endpoints.
- Delete the existing create-agency-inventory-config.js file in the bin folder.
- Move the new configuration file from the etc folder into the bin folder.
- Rename the configuration file to create-agency-inventory-config.js.
Running the application locally will help identify the specific repositories with missing or invalid code.json files. After all repositories have a valid code.json file, running the application will generate the Agency code.json locally.
The application provides command-line usage syntax help via the--help switch.
$ create-inventory.js --help Usage: create-inventory [options] Options: -V, --version output the version number -c, --configFile <file> Configuration file -h, --help output usage informationTo run the application locally, you need to:
Change the working directory to the bin folder.
To generate an inventory, run the application and provide a configuration fileusing the
--configFile <file>switch. The generated inventory will, bydefault, be printed to STDOUT. This output can be redirected to a filefor persistence.
$ create-inventory.js --configFile ./create-agency-inventory.config.json > code.jsonNote: This example assumes a file containing proper configuration is locatedin the current working directory and is called
create-agency-inventory.config.json.Adjust this usage to suit actual work environment.
To run the application with GitHub Actions, you need to:
Change the working directory to the .github/workflows folder.
Open the daily.yml file.
Update the
scheduleevent per the desired frequency.Save, commit and push your changes.
You can deploy the code.json generator to Cloud.gov with GitHub Actions.
To do this you will need a Cloud.gov account and theCloud Foundry Command Line Tools installed on your machine.
- Login to cloud.gov
- Navigate to your organization's space which the app will be deployed to.
- Follow the documentation to create a cloud.gov service account plan:space deployer for continuous deployment to Cloud.gov.
- Add the service key username as the CG_USERNAME secret in the GitHub Repository's Secrets.
- Add the service key username as the CG_USERNAME secret in the GitHub Repository's Secrets.
You need to establish the name of your application before deploying it to cloud.gov.
Change the working directory to the etc/manifests folder.
Open the frontend.yml file.
Update the applications "name:" to whatever subdomain you wish to use (e.g., your-app-name).
Save, commit and push your changes.
You will see the code.json file located at: your-app-name.app.cloud.gov/code.json
You can modify how frequently the Agency's code.json file is generated and published to cloud.gov.
Change the working directory to the .github/workflows folder.
Open the daily-deploy.yml file.
Update the
scheduleevent per the desired frequency.Save, commit and push your changes.
To develop and possibly contribute to this project please review thecode of conduct andcontributing guidelines.
Potentially fork this repository and then clone the fork to obtain the sourcecode.
$ git clone <fork_urn>/code-json-generator.git$ cd code-json-generatorUpdates should be developed in a feature branch on the local clone of a fork.
$ git checkout -b feature-XCommit changes to feature branches and push feature branches to the forkedremote. Submit a pull request back to this upstream repository for reviewand integration. If the pull request fixes an open issue on this repository,include the textfixes #N (whereN is the issue number fixed) in thepull request title or description.
$ git commit -am 'Implemented feature'$ git push origin feature-XThis repository reused material from theUSGS code-json-generator repository.
The United States Environmental Protection Agency (EPA) GitHub project code is provided on an "as is" basis and the user assumes responsibility for its use. EPA has relinquished control of the information and no longer has responsibility to protect the integrity , confidentiality, or availability of the information. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation or favoring by EPA. The EPA seal and logo shall not be used in any manner to imply endorsement of any commercial product or activity by EPA or the United States Government.
About
Automation that scrapes USEPA github and provides that metadata for code.gov
Topics
Resources
License
Code of conduct
Contributing
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Releases
Packages0
Uh oh!
There was an error while loading.Please reload this page.
Languages
- JavaScript93.1%
- Shell6.9%