Repository files navigation

Developers Italia providesa catalog of Free and Open Sourcesoftware aimed to Public Administrations.

publiccode-crawler retrieves thepubliccode.yml files from therepositories of publishers found in theDevelopers Italia API.

publiccode-crawler can either run manually on the target machine or it can be deployedfrom a Docker container.

1. Renameconfig.toml.example toconfig.toml and set the variables

   NOTE: The application also supports environment variables in substitutionto config.toml file. Remember: "environment variables get higher priority thanthe ones in configuration file"

2. Build the binary withgo build

You can build the Docker image using

docker build .

or use the image published to DockerHub:

docker run -it italia/publiccode-crawler

Gets the list of publishers fromhttps://api.developers.italia.it/v1/publishersand starts to crawl their repositories.

Gets the list of publishers inpublishers*.yml and starts to crawltheir repositories.

Crawl just the software specified as parameter.It takes the software URL and its publisher id as parameters.

Ex.publiccode-crawler crawl-software https://api.developers.italia.it/v1/software/a2ea59b0-87cd-4419-b93f-00bed8a7b859 edb66b3d-3e36-4b69-aba9-b7c4661b3fdd

• crawler download-publishers downloads organizations and repositories fromtheonboarding portal repositoryand saves them to a publishers YAML file.

• developers-italia-api: the APIused to store the results of the crawling
• publiccode-parser-go: the Gopackage for parsing publiccode.yml files

Developers Italia is a project byAgID and theItalian Digital Team, which developed thecrawler and maintains this repository.