Repository files navigation

Repology is a service which monitorsa lot of package repositoriesand other sources and aggregates data on software package versions,reporting new releases and packaging problems.

This repository contains Repology updater code, a backend servicewhich updates the repository information. See also theweb application code.

• Python 3.11+
• Python moduleJinja2
• Python modulelibversion (also requireslibversion C library)
• Python modulepsycopg2
• Python modulepyyaml
• Python modulexxhash
• Python modulepydantic
• PostgreSQL 16.0+
• PostgreSQL extensionlibversion

Needed for fetching/parsing repository data:

• Python modulejsonslicer
• Python modulelxml
• Python moduleprotobuf
• Python modulepyparsing
• Python modulerequests
• Python modulerpm (comes with RPM package manager)
• Python modulerubymarshal
• Python module sqlite3 (part of Python, sometimes packaged separately)
• Python moduletomli
• Python moduleyarl
• Python modulezstandard
• git
• rsync

Optional, for doing HTML validation when running tests:

• Python modulepytidylib andtidy-html5 library

Optional, for checking schemas of configuration files:

• Python modulevoluptuous

Optional, for python code linting:

• Python moduleflake8
• Python moduleflake8-builtins
• Python moduleflake8-import-order
• Python moduleflake8-quotes
• Python modulemypy

Since repology rules live in separate repository you'll need toclone it first. The location may be arbitrary, butrules.dsubdirectory is what default configuration file points to, sousing it is the most simple way.

git clone https://github.com/repology/repology-rules.git rules.d

First, you may need to tune settings which are shared by all repologyutilities, such as directory for storing downloaded repository stateor DSN (string which specifies how to connect to PostgreSQL database).Seerepology.conf.default for default values, createrepology.confin the same directory to override them (don't editrepology.conf.default!)or specify path to alternative config viaREPOLOGY_SETTINGSenvironment variable, or override settings via command line.

By default, repology uses./_state directory for storing raw and parsedrepository data andrepology/repology/repology database/user/passwordon localhost.

For the following steps you'll need to set up the database. EnsurePostgreSQL server is up and running, and execute the followingcommands to create the database for repology:

psql --username postgres -c"CREATE DATABASE repology"psql --username postgres -c"CREATE USER repology WITH PASSWORD 'repology'"psql --username postgres -c"GRANT ALL ON DATABASE repology TO repology"psql --username postgres --dbname repology -c"GRANT CREATE ON SCHEMA public TO PUBLIC"psql --username postgres --dbname repology -c"CREATE EXTENSION pg_trgm"psql --username postgres --dbname repology -c"CREATE EXTENSION libversion"

in the case you want to change the credentials, don't forget to addthe actual ones torepology.conf.

Note that you need more than 11GiB of disk space for RepologyPostgreSQL database and additionally more than 11GiB space for rawand parsed repository data if you decide to run a complete updateprocess.

The fastest and most simple way to fill the database would be touse a databasedump of main Repologyinstance:

curl -s https://dumps.repology.org/repology-database-dump-latest.sql.zst| unzstd| psql -U repology

Another option would be to go through complete update process whichincludes fetching and parsing all repository data from scratch andpushing it to the database.

First, init the database schema:

./repology-update.py --initdb

Note that this command drops all existing data in Repology database,if any. You only need to run this command once.

Next, run the update process:

./repology-update.py --fetch --fetch --parse --database --postupdate

Expect it to take several hours the first time, subsequent updateswill be faster. You can use the same command to updated. Briefexplanation of options used here:

• --fetch tells the utility to fetch raw repository data(download files, scrape websites, clone git repos) into statedirectory. Note that it needs to be specified twice to allowupdating.
• --parse enables parsing downloaded data into internal formatwhich is also saved into state directory.
• --database pushes processed package data into the database.
• --postupdate runs optional cleanup tasks.

• How to extend or fixrules for package matching
• How repologycompares versions

• Dmitry Marakasovamdmi3@amdmi3.ru

GPLv3 or later, seeCOPYING.