Repository files navigation

This directory contains the code of Wikipedia 1.0 supportingsoftware. More information about the Wikipedia 1.0 project can befoundon the Wikipedia inEnglish.

Thewp1 subdirectory includes code for updating theenwp10database, specifically theratings table (but also othertables). The library code itself isn't directly runnable, but insteadis loaded and run in various docker images that are maintained in thedocker directory.

requirements.txt is a list of python dependencies in pip format thatneed to be installed in a virtual env in order to run the library code.Both theweb andworkers docker images use the same requirements,thoughFlask and itsdependencies are not utilized by the worker code.

Thecron directory contains wrapper scripts for cron jobs that areruninside the workers image.

Thesetup directory contains a historical record of the databaseschema used by the tool for what is refered to in code as thewp10database. This file has been heavily edited, but should be able to beused to re-create theenwp10 database if necessary.

wp1-frontend contains the code for the Vue-CLI based frontend,which is encapsulated and served from thefrontend docker image.See that directory for instructions on how to setup a developmentenvironment for the frontend.

conf.json is a configuration file that is used by thewp1library code.

docker-compose.yml is a file read by thedocker-composecommand in order to generate thegraph of required docker images that represent the production environment.

docker-compose-dev.yml is a similar file which sets up a dev environment,with Redis and a MariaDB server for theenwp10 database. Use it like so

docker-compose -f docker-compose-dev.yml up -d

docker-compose-test.yml is a another docker file which sets up the test dbfor python "nosetests" (unit tests). Run it similarly:

docker-compose -f docker-compose-test.yml up -d

The*.dockerfile symlinks allow for each docker image in this repository(there are many) to be more easily organized.

openapi.yml is a YAML file that describes the API of theweb imageinOpenAPI format. If you visittheindex of the API server you willget a swagger-ui documentation frontend that utilizes this file. Itis symlinked into thewp1/web directory.

Thewp10_test.*.sql andwiki_test.*.sql files are roughapproximations of the schemas of the two databases that the libraryinterfaces with. They are used for unit testing.

This code is targeted to and tested on Python 3.12.0. For now, all developmenthas been on Linux, use other platforms at your own risk.

WP1 usesPipenv to managed dependencies.APipfile andPipfile.lock are provided. You should have the pipenv toolinstalled in your global Python install (not in a virtualenv):

pip3 install pipenv

Then you can use:

pipenv install --dev

Which will install the dependencies at the precise versions specified in thePipfile.lock file. Behind the scenes, Pipenv creates a virtualenv for youautomatically, which it keeps up to date when you run Pipenv commands. Youcan use thepipenv shell command to start a shell using the environment,which is similar to "activating" a virtualenv. You can also usepipenv runto run arbitrary individual shell commands within that environment. In manycases, it will be more convenient to use commands likepipenv run pytestthen actually spawning a subshell.

The frontend requiresNode.js version 18 to build andrun. Once node is installed, to install the requirements for the frontendserver, cd intowp1-frontend and use:

yarn install

If you do not have yarn, it can be installed with:

npm i -g yarn

You will also need to haveDocker on your systemin order to run the development server.

The script needs access to the enwiki_p replica database (referred toin the code aswikidb), as well as its own toolsdb application database(referred to in the code aswp10db). If you are a part of the toolforgeenwp10project, you canfind the credentials for these on toolforge in the replica.my.cnf file inthe tool's home directory. They need to be formatted in a way that isconsumable by the library and pymysql. Look atcredentials.py.exampleand create a copy calledcredentials.py with the relevant informationfilled in. The production version of this code also requires English WikipediaAPI credentials for automatically editing and updatingtables like this one.Currently, if your environment is DEVELOPMENT, jobs that utilize the APIto edit Wikipedia are disabled. There is no development wiki that gets editedat this time.

The "development" credentials files,credentials.py.dev andcredentials.py.dev.example are for running the docker graph of developmentresources. They are copied into the docker container that is run when usingdocker-compose-dev.yml.

Thecredentials.py file proper also contains a section for TEST databasecredentials. These are used in unit tests. If you use the database providedindocker-compose-test.yml you can copy these directly from the examplefile. However, you are free to provide your own test database that willbe destroyed after every test run. See the next section on running the tests.

The backend/python tests require a MariaDB or MySQL instance to connect to inorder to verify various statements and logic. This database does not need to bepersistent and in fact part of the test setup and teardown is to recreate (destroy)a fresh schema for the test databases each time. You also will need two databasesin your server:enwp10_test andenwikip_test. They can use default settingsand be empty.If you've followed the steps under 'Development' below tocreate a running dev database with docker-compose, you're all set.

If you have that, and you've already installed the requirements above,you should be able to simply run the following command from thisdirectory to run the tests:

pipenv run pytest

For frontend tests, you need to have a full working local developmentenvironment. You should follow the steps in 'Installation' above, as well as thesteps in 'Development' below. Your frontend should be running on port 5173 (thedefault) and the backend should be on port 5000 (also the default).

To run the tests:

cd wp1-frontend$(yarn bin)/cypress run

Then follow the GUI prompts to run "Electron E2E tests".

For development, you will need to have Docker installed as explained above.

There is a Docker setup for a development database. It lives indocker-compose-dev.yml.

Before you run the docker-compose command below, you must copy the filewp1/credentials.py.dev.example towp1/credentials.py.dev and fill out thesection forSTORAGE, if you wish to properly materialize builder lists intobackend selections.

After that is done, use the following command to run the dev environment:

docker-compose -f docker-compose-dev.yml up -d

See the instructions in the associatedREADME file

Using pipenv, you can start the API server with:

pipenv run flask --app wp1.web.app --debug run

Assuming you've installed the frontend deps (yarn install), the web frontendcan be started with the following command in thewp1-frontend directory:

yarn dev

The DEVELOPMENT section of credentials.py.example is already filled out withthe proper values for the servers listed in docker-compose-dev.yml. You shouldbe able to simply copy it to credentials.py.

If you wish to connect to a wiki replica database on toolforge, you will needto fill out your credentials in WIKIDB section. This is not required fordeveloping the frontend.

The API server has a built-in development overlay, currently used for manualupdate endpoints. What this means is that the endpoints defined inwp1.web.dev.projects are used with priority, instead of the production endpoints,only if the credentials.py ENV == Environment.DEVELOPMENT. This is to allowfor easier manual and CI testing of the manual update page.

If you wish to test the manual update job with a real Wikipedia replica databaseand RQ jobs, you will have to disable this overlay. The easiest way would be tochange the following line in wp1.web.app:

  if ENV == environment.Environment.DEVELOPMENT:    # In development, override some project endpoints, mostly manual    # update, to provide an easier env for developing the frontend.    print('DEVELOPMENT: overlaying dev_projects blueprint. '          'Some endpoints will be replaced with development versions')    app.register_blueprint(dev_projects, url_prefix='/v1/projects')

to something like:

  if false:  # false while manually testing    # In development, override some project endpoints, mostly manual    ...

Documentation lives atRead the Docs. It isbuilt usingmkdocs. The Read the Docs site automaticallymonitors the WP1 github HEAD and re-builds the documentation on every push.

If you are editing the docs and would like to view them locally before pushing:

$cd docs$ python -m venv venv$source venv/bin/activate$ pip install -r requirements.txt$cd ..$ mkdocs serve

Theserve command should print out the port to view the docs at, likely localhost:8000.

• Push to the release branch of the github repository:
  • git checkout main
  • git pull origin main
  • git checkout release
  • git merge main
  • git push origin release
• Wait for the release imagesto be built
• Log in to the box that contains the production docker images. It iscalled mwcurator.
• cd /data/code/wp1/
• sudo git pull origin main
• Pull the docker images from docker hub:
  • sudo docker pull ghcr.io/openzim/wp1-workers:release
  • sudo docker pull ghcr.io/openzim/wp1-web:release
  • sudo docker pull ghcr.io/openzim/wp1-frontend:release
• If you've made changes to the format or contents ofcredentials.py, update/data/wp1bot/credentials.py.
• Run docker-compose to bring the production images online.
  • sudo docker-compose up -d
• Run the production database migrations in the worker container:
  • sudo docker exec -ti -e PYTHONPATH=app wp1bot-workers yoyo -c /usr/src/app/db/production/yoyo.ini apply

This project is configured to use git pre-commit hooks managed by thePython programpre-commit (website). Pre-commit checks let us ensure that the code is properly formatted withyapf amongst other things.

If you've installed the requirements for this repository, the pre-commitbinary should be available to you. To install the hooks, use:

pre-commit install

Then, when you try to commit a change that would fail pre-commit, you get:

(venv) host:wikimedia_wp1_bot audiodude$ git commit -am 'Test commit'Trim Trailing Whitespace.................................................PassedFix End of Files.........................................................Passedyapf.....................................................................Failedhookid: yapf

From there, the pre-commit hook will have modified and thus unstaged some or allof the files you were trying to commit. Look through the changes to make surethey are sane, then re-add them with git add, before trying your commit again.

GPLv2 or later, seeLICENSE for more details.