Repository files navigation

The ZIM farm (zimfarm) is a semi-decentralised software solution tobuildZIM files efficiently. This means scraping Web contents,packaging them into a ZIM file and uploading the result to an onlineZIM files repository.

The Zimfarm platform is a combination of different tools:

Thedispatcher is a central database andAPI that recordsrecipes (metadata of ZIM to produce) andtasks. It includes a scheduler that decides when a ZIM file should be recreated (based on the recipe) and a dispatcher that creates and assignstasks toworkers.

Thefrontend, available atfarm.openzim.org is a simple consumer of the API.

It is used to create, clone and edit recipes, but also to monitor the evolution of tasks andworkers.

Anybody can use it in read-only mode.

Workers are always-running computers which gets assigned ZIM creation tasks by the dispatcher. If you are interested in providing us worker resources, pleaseread these instructions.

A worker is made of two software components:

Themanager is responsible for declaring its available resources and configuration and receives tasks assigned to it by the dispatcher. It's a very-low resources container whose job is to spawntask-worker ones.

Thetask-worker is responsible for running a specific task. It's also a very-low resources container but contrary to the manager, one is spawned for each task assigned to the worker (the manager defines the concurrency based on resources).

The task-worker's role is to start and monitor the scraper's container for the task and to spawn uploader containers for both created ZIM files and logs.

Theuploader is instantiated by the task-worker to upload, individually, each created ZIM files, as well as the scraper's container log.

The uploader supports both SCP and SFTP. We are currently using SFTP for all uploads due to a slight speed gain.

Uploader is very fast and convenient (can watch and resumes files) but works only off files at the moment.

Thereceiver is a jailed OpenSSH-server that receives scraper logs and ZIM files and pass the latter through a quarantine via thezimcheck tool which eventually either put them aside (invalid ZIM) or move those to thepublic download server.

Scrapers are the tools used to actually convert ascraping request (recorded in a Zimfarm recipe) into one or several ZIM files.

The most important one is the Mediawiki scraper, calledmwoffliner but there are many of them for Stack-Exchange, Project Gutenberg, PhET and others.

Scrapers are not part of the Zimfarm. Those are completely independent projects for which the requirements to integrate into the Zimfarm are minimal:

• Works completely off a docker image
• Arguments should be set on the command line
• ZIM output folder should be settable via an argument

ZIM file requests are handled onzim-requests repository.

If there's already a scraper for the website you want to convert to ZIM, someone with editor access to the Zimfarm will create the recipe and in a few days, a ZIM file should be available.