Repository files navigation

• Running the MetaCPAN stack with Docker (via Docker Compose)
• Quick Start
• Working with Containers
  • Building Containers
  • Accessing Containers
  • Accessing Services
    • web
    • api
    • Elasticsearch
    • PostgreSQL
    • grep
• System architecture
  • Thebin/metacpan-docker script
    • bin/metacpan init
    • bin/metacpan localapi
    • bin/metacpan-docker pull
    • bin/metacpan-docker reset
    • bin/metacpan-docker build/up/down/start/stop/run/ps/top...
  • Services
    • web
    • api
    • grep
      • Setting up a partial CPAN in theapi service
      • Bootstrapping theelasticsearch indices
      • Putting the above all together
    • elasticsearch and elasticsearch_test
• Tips and tricks
  • Running your own miniCPAN inside metacpan-docker
  • Running tests
  • Updating Carton dependencies
  • Updating the git repositories
  • Running Kibana to peek into Elasticsearch data
• Peeking Inside the Container
• To Do
• See also

Notice: This project is in experimental stage. It works, but there are a lotof things to be done better. Please use it and create Issues with your problems.

InstallDocker andDocker Compose (v2+) for your platform.Docker forMac orDocker for Windows will install both tools for you, if you areon either of these environments.

On Debian / Ubuntu, install using:

apt install docker-compose-v2

On Linux, Docker's default implementation only allowsroot user access toDocker commands and to control containers. In order to allow a regular user toaccess docker follow thepost-installation instructions.This document assumes the post-installation steps have been followed for thecurrent user.

You will also need Docker buildx, and to enable Docker BuildKit. They should beset up by default when using Docker Desktop, but on Linux you may need toinstall them. buildx is thedocker-buildx package on Debian based systems.Docker BuildKit can be enabled by following theGetting Startedinstructions.

On Debian / Ubuntu, install using:

apt install docker-buildx

If you are running a Mac ARM64 system, you will need to manually tell docker touse the x86_64 version of Elasticsearch 2.4. This can be done by running thecommand:

docker pull elasticsearch:2.4 --platform=linux/x86_64

It is highly recommended that you aliasdocker compose tofig (its originalname) and use it whereverdocker compose is used. You are going to have totype this command a lot.

Then, clone this repo and set up the environment:

git clone https://github.com/metacpan/metacpan-docker.gitcd metacpan-dockerbin/metacpan-docker init

Thebin/metacpan-docker init command clones the source repositories for:

• metacpan-web
• metacpan-api
• metacpan-grep-front-end
• metacpan-cpan-extracted-lite

These repositories are automatically mounted into the appropriate dockercontainers allowing the developer to use their preferred tools to work with thesource code.

Thedocker compose --profile dev up command on its own will bring up theentire stack in the foreground (logs will be displayed).

Thedocker compose --profile dev up command will also fetch the officialcontainer images fromMetaCPAN Docker Hubrepositories.

This will build the Docker containers for MetaCPAN, PostgreSQL and Elasticsearchservices (which will take a while, especially on a fresh first time install ofDocker) and run the services.

Don't forget to seed the localmetacpan-api with a partial CPAN; run thefollowing command in a separate terminal to get yourself up to speed:

docker compose exec api index-cpan.sh

This will prompt you to confirm removing old indices and setting up mappings onthe Elasticsearch service (sayYES). It will then proceed to rsync a partialCPAN in/CPAN for its metadata to be imported.

After the initialization above completes, the next step is to start the webfrontend with the following command:

docker compose up web-server

Once that is done, you should be able to see your local partial CPAN datain e.g.http://localhost:5001/recent andelsewhere.

From here, you can proceed and hack on the MetaCPAN code atsrc/metacpan-apiand/orsrc/metacpan-web directories, and saving edits will reload thecorresponding apps automatically!

When done hacking (or, more likely, when you need to rebuild/refresh your Dockerenvironment) you can then run

docker compose --profile dev down

in another terminal to stop all MetaCPAN services and remove the containers.

For further details, read on!

You can (re)build arbitrary containers. For instance, if you want to rebuild theapi container:

docker compose build api

Containers are accessible via thedocker compose exec command followed by thecontainer and then the command to execute. For example, to start a shell promptin theapi container:

docker compose exec api /bin/bash

Executing tests viaprove inside the API container:

docker compose exec api_test prove -lvr \  t/00_setup.t \  t/01_darkpan.t \  t/api/controller/cover.t

To access thepsql command line client in the PostgreSQL container:

docker compose exec pgdb psql

Each container is responsible for a different service. Some of these servicesare available in the developer environment via ports on the host system.

The local instance of the web front end is accessible via:

• http://localhost:5001
• http://web.metacpan.localhost

• http://localhost:5000
• http://api.metacpan.localhost

Theelasticsearch andelasticsearch_test containers are not exposeddirectly. They are available via theapi andapi_test containers.

You can query theelasticsearch container via:

docker compose exec elasticsearch curl http://localhost:9200

You can query theelasticsearch_test container via:

docker compose exec elasticsearch_test curl http://localhost:9200

The PostgreSQL service by default is only accessible from other containers.

The grep metacpan front end is accessible via:

• http://localhost:3000

Note: this is using a smaller, frozen version ofmetacpan-cpan-extracted viametacpan-cpan-extracted-lite.

The system consists of several services that live in docker containers:

• web — the web interface onhttp://localhost:5001
• api — the main server onhttp://localhost:5000
• api_test — the api server for running tests viaprove
• elasticsearch — database forapi
• elasticsearch_test — database forapi_test
• pgdb - PostgreSQL database container
• logspout - Docker log interface tohoneycomb.io
• grep - the web interface for grep.metacpan onhttp://localhost:3001

These services use one or more Docker volumes:

• metacpan_cpan: holds the CPAN archive, mounted in/CPAN
• metacpan_elasticsearch: holds the Elasticsearch database files
• metacpan_elasticsearch_test: holds the Elasticsearch test database files
• metacpan_api_carton andmetacpan_web_carton: holds the dependenciesinstalled byCarton for theapi andweb services, respectively;mounted on/carton instead oflocal, to prevent clashing with the hostuser's Carton
• metacpan_git_shared: points to the git repo containing all extracted CPANversions. This is mounted in/shared/metacpan_git. This can be eithermetacpan-cpan-extracted ormetacpan-cpan-extracted-lite. The volume isbound to the local repo at${PWD}/src/metacpan-cpan-extracted.

Docker Compose is used to, uh,compose them all together into one system.Usingdocker compose directly is a mouthful, however, so putting this alltogether is done via thebin/metacpan-docker script to simplify setup andusage (and to get you started hacking on the MetaCPAN sooner!)

bin/metacpan-docker is a thin wrapper around thedocker compose command,providing the environment variables necessary to run a basic MetaCPANenvironment. It provides these subcommands:

Theinit subcommand basically clones themetacpan-api andmetacpan-web repositories, and sets up the git commit hooks for each ofthem, in preparation for futuredocker compose orbin/metacpan-docker localapi commands.

It also clones themetacpan-grep-front-end andmetacpan-cpan-extracted-literepositories.

Thelocalapi subcommand adds the necessary configuration fordocker composeto run both themetacpan-web andmetacpan-api services, along withelasticsearch and Docker volumes. Under the hood, it customizes theCOMPOSE_FILE andCOMPOSE_PROJECT_NAME environment variables used bydocker compose to use additional YAML configuration files aside from thedefaultdocker compose.yml.

This is used to update all the git repository insrc/*. This will stay on yourcurrent local branch.

This is used to reset all the git repositories insrc/* to their latestversion onupstream/master. This will fail if you have some uncommitted localchanges. You should then commit or cancel your changes before re-running thecommand.

As noted earlier,bin/metacpan-docker is a thin wrapper arounddocker compose, so commands likeup,down, andrun will work as expectedfromdocker compose. See thedocker compose docs for an overview ofavailable commands.

Theweb service is a checkout ofmetacpan-web, built as a Docker image.Running this service alone is enough if you want to just hack on the frontend,since by default the service is configured to talk tohttps://fastapi.metacpan.org for its backend; ifthis is what you want, then you can simply invokedocker compose up ordocker compose up web.

Theapi service is a checkout ofmetacpan-api, built as a Docker image, justlike theweb service.

If using this service to run a local backend, you will need to run someadditional commands in a separate terminal oncebin/metacpan-docker localapi up runs.

Thegrep service is a checkout ofmetacpan-grep-front-end, built as a Dockerimage. Note that this is using themetacpan_git_shared volume, which requiresthe git repo formetacpan-cpan-extracted which can be initialized by running:

./bin/metacpan-docker init

Running

bin/metacpan-docker localapi exec api partial-cpan-mirror.sh

willrsync modules from selected CPAN authors, plus the package and authorindices, into theapi service's/CPAN directory. This is nearly equivalentto the same script in the (now deprecated) [metacpan-developer][8] repository.[8]:https://github.com/metacpan/metacpan-developer

Running

bin/metacpan-docker localapi exec api bin/run bin/metacpan mapping --deletebin/metacpan-docker localapi exec api bin/run bin/metacpan release /CPAN/authors/idbin/metacpan-docker localapi exec api bin/run bin/metacpan latestbin/metacpan-docker localapi exec api bin/run bin/metacpan author

in sequence will create the indices and mappings in theelasticsearch service,and import the/CPAN data intoelasticsearch.

If you're impatient or too lazy to do all the above, just running

bin/metacpan-docker localapi exec api index-cpan.sh

instead will set it all up for you.

Theelasticsearch andelasticsearch_test services use the officialElasticsearch Docker image, configured with settings and scripts taken fromthemetacpan-puppet repository. Theapi service depends on theelasticsearch service and theapi_test service depends on theelasticsearch_test services.

Suppose you have a local minicpan in/home/ftp/pub/CPAN. If you would like touse this inmetacpan-docker, then edit thedocker compose.localapi.yml tochange theapi service's volume mounts to use your local minicpan as/CPAN,e.g.:

services:api:volumes:      -/home/ftp/pub/CPAN:/CPAN...

Note that if you want CPAN author data indexed into Elasticsearch, your minicpanshould includeauthors/00whois.xml. Full indexing would take a better part ofa day or two, depending on your hardware.

Usebin/metacpan-docker run and similar:

# Run tests for metacpan-web against fastapi.metacpan.orgbin/metacpan-docker exec web bin/prove# Run tests for metacpan-web against a local apibin/metacpan-docker localapi exec web bin/prove# Run tests for metacpan-api against a local elasticsearch_testbin/metacpan-docker localapi exec api bin/prove

Because both theapi andweb services are running inside cleanPerlcontainers, it is possible to maintain a clean set of Carton dependenciesindependent of your host machine's perl. Just update thecpanfile of theproject, and run

bin/metacpan-docker exec web carton install# orbin/metacpan-docker exec api carton install

Due to the way the Compose services are configured, these commands will updatethe correspondingcpanfile.snapshot safely, even if you door don't have alocal directory (internally, the containers'local directory is placed in/carton instead, to prevent interfering with the host user's ownlocalCarton directory.)

You can usebin/metacpan-docker pull to update allsrc/* directories.

By default, thedocker compose.localapi.yml configures theelasticsearchservice to listen on the Docker host athttp://localhost:9200, and is also accessible via theDockerdefault network address ofhttp://172.17.0.1:9200; you can inspect it via simplecurl orwget requests, or use aKibana container, e.g.

docker run --rm -p 5601:5601 -e ELASTICSEARCH_URL=http://172.17.0.1:9200 -it kibana:4.6

Running the above will provide a Kibana container athttp://localhost:5601, which you can configure to haveit read thecpan* index in theelasticsearch service.

It is also certainly possible to run Kibana as part of the compose setup, byconfiguring e.g. akibana service.

If you rundocker ps you'll see the containers. You might see something like:

$ docker psCONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS                          PORTS                              NAMES2efb9c475c83        metacpan-web:latest"carton exec plackup…"   12 hours ago        Up 12 hours                     0.0.0.0:5001->5001/tcp             metacpan_web_18850110e06d8        metacpan-api:latest"/wait-for-it.sh db:…"   12 hours ago        Up 12 hours                     0.0.0.0:5000->5000/tcp             metacpan_api_17686d7ea03c6        postgres:9.6-alpine"docker-entrypoint.s…"   12 hours ago        Up 12 hours (healthy)           0.0.0.0:5432->5432/tcp             metacpan_pgdb_1c7de256d29b2        elasticsearch:2.4"/docker-entrypoint.…"   5 months ago        Up 26 hours                     0.0.0.0:9200->9200/tcp, 9300/tcp   metacpan_elasticsearch_1f1e04fe53598        elasticsearch:2.4"/docker-entrypoint.…"   5 months ago        Up 26 hours                     9300/tcp, 0.0.0.0:9900->9200/tcp   metacpan_elasticsearch_test_1

You can then use the container name to get shell access. For instance, to log into the API container:

docker exec -it metacpan_api_1 /bin/bash

• Integrate all other MetaCPAN services
• Add more Tips and tricks (as we continue hacking MetaCPAN in Docker)
• Provide a "near-production" Docker Compose configuration, suitable for DockerSwarm, and/or
• Refactor configuration to be suitable for Kubernetes (Google Cloud)deployments

• Docker Compose documentation
• metacpan-developer andmetacpan-puppet from which much informationabout the architecture is based on