Repository files navigation

Demonstration of distributed web scraper forhttps://www.techinasia.com/jobs and its analytics data pipeline.

This guide assumes you have working installation of:

• Docker Compose
• Postgres
• Redis

• Create 2 databases in yourPostgres instance namedscraper andetl
• Clone this repo by rungit clone https://github.com/pythonjokeun/web-scraper-data-pipeline
• Enter the cloned repo.
• Open updocker-compose.yml in your favorite editor.
• Configure all the environment variables values accordingly.
• Rundocker-compose up --scale scraper-consumer=2

Notes:

• If your working installation ofPostgres orRedis running at your local machine, you can usehost.docker.internal as the[host] value.
• The scraper and ETL data pipeline execution schedules are defined in theCRON_SCHEDULE variables usingcron expression.
• The--scale scraper-consumer=2 argument is used to define number ofscraper-consumer instance.

Following image is the overview how the system looks like,