Repository files navigation

Check out theblog post.

1. Fork/Clone

2. Create and activate a virtual environment

3. Install the requirements

4. Run the scrapers:

   # sync(env)$ python script.py headless# parallel with multiprocessing(env)$ python script_parallel_1.py headless# parallel with concurrent.futures(env)$ python script_parallel_2.py headless# concurrent with concurrent.futures (should be the fastest!)(env)$ python script_concurrent.py headless# parallel with concurrent.futures and concurrent with asyncio(env)$ python script_asyncio.py headless

5. Run the tests:

   (env)$ python -m pytest test/test_scraper.py(env)$ python -m pytest test/test_scraper_mock.py