flags/

Setting up Nginx and Vaurien

This text explains how to configure Nginx and Vaurien to test HTTP client code while avoiding network traffic and introducing simulated delays and errors. This setup is necessary if you want to experiment with the ``flags2*.py`` image download examples in this directory -- covered in chapters 17 and 18 of Fluent Python.

Overview

The flag download examples are designed to compare the performance of different approaches to finding and downloading files from the Web. However, we don't want to hit a public server with multiple requests per second while testing, and we want to be able to simulate high latency and random network errors.

For this setup I chose Nginx as the HTTP server because it is very fast and easy to configure, and Toxiproxy — designed by Shopify to introduce delays and network errors for testing distributed systems.

The archive ``flags.zip``, contains a directory ``flags/`` with 194 subdirectories, each containing a ``.gif`` image and a ``metadata.json`` file. These are public-domain images copied from the `CIA World Fact Book<https://www.cia.gov/library/publications/the-world-factbook/>`_.

Once these files are unpacked to the ``flags/`` directory and Nginx is configured, you can experiment with the ``flags2*.py`` examples without hitting the network.

Procedure

1. Unpack test data

The instructions in this section are for GNU/Linux or OSX using the command line. Windows users should have no difficulty doing the same operations with the Windows Exporer GUI.

Unpack the initial data in the ``fancy_flags/`` directory::

Verify that 194 directories are created in ``fancy_flags/flags/``, each with a ``.gif`` and a ``metadata.json`` file::

2. Install Nginx

Download and install Nginx. I used version 1.6.2 -- the latest stable version as I write this.

* Download page: http://nginx.org/en/download.html

* Beginner's guide: http://nginx.org/en/docs/beginners_guide.html

3. Configure Nginx

Edit the the ``nginx.conf`` file to set the port and document root. You can determine which ``nginx.conf`` is in use by running::

The output starts with::

Among the configure arguments you'll see ``--conf-path=``. That's the file you will edit.

Most of the content in ``nginx.conf`` is within a block labeled ``http`` and enclosed in curly braces. Within that block there can be multiple blocks labeled ``server``. Add another ``server`` block like this one::

After editing ``nginx.conf`` the server must be started (if it's not running) or told to reload the configuration file::

To test the configuration, open the URL http://localhost:8001/flags/ad/ad.gif in a browser. You should see the blue, yellow and red flag of Andorra.

If the test fails, please double check the procedure just described and refer to the Nginx documentation.

At this point you may run the ``flags_*2.py`` examples against the Nginx install by providing the ``--server LOCAL`` command line option. For example::

Note that Nginx is so fast that you will not see much difference in run time between the sequential and the concurrent versions. For more realistic testing with simulated network lag, we need to set up Toxiproxy.

4. Install and run Toxiproxy

Install...

In one terminal:

$ toxiproxy-server

In another terminal:

$ toxiproxy-cli create nginx_flags_delay -l localhost:8002 -u localhost:8001

Created new proxy nginx_flags_delay

$ toxiproxy-cli toxic add nginx_flags_delay -t latency -a latency=500

Added downstream latency toxic 'latency_downstream' on proxy 'nginx_flags_delay'

This creates an HTTP proxy on port 8002 which adds a 0.5s delay to every response. You can test it with a browser on port 8002: http://localhost:8002/flags/ad/ad.gif -- the flag of Andorra should appear after ½ second.

TODO: UPDATE NEXT PARAGRAPH

There is also the ``XXX`` script which runs a proxy on port 8003 producing errors in 25% of the responses and a .5 s delay to 50% of the responses. You can also test it with the browser on port 8003, but rememeber that errors are expected.

Platform-specific instructions

Nginx setup on Mac OS X

Homebrew (copy & paste code at the bottom of http://brew.sh/)::

Download and unpack::

Docroot is: /usr/local/var/www

/usr/local/etc/nginx/nginx.conf

::

Nginx setup on Lubuntu 14.04.1 LTS

Docroot is: /usr/share/nginx/html

AD AE AF AG AL AM AO AR AT AU AZ BA BB BD BE BF BG BH BI BJ BN BO BR BS BT

BW BY BZ CA CD CF CG CH CI CL CM CN CO CR CU CV CY CZ DE DJ DK DM DZ EC EE

EG ER ES ET FI FJ FM FR GA GB GD GE GH GM GN GQ GR GT GW GY HN HR HT HU ID

IE IL IN IQ IR IS IT JM JO JP KE KG KH KI KM KN KP KR KW KZ LA LB LC LI LK

LR LS LT LU LV LY MA MC MD ME MG MH MK ML MM MN MR MT MU MV MW MX MY MZ NA

NE NG NI NL NO NP NR NZ OM PA PE PG PH PK PL PT PW PY QA RO RS RU RW SA SB

SC SD SE SG SI SK SL SM SN SO SR SS ST SV SY SZ TD TG TH TJ TL TM TN TO TR

TT TV TW TZ UA UG US UY UZ VA VC VE VN VU WS YE ZA ZM ZW

fromflags2_commonimportmain,HTTPStatus,Result,save_flag

classFetchError(Exception):# <1>

def__init__(self,country_code):

self.country_code=country_code

asyncdefget_flag(session,base_url,cc):# <2>

url='{}/{cc}/{cc}.gif'.format(base_url,cc=cc.lower())

asyncwithsession.get(url)asresp:

ifresp.status==200:

returnawaitresp.read()

elifresp.status==404:

raiseweb.HTTPNotFound()

else:

raiseaiohttp.HttpProcessingError(

code=resp.status,message=resp.reason,

headers=resp.headers)

asyncdefdownload_one(session,cc,base_url,semaphore,verbose):# <3>

try:

asyncwithsemaphore:# <4>

image=awaitget_flag(session,base_url,cc)# <5>

exceptweb.HTTPNotFound:# <6>

status=HTTPStatus.not_found

exceptExceptionasexc:

raiseFetchError(cc)fromexc# <7>

else:

save_flag(image,cc.lower()+'.gif')# <8>

status=HTTPStatus.ok

ifverboseandmsg:

print(cc,msg)

returnResult(status,cc)

asyncdefdownloader_coro(cc_list,base_url,verbose,concur_req):# <1>

counter=collections.Counter()

semaphore=asyncio.Semaphore(concur_req)# <2>

asyncwithaiohttp.ClientSession()assession:# <8>

to_do= [download_one(session,cc,base_url,semaphore,verbose)

forccinsorted(cc_list)]# <3>

to_do_iter=asyncio.as_completed(to_do)# <4>

ifnotverbose:

to_do_iter=tqdm.tqdm(to_do_iter,total=len(cc_list))# <5>

forfutureinto_do_iter:# <6>

try:

exceptFetchErrorasexc:# <8>

country_code=exc.country_code# <9>

try:

error_msg=exc.__cause__.args[0]# <10>

exceptIndexError:

error_msg=exc.__cause__.__class__.__name__# <11>

ifverboseanderror_msg:

print(msg.format(country_code,error_msg))

status=HTTPStatus.error

else:

status=res.status

counter[status]+=1# <12>

defdownload_many(cc_list,base_url,verbose,concur_req):

loop=asyncio.get_event_loop()

coro=downloader_coro(cc_list,base_url,verbose,concur_req)

counts=loop.run_until_complete(coro)# <14>

loop.close()# <15>

if__name__=='__main__':

main(download_many,DEFAULT_CONCUR_REQ,MAX_CONCUR_REQ)