Make a Request¶

Making a request with Requests is very simple.

Begin by importing the Requests module:

>>>importrequests

Now, let’s try to get a webpage. For this example, let’s get GitHub’s publictimeline:

>>>r=requests.get('https://api.github.com/events')

Now, we have aResponse object calledr. We canget all the information we need from this object.

Requests’ simple API means that all forms of HTTP request are as obvious. Forexample, this is how you make an HTTP POST request:

>>>r=requests.post('https://httpbin.org/post',data={'key':'value'})

Nice, right? What about the other HTTP request types: PUT, DELETE, HEAD andOPTIONS? These are all just as simple:

>>>r=requests.put('https://httpbin.org/put',data={'key':'value'})>>>r=requests.delete('https://httpbin.org/delete')>>>r=requests.head('https://httpbin.org/get')>>>r=requests.options('https://httpbin.org/get')

That’s all well and good, but it’s also only the start of what Requests cando.

Passing Parameters In URLs¶

You often want to send some sort of data in the URL’s query string. Ifyou were constructing the URL by hand, this data would be given as key/valuepairs in the URL after a question mark, e.g.httpbin.org/get?key=val.Requests allows you to provide these arguments as a dictionary of strings,using theparams keyword argument. As an example, if you wanted to passkey1=value1 andkey2=value2 tohttpbin.org/get, you would use thefollowing code:

>>>payload={'key1':'value1','key2':'value2'}>>>r=requests.get('https://httpbin.org/get',params=payload)

You can see that the URL has been correctly encoded by printing the URL:

>>>print(r.url)https://httpbin.org/get?key2=value2&key1=value1

Note that any dictionary key whose value isNone will not be added to theURL’s query string.

You can also pass a list of items as a value:

>>>payload={'key1':'value1','key2':['value2','value3']}>>>r=requests.get('https://httpbin.org/get',params=payload)>>>print(r.url)https://httpbin.org/get?key1=value1&key2=value2&key2=value3

Response Content¶

We can read the content of the server’s response. Consider the GitHub timelineagain:

>>>importrequests>>>r=requests.get('https://api.github.com/events')>>>r.text'[{"repository":{"open_issues":0,"url":"https://github.com/...

Requests will automatically decode content from the server. Most unicodecharsets are seamlessly decoded.

When you make a request, Requests makes educated guesses about the encoding ofthe response based on the HTTP headers. The text encoding guessed by Requestsis used when you accessr.text. You can find out what encoding Requests isusing, and change it, using ther.encoding property:

>>>r.encoding'utf-8'>>>r.encoding='ISO-8859-1'

If you change the encoding, Requests will use the new value ofr.encodingwhenever you callr.text. You might want to do this in any situation whereyou can apply special logic to work out what the encoding of the content willbe. For example, HTML and XML have the ability to specify their encoding intheir body. In situations like this, you should user.content to find theencoding, and then setr.encoding. This will let you user.text withthe correct encoding.

Requests will also use custom encodings in the event that you need them. Ifyou have created your own encoding and registered it with thecodecsmodule, you can simply use the codec name as the value ofr.encoding andRequests will handle the decoding for you.

Binary Response Content¶

You can also access the response body as bytes, for non-text requests:

>>>r.contentb'[{"repository":{"open_issues":0,"url":"https://github.com/...

Thegzip anddeflate transfer-encodings are automatically decoded for you.

Thebr transfer-encoding is automatically decoded for you if a Brotli librarylikebrotli orbrotlicffi is installed.

For example, to create an image from binary data returned by a request, you canuse the following code:

>>>fromPILimportImage>>>fromioimportBytesIO>>>i=Image.open(BytesIO(r.content))

JSON Response Content¶

There’s also a builtin JSON decoder, in case you’re dealing with JSON data:

>>>importrequests>>>r=requests.get('https://api.github.com/events')>>>r.json()[{'repository': {'open_issues': 0, 'url': 'https://github.com/...

In case the JSON decoding fails,r.json() raises an exception. For example, ifthe response gets a 204 (No Content), or if the response contains invalid JSON,attemptingr.json() raisesrequests.exceptions.JSONDecodeError. This wrapper exceptionprovides interoperability for multiple exceptions that may be thrown by differentpython versions and json serialization libraries.

It should be noted that the success of the call tor.json() doesnotindicate the success of the response. Some servers may return a JSON object in afailed response (e.g. error details with HTTP 500). Such JSON will be decodedand returned. To check that a request is successful, user.raise_for_status() or checkr.status_code is what you expect.

Raw Response Content¶

In the rare case that you’d like to get the raw socket response from theserver, you can accessr.raw. If you want to do this, make sure you setstream=True in your initial request. Once you do, you can do this:

>>>r=requests.get('https://api.github.com/events',stream=True)>>>r.raw<urllib3.response.HTTPResponse object at 0x101194810>>>>r.raw.read(10)b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03'

In general, however, you should use a pattern like this to save what is beingstreamed to a file:

withopen(filename,'wb')asfd:forchunkinr.iter_content(chunk_size=128):fd.write(chunk)

UsingResponse.iter_content will handle a lot of what you would otherwisehave to handle when usingResponse.raw directly. When streaming adownload, the above is the preferred and recommended way to retrieve thecontent. Note thatchunk_size can be freely adjusted to a number thatmay better fit your use cases.

Custom Headers¶

If you’d like to add HTTP headers to a request, simply pass in adict to theheaders parameter.

For example, we didn’t specify our user-agent in the previous example:

>>>url='https://api.github.com/some/endpoint'>>>headers={'user-agent':'my-app/0.0.1'}>>>r=requests.get(url,headers=headers)

Note: Custom headers are given less precedence than more specific sources of information. For instance:

• Authorization headers set withheaders= will be overridden if credentialsare specified in.netrc, which in turn will be overridden by theauth=parameter. Requests will search for the netrc file at~/.netrc,~/_netrc,or at the path specified by theNETRC environment variable.Check details innetrc authentication.

• Authorization headers will be removed if you get redirected off-host.

• Proxy-Authorization headers will be overridden by proxy credentials provided in the URL.

• Content-Length headers will be overridden when we can determine the length of the content.

Furthermore, Requests does not change its behavior at all based on which custom headers are specified. The headers are simply passed on into the final request.

Note: All header values must be astring, bytestring, or unicode. While permitted, it’s advised to avoid passing unicode header values.

More complicated POST requests¶

Typically, you want to send some form-encoded data — much like an HTML form.To do this, simply pass a dictionary to thedata argument. Yourdictionary of data will automatically be form-encoded when the request is made:

>>>payload={'key1':'value1','key2':'value2'}>>>r=requests.post('https://httpbin.org/post',data=payload)>>>print(r.text){  ...  "form": {    "key2": "value2",    "key1": "value1"  },  ...}

Thedata argument can also have multiple values for each key. This can bedone by makingdata either a list of tuples or a dictionary with listsas values. This is particularly useful when the form has multiple elements thatuse the same key:

>>>payload_tuples=[('key1','value1'),('key1','value2')]>>>r1=requests.post('https://httpbin.org/post',data=payload_tuples)>>>payload_dict={'key1':['value1','value2']}>>>r2=requests.post('https://httpbin.org/post',data=payload_dict)>>>print(r1.text){  ...  "form": {    "key1": [      "value1",      "value2"    ]  },  ...}>>>r1.text==r2.textTrue

There are times that you may want to send data that is not form-encoded. Ifyou pass in astring instead of adict, that data will be posted directly.

For example, the GitHub API v3 accepts JSON-Encoded POST/PATCH data:

>>>importjson>>>url='https://api.github.com/some/endpoint'>>>payload={'some':'data'}>>>r=requests.post(url,data=json.dumps(payload))

Please note that the above code will NOT add theContent-Type header(so in particular it will NOT set it toapplication/json).

If you need that header set and you don’t want to encode thedict yourself,you can also pass it directly using thejson parameter (added in version 2.4.2)and it will be encoded automatically:

>>>url='https://api.github.com/some/endpoint'>>>payload={'some':'data'}

>>>r=requests.post(url,json=payload)

Note, thejson parameter is ignored if eitherdata orfiles is passed.

POST a Multipart-Encoded File¶

Requests makes it simple to upload Multipart-encoded files:

>>>url='https://httpbin.org/post'>>>files={'file':open('report.xls','rb')}>>>r=requests.post(url,files=files)>>>r.text{  ...  "files": {    "file": "<censored...binary...data>"  },  ...}

You can set the filename, content_type and headers explicitly:

>>>url='https://httpbin.org/post'>>>files={'file':('report.xls',open('report.xls','rb'),'application/vnd.ms-excel',{'Expires':'0'})}>>>r=requests.post(url,files=files)>>>r.text{  ...  "files": {    "file": "<censored...binary...data>"  },  ...}

If you want, you can send strings to be received as files:

>>>url='https://httpbin.org/post'>>>files={'file':('report.csv','some,data,to,send\nanother,row,to,send\n')}>>>r=requests.post(url,files=files)>>>r.text{  ...  "files": {    "file": "some,data,to,send\\nanother,row,to,send\\n"  },  ...}

In the event you are posting a very large file as amultipart/form-datarequest, you may want to stream the request. By default,requests does notsupport this, but there is a separate package which does -requests-toolbelt. You should readthe toolbelt’s documentation for more details about how to use it.

For sending multiple files in one request refer to theadvancedsection.

Response Status Codes¶

We can check the response status code:

>>>r=requests.get('https://httpbin.org/get')>>>r.status_code200

Requests also comes with a built-in status code lookup object for easyreference:

>>>r.status_code==requests.codes.okTrue

If we made a bad request (a 4XX client error or 5XX server error response), wecan raise it withResponse.raise_for_status():

>>>bad_r=requests.get('https://httpbin.org/status/404')>>>bad_r.status_code404>>>bad_r.raise_for_status()Traceback (most recent call last):  File"requests/models.py", line832, inraise_for_statusraisehttp_errorrequests.exceptions.HTTPError:404 Client Error

But, since ourstatus_code forr was200, when we callraise_for_status() we get:

>>>r.raise_for_status()None

All is well.

Response Headers¶

We can view the server’s response headers using a Python dictionary:

>>>r.headers{    'content-encoding': 'gzip',    'transfer-encoding': 'chunked',    'connection': 'close',    'server': 'nginx/1.0.4',    'x-runtime': '148ms',    'etag': '"e1ca502697e5c9317743dc078f67693f"',    'content-type': 'application/json'}

The dictionary is special, though: it’s made just for HTTP headers. According toRFC 7230, HTTP Header namesare case-insensitive.

So, we can access the headers using any capitalization we want:

>>>r.headers['Content-Type']'application/json'>>>r.headers.get('content-type')'application/json'

It is also special in that the server could have sent the same header multipletimes with different values, but requests combines them so they can berepresented in the dictionary within a single mapping, as perRFC 7230:

A recipient MAY combine multiple header fields with the same field nameinto one “field-name: field-value” pair, without changing the semanticsof the message, by appending each subsequent field value to the combinedfield value in order, separated by a comma.

Cookies¶

If a response contains some Cookies, you can quickly access them:

>>>url='http://example.com/some/cookie/setting/url'>>>r=requests.get(url)>>>r.cookies['example_cookie_name']'example_cookie_value'

To send your own cookies to the server, you can use thecookiesparameter:

>>>url='https://httpbin.org/cookies'>>>cookies=dict(cookies_are='working')>>>r=requests.get(url,cookies=cookies)>>>r.text'{"cookies": {"cookies_are": "working"}}'

Cookies are returned in aRequestsCookieJar,which acts like adict but also offers a more complete interface,suitable for use over multiple domains or paths. Cookie jars canalso be passed in to requests:

>>>jar=requests.cookies.RequestsCookieJar()>>>jar.set('tasty_cookie','yum',domain='httpbin.org',path='/cookies')>>>jar.set('gross_cookie','blech',domain='httpbin.org',path='/elsewhere')>>>url='https://httpbin.org/cookies'>>>r=requests.get(url,cookies=jar)>>>r.text'{"cookies": {"tasty_cookie": "yum"}}'

Redirection and History¶

By default Requests will perform location redirection for all verbs exceptHEAD.

We can use thehistory property of the Response object to track redirection.

TheResponse.history list contains theResponse objects that were created in order tocomplete the request. The list is sorted from the oldest to the most recentresponse.

For example, GitHub redirects all HTTP requests to HTTPS:

>>>r=requests.get('http://github.com/')>>>r.url'https://github.com/'>>>r.status_code200>>>r.history[<Response [301]>]

If you’re using GET, OPTIONS, POST, PUT, PATCH or DELETE, you can disableredirection handling with theallow_redirects parameter:

>>>r=requests.get('http://github.com/',allow_redirects=False)>>>r.status_code301>>>r.history[]

If you’re using HEAD, you can enable redirection as well:

>>>r=requests.head('http://github.com/',allow_redirects=True)>>>r.url'https://github.com/'>>>r.history[<Response [301]>]

Timeouts¶

You can tell Requests to stop waiting for a response after a given number ofseconds with thetimeout parameter. Nearly all production code should usethis parameter in nearly all requests. Failure to do so can cause your programto hang indefinitely:

>>>requests.get('https://github.com/',timeout=0.001)Traceback (most recent call last):  File"<stdin>", line1, in<module>requests.exceptions.Timeout:HTTPConnectionPool(host='github.com', port=80): Request timed out. (timeout=0.001)

Errors and Exceptions¶

In the event of a network problem (e.g. DNS failure, refused connection, etc),Requests will raise aConnectionError exception.

Response.raise_for_status() willraise anHTTPError if the HTTP requestreturned an unsuccessful status code.

If a request times out, aTimeout exception israised.

If a request exceeds the configured number of maximum redirections, aTooManyRedirects exception is raised.

All exceptions that Requests explicitly raises inherit fromrequests.exceptions.RequestException.

Ready for more? Check out theadvanced section.