簡介¶

• Basic Authentication

  以 Python 為例的Basic Authentication 教學。

urllib.request 是一個用來從 URLs (Uniform Resource Locators) 取得資料的Python模組。它提供一個了非常簡單的介面能接受多種不同的協議，urlopen 函式。也提供了較複雜的介面用於處理一些常見的狀況，例如：基本的 authentication、cookies、proxies 等等，這些都可以由 handler 或 opener 物件操作。

urllib.request supports fetching URLs for many "URL schemes" (identified by the stringbefore the":" in URL - for example"ftp" is the URL scheme of"ftp://python.org/") using their associated network protocols (e.g. FTP, HTTP).This tutorial focuses on the most common case, HTTP.

一般情形下urlopen 是非常容易使用的，但當你遇到錯誤或者較複雜的情況下，你可能需要對超文本協議 HyperText Transfer Protocol 有一定的了解。最完整且具參考價值的是RFC 2616，不過它是一份技術文件並不容易閱讀，以下的教學會提供足夠的 HTTP 知識來幫助你使用urllib。這份教學並非要取代urllib.request 這份文件，你還是會需要它。

從 URL 取得資源¶

以下是使用 urllib.request 最簡單的方法:

importurllib.requestwithurllib.request.urlopen('http://python.org/')asresponse:html=response.read()

If you wish to retrieve a resource via URL and store it in a temporarylocation, you can do so via theshutil.copyfileobj() andtempfile.NamedTemporaryFile() functions:

importshutilimporttempfileimporturllib.requestwithurllib.request.urlopen('http://python.org/')asresponse:withtempfile.NamedTemporaryFile(delete=False)astmp_file:shutil.copyfileobj(response,tmp_file)withopen(tmp_file.name)ashtml:pass

Many uses of urllib will be that simple (note that instead of an 'http:' URL wecould have used a URL starting with 'ftp:', 'file:', etc.). However, it's thepurpose of this tutorial to explain the more complicated cases, concentrating onHTTP.

HTTP is based on requests and responses - the client makes requests and serverssend responses. urllib.request mirrors this with aRequest object which representsthe HTTP request you are making. In its simplest form you create a Requestobject that specifies the URL you want to fetch. Callingurlopen with thisRequest object returns a response object for the URL requested. This response isa file-like object, which means you can for example call.read() on theresponse:

importurllib.requestreq=urllib.request.Request('http://python.org/')withurllib.request.urlopen(req)asresponse:the_page=response.read()

Note that urllib.request makes use of the same Request interface to handle all URLschemes. For example, you can make an FTP request like so:

req=urllib.request.Request('ftp://example.com/')

In the case of HTTP, there are two extra things that Request objects allow youto do: First, you can pass data to be sent to the server. Second, you can passextra information ("metadata")about the data or about the request itself, tothe server - this information is sent as HTTP "headers". Let's look at each ofthese in turn.

importurllib.parseimporturllib.requesturl='http://www.someserver.com/cgi-bin/register.cgi'values={'name':'Michael Foord','location':'Northampton','language':'Python'}data=urllib.parse.urlencode(values)data=data.encode('ascii')# data should be bytesreq=urllib.request.Request(url,data)withurllib.request.urlopen(req)asresponse:the_page=response.read()

>>>importurllib.request>>>importurllib.parse>>>data={}>>>data['name']='Somebody Here'>>>data['location']='Northampton'>>>data['language']='Python'>>>url_values=urllib.parse.urlencode(data)>>>print(url_values)# The order may differ from below.name=Somebody+Here&language=Python&location=Northampton>>>url='http://www.example.com/example.cgi'>>>full_url=url+'?'+url_values>>>data=urllib.request.urlopen(full_url)

importurllib.parseimporturllib.requesturl='http://www.someserver.com/cgi-bin/register.cgi'user_agent='Mozilla/5.0 (Windows NT 6.1; Win64; x64)'values={'name':'Michael Foord','location':'Northampton','language':'Python'}headers={'User-Agent':user_agent}data=urllib.parse.urlencode(values)data=data.encode('ascii')req=urllib.request.Request(url,data,headers)withurllib.request.urlopen(req)asresponse:the_page=response.read()

Handling Exceptions¶

urlopen raisesURLError when it cannot handle a response (though asusual with Python APIs, built-in exceptions such asValueError,TypeError etc. may also be raised).

HTTPError is the subclass ofURLError raised in the specific case ofHTTP URLs.

The exception classes are exported from theurllib.error module.

>>>req=urllib.request.Request('http://www.pretend_server.org')>>>try:urllib.request.urlopen(req)...excepturllib.error.URLErrorase:...print(e.reason)...(4, 'getaddrinfo failed')

# Table mapping response codes to messages; entries have the# form {code: (shortmessage, longmessage)}.responses={100:('Continue','Request received, please continue'),101:('Switching Protocols','Switching to new protocol; obey Upgrade header'),200:('OK','Request fulfilled, document follows'),201:('Created','Document created, URL follows'),202:('Accepted','Request accepted, processing continues off-line'),203:('Non-Authoritative Information','Request fulfilled from cache'),204:('No Content','Request fulfilled, nothing follows'),205:('Reset Content','Clear input form for further input.'),206:('Partial Content','Partial content follows.'),300:('Multiple Choices','Object has several resources -- see URI list'),301:('Moved Permanently','Object moved permanently -- see URI list'),302:('Found','Object moved temporarily -- see URI list'),303:('See Other','Object moved -- see Method and URL list'),304:('Not Modified','Document has not changed since given time'),305:('Use Proxy','You must use proxy specified in Location to access this ''resource.'),307:('Temporary Redirect','Object moved temporarily -- see URI list'),400:('Bad Request','Bad request syntax or unsupported method'),401:('Unauthorized','No permission -- see authorization schemes'),402:('Payment Required','No payment -- see charging schemes'),403:('Forbidden','Request forbidden -- authorization will not help'),404:('Not Found','Nothing matches the given URI'),405:('Method Not Allowed','Specified method is invalid for this server.'),406:('Not Acceptable','URI not available in preferred format.'),407:('Proxy Authentication Required','You must authenticate with ''this proxy before proceeding.'),408:('Request Timeout','Request timed out; try again later.'),409:('Conflict','Request conflict.'),410:('Gone','URI no longer exists and has been permanently removed.'),411:('Length Required','Client must specify Content-Length.'),412:('Precondition Failed','Precondition in headers is false.'),413:('Request Entity Too Large','Entity is too large.'),414:('Request-URI Too Long','URI is too long.'),415:('Unsupported Media Type','Entity body in unsupported format.'),416:('Requested Range Not Satisfiable','Cannot satisfy request range.'),417:('Expectation Failed','Expect condition could not be satisfied.'),500:('Internal Server Error','Server got itself in trouble'),501:('Not Implemented','Server does not support this operation'),502:('Bad Gateway','Invalid responses from another server/proxy.'),503:('Service Unavailable','The server cannot process the request due to a high load'),504:('Gateway Timeout','The gateway server did not receive a timely response'),505:('HTTP Version Not Supported','Cannot fulfill request.'),}

>>>req=urllib.request.Request('http://www.python.org/fish.html')>>>try:...urllib.request.urlopen(req)...excepturllib.error.HTTPErrorase:...print(e.code)...print(e.read())...404b'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\n\n\n<html  ...  <title>Page Not Found</title>\n  ...

fromurllib.requestimportRequest,urlopenfromurllib.errorimportURLError,HTTPErrorreq=Request(someurl)try:response=urlopen(req)exceptHTTPErrorase:print('The server couldn\'t fulfill the request.')print('Error code: ',e.code)exceptURLErrorase:print('We failed to reach a server.')print('Reason: ',e.reason)else:# everything is fine

fromurllib.requestimportRequest,urlopenfromurllib.errorimportURLErrorreq=Request(someurl)try:response=urlopen(req)exceptURLErrorase:ifhasattr(e,'reason'):print('We failed to reach a server.')print('Reason: ',e.reason)elifhasattr(e,'code'):print('The server couldn\'t fulfill the request.')print('Error code: ',e.code)else:# everything is fine

info and geturl¶

The response returned by urlopen (or theHTTPError instance) has twouseful methodsinfo() andgeturl() and is defined in the moduleurllib.response.

• geturl - this returns the real URL of the page fetched. This is usefulbecauseurlopen (or the opener object used) may have followed aredirect. The URL of the page fetched may not be the same as the URL requested.

• info - this returns a dictionary-like object that describes the pagefetched, particularly the headers sent by the server. It is currently anhttp.client.HTTPMessage instance.

Typical headers include 'Content-length', 'Content-type', and so on. See theQuick Reference to HTTP Headersfor a useful listing of HTTP headers with brief explanations of their meaningand use.

Openers and Handlers¶

When you fetch a URL you use an opener (an instance of the perhapsconfusingly namedurllib.request.OpenerDirector). Normally we have been usingthe default opener - viaurlopen - but you can create customopeners. Openers use handlers. All the "heavy lifting" is done by thehandlers. Each handler knows how to open URLs for a particular URL scheme (http,ftp, etc.), or how to handle an aspect of URL opening, for example HTTPredirections or HTTP cookies.

You will want to create openers if you want to fetch URLs with specific handlersinstalled, for example to get an opener that handles cookies, or to get anopener that does not handle redirections.

To create an opener, instantiate anOpenerDirector, and then call.add_handler(some_handler_instance) repeatedly.

Alternatively, you can usebuild_opener, which is a convenience function forcreating opener objects with a single function call.build_opener addsseveral handlers by default, but provides a quick way to add more and/oroverride the default handlers.

Other sorts of handlers you might want to can handle proxies, authentication,and other common but slightly specialised situations.

install_opener can be used to make anopener object the (global) defaultopener. This means that calls tourlopen will use the opener you haveinstalled.

Opener objects have anopen method, which can be called directly to fetchurls in the same way as theurlopen function: there's no need to callinstall_opener, except as a convenience.

Basic Authentication¶

To illustrate creating and installing a handler we will use theHTTPBasicAuthHandler. For a more detailed discussion of this subject --including an explanation of how Basic Authentication works - see theBasicAuthentication Tutorial.

When authentication is required, the server sends a header (as well as the 401error code) requesting authentication. This specifies the authentication schemeand a 'realm'. The header looks like:WWW-Authenticate:SCHEMErealm="REALM".

例如

WWW-Authenticate: Basic realm="cPanel Users"

The client should then retry the request with the appropriate name and passwordfor the realm included as a header in the request. This is 'basicauthentication'. In order to simplify this process we can create an instance ofHTTPBasicAuthHandler and an opener to use this handler.

TheHTTPBasicAuthHandler uses an object called a password manager to handlethe mapping of URLs and realms to passwords and usernames. If you know what therealm is (from the authentication header sent by the server), then you can use aHTTPPasswordMgr. Frequently one doesn't care what the realm is. In thatcase, it is convenient to useHTTPPasswordMgrWithDefaultRealm. This allowsyou to specify a default username and password for a URL. This will be suppliedin the absence of you providing an alternative combination for a specificrealm. We indicate this by providingNone as the realm argument to theadd_password method.

The top-level URL is the first URL that requires authentication. URLs "deeper"than the URL you pass to .add_password() will also match.

# create a password managerpassword_mgr=urllib.request.HTTPPasswordMgrWithDefaultRealm()# Add the username and password.# If we knew the realm, we could use it instead of None.top_level_url="http://example.com/foo/"password_mgr.add_password(None,top_level_url,username,password)handler=urllib.request.HTTPBasicAuthHandler(password_mgr)# create "opener" (OpenerDirector instance)opener=urllib.request.build_opener(handler)# use the opener to fetch a URLopener.open(a_url)# Install the opener.# Now all calls to urllib.request.urlopen use our opener.urllib.request.install_opener(opener)

top_level_url is in facteither a full URL (including the 'http:' schemecomponent and the hostname and optionally the port number)e.g."http://example.com/"or an "authority" (i.e. the hostname,optionally including the port number) e.g."example.com" or"example.com:8080"(the latter example includes a port number). The authority, if present, mustNOT contain the "userinfo" component - for example"joe:password@example.com" isnot correct.

Proxies¶

urllib will auto-detect your proxy settings and use those. This is throughtheProxyHandler, which is part of the normal handler chain when a proxysetting is detected. Normally that's a good thing, but there are occasionswhen it may not be helpful[5]. One way to do this is to setup our ownProxyHandler, with no proxies defined. This is done using similar steps tosetting up aBasic Authentication handler:

>>>proxy_support=urllib.request.ProxyHandler({})>>>opener=urllib.request.build_opener(proxy_support)>>>urllib.request.install_opener(opener)

Sockets and Layers¶

The Python support for fetching resources from the web is layered. urllib usesthehttp.client library, which in turn uses the socket library.

As of Python 2.3 you can specify how long a socket should wait for a responsebefore timing out. This can be useful in applications which have to fetch webpages. By default the socket module hasno timeout and can hang. Currently,the socket timeout is not exposed at the http.client or urllib.request levels.However, you can set the default timeout globally for all sockets using

importsocketimporturllib.request# timeout in secondstimeout=10socket.setdefaulttimeout(timeout)# this call to urllib.request.urlopen now uses the default timeout# we have set in the socket modulereq=urllib.request.Request('http://www.voidspace.org.uk')response=urllib.request.urlopen(req)

註腳¶

This document was reviewed and revised by John Lee.

Google for example.

Browser sniffing is a very bad practice for website design - buildingsites using web standards is much more sensible. Unfortunately a lot ofsites still send different versions to different browsers.

The user agent for MSIE 6 is'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)'

For details of more HTTP request headers, seeQuick Reference to HTTP Headers.

In my case I have to use a proxy to access the internet at work. If youattempt to fetchlocalhost URLs through this proxy it blocks them. IEis set to use the proxy, which urllib picks up on. In order to testscripts with a localhost server, I have to prevent urllib from usingthe proxy.

urllib opener for SSL proxy (CONNECT method):ASPN Cookbook Recipe.