Note
Theurllib2 module has been split across several modules inPython 3.0 namedurllib.request andurllib.error.The2to3 tool will automatically adapt imports when convertingyour sources to 3.0.
Theurllib2 module defines functions and classes which help in openingURLs (mostly HTTP) in a complex world — basic and digest authentication,redirections, cookies and more.
Theurllib2 module defines the following functions:
Open the URLurl, which can be either a string or aRequest object.
data may be a string specifying additional data to send to the server, orNone if no such data is needed. Currently HTTP requests are the only onesthat usedata; the HTTP request will be a POST instead of a GET when thedata parameter is provided.data should be a buffer in the standardapplication/x-www-form-urlencoded format. Theurllib.urlencode() function takes a mapping or sequence of 2-tuples andreturns a string in this format.
The optionaltimeout parameter specifies a timeout in seconds for blockingoperations like the connection attempt (if not specified, the global defaulttimeout setting will be used). This actually only works for HTTP, HTTPS,FTP and FTPS connections.
This function returns a file-like object with two additional methods:
RaisesURLError on errors.
Note thatNone may be returned if no handler handles the request (though thedefault installed globalOpenerDirector usesUnknownHandler toensure this never happens).
Changed in version 2.6:timeout was added.
Return anOpenerDirector instance, which chains the handlers in theorder given.handlers can be either instances ofBaseHandler, orsubclasses ofBaseHandler (in which case it must be possible to callthe constructor without any parameters). Instances of the following classeswill be in front of thehandlers, unless thehandlers contain them,instances of them or subclasses of them:ProxyHandler,UnknownHandler,HTTPHandler,HTTPDefaultErrorHandler,HTTPRedirectHandler,FTPHandler,FileHandler,HTTPErrorProcessor.
If the Python installation has SSL support (i.e., if thessl module can be imported),HTTPSHandler will also be added.
Beginning in Python 2.3, aBaseHandler subclass may also change itshandler_order member variable to modify its position in the handlerslist.
The following exceptions are raised as appropriate:
The handlers raise this exception (or derived exceptions) when they run into aproblem. It is a subclass ofIOError.
Though being an exception (a subclass ofURLError), anHTTPErrorcan also function as a non-exceptional file-like return value (the same thingthaturlopen() returns). This is useful when handling exotic HTTPerrors, such as requests for authentication.
The following classes are provided:
This class is an abstraction of a URL request.
url should be a string containing a valid URL.
data may be a string specifying additional data to send to the server, orNone if no such data is needed. Currently HTTP requests are the only onesthat usedata; the HTTP request will be a POST instead of a GET when thedata parameter is provided.data should be a buffer in the standardapplication/x-www-form-urlencoded format. Theurllib.urlencode() function takes a mapping or sequence of 2-tuples andreturns a string in this format.
headers should be a dictionary, and will be treated as ifadd_header()was called with each key and value as arguments. This is often used to “spoof”theUser-Agent header, which is used by a browser to identify itself –some HTTP servers only allow requests coming from common browsers as opposedto scripts. For example, Mozilla Firefox may identify itself as"Mozilla/5.0(X11;U;Linuxi686)Gecko/20071127Firefox/2.0.0.11", whileurllib2‘sdefault user agent string is"Python-urllib/2.6" (on Python 2.6).
The final two arguments are only of interest for correct handling of third-partyHTTP cookies:
origin_req_host should be the request-host of the origin transaction, asdefined byRFC 2965. It defaults tocookielib.request_host(self). Thisis the host name or IP address of the original request that was initiated by theuser. For example, if the request is for an image in an HTML document, thisshould be the request-host of the request for the page containing the image.
unverifiable should indicate whether the request is unverifiable, as definedby RFC 2965. It defaults to False. An unverifiable request is one whose URLthe user did not have the option to approve. For example, if the request is foran image in an HTML document, and the user had no option to approve theautomatic fetching of the image, this should be true.
The following methods describe all ofRequest‘s public interface, andso all must be overridden in subclasses.
Add a header that will not be added to a redirected request.
New in version 2.4.
Return whether the instance has the named header (checks both regular andunredirected).
New in version 2.4.
OpenerDirector instances have the following methods:
handler should be an instance ofBaseHandler. The following methodsare searched, and added to the possible chains (note that HTTP errors are aspecial case).
Open the givenurl (which can be a request object or a string), optionallypassing the givendata. Arguments, return values and exceptions raised arethe same as those ofurlopen() (which simply calls theopen()method on the currently installed globalOpenerDirector). Theoptionaltimeout parameter specifies a timeout in seconds for blockingoperations like the connection attempt (if not specified, the global defaulttimeout setting will be usedi). The timeout feature actually works only forHTTP, HTTPS, FTP and FTPS connections).
Changed in version 2.6:timeout was added.
Handle an error of the given protocol. This will call the registered errorhandlers for the given protocol with the given arguments (which are protocolspecific). The HTTP protocol is a special case which uses the HTTP responsecode to determine the specific error handler; refer to thehttp_error_*()methods of the handler classes.
Return values and exceptions raised are the same as those ofurlopen().
OpenerDirector objects open URLs in three stages:
The order in which these methods are called within each stage is determined bysorting the handler instances.
Every handler with a method named likeprotocol_request() has thatmethod called to pre-process the request.
Handlers with a method named likeprotocol_open() are called to handlethe request. This stage ends when a handler either returns a non-Nonevalue (ie. a response), or raises an exception (usuallyURLError).Exceptions are allowed to propagate.
In fact, the above algorithm is first tried for methods nameddefault_open(). If all such methods returnNone, the algorithmis repeated for methods named likeprotocol_open(). If all such methodsreturnNone, the algorithm is repeated for methods namedunknown_open().
Note that the implementation of these methods may involve calls of the parentOpenerDirector instance’sopen() anderror() methods.
Every handler with a method named likeprotocol_response() has thatmethod called to post-process the response.
BaseHandler objects provide a couple of methods that are directlyuseful, and others that are meant to be used by derived classes. These areintended for direct use:
The following members and methods should only be used by classes derived fromBaseHandler.
Note
The convention has been adopted that subclasses definingprotocol_request() orprotocol_response() methods are named*Processor; all others are named*Handler.
This method isnot defined inBaseHandler, but subclasses shoulddefine it if they want to catch all URLs.
This method, if implemented, will be called by the parentOpenerDirector. It should return a file-like object as described inthe return value of theopen() ofOpenerDirector, orNone.It should raiseURLError, unless a truly exceptional thing happens (forexample,MemoryError should not be mapped toURLError).
This method will be called before any protocol-specific open method.
This method isnot defined inBaseHandler, but subclasses shoulddefine it if they want to handle URLs with the given protocol.
This method, if defined, will be called by the parentOpenerDirector.Return values should be the same as fordefault_open().
This method isnot defined inBaseHandler, but subclasses shoulddefine it if they want to catch all URLs with no specific registered handler toopen it.
This method, if implemented, will be called by theparentOpenerDirector. Return values should be the same as fordefault_open().
This method isnot defined inBaseHandler, but subclasses shouldoverride it if they intend to provide a catch-all for otherwise unhandled HTTPerrors. It will be called automatically by theOpenerDirector gettingthe error, and should not normally be called in other circumstances.
req will be aRequest object,fp will be a file-like object withthe HTTP error body,code will be the three-digit code of the error,msgwill be the user-visible explanation of the code andhdrs will be a mappingobject with the headers of the error.
Return values and exceptions raised should be the same as those ofurlopen().
nnn should be a three-digit HTTP error code. This method is also not definedinBaseHandler, but will be called, if it exists, on an instance of asubclass, when an HTTP error with codennn occurs.
Subclasses should override this method to handle specific HTTP errors.
Arguments, return values and exceptions raised should be the same as forhttp_error_default().
This method isnot defined inBaseHandler, but subclasses shoulddefine it if they want to pre-process requests of the given protocol.
This method, if defined, will be called by the parentOpenerDirector.req will be aRequest object. The return value should be aRequest object.
This method isnot defined inBaseHandler, but subclasses shoulddefine it if they want to post-process responses of the given protocol.
This method, if defined, will be called by the parentOpenerDirector.req will be aRequest object.response will be an objectimplementing the same interface as the return value ofurlopen(). Thereturn value should implement the same interface as the return value ofurlopen().
Note
Some HTTP redirections require action from this module’s client code. If thisis the case,HTTPError is raised. SeeRFC 2616 for details of theprecise meanings of the various redirection codes.
Return aRequest orNone in response to a redirect. This is calledby the default implementations of thehttp_error_30*() methods when aredirection is received from the server. If a redirection should take place,return a newRequest to allowhttp_error_30*() to perform theredirect. Otherwise, raiseHTTPError if no other handler should try tohandle this URL, or returnNone if you can’t but another handler might.
Note
The default implementation of this method does not strictly followRFC 2616,which says that 301 and 302 responses toPOST requests must not beautomatically redirected without confirmation by the user. In reality, browsersdo allow automatic redirection of these responses, changing the POST to aGET, and the default implementation reproduces this behavior.
New in version 2.4.
HTTPCookieProcessor instances have one attribute:
These methods are available onHTTPPasswordMgr andHTTPPasswordMgrWithDefaultRealm objects.
Get user/password for given realm and URI, if any. This method will return(None,None) if there is no matching user/password.
ForHTTPPasswordMgrWithDefaultRealm objects, the realmNone will besearched if the givenrealm has no matching user/password.
Handle an authentication request by getting a user/password pair, and re-tryingthe request.authreq should be the name of the header where the informationabout the realm is included in the request,host specifies the URL and path toauthenticate for,req should be the (failed)Request object, andheaders should be the error headers.
host is either an authority (e.g."python.org") or a URL containing anauthority component (e.g."http://python.org/"). In either case, theauthority must not contain a userinfo component (so,"python.org" and"python.org:80" are fine,"joe:password@python.org" is not).
CacheFTPHandler objects areFTPHandler objects with thefollowing additional methods:
New in version 2.4.
Process HTTP error responses.
For 200 error codes, the response object is returned immediately.
For non-200 error codes, this simply passes the job on to theprotocol_error_code() handler methods, viaOpenerDirector.error().Eventually,urllib2.HTTPDefaultErrorHandler will raise anHTTPError if no other handler handles the error.
This example gets the python.org main page and displays the first 100 bytes ofit:
>>>importurllib2>>>f=urllib2.urlopen('http://www.python.org/')>>>printf.read(100)<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><?xml-stylesheet href="./css/ht2html
Here we are sending a data-stream to the stdin of a CGI and reading the data itreturns to us. Note that this example will only work when the Pythoninstallation supports SSL.
>>>importurllib2>>>req=urllib2.Request(url='https://localhost/cgi-bin/test.cgi',...data='This data is passed to stdin of the CGI')>>>f=urllib2.urlopen(req)>>>printf.read()Got Data: "This data is passed to stdin of the CGI"
The code for the sample CGI used in the above example is:
#!/usr/bin/env pythonimportsysdata=sys.stdin.read()print'Content-type: text-plain\n\nGot Data: "%s"'%data
Use of Basic HTTP Authentication:
importurllib2# Create an OpenerDirector with support for Basic HTTP Authentication...auth_handler=urllib2.HTTPBasicAuthHandler()auth_handler.add_password(realm='PDQ Application',uri='https://mahler:8092/site-updates.py',user='klem',passwd='kadidd!ehopper')opener=urllib2.build_opener(auth_handler)# ...and install it globally so it can be used with urlopen.urllib2.install_opener(opener)urllib2.urlopen('http://www.example.com/login.html')
build_opener() provides many handlers by default, including aProxyHandler. By default,ProxyHandler uses the environmentvariables named<scheme>_proxy, where<scheme> is the URL schemeinvolved. For example, thehttp_proxy environment variable is read toobtain the HTTP proxy’s URL.
This example replaces the defaultProxyHandler with one that usesprogrammatically-supplied proxy URLs, and adds proxy authorization support withProxyBasicAuthHandler.
proxy_handler=urllib2.ProxyHandler({'http':'http://www.example.com:3128/'})proxy_auth_handler=urllib2.HTTPBasicAuthHandler()proxy_auth_handler.add_password('realm','host','username','password')opener=build_opener(proxy_handler,proxy_auth_handler)# This time, rather than install the OpenerDirector, we use it directly:opener.open('http://www.example.com/login.html')
Adding HTTP headers:
Use theheaders argument to theRequest constructor, or:
importurllib2req=urllib2.Request('http://www.example.com/')req.add_header('Referer','http://www.python.org/')r=urllib2.urlopen(req)
OpenerDirector automatically adds aUser-Agent header toeveryRequest. To change this:
importurllib2opener=urllib2.build_opener()opener.addheaders=[('User-agent','Mozilla/5.0')]opener.open('http://www.example.com/')
Also, remember that a few standard headers (Content-Length,Content-Type andHost) are added when theRequest is passed tourlopen() (orOpenerDirector.open()).
urllib — Open arbitrary resources by URL
httplib — HTTP protocol client
