Repository files navigation

TheDownloads package provides a single function,download, which providescross-platform, multi-protocol, in-process download functionality implementedwithlibcurl. It uses libcurl's multi-handlecallback API to present a Julian API:download(url) blocks the task in whichit occurs but yields to Julia's scheduler, allowing arbitrarily many tasks todownload URLs concurrently and efficiently. As of Julia 1.6, this package is astandard library that is included with Julia, but this package can be used withJulia 1.3 through 1.5 as well.

The public API ofDownloads consists of three functions and three types:

• download — download a file from a URL, erroring if it can't be downloaded
• request — request a URL, returning aResponse object indicating success
• default_downloader! - set the defaultDownloader object
• Response — a type capturing the status and other metadata about a request
• RequestError — an error type thrown bydownload andrequest on error
• Downloader — an object encapsulating shared resources for downloading

download(url, [ output=tempname() ];    [ method="GET", ]    [ headers=<none>, ]    [ timeout=<none>, ]    [ progress=<none>, ]    [ verbose=false, ]    [ debug=<none>, ]    [ downloader=<default>, ])-> output

• url :: AbstractString
• output :: Union{AbstractString, AbstractCmd, IO}
• method :: AbstractString
• headers :: Union{AbstractVector, AbstractDict}
• timeout :: Real
• progress :: (total::Integer, now::Integer) --> Any
• verbose :: Bool
• debug :: (type, message) --> Any
• downloader :: Downloader

Download a file from the given url, saving it tooutput or if not specified, atemporary path. Theoutput can also be anIO handle, in which case the bodyof the response is streamed to that handle and the handle is returned. Ifoutput is a command, the command is run and output is sent to it on stdin.

If thedownloader keyword argument is provided, it must be aDownloaderobject. Resources and connections will be shared between downloads performed bythe sameDownloader and cleaned up automatically when the object is garbagecollected or there have been no downloads performed with it for a grace period.SeeDownloader for more info about configuration and usage.

If theheaders keyword argument is provided, it must be a vector or dictionarywhose elements are all pairs of strings. These pairs are passed as headers whendownloading URLs with protocols that supports them, such as HTTP/S.

Thetimeout keyword argument specifies a timeout for the download in seconds,with a resolution of milliseconds. By default no timeout is set, but this canalso be explicitly requested by passing a timeout value ofInf.

If theprogress keyword argument is provided, it must be a callback functionwhich will be called whenever there are updates about the size and status of theongoing download. The callback must take two integer arguments:total andnow which are the total size of the download in bytes, and the number of byteswhich have been downloaded so far. Note thattotal starts out as zero andremains zero until the server gives an indication of the total size of thedownload (e.g. with aContent-Length header), which may never happen. So awell-behaved progress callback should handle a total size of zero gracefully.

If theverbose option is set to true,libcurl, which is used to implementthe download functionality will print debugging information tostderr. If thedebug option is set to a function accepting twoString arguments, then theverbose option is ignored and instead the data that would have been printed tostderr is passed to thedebug callback withtype andmessage arguments.Thetype argument indicates what kind of event has occurred, and is one of:TEXT,HEADER IN,HEADER OUT,DATA IN,DATA OUT,SSL DATA IN orSSL DATA OUT. Themessage argument is the description of the debug event.

request(url;    [ input=<none>, ]    [ output=<none>, ]    [ method= input?"PUT": output?"GET":"HEAD", ]    [ headers=<none>, ]    [ timeout=<none>, ]    [ progress=<none>, ]    [ verbose=false, ]    [ debug=<none>, ]    [ throw=true, ]    [ downloader=<default>, ]    [ interrupt=<none>, ])-> Union{Response, RequestError}

• url :: AbstractString
• input :: Union{AbstractString, AbstractCmd, IO}
• output :: Union{AbstractString, AbstractCmd, IO}
• method :: AbstractString
• headers :: Union{AbstractVector, AbstractDict}
• timeout :: Real
• progress :: (dl_total, dl_now, ul_total, ul_now) --> Any
• verbose :: Bool
• debug :: (type, message) --> Any
• throw :: Bool
• downloader :: Downloader
• interrupt :: Base.Event

Make a request to the given url, returning aResponse object capturing thestatus, headers and other information about the response. The body of theresponse is written tooutput if specified and discarded otherwise. For HTTP/Srequests, if aninput stream is given, aPUT request is made; otherwise ifanoutput stream is given, aGET request is made; if neither is given aHEAD request is made. For other protocols, appropriate default methods areused based on what combination of input and output are requested. The followingoptions differ from thedownload function:

• input allows providing a request body; if provided default toPUT request
• progress is a callback taking four integers for upload and download progress
• throw controls whether to throw or return aRequestError on request error

Note that unlikedownload which throws an error if the requested URL could notbe downloaded (indicated by non-2xx status code),request returns aResponseobject no matter what the status code of the response is. If there is an errorwith getting a response at all, then aRequestError is thrown or returned.

If theinterrupt keyword argument is provided, it must be aBase.Event object.If the event is triggered while the request is in progress, the request will becancelled and an error will be thrown. This can be used to interrupt a longrunning request, for example if the user wants to cancel a download.

default_downloader!(        downloader=<none>    )

• downloader :: Downloader

Set the defaultDownloader. If no argument is provided, resets the default downloaderso that a fresh one is created the next time the default downloader is needed.

struct Response    proto::String    url::String    status::Int    message::String    headers::Vector{Pair{String,String}}end

Response is a type capturing the properties of a successful response to arequest as an object. It has the following fields:

• proto: the protocol that was used to get the response
• url: the URL that was ultimately requested after following redirects
• status: the status code of the response, indicating success, failure, etc.
• message: a textual message describing the nature of the response
• headers: any headers that were returned with the response

The meaning and availability of some of these responses depends on the protocolused for the request. For many protocols, including HTTP/S and S/FTP, a 2xxstatus code indicates a successful response. For responses in protocols that donot support headers, the headers vector will be empty. HTTP/2 does not include astatus message, only a status code, so the message will be empty.

struct RequestError<:ErrorException    url::String    code::Int    message::String    response::Responseend

RequestError is a type capturing the properties of a failed response to arequest as an exception object:

• url: the original URL that was requested without any redirects
• code: the libcurl error code;0 if a protocol-only error occurred
• message: the libcurl error message indicating what went wrong
• response: response object capturing what response info is available

The sameRequestError type is thrown bydownload if the request wassuccessful but there was a protocol-level error indicated by a status code thatis not in the 2xx range, in which casecode will be zero and themessagefield will be the empty string. Therequest API only throws aRequestErrorif the libcurl errorcode is non-zero, in which case the includedresponseobject is likely to have astatus of zero and an empty message. There are,however, situations where a curl-level error is thrown due to a protocol error,in which case both the inner and outer code and message may be of interest.

Downloader(; [ grace::Real=30 ])

Downloader objects are used to perform individualdownload operations.Connections, name lookups and other resources are shared within aDownloader.These connections and resources are cleaned up after a configurable grace period(default: 30 seconds) since anything was downloaded with it, or when it isgarbage collected, whichever comes first. If the grace period is set to zero,all resources will be cleaned up immediately as soon as there are no moreongoing downloads in progress. If the grace period is set toInf thenresources are not cleaned up untilDownloader is garbage collected.

using Downloads: Curl, Downloader, downloadeasy_hook= (easy, info)->begin    Curl.setopt(easy, Curl.CURLOPT_SSLKEY,"client.key")    Curl.setopt(easy, Curl.CURLOPT_SSLCERT,"client.crt")enddownloader=Downloader()downloader.easy_hook= easy_hookdownload(url; downloader)

Here, client.key and client.crt are the private and public keys for the client.

It’s also possible currently to make the default downloader use that hook by putting it into Downloads.EASY_HOOK. Here’s an example:

using Downloads: Downloads, Curl, downloadDownloads.EASY_HOOK[]= (easy, info)->begin    Curl.setopt(easy, Curl.CURLOPT_SSLKEY,"client.key")    Curl.setopt(easy, Curl.CURLOPT_SSLCERT,"client.crt")enddownload(url)

There is no difference between usage on Windows, MacOS, and Linux.