Repository files navigation

Requests-Scala is a Scala port of the popular PythonRequests HTTP client. Requests-Scala aims toprovide the same API and user-experience as the original Requests: flexible,intuitive, and straightforward to use.

If you use Requests-Scala and like it, you will probably enjoy the following book by the Author:

• Hands-on Scala Programming

Hands-on Scala has uses Requests-Scala extensively throughout the book, and hasthe entirety ofChapter 12: Working with HTTP APIs dedicated tothe library.Hands-on Scala is a great way to level up your skills in Scalain general and Requests-Scala in particular.

You can also support it by donating to our Patreon:

• https://www.patreon.com/lihaoyi

For a hands-on introduction to this library, take a look at the following blog post:

• How to work with HTTP JSON APIs in Scala

• Requests-Scala 0.9.0
  • Contents
  • Getting Started
  • Making a Request
    • Passing in Parameters
    • Response Content
  • Streaming Requests
  • Handling JSON
  • Multipart Uploads
  • Misc Configuration
    • Custom Headers
    • Timeouts
    • Compression
    • Cookies
    • Redirects
    • Client Side Certificates
  • Sessions
  • Why Requests-Scala?
  • Changelog
    • 0.9.0
    • 0.8.0
    • 0.7.1
    • 0.7.0
    • 0.6.7
    • 0.6.5
    • 0.5.1
    • 0.4.7
    • 0.3.0
    • 0.2.0
    • 0.1.9
    • 0.1.8
    • 0.1.7
    • 0.1.6
    • 0.1.5

Use the following import to get you started:

ivy"com.lihaoyi::requests:0.9.0"// mill"com.lihaoyi"%%"requests"%"0.9.0"// sbtcompile"com.lihaoyi:requests_2.12:0.9.0"//gradle

valr= requests.get("https://api.github.com/users/lihaoyi")r.statusCode// 200r.headers("content-type")// Buffer("application/json; charset=utf-8")r.text()// {"login":"lihaoyi","id":934140,"node_id":"MDQ6VXNlcjkzNDE0MA==",...

Making your first HTTP request is simple: simply callrequests.get with theURL you want, and requests will fetch it for you.

You can also callrequests.post,requests.put, etc. to make other kinds ofHTTP requests:

valr= requests.post("http://httpbin.org/post", data=Map("key"->"value"))valr= requests.put("http://httpbin.org/put", data=Map("key"->"value"))valr= requests.delete("http://httpbin.org/delete")valr= requests.head("http://httpbin.org/head")valr= requests.options("http://httpbin.org/get")// dynamically choose what HTTP method to usevalr= requests.send("put")("http://httpbin.org/put", data=Map("key"->"value"))

valr= requests.get("http://httpbin.org/get",    params=Map("key1"->"value1","key2"->"value2"))

You can pass in URL parameters to GET requests via theparams argument; simplypass in aMap[String, String]. As seen earlier, when passing in POST or PUTparameters, you instead need thedata argument:

valr= requests.post("http://httpbin.org/post", data=Map("key"->"value"))valr= requests.put("http://httpbin.org/put", data=Map("key"->"value"))

Apart from POSTing key-value pairs, you can also POSTStrings,Array[Byte]s,java.io.Files,java.nio.file.Paths, andrequests.MultiPart uploads:

requests.post("https://httpbin.org/post", data="Hello World")requests.post("https://httpbin.org/post", data=Array[Byte](1,2,3))requests.post("https://httpbin.org/post", data=new java.io.File("thing.json"))requests.post("https://httpbin.org/post", data= java.nio.file.Paths.get("thing.json"))

Thedata parameter also supports anything that implements theWritable interface, such asujson.Values,uPickle'supickle.default.writable values,orScalatags'sTags

valr= requests.get("https://api.github.com/events")r.statusCode// 200r.headers("content-type")// Buffer("application/json; charset=utf-8")

As seen earlier, you can use.statusCode and.headers to see the relevantmetadata of your HTTP response. The response data is in the.data field of theResponse object. Most often, it's text, which you can decode using the.text()property as shown below:

r.text()// [{"id":"7990061484","type":"PushEvent","actor":{"id":6242317,"login":...

If you want the raw bytes of the response, user.contents

r.contents// Array(91, 123, 34, 105, 100, 34, 58, 34, 55, 57,  57, 48, 48, 54, 49, ...

os.write(  os.pwd/"file.json",  requests.get.stream("https://api.github.com/events"))

Requests exposes therequests.get.stream (and equivalentrequests.post.stream,requests.put.stream, etc.) functions for you toperform streaming uploads/downloads without needing to load the entirerequest/response into memory. This is useful if you are upload/downloading largefiles or data blobs..stream returns aReadable value, that can be thenpassed to methods likeos.write,fastparse.parse orupickle.default.read to handle the received data in astreaming fashion:

ujson.read(requests.get.stream("https://api.github.com/events"))

Sincerequests.post andrequests.put both take adata: geny.Writableparameter, you can even chain requests together, taking the data returned fromone HTTP request and feeding it into another:

os.write(  os.pwd/"chained.json",  requests.post.stream("https://httpbin.org/post",    data= requests.get.stream("https://api.github.com/events")  ))

requests.*.stream should make it easy for you to work with datatoo big to fit in memory, while still benefiting from most of Requests' friendly& intuitive API.

Requests does not provide any built-in JSON support, but you can easily use athird-party JSON library to work with it. This example shows how to useuJson talk to a HTTP endpoint that requires aJSON-formatted body, either usingupickle.default.stream:

requests.post("https://api.github.com/some/endpoint",  data= upickle.default.stream(Map("user-agent"->"my-app/0.0.1")))

Or by constructingujson.Values directly

requests.post("https://api.github.com/some/endpoint",  data= ujson.Obj("user-agent"->"my-app/0.0.1"))

In both cases, the upload occurs efficiently in a streaming fashion, withoutmaterializing the entire JSON blob in memory.

It is equally easy ot use uJson to deal with JSON returned in the response fromthe server:

valr= requests.get("https://api.github.com/events")valjson= ujson.read(r.text())json.arr.length// 30json.arr(0).obj.keys// Set("id", "type", "actor", "repo", "payload", "public", "created_at")

While Requests-Scala doesn't come bundled with JSON functionality, it is trivialto use it together with any other 3rd party JSON library (I likeuJson) So just pick whatever library youwant.

valr= requests.post("http://httpbin.org/post",  data= requests.MultiPart(    requests.MultiItem("name",new java.io.File("build.sc"),"file.txt"),// you can upload strings, and file name is optional    requests.MultiItem("name2","Hello"),// bytes arrays are ok too    requests.MultiItem("name3",Array[Byte](1,2,3,4))  ))

Multipart uploads are done by passingrequests.MultiPart/requests.MultiItemto thedata parameter. EachMultiItem needs a name and a data-source, whichcan be aString,Array[Byte],java.io.File, orjava.nio.file.Path. EachMultiItem can optionally take a file name that will get sent to the server

Earlier you already saw how to use theparams anddata arguments. Apart fromthose, therequests.get method takes in a lot of arguments you can use toconfigure it, e.g. passing in custom headers:

requests.get("https://api.github.com/some/endpoint",  headers=Map("user-agent"->"my-app/0.0.1"))

To pass in a single header multiple times, you can pass them as a comma separated list:

requests.get("https://api.github.com/some/endpoint",  headers=Map("user-agent"->"my-app/0.0.1,other-app/0.0.2"))

readTimeouts andconnectTimeouts:

requests.get("https://httpbin.org/delay/1", readTimeout=10)// TimeoutExceptionrequests.get("https://httpbin.org/delay/1", readTimeout=1500)// okrequests.get("https://httpbin.org/delay/3", readTimeout=1500)// TimeoutException

requests.get("https://httpbin.org/delay/1", connectTimeout=10)// TimeoutExceptionrequests.get("https://httpbin.org/delay/1", connectTimeout=1500)// okrequests.get("https://httpbin.org/delay/3", connectTimeout=1500)// ok

Configuration for compressing the requestdata upload with Gzip or Deflate viathecompress parameter:

requests.post("https://httpbin.org/post",  compress= requests.Compress.None,  data="Hello World")requests.post("https://httpbin.org/post",  compress= requests.Compress.Gzip,  data="I am cow")requests.post("https://httpbin.org/post",  compress= requests.Compress.Deflate,  data="Hear me moo")

Or to disabling the de-compression of the responsedata being downloaded viatheautoCompress parameter, in case you want the un-compressed data blob forwhatever reason:

requests.get("https://httpbin.org/gzip").contents.length// 250requests.get("https://httpbin.org/gzip", autoDecompress=false).contents.length// 201requests.get("https://httpbin.org/deflate").contents.length// 251requests.get("https://httpbin.org/deflate", autoDecompress=false).contents.length// 188

Note that by default, compression of fixed-size in-memory input (Strings,Array[Byte]s, ...) buffers up the compressed data in memory before uploadingit. Compression of unknown-length/not-in-memory data (files,InputStreams,...) doesn't perform this buffering and uses chunked transfer encoding, asnormal. If you want to avoid buffering in memory and are willing to use chunkedtransfer encoding for in-memory data, wrap it in an inputstream (e.g.Array[Byte] can be wrapped in aByteArrayInputStream)

You can take the cookies that result from one HTTP request and pass them into asubsequent HTTP request:

valr= requests.get("https://httpbin.org/cookies/set?freeform=test")r.cookies// Map("freeform" -> freeform=test)

valr2= requests.get("https://httpbin.org/cookies", cookies= r.cookies)r2.text()// {"cookies":{"freeform":"test"}}

This is a common pattern, e.g. to maintain an authentication/login sessionacross multiple requests. However, it may be easier to instead use Sessions...

Requests handles redirects automatically for you, up to a point:

valr= requests.get("http://www.github.com")r.url// https://github.com/r.history// Some(Response("https://www.github.com", 301, "Moved Permanently", ...r.history.get.history// Some(Response("http://www.github.com", 301, "Moved Permanently", ...r.history.get.history.get.history// None

As you can see, the request tohttp://www.github.com was first redirected tohttps://www.github.com, and then tohttps://github.com/. Requests by defaultonly follows up to 5 redirects in a row, though this is configurable via themaxRedirects parameter:

valr0= requests.get("http://www.github.com", maxRedirects=0)// Response("http://www.github.com", 301, "Moved Permanently", ...r0.history// Nonevalr1= requests.get("http://www.github.com", maxRedirects=1)// Response("http://www.github.com", 301, "Moved Permanently", ...r1.history// Some(Response("http://www.github.com", 301, "Moved Permanently", ...r1.history.get.history// None

As you can see, you can usemaxRedirects = 0 to disable redirect handlingcompletely, or use another number to control how many redirects Requests followsbefore giving up.

All of the intermediate responses in a redirect chain are available in aResponse's.history field; each.history points 1 response earlier, forminga linked list ofResponse objects until the earliest response has a value ofNone. You can crawl up this linked list if you want to inspect the headers orother metadata of the intermediate redirects that brought you to your final value.

To use client certificate you need a PKCS 12 archive with private key and certificate.

requests.get("https://client.badssl.com",  cert="./badssl.com-client.p12")

If the p12 archive is password protected you can provide a second parameter:

requests.get("https://client.badssl.com",  cert= ("./badssl.com-client.p12","password"))

For test environments you may want to combinecert with theverifySslCerts = false option (if you have self signed SSL certificates on test servers).

requests.get("https://client.badssl.com",  cert= ("./badssl.com-client.p12","password"),  verifySslCerts=false)

You can also use a sslContext to provide a more customized ssl configuration

valsslContext:SSLContext=//initialized sslContextrequests.get("https://client.badssl.com",  sslContext= sslContext)

Arequests.Session automatically handles sending/receiving/persisting cookiesfor you across multiple requests:

vals= requests.Session()valr= s.get("https://httpbin.org/cookies/set?freeform=test")valr2= s.get("https://httpbin.org/cookies")r2.text()// {"cookies":{"freeform":"test"}}

If you want to deal with a website that uses cookies, it's usually easier to usearequests.Session rather than passing aroundcookie variables manually.

Apart from persisting cookies, sessions are also useful for consolidating commonconfiguration that you want to use across multiple requests, e.g. customheaders, cookies or other things:

vals= requests.Session(  headers=Map("x-special-header"->"omg"),  cookieValues=Map("cookie"->"vanilla"))valr1= s.get("https://httpbin.org/cookies")r1.text()// {"cookies":{"cookie":"vanilla"}}valr2= s.get("https://httpbin.org/headers")r2.text()// {"headers":{"X-Special-Header":"omg", ...}}

There is a whole zoo of HTTP clients in the Scala ecosystem. Akka-http, Play-WS,STTP, HTTP4S, Scalaj-HTTP, RosHTTP, Dispatch. Nevertheless, none of them comeclose to the ease and weightlessness of using Kenneth Reitz'sRequests library: too many implicits,operators, builders, monads, and other things.

When I want to make a HTTP request, I do not want to know about.unsafeRunSync, infix methods likesvc OK as.String, or define implicitActorSystems,ActorMaterializers, andExecutionContexts. So farsttp andscalaj-http come closest to what Iwant, but still fall short: both still use a pattern of fluent builders that tome doesn't fit how I think when making a HTTP request. I just want to call onefunction to make a HTTP request, and get back my HTTP response.

Most people will never reach the scale that asynchrony matters, and most ofthose who do reach that scale will only need it in a small number of specializedplaces, not everywhere.

Compare the getting-started code necessary for Requests-Scala against some othercommon Scala HTTP clients:

// Requests-Scalavalr= requests.get("https://api.github.com/search/repositories",  params=Map("q"->"http language:scala","sort"->"stars"))r.text()// {"login":"lihaoyi","id":934140,"node_id":"MDQ6VXNlcjkzNDE0MA==",...

// Akka-Httpimportakka.actor.ActorSystemimportakka.http.scaladsl.Httpimportakka.http.scaladsl.model._importakka.stream.ActorMaterializerimportscala.concurrent.Futureimportscala.util.{Failure,Success }implicitvalsystem=ActorSystem()implicitvalmaterializer=ActorMaterializer()// needed for the future flatMap/onComplete in the endimplicitvalexecutionContext= system.dispatchervalresponseFuture:Future[HttpResponse]=Http().singleRequest(HttpRequest(uri="http://akka.io"))responseFuture  .onComplete {caseSuccess(res)=> println(res)caseFailure(_)=> sys.error("something wrong")  }

// Play-WSimportakka.actor.ActorSystemimportakka.stream.ActorMaterializerimportplay.api.libs.ws._importplay.api.libs.ws.ahc._importscala.concurrent.FutureimportDefaultBodyReadables._importscala.concurrent.ExecutionContext.Implicits._// Create Akka system for thread and streaming managementimplicitvalsystem=ActorSystem()implicitvalmaterializer=ActorMaterializer()// Create the standalone WS client// no argument defaults to a AhcWSClientConfig created from// "AhcWSClientConfigFactory.forConfig(ConfigFactory.load, this.getClass.getClassLoader)"valwsClient=StandaloneAhcWSClient()wsClient.url("http://www.google.com").get()  .map { response⇒valstatusText:String= response.statusTextvalbody= response.body[String]    println(s"Got a response$statusText")  }.  andThen {case _=> wsClient.close() }  andThen {case _=> system.terminate() }

// Http4simportorg.http4s.client.dsl.io._importorg.http4s.headers._importorg.http4s.MediaTypevalrequest=GET(Uri.uri("https://my-lovely-api.com/"),Authorization(Credentials.Token(AuthScheme.Bearer,"open sesame")),Accept(MediaType.application.json))httpClient.expect[String](request)

// sttpimportsttp.client3._valrequest= basicRequest.response(asStringAlways)  .get(uri"https://api.github.com/search"    .addParams(Map("q"->"http language:scala","sort"->"stars")))valbackend=HttpURLConnectionBackend()valresponse= backend.send(request)println(response.body)

// Dispatchimportdispatch._,Defaults._valsvc= url("http://api.hostip.info/country.php")valcountry=Http.default(svcOK as.String)

The existing clients require a complex mix of imports, implicits, operators, andDSLs. The goal of Requests-Scala is to do away with all of that: your HTTPrequest is just a function call that takes parameters; that is all you need toknow.

As it turns out, Kenneth Reitz's Requests isnot a lot of code.Most of the heavy lifting is done in other libraries, and his library is a justthin-shim that makes the API 10x better. Similarly, it turns out on the JVM most of theheavy lifting is also done for you. There have always been options, butsince JDK 11 a decent HTTP client is provided in the standard library.

Given that's the case, how hard can it be to port over a dozen Python files toScala? This library attempts to do that: class by class, method by method,keyword-argument by keyword-argument. Not everything has been implemented yet,some things differ (some avoidably, some unavoidably), and it's nowhere near aspolished, but you should definitely try it out as the HTTP client for your nextcodebase or project!

• Use JDK 11 HttpClient (#158). Notethat this means we are dropping compatibility with JDK 8, and will require JDK 11 and abovegoing forward. People who need to use JDK 8 can continue using version 0.8.3

• Fix handling of HTTP 304 (#159)

• fix: content type header not present in multipart item (#154)

• Update Geny to 1.0.0#120

• Fix issue with data buffers not being flushed when compression is enabled#108

• Allowrequests.send(method)(...) to dynamically choose a HTTP method#94
• Avoid crashing on gzipped HEAD requests#95
• All exceptions now inherit from aRequestsException base class

• Add support for Scala 3.0.0-RC2

• requests.Response now implements thegeny.Readable interface, and can bedirectly passed to compatible APIs likeujson.read oros.write

• Add support for custom SSL certs

• Allow body content for DELETE requests

• Maderequests.{get,post,put,delete,head,options,patch}.stream return aReadable, allowing upickle andfastparse to operate directly on the streaming input

• requests.{get,post,put,delete,head,options,patch} now throw arequests.RequestFailedException(val response: Response) if a non-2xx statuscode is received. You can disable throwing the exception by passing incheck = false
• requests.{get,post,put,delete,head,options,patch}.stream now returns aWritable instead of takingcallbacks.

• Support for uploadinggeny.Writable data types inrequest bodies.

• Support for Scala 2.13.0 final

• SupportPATCH and other verbs

• Support forBearer token auth

• RequestBlob headers no longer over-write session headers

• Allow POSTs to take URL parameters
• Return response body for all 2xx response codes
• Always setContent-Length to 0 when request body is empty

• First Release