imroc/reqPublic

NotificationsYou must be signed in to change notification settings
Fork361
Star4.4k

Simple Go HTTP client with Black Magic

License

MIT license

4.4k stars 361 forks Branches Tags Activity

Star

Notifications

You must be signed in to change notification settings

Branches Tags

Folders and files

Name		Name	Last commit message	Last commit date
Latest commit History 1,028 Commits
.github/workflows		.github/workflows
.testdata		.testdata
examples		examples
http2		http2
internal		internal
pkg		pkg
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
client.go		client.go
client_impersonate.go		client_impersonate.go
client_test.go		client_test.go
client_wrapper.go		client_wrapper.go
decode.go		decode.go
decode_test.go		decode_test.go
digest.go		digest.go
dump.go		dump.go
go.mod		go.mod
go.sum		go.sum
header.go		header.go
http.go		http.go
http_request.go		http_request.go
logger.go		logger.go
logger_test.go		logger_test.go
middleware.go		middleware.go
parallel_download.go		parallel_download.go
redirect.go		redirect.go
req.go		req.go
req_test.go		req_test.go
request.go		request.go
request_test.go		request_test.go
request_wrapper.go		request_wrapper.go
response.go		response.go
retry.go		retry.go
retry_test.go		retry_test.go
roundtrip.go		roundtrip.go
roundtrip_js.go		roundtrip_js.go
server.go		server.go
textproto_reader.go		textproto_reader.go
trace.go		trace.go
transfer.go		transfer.go
transport.go		transport.go
transport_default_other.go		transport_default_other.go
transport_default_wasm.go		transport_default_wasm.go

Repository files navigation

req

Simple Go HTTP client with Black Magic

Documentation

Full documentation is available on the official website:https://req.cool.

Features

Simple and Powerful: Simple and easy to use, providing rich client-level and request-level settings, all of which are intuitive and chainable methods.
Easy Debugging: Powerful and convenient debug utilities, including debug logs, performance traces, and even dump the complete request and response content (seeDebugging).
Easy API Testing: API testing can be done with minimal code, no need to explicitly create any Request or Client, or even to handle errors (SeeQuick HTTP Test)
Smart by Default: Detect and decode to utf-8 automatically if possible to avoid garbled characters (SeeAuto Decode), marshal request body and unmarshal response body automatically according to the Content-Type.
Support Multiple HTTP Versions: SupportHTTP/1.1,HTTP/2, andHTTP/3, and can automatically detect the server side and select the optimal HTTP version for requests, you can also force the protocol if you want (SeeForce HTTP version).
Support Retry: Support automatic request retry and is fully customizable (SeeRetry).
HTTP Fingerprinting: Support http fingerprint impersonation, so that we can access websites that prohibit crawler programs by identifying http fingerprints (SeeHTTP Fingerprint).
Multiple Authentication Methods: You can use HTTP Basic Auth, Bearer Auth Token and Digest Auth out of box (seeAuthentication).
Easy Download and Upload: You can download and upload files with simple request settings, and even set a callback to show real-time progress (SeeDownload andUpload).
Exportable:req.Transport is exportable. Compared withhttp.Transport, it also supports HTTP3, dump content, middleware, etc. It can directly replace the Transport ofhttp.Client in existing projects, and obtain more powerful functions with minimal code change.
Extensible: Support Middleware for Request, Response, Client and Transport (SeeRequest and Response Middleware) andClient and Transport Middleware).

Get Started

Install

You first needGo installed (version 1.22+ is required), then you can use the below Go command to install req:

go get github.com/imroc/req/v3

Import

Import req to your code:

import"github.com/imroc/req/v3"

Basic Usage

# assume the following codes in main.go file$ cat main.go

package mainimport ("github.com/imroc/req/v3")funcmain() {req.DevMode()// Treat the package name as a Client, enable development modereq.MustGet("https://httpbin.org/uuid")// Treat the package name as a Request, send GET request.req.EnableForceHTTP1()// Force using HTTP/1.1req.MustGet("https://httpbin.org/uuid")}

$ go run main.go2022/05/19 10:05:07.920113 DEBUG [req] HTTP/2 GET https://httpbin.org/uuid:authority: httpbin.org:method: GET:path: /uuid:scheme: httpsuser-agent: req/v3 (https://github.com/imroc/req/v3)accept-encoding: gzip:status: 200date: Thu, 19 May 2022 02:05:08 GMTcontent-type: application/jsoncontent-length: 53server: gunicorn/19.9.0access-control-allow-origin:*access-control-allow-credentials:true{"uuid":"bd519208-35d1-4483-ad9f-e1555ae108ba"}2022/05/19 10:05:09.340974 DEBUG [req] HTTP/1.1 GET https://httpbin.org/uuidGET /uuid HTTP/1.1Host: httpbin.orgUser-Agent: req/v3 (https://github.com/imroc/req/v3)Accept-Encoding: gzipHTTP/1.1 200 OKDate: Thu, 19 May 2022 02:05:09 GMTContent-Type: application/jsonContent-Length: 53Connection: keep-aliveServer: gunicorn/19.9.0Access-Control-Allow-Origin:*Access-Control-Allow-Credentials:true{"uuid":"49b7f916-c6f3-49d4-a6d4-22ae93b71969"}

The sample code above is good for quick testing purposes, which useDevMode() to see request details, and send requests using global wrapper methods that use the default client behind the scenes to initiate the request.

In production, it is recommended to explicitly create a client, and then use the same client to send all requests, please see other examples below.

Videos

The following is a series of video tutorials for req:

Youtube Play List
BiliBili 播放列表 (Chinese)

More

Check more introduction, tutorials, examples, best practices and API references on theofficial website.

Simple GET

package mainimport ("fmt""github.com/imroc/req/v3""log")funcmain() {client:=req.C()// Use C() to create a client.resp,err:=client.R().// Use R() to create a request.Get("https://httpbin.org/uuid")iferr!=nil {log.Fatal(err)}fmt.Println(resp)}

{  "uuid": "a4d4430d-0e5f-412f-88f5-722d84bc2a62"}

Advanced GET

package mainimport ("fmt""github.com/imroc/req/v3""log""time")typeErrorMessagestruct {Messagestring`json:"message"`}typeUserInfostruct {Namestring`json:"name"`Blogstring`json:"blog"`}funcmain() {client:=req.C().SetUserAgent("my-custom-client").// Chainable client settings.SetTimeout(5*time.Second)varuserInfoUserInfovarerrMsgErrorMessageresp,err:=client.R().SetHeader("Accept","application/vnd.github.v3+json").// Chainable request settings.SetPathParam("username","imroc").// Replace path variable in url.SetSuccessResult(&userInfo).// Unmarshal response body into userInfo automatically if status code is between 200 and 299.SetErrorResult(&errMsg).// Unmarshal response body into errMsg automatically if status code >= 400.EnableDump().// Enable dump at request level, only print dump content if there is an error or some unknown situation occurs to help troubleshoot.Get("https://api.github.com/users/{username}")iferr!=nil {// Error handling.log.Println("error:",err)log.Println("raw content:")log.Println(resp.Dump())// Record raw content when error occurs.return  }ifresp.IsErrorState() {// Status code >= 400.fmt.Println(errMsg.Message)// Record error message returned.return  }ifresp.IsSuccessState() {// Status code is between 200 and 299.fmt.Printf("%s (%s)\n",userInfo.Name,userInfo.Blog)return  }// Unknown status code.log.Println("unknown status",resp.Status)log.Println("raw content:")log.Println(resp.Dump())// Record raw content when server returned unknown status code.}

Normally it will output (SuccessState):

roc (https://imroc.cc)

More Advanced GET

You can set up a unified logic for error handling on the client, so that each time you send a request you only need to focus on the success situation, reducing duplicate code.

package mainimport ("fmt""github.com/imroc/req/v3""log""time")typeErrorMessagestruct {Messagestring`json:"message"`}func (msg*ErrorMessage)Error()string {returnfmt.Sprintf("API Error: %s",msg.Message)}typeUserInfostruct {Namestring`json:"name"`Blogstring`json:"blog"`}varclient=req.C().SetUserAgent("my-custom-client").// Chainable client settings.SetTimeout(5*time.Second).EnableDumpEachRequest().SetCommonErrorResult(&ErrorMessage{}).OnAfterResponse(func(client*req.Client,resp*req.Response)error {ifresp.Err!=nil {// There is an underlying error, e.g. network error or unmarshal error.returnnil}iferrMsg,ok:=resp.ErrorResult().(*ErrorMessage);ok {resp.Err=errMsg// Convert api error into go errorreturnnil}if!resp.IsSuccessState() {// Neither a success response nor a error response, record details to help troubleshootingresp.Err=fmt.Errorf("bad status: %s\nraw content:\n%s",resp.Status,resp.Dump())}returnnil})funcmain() {varuserInfoUserInforesp,err:=client.R().SetHeader("Accept","application/vnd.github.v3+json").// Chainable request settingsSetPathParam("username","imroc").SetSuccessResult(&userInfo).// Unmarshal response body into userInfo automatically if status code is between 200 and 299.Get("https://api.github.com/users/{username}")iferr!=nil {// Error handling.log.Println("error:",err)return}ifresp.IsSuccessState() {// Status code is between 200 and 299.fmt.Printf("%s (%s)\n",userInfo.Name,userInfo.Blog)}}

Simple POST

package mainimport ("fmt""github.com/imroc/req/v3""log")typeRepostruct {Namestring`json:"name"`Urlstring`json:"url"`}typeResultstruct {Datastring`json:"data"`}funcmain() {client:=req.C().DevMode()varresultResultresp,err:=client.R().SetBody(&Repo{Name:"req",Url:"https://github.com/imroc/req"}).SetSuccessResult(&result).Post("https://httpbin.org/post")iferr!=nil {log.Fatal(err)  }if!resp.IsSuccessState() {fmt.Println("bad response status:",resp.Status)return  }fmt.Println("++++++++++++++++++++++++++++++++++++++++++++++++")fmt.Println("data:",result.Data)fmt.Println("++++++++++++++++++++++++++++++++++++++++++++++++")}

2022/05/19 20:11:00.151171 DEBUG [req] HTTP/2 POST https://httpbin.org/post:authority: httpbin.org:method: POST:path: /post:scheme: httpsuser-agent: req/v3 (https://github.com/imroc/req/v3)content-type: application/json; charset=utf-8content-length: 55accept-encoding: gzip{"name":"req","website":"https://github.com/imroc/req"}:status: 200date: Thu, 19 May 2022 12:11:00 GMTcontent-type: application/jsoncontent-length: 651server: gunicorn/19.9.0access-control-allow-origin:*access-control-allow-credentials: true{  "args": {},  "data": "{\"name\":\"req\",\"website\":\"https://github.com/imroc/req\"}",  "files": {},  "form": {},  "headers": {    "Accept-Encoding": "gzip",    "Content-Length": "55",    "Content-Type": "application/json; charset=utf-8",    "Host": "httpbin.org",    "User-Agent": "req/v3 (https://github.com/imroc/req/v3)",    "X-Amzn-Trace-Id": "Root=1-628633d4-7559d633152b4307288ead2e"  },  "json": {    "name": "req",    "website": "https://github.com/imroc/req"  },  "origin": "103.7.29.30",  "url": "https://httpbin.org/post"}++++++++++++++++++++++++++++++++++++++++++++++++data: {"name":"req","url":"https://github.com/imroc/req"}++++++++++++++++++++++++++++++++++++++++++++++++

Do API Style

If you like, you can also use a Do API style like the following to make requests:

package mainimport ("fmt""github.com/imroc/req/v3")typeAPIResponsestruct {Originstring`json:"origin"`Urlstring`json:"url"`}funcmain() {varrespAPIResponsec:=req.C().SetBaseURL("https://httpbin.org/post")err:=c.Post().SetBody("hello").Do().Into(&resp)iferr!=nil {panic(err)}fmt.Println("My IP is",resp.Origin)}

My IP is 182.138.155.113

The order of chain calls is more intuitive: first call Client to create a request with a specified Method, then use chain calls to set the request, then useDo() to fire the request, return Response, and finally callResponse.Into to unmarshal response body into specified object.
Response.Into will return an error if an error occurs during sending the request or during unmarshalling.
The url of some APIs is fixed, and different types of requests are implemented by passing different bodies. In this scenario,Client.SetBaseURL can be used to set a unified url, and there is no need to set the url for each request when initiating a request. Of course, you can also callRequest.SetURL to set it if you need it.

Build SDK With Req

Here is an example of building GitHub's SDK with req, using two styles (GetUserProfile_Style1,GetUserProfile_Style2).

import ("context""fmt""github.com/imroc/req/v3")typeErrorMessagestruct {Messagestring`json:"message"`}// Error implements go error interface.func (msg*ErrorMessage)Error()string {returnfmt.Sprintf("API Error: %s",msg.Message)}typeGithubClientstruct {*req.Client}funcNewGithubClient()*GithubClient {return&GithubClient{Client:req.C().SetBaseURL("https://api.github.com").SetCommonErrorResult(&ErrorMessage{}).EnableDumpEachRequest().OnAfterResponse(func(client*req.Client,resp*req.Response)error {ifresp.Err!=nil {// There is an underlying error, e.g. network error or unmarshal error.returnnil}iferrMsg,ok:=resp.ErrorResult().(*ErrorMessage);ok {resp.Err=errMsg// Convert api error into go errorreturnnil}if!resp.IsSuccessState() {// Neither a success response nor a error response, record details to help troubleshootingresp.Err=fmt.Errorf("bad status: %s\nraw content:\n%s",resp.Status,resp.Dump())}returnnil}),}}typeUserProfilestruct {Namestring`json:"name"`Blogstring`json:"blog"`}// GetUserProfile_Style1 returns the user profile for the specified user.// Github API doc: https://docs.github.com/en/rest/users/users#get-a-userfunc (c*GithubClient)GetUserProfile_Style1(ctx context.Context,usernamestring) (user*UserProfile,errerror) {_,err=c.R().SetContext(ctx).SetPathParam("username",username).SetSuccessResult(&user).Get("/users/{username}")return}// GetUserProfile_Style2 returns the user profile for the specified user.// Github API doc: https://docs.github.com/en/rest/users/users#get-a-userfunc (c*GithubClient)GetUserProfile_Style2(ctx context.Context,usernamestring) (user*UserProfile,errerror) {err=c.Get("/users/{username}").SetPathParam("username",username).Do(ctx).Into(&user)return}