- Notifications
You must be signed in to change notification settings - Fork0
💣 INFINI Loadgen for API Benchmark & API Testing.
License
infinilabs/loadgen
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
Highlights of Loadgen:
- Robust performance
- Lightweight and dependency-free
- Random selection of template-based parameters
- High concurrency
- Balanced traffic control at the benchmark end
- Validate server responses.
Install with script:
curl -sSL http://get.infini.cloud | bash -s -- -p loadgenOr download from here:http://release.infinilabs.com/loadgen/
➜ /tmp mkdir loadgen➜ /tmp curl -sSL http://get.infini.cloud | bash -s -- -p loadgen -d /tmp/loadgen @@@@@@@@@@@ @@@@@@@@@@@@ @@@@@@@@@@@@ @@@@@@@@@&@@@ #@@@@@@@@@@@@@ @@@ @@@@@@@@@@@@@ &@@@@@@@ &@@@@@@@@@@@@@ @&@@@@@@@&@ @@@&@@@@@@@&@ @@@@@@@@@@@@@@@@ @@@@@@@@@@@@@@ @@@@@@@@@@@@@@@@@@& @@@@@@@@@@@@@ %@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@@@&@@@@@@@@@@@@@@@ @@ ,@@@@@@@@@@@@@@@@@@@@@@@& @@@@@. @@@@@&@@@@@@@@@@@@@@ @@@@@@@@@@ @@@@@@@@@@@@@@@# @&@@@&@@@&@@@ &@&@@@&@@@&@ @@@@@@@@@@@@@. @@@@@@@* @@@@@@@@@@@@@ %@@@ @@@@@@@@@@@@@/@@@@@@@&@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Welcome to INFINI Labs!Now attempting the installation...Name: [loadgen], Version: [1.26.1-598], Path: [/tmp/loadgen]File: [https://release.infinilabs.com/loadgen/stable/loadgen-1.26.1-598-mac-arm64.zip]##=O#- #Installation complete. [loadgen] is ready to use!----------------------------------------------------------------cd /tmp/loadgen && ./loadgen-mac-arm64---------------------------------------------------------------- __ _ __ ____ __ _ __ __ / // |/ // __// // |/ // / / // || // _/ / // || // //_//_/|_//_/ /_//_/|_//_/©INFINI.LTD, All Rights Reserved.Loadgen is easy to use. After the tool is downloaded and decompressed, two files are obtained: one executable program and one configuration fileloadgen.yml. An example of the configuration file is as follows:
env: ES_USERNAME: elastic ES_PASSWORD: elasticrunner: # total_rounds: 1 no_warm: false log_requests: false assert_invalid: false assert_error: falsevariables: - name: ip type: file path: test/ip.txt - name: user type: file path: test/user.txt - name: id type: sequence - name: uuid type: uuid - name: now_local type: now_local - name: now_utc type: now_utc - name: now_unix type: now_unixrequests: - request: method: GET basic_auth: username: $[[env.ES_USERNAME]] password: $[[env.ES_PASSWORD]] url: http://localhost:8000/medcl/_search body: '{ "query": {"match": { "name": "$[[user]]" }}}'By default,loadgen will run under the benchmarking mode, repeating through all therequests during the specified duration (-d). If you only need to test the responses, settingrunner.total_rounds: 1 will letloadgen run for only once.
By default,loadgen will canonilize the HTTP response header keys received from the server side (user-agent: xxx ->User-Agent: xxx). If you need to assert the header keys exactly, you can setrunner.disable_header_names_normalizing: true to disable this behavior.
In the above configuration,variables is used to define variable parameters and variables are identified byname. In a constructed request,$[[Variable name]] can be used to access the value of the variable. Supported variable types are as follows:
| Type | Description | Parameters |
|---|---|---|
file | Load variables from file | path: the path of the data filesdata: a list of values, will get appended to the end of the data specified bypath file |
list | Defined variables inline | usedata to define a string array |
sequence | 32-bit Variable of the auto incremental numeric type | from: the minimum of the valuesto: the maximum of the values |
sequence64 | 64-bit Variable of the auto incremental numeric type | from: the minimum of the valuesto: the maximum of the values |
range | Variable of the range numbers, support parametersfrom andto to define the range | from: the minimum of the valuesto: the maximum of the values |
random_array | Generate a random array from the variable specified byvariable_key | variable_key: the variable name for the source of array valuessize: the size of arraysquare_bracket:true/false, whether to add[] for the outputed arraystring_bracket: the string to surround the outputed elements. |
uuid | Variable of the UUID character type | |
now_local | Current time and local time zone | |
now_utc | Current time and UTC time zone | |
now_unix | Current time and Unix timestamp | |
now_with_format | Current time，support parameterformat to customize the output format， eg:2006-01-02T15:04:05-0700 | format: the format of the time output (Example) |
Variable parameters of thefile type are loaded from an external text file. One variable parameter occupies one line. When one variable of the file type is accessed, one variable value is taken randomly. An example of the variable format is as follows:
➜ loadgen git:(master) ✗ cat test/user.txtmedclelasticTips about how to generate a random string of fixed length, such as 1024 per line:
LC_CTYPE=C tr -dc A-Za-z0-9_\!\@\#\$\%\^\&\*\(\)-+= < /dev/random | head -c 1024 >> 1k.txtloadgen supporting loading and using environment variables inloadgen.yml, you can specify the default values inenv configuration.loadgen will overwrite the variables at runtime if they're also specified by the command-line environment.
The environment variables can be access by$[[env.ENV_KEY]]:
# Default values for the environment variables.env: ES_USERNAME: elastic ES_PASSWORD: elastic ES_ENDPOINT: http://localhost:8000requests: - request: method: GET basic_auth: username: $[[env.ES_USERNAME]] # Use environment variables password: $[[env.ES_PASSWORD]] # Use environment variables url: $[[env.ES_ENDPOINT]]/medcl/_search # Use environment variables body: '{ "query": {"match": { "name": "$[[user]]" }}}'Therequests node is used to set requests to be executed by Loadgen in sequence. Loadgen supports fixed-parameter requests and requests constructed using template-based variable parameters. The following is an example of a common query request.
requests: - request: method: GET basic_auth: username: elastic password: pass url: http://localhost:8000/medcl/_search?q=name:$[[user]]In the above query, Loadgen conducts queries based on themedcl index and executes one query based on thename field. The value of each request is from the random variableuser.
It is very easy to use Loadgen to simulate bulk ingestion. Configure one index operation in the request body and then use thebody_repeat_times parameter to randomly replicate several parameterized requests to complete the preparation of a batch of requests. See the following example.
- request: method: POST basic_auth: username: test password: testtest url: http://localhost:8000/_bulk body_repeat_times: 1000 body: | { "index" : { "_index" : "medcl-y4","_type":"doc", "_id" : "$[[uuid]]" } } { "id" : "$[[id]]","field1" : "$[[user]]","ip" : "$[[ip]]","now_local" : "$[[now_local]]","now_unix" : "$[[now_unix]]" }You can use theassert configuration to check the response values.assert now supports most of all thecondition checkers of INFINI Gateway.
requests: - request: method: GET basic_auth: username: elastic password: pass url: http://localhost:8000/medcl/_search?q=name:$[[user]] assert: equals: _ctx.response.status: 201The response value can be accessed from the_ctx value, currently it contains these values:
| Parameter | Description |
|---|---|
_ctx.response.status | HTTP response status code |
_ctx.response.header | HTTP response headers |
_ctx.response.body | HTTP response body text |
_ctx.response.body_json | If the HTTP response body is a valid JSON string, you can access the JSON fields bybody_json |
_ctx.elapsed | The time elapsed since request sent to the server (milliseconds) |
If the request failed (e.g. the host is not reachable),loadgen will record it underNumber of Errors as part of the testing output. If you configuredrunner.assert_error: true,loadgen will exit asexit(2) when there're any requests failed.
If the assertion failed,loadgen will record it underNumber of Invalid as part of the testing output and skip the subsequent requests in this round. If you configuredrunner.assert_invalid: true,loadgen will exit asexit(1) when there're any assertions failed.
Each request can useregister to dynamically set the variables based on the response value, a common usage is to update the parameters of the later requests based on the previous responses.
In the below example, we're registering the response value_ctx.response.body_json.test.settings.index.uuid of the$[[env.ES_ENDPOINT]]/test to theindex_id variable, then we can access it by$[[index_id]].
requests: - request: method: GET url: $[[env.ES_ENDPOINT]]/test assert: equals: _ctx.response.status: 200 register: - index_id: _ctx.response.body_json.test.settings.index.uuidRun Loadgen to perform the benchmark test as follows:
➜ loadgen git:(master) ✗ ./bin/loadgen -d 30 -c 100 -compress __ ___ _ ___ ___ __ __ / / /___\/_\ / \/ _ \ /__\/\ \ \ / / // ///_\\ / /\ / /_\//_\ / \/ // /__/ \_// _ \/ /_// /_\\//__/ /\ /\____|___/\_/ \_/___,'\____/\__/\_\ \/[LOADGEN] A http load generator and testing suit.[LOADGEN] 1.0.0_SNAPSHOT, 83f2cb9, Sun Jul 4 13:52:42 2021 +0800, medcl, support single item in dict files[07-19 16:15:00] [INF] [instance.go:24] workspace: data/loadgen/nodes/0[07-19 16:15:00] [INF] [loader.go:312] warmup started[07-19 16:15:00] [INF] [app.go:306] loadgen now started.[07-19 16:15:00] [INF] [loader.go:316] [GET] http://localhost:8000/medcl/_search[07-19 16:15:00] [INF] [loader.go:317] status: 200,<nil>,{"took":1,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":null,"hits":[]}}[07-19 16:15:00] [INF] [loader.go:316] [GET] http://localhost:8000/medcl/_search?q=name:medcl[07-19 16:15:00] [INF] [loader.go:317] status: 200,<nil>,{"took":1,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":null,"hits":[]}}[07-19 16:15:01] [INF] [loader.go:316] [POST] http://localhost:8000/_bulk[07-19 16:15:01] [INF] [loader.go:317] status: 200,<nil>,{"took":120,"errors":false,"items":[{"index":{"_index":"medcl-y4","_type":"doc","_id":"c3qj9123r0okahraiej0","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":5735852,"_primary_term":3,"status":201}}]}[07-19 16:15:01] [INF] [loader.go:325] warmup finished209 requests finished in 10.031365126s, 0.00bytes sent, 32.86KB received[Loadgen Client Metrics]Requests/sec:20.82Request Traffic/sec:0.00bytesTotal Transfer/sec:3.27KBFastest Request:1msSlowest Request:182.437792msStatus 302:209[Latency Metrics]209 samples of 209 eventsCumulative:10.031365126sHMean:46.31664msAvg.:47.996962msp50: 45.712292msp75:51.6065msp95:53.05475msp99:118.162416msp999:182.437792msLong 5%:87.678145msShort 5%:39.11217msMax:182.437792msMin:38.257791msRange:144.180001msStdDev:14.407579msRate/sec.:20.82[Latency Distribution] 38.257ms - 52.675ms ------------------------------ 52.675ms - 67.093ms -- 67.093ms - 81.511ms - 81.511ms - 95.929ms - 95.929ms - 110.347ms - 110.347ms - 124.765ms -[Estimated Server Metrics]Requests/sec:20.83Avg Req Time:47.996962msTransfer/sec:3.28KBLoadgen executes all requests once to warm up before the formal benchmark test. If an error occurs, a prompt is displayed, asking you whether to continue.The warm-up request results are also output to the terminal. After execution, an execution summary is output.You can setrunner.no_warm: true to skip the warm-up stage.
The final results of Loadgen are the cumulative statistics after all requests are executed, and they may be inaccurate. You are advised to start the Kibana dashboard to check all operating indicators of Elasticsearch in real time.
Loadgen cyclically executes requests defined in the configuration file. By default, Loadgen runs for5s and then automatically exits. If you want to prolong the running time or increase the concurrency, you can set the tool's startup parameters. The help commands are as follows:
➜ loadgen git:(master) ✗ ./bin/loadgen --helpUsage of ./bin/loadgen: -c int Number of concurrent threads (default 1) -compress Compress requests with gzip -config string the location of config file, default: loadgen.yml (default "loadgen.yml") -d int Duration of tests in seconds (default 5) -debug run in debug mode, loadgen will quit with panic error -l int Limit total requests (default -1) -log string the log level,options:trace,debug,info,warn,error (default "info") -r int Max requests per second (fixed QPS) (default -1) -vversionYou can use Loadgen and set the CLI parameter-r to restrict the number of requests that can be sent by the client per second, so as to evaluate the response time and load of Elasticsearch under fixed pressure. See the following example.
➜ loadgen git:(master) ✗ ./bin/loadgen -d 30 -c 100 -r 100Note: The client throughput limit may not be accurate enough in the case of massive concurrencies.
You can set the-l parameter to control the total number of requests that can be sent by the client, so as to generate a fixed number of documents. Modify the configuration as follows:
requests: - request: method: POST basic_auth: username: test password: testtest url: http://localhost:8000/medcl-test/doc2/_bulk body_repeat_times: 1 body: | { "index" : { "_index" : "medcl-test", "_id" : "$[[uuid]]" } } { "id" : "$[[id]]","field1" : "$[[user]]","ip" : "$[[ip]]" }Configured parameters use the content of only one document for each request. Then, the system executes Loadgen.
./bin/loadgen -config loadgen-gw.yml -d 600 -c 100 -l 50000After execution,50000 records are added for the Elasticsearch indexmedcl-test.
If the IDs of generated documents need to increase regularly to facilitate comparison, you can use the auto incremental IDs of thesequence type as the primary key and avoid using random numbers in the content. See the following example.
requests: - request: method: POST basic_auth: username: test password: testtest url: http://localhost:8000/medcl-test/doc2/_bulk body_repeat_times: 1 body: | { "index" : { "_index" : "medcl-test", "_id" : "$[[id]]" } } { "id" : "$[[id]]" }In a request, we might want use the same variable value, such as therouting parameter to control the shard destination, also store the field in the JSON document.You can useruntime_variables to set request-level variables, orruntime_body_line_variables to define request-body-level variables.If the request body setbody_repeat_times, each line will be different, as shown in the following example:
variables: - name: id type: sequence - name: uuid type: uuid - name: now_local type: now_local - name: now_utc type: now_utc - name: now_unix type: now_unix - name: suffix type: range from: 10 to: 15requests: - request: method: POST runtime_variables: batch_no: id runtime_body_line_variables: routing_no: uuid basic_auth: username: ingest password: password #url: http://localhost:8000/_search?q=$[[id]] url: http://192.168.3.188:9206/_bulk body_repeat_times: 10 body: | { "create" : { "_index" : "test-$[[suffix]]","_type":"doc", "_id" : "$[[uuid]]" , "routing" : "$[[routing_no]]" } } { "id" : "$[[uuid]]","routing_no" : "$[[routing_no]]","batch_number" : "$[[batch_no]]", "random_no" : "$[[suffix]]","ip" : "$[[ip]]","now_local" : "$[[now_local]]","now_unix" : "$[[now_unix]]" }We defined thebatch_no variable to represent the same batch number in a batch of documents, and therouting_no variable to represent the routing value at each document level.
requests: - request: method: GET url: http://localhost:8000/test/_search headers: - Agent: "Loadgen-1" disable_header_names_normalizing: falseBy default,loadgen will canonilize the HTTP header keys before sending the request (user-agent: xxx ->User-Agent: xxx), if you need to set the header keys exactly as is, setdisable_header_names_normalizing: true.
Loadgen also support simply the requests called DSL,for example, prepare a dsl file for loadgen, save asbulk.dsl:
POST /_bulk{"index": {"_index": "$[[env.INDEX_NAME]]", "_type": "_doc", "_id": "$[[uuid]]"}}{"id": "$[[id]]", "routing": "$[[routing_no]]", "batch": "$[[batch_no]]", "now_local": "$[[now_local]]", "now_unix": "$[[now_unix]]"}And specify the dsl file with parameterrun:
$ INDEX_NAME=medcl123 ES_ENDPOINT=https://localhost:9200 ES_USERNAME=admin ES_PASSWORD=b14612393da0d4e7a70b ./bin/loadgen -run bulk.dslNow you should ready to rock~
About
💣 INFINI Loadgen for API Benchmark & API Testing.