- Notifications
You must be signed in to change notification settings - Fork20
Highly configurable JSON format logging per Location - nginx logging module - aka. kasha 🍲
License
fooinha/nginx-json-log
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
nginx modules for logging in custom json format - aka kasha (🍲)
This module adds to nginx the ability of advanced JSON logging of HTTP requests per location.
This module adds to nginx the ability of advanced JSON logging of Stream connections per server.
It's possible to log to a destination ouput any request/session made to a specific nginx location.
The output format is configurable.
It also allows to log complex and multi-level JSON documents.
It supports logging to text file, to syslog or to a kafka topic.
It supports multiple output destinations with multiple formats for a location.
Current version released is 0.0.8.
Stream logging is only available when using nginx (>= 1.11.2).
That are many use cases.
Many things can be done by using the access log data.
Having it in JSON format makes easier for integration with other platforms and applications.
A quick example:
Each logging configuration is based on a json_log_format. (🍲)
A json_log_format is a ';' separated list of items to include in the logging preparation.
The left hand side part of item will be the JSON Path for the variable name
The left hand side part can be prefixed with 's:', 'i:', 'r:', 'b:' or 'n:', so, the JSON encoding type can be controlled.
- 's:' - JSON string ( default )
- 'i:' - JSON integer
- 'r:' - JSON real
- 'b:' - JSON boolean
- 'n:' - JSON null
Additional prefix:
- 'a:' - JSON Array - MUST be used before other prefixes. All keys with same name and defined as array will be its values grouped together in an array. ( see example below )
The right hand side will be the variable's name or literal value.For this, known or previously setted variables, can be used by using the '$' before name.
Common HTTP nginx builtin variables like $uri, or any other variable set by other handler modules can be used.
Common STREAM nginx builtin variables like $remote_addr, or any other variable set by other handler modules can be used.
Additional variables are provided by this module. See the available variables below atVariables section.
The output is sent to the location specified by the first json_log_format argument.The possible output locations are:
- "file:" - The logging location will be a local filesystem file.
- "kafka:" - The logging location will be a Kafka topic.
For HTTP logging, if kafka output is used the value from $request_id nginx variable will be used to set kafka's message id.
The $request_id is only available for nginx (>=1.11.0).
json_log_format my_log 'src.ip $remote_addr;src.port $remote_port;dst.ip $server_addr;dst.port $server_port;_date $time_iso8601;r:_real 1.1;i:_int 2016;i:_status $status;b:_notrack false;_literal root;comm.proto http;comm.http.method $request_method;comm.http.path $uri;comm.http.host $host;comm.http.server_name $server_name;a:i:list 1;a:list string;'; json_log file:/tmp/log my_log;'json_log syslog:server=unix:/dev/log my_log;'
This will produce the following JSON line to '/tmp/log' file .To ease reading, it's shown here formatted with newlines.
{"_date":"2016-12-11T18:06:54+00:00","_int":2016,"_literal":"root","_real":1.1,"_status":200,"_notrack":false,"comm": {"http": {"host":"localhost","method":"HEAD","path":"/index.html","server_name":"localhost" },"proto":"http" },"dst": {"ip":"127.0.0.1","port":"80" },"src": {"ip":"127.0.0.1","port":"52136" },"list": [1,"string" ]}perl_set $bar 'sub {my $r = shift;my $uri = $r->uri;return "yes" if $uri =~ /^\/bar/;return "";}';json_log_format with_perl 'comm.http.server_name $server_name;perl.bar $bar;'; json_log file:/tmp/log with_perl'
A request sent to/bar .
{"comm": {"http": {"server_name":"localhost" } },"perl": {"bar":"yes" }}json_log_format stream '_date $time_iso8601;src.addr $remote_addr;src.port $remote_portsession.bytes.received $bytes_received;session.bytes.sent $bytes_received;session.connection $connection;session.protocol $protocol;session.time $session_time;session.status $status;'; json_log file:/tmp/stream.log stream'
{"_date":"2017-04-09T20:52:41+00:00","session": {"bytes": {"received":"23","sent":"23" },"connection":"1","protocol":"TCP","status":"200","time":"6.875" },"src": {"addr":"127.0.0.1" }}The value of key of typestring can be encoded in 3 ways.To change the encoding output this should be configured per output key by specifying the encoding after a "|" character.For instance forbase64 output the key should be like thiskey|base64
The default encoding isplain when no encoding is supplied.
- plain
- base64
- hex
- hexdump
Format example:
json_log_format body_log 'http.req.body_plain $http_json_log_req_body;http.req.body_hex|hex $http_json_log_req_body;http.req.body_hexdump|hexdump $http_json_log_req_body;http.req.body_base64|base64 $http_json_log_req_body;';
Note: use a different key name if the output with different encoding is wanted for the same $var .
Response:
{"http": {"req": {"body_base64":"Zm9vPWJhcg==","body_hex":"666f6f3d626172","body_hexdump": ["66 6f 6f 3d 62 61 72 .. .. .. .. .. .. .. .. .. |foo=bar |" ],"body_plain":"foo=bar" } }}- Syntax:json_log_formatformat_name {format }if=...;
- Default: —
- Context: http main
- Context: stream main
Specifies a format name that can be used by ajson_log directive
See details above.
Works the same way asif argument from httpaccess_log directive.
- Syntax:json_loglocationformat_name
- Default: —
- Context: http location
- Context: stream server
Specifies the location for the output...
The output location type should be prefixed with supported location types. (file: orkafka: )
For afile: type the value part will be a local file name. e.g.file:/tmp/log
For akafka: type the value part will be the topic name. e.g.kafka: topic
The kafka output only happens if a list of brokers is defined byjson_log_kafka_brokers directive.
The format to use when writing to output destination.
- Syntax:json_log_kafka_brokers list of brokers separated by spaces;
- Default: —
- Context: http main
- Context: stream main
- Syntax:json_log_kafka_client_idid;
- Default: nginx
- Context: http main
- Context: stream main
- Syntax:json_log_kafka_compressioncompression_codec;
- Default: snappy
- Context: http main
- Context: stream main
- Syntax:json_log_kafka_log_levelnumeric_log_level;
- Default: 6
- Context: http main
- Context: stream main
- Syntax:json_log_kafka_max_retriesnumeric;
- Default: 0
- Context: http main
- Context: stream main
- Syntax:json_log_kafka_buffer_max_messagesnumeric;
- Default: 100000
- Context: http main
- Context: stream main
- Syntax:json_log_kafka_backoff_msnumeric;
- Default: 10
- Context: http main
- Context: stream main
- Syntax:json_log_kafka_partitionpartition;
- Default: RD_KAFKA_PARTITION_UA
- Context: http main
- Context: stream main
Creates a json object with all request headers.
Example:
"req": { "headers": { "Host": "localhost", "User-Agent": "curl/7.52.1", "Accept": "*/*" } }Log request body.
Example:
"req": { "body": "foo=bar" }Log request body encoded as hexdump array.
Example:
"hex": [ "31 32 33 34 35 36 37 38 39 41 42 43 44 45 46 30 |123456789ABCDEF0|", "31 32 33 34 35 36 .. .. .. .. .. .. .. .. .. .. |123456 |" ]Creates a json string with all response headers.
Example:
resp": { "headers": "HTTP/1.1 204 No Content\r\nServer: nginx/1.21.2\r\nDate: Mon, 09 Aug 2021 15:24:39 GMT\r\nConnection: keep-alive\r\nX-Foo: bar\r\n\r\n", }If nginx is used for mail proxy andauth_http is set, then, it's possible to log the HTTP requests for authentication service.
An example with full configuration.
mail { server_name mail.local; auth_http localhost:9876/; server { listen 25; protocol smtp; smtp_auth none; }}http { server { listen 9876; location / { add_header Auth-Status OK; add_header Auth-Server 127.0.0.1; add_header Auth-Port 2525; root html; index index.html index.htm; json_log_format mail_log ' _date $time_iso8601; headers $http_json_log_req_headers; '; json_log file:/tmp/mail.log mail_log; } error_page 405 =200 $uri ; error_page 404 =200 /index.html ; }}{ "_date": "2017-04-12T23:58:07+00:00", "headers": { "Host": "localhost", "Auth-Method": "none", "Auth-User": "", "Auth-Pass": "", "Auth-Protocol": "smtp", "Auth-Login-Attempt": "1", "Client-IP": "127.0.0.1", "Client-Host": "[UNAVAILABLE]", "Auth-SMTP-Helo": "atacker.cloud", "Auth-SMTP-From": "mail from: <bob>", "Auth-SMTP-To": "rcpt to: <alice>" }}Attention: The Auth-Pass is not masked in any way, nor removed, so the client's value will be logged.
For Ubuntu or Debian install development packages.
$ sudo apt-get install libjansson-dev librdkafka-dev
Build as a common nginx module.After configure we need to modify the default order of the modules.
$ ./configure --add-module=/build/ngx_json_log$ cat objs/ngx_modules.c| awk'{ sub("\"ngx_http_json_log_module\",", ""); sub("\"ngx_http_header_filter_module\"", "\"ngx_http_json_log_module\", \"ngx_http_header_filter_module\""); print}'> objs/ngx_modules.c.dat&& mv objs/ngx_modules.c.dat objs/ngx_modules.c$ make&& make install
THIS IS NOT PRODUCTION ready.
So there's no guarantee of success. It most probably blow up when running in real life scenarios.
The unit tests usehttps://github.com/openresty/test-nginx framework.
$ git clone https://github.com/openresty/test-nginx.git$ cd test-nginx/$ cpanm .$ export PATH=$PATH:/usr/local/nginx/sbin/At project root just run the prove command:
$ provet/0001_simple_file_log.t .. okAll tests successful.Files=1, Tests=12, 0 wallclock secs ( 0.02 usr 0.01 sys + 0.15 cusr 0.00 csys = 0.18 CPU)Result: PASSMaking 1M requests to local destination.
$ time ab -n 1000000 -c 4 http://127.0.0.1/megaThis is ApacheBench, Version 2.3 <$Revision: 1757674 $>Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/Licensed to The Apache Software Foundation, http://www.apache.org/Benchmarking 127.0.0.1 (be patient)Completed 100000 requestsCompleted 200000 requestsCompleted 300000 requestsCompleted 400000 requestsCompleted 500000 requestsCompleted 600000 requestsCompleted 700000 requestsCompleted 800000 requestsCompleted 900000 requestsCompleted 1000000 requestsFinished 1000000 requestsServer Software: nginx/1.10.3Server Hostname: 127.0.0.1Server Port: 80Document Path: /megaDocument Length: 169 bytesConcurrency Level: 4Time taken for tests: 95.224 secondsComplete requests: 1000000Failed requests: 0Non-2xx responses: 1000000Total transferred: 319000000 bytesHTML transferred: 169000000 bytesRequests per second: 10501.55 [#/sec] (mean)Time per request: 0.381 [ms] (mean)Time per request: 0.095 [ms] (mean, across all concurrent requests)Transfer rate: 3271.48 [Kbytes/sec] receivedConnection Times (ms) min mean[+/-sd] median maxConnect: 0 0 0.2 0 64Processing: 0 0 0.4 0 98Waiting: 0 0 0.4 0 97Total: 0 0 0.5 0 115Percentage of the requests served within a certain time (ms) 50% 0 66% 0 75% 0 80% 0 90% 1 95% 1 98% 1 99% 1 100% 115 (longest request)real 1m36.057suser 0m5.390ssys 1m22.040s$ wc -l /tmp/1M.log1000000 /tmp/1M.log$ time ab -n 1000000 -c 4 http://127.0.0.1/megaThis is ApacheBench, Version 2.3 <$Revision: 1757674 $>Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/Licensed to The Apache Software Foundation, http://www.apache.org/Benchmarking 127.0.0.1 (be patient)Completed 100000 requestsCompleted 200000 requestsCompleted 300000 requestsCompleted 400000 requestsCompleted 500000 requestsCompleted 600000 requestsCompleted 700000 requestsCompleted 800000 requestsCompleted 900000 requestsCompleted 1000000 requestsFinished 1000000 requestsServer Software: nginx/1.10.3Server Hostname: 127.0.0.1Server Port: 80Document Path: /megaDocument Length: 169 bytesConcurrency Level: 4Time taken for tests: 102.439 secondsComplete requests: 1000000Failed requests: 0Non-2xx responses: 1000000Total transferred: 319000000 bytesHTML transferred: 169000000 bytesRequests per second: 9761.95 [#/sec] (mean)Time per request: 0.410 [ms] (mean)Time per request: 0.102 [ms] (mean, across all concurrent requests)Transfer rate: 3041.08 [Kbytes/sec] receivedConnection Times (ms) min mean[+/-sd] median maxConnect: 0 0 0.3 0 93Processing: 0 0 2.1 0 1021Waiting: 0 0 2.1 0 1021Total: 0 0 2.1 0 1022Percentage of the requests served within a certain time (ms) 50% 0 66% 0 75% 0 80% 0 90% 1 95% 1 98% 1 99% 2 100% 1022 (longest request)real 1m43.328suser 0m5.770ssys 1m21.380sDocker images and a docker compose file is available at the ./docker directory.
$ docker-compose up --build -dCreating nginx-json-zookeeperCreating nginx-json-kafkaCreating nginx-json...
$ docker-compose ps Name Command State Ports------------------------------------------------------------------------------------------------------------------nginx-json /bin/sh -c /usr/local/ngin ... Up 0.0.0.0:80->80/tcpnginx-json-kafka start-kafka.sh Up 0.0.0.0:9092->9092/tcpnginx-json-zookeeper /bin/sh -c /usr/sbin/sshd ... Up 0.0.0.0:2181->2181/tcp, 22/tcp, 2888/tcp, 3888/tcpAn additional docker service for development it's available.
Just uncomment the nginx-json-dev service block.
Also docker services for the ELK stack are available.
Name Command State Ports------------------------------------------------------------------------------------------------------------------------------------------nginx-json /bin/sh -c /usr/local/ngin ... Up 0.0.0.0:80->80/tcpnginx-json-dev /bin/bash Up 0.0.0.0:81->81/tcpnginx-json-elasticsearch /bin/bash bin/es-docker Up 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcpnginx-json-kafka start-kafka.sh Up 0.0.0.0:9092->9092/tcpnginx-json-kibana /bin/sh -c /usr/local/bin/ ... Up 0.0.0.0:5601->5601/tcpnginx-json-logstash /usr/local/bin/docker-entr ... Up 0.0.0.0:5044->5044/tcp, 0.0.0.0:5050->5050/tcp, 0.0.0.0:9600->9600/tcpnginx-json-zookeeper /bin/sh -c /usr/sbin/sshd ... Up 0.0.0.0:2181->2181/tcp, 22/tcp, 2888/tcp, 3888/tcpAbout
Highly configurable JSON format logging per Location - nginx logging module - aka. kasha 🍲
Topics
Resources
License
Uh oh!
There was an error while loading.Please reload this page.