Nginx Integration
| Version | 3.0.0 (View all) |
| Subscription level What's this? | Basic |
| Developed by What's this? | Elastic |
| Ingestion method(s) | API, File |
| Minimum Kibana version(s) | 9.0.0 8.13.0 |
The Nginx integration allows you to monitorNginx servers. Time seriesindex mode enabled for metrics data stream.
Use the Nginx integration to collect metrics and logs from your server.Then visualize that data in Kibana, use the Machine Learning app to find unusual activity in HTTP access logs,create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue.
For example, if you wanted to be notified if a certain number of client requests failed in a given time period,you could install the Nginx integration to send logs to Elastic.Then, you could view the logs stream into Elastic in real time in the Observability Logs app.You could also set up a new log threshold rule in the Logs app to alert you when there are more thana certain number of events with a failing status in a given time period.
The Nginx integration collects two types of data: logs and metrics.
Logs help you keep a record of events that happen in your Nginx servers.This includes when a client request or error occurs.
Metrics give you insight into the state of your Nginx servers.This includes information like the total number of active client connections by status,the total number of client requests, and more.
You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it.You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.
Each data source was tested with a specific Nginx version.For more information see theLogs reference andMetrics reference.
On Windows, the module was tested with Nginx installed from the Chocolatey repository.
For step-by-step instructions on how to set up an integration, see theGetting started guide.
Timezone support
This data source parses logs that don’t contain timezone information. For these logs, the Elastic Agent reads the localtimezone and uses it when parsing to convert the timestamp to UTC. The timezone to be used for parsing is includedin the event in theevent.timezone field.
To disable this conversion, theevent.timezone field can be removed using thedrop_fields processor.
If logs originate from systems or applications with a timezone that is different than the local one,theevent.timezone field can be overwritten with the original timezone using theadd_fields processor.
Access logs collects the Nginx access logs.
The Nginx access logs stream was tested with Nginx 1.19.5.
Example
{ "@timestamp": "2022-12-09T10:39:23.000Z", "_tmp": {}, "agent": { "ephemeral_id": "34369a4a-4f24-4a39-9758-85fc2429d7e2", "id": "ef5e274d-4b53-45e6-943a-a5bcf1a6f523", "name": "docker-fleet-agent", "type": "filebeat", "version": "8.5.0" }, "data_stream": { "dataset": "nginx.access", "namespace": "ep", "type": "logs" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { "id": "ef5e274d-4b53-45e6-943a-a5bcf1a6f523", "snapshot": false, "version": "8.5.0" }, "event": { "agent_id_status": "verified", "category": [ "web" ], "created": "2022-12-09T10:39:38.896Z", "dataset": "nginx.access", "ingested": "2022-12-09T10:39:40Z", "kind": "event", "outcome": "success", "timezone": "+00:00", "type": [ "access" ] }, "host": { "architecture": "x86_64", "containerized": false, "hostname": "docker-fleet-agent", "id": "66392b0697b84641af8006d87aeb89f1", "ip": [ "172.18.0.7" ], "mac": [ "02-42-AC-12-00-07" ], "name": "docker-fleet-agent", "os": { "codename": "focal", "family": "debian", "kernel": "5.15.49-linuxkit", "name": "Ubuntu", "platform": "ubuntu", "type": "linux", "version": "20.04.5 LTS (Focal Fossa)" } }, "http": { "request": { "method": "GET" }, "response": { "body": { "bytes": 97 }, "status_code": 200 }, "version": "1.1" }, "input": { "type": "log" }, "log": { "file": { "path": "/tmp/service_logs/access.log" }, "offset": 0 }, "nginx": { "access": { "remote_ip_list": [ "127.0.0.1" ] } }, "related": { "ip": [ "127.0.0.1" ] }, "source": { "address": "127.0.0.1", "ip": "127.0.0.1" }, "tags": [ "nginx-access" ], "url": { "original": "/server-status", "path": "/server-status" }, "user_agent": { "device": { "name": "Other" }, "name": "curl", "original": "curl/7.64.0", "version": "7.64.0" }}ECS Field Reference
Please refer to the followingdocument for detailed information on ECS fields.
Exported fields
| Field | Description | Type | Unit |
|---|---|---|---|
| @timestamp | Event timestamp. | date | |
| cloud.image.id | Image ID for the cloud instance. | keyword | |
| data_stream.dataset | Data stream dataset. | constant_keyword | |
| data_stream.namespace | Data stream namespace. | constant_keyword | |
| data_stream.type | Data stream type. | constant_keyword | |
| event.dataset | Event dataset | constant_keyword | |
| event.module | Event module | constant_keyword | |
| host.containerized | If the host is a container. | boolean | |
| host.os.build | OS build information. | keyword | |
| host.os.codename | OS codename, if any. | keyword | |
| input.type | Input type | keyword | |
| log.offset | Log offset | long | |
| nginx.access.remote_ip_list | An array of remote IP addresses. It is a list because it is common to include, besides the client IP address, IP addresses from headers likeX-Forwarded-For. Real source IP is restored tosource.ip. | keyword | |
| nginx.access.response_time | Time to serve the request in milliseconds. | long | ms |
Error logs collects the Nginx error logs.
The Nginx error logs stream was tested with Nginx 1.19.5.
Example
{ "@timestamp": "2022-12-09T10:40:03.000Z", "agent": { "ephemeral_id": "34369a4a-4f24-4a39-9758-85fc2429d7e2", "id": "ef5e274d-4b53-45e6-943a-a5bcf1a6f523", "name": "docker-fleet-agent", "type": "filebeat", "version": "8.5.0" }, "data_stream": { "dataset": "nginx.error", "namespace": "ep", "type": "logs" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { "id": "ef5e274d-4b53-45e6-943a-a5bcf1a6f523", "snapshot": false, "version": "8.5.0" }, "event": { "agent_id_status": "verified", "category": [ "web" ], "created": "2022-12-09T10:40:18.973Z", "dataset": "nginx.error", "ingested": "2022-12-09T10:40:24Z", "kind": "event", "timezone": "+00:00", "type": [ "error" ] }, "host": { "architecture": "x86_64", "containerized": false, "hostname": "docker-fleet-agent", "id": "66392b0697b84641af8006d87aeb89f1", "ip": [ "172.18.0.7" ], "mac": [ "02-42-AC-12-00-07" ], "name": "docker-fleet-agent", "os": { "codename": "focal", "family": "debian", "kernel": "5.15.49-linuxkit", "name": "Ubuntu", "platform": "ubuntu", "type": "linux", "version": "20.04.5 LTS (Focal Fossa)" } }, "input": { "type": "log" }, "log": { "file": { "path": "/tmp/service_logs/error.log" }, "level": "warn", "offset": 0 }, "message": "conflicting server name \"localhost\" on 0.0.0.0:80, ignored", "nginx": { "error": {} }, "process": { "pid": 1, "thread": { "id": 1 } }, "tags": [ "nginx-error" ]}ECS Field Reference
Please refer to the followingdocument for detailed information on ECS fields.
Exported fields
| Field | Description | Type |
|---|---|---|
| @timestamp | Event timestamp. | date |
| cloud.image.id | Image ID for the cloud instance. | keyword |
| data_stream.dataset | Data stream dataset. | constant_keyword |
| data_stream.namespace | Data stream namespace. | constant_keyword |
| data_stream.type | Data stream type. | constant_keyword |
| event.dataset | Event dataset | constant_keyword |
| event.module | Event module | constant_keyword |
| host.containerized | If the host is a container. | boolean |
| host.os.build | OS build information. | keyword |
| host.os.codename | OS codename, if any. | keyword |
| input.type | Input type | keyword |
| log.offset | Log offset | long |
| nginx.error.connection_id | Connection identifier. | long |
The Nginxstubstatus stream collects data from the Nginxngx_http_stub_status module. It scrapes the server statusdata from the web page generated byngx_http_stub_status. Please verify that your Nginx distribution comes with the mentionedmodule and it's enabled in the Nginx configuration file:
location /nginx_status { stub_status; allow 127.0.0.1; deny all;}- only allow requests from localhost
- deny all other hosts
Replace127.0.0.1 with your server’s IP address and make sure that this page accessible to only you.
The Nginxstubstatus stream was tested with Nginx 1.19.5 and is expected to work with all versions >= 1.19.
Example
{ "@timestamp": "2024-02-08T08:12:10.668Z", "agent": { "ephemeral_id": "c3d516ba-b659-4190-a29d-d28200d74d48", "id": "2ea50bee-9250-43d1-8d70-949f242aa275", "name": "docker-fleet-agent", "type": "metricbeat", "version": "8.10.1" }, "data_stream": { "dataset": "nginx.stubstatus", "namespace": "ep", "type": "metrics" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { "id": "2ea50bee-9250-43d1-8d70-949f242aa275", "snapshot": false, "version": "8.10.1" }, "event": { "agent_id_status": "verified", "dataset": "nginx.stubstatus", "duration": 1156104, "ingested": "2024-02-08T08:12:13Z", "module": "nginx" }, "host": { "architecture": "x86_64", "containerized": false, "hostname": "docker-fleet-agent", "id": "28da52b32df94b50aff67dfb8f1be3d6", "ip": [ "172.24.0.7" ], "mac": [ "02-42-AC-18-00-07" ], "name": "docker-fleet-agent", "os": { "codename": "focal", "family": "debian", "kernel": "5.15.0-89-generic", "name": "Ubuntu", "platform": "ubuntu", "type": "linux", "version": "20.04.6 LTS (Focal Fossa)" } }, "metricset": { "name": "stubstatus", "period": 10000 }, "nginx": { "stubstatus": { "accepts": 33, "active": 1, "current": 33, "dropped": 0, "handled": 33, "hostname": "elastic-package-service-nginx-1:80", "reading": 0, "requests": 33, "waiting": 0, "writing": 1 } }, "service": { "address": "http://elastic-package-service-nginx-1:80/server-status", "type": "nginx" }, "tags": [ "nginx-stubstatus" ]}ECS Field Reference
Please refer to the followingdocument for detailed information on ECS fields.
Exported fields
| Field | Description | Type | Metric Type |
|---|---|---|---|
| @timestamp | Event timestamp. | date | |
| agent.id | keyword | ||
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | |
| cloud.availability_zone | Availability zone in which this host is running. | keyword | |
| cloud.image.id | Image ID for the cloud instance. | keyword | |
| cloud.instance.id | Instance ID of the host machine. | keyword | |
| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | |
| cloud.region | Region in which this host is running. | keyword | |
| container.id | Unique container id. | keyword | |
| data_stream.dataset | Data stream dataset. | constant_keyword | |
| data_stream.namespace | Data stream namespace. | constant_keyword | |
| data_stream.type | Data stream type. | constant_keyword | |
| event.dataset | Event dataset | constant_keyword | |
| event.module | Event module | constant_keyword | |
| host.containerized | If the host is a container. | boolean | |
| host.name | Name of the host. It can contain whathostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | |
| host.os.build | OS build information. | keyword | |
| host.os.codename | OS codename, if any. | keyword | |
| nginx.stubstatus.accepts | The total number of accepted client connections. | long | counter |
| nginx.stubstatus.active | The current number of active client connections including Waiting connections. | long | gauge |
| nginx.stubstatus.current | The current number of client requests. | long | gauge |
| nginx.stubstatus.dropped | The total number of dropped client connections. | long | counter |
| nginx.stubstatus.handled | The total number of handled client connections. | long | counter |
| nginx.stubstatus.hostname | Nginx hostname. | keyword | |
| nginx.stubstatus.reading | The current number of connections where Nginx is reading the request header. | long | gauge |
| nginx.stubstatus.requests | The total number of client requests. | long | counter |
| nginx.stubstatus.waiting | The current number of idle client connections waiting for a request. | long | gauge |
| nginx.stubstatus.writing | The current number of connections where Nginx is writing the response back to the client. | long | gauge |
| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword |
These anomaly detection jobs are available in the Machine Learning app in Kibanawhen you have data that matches the query specified in themanifest.
Find unusual activity in HTTP access logs.
| Job | Description |
|---|---|
visitor_rate_nginx | HTTP Access Logs: Detect unusual visitor rates |
status_code_rate_nginx | HTTP Access Logs: Detect unusual status code rates |
source_ip_url_count_nginx | HTTP Access Logs: Detect unusual source IPs - high distinct count of URLs |
source_ip_request_rate_nginx | HTTP Access Logs: Detect unusual source IPs - high request rates |
low_request_rate_nginx | HTTP Access Logs: Detect low request rates |
This integration includes one or more Kibana dashboards that visualizes the data collected by the integration. The screenshots below illustrate how the ingested data is displayed.
Changelog
| Version | Details | Minimum Kibana version |
|---|---|---|
| 3.0.0 | Breaking change (View pull request) Remove third-party pipeline for previously removed 'third-party REST API' input. | 9.0.0 8.13.0 |
| 2.3.2 | Bug fix (View pull request) Remove unused agent files. | 9.0.0 8.13.0 |
| 2.3.1 | Bug fix (View pull request) Fix Dashboard Links Panel by setting useCurrentFilters tofalse | 9.0.0 8.13.0 |
| 2.3.0 | Enhancement (View pull request) Use links panel in Dashboards. | 9.0.0 8.13.0 |
| 2.2.1 | Bug fix (View pull request) Update tests by using debian archive. | 9.0.0 8.13.0 |
| 2.2.0 | Enhancement (View pull request) Added optional parsing for request duration in access-log. | 9.0.0 8.13.0 |
| 2.1.0 | Enhancement (View pull request) Allow @custom pipeline access to event.original without setting preserve_original_event. | 9.0.0 8.13.0 |
| 2.0.0 | Enhancement (View pull request) Remove deprecated httpjson input. | 9.0.0 8.13.0 |
| 1.25.1 | Bug fix (View pull request) Added description to ssl nodes including links to documentation. | 9.0.0 8.13.0 |
| 1.25.0 | Enhancement (View pull request) Add support for Kibana 9.0.0. | 9.0.0 8.13.0 |
| 1.24.1 | Bug fix (View pull request) Update links to getting started docs | 8.13.0 |
| 1.24.0 | Enhancement (View pull request) Deprecate third-party REST API import option. | 8.13.0 |
| 1.23.0 | Enhancement (View pull request) Add processor support for stubstatus data stream. | 8.13.0 |
| 1.22.0 | Enhancement (View pull request) ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template. | 8.13.0 |
| 1.21.0 | Enhancement (View pull request) Add global filter on data_stream.dataset to improve performance. | 8.12.0 |
| 1.20.0 | Enhancement (View pull request) Enable secrets for sensitive fields. For more details, referhttps://www.elastic.co/guide/en/fleet/current/agent-policy.html#agent-policy-secret-values | 8.12.0 |
| 1.19.1 | Bug fix (View pull request) Disable secrets for older stack versions due to errors. | 8.8.0 |
| 1.19.0 | Enhancement (View pull request) Add support for tags in stub status metrics | 8.8.0 |
| 1.18.0 | Enhancement (View pull request) Enable 'secret' for the sensitive fields, supported from 8.12. | 8.8.0 |
| 1.17.0 | Enhancement (View pull request) Limit request tracer log count to five. | 8.8.0 |
| 1.16.0 | Enhancement (View pull request) Update the package format_version to 3.0.0. | 8.8.0 |
| 1.15.1 | Bug fix (View pull request) Add null check and ignore_missing check to the rename processor | 8.8.0 |
| 1.15.0 | Enhancement (View pull request) Enable time series data streams for the metrics datasets. This improves storage usage and query performance. For more details, seehttps://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html | 8.8.0 |
| 1.14.1-beta | Bug fix (View pull request) Add beta flag | — |
| 1.14.0 | Enhancement (View pull request) Rename ownership from obs-service-integrations to obs-infraobs-integrations | 8.8.0 |
| 1.13.0-beta | Enhancement (View pull request) Add a new flag to enable request tracing | — |
| 1.12.0-beta | Enhancement (View pull request) Enable TSDB on Nginx for beta testing. | — |
| 1.11.1 | Bug fix (View pull request) Modifed the dimension field mapping to support public cloud deployment. | 8.5.0 |
| 1.11.0 | Enhancement (View pull request) Migrate "Access and error logs" dashboard visualizations to lens. | 8.5.0 |
| 1.10.0 | Enhancement (View pull request) Migrate "Logs Overview" dashboard visualizations to lens. | 8.5.0 |
| 1.9.0 | Enhancement (View pull request) Migrate "Metrics Overview" dashboard visualizations to lens. | 8.5.0 |
| 1.8.0 | Enhancement (View pull request) Added changes to support TSDB Enablement. | 8.0.0 |
| 1.7.1 | Enhancement (View pull request) Added categories and/or subcategories. | 8.0.0 |
| 1.7.0 | Enhancement (View pull request) Update ECS version to 8.5.1. | 8.0.0 |
| 1.6.0 | Enhancement (View pull request) Added infrastructure category. | 8.0.0 |
| 1.5.1 | Enhancement (View pull request) Remove unused visualizations | 8.0.0 |
| 1.5.0 | Enhancement (View pull request) Add ignore_older to log data streams | 8.0.0 |
| 1.4.1 | Enhancement (View pull request) Update documentation with additional context for new users. | 8.0.0 |
| 1.4.0 | Enhancement (View pull request) Migration of tile map to map object in dashboards | — |
| 1.3.2 | Enhancement (View pull request) Add documentation for multi-fields | — |
| 1.3.1 | Enhancement (View pull request) Add ML modules to readme | 8.0.0 7.14.0 |
| 1.3.0 | Enhancement (View pull request) Update to ECS 8.0 | — |
| 1.2.3 | Bug fix (View pull request) Regenerate test files using the new GeoIP database | — |
| 1.2.2 | Bug fix (View pull request) Change test public IPs to the supported subset | — |
| 1.2.1 | Bug fix (View pull request) Fix ML module manifest query to ignore frozen and cold tiers | 8.0.0 7.14.0 |
| 1.2.0 | Enhancement (View pull request) Release nginx package for v8.0.0 | 8.0.0 7.14.0 |
| 1.1.2 | Enhancement (View pull request) Uniform with guidelines | — |
| 1.1.1 | Bug fix (View pull request) Fix logic that checks for the 'forwarded' tag | — |
| 1.1.0 | Enhancement (View pull request) Update to ECS 1.12.0 | 7.14.0 |
| 1.0.0 | Enhancement (View pull request) Release Nginx as GA | — |
| 0.8.2 | Enhancement (View pull request) Convert to generated ECS fields | — |
| 0.8.1 | Enhancement (View pull request) update to ECS 1.11.0 | — |
| 0.8.0 | Enhancement (View pull request) Update integration description | — |
| 0.7.0 | Enhancement (View pull request) Set "event.module" and "event.dataset" | — |
| 0.6.2 | Enhancement (View pull request) Add support for Splunk authorization tokens | — |
| 0.6.1 | Bug fix (View pull request) Fix bug in Third Party REST API ingest pipeline | — |
| 0.6.0 | Enhancement (View pull request) update to ECS 1.10.0 and add event.original options | — |
| 0.5.0 | Enhancement (View pull request) Adds ML jobs for finding unusual activity in HTTP access logs | — |
| 0.4.1 | Enhancement (View pull request) update to ECS 1.9.0 | — |
| 0.3.11 | Enhancement (View pull request) Updating package owner | — |
| 0.3.10 | Enhancement (View pull request) Fix compatibility with Kibana | — |
| 0.1.0 | Enhancement (View pull request) initial release | — |