ZooKeeper Integration
| Version | 1.14.0 (View all) |
| Subscription level What's this? | Basic |
| Developed by What's this? | Elastic |
| Ingestion method(s) | Database |
| Minimum Kibana version(s) | 9.0.0 8.13.0 |
This integration periodically fetches metrics from theZooKeeper service.
The ZooKeeper integration is tested with ZooKeeper 3.4.8 and is expected to work with all version >= 3.4.0. Versions prior to 3.4 do not support the mntr command.
Theconnection dataset fetches the data returned by thecons admin keyword.
Example
{ "@timestamp": "2020-07-06T16:12:07.612Z", "agent": { "ephemeral_id": "4d221f8f-7147-4855-8ea3-b4d2a5b80ae0", "id": "2ff8a09c-c7f0-42f2-9fe1-65f7fd460651", "name": "zookeeper-01", "type": "metricbeat", "version": "8.0.0" }, "client": { "ip": "172.28.0.1", "port": 44338 }, "ecs": { "version": "8.11.0" }, "event": { "dataset": "zookeeper.connection", "duration": 3093417, "module": "zookeeper" }, "host": { "name": "zookeeper-01" }, "metricset": { "name": "connection", "period": 10000 }, "service": { "address": "localhost:2181", "type": "zookeeper" }, "zookeeper": { "connection": { "interest_ops": 0, "queued": 0, "received": 1, "sent": 0 } }}ECS Field Reference
Please refer to the followingdocument for detailed information on ECS fields.
Exported fields
| Field | Description | Type | Metric Type |
|---|---|---|---|
| @timestamp | Event timestamp. | date | |
| agent.id | keyword | ||
| client.ip | IP address of the client (IPv4 or IPv6). | ip | |
| client.port | Port of the client. | long | |
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | |
| cloud.availability_zone | Availability zone in which this host is running. | keyword | |
| cloud.image.id | Image ID for the cloud instance. | keyword | |
| cloud.instance.id | Instance ID of the host machine. | keyword | |
| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | |
| cloud.region | Region in which this host is running. | keyword | |
| container.id | Unique container id. | keyword | |
| data_stream.dataset | Data stream dataset. | constant_keyword | |
| data_stream.namespace | Data stream namespace. | constant_keyword | |
| data_stream.type | Data stream type. | constant_keyword | |
| event.dataset | Event dataset | constant_keyword | |
| event.module | Event module | constant_keyword | |
| host.containerized | If the host is a container. | boolean | |
| host.name | Name of the host. It can contain whathostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | |
| host.os.build | OS build information. | keyword | |
| host.os.codename | OS codename, if any. | keyword | |
| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | |
| zookeeper.connection.interest_ops | Interest ops | long | |
| zookeeper.connection.queued | Queued connections | long | gauge |
| zookeeper.connection.received | Received connections | long | counter |
| zookeeper.connection.sent | Connections sent | long | counter |
Themntr Metricset fetches the data returned by themntr admin keyword.
Example
{ "@timestamp": "2020-07-06T16:12:08.494Z", "agent": { "ephemeral_id": "4d221f8f-7147-4855-8ea3-b4d2a5b80ae0", "id": "2ff8a09c-c7f0-42f2-9fe1-65f7fd460651", "name": "zookeeper-01", "type": "metricbeat", "version": "8.0.0" }, "ecs": { "version": "8.11.0" }, "event": { "dataset": "zookeeper.mntr", "duration": 15795652, "module": "zookeeper" }, "host": { "name": "zookeeper-01" }, "metricset": { "name": "mntr", "period": 10000 }, "service": { "address": "localhost:2181", "type": "zookeeper", "version": "3.5.5-390fe37ea45dee01bf87dc1c042b5e3dcce88653, built on 05/03/2019 12:07 GMT" }, "zookeeper": { "mntr": { "approximate_data_size": 44, "ephemerals_count": 0, "latency": { "avg": 0, "max": 0, "min": 0 }, "max_file_descriptor_count": 1048576, "num_alive_connections": 1, "open_file_descriptor_count": 49, "outstanding_requests": 0, "packets": { "received": 152, "sent": 151 }, "server_state": "standalone", "watch_count": 0, "znode_count": 5 } }}ECS Field Reference
Please refer to the followingdocument for detailed information on ECS fields.
Exported fields
| Field | Description | Type | Metric Type |
|---|---|---|---|
| @timestamp | Event timestamp. | date | |
| agent.id | keyword | ||
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | |
| cloud.availability_zone | Availability zone in which this host is running. | keyword | |
| cloud.image.id | Image ID for the cloud instance. | keyword | |
| cloud.instance.id | Instance ID of the host machine. | keyword | |
| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | |
| cloud.region | Region in which this host is running. | keyword | |
| container.id | Unique container id. | keyword | |
| data_stream.dataset | Data stream dataset. | constant_keyword | |
| data_stream.namespace | Data stream namespace. | constant_keyword | |
| data_stream.type | Data stream type. | constant_keyword | |
| event.dataset | Event dataset | constant_keyword | |
| event.module | Event module | constant_keyword | |
| host.containerized | If the host is a container. | boolean | |
| host.name | Name of the host. It can contain whathostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | |
| host.os.build | OS build information. | keyword | |
| host.os.codename | OS codename, if any. | keyword | |
| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | |
| zookeeper.mntr.approximate_data_size | Approximate size of ZooKeeper data. | long | gauge |
| zookeeper.mntr.ephemerals_count | Number of ephemeral znodes. | long | gauge |
| zookeeper.mntr.followers | Number of followers seen by the current host. | long | gauge |
| zookeeper.mntr.latency.avg | Average latency between ensemble hosts in milliseconds. | long | gauge |
| zookeeper.mntr.latency.max | Maximum latency in milliseconds. | long | gauge |
| zookeeper.mntr.latency.min | Minimum latency in milliseconds. | long | gauge |
| zookeeper.mntr.max_file_descriptor_count | Maximum number of file descriptors allowed for the ZooKeeper process. | long | gauge |
| zookeeper.mntr.num_alive_connections | Number of connections to ZooKeeper that are currently alive. | long | gauge |
| zookeeper.mntr.open_file_descriptor_count | Number of file descriptors open by the ZooKeeper process. | long | gauge |
| zookeeper.mntr.outstanding_requests | Number of outstanding requests that need to be processed by the cluster. | long | gauge |
| zookeeper.mntr.packets.received | Number of ZooKeeper network packets received. | long | gauge |
| zookeeper.mntr.packets.sent | Number of ZooKeeper network packets sent. | long | gauge |
| zookeeper.mntr.pending_syncs | Number of pending syncs to carry out to ZooKeeper ensemble followers. | long | gauge |
| zookeeper.mntr.server_state | Role in the ZooKeeper ensemble. | keyword | |
| zookeeper.mntr.synced_followers | Number of synced followers reported when a node server_state is leader. | long | gauge |
| zookeeper.mntr.watch_count | Number of watches currently set on the local ZooKeeper process. | long | gauge |
| zookeeper.mntr.znode_count | Number of znodes reported by the local ZooKeeper process. | long | gauge |
Theserver Metricset fetches the data returned by thesrvr admin keyword.
Example
{ "@timestamp": "2020-07-06T16:12:12.409Z", "agent": { "ephemeral_id": "4d221f8f-7147-4855-8ea3-b4d2a5b80ae0", "id": "2ff8a09c-c7f0-42f2-9fe1-65f7fd460651", "name": "zookeeper-01", "type": "metricbeat", "version": "8.0.0" }, "ecs": { "version": "8.11.0" }, "event": { "dataset": "zookeeper.server", "duration": 3001938, "module": "zookeeper" }, "host": { "name": "zookeeper-01" }, "metricset": { "name": "server", "period": 10000 }, "service": { "address": "localhost:2181", "type": "zookeeper", "version": "3.5.5-390fe37ea45dee01bf87dc1c042b5e3dcce88653" }, "zookeeper": { "server": { "connections": 1, "count": 0, "epoch": 0, "latency": { "avg": 0, "max": 0, "min": 0 }, "mode": "standalone", "node_count": 5, "outstanding": 0, "received": 156, "sent": 155, "version_date": "2019-05-03T12:07:00Z", "zxid": "0x0" } }}ECS Field Reference
Please refer to the followingdocument for detailed information on ECS fields.
Exported fields
| Field | Description | Type | Metric Type |
|---|---|---|---|
| @timestamp | Event timestamp. | date | |
| agent.id | keyword | ||
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | |
| cloud.availability_zone | Availability zone in which this host is running. | keyword | |
| cloud.image.id | Image ID for the cloud instance. | keyword | |
| cloud.instance.id | Instance ID of the host machine. | keyword | |
| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | |
| cloud.region | Region in which this host is running. | keyword | |
| container.id | Unique container id. | keyword | |
| data_stream.dataset | Data stream dataset. | constant_keyword | |
| data_stream.namespace | Data stream namespace. | constant_keyword | |
| data_stream.type | Data stream type. | constant_keyword | |
| event.dataset | Event dataset | constant_keyword | |
| event.module | Event module | constant_keyword | |
| host.containerized | If the host is a container. | boolean | |
| host.name | Name of the host. It can contain whathostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | |
| host.os.build | OS build information. | keyword | |
| host.os.codename | OS codename, if any. | keyword | |
| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | |
| zookeeper.server.connections | Number of clients currently connected to the server | long | gauge |
| zookeeper.server.count | Total transactions of the leader in epoch | long | counter |
| zookeeper.server.epoch | Epoch value of the Zookeeper transaction ID. An epoch signifies the period in which a server is a leader | long | |
| zookeeper.server.latency.avg | Average amount of time taken for the server to respond to a client request | long | gauge |
| zookeeper.server.latency.max | Maximum amount of time taken for the server to respond to a client request | long | gauge |
| zookeeper.server.latency.min | Minimum amount of time taken for the server to respond to a client request | long | gauge |
| zookeeper.server.mode | Mode of the server. In an ensemble, this may either be leader or follower. Otherwise, it is standalone | keyword | |
| zookeeper.server.node_count | Total number of nodes | long | gauge |
| zookeeper.server.outstanding | Number of requests queued at the server. This exceeds zero when the server receives more requests than it is able to process | long | gauge |
| zookeeper.server.received | Number of requests received by the server | long | counter |
| zookeeper.server.sent | Number of requests sent by the server | long | counter |
| zookeeper.server.version_date | Date of the Zookeeper release currently in use | date | |
| zookeeper.server.zxid | Unique value of the Zookeeper transaction ID. The zxid consists of an epoch and a counter. It is established by the leader and is used to determine the temporal ordering of changes | keyword |
This integration includes one or more Kibana dashboards that visualizes the data collected by the integration. The screenshots below illustrate how the ingested data is displayed.
Changelog
| Version | Details | Minimum Kibana version |
|---|---|---|
| 1.14.0 | Enhancement (View pull request) Add support for Kibana 9.0.0. | 9.0.0 8.13.0 |
| 1.13.1 | Bug fix (View pull request) Remove hostname field from mntr data stream. | 8.13.0 |
| 1.13.0 | Enhancement (View pull request) Add processor support for connection, mntr and server data streams. | 8.13.0 |
| 1.12.0 | Enhancement (View pull request) ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template. | 8.13.0 |
| 1.11.0 | Enhancement (View pull request) Add global filter on data_stream.dataset to improve performance. | 8.8.0 |
| 1.10.0 | Enhancement (View pull request) Update the package format_version to 3.0.0. | 8.8.0 |
| 1.9.0 | Enhancement (View pull request) Enable time series data streams for the metrics datasets. This improves storage usage and query performance. For more details, seehttps://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html | 8.8.0 |
| 1.8.5 | Enhancement (View pull request) Added dimension mapping for server datastream. | 8.2.0 |
| 1.8.4 | Enhancement (View pull request) Added dimension mapping for mntr datastream. | 8.2.0 |
| 1.8.3 | Enhancement (View pull request) Added dimension mapping for connection datastream. | 8.2.0 |
| 1.8.2 | Enhancement (View pull request) Added metric_type mapping for server datastream. | 8.2.0 |
| 1.8.1 | Enhancement (View pull request) Added metric_type mapping for mntr datastream. | 8.2.0 |
| 1.8.0 | Enhancement (View pull request) Added metric_type mapping for connection datastream | 8.2.0 |
| 1.7.0 | Enhancement (View pull request) Rename ownership from obs-service-integrations to obs-infraobs-integrations | 8.2.0 |
| 1.6.0 | Enhancement (View pull request) Migrate visualizations to lens. | 8.2.0 |
| 1.5.1 | Enhancement (View pull request) Added categories and/or subcategories. | 8.0.0 7.14.0 |
| 1.5.0 | Enhancement (View pull request) Update ECS version to 8.5.1 | 8.0.0 7.14.0 |
| 1.4.0 | Enhancement (View pull request) Added infrastructure category. | 8.0.0 7.14.0 |
| 1.3.1 | Enhancement (View pull request) Add documentation for multi-fields | 8.0.0 7.14.0 |
| 1.3.0 | Enhancement (View pull request) Update to ECS 8.0 | — |
| 1.2.0 | Enhancement (View pull request) Support Kibana 8.0 | 8.0.0 7.14.0 |
| 1.1.1 | Enhancement (View pull request) Uniform with guidelines | — |
| 1.1.0 | Enhancement (View pull request) Update to ECS 1.12.0 | 7.14.0 |
| 1.0.0 | Enhancement (View pull request) Release Kafka as GA | — |
| 0.4.0 | Enhancement (View pull request) Update integration description | — |
| 0.3.1 | Bug fix (View pull request) Add missing data_stream fields | — |
| 0.3.0 | Enhancement (View pull request) Set "event.module" and "event.dataset" | — |
| 0.2.7 | Enhancement (View pull request) Updating package owner | — |
| 0.2.6 | Bug fix (View pull request) Correct sample event file. | — |
| 0.1.0 | Enhancement (View pull request) initial release | — |