Movatterモバイル変換

[0]ホーム
URL:

Loading
  1. Elastic Docs/
  2. Deploy and manage/
  3. Deploy/
  4. Elastic Cloud on Kubernetes/
  5. Orchestrate other Elastic applications/
  6. Logstash

Configuration for Logstash on Elastic Cloud on Kubernetes

You can upgrade the Logstash version or change settings by editing the YAML specification. ECK applies the changes by performing a rolling restart of Logstash Pods.

Define the Logstash configuration (the ECK equivalent tologstash.yml) in thespec.config section:

apiVersion: logstash.k8s.elastic.co/v1alpha1kind: Logstashmetadata:  name: quickstartspec:  version: 9.3.0  count: 1  elasticsearchRefs:  - name: quickstart    clusterName: qs  config:    pipeline.workers: 4    log.level: debug
  1. Customize Logstash configuration usinglogstash.yml settings here

Alternatively, you can provide the configuration through a Secret specified in thespec.configRef section. The Secret must have alogstash.yml entry with your settings:

apiVersion: logstash.k8s.elastic.co/v1alpha1kind: Logstashmetadata:  name: quickstartspec:  version: 9.3.0  count: 1  elasticsearchRefs:  - name: quickstart    clusterName: qs  configRef:    secretName: quickstart-config---apiVersion: v1kind: Secretmetadata:  name: quickstart-configstringData:  logstash.yml: |-    pipeline.workers: 4    log.level: debug

Define Logstash pipelines in thespec.pipelines section (the ECK equivalent topipelines.yml):

apiVersion: logstash.k8s.elastic.co/v1alpha1kind: Logstashmetadata:  name: quickstartspec:  version: 9.3.0  count: 1  elasticsearchRefs:    - clusterName: qs      name: quickstart  pipelines:    - pipeline.id: main      config.string: |        input {          beats {            port => 5044          }        }        output {          elasticsearch {            hosts => [ "${QS_ES_HOSTS}" ]            user => "${QS_ES_USER}"            password => "${QS_ES_PASSWORD}"            ssl_certificate_authorities => "${QS_ES_SSL_CERTIFICATE_AUTHORITY}"          }        }

Alternatively, you can provide the pipelines configuration through a Secret specified in thespec.pipelinesRef field. The Secret must have apipelines.yml entry with your configuration:

apiVersion: logstash.k8s.elastic.co/v1alpha1kind: Logstashmetadata:  name: quickstartspec:  version: 9.3.0  count: 1  elasticsearchRefs:    - clusterName: qs      name: quickstart  pipelinesRef:    secretName: quickstart-pipeline---apiVersion: v1kind: Secretmetadata:  name: quickstart-pipelinestringData:  pipelines.yml: |-    - pipeline.id: main      config.string: |        input {          beats {            port => 5044          }        }        output {          elasticsearch {            hosts => [ "${QS_ES_HOSTS}" ]            user => "${QS_ES_USER}"            password => "${QS_ES_PASSWORD}"            ssl_certificate_authorities => "${QS_ES_SSL_CERTIFICATE_AUTHORITY}"          }        }

Logstash on ECK supports all options present inpipelines.yml, including settings to update the number of workers, and the size of the batch that the pipeline will process. This also includes usingpath.config to point to volumes mounted on the Logstash container:

apiVersion: logstash.k8s.elastic.co/v1alpha1kind: Logstashmetadata:  name: quickstartspec:  version: 9.3.0  count: 1  elasticsearchRefs:    - clusterName: qs      name: quickstart  pipelines:    - pipeline.id: main      config.string: |        input {          beats {            port => 5044          }        }        output {          elasticsearch {            hosts => [ "${QS_ES_HOSTS}" ]            user => "${QS_ES_USER}"            password => "${QS_ES_PASSWORD}"            ssl_certificate_authorities => "${QS_ES_SSL_CERTIFICATE_AUTHORITY}"          }        }
Note

Logstash persistent queues (PQs) and dead letter queues (DLQs) are not currently managed by the Logstash operator, and using them will require you to create and manage your own Volumes and VolumeMounts

Added in 2.9.0

This was added in 2.9.0.

Warning

Volume support for Logstash is a breaking change to earlier versions of ECK and requires you to recreate your Logstash resources.

A PersistentVolume calledlogstash-data is created by default. It maps to/usr/share/logstash/data for persistent storage, which is typically used for storage from plugins.

By default, thelogstash-data volume claim is a1.5Gi volume, using the standard StorageClass of your Kubernetes cluster. You can override the default by adding aspec.volumeClaimTemplate section namedlogstash-data.

For production workloads, you should define your own volume claim template with the desired storage capacity and (optionally) the Kubernetesstorage class to associate with the persistent volume. To override this volume claim fordata usages, the name of this volume claim must belogstash-data.

This example updates the default data template to increase the storage to2Gi for the Logstash data folder:

apiVersion: logstash.k8s.elastic.co/v1alpha1kind: Logstashmetadata:  name: logstashspec:  # some configuration attributes omitted for brevity here  volumeClaimTemplates:    - metadata:        name: logstash-data      spec:        accessModes:          - ReadWriteOnce        resources:          requests:            storage: 2Gi
  1. Do not change this name unless you set up a volume mount for the data path.

The default volume size will likely be insufficient for production workloads, especially when you are using:

  • the persistent queue (PQ) feature
  • dead letter queues (DLQ), or
  • Logstash plugins that make heavy use of temporary storage.

Increase the storage capacity, or consider creating separate volumes for these use cases.

You can add separate storage by including an additionalspec.volumeClaimTemplate along with a correspondingspec.podTemplate.spec.containers.volumeMount for each requested volume.

This example shows how to setup separate storage for a PQ:

apiVersion: logstash.k8s.elastic.co/v1alpha1kind: Logstashmetadata:  name: logstashspec:  # some configuration attributes omitted for brevity here  volumeClaimTemplates:    - metadata:        name: pq      spec:        accessModes:        - ReadWriteOnce        resources:          requests:            storage: 10Gi  podTemplate:    spec:      containers:      - name: logstash        volumeMounts:        - mountPath: /usr/share/logstash/pq          name: pq          readOnly: false  config:    log.level: info    queue.type: persisted    path.queue: /usr/share/logstash/pq
  1. Thename values in thevolumeMount for the container in thepodTemplate section and the name of thevolumeClaimTemplate must match.
  2. Set thepath.queue setting in the configuration to match themountPath in thevolumeMount.

This example shows how to configure Logstash with a Dead Letter Queue setup on the main pipeline, and a separate pipeline to read items from the DLQ.

apiVersion: logstash.k8s.elastic.co/v1alpha1kind: Logstashmetadata:  name: logstashspec:   # some configuration attributes omitted for brevity here   podTemplate:    spec:      containers:      - name: logstash        volumeMounts:        - mountPath: /usr/share/logstash/dlq          name: dlq          readOnly: false  volumeClaimTemplates:    - metadata:        name: dlq      spec:        accessModes:        - ReadWriteOnce        resources:          requests:            storage: 10Gi  pipelines:    - pipeline.id: beats      dead_letter_queue.enable: true      path.dead_letter_queue: /usr/share/logstash/dlq      config.string: |        input {          beats {            port => 5044          }        }        output {          elasticsearch {            hosts => [ "${ECK_ES_HOSTS}" ]            user => "${ECK_ES_USER}"            password => "${ECK_ES_PASSWORD}"            ssl_certificate_authorities => "${ECK_ES_SSL_CERTIFICATE_AUTHORITY}"          }        }    - pipeline.id: dlq_read      dead_letter_queue.enable: false      config.string: |        input {          dead_letter_queue {            path => "/usr/share/logstash/dlq"            commit_offsets => true            pipeline_id => "beats"            clean_consumed => true          }        }        filter {          mutate {            remove_field => "[geoip][location]"          }        }        output {          elasticsearch {            hosts => [ "${ECK_ES_HOSTS}" ]            user => "${ECK_ES_USER}"            password => "${ECK_ES_PASSWORD}"            ssl_certificate_authorities => "${ECK_ES_SSL_CERTIFICATE_AUTHORITY}"          }        }
  1. Thename values in thevolumeMount for the container in thepodTemplate section and the name of thevolumeClaimTemplate must match.
  2. Set thepath.dead_letter_queue setting in the pipeline config to match themountPath in thevolumeMount for pipelines that are writing to the Dead Letter Queue, and set thepath setting of thedead_letter_queue plugin for the pipeline that will read from the Dead Letter Queue.

If the storage class allowsvolume expansion, you can increase the storage requests size inspec.volumeClaimTemplates. ECK updates the existing PersistentVolumeClaims accordingly, and recreates the StatefulSet automatically.

If the volume driver supportsExpandInUsePersistentVolumes, the filesystem is resized online. In this case, you do not need to restart the Logstash process or re-create the Pods.

If the volume driver does not supportExpandInUsePersistentVolumes, you must manually delete Pods after the resize so that they can be recreated automatically with the expanded filesystem.

Any other changes in the volumeClaimTemplates—such as changing the storage class or decreasing the volume size—are not allowed. To make changes such as these, you must fully delete the Logstash resource, delete and recreate or resize the volume, and create a new Logstash resource.

Before you delete a persistent queue (PQ) volume, ensure that the queue is empty. We recommend settingqueue.drain: true on the Logstash Pods to ensure that the queue is drained when Pods are shutdown. Note that you should also increase theterminationGracePeriodSeconds to a large enough value to allow the queue to drain.

This example shows how to configure a Logstash resource to drain the queue and increase the termination grace period.

apiVersion: logstash.k8s.elastic.co/v1alpha1kind: Logstashmetadata:  name: logstashspec:  # some configuration attributes omitted for brevity here  config:    queue.drain: true  podTemplate:    spec:      terminationGracePeriodSeconds: 604800
Note

AKubernetes known issue: Kubernetes may not honorterminationGracePeriodSeconds settings greater than 600. A queue of a terminated Pod may not be fully drained, even whenqueue.drain: true is set and a highterminationGracePeriodSeconds is configured.

Note

In this technical preview, there is currently no way to drain a dead letter queue (DLQ) automatically before Logstash shuts down. To manually drain the queue, first stop sending data to it, by either disabling the DLQ feature, or disabling any pipelines that send to a DLQ. Then wait for events to stop flowing through any pipelines reading from the input.

If you are not concerned about data loss, you can use anemptyDir volume for Logstash data.

Warning

The use ofemptyDir in a production environment may cause permanent data loss. Do not use with persistent queues (PQs), dead letter queues (DLQs), or with any plugin that requires persistent storage to keep track of state between restarts of Logstash.

Plugins that require persistent storage include any plugin that stores state locally. These plugins typically have a configuration parameter that includes the namepath ordirectory, not including paths to static content, such as certificates or keystores. Examples include thesincedb_path setting for thefile,dead_letter_queue ands3 inputs, thelast_run_metadata_path for theJDBC input,aggregate_maps_path for theaggregate filter, andtemporary_directory for thes3 output, used to aggregate content before uploading to s3.

spec:  count: 5  podTemplate:    spec:      volumes:      - name: logstash-data        emptyDir: {}

Thespec.elasticsearchRefs section provides a mechanism to help configure Logstash to establish a secured connection to one or more ECK managed Elasticsearch clusters. By default, eachelasticsearchRef will target all nodes in its referenced Elasticsearch cluster. If you want to direct traffic to specific nodes of your Elasticsearch cluster, refer toTraffic Splitting for more information and examples.

When you useelasticsearchRefs in a Logstash pipeline, the Logstash operator creates the necessary resources from the associated Elasticsearch cluster, and provides environment variables to allow these resources to be accessed from the pipeline configuration. Environment variables are replaced at runtime with the appropriate values. The environment variables have a fixed naming convention:

  • NORMALIZED_CLUSTERNAME_ES_HOSTS
  • NORMALIZED_CLUSTERNAME_ES_USER
  • NORMALIZED_CLUSTERNAME_ES_PASSWORD
  • NORMALIZED_CLUSTERNAME_ES_SSL_CERTIFICATE_AUTHORITY

where NORMALIZED_CLUSTERNAME is the value taken from theclusterName field of theelasticsearchRef property, capitalized, with- transformed to_. That is,prod-es would becomePROD_ES.

Note

  • TheclusterName value should be unique across all referenced Elasticsearch instances in the same Logstash spec.

  • The Logstash ECK operator creates a user calledeck_logstash_user_role when anelasticsearchRef is specified. This user has the following permissions:

      "cluster": ["monitor", "manage_ilm", "read_ilm", "manage_logstash_pipelines", "manage_index_templates", "cluster:admin/ingest/pipeline/get",]  "indices": [    {      "names": [ "logstash", "logstash-*", "ecs-logstash", "ecs-logstash-*", "logs-*", "metrics-*", "synthetics-*", "traces-*" ],      "privileges": ["manage", "write", "create_index", "read", "view_index_metadata"]    }]

    You canupdate user permissions to include more indices if the Elasticsearch plugin is expected to use indices other than the default. Check outLogstash configuration with a custom index sample configuration that creates a user that writes to a custom index.

This example demonstrates how to create a Logstash deployment that connects to different Elasticsearch instances, one of which is in a separate namespace:

apiVersion: logstash.k8s.elastic.co/v1alpha1kind: Logstashmetadata:  name: quickstartspec:  version: 9.3.0  count: 1  elasticsearchRefs:    - clusterName: prod-es      name: prod    - clusterName: qa-es      name: qa      namespace: qa  pipelines:    - pipeline.id: main      config.string: |        input {          beats {            port => 5044          }        }        output {          elasticsearch {            hosts => [ "${PROD_ES_ES_HOSTS}" ]            user => "${PROD_ES_ES_USER}"            password => "${PROD_ES_ES_PASSWORD}"            ssl_certificate_authorities => "${PROD_ES_ES_SSL_CERTIFICATE_AUTHORITY}"          }          elasticsearch {            hosts => [ "${QA_ES_ES_HOSTS}" ]            user => "${QA_ES_ES_USER}"            password => "${QA_ES_ES_PASSWORD}"            ssl_certificate_authorities => "${QA_ES_ES_SSL_CERTIFICATE_AUTHORITY}"          }        }
  1. Define Elasticsearch references in the CRD. This will create the appropriate Secrets to store certificate details and the rest of the connection information, and create environment variables to allow them to be referred to in Logstash pipeline configurations.
  2. This refers to an Elasticsearch cluster residing in the same namespace as the Logstash instances.
  3. This refers to an Elasticsearch cluster residing in a different namespace to the Logstash instances.
  4. Elasticsearch output definitions - use the environment variables created by the Logstash operator when specifying anElasticsearchRef. Note the use of "normalized" versions of theclusterName in the environment variables used to populate the relevant fields.

Logstash can connect to external Elasticsearch cluster that is not managed by ECK. You can reference a Secret instead of an Elasticsearch cluster in theelasticsearchRefs section through thesecretName attribute:

apiVersion: v1kind: Secretmetadata:  name: external-es-refstringData:  url: https://<example-url>.elastic-cloud.com:443  username: logstash_user  password: REDACTED  ca.crt: REDACTED---apiVersion: logstash.k8s.elastic.co/v1alpha1kind: Logstashmetadata:  name: quickstartspec:  version: 9.3.0  count: 1  elasticsearchRefs:    - clusterName: prod-es      secretName: external-es-ref  monitoring:    metrics:      elasticsearchRefs:      - secretName: external-es-ref    logs:      elasticsearchRefs:      - secretName: external-es-ref
  1. The URL to reach the Elasticsearch cluster.
  2. The username of the user to be authenticated to the Elasticsearch cluster.
  3. The password of the user to be authenticated to the Elasticsearch cluster.
  4. The CA certificate in PEM format to secure communication to the Elasticsearch cluster (optional).
  5. ThesecretName andname attributes are mutually exclusive, you have to choose one or the other.
Tip

Always specify the port in the URL when Logstash is connecting to an external Elasticsearch cluster.

By default, the Logstash operator creates a headless Service for the metrics endpoint to enable metric collection by the Metricbeat sidecar for Stack Monitoring:

kubectl get service quickstart-ls-api
NAME                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGEquickstart-ls-api   ClusterIP   None         <none>        9600/TCP   48s

Additional services can be added in thespec.services section of the resource:

services:  - name: beats    service:      spec:        ports:        - port: 5044          name: "winlogbeat"          protocol: TCP        - port: 5045          name: "filebeat"          protocol: TCP

You cancustomize the Logstash Pod using a Pod template, defined in thespec.podTemplate section of the configuration.

This example demonstrates how to create a Logstash deployment with increased heap size and resource limits.

apiVersion: logstash.k8s.elastic.co/v1alpha1kind: Logstashmetadata:  name: logstash-samplespec:  version: 9.3.0  count: 1  elasticsearchRefs:    - name: "elasticsearch-sample"      clusterName: "sample"  podTemplate:    spec:      containers:      - name: logtash        env:        - name: LS_JAVA_OPTS          value: "-Xmx2g -Xms2g"        resources:          requests:            memory: 1Gi            cpu: 0.5          limits:            memory: 4Gi            cpu: 2

The name of the container in the Pod template must belogstash.

[8]ページ先頭
©2009-2026 Movatter.jp