DocumentationGrafana LokiSet upUpgrade

Upgrade Loki

Every attempt is made to keep Grafana Loki backwards compatible, such that upgrades should be low risk and low friction.

Unfortunately Loki is software and software is hard and sometimes we are forced to make decisions between ease of use and ease of maintenance.

If we have any expectation of difficulty upgrading, we will document it here.

As more versions are released it becomes more likely unexpected problems arise moving between multiple versions at once.If possible try to stay current and do sequential updates. If you want to skip versions, try it in a development environment before attempting to upgrade production.

Checking for config changes

Using docker you can check changes between 2 versions of Loki with a command like this:

BashCopy

export OLD_LOKI=2.9.4export NEW_LOKI=3.0.0export CONFIG_FILE=local-config.yamldiff --color=always --side-by-side <(docker run --rm -t -v "${PWD}":/config grafana/loki:${OLD_LOKI} -config.file=/etc/loki/${CONFIG_FILE} -print-config-stderr 2>&1 | sed '/Starting Loki/q' | tr -d '\r') <(docker run --rm -t -v "${PWD}":/config grafana/loki:${NEW_LOKI} -config.file=/etc/loki/${CONFIG_FILE} -print-config-stderr 2>&1 | sed '/Starting Loki/q' | tr -d '\r') | less -R

Thetr -d '\r' is likely not necessary for most people, seems like WSL2 was sneaking in some windows newline characters…

The output is incredibly verbose as it shows the entire internal config struct used to run Loki, you can play around with the diff command if you prefer to only show changes or a different style output.

Main / Unreleased

Distributor Max Receive Limits for uncompressed bytes

The next Loki release introduces a new configuration option (i.e.-distibutor.max-recv-msg-size) for the distributors to control the max receive size of uncompressed stream data. The new options’s default value is set to100MB.

Supported clients should check the configuration options for max send message size if applicable.

Helm Chart Upgrades

Helm Chart 6.34.0 - Zone-aware Ingester Breaking Change

If you are using zone-aware ingesters (ingester.zoneAwareReplication.enabled: true), upgrading to Helm chart 6.34.0 requires manual StatefulSet deletion before the upgrade. This is due to a fix for theserviceName field in zone-aware ingester StatefulSets, which is an immutable field in Kubernetes.

For detailed upgrade instructions, see:Helm Chart 6.x Upgrade Guide - Zone-aware Ingester Breaking Change

Key points:

• Only affects deployments withingester.zoneAwareReplication.enabled: true
• Requires manual StatefulSet deletion with--cascade=orphan
• No data loss - PersistentVolumeClaims and data are preserved
• New StatefulSets will be created with correct service references

3.4.0

Loki 3.4.0

New Object Storage Clients

Loki release 3.4.0 introduces new object storage clients based on theThanos Object Storage Client Go module, this is an opt-in feature.In a future release, this will become the default way of configuring storage and the existing storage clients will be deprecated.

The new storage configuration deviates from the existing format. Refer to theThanos storage configuration reference to view the complete list of supported storage providers and their configuration options.

The documentation now also includes amigration guide andconfiguration examples for using Thanos-based storage clients.

3.3.0

Loki 3.3.0

Experimental Bloom Filters

With Loki 3.3.0, the bloom block format changed and any previously created block is incompatible with the new format.Before upgrading, we recommend deleting all the existing bloom blocks in the object store. We store bloom blocks andmetas inside thebloom path in the configured object store. To get rid of all the bloom blocks, delete all the objectsinside thebloom path in the object store.

3.2.0

Loki 3.2.0

HTTP API

The API endpoint for instant queries/api/v1/query now returns a HTTP status 400 (Bad Request) when the providedqueryparameter contains a log selector query instead of returning inconsistent results. Please use the range query endpoint/api/v1/query_range (Range type in Grafana Explore) instead.

Configuration

Loki changes the default value of-ruler.alertmanager-use-v2 fromfalse totrue. Alertmanager APIv1 was deprecated in Alertmanager 0.16.0 and is removed as of 0.27.0.

Experimental Bloom Filters

The bloom compactor component, which builds bloom filter blocks for query acceleration, has been removed in favor of two new components: bloom planner and bloom builder.Please consult theQuery Acceleration with Blooms docs for more information.

CLI arguments (and their YAML counterparts) of per-tenant settings that have been removed as part of this change:

• -bloom-compactor.enable-compaction
• -bloom-compactor.shard-size
• -bloom-compactor.shard-size
• -bloom-compactor.shard-size

CLI arguments of per-tenant settings that have been moved to a different prefix as part of this change:

• -bloom-compactor.max-page-size changed to-bloom-builder.max-page-size
• -bloom-compactor.max-block-size changed to-bloom-builder.max-block-size
• -bloom-compactor.ngram-length changed to-bloom-builder.ngram-length
• -bloom-compactor.ngram-skip changed to-bloom-builder.ngram-skip
• -bloom-compactor.false-positive-rate changed to-bloom-builder.false-positive-rate
• -bloom-compactor.block-encoding changed to-bloom-builder.block-encoding

Their YAML counterparts in thelimits_config block are kept identical.

All other CLI arguments (and their YAML counterparts) prefixed with-bloom-compactor. have been removed.

3.0.0

Loki 3.0 is a major version increase and comes with several breaking changes.

Here is the shortlist of things we think most people may encounter:

• Structured metadata is enabled by default and requirestsdb andv13 schema or Loki won’t start. Refer toStructured Metadata, Open Telemetry, Schemas and Indexes.
• Theshared_store config is removed. Refer toRemovedshared_store andshared_store_key_prefix from shipper configuration.
• Loki now enforces a max line size of 256KB by default (you can disable this or increase this but this is how Grafana Labs runs Loki). Refer toChanges to default configure values.
• Loki now enforces a max label limit of 15 labels per series, down from 30. Extra labels inflate the size of the index and reduce performance, you should almost never need more than 15 labels. Refer toChanges to default configure values.
• Loki will automatically attempt to populate aservice_name label on ingestion. Refer toservice_name label.
• There are many metric name changes. Refer toDistributor metric changes,Embedded cache metric changes, andMetrics namespace.

If you would like to see if your existing configuration will work with Loki 3.0:

1. In an empty directory on your computer, copy you configuration into a file namedloki-config.yaml.
2. Run this command from that directory:

BashCopy

docker run --rm -t -v "${PWD}":/config grafana/loki:3.0.0 -config.file=/config/loki-config.yaml -verify-config=true

TheHelm chart has gone through some significant changes and has a separate upgrade guide:Upgrading to Helm 6.x.

Loki 3.0.0

Structured Metadata, Open Telemetry, Schemas and Indexes

A flagship feature of Loki 3.0 is native support for the Open Telemetry Protocol (OTLP). This is made possible by a new feature in Loki calledStructured Metadata, a place for metadata which doesn’t belong in labels or log lines. OTel resources and attributes are often a great example of data which doesn’t belong in the index nor in the log line.

Structured Metadata is enabled by default in Loki 3.0, however, it requires your active schema be using both thetsdb index type AND thev13 storage schema. If you are not using both of these you have two options:

• Upgrade your index version and schema version before updating to 3.0, seeschema config upgrade.
• Disable Structured Metadata (and therefore OTLP support) and upgrade to 3.0 and perform the schema migration after. This can be done by settingallow_structured_metadata: false in thelimits_config section or set the command line argument-validation.allow-structured-metadata=false.

service_name label

Loki 3.0 will automatically assign aservice_name label to all ingested logs by default. A service name is something required by Open Telemetry semantic conventions and is something Grafana Labs is building into our future user interface and query experiences.

Loki will attempt to create theservice_name label by looking for the following labels in this order:

• service_name
• service
• app
• application
• name
• app_kubernetes_io_name
• container
• container_name
• component
• workload
• job

If no label is found matching the list, a value ofunknown_service is applied.

You can change this list by providing a list of labels todiscover_service_name in thelimits_config block.

You can disable this by providing an empty value fordiscover_service_name.

Removedshared_store andshared_store_key_prefix from shipper configuration

The following CLI flags and the corresponding YAML settings to configure shared store for TSDB and BoltDB shippers are now removed:

• -boltdb.shipper.shared-store
• -tsdb.shipper.shared-store

Going forward theobject_store setting in theperiod_config will be used to configure the store for the index.This enforces chunks and index files to reside together in the same storage bucket for a given period.

We are removing the shared store setting in an effort to simplify storage configuration and reduce the possibility for misconfiguration.

• If you have not configured-boltdb.shipper.shared-store,-tsdb.shipper.shared-store or their corresponding YAML setting before, no changes are required as part of the upgrade.
• If you have configured-boltdb.shipper.shared-store or its YAML setting:
  • If it matches the value ofobject_store for all the periods that useboltdb-shipper as index type, no additional changes are required besides removing the usage of the deleted configuration option.
  • If there is a mismatch, you lose access to the index for periods where-boltdb.shipper.shared-store does not matchobject_store.
    • To make these indexes queryable, index tables need to moved or copied to the store configured inobject_store.
• If you have configured-tsdb.shipper.shared-store or its YAML setting:
  • If it matches the value ofobject_store for all the periods that usetsdb as index type, no additional changes are required besides removing the usage of the deleted configuration option.
  • If there is a mismatch, you lose access to the index for periods where-tsdb.shipper.shared-store does not matchobject_store.
    • To make these indexes queryable, index tables need to moved or copied to the store configured inobject_store.

The following CLI flags and the corresponding YAML settings to configure a path prefix for TSDB and BoltDB shippers are now removed:

• -boltdb.shipper.shared-store.key-prefix
• -tsdb.shipper.shared-store.key-prefix

Path prefix for storing the index can now be configured by settingpath_prefix underindex key inperiod_config.This enables users to change the path prefix by adding a new period config.

YAMLCopy

period_config:  index:    path_prefix: "index/"    period: 24h

path_prefix defaults toindex/ which is same as the default value of the removed configurations.

• No changes are required if you have not configured-boltdb.shipper.shared-store.key-prefix,-tsdb.shipper.shared-store.key-prefix or the corresponding YAML setting previously.
• If you have configured-boltdb.shipper.shared-store.key-prefix or its YAML setting to a value other thanindex/, ensure that all the existing period configs that useboltdb-shipper as the index havepath_prefix set to the value previously configured.
• If you have configured-tsdb.shipper.shared-store.key-prefix or its YAML setting to a value other thanindex/, ensure that all the existing period configs that usetsdb as the index have thepath_prefix set to the value previously configured.

Removedshared_store andshared_store_key_prefix from compactor configuration

The following CLI flags and the corresponding YAML settings to configure the shared store and path prefix for compactor are now removed:

• -boltdb.shipper.compactor.shared-store
• -boltdb.shipper.compactor.shared-store.key-prefix

Going forward compactor will run compaction and retention on all the object stores configured inperiod configs where the index type is eithertsdb orboltdb-shipper.

delete_request_store should be explicitly configured

-compactor.delete-request-store or its YAML setting should be explicitly configured when retention is enabled, this is required for storing delete requests.The path prefix under which the delete requests are stored is decided by-compactor.delete-request-store.key-prefix, it defaults toindex/.

Configurationasync_cache_write_back_concurrency andasync_cache_write_back_buffer_size have been removed

These configurations were redundant with theBackground configuration in thecache-config.

async_cache_write_back_concurrency can be set withwriteback_goroutinesasync_cache_write_back_buffer_size can be set withwriteback_buffer

additionally theBackground configuration also lest you setwriteback_size_limit which can be used to set a maximum amount of memory to use for writeback objects vs a count of objects.

Legacy ingester shutdown handler is removed

The already deprecated handler/ingester/flush_shutdown is removed in favor of/ingester/shutdown?flush=true.

Ingester configurationmax_transfer_retries is removed

The settingmax_transfer_retries (-ingester.max-transfer-retries) is removed in favor of the Write Ahead log (WAL).It was used to allow transferring chunks to new ingesters when the old ingester was shutting down during a rolling restart.Alternatives to this setting are:

• A. (Preferred) Enable the WAL and rely on the new ingester to replay the WAL.
  • Optionally, you can enableflush_on_shutdown (-ingester.flush-on-shutdown) to flush to long-term storage on shutdowns.
• B. Manually flush during shutdowns viathe ingester/shutdown?flush=true endpoint.

Removed thedefault section of the runtime overrides config file

This was introduced in 2.9 and likely not widely used. This only affects you if you run Loki with a runtime config file AND you had populated the newdefault block added in 2.9.

Thedefault block was removed and instead a top level config now exists in the standard Loki config calledoperational_config, you can set default values here for runtime configs.

Configurationuse_boltdb_shipper_as_backup is removed

The settinguse_boltdb_shipper_as_backup (-tsdb.shipper.use-boltdb-shipper-as-backup) was a remnant from the development of the TSDB storage.It was used to allow writing to both TSDB and BoltDB when TSDB was still highly experimental.Since TSDB is now stable and the recommended index type, the setting has become irrelevant and therefore was removed.The previous default valuefalse is applied.

Deprecated configuration options are removed

1. Removed already deprecatedstore.max-look-back-period CLI flag and the corresponding YAML settings. Usequerier.max-query-lookback config instead.
2. Removes already deprecated-querier.engine.timeout CLI flag and the corresponding YAML setting.
3. Also removes thequery_timeout from the querier YAML section. Instead of configuringquery_timeout underquerier, you now configure it inLimits Config.
4. s3.sse-encryption is removed. AWS now defaults encryption of all buckets to SSE-S3. Usesse.type to set SSE type.
5. ruler.wal-cleaer.period is removed. Useruler.wal-cleaner.period instead.
6. experimental.ruler.enable-api is removed. Useruler.enable-api instead.
7. split_queries_by_interval is removed fromquery_range YAML section. You can instead configure it inLimits Config.
8. frontend.forward-headers-list CLI flag and its corresponding YAML setting are removed.
9. frontend.cache-split-interval CLI flag is removed. Results caching interval is now determined byquerier.split-queries-by-interval.
10. querier.worker-parallelism CLI flag and its corresponding yaml setting are now removed as it does not offer additional value to already existingquerier.max-concurrent.We recommend configuringquerier.max-concurrent to limit the max concurrent requests processed by the queriers.
11. ruler.evaluation-delay-duration CLI flag and the corresponding YAML setting are removed.
12. validation.enforce-metric-name CLI flag and the corresponding YAML setting are removed.
13. boltdb.shipper.compactor.deletion-mode CLI flag and the corresponding YAML setting are removed. You can instead configure thecompactor.deletion-mode CLI flag ordeletion_mode YAML setting inLimits Config.
14. Compactor CLI flags that use the prefixboltdb.shipper.compactor. are removed. You can instead use CLI flags with thecompactor. prefix.

Distributor metric changes

Theloki_distributor_ingester_append_failures_total metric has been removed in favour ofloki_distributor_ingester_append_timeouts_total.This new metric will provide a more clear signal that there is an issue with ingesters, and this metric can be used for high-signal alerting.

Changes to default configuration values in 3.0

Automatic stream sharding is enabled by default

Automatic stream sharding helps keep the write load of high volume streams balanced across ingesters and helps to avoid hot-spotting. Check out theoperations page for more information

More results caching is enabled by default

The TSDB index type has support for caching results for ‘stats’ and ‘volume’ queries which are now enabled by default.

’label’ and ‘series’ requests can be cached now too and this is enabled by default.

All of these are cached to theresults_cache which is configured in thequery_range config section. By default, an in memory cache is used.

Write dedupe cache is deprecated

Write dedupe cache is deprecated because it not required by the newer single store indexes (TSDB andboltdb-shipper).If you using alegacy index type, consider migrating to TSDB (recommended).

Embedded cache metric changes

• The following embedded cache metrics are removed. Instead useloki_cache_fetched_keys,loki_cache_hits,loki_cache_request_duration_seconds which instruments requests made to the configured cache (embeddedcache,memcached orredis).

  • querier_cache_added_total
  • querier_cache_gets_total
  • querier_cache_misses_total

• The following embedded cache metrics are renamed:

  • querier_cache_added_new_total is renamed toloki_embeddedcache_added_new_total
  • querier_cache_evicted_total is renamed toloki_embeddedcache_evicted_total
  • querier_cache_entries is renamed toloki_embeddedcache_entries
  • querier_cache_memory_bytes is renamed toloki_embeddedcache_memory_bytes

• Already deprecated metricquerier_cache_stale_gets_total is now removed.

Metrics namespace

Some Loki metrics started with the prefixcortex_. In this release they will be changed so they start withloki_. To keep them atcortex_ change themetrics_namespace from the defaultloki tocortex. These metrics will be changed:

• cortex_distributor_ingester_clients
• cortex_dns_failures_total
• cortex_dns_lookups_total
• cortex_dns_provider_results
• cortex_frontend_query_range_duration_seconds_bucket
• cortex_frontend_query_range_duration_seconds_count
• cortex_frontend_query_range_duration_seconds_sum
• cortex_ingester_flush_queue_length
• cortex_kv_request_duration_seconds_bucket
• cortex_kv_request_duration_seconds_count
• cortex_kv_request_duration_seconds_sum
• cortex_member_consul_heartbeats_total
• cortex_prometheus_last_evaluation_samples
• cortex_prometheus_notifications_alertmanagers_discovered
• cortex_prometheus_notifications_dropped_total
• cortex_prometheus_notifications_errors_total
• cortex_prometheus_notifications_latency_seconds
• cortex_prometheus_notifications_latency_seconds_count
• cortex_prometheus_notifications_latency_seconds_sum
• cortex_prometheus_notifications_queue_capacity
• cortex_prometheus_notifications_queue_length
• cortex_prometheus_notifications_sent_total
• cortex_prometheus_rule_evaluation_duration_seconds
• cortex_prometheus_rule_evaluation_duration_seconds_count
• cortex_prometheus_rule_evaluation_duration_seconds_sum
• cortex_prometheus_rule_evaluation_failures_total
• cortex_prometheus_rule_evaluations_total
• cortex_prometheus_rule_group_duration_seconds
• cortex_prometheus_rule_group_duration_seconds_count
• cortex_prometheus_rule_group_duration_seconds_sum
• cortex_prometheus_rule_group_interval_seconds
• cortex_prometheus_rule_group_iterations_missed_total
• cortex_prometheus_rule_group_iterations_total
• cortex_prometheus_rule_group_last_duration_seconds
• cortex_prometheus_rule_group_last_evaluation_timestamp_seconds
• cortex_prometheus_rule_group_rules
• cortex_query_frontend_connected_schedulers
• cortex_query_frontend_queries_in_progress
• cortex_query_frontend_retries_bucket
• cortex_query_frontend_retries_count
• cortex_query_frontend_retries_sum
• cortex_query_scheduler_connected_frontend_clients
• cortex_query_scheduler_connected_querier_clients
• cortex_query_scheduler_inflight_requests
• cortex_query_scheduler_inflight_requests_count
• cortex_query_scheduler_inflight_requests_sum
• cortex_query_scheduler_queue_duration_seconds_bucket
• cortex_query_scheduler_queue_duration_seconds_count
• cortex_query_scheduler_queue_duration_seconds_sum
• cortex_query_scheduler_queue_length
• cortex_query_scheduler_running
• cortex_ring_member_heartbeats_total
• cortex_ring_member_tokens_owned
• cortex_ring_member_tokens_to_own
• cortex_ring_members
• cortex_ring_oldest_member_timestamp
• cortex_ring_tokens_total
• cortex_ruler_client_request_duration_seconds_bucket
• cortex_ruler_client_request_duration_seconds_count
• cortex_ruler_client_request_duration_seconds_sum
• cortex_ruler_clients
• cortex_ruler_config_last_reload_successful
• cortex_ruler_config_last_reload_successful_seconds
• cortex_ruler_config_updates_total
• cortex_ruler_managers_total
• cortex_ruler_ring_check_errors_total
• cortex_ruler_sync_rules_total

Themetrics_namespace setting is deprecated already. It will be removed in the next minor release. The default prefix will beloki then.

LogCLI

Store for retrieving remote schema

Previously LogCLI used to fetch remote schema from the store configured in-boltdb.shipper.shared-store when-remote-schema is set totrue.A new CLI flag-schema-store is introduced as a replacement to configure the store for retrieving remote schema.

2.9.0

Loki 2.9.0

Index gateway shuffle sharding

The index gateway now supports shuffle sharding of index data when running in“ring” mode. The index data is sharded by tenant where each tenant getsassigned a sub-set of all available instances of the index gateways in the ring.

If you configured a high replication factor to accommodate for load, sincein the past this was the only option to give a tenant more instances forquerying, you should consider reducing the replication factor to a meaningfulvalue for replication (for example, from 12 to 3) and instead set the shard factor forindividual tenants as required.

If the global shard factor (no per-tenant) is 0 (default value), the globalshard factor is set to replication factor. It can still be overwritten pertenant.

In the context of the index gateway, sharding is synonymous to replication.

Index shipper multi-store support

In previous releases, if you did not explicitly configure-boltdb.shipper.shared-store,-tsdb.shipper.shared-store, those values default to theobject_store configured in the latestperiod_config of the corresponding index type.These defaults are removed in favor of uploading indexes to multiple stores. If you do not explicitly configure ashared-store, the boltdb and tsdb indexes will be shipped to theobject_store configured for that period.

Shutdown marker file

A shutdown marker file can be written by the/ingester/prepare_shutdown endpoint.If the newingester.shutdown_marker_path config setting has a value that value is used.If not thecommon.path_prefix config setting is used if it has a value. Otherwise a warning is shownin the logs on startup and the/ingester/prepare_shutdown endpoint will return a 500 status code.

Compactor multi-store support

In previous releases, setting-boltdb.shipper.compactor.shared-store configured the following:

• store used for managing delete requests.
• store on which index compaction should be performed.

If-boltdb.shipper.compactor.shared-store was not set, it used to default to theobject_store configured in the latestperiod_config that uses either the tsdb or boltdb-shipper index.

Compactor now supports index compaction on multiple buckets/object stores.And going forward loki will not set any defaults on-boltdb.shipper.compactor.shared-store, this has a couple of side effects detailed as follows:

store on which index compaction should be performed

If-boltdb.shipper.compactor.shared-store is configured by the user, loki would run index compaction only on the store specified by the config.If not set, compaction would be performed on all the object stores that contain either a boltdb-shipper or tsdb index.

store used for managing delete requests

A new config option-boltdb.shipper.compactor.delete-request-store decides where delete requests should be stored. This new option takes precedence over-boltdb.shipper.compactor.shared-store.

In the case where neither of these options are set, theobject_store configured in the latestperiod_config that uses either a tsdb or boltdb-shipper index is used for storing delete requests to ensure pending requests are processed.

logfmt parser non-strict parsing

logfmt parser now performs non-strict parsing which helps scan semi-structured log lines.It skips invalid tokens and tries to extract as many key/value pairs as possible from the rest of the log line.If you have a use-case that relies on strict parsing where you expect the parser to throw an error, use| logfmt --strict to enable strict mode.

logfmt parser doesn’t include standalone keys(keys without a value) in the resulting label set anymore.You can use--keep-empty flag to retain them.

Jsonnet 2..9.0

Deprecated PodDisruptionBudget definition has been removed

Thepolicy/v1beta1 API version of PodDisruptionBudget is no longer served as of Kubernetes v1.25.To support the latest versions of the Kubernetes, it was necessary to replacepolicy/v1beta1 with the new definitionpolicy/v1 that is available since v1.21.

No impact is expected if you use Kubernetes v1.21 or newer.

Please refer toofficial migration guide for more details.

2.8.0

Loki 2.8.0

Change in LogQL behavior

When there are duplicate labels in a log line, only the first value will be kept. Previously only the last valuewas kept.

Default retention_period has changed

This change will affect you if you have:

YAMLCopy

compactor:  retention_enabled: true

And didnot define aretention_period inlimits_config, thus relying on the previous default of744h

In this release the default has been changed to0s.

A value of0s is the same as “retain forever” or “disable retention”.

If,and only if, you wish to retain the previous default of 744h, apply this config.

YAMLCopy

limits_config:  retention_period: 744h

metrics.go log linesubqueries replaced withsplits andshards

The metrics.go log line emitted for every query had an entry calledsubqueries which was intended to represent the amount a query was parallelized on execution.

In the current form it only displayed the count of subqueries generated with the Loki split by time logic and did not include counts for shards.

There wasn’t a clean way to update subqueries to include sharding information and there is value in knowing the difference between the subqueries generated when we split by time vs sharding factors, especially now that TSDB can do dynamic sharding.

In 2.8 we no longer includesubqueries in metrics.go, it does still exist in the statistics API data returned but just for backwards compatibility, the value will always be zero now.

Instead, now you can usesplits to see how many split by time intervals were created andshards to see the total number of shards created for a query.

Promtail 2.8.0

The go build tagpromtail_journal_enabled was introduced

The go build tagpromtail_journal_enabled should be passed to include Journal support to the promtail binary.If you need Journal support you will need to run go build with tagpromtail_journal_enabled:

shellCopy

go build --tags=promtail_journal_enabled ./clients/cmd/promtail

Introducing this tag aims to relieve Linux/CentOS users with CGO enabled from installing libsystemd-dev/systemd-devel libraries if they don’t need Journal support.

Ruler

CLI flagruler.wal-cleaer.period deprecated

CLI flagruler.wal-cleaer.period is now deprecated and replaced with a typo fixruler.wal-cleaner.period.The yaml configuration remains unchanged:

YAMLCopy

ruler:  wal_cleaner:    period: 5s

Querier

query-frontend Kubernetes headless service changed to load balanced service

Thequery-frontend Kubernetes service was previously headless and was used for two purposes:

• Distributing the Loki query requests amongst all the available Query Frontend pods.
• Discover IPs of Query Frontend pods from Queriers to connect as workers.

The problem here is that a headless service does not support load balancing and leaves it up to the client to balance the load.Additionally, a load-balanced service does not let us discover the IPs of the underlying pods.

To meet both these requirements, we have made the following changes:

• Changed the existingquery-frontend Kubernetes service from headless to load-balanced to have a fair load distribution on all the Query Frontend instances.
• Addedquery-frontend-headless to discover QF pod IPs from queriers to connect as workers.

If you are deploying Loki with Query Scheduler by settingquery_scheduler_enabled config totrue, then there is nothing to do here for this change.If you are not using Query Scheduler, then to avoid any issues on the Read path until the rollout finishes, it would be good to follow below steps:

• Create just thequery-frontend-headless service without applying any changes to thequery-frontend service.
• Rollout changes toqueriers.
• Roll out the rest of the changes.

General

Store & Cache Statistics

Statistics are now logged inmetrics.go lines about how long it takes to download chunks from the store, as well as how long it takes to download chunks, index query, and result cache responses from cache.

Example (note the*_download_time fields):

BashCopy

level=info ts=2022-12-20T15:27:54.858554127Z caller=metrics.go:147 component=frontend org_id=docker latency=fast query="sum(count_over_time({job=\"generated-logs\"}[1h]))" query_type=metric range_type=range length=6h17m48.865587821s start_delta=6h17m54.858533178s end_delta=5.99294552s step=1m30s duration=5.990829396s status=200 limit=30 returned_lines=0 throughput=123MB total_bytes=738MB total_entries=1 store_chunks_download_time=2.319297059s queue_time=2m21.476090991s subqueries=8 cache_chunk_req=81143 cache_chunk_hit=32390 cache_chunk_bytes_stored=1874098 cache_chunk_bytes_fetched=94289610 cache_chunk_download_time=56.96914ms cache_index_req=994 cache_index_hit=710 cache_index_download_time=1.587842ms cache_result_req=7 cache_result_hit=0 cache_result_download_time=380.555µs

These statistics are also displayed when using--stats with LogCLI.

2.7.0

Loki 2.7.0

Loki Canary Permission

The newpush mode toLoki canary can push logs that are generated by a Loki canary directly to a given Loki URL. Previously, it only wrote to a local file and you needed some agent, such as promtail, to scrape and push it to Loki.So if you run Loki behind some proxy with different authorization policies to read and write to Loki, then auth credentials we pass to Loki canary now needs to have bothREAD andWRITE permissions.

engine.timeout andquerier.query_timeout are deprecated

Previously, we had two configurations to define a query timeout:engine.timeout andquerier.query-timeout.As they were conflicting andengine.timeout isn’t as expressive asquerier.query-timeout,we’re deprecating it and moving it toLimits Configlimits_config.query_timeout with same default values.

fifocache has been renamed

The in-memoryfifocache has been renamed toembedded-cache. This allows us to replace the implementation (currently a simple FIFO data structure) with something else in the future without causing confusion

Evenly spread Memcached pods for chunks across kubernetes nodes

We now evenly spread memcached_chunks pods across the available kubernetes nodes, but allowing more than one pod to be scheduled into the same node.If you want to run at most a single pod per node, set$.memcached.memcached_chunks.use_topology_spread to false.

While we attempt to schedule at most 1 memcached_chunks pod per Kubernetes node with thetopology_spread_max_skew: 1 field,if no more nodes are available then multiple pods will be scheduled on the same node.This can potentially impact your service’s reliability so consider tuning these values according to your risk tolerance.

Evenly spread distributors across kubernetes nodes

We now evenly spread distributors across the available kubernetes nodes, but allowing more than one distributors to be scheduled into the same node.If you want to run at most a single distributors per node, set$._config.distributors.use_topology_spread to false.

While we attempt to schedule at most 1 distributor per Kubernetes node with thetopology_spread_max_skew: 1 field,if no more nodes are available then multiple distributors will be scheduled on the same node.This can potentially impact your service’s reliability so consider tuning these values according to your risk tolerance.

Evenly spread queriers across kubernetes nodes

We now evenly spread queriers across the available kubernetes nodes, but allowing more than one querier to be scheduled into the same node.If you want to run at most a single querier per node, set$._config.querier.use_topology_spread to false.

While we attempt to schedule at most 1 querier per Kubernetes node with thetopology_spread_max_skew: 1 field,if no more nodes are available then multiple queriers will be scheduled on the same node.This can potentially impact your service’s reliability so consider tuning these values according to your risk tolerance.

Default value forserver.http-listen-port changed

This value now defaults to 3100, so the Loki process doesn’t require special privileges. Previously, it had been set to port 80, which is a privileged port. If you need Loki to listen on port 80, you can set it back to the previous default using-server.http-listen-port=80.

docker-compose setup has been updated

The docker-composesetup has been updated tov2.6.0 and includes many improvements.

Notable changes include:

• authentication (multi-tenancy) isenabled by default; you can disable it inproduction/docker/config/loki.yaml by settingauth_enabled: false
• storage is now using Minio instead of local filesystem
  • move your current storage into.data/minio and it should work transparently
• log-generator was added - if you don’t need it, simply remove the service fromdocker-compose.yaml or don’t start the service

Configuration for deletes has changed

The globaldeletion_mode option in the compactor configuration moved to runtime configurations.

• Thedeletion_mode option needs to be removed from your compactor configuration
• Thedeletion_mode global override needs to be set to the desired mode:disabled,filter-only, orfilter-and-delete. By default,filter-and-delete is enabled.
• Anyallow_delete per-tenant overrides need to be removed or changed todeletion_mode overrides with the desired mode.

Metric name forloki_log_messages_total changed

The name of this metric was changed toloki_internal_log_messages_total to reduce ambiguity. The previous name is still present but is deprecated.

Usage Report / Telemetry config has changed named

The configuration for anonymous usage statistics reporting to Grafana has changed fromusage_report toanalytics.

TLScipher_suites andtls_min_version have moved

These were previously configurable underserver.http_tls_config andserver.grpc_tls_config separately. They are now underserver.tls_cipher_suites andserver.tls_min_version. These values are also now configurable for individual clients, for example:distributor.ring.etcd orquerier.ingester_client.grpc_client_config.

ruler.storage.configdb has been removed

ConfigDB was disallowed as a Ruler storage option back in 2.0. The config struct has finally been removed.

ruler.remote_write.client has been removed

Can no longer specify a remote write client for the ruler.

Promtail 2.7.0

gcp_push_target_parsing_errors_total has a newreason label

Thegcp_push_target_parsing_errors_total GCP Push Target metrics has been added a new label namedreason. This includes detail on what might have caused the parsing to fail.

Windows event logs: now correctly includesuser_data

The contents of theuser_data field was erroneously set to the same value asevent_data in previous versions. This was fixed in#7461 and log queries relying on this broken behavior may be impacted.

2.6.0

Loki 2.6.0

Implementation of unwrappedrate aggregation changed

The implementation of therate() aggregation function changed back to the previous implementation prior to#5013.This means that the rate per second is calculated based on the sum of the extracted values, instead of the average increase over time.

If you want the extracted values to be treated asCounter metric, you should use the newrate_counter() aggregation function, which calculates the per-second average rate of increase of the vector.

Default value forazure.container-name changed

This value now defaults toloki, it was previously set tocortex. If you are relying on this container name for your chunks or ruler storage, you will have to manually specify-azure.container-name=cortex or-ruler.storage.azure.container-name=cortex respectively.

2.5.0

Loki 2.5.0

split_queries_by_interval yaml configuration has moved

It was previously possible to define this value in two places

YAMLCopy

query_range:  split_queries_by_interval: 10m

and/or

YAMLCopy

limits_config:  split_queries_by_interval: 10m

In 2.5.0 it can only be defined in thelimits_config section,Loki will fail to start if you do not remove thesplit_queries_by_interval config from thequery_range section.

Additionally, it has a new default value of30m rather than0.

The CLI flag is not changed and remainsquerier.split-queries-by-interval.

Dropped support for old Prometheus rules configuration format

Alerting rules previously could be specified in two formats: 1.x format (legacy one, namedv0 internally) and 2.x.We decided to drop support for format1.x as it is fairly old and keeping support for it required a lot of code.

In case you’re still using the legacy format, take a look atAlerting Rules for instructionson how to write alerting rules in the new format.

For reference, the newer format follows a structure similar to the one below:

YAMLCopy

 groups: - name: example   rules:   - alert: HighErrorRate     expr: job:request_latency_seconds:mean5m{job="myjob"} > 0.5     for: 10m     labels:       severity: page     annotations:       summary: High request latency

Meanwhile, the legacy format is a string in the following format:

Copy

 ALERT <alert name>   IF <expression>   [ FOR <duration> ]   [ LABELS <label set> ]   [ ANNOTATIONS <label set> ]

Changes to default configuration values

• parallelize_shardable_queries under thequery_range config now defaults totrue.
• split_queries_by_interval under thelimits_config config now defaults to30m, it was0s.
• max_chunk_age in theingester config now defaults to2h previously it was1h.
• query_ingesters_within under thequerier config now defaults to3h, previously it was0s. Any query (or subquery) that has an end time more than3h ago will not be sent to the ingesters, this saves work on the ingesters for data they normally don’t contain. If you regularly write old data to Loki you may need to return this value to0s to always query ingesters.
• max_concurrent under thequerier config now defaults to10 instead of20.
• match_max_concurrent under thefrontend_worker config now defaults to true, this supersedes theparallelism setting which can now be removed from your config. Controlling query parallelism of a single process can now be done with thequeriermax_concurrent setting.
• flush_op_timeout under theingester configuration block now defaults to10m, increased from10s. This can help when replaying a large WAL on Loki startup, and avoidmsg="failed to flush" ... context deadline exceeded errors.

Promtail 2.5.0

gcplog labels have changed

• Resource labels have been moved from__<NAME> to__gcp_resource_labels_<NAME> for example, if you previously used__project_id then you’ll need to update your relabel config to use__gcp_resource_labels_project_id.
• resource_type has been moved to__gcp_resource_type

promtail_log_entries_bytes_bucket histogram has been removed

This histogram reports the distribution of log line sizes by file. It has 8 buckets for every file being tailed.

This creates a lot of series and we don’t think this metric has enough value to offset the amount of series generated so we are removing it.

While this isn’t a direct replacement, two metrics we find more useful are size and line counters configured via pipeline stages, an example of how to configure these metrics can be found in themetrics pipeline stage docs.

added Docker target log message has been demoted from level=error to level=info

If you have dashboards that depended on the log level, change them to search for themsg="added Docker target" property.

Jsonnet 2.5.0

Compactor config defined as command line args moved to yaml config

Following 2 compactor configs that were defined as command line arguments in jsonnet are now moved to yaml config:

YAMLCopy

# Directory where files can be downloaded for compaction.# CLI flag: -boltdb.shipper.compactor.working-directory[working_directory: <string>]# The shared store used for storing boltdb files.# Supported types: gcs, s3, azure, swift, cos, filesystem.# CLI flag: -boltdb.shipper.compactor.shared-store[shared_store: <string>]

2.4.0

The following are important changes which should be reviewed and understood prior to upgrading Loki.

Loki 2.4.0

The following changes pertain to upgrading Loki.

The single binary no longer runs a table-manager

Single binary Loki means running loki with-target=all which is the default if no-target flag is passed.

This will impact anyone in the following scenarios:

1. Running a single binary Loki with any index type other thanboltdb-shipper orboltdb
2. Relying on retention with the configsretention_deletes_enabled andretention_period

Anyone in situation #1 who is not usingboltdb-shipper orboltdb (e.g.cassandra orbigtable) should modify their Loki command to include-target=all,table-manager this will instruct Loki to run a table-manager for you.

Anyone in situation #2, you have two options, the first (and not recommended) is to run Loki with a table-manager by adding-target=all,table-manager.

The second and recommended solution, is to use deletes via the compactor:

YAMLCopy

compactor:  retention_enabled: truelimits_config:  retention_period: [30d]

See theretention docs for more info.

Log messages on startup: proto: duplicate proto type registered

PR#3842cyriltovena: Fork cortex chunk storage into Loki.

Since Cortex doesn’t plan to use thechunk package anymore, we decided to fork it into our storage package tobe able to evolve and modify it easily. However, as a side-effect, we still vendor Cortex which includes this forkedcode and protobuf files resulting in log messages like these at startup:

BashCopy

2021-11-04 15:30:02.437911 I | proto: duplicate proto type registered: purgeplan.DeletePlan2021-11-04 15:30:02.437936 I | proto: duplicate proto type registered: purgeplan.ChunksGroup2021-11-04 15:30:02.437939 I | proto: duplicate proto type registered: purgeplan.ChunkDetails...

The messages are harmless and we will work to remove them in the future.

Change of some default limits to common values

PR4415DylanGuedes: the default value of some limits were changed to protect users from overwhelming their cluster with ingestion load caused by relying on default configs.

We suggest you double check if the following parameters arepresent in your Loki config:ingestion_rate_strategy,max_global_streams_per_usermax_query_lengthmax_query_parallelismmax_streams_per_userreject_old_samplesreject_old_samples_max_age. If they are not present, we recommend you double check that the new values will not negatively impact your system. The changes are:

Change of configuration defaults

• chunk_retain_period is necessary when using an index queries cache which is not enabled by default. If you have configured an index_queries_cache_config section make sure that you set chunk_retain_period larger than your cache TTL
• chunk_idle_period is how long before a chunk which receives no logs is flushed.
• chunk_target_size was increased to flush slightly larger chunks, if using memcache for a chunks store make sure it will accept files up to 1.5MB in size.

In memory FIFO caches enabled by default

Loki now enables a results cache and chunks cache in memory to improve performance. This can however increase memory usage as the cache’s by default are allowed to consume up to 1GB of memory.

If you would like to disable these caches or change this memory limit:

Disable:

YAMLCopy

chunk_store_config:  chunk_cache_config:    enable_fifocache: falsequery_range:  results_cache:    cache:      enable_fifocache: false

Resize:

YAMLCopy

chunk_store_config:  chunk_cache_config:    enable_fifocache: true    fifocache:      max_size_bytes: 500MBquery_range:  results_cache:    cache:      enable_fifocache: true      fifocache:        max_size_bytes: 500MB

Ingester Lifecyclerfinal_sleep now defaults to0s

• 4608trevorwhitney: Change default value of ingester lifecycler’sfinal_sleep from30s to0s

This final sleep exists to keep Loki running for long enough to get one final Prometheus scrape before shutting down, however it also causes Loki to sit idle for 30s on shutdown which is an annoying experience for many people.

We decided the default would be better to disable this sleep behavior but anyone can set this config variable directly to return to the previous behavior.

Ingester WAL now defaults to on, and chunk transfers are disabled by default

• 4543trevorwhitney: Change more default values and improve application of common storage config
• 4629owen-d: Default the WAL to enabled in the Loki jsonnet library
• 4624chaudum: Disable chunk transfers in jsonnet lib

This changes a few default values, resulting in the ingester WAL now being on by default,and chunk transfer retries are disabled by default. Note, this now means Loki will depend on local disk by default for its WAL (write ahead log) directory. This defaults towal but can be overridden via the--ingester.wal-dir or viapath_prefix in the common configuration section. Below are config snippets with the previous defaults, and another with the new values.

Previous defaults:

YAMLCopy

ingester:  max_transfer_retries: 10  wal:    enabled: false

New defaults:

YAMLCopy

ingester:  max_transfer_retries: 0  wal:    enabled: true

Using the write ahead log (WAL) is recommended and is now the default. However using the WAL is incompatible with chunk transfers, if you have explicitly configuredingester.max-transfer-retries to a non-zero value, you must set it to 0 to disable transfers.

Memberlist config now automatically applies to all non-configured rings

• 4400trevorwhitney: Config: automatically apply memberlist config too all rings when provided

This change affects the behavior of the ingester, distributor, and ruler rings. Previously, if you wanted to use memberlist for all of these rings, youhad to provide amemberlist configuration as well as specifystore: memberlist for thekvstore of each of the rings you wanted to use memberlist.For example, your configuration might look something like this:

YAMLCopy

memberlist:  join_members:    - loki.namespace.svc.cluster.localdistributor:  ring:    kvstore:      store: memberlistingester:    lifecycler:      ring:        kvstore:          store: memberlistruler:  ring:    kvstore:      store: memberlist

Now, if your provide amemberlist configuration with at least onejoin_members, loki will default all rings to use akvstore of typememberlist.You can change this behavior by overriding specific configurations. For example, if you wanted to useconsul for youruler rings, butmemberlistfor theingester anddistributor, you could do so with the following config (although we don’t know why someone would want to do this):

YAMLCopy

memberlist:  join_members:    - loki.namespace.svc.cluster.localruler:  ring:    kvstore:      store: consul      consul:        host: consul.namespace.svc.cluster.local:8500

Changed defaults for some GRPC server settings

• 4435trevorwhitney: Change default values for two GRPC settings so querier can connect to frontend/scheduler

This changes two default values,grpc_server_min_time_between_pings andgrpc_server_ping_without_stream_allowed used by the GRPC server.

Previous Values:

YAMLCopy

server:  grpc_server_min_time_between_pings: '5m'  grpc_server_ping_without_stream_allowed: false

New Values:

YAMLCopy

server:  grpc_server_min_time_between_pings: '10s'  grpc_server_ping_without_stream_allowed: true

This issue has some more information on the change.

Some metric prefixes have changed fromcortex_ toloki_

• #3842/#4253jordanrushing: Metrics related to chunk storage and runtime config have changed their prefixes fromcortex_ toloki_.

YAMLCopy

cortex_runtime_config* -> loki_runtime_config*cortex_chunks_store* -> loki_chunks_store*

Recording rules storage is now durable

• 4344dannykopping: per-tenant WAL

Previously, samples generated by recording rules would only be buffered in memory before being remote-written to Prometheus; from thisversion, theruler now writes these samples to a per-tenant Write-Ahead Log for durability. More details about theper-tenant WAL can be foundhere.

Theruler now requires persistent storage - see theOperations page for more details about deployment.

Promtail 2.4.0

The following changes pertain to upgrading Promtail.

Promtail no longer insertpromtail_instance label when scrapinggcplog target

• 4556james-callahan: Removepromtail_instance label that was being added by promtail when scrapinggcplog target.

2.3.0

Loki 2.3.0

Query restriction introduced for queries which do not have at least one equality matcher

PR3216sandeepsukhani: check for stream selectors to have at least one equality matcher

This change now rejects any query which does not contain at least one equality matcher, an example may better illustrate:

{namespace=~".*"}

This query will now be rejected, however there are several ways to modify it for it to succeed:

Add at least one equals label matcher:

{cluster="us-east-1",namespace=~".*"}

Use.+ instead of.*

{namespace=~".+"}

This difference may seem subtle but if we break it down. matches any character,* matches zero or more of the preceding character and+ matches one or more of the preceding character. The.* case will match empty values where.+ will not, this is the important difference.{namespace=""} is an invalid request (unless you add another equals label matcher like the example above)

The reasoning for this change has to do with how index lookups work in Loki, if you don’t have at least one equality matcher Loki has to perform a complete index table scan which is an expensive and slow operation.

2.2.0

Loki 2.2.0

Be sure to upgrade to 2.0 or 2.1 BEFORE upgrading to 2.2

In Loki 2.2 we changed the internal version of our chunk format from v2 to v3, this is a transparent change and is only relevant if you every try todowngrade a Loki installation. We incorporated the code to read v3 chunks in 2.0.1 and 2.1, as well as 2.2 and any future releases.

If you upgrade to 2.2+ any chunks created can only be read by 2.0.1, 2.1 and 2.2+

This makes it important to first upgrade to 2.0, 2.0.1, or 2.1before upgrading to 2.2 so that if you need to rollback for any reason you can do so easily.

Loki Config

Read this if you use the query-frontend and havesharded_queries_enabled: true

We discovered query scheduling related to sharded queries over long time ranges could lead to unfair work scheduling by one single query in the per tenant work queue.

Themax_query_parallelism setting is designed to limit how many split and sharded units of ‘work’ for a single query are allowed to be put into the per tenant work queue at one time. The previous behavior would split the query by time using thesplit_queries_by_interval and compare this value tomax_query_parallelism when filling the queue, however with sharding enabled, every split was then sharded into 16 additional units of work after themax_query_parallelism limit was applied.

In 2.2 we changed this behavior to apply themax_query_parallelism after splittingand sharding a query resulting a more fair and expected queue scheduling per query.

What this means Loki will be putting much less work into the work queue per query if you are using the query frontend and have sharding_queries_enabled (which you should).You may need to increase yourmax_query_parallelism setting if you are noticing slower query performance In practice, you may not see a difference unless you were running a cluster with a LOT of queriers or queriers with a very highparallelism frontend_worker setting.

You could consider multiplying your currentmax_query_parallelism setting by 16 to obtain the previous behavior, though in practice we suspect few people would really want it this high unless you have a significant querier worker pool.

Also be aware to make suremax_outstanding_per_tenant is always greater thanmax_query_parallelism or large queries will automatically fail with a 429 back to the user.

Promtail 2.2.0

For 2.0 we eliminated the long deprecatedentry_parser configuration in Promtail configs, however in doing so we introduced a very confusing and erroneous default behavior:

If you did not specify apipeline_stages entry you would be provided with a default which included thedocker pipeline stage. This can lead to some very confusing results.

In3404, we corrected this behavior

If you are using docker, and any of yourscrape_configs are missing apipeline_stages definition, you should add the following to obtain the correct behavior:

YAMLCopy

pipeline_stages:  - docker: {}

2.1.0

The upgrade from 2.0.0 to 2.1.0 should be fairly smooth, be aware of these two things:

Helm charts have moved

Helm charts are now located in theGrafana Helm charts repo.

The helm repo URL is now:https://grafana.github.io/helm-charts.

Fluent Bit plugin renamed

Fluent bit officially supports Loki as an output plugin now.

However this created a naming conflict with our existing output plugin (the new native output uses the nameloki) so we have renamed our plugin.

In time our plan is to deprecate and eliminate our output plugin in favor of the native Loki support. However until then you can continue using the plugin with the following change:

Old:

shellCopy

[Output]    Name loki

New:

shellCopy

[Output]    Name grafana-loki

2.0.0

This is a major Loki release and there are some very important upgrade considerations.For the most part, there are very few impactful changes and for most this will be a seamless upgrade.

2.0.0 Upgrade Topics:

• IMPORTANT If you are using a docker image, read this!
• IMPORTANT boltdb-shipper upgrade considerations
• IMPORTANT results_cachemax_freshness removed from yaml config
• Promtail removed entry_parser config
• If you would like to use the new single store index and v11 schema

IMPORTANT If you are using a docker image, read this!

(This includes, Helm, Tanka, docker-compose etc.)

The default config file in the docker image, as well as the default helm values.yaml and jsonnet for Tanka all specify a schema definition to make things easier to get started.

In 2.0 the defaults are now v11 schema and theboltdb-shipper index type.

If you are using an index type ofaws,bigtable, orcassandra this means you have already defined a custom schema and there isnothing further you need to do regarding the schema.You can consider however adding a new schema entry to use the newboltdb-shipper type if you want to move away from these separate index stores and instead use just one object store.

What to do

The minimum action required is to create a config which specifies the schema to match what the previous defaults were.

(Keep in mind this will only tell Loki to use the old schema default, if you would like to upgrade to v11 and/or move to the single store boltdb-shipper,see below)

There are three places we have hard coded the schema definition:

Helm

Helm has shipped with the same internal schema in the values.yaml file for a very long time.

If you are providing your own values.yaml file then there is norequired action because you already have a schema definition.

If you are not providing your own values.yaml file, you will need to make one!

We suggest using the includedvalues.yaml file from the 1.6.0 tag

This matches what the default values.yaml file had prior to 2.0 and is necessary for Loki to work post 2.0

As mentioned above, you should also consider looking at moving to the v11 schema and boltdb-shippersee below for more information.

Tanka

This likely only affects a small portion of tanka users because the default schema config for Loki was forcingGCS andbigtable.

If yourmain.jsonnet (or somewhere in your manually created jsonnet) does not have a schema config section then you will need to add one like this!

jsonnetCopy

{  _config+:: {    using_boltdb_shipper: false,    loki+: {      schema_config+: {        configs: [{          from: '2018-04-15',          store: 'bigtable',          object_store: 'gcs',          schema: 'v11',          index: {            prefix: '%s_index_' % $._config.table_prefix,            period: '168h',          },        }],      },    },  }}

Changing the jsonnet config to use theboltdb-shipper type is the same asbelow where you need to add a new schema section.

HOWEVER Be aware when you changeusing_boltdb_shipper: true the deployment type for the ingesters and queriers will change to StatefulSets! StatefulSets are required for the ingester and querier using boltdb-shipper.

Docker (e.g. docker-compose)

For docker related cases you will have to mount a Loki config file separate from what’s shipped inside the container

I would recommend taking the previous default file from the1.6.0 tag on github

How you get this mounted and in use by Loki might vary based on how you are using the image, but this is a common example:

shellCopy

docker run -d --name=loki --mount type=bind,source="path to loki-config.yaml",target=/etc/loki/local-config.yaml

The Loki docker image is expecting to find the config file at/etc/loki/local-config.yaml

IMPORTANT: boltdb-shipper upgrade considerations

Significant changes have taken place between 1.6.0 and 2.0.0 for boltdb-shipper index type, if you are already running this index and are upgrading some extra caution is warranted.

Take a complete backup of theindex directory in your object store, this location might be slightly different depending on what store you use.It should be a folder named index with a bunch of folders inside with names likeindex_18561,index_18560…

The chunks directory should not need any special backups.

If you have an environment to test this in, do so before upgrading against critical data.

There are 2 significant changes warranting the backup of this data because they will make rolling back impossible:

• A compactor is included which will take existing index files and compact them to one per day and remove non compacted files
• All index files are now gzipped before uploading

The second part is important because 1.6.0 does not understand how to read the gzipped files, so any new files uploaded or any files compacted become unreadable to 1.6.0 or earlier.

THIS BEING SAID we are not expecting problems, our testing so far has not uncovered any problems, but some extra precaution might save data loss in unforeseen circumstances!

Report any problems via GitHub issues or reach us on the #loki slack channel.

To use boltdb-shipper in 2.0 you need a shared storage (S3, GCS, etc), the mode of running with separate filesystem stores in HA using a ring is not officially supported.

We didn’t do anything explicitly to limit this functionality however we have not had any time to actually test this which is why we removed the docs and are listing it as not supported.

If running in microservices, deploy ingesters before queriers

Ingesters now expose a new RPC method that queriers use when the index type isboltdb-shipper.Queriers generally roll out faster than ingesters, so if new queriers query older ingesters using the new RPC, the queries would fail.To avoid any query downtime during the upgrade, rollout ingesters before queriers.

If running the compactor, ensure it has delete permissions for the object storage

The compactor is an optional but suggested component that combines and deduplicates the boltdb-shipper index files. When compacting index files, the compactor writes a new file and deletes unoptimized files. Ensure that the compactor has appropriate permissions for deleting files, for example, s3:DeleteObject permission for AWS S3.

IMPORTANT:results_cache.max_freshness removed from YAML config

Themax_freshness config fromresults_cache has been removed in favour of another flag calledmax_cache_freshness_per_query inlimits_config which has the same effect.If you happen to haveresults_cache.max_freshness set, uselimits_config.max_cache_freshness_per_query YAML config instead.

Promtail config removed

The long deprecatedentry_parser config in Promtail has been removed, usepipeline_stages instead.

Upgrading schema to use boltdb-shipper and/or v11 schema

If you would also like to take advantage of the new Single Store (boltdb-shipper) index, as well as the v11 schema if you aren’t already using it.

You can do this by adding a new schema entry.

Here is an example:

YAMLCopy

schema_config:  configs:    - from: 2018-04-15           ①      store: boltdb              ①④      object_store: filesystem   ①④      schema: v11                ②      index:        prefix: index_           ①        period: 168h             ①    - from: 2020-10-24           ③      store: boltdb-shipper      object_store: filesystem   ④      schema: v11      index:        prefix: index_        period: 24h              ⑤

① Make sure all of these match your current schema config② Make sure this matches your previous schema version, Helm for example is likely v9③ Make sure this is a date in theFUTURE keep in mind Loki only knows UTC so make sure it’s a future UTC date④ Make sure this matches your existing config (e.g. maybe you were using gcs for your object_store)⑤ 24h is required for boltdb-shipper

There are more examples on theStorage description page including the information you need to setup thestorage section for boltdb-shipper.

1.6.0

Important: Ksonnet port changed and removed NET_BIND_SERVICE capability from Docker image

In 1.5.0 we changed the Loki user to not run as root which created problems binding to port 80.To address this we updated the docker image to add the NET_BIND_SERVICE capability to the loki processwhich allowed Loki to bind to port 80 as a non root user, so long as the underlying system allowed thatlinux capability.

This has proved to be a problem for many reasons and in PR2294the capability was removed.

It is now no longer possible for the Loki to be started with a port less than 1024 with the published docker image.

The default for Helm has always been port 3100, and Helm users should be unaffected unless they changed the default.

Ksonnet users however should closely check their configuration, in PR 2294 the loki port was changed from 80 to 3100

IMPORTANT: If you run Loki in microservices mode, special rollout instructions

A new ingester GRPC API has been added allowing to speed up metric queries, to ensure a rollout without query errorsmake sure you upgrade all ingesters first.Once this is done you can then proceed with the rest of the deployment, this is to ensure that queriers won’t look for an API not yet available.

If you roll out everything at once, queriers with this new code will attempt to query ingesters which may not have the new method on the API and queries will fail.

This will only affect reads(queries) and not writes and only for the duration of the rollout.

IMPORTANT: Scrape config changes to both Helm and Ksonnet will affect labels created by Promtail

Loki PR2091 makes several changes to the Promtail scrape config:

This is triggered by jsonnet-libs PR261:

The above PR changes the instance label to be actually unique withina scrape config. It also adds a pod and a container target labelso that metrics can easily be joined with metrics from cAdvisor, KSM,and the Kubelet.

This commit adds the same to the Loki scrape config. It also removesthe container_name label. It is the same as the container labeland was already added to Loki previously. However, thecontainer_name label is deprecated and has disappeared in Kubernetes 1.16,so that it will soon become useless for direct joining.

TL;DR

The following label have been changed in both the Helm and Ksonnet Promtail scrape configs:

instance ->podcontainer_name ->container

Experimental boltdb-shipper changes

PR2166 now forces the index to have a period of exactly24h:

Loki will fail to start with an error if the active schema or upcoming schema are not set to a period of24h

You can add a new schema config like this:

YAMLCopy

schema_config:  configs:    - from: 2020-01-01      <----- This is your current entry, date will be different      store: boltdb-shipper      object_store: aws      schema: v11      index:        prefix: index_        period: 168h    - from: [INSERT FUTURE DATE HERE]   <----- Add another entry, set a future date      store: boltdb-shipper      object_store: aws      schema: v11      index:        prefix: index_        period: 24h   <--- This must be 24h

If you are not onschema: v11 this would be a good opportunity to make that changein the new schema config also.

There was also a significant overhaul to how boltdb-shipper internals, this should not be visible to a user but as thisfeature is experimental and under development bug are possible!

The most noticeable change if you look in the storage, Loki no longer updates an existing file and instead creates anew index file every 15mins, this is an important move to make sure objects in the object store are immutable andwill simplify future operations like compaction and deletion.

Breaking CLI flags changes

The following CLI flags where changed to improve consistency, they are not expected to be widely used

diffCopy

- querier.query_timeout+ querier.query-timeout- distributor.extra-query-delay+ querier.extra-query-delay- max-chunk-batch-size+ store.max-chunk-batch-size- ingester.concurrent-flushed+ ingester.concurrent-flushes

Loki Canary metric name changes

When adding some new features to the canary we realized the existing metrics were not compliant with standards for counter names, the following metrics have been renamed:

nohighlightCopy

loki_canary_total_entries               ->      loki_canary_entries_totalloki_canary_out_of_order_entries        ->      loki_canary_out_of_order_entries_totalloki_canary_websocket_missing_entries   ->      loki_canary_websocket_missing_entries_totalloki_canary_missing_entries             ->      loki_canary_missing_entries_totalloki_canary_unexpected_entries          ->      loki_canary_unexpected_entries_totalloki_canary_duplicate_entries           ->      loki_canary_duplicate_entries_totalloki_canary_ws_reconnects               ->      loki_canary_ws_reconnects_totalloki_canary_response_latency            ->      loki_canary_response_latency_seconds

Ksonnet Changes

Inproduction/ksonnet/loki/config.libsonnet the variablestorage_backend used to have a default value of'bigtable,gcs'.This has been changed to providing no default and will error if not supplied in your environment jsonnet,here is an example of what you should add to have the same behavior as the default (namespace and cluster should already be defined):

jsonnetCopy

_config+:: {    namespace: 'loki-dev',    cluster: 'us-central1',    storage_backend: 'gcs,bigtable',

Defaulting togcs,bigtable was confusing for anyone using ksonnet with other storage backends as it would manifest itself with obscure bigtable errors.

1.5.0

Breaking config changes

Loki 1.5.0 vendors Cortex v1.0.0 (congratulations!), which has amassive list of changes.

While changes in the command line flags affect Loki as well, we usually recommend people to use configuration file instead.

Cortex has done lot of cleanup in the configuration files, and you are strongly urged to take a look at theannotated diff for config file before upgrading to Loki 1.5.0.

Following fields were removed from YAML configuration completely:claim_on_rollout (always true),normalise_tokens (always true).

Test Your Config

To see if your config needs to change, one way to quickly test is to download a 1.5.0 (or newer) binary from therelease page

Then run the binary providing your config file./loki-linux-amd64 -config.file=myconfig.yaml

If there are configs which are no longer valid you will see errors immediately:

shellCopy

./loki-linux-amd64 -config.file=loki-local-config.yamlfailed parsing config: loki-local-config.yaml: yaml: unmarshal errors:  line 35: field dynamodbconfig not found in type aws.StorageConfig

Referencing thelist of diffs I can see this config changed:

diffCopy

-  dynamodbconfig:+  dynamodb:

Also several other AWS related configs changed and would need to update those as well.

Loki Docker Image User and File Location Changes

To improve security concerns, in 1.5.0 the Docker container no longer runs the loki process asroot and instead the process runs as userloki with UID10001 and GID10001

This may affect people in a couple ways:

Loki Port

If you are running Loki with a config that opens a port number above 1024 (which is the default, 3100 for HTTP and 9095 for GRPC) everything should work fine in regards to ports.

If you are running Loki with a config that opens a port number less than 1024 Linux normally requires root permissions to do this, HOWEVER in the Docker container we runsetcap cap_net_bind_service=+ep /usr/bin/loki

This capability lets the loki process bind to a port less than 1024 when run as a non root user.

Not every environment will allow this capability however, it’s possible to restrict this capability in linux. If this restriction is in place, you will be forced to run Loki with a config that has HTTP and GRPC ports above 1024.

Filesystem

In 1.4.0 and earlier the included config file in the docker container was using directories:

shellCopy

/tmp/loki/index/tmp/loki/chunks

In 1.5.0 this has changed:

shellCopy

/loki/index/loki/chunks

This will mostly affect anyone using docker-compose or docker to run Loki and are specifying a volume to persist storage.

There are two concerns to track here, one is the correct ownership of the files and the other is making sure your mounts updated to the new location.

One possible upgrade path would look like this:

If I were running Loki with this commanddocker run -d --name=loki --mount source=loki-data,target=/tmp/loki -p 3100:3100 grafana/loki:1.4.0

This would mount a docker volume namedloki-data to the/tmp/loki folder which is where Loki will persist theindex andchunks folder in 1.4.0

To move to 1.5.0 I can do the following (note that your container names and paths and volumes etc may be different):

BashCopy

docker stop lokidocker rm lokidocker run --rm --name="loki-perm" -it --mount source=loki-data,target=/mnt ubuntu /bin/bashcd /mntchown -R 10001:10001 ./*exitdocker run -d --name=loki --mount source=loki-data,target=/loki -p 3100:3100 grafana/loki:1.5.0

Notice the change in thetarget=/loki for 1.5.0 to the new data directory location specified in theincluded Loki config file.

The intermediate step of using an ubuntu image to change the ownership of the Loki files to the new user might not be necessary if you can easily access these files to run thechown command directly.That is if you have access to/var/lib/docker/volumes or if you mounted to a different local filesystem directory, you can change the ownership directly without using a container.

Loki Duration Configs

If you get an error like:

nohighlightCopy

 ./loki-linux-amd64-1.5.0 -log.level=debug -config.file=/etc/loki/config.ymlfailed parsing config: /etc/loki/config.yml: not a valid duration string: "0"

This is because of some underlying changes that no longer allow durations without a unit.

Unfortunately the yaml parser doesn’t give a line number but it’s likely to be one of these two:

YAMLCopy

chunk_store_config:  max_look_back_period: 0s # DURATION VALUES MUST HAVE A UNIT EVEN IF THEY ARE ZEROtable_manager:  retention_deletes_enabled: false  retention_period: 0s # DURATION VALUES MUST HAVE A UNIT EVEN IF THEY ARE ZERO

Promtail Config Changes

The underlying backoff library used in Promtail had a config change which wasn’t originally noted in the release notes:

If you get this error:

nohighlightCopy

Unable to parse config: /etc/promtail/promtail.yaml: yaml: unmarshal errors:  line 3: field maxbackoff not found in type util.BackoffConfig  line 4: field maxretries not found in type util.BackoffConfig  line 5: field minbackoff not found in type util.BackoffConfig

The new values are:

YAMLCopy

min_period:max_period:max_retries:

1.4.0

Loki 1.4.0 vendors Cortex v0.7.0-rc.0 which containsseveral breaking config changes.

In thecache_config,defaul_validity has changed todefault_validity.

If you configured your schema via arguments and not a config file, this is no longer supported. This is not something we had ever provided as an option via docs and is unlikely anyone is doing, but worth mentioning.

The other config changes should not be relevant to Loki.

Required Upgrade Path

The newly vendored version of Cortex removes code related to de-normalized tokens in the ring. What you need to know is this:

A “shared ring” as mentioned below refers to usingconsul oretcd for values in the following config:

YAMLCopy

kvstore:  # The backend storage to use for the ring. Supported values are  # consul, etcd, inmemory  store: <string>

• Running without using a shared ring (inmemory): No action required
• Running with a shared ring and upgrading from v1.3.0 -> v1.4.0: No action required
• Running with a shared ring and upgrading from any version less than v1.3.0 (e.g. v1.2.0) -> v1.4.0:ACTION REQUIRED

There are two options for upgrade if you are not on version 1.3.0 and are using a shared ring:

• Upgrade first to v1.3.0BEFORE upgrading to v1.4.0

OR

• If you are running a single binary you only need to add this flag to your single binary command.

1. Add the following configuration to your ingesters command:-ingester.normalise-tokens=true
2. Restart your ingesters with this config
3. Proceed with upgrading to v1.4.0
4. Remove the config option (only do this after everything is running v1.4.0)

It is also possible to enable this flag via config file, see thelifecycler_config configuration option.

If using the Helm Loki chart:

YAMLCopy

extraArgs:  ingester.normalise-tokens: true

If using the Helm Loki-Stack chart:

YAMLCopy

loki:  extraArgs:    ingester.normalise-tokens: true

What will go wrong

If you attempt to add a v1.4.0 ingester to a ring created by Loki v1.2.0 or older which does not have the commandline argument-ingester.normalise-tokens=true (or configured viaconfig file), the v1.4.0 ingester will remove all the entries in the ring for all the other ingesters as it cannot “see” them.

This will result in distributors failing to write and a general ingestion failure for the system.

If this happens to you, you will want to rollback your deployment immediately. You need to remove the v1.4.0 ingester from the ring ASAP, this should allow the existing ingesters to re-insert their tokens. You will also want to remove any v1.4.0 distributors as they will not understand the old ring either and will fail to send traffic.