Single-node version#

VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database.Seecase studies for VictoriaMetrics.

VictoriaMetrics is available inbinary releases,Docker images atDocker HubandQuay,source code.

Documentation for the cluster version of VictoriaMetrics is available onVictoriaMetrics Cluster.

Learn more aboutkey conceptsof VictoriaMetrics and follow thequick start guidefor a better experience.

If you have questions about VictoriaMetrics, then feel free asking them in theVictoriaMetrics community Slack chat,you can join it viaSlack Inviter.

Contact usif you need enterprise support for VictoriaMetrics.Seefeatures available in enterprise package.Enterprise binaries can be downloaded and evaluated for freefromthe releases page.You can also request afree trial license.

VictoriaMetrics is developed at a fast pace, so it is recommended to check theCHANGELOGperiodically,and to performregular upgrades.

VictoriaMetrics Enterpriseprovides long-term support lines of releases (LTS releases) -seethese docs.

VictoriaMetrics has achieved security certifications for Database Software Development and Software-Based Monitoring Services.We apply strict security measures in everything we do. SeeSecurity pagefor more details.

Prominent features#

VictoriaMetrics has the following prominent features:

Seecase studies for VictoriaMetricsandvarious Articles about VictoriaMetrics.

Components#

VictoriaMetrics ecosystem contains the following components additionally tosingle-node VictoriaMetrics:

  • vmagent- lightweight agent for receiving metrics viapull-basedandpush-basedprotocols, transforming and sending them to the configured Prometheus-compatibleremote storage systems such as VictoriaMetrics.
  • vmalert- a service for processing Prometheus-compatible alerting and recording rules.
  • vmalert-tool- a tool for validating alerting and recording rules.
  • vmauth- authorization proxy and load balancer optimized for VictoriaMetrics products.
  • vmgateway- authorization proxy with per-tenantrate limiting capabilities.
  • vmctl- a tool for migrating and copying data between different storage systems for metrics.
  • vmbackup,vmrestoreandvmbackupmanager-tools for creating backups and restoring from backups for VictoriaMetrics data.
  • vminsert,vmselect andvmstorage - components ofVictoriaMetrics cluster.
  • VictoriaLogs- user-friendly cost-efficient database for logs.

Operation#

Install#

To quickly try VictoriaMetrics, just download theVictoriaMetrics executableor docker image fromDocker HuborQuayand start it with the desired command-line flags.See alsoQuickStart guidefor additional information.

VictoriaMetrics can also be installed via these installation methods:

How to start VictoriaMetrics#

The following command-line flags are used the most:

  • -storageDataPath - VictoriaMetrics stores all the data in this directory. The default path isvictoria-metrics-data in the current working directory.
  • -retentionPeriod - retention for stored data. Older data is automatically deleted. Default retention is 1 month (31 days). The minimum retention period is 24h or 1d. Seethese docsfor more details.

Other flags have good enough default values, so set them only if you really need to. Pass-help to seeall the available flags with description and default values.

The following docs may be useful during initial VictoriaMetrics setup:

VictoriaMetrics acceptsPrometheus querying API requestson port8428 by default.

It is recommended setting upmonitoringfor VictoriaMetrics.

Environment variables#

All the VictoriaMetrics components allow referring environment variables inyaml configuration files (such as-promscrape.config)and in command-line flags via%{ENV_VAR} syntax.For example,-metricsAuthKey=%{METRICS_AUTH_KEY} is automatically expanded to-metricsAuthKey=top-secretifMETRICS_AUTH_KEY=top-secret environment variable exists at VictoriaMetrics startup.This expansion is performed by VictoriaMetrics itself.

VictoriaMetrics recursively expands%{ENV_VAR} references in environment variables on startup.For example,FOO=%{BAR} environment variable is expanded toFOO=abc ifBAR=a%{BAZ} andBAZ=bc environment variables exist.

Additionally, all the VictoriaMetrics components allow setting flag values via environment variables according to these rules:

  • The-envflag.enable flag must be set.
  • Each. char in flag name must be substituted with_ (for example-insert.maxQueueDuration <duration> will translate toinsert_maxQueueDuration=<duration>).
  • Repeated flags can be replaced by an environment variable with comma separated values for the repeated flags.For example-storageNode <nodeA> -storageNode <nodeB> command-line flags can be set asstorageNode=<nodeA>,<nodeB> environment variable.
  • Environment var prefix can be set via-envflag.prefix flag. For instance, if-envflag.prefix=VM_, then env vars must be prepended withVM_.

Setting up service#

Readinstructionson how to set up VictoriaMetricsas a service for your OS. See alsoansible playbooks.

Running as Windows service#

In order to run VictoriaMetrics as a Windows service it is required to create a service configuration forWinSWand then install it as a service according to the following guide:

  1. Create a service configuration:

    <service><id>VictoriaMetrics</id><name>VictoriaMetrics</name><description>VictoriaMetrics</description><executable>%BASE%\victoria-metrics-windows-amd64-prod.exe"</executable><onfailureaction="restart"delay="10 sec"/><onfailureaction="restart"delay="20 sec"/><resetfailure>1 hour</resetfailure><arguments>-envflag.enable</arguments><priority>Normal</priority><stoptimeout>15 sec</stoptimeout><stopparentprocessfirst>true</stopparentprocessfirst><startmode>Automatic</startmode><waithint>15 sec</waithint><sleeptime>1 sec</sleeptime><logpath>%BASE%\logs</logpath><logmode="roll"><sizeThreshold>10240</sizeThreshold><keepFiles>8</keepFiles></log><envname="loggerFormat"value="json"/><envname="loggerOutput"value="stderr"/><envname="promscrape_config"value="C:\Program Files\victoria-metrics\promscrape.yml"/></service>

  2. Install WinSW by following thisdocumentation.

  3. Install VictoriaMetrics as a service by running the following from elevated PowerShell:

    winsw install VictoriaMetrics.xmlGet-Service VictoriaMetrics| Start-Service

Seethis issuefor more details.

Start with docker-compose#

Docker-composehelps to spin up VictoriaMetrics,vmagentand Grafana with one command.

Playgrounds#

VictoriaMetrics has the following publicly available demo resources:

  1. https://play.victoriametrics.com/-VMUIof VictoriaMetrics cluster installation.It is available for testing the query engine, relabeling debugger, other tools and pages provided by VMUI.
  2. https://play-grafana.victoriametrics.com/- Grafana configured with manytypical dashboards using VictoriaMetrics and VictoriaLogs as datasource. It contains VictoriaMetrics cluster dashboard with3 cluster installations for the recent OS and LTS versions running under the constant benchmark.
  3. https://play-vmlogs.victoriametrics.com/-VMUIof VictoriaLogs installation.It is available for testing the query engine on demo logs set.
  4. https://play-vtraces.victoriametrics.com/-VMUIof VictoriaTraces installation.It is available for testing the query engine on demo traces set.

Additionally, we provide a docker-compose environment forVictoriaMetrics,VictoriaLogsandVictoriaTraces.They are already configured, provisioned and interconnected. It can be used as an example for aquick start.

How to upgrade VictoriaMetrics#

VictoriaMetrics is developed at a fast pace, so it is recommended periodically checkingthe CHANGELOG pageand performing regular upgrades.

It is safe upgrading VictoriaMetrics to new versions unlessrelease notessay otherwise.It is safe skipping multiple versions during the upgrade unlessrelease notessay otherwise.It is recommended performing regular upgrades to the latest version, since it may contain important bug fixes, performance optimizations or new features.

It is also safe downgrading to older versions unlessrelease notessay otherwise.

The following steps must be performed during the upgrade / downgrade procedure:

  • SendSIGINT signal to VictoriaMetrics process in order to gracefully stop it. Seehow to send signals to processes.
  • Wait until the process stops. This can take a few seconds.
  • Start the upgraded VictoriaMetrics.

Prometheus doesn’t drop data during VictoriaMetrics restart. Seethis articlefor details. The same applies also tovmagent.

vmui#

VictoriaMetrics provides UI for query troubleshooting and exploration. The UI is available athttp://victoriametrics:8428/vmui(or athttp://<vmselect>:8481/select/<accountID>/vmui/ incluster version of VictoriaMetrics).

Note

SeeVMUI at VictoriaMetrics playground.

VMUI provides the following features:

Querying:

Enter the MetricsQL query inQuery field and hitEnter. Multi-line queries can be entered by pressingShift-Enter.

VMUI provides auto-completion forMetricsQLfunctions, metric names, label names and label values.The auto-completion can be enabled by checking theAutocomplete toggle. When the auto-completion is disabled, it canstill be triggered for the current cursor position by pressingctrl+space.

To correlate between multiple queries on the same graph clickAdd Query button and enter an additional query.Results for all the queries are displayed simultaneously on the same graph.

Results of a particular query can be hidden by clicking theeye icon on the right side of the input field.Clicking on theeye icon while holding thectrl key hides results of all other queries.

VMUI automatically adjusts the interval between datapoints on the graph depending on the horizontal resolution and on the selected time range.The step value can be customized by changingStep value in the top-right corner.

Clicking on the line on graph pins the tooltip. User can pin multiple tooltips. Pressx icon to unpin the tooltip.

Query history can be navigated by holdingCtrl (orCmd on MacOS) and pressingup ordown arrows on the keyboard while the cursor is located in the query input field.

VMUI automatically switches from graph view to heatmap view when the query returnshistogrambuckets(bothPrometheus histogramsandVictoriaMetrics histogramsare supported).Try, for example,this query.To disable heatmap view press on settings icon in the top-right corner of graph area and disableHistogram mode toggle.

Time range:

The time range for graphs can be adjusted in multiple ways:

  • Click on time picker in the top-right corner to select a relative (Last N minutes) or absolute time range (specifyFrom andTo);
  • Zoom-in into graph by click-and-drag motion over the graph area;
  • When hovering cursor over the graph area, holdctrl (orcmd on MacOS) and scroll up or down to zoom out or zoom in;
  • When hovering cursor over the graph area, holdctrl (orcmd on MacOS) and drag the graph to the left / right to move the displayed time range into the future / past.

Legend:

Legend is displayed below the graph area.Clicking on item in legend hides all other items from displaying. Clicking on the item while holding thectrl key hidesonly this item.

Clicking on the label-value pair in item automatically copies it into buffer, so it can be pasted later.

There are additional visualization settings in the top right-corner of the legend view: switching to table view,hiding common labels, etc.

Troubleshooting:

When querying thebackfilled dataor duringquery troubleshooting,it may be useful disabling response cache by clickingDisable cache checkbox.

Query can betracedby clicking onTrace query toggle below query input area and executing query again. Once trace is generated, clickon it to expand for more details.

The query and its trace can be exported by clicking ondebug icon in top right corner of trace block. The exported filefile can be loaded again in VMUI onTools=>Query Analyzer page.

Raw query page allows displaying raw, unmodified data. It can be useful for seeing the actual scrape interval or detectingsample duplicates.

Top queries#

VMUIprovidestop queries tab, which can help determining the following query types:

  • the most frequently executed queries;
  • queries with the biggest average execution duration;
  • queries that took the most summary time for execution.

This information is obtained from the/api/v1/status/top_queries HTTP endpoint.

Active queries#

VMUIprovidesactive queries tab, which shows currently execute queries.It provides the following information per each query:

  • The query itself, together with the time range and step args passed to/api/v1/query_range.
  • The duration of the query execution.
  • The client address, who initiated the query execution.

This information is obtained from the/api/v1/status/active_queries HTTP endpoint.

Metrics explorer#

VMUIprovides an ability to explore metrics exported by a particularjob /instance in the following way:

  1. Open thevmui athttp://victoriametrics:8428/vmui/.
  2. Click theExplore Prometheus metrics tab.
  3. Select thejob you want to explore.
  4. Optionally select theinstance for the selected job to explore.
  5. Select metrics you want to explore and compare.

It is possible to change the selected time range for the graphs in the top right corner.

Cardinality explorer#

VictoriaMetrics provides an ability to exploretime series cardinalityatExplore cardinality tab invmui:

  • Metricnames with the highest number of series.
  • Labelswith the highest number of series.
  • Values with the highest number of series for the selected label (akafocusLabel).
  • label=name pairs with the highest number of series.
  • Labels with the highest number of unique values.
  • Read usage statistics of metric names, based onmetric name usage tracker.Shows the number of times the metric name was queried (Requests count), and the last time (Last request) when it was queried.

Note thatcluster version of VictoriaMetricsmay show lower than expected number of unique label values for labels with small number of unique values.This is because ofimplementation limits.

By default, cardinality explorer analyzes time series for thecurrent date. It provides the ability toselect different day at the top right corner.By default, all the time series for the selected date are analyzed. To narrow down the analysis specifyseries selector.

Cardinality explorer is built on top of/api/v1/status/tsdb.

Resources:

Cardinality explorer statistic inaccuracy#

Incluster version of VictoriaMetricseach vmstorage tracks the stored time series individually.vmselect requests stats via/api/v1/status/tsdbAPI from each vmstorage node and merges the results by summing per-series stats.This may lead to inflated values when samples for the same time series are spread across multiple vmstorage nodesdue toreplicationorrerouting.

How to apply new config to VictoriaMetrics#

VictoriaMetrics is configured via command-line flags, so it must be restarted when new command-line flags should be applied:

  • SendSIGINT signal to VictoriaMetrics process in order to gracefully stop it.
  • Wait until the process stops. This can take a few seconds.
  • Start VictoriaMetrics with the new command-line flags.

Prometheus doesn’t drop data during VictoriaMetrics restart. Seethis articlefor details. The same applies also tovmagent.

How to scrape Prometheus exporters such asnode-exporter#

VictoriaMetrics can be used as drop-in replacement for Prometheus for scraping targets configured inprometheus.yml config fileaccording tothe specification.Just set-promscrape.config command-line flag to the path toprometheus.yml config - and VictoriaMetrics should start scraping the configured targets.If the provided configuration file containsunsupported options,then either delete them from the file or just pass-promscrape.config.strictParse=false command-line flag to VictoriaMetrics, so it will ignore unsupported options.

The file pointed by-promscrape.config may contain%{ENV_VAR} placeholders, which are substituted by the correspondingENV_VAR environment variable values.

See also:

VictoriaMetrics also supportsimporting data in Prometheus exposition format.

See alsovmagent, which can be used as drop-in replacement for Prometheus.

Prometheus querying API usage#

VictoriaMetrics supports the following handlers fromPrometheus querying API:

These handlers can be queried from Prometheus-compatible clients such as Grafana or curl.All the Prometheus querying API handlers can be prepended with/prometheus prefix. For example, both/prometheus/api/v1/query and/api/v1/query should work.

Prometheus querying API enhancements#

VictoriaMetrics accepts optionalextra_label=<label_name>=<label_value> query arg, which can be usedfor enforcing additional label filters for queries. For example,/api/v1/query_range?extra_label=user_id=123&extra_label=group_id=456&query=<query>would automatically add{user_id="123",group_id="456"} label filters to the given<query>.This functionality can be used for limiting the scope of time series visible to the given tenant.It is expected that theextra_label query args are automatically set by auth proxy sitting in front of VictoriaMetrics.Seevmauthandvmgatewayas examples of such proxies.

VictoriaMetrics accepts optionalextra_filters[]=series_selector query arg, which can be used for enforcing arbitrary label filters for queries.For example,/api/v1/query_range?extra_filters[]={env=~"prod|staging",user="xyz"}&query=<query> would automaticallyadd{env=~"prod|staging",user="xyz"} label filters to the given<query>. This functionality can be used for limitingthe scope of time series visible to the given tenant. It is expected that theextra_filters[] query args are automaticallyset by auth proxy sitting in front of VictoriaMetrics.Seevmauthandvmgatewayas examples of such proxies.

VictoriaMetrics accepts multiple formats fortime,start andend query args - seethese docs.

VictoriaMetrics acceptsround_digits query arg for/api/v1/queryand/api/v1/query_rangehandlers. It can be used for rounding response valuesto the given number of digits after the decimal point.For example,/api/v1/query?query=avg_over_time(temperature[1h])&round_digits=2 would round response values to up to two digits after the decimal point.

VictoriaMetrics acceptslimit query arg for/api/v1/labelsand/api/v1/label/<labelName>/valueshandlers for limiting the number of returned entries.For example, the query to/api/v1/labels?limit=5 returns a sample of up to 5 unique labels, while ignoring the rest of labels.If the providedlimit value exceeds the corresponding-search.maxTagKeys /-search.maxTagValues command-line flag values,then limits specified in the command-line flags are used.

By default, VictoriaMetrics returns time series for the last day starting at 00:00 UTCfrom/api/v1/series,/api/v1/labelsand/api/v1/label/<labelName>/values,while the Prometheus API defaults to all time. Explicitly setstart andend to select the desired time range.VictoriaMetrics rounds the specifiedstart..end time range to day granularity because of performance optimization concerns.If you need the exact set of label names and label values on the given time range, then send queriesto/api/v1/queryor to/api/v1/query_range.

VictoriaMetrics acceptslimit query arg at/api/v1/seriesfor limiting the number of returned entries. For example, the query to/api/v1/series?limit=5 returns a sample of up to 5 series, while ignoring the rest of series.If the providedlimit value exceeds the corresponding-search.maxSeries command-line flag values, then limits specified in the command-line flags are used.

VictoriaMetrics returns an extra objectstats in JSON response for/api/v1/queryand/api/v1/query_rangeAPIs. This object contains twofields:executionTimeMsec with number of milliseconds the request took andseriesFetched with number of series thatwere fetched from database before filtering. TheseriesFetched field is effectively used by vmalert for detectingmisconfigured rule expressions. Please note,seriesFetchedprovides approximate number of series, it is not recommended to rely on it in tests.

Additionally, VictoriaMetrics provides the following handlers:

  • /vmui - Basic Web UI. Seethese docs.

  • /api/v1/series/count - returns the total number of time series in the database. Some notes:

    • the handler scans allIndexDBsentirely, so it can be slow if the database contains tens of millions of time series;
    • it can return an inflated value if the same time series is stored in more than one IndexDB.
    • the handler may countdeleted time seriesadditionally to normal time series due to internal implementation restrictions;

  • /api/v1/status/active_queries - returns the list of currently running queries. This list is also available atactive queries page at VMUI.

  • /api/v1/status/top_queries - returns the following query lists:

    • the most frequently executed queries -topByCount
    • queries with the biggest average execution duration -topByAvgDuration
    • queries that took the most time for execution -topBySumDuration

    The number of returned queries can be limited viatopN query arg. Old queries can be filtered out withmaxLifetime query arg.For example, request to/api/v1/status/top_queries?topN=5&maxLifetime=30s would return up to 5 queries per list, which were executed during the last 30 seconds.VictoriaMetrics tracks the last-search.queryStats.lastQueriesCount queries with durations at least-search.queryStats.minQueryDuration.

    See alsotop queries page at VMUI.

Timestamp formats#

VictoriaMetrics accepts the following formats fortime,start andend query argsinquery APIsandinexport APIs.

  • Unix timestamps in seconds with optional milliseconds after the point. For example,1562529662.678.
  • Unix timestamps in milliseconds. For example,1562529662678.
  • Unix timestamps in microseconds. For example,1562529662678901.
  • Unix timestamps in nanoseconds. For example,1562529662678901234.
  • RFC3339. For example,2022-03-29T01:02:03Z or2022-03-29T01:02:03+02:30.
  • Partial RFC3339. Examples:2022,2022-03,2022-03-29,2022-03-29T01,2022-03-29T01:02,2022-03-29T01:02:03.The partial RFC3339 time is in local timezone of the host where VictoriaMetrics runs.It is possible to specify the needed timezone by addingZ (UTC),+hh:mm or-hh:mm suffix to partial time.For example,2022-03-01Z corresponds to the given date in UTC timezone, while2022-03-01+06:30 corresponds to2022-03-01 date at06:30 timezone.
  • Relative duration comparing to the current time. For example,1h5m,-1h5m ornow-1h5m meansone hour and five minutes ago, whilenow meansnow.

How to build from sources#

We recommend using eitherbinary releasesordocker images (Docker HubandQuay) instead of building VictoriaMetricsfrom sources. Building from sources is reasonable when developing additional features specificto your needs or when testing bugfixes.

Development build#

  1. Install Go.
  2. Runmake victoria-metrics from the root folder ofthe repository.It buildsvictoria-metrics binary and puts it into thebin folder.

Production build#

  1. Install docker.
  2. Runmake victoria-metrics-prod from the root folder ofthe repository.It buildsvictoria-metrics-prod binary and puts it into thebin folder.

ARM build#

ARM build may run on Raspberry Pi or onenergy-efficient ARM servers.

Development ARM build#

  1. Install Go.
  2. Runmake victoria-metrics-linux-arm ormake victoria-metrics-linux-arm64 from the root folder ofthe repository.It buildsvictoria-metrics-linux-arm orvictoria-metrics-linux-arm64 binary respectively and puts it into thebin folder.

Production ARM build#

  1. Install docker.
  2. Runmake victoria-metrics-linux-arm-prod ormake victoria-metrics-linux-arm64-prod from the root folder ofthe repository.It buildsvictoria-metrics-linux-arm-prod orvictoria-metrics-linux-arm64-prod binary respectively and puts it into thebin folder.

Pure Go build (CGO_ENABLED=0)#

Pure Go mode builds only Go code withoutcgodependencies.

  1. Install Go.
  2. Runmake victoria-metrics-pure from the root folder ofthe repository.It buildsvictoria-metrics-pure binary and puts it into thebin folder.

Building docker images#

Runmake package-victoria-metrics. It buildsvictoriametrics/victoria-metrics:<PKG_TAG> docker image locally.<PKG_TAG> is auto-generated image tag, which depends on source code in the repository.The<PKG_TAG> may be manually set viaPKG_TAG=foobar make package-victoria-metrics.

The base docker image isalpinebut it is possible to use any other base imageby setting it via<ROOT_IMAGE> environment variable.For example, the following command builds the image on top ofscratchimage:

ROOT_IMAGE=scratch make package-victoria-metrics

Building VictoriaMetrics with Podman#

VictoriaMetrics can be built with Podman in either rootful or rootless mode.

When building via rootful Podman, simply addDOCKER=podman to the relevantmake commandline. To buildvia rootless Podman, addDOCKER=podman DOCKER_RUN="podman run --userns=keep-id" to themakecommandline.

For example:make victoria-metrics-pure DOCKER=podman DOCKER_RUN="podman run --userns=keep-id"

Note thatproduction builds are not supported via Podman because Podman does not supportbuildx.

How to work with snapshots#

Create snapshot#

Send a request tohttp://<victoriametrics-addr>:8428/snapshot/create endpoint in order to createaninstant snapshot.The page returns the following JSON response on successful creation of snapshot:

{"status":"ok","snapshot":"<snapshot-name>"}

Snapshots are created under<-storageDataPath>/snapshots directory, where<-storageDataPath>is the corresponding command-line flag value. Snapshots can be archived to backup storage at any timewithvmbackup.

Snapshots consist of a mix of hard-links and soft-links to various files and directories inside-storageDataPath.Seethis articlefor more details. This adds some restrictions on what can be done with the contents of<-storageDataPath>/snapshots directory:

  • Do not delete subdirectories inside<-storageDataPath>/snapshots withrm or similar commands, since this will leave some snapshot data undeleted.Prefer using the/snapshot/delete API for deleting snapshot. See below for more details about this API.
  • Do not copy subdirectories inside<-storageDataPath>/snapshot withcp,rsync or similar commands, since there are high chancesthat these commands won’t copy some data stored in the snapshot. Prefer usingvmbackupfor making copies of snapshot data.

See alsosnapshot troubleshooting.

Thehttp://<victoriametrics-addr>:8428/snapshot/list endpoint returns the list of available snapshots.

Delete snapshot#

Send a query tohttp://<victoriametrics-addr>:8428/snapshot/delete?snapshot=<snapshot-name> in orderto delete the snapshot with<snapshot-name> name.

Navigate tohttp://<victoriametrics-addr>:8428/snapshot/delete_all in order to delete all the snapshots.

How to restore from a snapshot#

  1. Stop VictoriaMetrics withkill -INT.
  2. Restore snapshot contents from backup withvmrestoreto the directory pointed by-storageDataPath.
  3. Start VictoriaMetrics.

Snapshot troubleshooting#

Snapshot doesn’t occupy disk space just after its’ creation thanks to theused approach.Old snapshots may start occupying additional disk space if they refer to old parts, which were already deleted duringbackground merge.That’s why it is recommended deleting old snapshots after they are no longer needed in order to free up disk space used by old snapshots.This can be done either manually or automatically if the-snapshotsMaxAge command-line flag is set. Make sure that the backup process has enough time to completewhen setting-snapshotsMaxAge command-line flag.

VictoriaMetrics exposes the current number of available snapshots viavm_snapshots metric at/metricspage.

How to delete time series#

Send a request tohttp://<victoriametrics-addr>:8428/api/v1/admin/tsdb/delete_series?match[]=<timeseries_selector_for_delete>,where<timeseries_selector_for_delete> may contain anytime series selectorfor metrics to delete. Delete API doesn’t support the deletion of specific time ranges, the series can only be deleted completely.Storage space for the deleted time series isn’t freed instantly - it is freed during subsequentbackground merges of data files.

Note that background merges may never occur for data from previous months, so storage space won’t be freed for historical data.In this caseforced mergemay help freeing up storage space.

It is recommended verifying which metrics will be deleted with the call tohttp://<victoria-metrics-addr>:8428/api/v1/series?match[]=<timeseries_selector_for_delete>before actually deleting the metrics. By default, this query will only scan series in the past 5 minutes, so you may need toadjuststart andend to a suitable range to achieve match hits. Also, if the number of returned time series israther big you will need to set-search.maxDeleteSeries flag (seeResource usage limits).

The/api/v1/admin/tsdb/delete_series handler may be protected withauthKey if-deleteAuthKey command-line flag is set.Note that handler accepts any HTTP method, so sending aGET request to/api/v1/admin/tsdb/delete_series will result in deletion of time series.

The delete API is intended mainly for the following cases:

  • One-off deleting of accidentally written invalid (or undesired) time series.
  • One-off deleting of user data due toGDPR.

Using the delete API is not recommended in the following cases, since it brings a non-zero overhead:

  • Regular cleanups for unneeded data. Just prevent writing unneeded data into VictoriaMetrics.This can be done withrelabeling.Seethis articlefor details.
  • Reducing disk space usage by deleting unneeded time series. This doesn’t work as expected, since the deletedtime series occupy disk space until the next merge operation, which can never occur when deleting too old data.Forced mergemay be used for freeing up disk space occupied by old data.Note that VictoriaMetrics doesn’t delete entries fromIndexDBfor the deleted time series.IndexDB is cleaned up once per the configuredretention.

It’s better to use the-retentionPeriod command-line flag for efficient pruning of old data.

Forced merge#

VictoriaMetrics performsdata compactions in backgroundin order to keep good performance characteristics when accepting new data. These compactions (merges) are performed independently on per-month partitions.This means that compactions are stopped for per-month partitions if no new data is ingested into these partitions.Sometimes it is necessary to trigger compactions for old partitions. For instance, in order to free up disk space occupied bydeleted time series.In this case forced compaction may be initiated on the specified per-month partition by sending request to/internal/force_merge?partition_prefix=YYYY_MM,whereYYYY_MM is per-month partition name. For example,http://victoriametrics:8428/internal/force_merge?partition_prefix=2020_08 would initiate forcedmerge for August 2020 partition. The call to/internal/force_merge returns immediately, while the corresponding forced merge continues running in background.

Forced merges may require additional CPU, disk IO and storage space resources. It is unnecessary to run forced merge under normal conditions,since VictoriaMetrics automatically performsoptimal merges in backgroundwhen new data is ingested into it.

How to export time series#

VictoriaMetrics provides the following handlers for exporting data:

  • /api/v1/export for exporting data in JSON line format. Seethese docsfor details.
  • /api/v1/export/csv for exporting data in CSV. Seethese docsfor details.
  • /api/v1/export/native for exporting data in native binary format. This is the most efficient format for data export.Seethese docsfor details.

How to export data in JSON line format#

Send a request tohttp://<victoriametrics-addr>:8428/api/v1/export?match[]=<timeseries_selector_for_export>,where<timeseries_selector_for_export> may contain anytime series selectorfor metrics to export. Use{__name__!=""} selector for fetching all the time series.

The response would contain all the data for the selected time series in JSON line format - seethese docsfor details on this format.

Each JSON line contains samples for a single time series. An example output:

{"metric":{"__name__":"up","job":"node_exporter","instance":"localhost:9100"},"values":[0,0,0],"timestamps":[1549891472010,1549891487724,1549891503438]}{"metric":{"__name__":"up","job":"prometheus","instance":"localhost:9090"},"values":[1,1,1],"timestamps":[1549891461511,1549891476511,1549891491511]}

Optionalstart andend args may be added to the request in order to limit the time frame for the exported data.Seeallowed formatsfor these args.

For example:

curl http://<victoriametrics-addr>:8428/api/v1/export -d'match[]=<timeseries_selector_for_export>' -d'start=1654543486' -d'end=1654543486'curl http://<victoriametrics-addr>:8428/api/v1/export -d'match[]=<timeseries_selector_for_export>' -d'start=2022-06-06T19:25:48' -d'end=2022-06-06T19:29:07'

Optionalmax_rows_per_line arg may be added to the request for limiting the maximum number of rows exported per each JSON line.Optionalreduce_mem_usage=1 arg may be added to the request for reducing memory usage when exporting big number of time series.In this case the output may contain multiple lines with samples for the same time series.

PassAccept-Encoding: gzip HTTP header in the request to/api/v1/export in order to reduce network bandwidth during exporting big amountsof time series data. This enables gzip compression for the exported data. Example for exporting gzipped data:

curl -H'Accept-Encoding: gzip' http://localhost:8428/api/v1/export -d'match[]={__name__!=""}' > data.jsonl.gz

The maximum duration for each request to/api/v1/export is limited by-search.maxExportDuration command-line flag.

Exported data can be imported via POST’ing it to/api/v1/import.

By default, data exported via/api/v1/export is deduplicated according to-dedup.minScrapeIntervalsetting.Pass GET paramreduce_mem_usage=1 in export request to disable deduplication for recently written data.Afterbackground mergesdeduplication becomes permanent.

How to export CSV data#

Send a request tohttp://<victoriametrics-addr>:8428/api/v1/export/csv?format=<format>&match=<timeseries_selector_for_export>,where:

  • <format> must contain comma-delimited label names for the exported CSV. The following special label names are supported:

    • __name__ - metric name
    • __value__ - sample value
    • __timestamp__:<ts_format> - sample timestamp.<ts_format> can have the following values:
      • unix_s - unix seconds
      • unix_ms - unix milliseconds
      • unix_ns - unix nanoseconds
      • rfc3339 -RFC3339time (in the timezone of the server)
      • custom:<layout> - custom layout for time that is supported bytime.Formatfunction from Go.

  • <timeseries_selector_for_export> may contain anytime series selectorfor metrics to export.

Optionalstart andend args may be added to the request in order to limit the time frame for the exported data.Seeallowed formatsfor these args.

For example:

curl http://<victoriametrics-addr>:8428/api/v1/export/csv -d'format=<format>' -d'match[]=<timeseries_selector_for_export>' -d'start=1654543486' -d'end=1654543486'curl http://<victoriametrics-addr>:8428/api/v1/export/csv -d'format=<format>' -d'match[]=<timeseries_selector_for_export>' -d'start=2022-06-06T19:25:48' -d'end=2022-06-06T19:29:07'

The exported CSV data can be imported to VictoriaMetrics via/api/v1/import/csv.

Thededuplicationis applied for the data exported in CSV by default. It is possible to export raw data without de-duplication by passingreduce_mem_usage=1 query arg to/api/v1/export/csv.

How to export data in native format#

Send a request tohttp://<victoriametrics-addr>:8428/api/v1/export/native?match[]=<timeseries_selector_for_export>,where<timeseries_selector_for_export> may contain anytime series selectorfor metrics to export. Use{__name__=~".*"} selector for fetching all the time series.

On large databases you may experience problems with limit on the number of time series, which can be exported. In this case you need to adjust-search.maxExportSeries command-line flag:

# count unique time series in databasewget -O- -q'http://your_victoriametrics_instance:8428/api/v1/series/count'| jq'.data[0]'# relaunch victoriametrics with search.maxExportSeries more than value from previous command

Optionalstart andend args may be added to the request in order to limit the time frame for the exported data.Seeallowed formatsfor these args.

For example:

curl http://<victoriametrics-addr>:8428/api/v1/export/native -d'match[]=<timeseries_selector_for_export>' -d'start=1654543486' -d'end=1654543486'curl http://<victoriametrics-addr>:8428/api/v1/export/native -d'match[]=<timeseries_selector_for_export>' -d'start=2022-06-06T19:25:48' -d'end=2022-06-06T19:29:07'

The exported data can be imported to VictoriaMetrics via/api/v1/import/native.The native export format may change in incompatible way between VictoriaMetrics releases, so the data exported from the release Xcan fail to be imported into VictoriaMetrics release Y.

Thededuplicationisn’t applied for the data exported in native format. It is expected that the de-duplication is performed during data import.

How to import time series data#

VictoriaMetrics can discover and scrape metrics from Prometheus-compatible targets (aka “pull” protocol) -seethese docs.Additionally, VictoriaMetrics can accept metrics via the following popular data ingestion protocols (aka “push” protocols):

Please note, most of the ingestion APIs (exceptPrometheus remote_write API,OpenTelemetryandInflux Line Protocol)are optimized for performance and processes data in a streaming fashion.It means that client can transfer unlimited amount of data through the open connection. Because of this, import APIsmay not return parsing errors to the client, as it is expected for data stream to be not interrupted.Instead, look for parsing errors on the server side (VictoriaMetrics single-node or vminsert) orcheck for changes invm_rows_invalid_total (exported by server side) metric.

How to import data in JSON line format#

VictoriaMetrics accepts metrics data in JSON line format at/api/v1/import endpoint. Seethese docsfor details on this format.

Example for importing data obtained via/api/v1/export:

# Export the data from <source-victoriametrics>:curl http://source-victoriametrics:8428/api/v1/export -d'match={__name__!=""}' > exported_data.jsonl# Import the data to <destination-victoriametrics>:curl -X POST http://destination-victoriametrics:8428/api/v1/import -T exported_data.jsonl

PassContent-Encoding: gzip HTTP request header to/api/v1/import for importing gzipped data:

# Export gzipped data from <source-victoriametrics>:curl -H'Accept-Encoding: gzip' http://source-victoriametrics:8428/api/v1/export -d'match={__name__!=""}' > exported_data.jsonl.gz# Import gzipped data to <destination-victoriametrics>:curl -X POST -H'Content-Encoding: gzip' http://destination-victoriametrics:8428/api/v1/import -T exported_data.jsonl.gz

Extra labels may be added to all the imported time series by passingextra_label=name=value query args.For example,/api/v1/import?extra_label=foo=bar would add"foo":"bar" label to all the imported time series.

Note that it could be required to flush response cache after importing historical data. Seethese docsfor detail.

VictoriaMetrics parses input JSON lines one-by-one. It loads the whole JSON line in memory, then parses it and then saves the parsed samples into persistent storage.This means that VictoriaMetrics can occupy big amounts of RAM when importing too long JSON lines.The solution is to split too long JSON lines into shorter lines. It is OK if samples for a single time series are split among multiple JSON lines.JSON line length can be limited viamax_rows_per_line query arg when exporting via/api/v1/export.

The maximum JSON line length, which can be parsed by VictoriaMetrics, is limited by-import.maxLineLen command-line flag value.

How to import data in native format#

The specification of VictoriaMetrics’ native format may yet change and is not formally documented yet. So currently we do not recommend that external clients attempt to pack their own metrics in native format file.

If you have a native format file obtained via/api/v1/export/nativehowever this is the most efficient protocol for importing data in.

# Export the data from <source-victoriametrics>:curl http://source-victoriametrics:8428/api/v1/export/native -d'match={__name__!=""}' > exported_data.bin# Import the data to <destination-victoriametrics>:curl -X POST http://destination-victoriametrics:8428/api/v1/import/native -T exported_data.bin

Extra labels may be added to all the imported time series by passingextra_label=name=value query args.For example,/api/v1/import/native?extra_label=foo=bar would add"foo":"bar" label to all the imported time series.

Note that it could be required to flush response cache after importing historical data. Seethese docsfor detail.

How to import CSV data#

Arbitrary CSV data can be imported via/api/v1/import/csv. The CSV data is imported according to the providedformat query arg.Theformat query arg must contain comma-separated list of parsing rules for CSV fields. Each rule consists of three parts delimited by a colon:

<column_pos>:<type>:<context>
  • <column_pos> is the position of the CSV column (field). Column numbering starts from 1. The order of parsing rules may be arbitrary.
  • <type> describes the column type. Supported types are:
    • metric - the corresponding CSV column at<column_pos> contains metric value, which must be integer or floating-point number.The metric name is read from the<context>. CSV line must have at least a single metric field. Multiple metric fields per CSV line is OK.
    • label - the corresponding CSV column at<column_pos> contains label value. The label name is read from the<context>.CSV line may have arbitrary number of label fields. All these labels are attached to all the configured metrics.
    • time - the corresponding CSV column at<column_pos> contains metric time. CSV line may contain either one or zero columns with time.If CSV line has no time, then the current time is used. The time is applied to all the configured metrics.The format of the time is configured via<context>. Supported time formats are:
      • unix_s - unix timestamp in seconds.
      • unix_ms - unix timestamp in milliseconds.
      • unix_ns - unix timestamp in nanoseconds. Note that VictoriaMetrics rounds the timestamp to milliseconds.
      • rfc3339 - timestamp inRFC3339format, i.e.2006-01-02T15:04:05Z.
      • custom:<layout> - custom layout for the timestamp. The<layout> may contain arbitrary time layout according totime.Parse rules in Go.

Each request to/api/v1/import/csv may contain arbitrary number of CSV lines.

Example for importing CSV data via/api/v1/import/csv:

# Import via POST data:curl -d"GOOG,1.23,4.56,NYSE"'http://localhost:8428/api/v1/import/csv?format=2:metric:ask,3:metric:bid,1:label:ticker,4:label:market'curl -d"MSFT,3.21,1.67,NASDAQ"'http://localhost:8428/api/v1/import/csv?format=2:metric:ask,3:metric:bid,1:label:ticker,4:label:market'# Import via file upload:curl -X POST'http://localhost:8428/api/v1/import/csv?format=2:metric:ask,3:metric:bid,1:label:ticker,4:label:market' -T exported_data.csv

After that the data may be read via/api/v1/exportendpoint:

curl -G'http://localhost:8428/api/v1/export' -d'match[]={ticker!=""}'

The following response should be returned:

{"metric":{"__name__":"bid","market":"NASDAQ","ticker":"MSFT"},"values":[1.67],"timestamps":[1583865146520]}{"metric":{"__name__":"bid","market":"NYSE","ticker":"GOOG"},"values":[4.56],"timestamps":[1583865146495]}{"metric":{"__name__":"ask","market":"NASDAQ","ticker":"MSFT"},"values":[3.21],"timestamps":[1583865146520]}{"metric":{"__name__":"ask","market":"NYSE","ticker":"GOOG"},"values":[1.23],"timestamps":[1583865146495]}

Extra labels may be added to all the imported lines by passingextra_label=name=value query args.For example,/api/v1/import/csv?extra_label=foo=bar would add"foo":"bar" label to all the imported lines.

Note that it could be required to flush response cache after importing historical data. Seethese docsfor detail.

How to import data in Prometheus exposition format#

VictoriaMetrics accepts data inPrometheus exposition format,inOpenMetrics formatand inPushgateway formatvia/api/v1/import/prometheus path.

For example, the following command imports a single line in Prometheus exposition format into VictoriaMetrics:

curl -d'foo{bar="baz"} 123' -X POST'http://localhost:8428/api/v1/import/prometheus'

The following command may be used for verifying the imported data:

curl -G'http://localhost:8428/api/v1/export' -d'match={__name__=~"foo"}'

It should return something like the following:

{"metric":{"__name__":"foo","bar":"baz"},"values":[123],"timestamps":[1594370496905]}

The following command imports a single metric viaPushgateway formatwith{job="my_app",instance="host123"} labels:

curl -d'metric{label="abc"} 123' -X POST'http://localhost:8428/api/v1/import/prometheus/metrics/job/my_app/instance/host123'

PassContent-Encoding: gzip HTTP request header to/api/v1/import/prometheus for importing gzipped data:

# Import gzipped data to <destination-victoriametrics>:curl -X POST -H'Content-Encoding: gzip' http://destination-victoriametrics:8428/api/v1/import/prometheus -T prometheus_data.gz

Extra labels may be added to all the imported metrics either viaPushgateway formator by passingextra_label=name=value query args. For example,/api/v1/import/prometheus?extra_label=foo=bar would add{foo="bar"} label to all the imported metrics.

If timestamp is missing in<metric> <value> <timestamp> Prometheus exposition format line, then the current timestamp is used during data ingestion.It can be overridden by passing unix timestamp inmilliseconds viatimestamp query arg. For example,/api/v1/import/prometheus?timestamp=1594370496905.

VictoriaMetrics accepts arbitrary number of lines in a single request to/api/v1/import/prometheus, i.e. it supports data streaming.

Note that it could be required to flush response cache after importing historical data. Seethese docsfor detail.

VictoriaMetrics also may scrape Prometheus targets - seethese docs.

Sending data via OpenTelemetry#

VictoriaMetrics supports data ingestion viaOpenTelemetry protocol for metricsat/opentelemetry/v1/metrics path.

VictoriaMetrics expectsprotobuf-encoded requests at/opentelemetry/v1/metrics.Set HTTP request headerContent-Encoding: gzip when sending gzip-compressed data to/opentelemetry/v1/metrics.

VictoriaMetrics stores the ingested OpenTelemetryraw samplesas is without any transformations.Pass-opentelemetry.usePrometheusNaming command-line flag to VictoriaMetrics for automatic conversion of metric names and labels into Prometheus-compatible format.Pass-opentelemetry.convertMetricNamesToPrometheus command-line flag to VictoriaMetrics for applying Prometheus-compatible format conversion only for metrics names.OpenTelemetryexponential histogramis automatically convertedtoVictoriaMetrics histogram format.

Using the following exporter configuration in the OpenTelemetry collector will allow you to send metrics into VictoriaMetrics:

exporters:otlphttp/victoriametrics:compression:gzipencoding:protoendpoint:http://<collector/vmagent>.<namespace>.svc.cluster.local:<port>/opentelemetry
Note

Note,cluster version of VMexpects specifying tenant ID, i.e.http://<vminsert>:<port>/insert/<accountID>/opentelemetry.See more aboutmultitenancy.

Remember to add the exporter to the desired service pipeline in order to activate the exporter.

service:pipelines:metrics:exporters:-otlphttp/victoriametricsreceivers:-otlp

SeeHow to use OpenTelemetry metrics with VictoriaMetrics.

JSON line format#

VictoriaMetrics accepts data in JSON line format at/api/v1/importand exports data in this format at/api/v1/export.

The format followsJSON streaming concept, e.g. each line contains JSON object with metrics data in the following format:

{// metric contains metric name plus labels for a particular time series"metric":{"__name__":"metric_name",// <- this is metric name// Other labels for the time series"label1":"value1","label2":"value2",..."labelN":"valueN"},// values contains raw sample values for the given time series"values":[1,2.345,-678],// timestamps contains raw sample UNIX timestamps in milliseconds for the given time series// every timestamp is associated with the value at the corresponding position"timestamps":[1549891472010,1549891487724,1549891503438]}

Note that every JSON object must be written in a single line, e.g. all the newline chars must be removed from it./api/v1/importhandler doesn’t accept JSON lines longer than the valuepassed to-import.maxLineLen command-line flag (by default this is 10MB).

It is recommended passing 1K-10K samples per line for achieving the maximum data ingestion performance at/api/v1/import.Too long JSON lines may increase RAM usage at VictoriaMetrics side.

/api/v1/exporthandler acceptsmax_rows_per_line query arg, which allows limiting the number of samples per each exported line.

It is OK to splitraw samplesfor the sametime seriesacross multiple lines.

The number of lines in the request to/api/v1/importcan be arbitrary - they are imported in streaming manner.

Relabeling#

VictoriaMetrics supports Prometheus-compatible relabeling for all the ingested metrics if-relabelConfig command-line flag pointsto a file containing a list ofrelabel_configentries.The-relabelConfig also can point to http or https url. For example,-relabelConfig=https://config-server/relabel_config.yml.

The following docs can be useful in understanding the relabeling:

The-relabelConfig files can contain special placeholders in the form%{ENV_VAR}, which are replaced by the corresponding environment variable values.

Example contents for-relabelConfig file:

# Add {cluster="dev"} label.-target_label:clusterreplacement:dev# Drop the metric (or scrape target) with `{__meta_kubernetes_pod_container_init="true"}` label.-action:dropsource_labels:[__meta_kubernetes_pod_container_init]regex:true

VictoriaMetrics provides additional relabeling features such as Graphite-style relabeling.Seethese docsfor more details.

The relabeling can be debugged athttp://victoriametrics:8428/metric-relabel-debug pageor at ourpublic demo playground.Seethese docsfor more details.

Federation#

VictoriaMetrics exportsPrometheus-compatible federation dataathttp://<victoriametrics-addr>:8428/federate?match[]=<timeseries_selector_for_federation>.

Optionalstart andend args may be added to the request in order to scrape the last point for each selected time series on the[start ... end] interval.Seeallowed formatsfor these args.

For example:

curl http://<victoriametrics-addr>:8428/federate -d'match[]=<timeseries_selector_for_export>' -d'start=1654543486' -d'end=1654543486'curl http://<victoriametrics-addr>:8428/federate -d'match[]=<timeseries_selector_for_export>' -d'start=2022-06-06T19:25:48' -d'end=2022-06-06T19:29:07'

By default, the last point on the interval[now - max_lookback ... now] is scraped for each time series. The default value formax_lookback is5m (5 minutes), but it can be overridden withmax_lookback query arg.For instance,/federate?match[]=up&max_lookback=1h would return last points on the[now - 1h ... now] interval. This may be useful for time series federationwith scrape intervals exceeding5m.

Capacity planning#

VictoriaMetrics uses lower amounts of CPU, RAM and storage space on production workloads compared to competing solutions (Prometheus, Thanos, Cortex, TimescaleDB, InfluxDB, QuestDB, M3DB) according toour case studies.

VictoriaMetrics capacity scales linearly with the available resources. The needed amounts of CPU and RAM highly depends on the workload - the number ofactive time series, serieschurn rate, query types, query qps, etc. It is recommended setting up a test VictoriaMetrics for your production workload and iteratively scaling CPU and RAM resources until it becomes stable according totroubleshooting docs. A single-node VictoriaMetrics works perfectly with the following production workload according toour case studies:

  • Ingestion rate: 1.5+ million samples per second
  • Active time series: 50+ million
  • Total time series: 5+ billion
  • Time series churn rate: 150+ million of new series per day
  • Total number of samples: 10+ trillion
  • Queries: 200+ qps
  • Query latency (99th percentile): 1 second

The needed storage space for the given retention (the retention is set via-retentionPeriod command-line flag) can be extrapolated from disk space usage in a test run. For example, if-storageDataPath directory size becomes 10GB after a day-long test run on a production workload, then it will need at least10GB*100=1TB of disk space for-retentionPeriod=100d (100-days retention period).

It is recommended leaving the following amounts of spare resources:

  • 50% of free RAM for reducing the probability of OOM (out of memory) crashes. Exceeding 50% of free RAM may cause cache evictions, excessive I/O and overall slowdown (see#9895-commentfor more details).
  • 50% of spare CPU for reducing the probability of slowdowns during temporary spikes in workload.
  • At least20% of free storage spaceat the directory pointed by-storageDataPath command-line flag. See also-storage.minFreeDiskSpaceBytes command-lineflag description.

See alsoresource usage limits docs.

Resource usage limits#

By default, VictoriaMetrics is tuned for an optimal resource usage under typical workloads. Some workloads may need fine-grained resource usage limits. In these cases the following command-line flags may be useful:

  • -maxIngestionRate limits samples/second ingested. This may be useful when CPU resources are limited or overloaded.
  • -memory.allowedPercent and-memory.allowedBytes limit the amounts of memory, which may be used for various internal caches at VictoriaMetrics.Note that VictoriaMetrics may use more memory, since these flags don’t limit additional memory, which may be needed on a per-query basis.
  • -search.maxMemoryPerQuery limits the amounts of memory, which can be used for processing a single query. Queries, which need more memory, are rejected.Heavy queries, which select big number of time series, may exceed the per-query memory limit by a small percent. The total memory limitfor concurrently executed queries can be estimated as-search.maxMemoryPerQuery multiplied by-search.maxConcurrentRequests.
  • -search.maxUniqueTimeseries limits the number of unique time series a single query can find and process. By default, VictoriaMetrics calculates the limit automaticallybased on the available memory and the maximum number of concurrent requests it can process (see-search.maxConcurrentRequests). VictoriaMetrics keeps in memorysome metainformation about the time series located by each query and spends some CPU time for processing the found time series.This means that the maximum memory usage and CPU usage a single query can use is proportional to-search.maxUniqueTimeseries.
  • -search.maxQueryDuration limits the duration of a single query. If the query takes longer than the given duration, then it is canceled.This allows saving CPU and RAM when executing unexpected heavy queries.The limit can be overridden to a smaller value by passingtimeout GET parameter.
  • -search.maxConcurrentRequests limits the number of concurrent requests VictoriaMetrics can process. Bigger number of concurrent requests usually meansbigger memory usage. For example, if a single query needs 100 MiB of additional memory during its execution, then 100 concurrent queries may need100 * 100 MiB = 10 GiBof additional memory. So it is better to limit the number of concurrent queries, while pausing additional incoming queries if the concurrency limit is reached.VictoriaMetrics provides-search.maxQueueDuration command-line flag for limiting the max wait time for paused queries. See also-search.maxMemoryPerQuery command-line flag.
  • -search.maxQueueDuration limits the maximum duration queries may wait for execution when-search.maxConcurrentRequests concurrent queries are executed.
  • -search.ignoreExtraFiltersAtLabelsAPI enables ignoring ofmatch[],extra_filters[] andextra_labelquery args at/api/v1/labelsand/api/v1/label/…/values.This may be useful for reducing the load on VictoriaMetrics if the provided extra filters match too many time series.The downside is that the endpoints can return labels and series, which do not match the provided extra filters.
  • -search.maxSamplesPerSeries limits the number of raw samples the query can process per each time series. VictoriaMetrics sequentially processesraw samples per each found time series during the query. It unpacks raw samples on the selected time range per each time series into memoryand then applies the givenrollup function. The-search.maxSamplesPerSeries command-line flagallows limiting memory usage in the case when the query is executed on a time range, which contains hundreds of millions of raw samples per each located time series.
  • -search.maxSamplesPerQuery limits the number of raw samples a single query can process. This allows limiting CPU usage for heavy queries.
  • -search.maxResponseSeries limits the number of time series a single query can return from/api/v1/queryand/api/v1/query_range.
  • -search.maxPointsPerTimeseries limits the number of calculated points, which can be returned per each matching time seriesfromrange query.
  • -search.maxPointsSubqueryPerTimeseries limits the number of calculated points, which can be generated per each matching time seriesduringsubqueryevaluation.
  • -search.maxSeriesPerAggrFunc limits the number of time series, which can be generated byMetricsQL aggregate functionsin a single query.
  • -search.maxSeries limits the number of time series, which may be returned from/api/v1/series.This endpoint is used mostly by Grafana for auto-completion of metric names, label names and label values. Queries to this endpoint may take big amountsof CPU time and memory when the database contains big number of unique time series because ofhigh churn rate.In this case it might be useful to set the-search.maxSeries to quite low value in order limit CPU and memory usage.See also-search.maxLabelsAPIDuration and-search.maxLabelsAPISeries.
  • -search.maxDeleteSeries limits the number of unique time series that can bedeleted by a single/api/v1/admin/tsdb/delete_seriescall. The duration is limited via-search.maxDeleteDuration flagAvailable fromv1.110.0. Deleting too many time series may require bigamount of CPU and memory and this limit guards against unplanned resource usage spikes. Also seeHow to delete time seriessection to learn aboutdifferent ways of deleting series.
  • -search.maxTSDBStatusTopNSeries atvmselect limits the number of unique timeseries that can be queried with topN argument by a single/api/v1/status/tsdb?topN=Ncall.
  • -search.maxTagKeys limits the number of items, which may be returned from/api/v1/labels.This endpoint is used mostly by Grafana for auto-completion of label names. Queries to this endpoint may take big amounts of CPU time and memorywhen the database contains big number of unique time series because ofhigh churn rate.In this case it might be useful to set the-search.maxTagKeys to quite low value in order to limit CPU and memory usage.See also-search.maxLabelsAPIDuration and-search.maxLabelsAPISeries.
  • -search.maxTagValues limits the number of items, which may be returned from/api/v1/label/…/values.This endpoint is used mostly by Grafana for auto-completion of label values. Queries to this endpoint may take big amounts of CPU time and memorywhen the database contains big number of unique time series because ofhigh churn rate.In this case it might be useful to set the-search.maxTagValues to quite low value in order to limit CPU and memory usage.See also-search.maxLabelsAPIDuration and-search.maxLabelsAPISeries.
  • -search.maxLabelsAPISeries limits the number of time series, which can be scanned when performing/api/v1/labelsor/api/v1/label/…/valuesrequests.These endpoints are used mostly by Grafana for auto-completion of label names and label values. Queries to these endpoints may take big amounts of CPU time and memorywhen the database contains big number of unique time series because ofhigh churn rate.In this case it might be useful to set the-search.maxLabelsAPISeries to quite low value in order to limit CPU and memory usage.See also-search.maxLabelsAPIDuration and-search.ignoreExtraFiltersAtLabelsAPI.
  • -search.maxLabelsAPIDuration limits the duration for requests to/api/v1/labels,/api/v1/label/…/valuesor/api/v1/series.The limit can be overridden to a smaller value by passingtimeout GET parameter.These endpoints are used mostly by Grafana for auto-completion of label names and label values. Queries to these endpoints may take big amounts of CPU time and memorywhen the database contains big number of unique time series because ofhigh churn rate.In this case it might be useful to set the-search.maxLabelsAPIDuration to quite low value in order to limit CPU and memory usage.See also-search.maxLabelsAPISeries and-search.ignoreExtraFiltersAtLabelsAPI.
  • -search.maxTagValueSuffixesPerSearch limits the number of entries, which may be returned from/metrics/find endpoint. SeeGraphite Metrics API usage docs.
  • -search.maxFederateSeries limits maximum number of time series, which can be returned via/federate API.The duration of the/federate queries is limited via-search.maxQueryDuration flag. This option allows limiting memory usage.
  • -search.maxExportSeries limits maximum number of time series, which can be returned from/api/v1/export* APIs.The duration of the export queries is limited via-search.maxExportDuration flag. This option allows limiting memory usage.
  • -search.maxTSDBStatusSeries limits maximum number of time series, which can be processed during the call to/api/v1/status/tsdb.The duration of the status queries is limited via-search.maxStatusRequestDuration flag. This option allows limiting memory usage.

See alsoresource usage limits at VictoriaMetrics cluster,cardinality limiterandcapacity planning docs.

High availability#

The general approach for achieving high availability is the following:

  • To run two identically configured VictoriaMetrics instances in distinct datacenters (availability zones);
  • To store the collected data simultaneously into these instances viavmagentor Prometheus.
  • To query the first VictoriaMetrics instance and to fail over to the second instance when the first instance becomes temporarily unavailable.This can be done viavmauthaccording tothese docs.

Such a setup guarantees that the collected data isn’t lost when one of VictoriaMetrics instance becomes unavailable.The collected data continues to be written to the available VictoriaMetrics instance, so it should be available for querying.Bothvmagentand Prometheus buffer the collected data locally if they cannot send itto the configured remote storage. So the collected data will be written to the temporarily unavailable VictoriaMetrics instanceafter it becomes available.

If you usevmagentfor storing the data into VictoriaMetrics,then it can be configured with multiple-remoteWrite.url command-line flags, where every flag points to the VictoriaMetricsinstance in a particular availability zone, in order to replicate the collected data to all the VictoriaMetrics instances.For example, the following command instructsvmagent to replicate data tovm-az1 andvm-az2 instances of VictoriaMetrics:

/path/to/vmagent\  -remoteWrite.url=http://<vm-az1>:8428/api/v1/write\  -remoteWrite.url=http://<vm-az2>:8428/api/v1/write

If you use Prometheus for collecting and writing the data to VictoriaMetrics,then the followingremote_writesectionin Prometheus config can be used for replicating the collected data tovm-az1 andvm-az2 VictoriaMetrics instances:

remote_write:-url:http://<vm-az1>:8428/api/v1/write-url:http://<vm-az2>:8428/api/v1/write

It is recommended to usevmagentinstead of Prometheus for highly loaded setups,since it uses lower amounts of RAM, CPU and network bandwidth than Prometheus.

If you use identically configuredvmagentinstances for collecting the same dataand sending it to VictoriaMetrics, then do not forget enablingdeduplicationat VictoriaMetrics side.

Seevictoria-metrics-distributed chartfor an example.

Deduplication#

VictoriaMetrics leaves a singleraw samplewith the biggesttimestampfor eachtime seriesper each-dedup.minScrapeInterval discrete interval if-dedup.minScrapeInterval is set to positive duration.For example,-dedup.minScrapeInterval=60s would leave a single raw sample with the biggest timestamp per each discrete60s interval.This aligns with thestaleness rules in Prometheus.

If multiple raw samples havethe same timestamp on the given-dedup.minScrapeInterval discrete interval,then the sample withthe biggest value is kept.Stale markersare preferred over any other value.

Prometheus staleness markersare processed as any other value during de-duplication.If raw sample with the biggest timestamp on-dedup.minScrapeInterval contains a stale marker, then it is kept after the deduplication.This allows properly preserving staleness markers during the de-duplication.

Please note,labelsof raw samples should be identicalin order to be deduplicated. For example, this is whyHA pair of vmagentsneeds to be identically configured.

The-dedup.minScrapeInterval=D is equivalent to-downsampling.period=0s:D indownsampling.It is also safe to use deduplication and downsampling simultaneously.

The recommended value for-dedup.minScrapeInterval must equal toscrape_interval config from Prometheus configs.It is recommended to have a singlescrape_interval across all the scrape targets.Seethis articlefor details.

The de-duplication reduces disk space usage if multipleidentically configuredvmagentor Prometheus instances in HA pair write data to the same VictoriaMetrics instance.These vmagent or Prometheus instances must haveidenticalexternal_labels section in their configs,so they write data to the same time series.See alsohow to set up multiple vmagent instances for scraping the same targets.Note that de-duplication doesn’t reduce theindexdbsize -seewhy IndexDB size is so large?.

It is recommended passing different-promscrape.cluster.name values to each distinct HA pair ofvmagent instances,so the de-duplication consistently leaves samples for onevmagent instance and removes duplicate samplesfrom othervmagent instances.Seethese docsfor details.

VictoriaMetrics stores all the ingested samples to disk even if-dedup.minScrapeInterval command-line flag is set.The ingested samples are de-duplicated duringbackground mergesand during query execution.VictoriaMetrics also supports de-duplication during data ingestion before the data is stored to disk, via-streamAggr.dedupInterval command-line flag -seethese docs.

Metrics Metadata#

Single-node VictoriaMetrics can store metric metadata (TYPE, HELP, UNIT)Available fromv1.130.0.Metadata ingestion and querying are disabled by default. To enable them, set-enableMetadata=true.

The metadata is stored in memory and can use up to 1% of available memory by default. The size could be adjusted by-storage.maxMetadataStorageSize flag.Please note that metadata is lost after restarts. It is ingested independently from metrics, so a metric may exist without metadata, and vice versa.

Metadata can be queried via the/api/v1/metadata endpoint, which provides a response compatible with the Prometheusmetadata API.See/api/v1/metadataexample.

Storage#

VictoriaMetrics buffers the ingested data in memory for up to a second. Then the buffered data is written to in-memoryparts,which can be searched during queries. The in-memoryparts are periodically persisted to disk, so they could survive unclean shutdownsuch as out of memory crash, hardware power loss orSIGKILL signal. The interval for flushing the in-memory data to diskcan be configured with the-inmemoryDataFlushInterval command-line flag (note that too short flush interval may significantly increase disk IO).

In-memory parts are persisted to disk intopart directories under the<-storageDataPath>/data/small/YYYY_MM/ folder,whereYYYY_MM is the month partition for the stored data. For example,2022_11 is the partition forpartswithraw samplesfromNovember 2022.Each partition directory containsparts.json file with the actual list of parts in the partition.

Everypart directory containsmetadata.json file with the following fields:

  • RowsCount - the number ofraw samplesstored in the part
  • BlocksCount - the number of blocks stored in the part (see details about blocks below)
  • MinTimestamp andMaxTimestamp - minimum and maximum timestamps across raw samples stored in the part
  • MinDedupInterval - thededuplication intervalapplied to the given part.

Eachpart consists ofblocks sorted by internal time series id (akaTSID).Eachblock contains up to 8Kraw samples,which belong to a singletime series.Raw samples in each block are sorted bytimestamp. Blocks for the same time series are sortedby thetimestamp of the first sample. Timestamps and values for all the blocksare stored incompressed formin separate files underpart directory -timestamps.bin andvalues.bin.

Thepart directory also containsindex.bin andmetaindex.bin files - these files contain indexfor fast block lookups, which belong to the givenTSID and cover the given time range.

Parts are periodically merged into bigger parts in background. The background merge provides the following benefits:

  • keeping the number of data files under control, so they don’t exceed limits on open files
  • improved data compression, since bigger parts are usually compressed better than smaller parts
  • improved query speed, since queries over smaller number of parts are executed faster
  • various background maintenance tasks such asde-duplication,downsamplingandfreeing up disk space for the deleted time seriesare performed during the merge

Newly addedparts either successfully appear in the storage or fail to appear.The newly addedpart is atomically registered in theparts.json file under the corresponding partitionafter it is fully written andfsyncedto the storage.Thanks to this algorithm, storage never contains partially created parts, even if hardware power offoccurs in the middle of writing thepart to disk - such incompletely writtenpartsare automatically deleted on the next VictoriaMetrics start.

The same applies to merge process —parts are either fully merged into a newpart or fail to merge,leaving the sourceparts untouched. However, due to hardware issues data on disk may be corrupted regardless ofVictoriaMetrics process. VictoriaMetrics can detect corruption during decompressing, decoding or sanity checkingof the data blocks. Butit cannot fix the corrupted data. Data parts that fail to load on startup need to be deletedor restored from backups. This is why it is recommended performingregular backups.

VictoriaMetrics doesn’t use checksums for stored data blocks. See why in thisGitHub Issue.

VictoriaMetrics doesn’t merge parts if their summary size exceeds free disk space.This prevents from potential out of disk space errors during merge.The number of parts may significantly increase over time under free disk space shortage.This increases overhead during data querying, since VictoriaMetrics needs to read data frombigger number of parts per each request. That’s why it is recommended to have at least 20%of free disk space under directory pointed by-storageDataPath command-line flag.

Information about merging process is available inthe dashboard for single-node VictoriaMetricsandthe dashboard for VictoriaMetrics cluster.See more details inmonitoring docs.

Seethis articlefor more details.

See alsohow to work with snapshotsandIndexDB.

IndexDB#

VictoriaMetrics identifiestime seriesbyTSID (time series ID) and storesraw samplessortedby TSID (seeStorage). Thus, the TSID is a primary index and couldbe used for searching and retrieving raw samples. However, the TSID is neverexposed to the clients, i.e. it is for internal use only.

Instead, VictoriaMetrics maintains aninverted index that enables searchingthe raw samples by metric name, label name, and label value by mapping thesevalues to the corresponding TSIDs.

VictoriaMetrics uses two types of inverted indexes:

  • Global index. Searches using this index is performed across the entireretention period.
  • Per-day index. This index stores mappings similar to ones in global indexbut also includes the date in each mapping. This speeds up data retrievalfor queries within a shorter time range (which is often just the last day).

When the search query is executed, VictoriaMetrics decides which index to usebased on the time range of the query:

  • Per-day index is used if the search time range is 40 days or less.
  • Global index is used for search queries with a time range greater than 40days.

Mappings are added to the indexes during the data ingestion:

  • In global index each mapping is created only once per retention period.
  • In the per-day index each mapping is created for each unique date thathas been seen in the samples for the corresponding time series.

IndexDB respectsretention periodand once it is over, the indexesare dropped. For the new retention period, the indexes are gradually populatedagain as the new samples arrive.

See alsoWhy IndexDB size is so large?.

Index tuning for low churn rate#

By default, VictoriaMetrics uses the following indexes for data retrieval:global andper-day.Both store the same data and on query time VictoriaMetrics can choose between indexes for optimal performance.SeeIndexDBfor details.

If your use case involveshigh cardinalitywithhigh churn ratethen this default setting should be ideal for you.

A prominent example is Kubernetes. Services in k8s expose big number of series with short lifetime, significantlyincreasing churn rate. The per-day index speeds up data retrieval in this case.

But if your use case assumes low or no churn rate, then you might benefit from disabling the per-day index by settingthe flag-disablePerDayIndexAvailable fromv1.112.0. This will improve the time series ingestion speed and decrease disk space usage,since no time or disk space is spent maintaining the per-day index.

Example use cases:

  • Historical weather data, such asERA5.It consists of millions time series whose hourly values span tens of years. The time series set never changes.If the per-day index is disabled, once the first hour of data is ingested the entire time series set will be writteninto the global index and subsequent portions of data will not result in index update. But if the per-day indexis enabled, the same set of time-series will be written to the per-day index for every day of data.

  • IoT: a huge set of sensors exports time series with the sensor ID used as a metric label value. Since sensor additionsor removals happen infrequently, the time series churn rate will be low. With the per-day index disabled, the entiretime series set will be registered in global index during the initial data ingestion and the global index will receivesmall updates when a sensor is added or removed.

What to expect:

  • Prefer setting this flag on fresh installations.
  • Disabling per-day index on installations with historical data is Ok.
  • Re-enabling per-day index on installations with historical data will make it unsearchable.

See alsoWhy IndexDB size is so large?.

Retention#

Retention is configured with the-retentionPeriod command-line flag, which takes a number followed by a time unitcharacter -h(ours),d(ays),w(eeks),y(ears). If the time unit is not specified, a month (31 days) is assumed.For instance,-retentionPeriod=3 means that the data will be stored for 3 months (93 days) and then deleted.The default retention period is one month. Theminimum retention period is 24h or 1d.

Data is split in per-month partitions inside<-storageDataPath>/data/{small,big} folders.Data partitions outside the configured retention are deletedon the first day of the new month.Each partition consists of one or moredata parts. Data parts outside the configured retentionareeventually deleted duringbackground merge.The time range covered by data part isnot limited by retention period unit. One data part can cover hours or days ofdata. Hence, a data part can be deleted onlywhen fully outside the configured retention.See more about partitions and parts in theStorage section.

The maximum disk space usage for a given-retentionPeriod is going to be (-retentionPeriod + 1) months.For example, if-retentionPeriod is set to 1, data for January is deleted on March 1st.

It is safe to extend-retentionPeriod on existing data. If-retentionPeriod is set to a lowervalue than before, then data outside the configured period will be eventually deleted.

VictoriaMetrics does not support indefinite retention, but you can specify an arbitrarily high duration, e.g.-retentionPeriod=100y.

Multiple retentions#

Distinct retentions for distinct time series can be configured viaretention filtersinVictoriaMetrics Enterprise.

Community version of VictoriaMetrics supports only a single retention, which can be configured via-retentionPeriodcommand-line flag.If you need multiple retentions in community version of VictoriaMetrics, then you may start multiple VictoriaMetrics instances with distinct values for the following flags:

  • -retentionPeriod
  • -storageDataPath, so the data for each retention period is saved in a separate directory
  • -httpListenAddr, so clients may reach VictoriaMetrics instance with proper retention

Then set upvmauthin front of VictoriaMetrics instances,so it could route requests from particular user to VictoriaMetrics with the desired retention.

Similar scheme can be applied for multiple tenants inVictoriaMetrics cluster.Seethese docsfor multi-retention setup details.

Retention filters#

Enterprise version of VictoriaMetricssupportsretention filters,which allow configuring multiple retentions for distinct sets of time series matching the configuredseries filtersvia-retentionFilter command-line flag. This flag acceptsfilter:duration options, wherefilter must bea validseries filter, while thedurationmust contain validretentionfor time series matching the givenfilter.Theduration of the-retentionFilter must be lower or equal to-retentionPeriodflag value.If series doesn’t match any configured-retentionFilter, then the retention configured via-retentionPeriodcommand-line flag is applied to it. If series matches multiple configured retention filters, then the smallest retention is applied.

For example, the following config sets 3 days retention for time series withteam="juniors" label,30 days retention for time series withenv="dev" orenv="staging" label and 1 year retention for the remaining time series:

-retentionFilter='{team="juniors"}:3d' -retentionFilter='{env=~"dev|staging"}:30d' -retentionPeriod=1y

There are two gauge metrics to monitor the retention filters process:

  • vm_retention_filters_partitions_scheduled shows the total number of partitions scheduled for retention filters
  • vm_retention_filters_partitions_scheduled_size_bytes shows the total size of scheduled partitions.

Additionally, a log message with the filter expression and the partition name is written to the log on the start and completion of the operation.

Important notes:

  • The data outside the configured retention isn’t deleted instantly - it is deleted eventually duringbackground merges.
  • The-retentionFilter doesn’t remove old data fromIndexDBuntil the configured-retentionPeriod.So the IndexDB size can grow big underhigh churn rateeven for small retentions configured via-retentionFilter.

Retention filters configuration can be tested in enterprise version of vmui on the pageTools.Retention filters debug.It is safe updating-retentionFilter during VictoriaMetrics restarts - the updated retention filters are applied eventuallyto historical data.

It’s expected that resource usage will temporarily increase when-retentionFilter is applied.This is because additional operations are required to read the data, filter and apply retention to partitions,which will cost extra CPU and memory.

Seehow to configure multiple retentions in VictoriaMetrics cluster.

See alsodownsampling.

Retention filters can be evaluated for free by downloading and using enterprise binaries fromthe releases page.See how to request afree trial license.

Downsampling#

VictoriaMetrics Enterprisesupports multi-level downsampling via-downsampling.period=offset:interval command-line flag.This command-line flag instructs leaving the last sample per eachinterval fortime seriessamplesolder than theoffset. Theoffset must be a multiple ofinterval. For example,-downsampling.period=30d:5m instructs leaving the last sampleper each 5-minute interval for samples older than 30 days, while the rest of samples are dropped.

The-downsampling.period command-line flag can be specified multiple times in order to apply different downsampling levels for different time ranges (aka multi-level downsampling).For example,-downsampling.period=30d:5m,180d:1h instructs leaving the last sample per each 5-minute interval for samples older than 30 days,while leaving the last sample per each 1-hour interval for samples older than 180 days.

VictoriaMetrics supportsAvailable fromv1.100.0configuring independent downsampling per different sets oftime seriesvia-downsampling.period=filter:offset:interval syntax. In this case the givenoffset:interval downsampling is applied only to time series matching the givenfilter.Thefilter can contain arbitraryseries filter.For example,-downsampling.period='{__name__=~"(node|process)_.*"}:1d:1m instructs VictoriaMetrics to downsample samples older than one day with one minute intervalonly fortime serieswith names starting withnode_ orprocess_ prefixes.The downsampling for other time series can be configured independently via additional-downsampling.period command-line flags.Downsampling configuration can be tested in enterprise version of vmui on the pageTools.Downsampling filters debug.

If the time series doesn’t match anyfilter, then it isn’t downsampled. If the time series matches multiple filters, then the downsamplingfor the first matchingfilter is applied. For example,-downsampling.period='{env="prod"}:1d:30s,{__name__=~"node_.*"}:1d:5m' de-duplicatessamples older than one day with 30 seconds interval across all the time series withenv="prod"label,even if their names start withnode_ prefix. All the other time series with names starting withnode_ prefix are de-duplicated with 5 minutes interval.

If downsampling shouldn’t be applied to some time series matching the givenfilter, then pass-downsampling.period=filter:0s:0s command-line flag to VictoriaMetrics.For example, if series withenv="prod" label shouldn’t be downsampled, then pass-downsampling.period='{env="prod"}:0s:0s' command-line flag in front of other-downsampling.period flags.But-downsampling.period=0s:interval or-downsampling.period=filter:0s:0s cannot be used withdeduplicationsimultaneously as they could conflict.

Downsampling is applied independently per each time series and leaves a singleraw samplewith the biggesttimestampon the configured interval, in the same way asdeduplicationdoes.It works the best forcountersandhistograms,as their values are always increasing. Downsamplinggaugesandsummarieslose some changes within the downsampling interval,since only the last sample on the given interval is left and the rest of samples are dropped.

You can userecording rulesorstreaming aggregationto apply custom aggregation functions, like min/max/avg etc., in order to make gauges more resilient to downsampling.

Downsampling can reduce disk space usage and improve query performance if it is applied to time series with big numberof samples per each series. The downsampling doesn’t improve query performance and doesn’t reduce disk space if the database contains big numberof time series with small number of samples per each series, since downsampling doesn’t reduce the number of time series.So there is little sense in applying downsampling to time series withhigh churn rate.In this case the majority of query time is spent on searching for the matching time series instead of processing the found samples.SeeWhy IndexDB size is so large?.

Downsampling is performed duringbackground merges.It cannot be performed if there is not enough of free disk space or if vmstorage is inread-only mode.

It’s expected that resource usage will temporarily increase whendownsampling with filters is applied.This is because additional operations are required to read historical data, downsample, and persist it back,which will cost extra CPU and memory.

Please, note that intervals of-downsampling.period for a single filter must be multiples of each other.In casededuplicationis enabled, value of-dedup.minScrapeInterval command-line flag must alsobe multiple of-downsampling.period intervals. This is required to ensure consistency of deduplication and downsampling results.

It is safe updating-downsampling.period during VictoriaMetrics restarts - the updated downsampling configuration will beapplied eventually to historical data duringbackground merges.

Seehow to configure downsampling in VictoriaMetrics cluster.

See alsoretention filters.

The downsampling can be evaluated for free by downloading and using enterprise binaries fromthe releases page.See how to request afree trial license.

Multi-tenancy#

Single-node VictoriaMetrics doesn’t support multi-tenancy. Use thecluster versioninstead.

Scalability and cluster version#

Though single-node VictoriaMetrics cannot scale to multiple nodes, it is optimized for resource usage - storage size / bandwidth / IOPS, RAM, CPU.This means that a single-node VictoriaMetrics may scale vertically and substitute a moderately sized cluster built with competing solutionssuch as Thanos, Uber M3, InfluxDB or TimescaleDB. Seevertical scalability benchmarks.

So try single-node VictoriaMetrics at first and thenswitch to the cluster versionif you still needhorizontally scalable long-term remote storage for really large Prometheus deployments.Contact usfor enterprise support.

Alerting#

It is recommended usingvmalertfor alerting.

Additionally, alerting can be set up with the following tools:

Security#

General security recommendations:

  • All the VictoriaMetrics components must run in protected private networks without direct access from untrusted networks such as Internet.The exception isvmauthandvmgateway,which are intended for serving public requests and performing authorization withTLS termination.
  • All the requests from untrusted networks to VictoriaMetrics components must go through auth proxy such asvmauthorvmgateway. The proxy must be set up with proper authentication and authorization.
  • Prefer using lists of allowed API endpoints, while disallowing access to other endpoints when configuringvmauthin front of VictoriaMetrics components.
  • Set reasonableStrict-Transport-Securityheader value to all the components to mitigateMitM attacks, for example:max-age=31536000; includeSubDomains. See-http.header.hsts flag.
  • Set reasonableContent-Security-Policyheader value to mitigateXSS attacks. See-http.header.csp flag.
  • Set reasonableX-Frame-Optionsheader value to mitigateclickjacking attacks, for exampleDENY. See-http.header.frameOptions flag.

VictoriaMetrics provides the following security-related command-line flags:

  • -tls,-tlsCertFile and-tlsKeyFile for switching from HTTP to HTTPS at-httpListenAddr (TCP port 8428 is listened by default).Enterprise version of VictoriaMetricssupports automatic issuing of TLS certificates.Seethese docs.
  • -mtls and-mtlsCAFile for enablingmTLSfor requests to-httpListenAddr. Seethese docs.
  • -httpAuth.username and-httpAuth.password for protecting all the HTTP endpointswithHTTP Basic Authentication.
  • -deleteAuthKey for protecting/api/v1/admin/tsdb/delete_series endpoint. Seehow to delete time series.
  • -snapshotAuthKey for protecting/snapshot* endpoints. Seehow to work with snapshots.
  • -forceFlushAuthKey for protecting/internal/force_flush endpoint. Seethese docs.
  • -forceMergeAuthKey for protecting/internal/force_merge endpoint. Seeforce merge docs.
  • -search.resetCacheAuthKey for protecting/internal/resetRollupResultCache endpoint. Seebackfillingfor more details.
  • -reloadAuthKey for protecting/-/reload endpoint, which is used for force reloading of-promscrape.config.
  • -configAuthKey for protecting/config endpoint, since it may contain sensitive information such as passwords.
  • -flagsAuthKey for protecting/flags endpoint.
  • -pprofAuthKey for protecting/debug/pprof/* endpoints, which can be used forprofiling.
  • -metricNamesStatsResetAuthKey for protecting/api/v1/admin/status/metric_names_stats/reset endpoint, used forMetric Names Tracker.
  • -denyQueryTracing for disallowingquery tracing.
  • -http.header.hsts,-http.header.csp, and-http.header.frameOptions for servingStrict-Transport-Security,Content-Security-PolicyandX-Frame-Options HTTP response headers.

Explicitly set internal network interface for TCP and UDP ports for data ingestion with Graphite and OpenTSDB formats.For example, substitute-graphiteListenAddr=:2003 with-graphiteListenAddr=<internal_iface_ip>:2003. This protects from unexpected requests from untrusted network interfaces.

See alsosecurity recommendation for VictoriaMetrics clusterandthe general security page at VictoriaMetrics website.

CVE handling policy#

Source code: Go dependencies are scanned bygovulncheckin CI.All vulnerabilities must be fixed before next scheduled release and backported toLTS releases.

Docker images: CVE findings inAlpinebase image pose minimal risk since VictoriaMetrics binaries are statically compiled with no OS dependencies.When detected, only the Alpine base tag is updated.Releases proceed as planned even if upstream fixes are not yet available.For maximum security, hardenedscratch-based images are also provided.All images are continuously scanned by Docker Hub and verified before release usinggrype.

mTLS protection#

By defaultVictoriaMetrics accepts http requests at8428 port (this port can be changed via-httpListenAddr command-line flags).Enterprise version of VictoriaMetricssupports the ability to acceptmTLSrequests at this port, by specifying-tls and-mtls command-line flags. For example, the following command runsVictoriaMetrics, which accepts only mTLS requests at port8428:

./victoria-metrics -tls -mtls

By default, system-wideTLS Root CAis used for verifying client certificates if-mtls command-line flag is specified.It is possible to specify custom TLS Root CA via-mtlsCAFile command-line flag.

Automatic issuing of TLS certificates#

All the VictoriaMetricsEnterprisecomponents support automatic issuing of TLS certificates for public HTTPS server running at-httpListenAddrviaLet’s Encrypt service. The following command-line flags must be set in order to enable automatic issuing of TLS certificates:

  • -httpListenAddr must be set for listening TCP port443. For example,-httpListenAddr=:443. This port must be accessible by theLet’s Encrypt service.
  • -tls must be set in order to accept HTTPS requests at-httpListenAddr. Note that-tlcCertFile and-tlsKeyFile aren’t needed when automatic TLS certificate issuing is enabled.
  • -tlsAutocertHosts must be set to comma-separated list of hosts, which can be reached via-httpListenAddr. TLS certificates are automatically issued for these hosts.
  • -tlsAutocertEmail must be set to contact email for the issued TLS certificates.
  • -tlsAutocertCacheDir may be set to the directory path for persisting the issued TLS certificates between VictoriaMetrics restarts. If this flag isn’t set,then TLS certificates are re-issued on every restart.

This functionality can be evaluated for free according tothese docs.

See alsosecurity recommendations.

Tuning#

  • No need in tuning for VictoriaMetrics - it uses reasonable defaults for command-line flags,which are automatically adjusted for the available CPU and RAM resources.
  • No need in tuning for Operating System - VictoriaMetrics is optimized for default OS settings.The only option is increasing the limit onthe number of open files in the OS.The recommendation is not specific for VictoriaMetrics only but also for any service which handles many HTTP connections and stores data on disk.
  • VictoriaMetrics is a write-heavy application and its performance depends on disk performance. So be careful with otherapplications or utilities (likefstrim)which couldexhaust disk resources.
  • The recommended filesystem isext4, the recommended persistent storage ispersistent HDD-based disk on GCP,since it is protected from hardware failures via internal replication and it can beresized on the fly.If you plan to store more than 1TB of data onext4 partition, then the following options are recommended to pass tomkfs.ext4:
mkfs.ext4 ... -O 64bit,huge_file,extent -T huge

Monitoring#

VictoriaMetrics exports internal metrics in Prometheus exposition format at/metrics page.These metricscan be scrapedviavmagentor any other Prometheus-compatible scraper.

Note

Single-node VictoriaMetrics can self-scrape its metrics when-selfScrapeInterval command-line flag isset to duration greater than 0. For example,-selfScrapeInterval=10s scrapes/metrics page every 10 seconds.

See the list of officialGrafana dashboards for VictoriaMetrics components.

Please follow the monitoring recommendations below:

  • Prefer giving distinct scrape job names per each component type. I.e.vmagent andvmalert should have corresponding job names.
  • Never use load balancer address for scraping metrics. All the monitored components should be scraped directly by their address.
  • Set uprecommended alertsviavmalertor via Prometheus.
  • See currently running queries and their execution times atactive queries page.
  • See queries that take the most time to execute attop queries page.

See alsoVictoriaMetrics Monitoringandtroubleshooting docs.

Note

VictoriaMetrics components do not expose metadataTYPE andHELP fields on/metrics page.Services like Google Cloud Managed Prometheus could require metadata to be present for scraping. In this case, pass-metrics.exposeMetadatacommand-line to them. Seethese docsfor details.

TSDB stats#

VictoriaMetrics returns TSDB stats at/api/v1/status/tsdb page in the way similar to Prometheus - seethese Prometheus docs. VictoriaMetrics accepts the following optional query args at/api/v1/status/tsdb page:

  • topN=N whereN is the number of top entries to return in the response. By default, top 10 entries are returned.
  • date=YYYY-MM-DD whereYYYY-MM-DD is the date for collecting the stats. By default, the stats is collected for the current day. Passdate=1970-01-01 in order to collect global stats across all the days.
  • focusLabel=LABEL_NAME returns label values with the highest number of time series for the givenLABEL_NAME in theseriesCountByFocusLabelValue list.
  • match[]=SELECTOR whereSELECTOR is an arbitrarytime series selectorfor series to take into account during stats calculation. By default all the series are taken into account.
  • extra_label=LABEL=VALUE. Seethese docsfor more details.

Incluster version of VictoriaMetricseach vmstorage tracks the stored time series individually.vmselect requests stats via/api/v1/status/tsdbAPI from each vmstorage node and merges the results by summing per-series stats.This may lead to inflated values when samples for the same time series are spread across multiple vmstorage nodesdue toreplicationorrerouting.

VictoriaMetrics provides UI on top of/api/v1/status/tsdb - seecardinality explorer docs.

VictoriaMetrics enhances Prometheus stats withrequestsCount andlastRequestTimestamp forseriesCountByMetricName. This stats added iftracking metric names statsis configured.

Track ingested metrics usage#

VictoriaMetrics can track statistics of fetchedmetric namesduringqueryingAvailable fromv1.113.0. It tracksonly metric names, as the number of names is usually limited (thousands) compared to time series (millions or billions).This feature can be disabled via the flag--storage.trackMetricNamesStats=false (enabled by default) on a single-nodeVictoriaMetrics orvmstorage.

During querying, VictoriaMetrics tracks how many times the requested metric name was fetched from the database andwhen was the last time it happened. In this way, it is possible to identify metric names that were never queried.Or if metric was queried occasionally - when the last time it happened.

The usage stats for a metric won’t update in these two cases:

  • Querying a metric with non-matching filters. For example, querying forvm_log_messages_total{level!="info"} won’t update usage statsforvm_log_messages_total if there is no{level!="info"} series yet.
  • The query response is fully cached in therollup result cache.

To get metric names usage statistics, use the/prometheus/api/v1/status/metric_names_stats API endpoint fora single-node VictoriaMetrics (or athttp://<vmselect>:8481/select/<accountID>/prometheus/api/v1/status/metric_names_stats incluster version of VictoriaMetrics).It accepts the following query parameters:

  • limit - integer value to limit the number of metric names in response. By default, API returns 1000 records.
  • le -less than or equal, is an integer threshold for filtering metric names by their usage count in queries.For example, with?le=1 API returns metric names that were queried <=1 times.
  • match_pattern - a regex pattern to match metric names. For example,?match_pattern=vm_ will match any metricnames withvm_ pattern, likevm_http_requests,max_vm_memory_available.

The API endpoint returns the followingJSON response:

{"status":"success","statsCollectedSince":1737534094,"statsCollectedRecordsTotal":2,"records":[{"metricName":"node_disk_writes_completed_total","queryRequestsCount":50,"lastRequestTimestamp":1737534262},{"metricName":"node_network_transmit_errs_total","queryRequestsCount":100,"lastRequestTimestamp":1737534262}]}
  • statsCollectedSince is a timestamp since tracker was enabled (or reset, see below);
  • statsCollectedRecordsTotal total number of metric names it contains;
  • records:
    • metricName a metric name;
    • queryRequestsCount a cumulative counter of times the metric was fetched. If metric namefoo has 10 time series,then one read queryfoo will increment counter by 10.
    • lastRequestTimestamp a timestamp when last time this statistic was updated.

VictoriaMetrics tracks metric names query statistics for/api/v1/query,/api/v1/query_range,/render,/federate and/api/v1/export API calls.

VictoriaMetrics stores tracked metric names in memory and saves the state to disk in the<-storageDataPath>/cache folder during restarts.The size of the in-memory state is limited to1% of the available memory by default.This limit can be adjusted using the-storage.cacheSizeMetricNamesStats flag.

When the maximum state capacity is reached, VictoriaMetrics will stop tracking stats for newly registered time series.However, read request statistics for already tracked time series will continue to work as expected.

VictoriaMetrics exposes the following metrics for the metric name tracker:

  • vm_cache_size_bytes{type="storage/metricNamesStatsTracker"}
  • vm_cache_size{type="storage/metricNamesStatsTracker"}
  • vm_cache_size_max_bytes{type="storage/metricNamesStatsTracker"}

An alerting rule with queryvm_cache_size_bytes{type="storage/metricNamesStatsTracker"} \ vm_cache_size_max_bytes{type="storage/metricNamesStatsTracker"} > 0.9can be used to notify the user of cache utilization exceeding 90%.

The metric name tracker state can bereset via the API endpoint/api/v1/admin/status/metric_names_stats/resetfor a single-node VictoriaMetrics (or athttp://<vmselect>:8481/admin/api/v1/admin/status/metric_names_stats/resetincluster version of VictoriaMetrics) orviacache removalprocedure. This reset state endpoint can be protected via-metricNamesStatsResetAuthKeycmd-line flag. SeeSecurityfor details.

Query tracing#

VictoriaMetrics supports query tracing, which can be used for determining bottlenecks during query processing.This is likeEXPLAIN ANALYZE from Postgresql.

Query tracing can be enabled for a specific query by passingtrace=1 query arg.In this case VictoriaMetrics puts query trace intotrace field in the output JSON.

For example, the following command:

curl http://localhost:8428/api/v1/query_range -d'query=2*rand()' -d'start=-1h' -d'step=1m' -d'trace=1'| jq'.trace'

would return the following trace:

{"duration_msec":0.099,"message":"/api/v1/query_range: start=1654034340000, end=1654037880000, step=60000, query=\"2*rand()\": series=1","children":[{"duration_msec":0.034,"message":"eval: query=2 * rand(), timeRange=[1654034340000..1654037880000], step=60000, mayCache=true: series=1, points=60, pointsPerSeries=60","children":[{"duration_msec":0.032,"message":"binary op \"*\": series=1","children":[{"duration_msec":0.009,"message":"eval: query=2, timeRange=[1654034340000..1654037880000], step=60000, mayCache=true: series=1, points=60, pointsPerSeries=60"},{"duration_msec":0.017,"message":"eval: query=rand(), timeRange=[1654034340000..1654037880000], step=60000, mayCache=true: series=1, points=60, pointsPerSeries=60","children":[{"duration_msec":0.015,"message":"transform rand(): series=1"}]}]}]},{"duration_msec":0.004,"message":"sort series by metric name and labels"},{"duration_msec":0.044,"message":"generate /api/v1/query_range response for series=1, points=60"}]}

All the durations and timestamps in traces are in milliseconds.

Query tracing is allowed by default. It can be denied by passing-denyQueryTracing command-line flag to VictoriaMetrics.

VMUIprovides an UI:

  • for query tracing - just clickTrace query checkbox and re-run the query in order to investigate its’ trace.
  • for exploring custom trace - go to the tabTrace analyzer and upload or paste JSON with trace information.

Cardinality limiter#

By default, VictoriaMetrics doesn’t limit the number of stored time series. The limit can be enforced by setting the following command-line flags:

  • -storage.maxHourlySeries - limits the number of time series that can be added during the last hour. Useful for limiting the number ofactive time series.
  • -storage.maxDailySeries - limits the number of time series that can be added during the last day. Useful for limiting dailychurn rate.

Both limits can be set simultaneously. If any of these limits is reached, then incoming samples for new time series are dropped. A sample of dropped series is put in the log withWARNING level.

The exceeded limits can bemonitoredwith the following metrics:

  • vm_hourly_series_limit_rows_dropped_total - the number of metrics dropped due to exceeded hourly limit on the number of unique time series.

  • vm_hourly_series_limit_max_series - the hourly series limit set via-storage.maxHourlySeries command-line flag.

  • vm_hourly_series_limit_current_series - the current number of unique series during the last hour.The following query can be useful for alerting when the number of unique series during the last hour exceeds 90% of the-storage.maxHourlySeries:

    vm_hourly_series_limit_current_series / vm_hourly_series_limit_max_series > 0.9

  • vm_daily_series_limit_rows_dropped_total - the number of metrics dropped due to exceeded daily limit on the number of unique time series.

  • vm_daily_series_limit_max_series - the daily series limit set via-storage.maxDailySeries command-line flag.

  • vm_daily_series_limit_current_series - the current number of unique series during the last day.The following query can be useful for alerting when the number of unique series during the last day exceeds 90% of the-storage.maxDailySeries:

    vm_daily_series_limit_current_series / vm_daily_series_limit_max_series > 0.9

These limits are approximate, so VictoriaMetrics can underflow/overflow the limit by a small percentage (usually less than 1%).

See also more advancedcardinality limiter in vmagentandcardinality explorer docs.

Troubleshooting#

  • It is recommended to use default command-line flag values (i.e. don’t set them explicitly) until the needof tweaking these flag values arises.

  • It is recommended inspecting logs during troubleshooting, since they may contain useful information.

  • It is recommended upgrading to the latest available release fromthis page,since the encountered issue could be already fixed there.

  • It is recommended to have at least 50% of spare resources for CPU, disk IO and RAM, so VictoriaMetrics could handle short spikes in the workload without performance issues.

  • VictoriaMetrics requires free disk space formerging data files to bigger ones.It may slow down when there is no enough free space left. So make sure-storageDataPath directoryhas at least 20% of free space. The remaining amount of free spacecan bemonitoredviavm_free_disk_space_bytes metric. The total size of datastored on the disk can be monitored via sum ofvm_data_size_bytes metrics.

  • If you run VictoriaMetrics on a host with 16 or more CPU cores, then it may be needed to tune the-search.maxWorkersPerQuery command-line flagin order to improve query performance. If VictoriaMetrics serves big number of concurrentselect queries, then try reducing the value for this flag.If VictoriaMetrics serves heavy queries, which select>10K oftime seriesand/or process>100Mofraw samplesper query, then try setting the value for this flag to the number of available CPU cores.

  • VictoriaMetrics buffers incoming data in memory for up to a few seconds before flushing it to persistent storage.This may lead to the following “issues”:

    • Data becomes available for querying in a few seconds after inserting. It is possible to flush in-memory buffers to searchable partsby requesting/internal/force_flush http handler. This handler is mostly needed for testing and debugging purposes.
    • The last few seconds of inserted data may be lost on unclean shutdown (i.e. OOM,kill -9 or hardware reset).The-inmemoryDataFlushInterval command-line flag allows controlling the frequency of in-memory data flush to persistent storage.Seestorage docsandthis articlefor more details.

  • If VictoriaMetrics works slowly and eats more than a CPU core per 100K ingested data points per second,then it is likely you have too manyactive time seriesfor the current amount of RAM.VictoriaMetricsexposesvm_slow_* metrics such asvm_slow_row_inserts_total andvm_slow_metric_name_loads_total, which could be usedas an indicator of low amounts of RAM. It is recommended increasing the amount of RAM on the node with VictoriaMetrics in order to improveingestion and query performance in this case.

  • If the order of labels for the same metrics can change over time (e.g. ifmetric{k1="v1",k2="v2"} may becomemetric{k2="v2",k1="v1"}),then it is recommended running VictoriaMetrics with-sortLabels command-line flag in order to reduce memory usage and CPU usage.

  • VictoriaMetrics prioritizes data ingestion over data querying. So if it has no enough resources for data ingestion,then data querying may slow down significantly.

  • If VictoriaMetrics doesn’t work because of certain parts are corrupted due to disk errors,then just remove directories with broken parts. It is safe removing subdirectories under<-storageDataPath>/data/{big,small}/YYYY_MM directorieswhen VictoriaMetrics isn’t running. This recovers VictoriaMetrics at the cost of data loss stored in the deleted broken parts.The names of broken parts should be present in the error message. If you see that error message is truncated and doesn’t contain all the informationtry increasing-loggerMaxArgLen cmd-line flag to higher values to avoid error messages truncation.

  • If you see gaps on the graphs, try resetting the cache by sending request to/internal/resetRollupResultCache.If this removes gaps on the graphs, then it is likely data with timestamps older than-search.cacheTimestampOffsetis ingested into VictoriaMetrics. Make sure that data sources have synchronized time with VictoriaMetrics.

    If the gaps are related to irregular intervals between samples, then try adjusting-search.minStalenessInterval command-line flagto value close to the maximum interval between samples.

  • If you are switching from InfluxDB or TimescaleDB, then it may be needed to set-search.setLookbackToStep command-line flag.This suppresses default gap filling algorithm used by VictoriaMetrics - by default it assumeseach time series is continuous instead of discrete, so it fills gaps between real samples with regular intervals.

  • Metrics and labels leading tohigh cardinalityorhigh churn ratecan be determinedviacardinality explorerand via/api/v1/status/tsdbendpoint.

  • New time series can be logged if-logNewSeries command-line flag is passed to VictoriaMetrics or temporary enabled via/internal/log_new_series API call./internal/log_new_series API accepts query parameterseconds, with default value of60, which defines a duration for logging newly created series.

  • VictoriaMetrics limits the number of labels per each series, label name length and label value lengthvia-maxLabelsPerTimeseries,-maxLabelNameLen and-maxLabelValueLen command-line flags respectively.Series that exceed the limits are ignored on ingestion. This prevents from ingesting malformed series.It is recommendedmonitoringvm_rows_ignored_total metric and VictoriaMetrics logs in orderto determine whether limits must be adjusted for your workload.Alternatively, you can userelabelingto change metric target labels.

  • If you store Graphite metrics likefoo.bar.baz in VictoriaMetrics, then{__graphite__="foo.*.baz"} filter can be used for selecting such metrics.Seethese docsfor details. You can also query Graphite metrics withGraphite querying API.

  • VictoriaMetrics ignoresNaN values during data ingestion.

See also:

Push metrics#

All the VictoriaMetrics components support pushing their metrics exposed at/metrics page to remote storage in Prometheus text exposition format.This functionality may be used instead ofclassic Prometheus-like metrics scrapingif VictoriaMetrics components are located in isolated networks, so they cannot be scraped by localvmagent.

The following command-line flags are related to pushing metrics from VictoriaMetrics components:

  • -pushmetrics.url - the url to push metrics to. For example,-pushmetrics.url=http://victoria-metrics:8428/api/v1/import/prometheus instructsto push internal metrics to/api/v1/import/prometheus endpoint according tothese docs.The-pushmetrics.url can be specified multiple times. In this case metrics are pushed to all the specified urls.The url can contain basic auth params in the formhttp://user:pass@hostname/api/v1/import/prometheus.Metrics are pushed to the provided-pushmetrics.url in a compressed form withContent-Encoding: gzip request header.This allows reducing the required network bandwidth for metrics push. The compression can be disabled by passing-pushmetrics.disableCompression command-line flag.
  • -pushmetrics.extraLabel - labels to add to all the metrics before sending them to every-pushmetrics.url. Each label must be specified in the formatlabel="value".It is OK to specify multiple-pushmetrics.extraLabel command-line flags. In this case all the specified labelsare added to all the metrics before sending them to all the configured-pushmetrics.url addresses.
  • -pushmetrics.interval - the interval between pushes. By default it is set to 10 seconds.
  • -pushmetrics.header - an optional HTTP header to send to every-pushmetrics.url. For example,-pushmetrics.header='Authorization: Basic foo' instructs to sendAuthorization: Basic foo HTTP header with every request to every-pushmetrics.url. It is possible to set multiple-pushmetrics.header command-line flagsfor sending multiple different HTTP headers to-pushmetrics.url.

For example, the following command instructs VictoriaMetrics to push metrics from/metrics page tohttps://maas.victoriametrics.com/api/v1/import/prometheuswithuser:passBasic auth. Theinstance="foobar" andjob="vm" labelsare added to all the metrics before sending them to the remote storage:

/path/to/victoria-metrics\  -pushmetrics.url=https://user:pass@maas.victoriametrics.com/api/v1/import/prometheus\  -pushmetrics.extraLabel='instance="foobar"'\  -pushmetrics.extraLabel='job="vm"'

Caches#

Cache removal#

VictoriaMetrics uses various internal caches. These caches are stored to<-storageDataPath>/cache directory during graceful shutdown(e.g. when VictoriaMetrics is stopped by sendingSIGINT signal). The caches are read on the next VictoriaMetrics startup.Sometimes it is needed to remove such caches on the next startup. This can be done in the following ways:

  • By manually removing the<-storageDataPath>/cache directory when VictoriaMetrics is stopped.
  • By placingreset_cache_on_startup file inside the<-storageDataPath>/cache directory before the restart of VictoriaMetrics.In this case VictoriaMetrics will automatically remove all the caches on the next start.Seethis issuefor details.

It is also possible removingrollup result cacheon startup by passing-search.resetRollupResultCacheOnStartup command-line flag to VictoriaMetrics.

Rollup result cache#

VictoriaMetrics caches query responses by default. This allows increasing performance for repeated queriesto/api/v1/queryand/api/v1/query_rangewith the increasingtime,start andend query args.

This cache may work incorrectly when ingesting historical data into VictoriaMetrics. Seethese docsfor details.

The rollup cache can be disabled either globally by running VictoriaMetrics with-search.disableCache command-line flagor on a per-query basis by passingnocache=1 query arg to/api/v1/query and/api/v1/query_range.

See alsocache removal docs.

Cache tuning#

VictoriaMetrics uses various in-memory caches for faster data ingestion and query performance.The following metrics for each type of cache are exported at/metrics page:

  • vm_cache_size_bytes - the actual cache size
  • vm_cache_size_max_bytes - cache size limit
  • vm_cache_requests_total - the number of requests to the cache
  • vm_cache_misses_total - the number of cache misses
  • vm_cache_entries - the number of entries in the cache

Both GrafanaVictoriaMetrics - single-nodeandVictoriaMetrics - clusterdashboardscontainTroubleshooting section where the cache metrics are visualized. TheCache usage %panel shows the percentage of used cache size from the allowed size by type. If the percentageis below 100%, then no further tuning needed. TheCache miss ratio panel shows the percentageof reads for which no value was found in the cache. If the cache utilization is 100% and there arecache misses, then the cache is either not accepting new entries or evicting existing ones. Itssize may need to be increased.

Please note, default cache sizes were carefully adjusted accordingly to the mostpractical scenarios and workloads. Change the defaults only if you understand the implicationsand vmstorage has enough free memory to accommodate new cache sizes.

To override the default values see command-line flags with-storage.cacheSize prefix.See the full description ofcommand-line flags.

Data migration#

From VictoriaMetrics#

The simplest way to migrate data from one single-node (source) to another (destination), or from one vmstorage nodeto another is to do the following:

  1. Stop the VictoriaMetrics (source) withkill -INT;
  2. Copy (viarsyncor any other tool) the entire folder specifiedvia-storageDataPath from the source node toan empty folder at the destination node.
  3. Once copy is done, stop the VictoriaMetrics (destination) withkill -INT and verify thatits-storageDataPath points to the copied folder from p.2;
  4. Start the VictoriaMetrics (destination). The copied data should be now available.

Things to consider when copying data:

  • Data formats between single-node and vmstorage node aren’t compatible and can’t be copied.
  • Copying a data folder means complete replacement of the previous data on destination VictoriaMetrics.
  • Data can’t be mixed: make sure that the destination folderis empty before copying.

For scenarios like single-to-cluster, cluster-to-single, re-sharding or migrating only a fraction of data:see how to migrate data from VictoriaMetrics via vmctl.

From other systems#

Usevmctlto migrate data from other systems to VictoriaMetrics.

Backfilling#

VictoriaMetrics accepts out-of-order historical data viaany supported ingestion methodwithout limitations. Only make sure that backfilled data is within of the configuredretention period.

Note

Seehow to backfill recording rules via vmalert.

It is recommended disablingquery cachewith-search.disableCache command-line flag when writinghistorical data with timestamps from the past, since the cache assumes that the data is written withthe current timestamps. Query cache can be enabled after the backfilling is complete.

An alternative solution is to query/internal/resetRollupResultCacheafter the backfilling is complete. This will reset thequery cache, which could contain incomplete data cached during the backfilling.

Yet another solution is to increase-search.cacheTimestampOffset flag value to disable cachingfor data with timestamps close to the current time. Single-node VictoriaMetrics automatically resets responsecache when samples with timestamps older thannow - search.cacheTimestampOffset are ingested to it.

Data updates#

VictoriaMetrics doesn’t support updating already existing sample values to new ones. It stores all the ingested data pointsfor the same time series with identical timestamps. While it is possible substituting old time series with new time series viaremoval of old time seriesand thenwriting new time series, this approachshould be used only for one-off updates. It shouldn’t be used for frequent updates because of non-zero overhead related to data removal.

Replication#

Single-node VictoriaMetrics doesn’t support application-level replication. Use cluster version instead.Seethese docsfor details.

Storage-level replication may be offloaded to durable persistent storage such asGoogle Cloud disks.

See alsohigh availability docsandbackup docs.

Backups#

For backup configuration and setup, please refer tovmbackup documentation.

vmalert#

A single-node VictoriaMetrics is capable of proxying requests tovmalertwhen-vmalert.proxyURL flag is set. Use this feature for the following cases:

  • for proxying requests fromGrafana Alerting UI;
  • for accessing vmalerts UI through single-node VictoriaMetrics Web interface.

For accessing vmalerts UI through single-node VictoriaMetrics configure-vmalert.proxyURL flag and visithttp://<victoriametrics-addr>:8428/vmalert/ link.

Benchmarks#

Note, that vendors (including VictoriaMetrics) are often biased when doing such tests. E.g. they try highlightingthe best parts of their product, while highlighting the worst parts of competing products.So we encourage users and all independent third parties to conduct their benchmarks for various productsthey are evaluating in production and publish the results.

As a reference, please seebenchmarksconducted byVictoriaMetrics team. Please also see thehelm chartfor running ingestion benchmarks based on node_exporter metrics.

Profiling#

VictoriaMetrics provides handlers for collecting the followingGo profiles:

  • Memory profile. It can be collected with the following command (replace0.0.0.0 with hostname if needed):
curl http://0.0.0.0:8428/debug/pprof/heap > mem.pprof
  • CPU profile. It can be collected with the following command (replace0.0.0.0 with hostname if needed):
curl http://0.0.0.0:8428/debug/pprof/profile > cpu.pprof

The command for collecting CPU profile waits for 30 seconds before returning.

The collected profiles may be analyzed withgo tool pprof.It is safe sharing the collected profiles from security point of view, since they do not contain sensitive information.

Third-party contributions#

Contacts#

Contact us with any questions regarding VictoriaMetrics atinfo@victoriametrics.com.

Community and contributions#

Feel free asking any questions regarding VictoriaMetrics:

If you like VictoriaMetrics and want to contribute, then pleaseread these docs.

Reporting bugs#

Report bugs and propose new features in ourGitHub Issues.

Documentation#

VictoriaMetrics documentation is available athttps://docs.victoriametrics.com/victoriametrics/.It is built from*.md files located indocsfolderand gets automatically updated once changes are merged tomasterbranch.To update the documentation follow the steps below:

Requirements for changes to docs:

  • Keep backward compatibility of existing links. Avoid changing anchors or deleting pages as they could have beenused or posted in other docs, GitHub issues, Stack Overflow answers, etc.
  • Keep docs clear, concise and simple. Try using as simple wording as possible, without sacrificing clarity.
  • Keep docs consistent. When modifying existing docs, verify that other places referencing to this doc are still relevant.
  • Prefer improving the existing docs instead of adding new ones.
  • Use absolute links. This simplifies moving docs between different files.

Periodically runmake spellcheck - this command detects spelling errors atdocs/ folder. Please fix the found spelling errorsand commit the fixes in a separate commit.

Images in documentation#

Please, keep image size and number of images per single page low. Keep the docs page as lightweight as possible.

Image files must be placed in the same folder as the doc itself and they must have the same prefix as the doc filename.For example, all the images fordocs/foo/bar.md should have filenames starting fromdocs/foo/bar.This simplifies lifetime management of the images:

  • when the corresponding doc is removed, then it is clear how to remove the associated images
  • when the corresponding doc is renamed, then it is clear how to rename the associated images.

If the page needs to have many images, consider using WEB-optimized image formatwebp.When adding a new doc with many images usewebp format right away. Or use a Makefile command below toconvert already existing images atdocs folder automatically toweb format:

make docs-images-to-webp

Once conversion is done, update the path to images in your docs and verify everything is correct.

Zipcontains three folders with different image orientations (main color and inverted version).

Files included in each folder:

  • 2 JPEG Preview files
  • 2 PNG Preview files with transparent background
  • 2 EPS Adobe Illustrator EPS10 files

Logo Usage Guidelines#

Font used#

  • Lato Black
  • Lato Regular

Color Palette#

We kindly ask#

  • Please don’t use any other font instead of suggested.
  • To keep enough clear space around the logo.
  • Do not change spacing, alignment, or relative locations of the design elements.
  • Do not change the proportions for any of the design elements or the design itself.You may resize as needed but must retain all proportions.

List of command-line flags#

Pass-help to VictoriaMetrics in order to see the list of supported command-line flags with their description:

victoria-metricsisatimeseriesdatabaseandmonitoringsolution.Seethedocsathttps://docs.victoriametrics.com/victoriametrics/-bigMergeConcurrencyintDeprecated:thisflagdoesnothing-blockcache.missesBeforeCachingintThenumberofcachemissesbeforeputtingtheblockintocache.Highervaluesmayreduceindexdb/dataBlockscachesizeatthecostofhigherCPUanddiskreadusage(default2)-cacheExpireDurationdurationItemsareremovedfromin-memorycachesaftertheyaren't accessed for this duration. Lower values may reduce memory usage at the cost of higher CPU usage. See also -prevCacheRemovalPercent (default 30m0s)-configAuthKeyvalueAuthorizationkeyforaccessing/configpage.ItmustbepassedviaauthKeyqueryarg.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-configAuthKey=file:///abs/path/to/fileor-configAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-configAuthKey=http://host/pathor-configAuthKey=https://host/path-csvTrimTimestampdurationTrimtimestampswhenimportingcsvdatatothisduration.Minimumpracticaldurationis1ms.Higherduration(i.e.1s)maybeusedforreducingdiskspaceusagefortimestampdata(default1ms)-datadog.maxInsertRequestSizesizeThemaximumsizeinbytesofasingleDataDogPOSTrequestto/datadog/api/v2/seriesSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default67108864)-datadog.sanitizeMetricNameSanitizemetricnamesfortheingestedDataDogdatatocomplywithDataDogbehaviourdescribedathttps://docs.datadoghq.com/metrics/custom_metrics/#naming-custom-metrics (default true)-dedup.minScrapeIntervaldurationLeaveonlythelastsampleineverytimeseriespereachdiscreteintervalequalto-dedup.minScrapeInterval>0.Seealso-streamAggr.dedupIntervalandhttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#deduplication-deleteAuthKeyvalueauthKeyformetrics' deletion via /api/v1/admin/tsdb/delete_series and /tags/delSeries. It could be passed via authKey query arg. It overrides -httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-deleteAuthKey=file:///abs/path/to/fileor-deleteAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-deleteAuthKey=http://host/pathor-deleteAuthKey=https://host/path-denyQueriesOutsideRetentionWhethertodenyqueriesoutsidetheconfigured-retentionPeriod.Whenset,then/api/v1/query_rangewouldreturn'503 Service Unavailable'errorforquerieswith'from'valueoutside-retentionPeriod.Thismaybeusefulwhenmultipledatasourceswithdistinctretentionsarehiddenbehindquery-tee-denyQueryTracingWhethertodisabletheabilitytotracequeries.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#query-tracing-disablePerDayIndexDisableper-dayindexanduseglobalindexforallsearches.Thismayimproveperformanceanddecreasediskspaceusagefortheusecaseswithfixedsetoftimeseriesscatteredacrossabigtimerange(forexample,whenloadingyearsofhistoricaldata).Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#index-tuning-downsampling.periodarrayComma-separateddownsamplingperiodsintheformat'offset:period'.Forexample,'30d:10m'instructstoleaveasinglesampleper10minutesforsamplesolderthan30days.The'offset'mustbeamultipleof'interval',andwhensettingmultipledownsamplingperiodsforasinglefilter,thoseperiodsmustalsobemultiplesofeachother.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#downsampling for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/victoriametrics/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Eacharrayitemcancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-dryRunWhethertocheckconfigfileswithoutrunningVictoriaMetrics.Thefollowingconfigfilesarechecked:-promscrape.config,-relabelConfigand-streamAggr.config.Unknownconfigentriesaren't allowed in -promscrape.config by default. This can be changed with -promscrape.config.strictParse=false command-line flag-enableMetadataWhethertoenablemetadataprocessingformetricsscrapedfromtargets,receivedviaVictoriaMetricsremotewrite,Prometheusremotewritev1orOpenTelemetryprotocol.SeealsoremoteWrite.maxMetadataPerBlock-enableTCP6WhethertoenableIPv6forlisteninganddialing.Bydefault,onlyIPv4TCPandUDPareused-envflag.enableWhethertoenablereadingflagsfromenvironmentvariablesinadditiontothecommandline.Commandlineflagvalueshavepriorityovervaluesfromenvironmentvars.Flagsarereadonlyfromthecommandlineifthisflagisn't set. See https://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#environment-variables for more details-envflag.prefixstringPrefixforenvironmentvariablesif-envflag.enableisset-eulaDeprecated,pleaseuse-licenseor-licenseFileflagsinstead.Byspecifyingthisflag,youconfirmthatyouhaveanenterpriselicenseandaccepttheESAhttps://victoriametrics.com/legal/esa/.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/victoriametrics/enterprise/-filestream.disableFadviseWhethertodisablefadvise()syscallwhenreadinglargedatafiles.Thefadvise()syscallpreventsfromevictionofrecentlyaccesseddatafromOSpagecacheduringbackgroundmergesandbackups.InsomerarecasesitisbettertodisablethesyscallifitusestoomuchCPU-finalMergeDelaydurationDeprecated:thisflagdoesnothing-flagsAuthKeyvalueAuthkeyfor/flagsendpoint.ItmustbepassedviaauthKeyqueryarg.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-flagsAuthKey=file:///abs/path/to/fileor-flagsAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-flagsAuthKey=http://host/pathor-flagsAuthKey=https://host/path-forceFlushAuthKeyvalueauthKey,whichmustbepassedinquerystringto/internal/force_flushpages.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-forceFlushAuthKey=file:///abs/path/to/fileor-forceFlushAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-forceFlushAuthKey=http://host/pathor-forceFlushAuthKey=https://host/path-forceMergeAuthKeyvalueauthKey,whichmustbepassedinquerystringto/internal/force_mergepages.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-forceMergeAuthKey=file:///abs/path/to/fileor-forceMergeAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-forceMergeAuthKey=http://host/pathor-forceMergeAuthKey=https://host/path-fs.disableMmapWhethertousepread()insteadofmmap()forreadingdatafiles.Bydefault,mmap()isusedfor64-bitarchesandpread()isusedfor32-bitarches,sincetheycannotreaddatafilesbiggerthan2^32bytesinmemory.mmap()isusuallyfasterforreadingsmalldatachunksthanpread()-fs.maxConcurrencyintThemaximumnumberofconcurrentgoroutinestoworkwithfiles;smallervaluesmayhelpreducingGoschedulinglatencyonsystemswithsmallnumberofCPUcores;highervaluesmayhelpreducingdataingestionlatencyonsystemswithhigh-latencystoragesuchasNFSorCeph(defaultfsutil.getDefaultConcurrency())-graphite.sanitizeMetricNameSanitizemetricnamesfortheingestedGraphitedata.Seehttps://docs.victoriametrics.com/victoriametrics/integrations/graphite/#ingesting-graphiteListenAddrstringTCPandUDPaddresstolistenforGraphiteplaintextdata.Usually:2003mustbeset.Doesn't work if empty. See also -graphiteListenAddr.useProxyProtocol-graphiteListenAddr.useProxyProtocolWhethertouseproxyprotocolforconnectionsacceptedat-graphiteListenAddr.Seehttps://www.haproxy.org/download/1.8/doc/proxy-protocol.txt-graphiteTrimTimestampdurationTrimtimestampsforGraphitedatatothisduration.Minimumpracticaldurationis1s.Higherduration(i.e.1m)maybeusedforreducingdiskspaceusagefortimestampdata(default1s)-http.connTimeoutdurationIncomingconnectionsto-httpListenAddrareclosedaftertheconfiguredtimeout.ThismayhelpevenlyspreadingloadamongaclusterofservicesbehindTCP-levelloadbalancer.Zerovaluedisablesclosingofincomingconnections(default2m0s)-http.disableCORSDisableCORSforallorigins(*)-http.disableKeepAliveWhethertodisableHTTPkeep-aliveforincomingconnectionsat-httpListenAddr-http.disableResponseCompressionDisablecompressionofHTTPresponsestosaveCPUresources.Bydefault,compressionisenabledtosavenetworkbandwidth-http.header.cspstringValuefor'Content-Security-Policy'header,recommended:"default-src 'self'"-http.header.frameOptionsstringValuefor'X-Frame-Options'header-http.header.hstsstringValuefor'Strict-Transport-Security'header,recommended:'max-age=31536000; includeSubDomains'-http.idleConnTimeoutdurationTimeoutforincomingidlehttpconnections(default1m0s)-http.maxGracefulShutdownDurationdurationThemaximumdurationforagracefulshutdownoftheHTTPserver.Ahighlyloadedservermayrequireincreasedvalueforagracefulshutdown(default7s)-http.pathPrefixstringAnoptionalprefixtoaddtoallthepathshandledbyhttpserver.Forexample,if'-http.pathPrefix=/foo/bar'isset,thenallthehttprequestswillbehandledon'/foo/bar/*'paths.Thismaybeusefulforproxiedrequests.Seehttps://www.robustperception.io/using-external-urls-and-proxies-with-prometheus-http.shutdownDelaydurationOptionaldelaybeforehttpservershutdown.Duringthisdelay,theserverreturnsnon-OKresponsesfrom/healthpage,soloadbalancerscanroutenewrequeststootherservers-httpAuth.passwordvaluePasswordforHTTPserver's Basic Auth. The authentication is disabled if -httpAuth.username is emptyFlagvaluecanbereadfromthegivenfilewhenusing-httpAuth.password=file:///abs/path/to/fileor-httpAuth.password=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-httpAuth.password=http://host/pathor-httpAuth.password=https://host/path-httpAuth.usernamestringUsernameforHTTPserver's Basic Auth. The authentication is disabled if empty. See also -httpAuth.password-httpListenAddrarrayTCPaddressestolistenforincominghttprequests.Seealso-tlsand-httpListenAddr.useProxyProtocolSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Eacharrayitemcancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-httpListenAddr.useProxyProtocolarrayWhethertouseproxyprotocolforconnectionsacceptedatthecorresponding-httpListenAddr.Seehttps://www.haproxy.org/download/1.8/doc/proxy-protocol.txt.Withenabledproxyprotocolhttpservercannotserveregular/metricsendpoint.Use-pushmetrics.urlformetricspushingSupportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-import.maxLineLensizeThemaximumlengthinbytesofasinglelineacceptedby/api/v1/import;thelinelengthcanbelimitedwith'max_rows_per_line'queryargpassedto/api/v1/exportSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default10485760)-influx.databaseNamesarrayComma-separatedlistofdatabasenamestoreturnfrom/queryand/influx/queryAPI.ThiscanbeneededforacceptingdatafromTelegrafpluginssuchashttps://github.com/fangli/fluent-plugin-influxdbSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Eacharrayitemcancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-influx.forceStreamModeForcestreammodeparsingforingesteddata.Seehttps://docs.victoriametrics.com/victoriametrics/integrations/influxdb/-influx.maxLineSizesizeThemaximumsizeinbytesforasingleInfluxDBlineduringparsing.Applicableforstreammodeonly.Seehttps://docs.victoriametrics.com/victoriametrics/integrations/influxdb/Supportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default262144)-influx.maxRequestSizesizeThemaximumsizeinbytesofasingleInfluxDBrequest.Applicableforbatchmodeonly.Seehttps://docs.victoriametrics.com/victoriametrics/integrations/influxdb/Supportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default67108864)-influxDBLabelstringDefaultlabelfortheDBnamesentover'?db={db_name}'queryparameter(default"db")-influxListenAddrstringTCPandUDPaddresstolistenforInfluxDBlineprotocoldata.Usually:8089mustbeset.Doesn't work if empty. This flag isn'tneededwheningestingdataoverHTTP-justsendittohttp://<victoriametrics>:8428/write.Seealso-influxListenAddr.useProxyProtocol-influxListenAddr.useProxyProtocolWhethertouseproxyprotocolforconnectionsacceptedat-influxListenAddr.Seehttps://www.haproxy.org/download/1.8/doc/proxy-protocol.txt-influxMeasurementFieldSeparatorstringSeparatorfor'{measurement}{separator}{field_name}'metricnamewheninsertedviaInfluxDBlineprotocol(default"_")-influxSkipMeasurementUses'{field_name}'asametricnamewhileignoring'{measurement}'and'-influxMeasurementFieldSeparator'-influxSkipSingleFieldUses'{measurement}'insteadof'{measurement}{separator}{field_name}'formetricnameifInfluxDBlinecontainsonlyasinglefield-influxTrimTimestampdurationTrimtimestampsforInfluxDBlineprotocoldatatothisduration.Minimumpracticaldurationis1ms.Higherduration(i.e.1s)maybeusedforreducingdiskspaceusagefortimestampdata(default1ms)-inmemoryDataFlushIntervaldurationTheintervalforguaranteedsavingofin-memorydatatodisk.ThesaveddatasurvivesuncleanshutdownssuchasOOMcrash,hardwarereset,SIGKILL,etc.Biggerintervalsmayhelpincreasethelifetimeofflashstoragewithlimitedwritecycles(e.g.RaspberryPI).SmallerintervalsincreasediskIOload.Minimumsupportedvalueis1s(default5s)-insert.maxQueueDurationdurationThemaximumdurationtowaitinthequeuewhen-maxConcurrentInsertsconcurrentinsertrequestsareexecuted(default1m0s)-internStringCacheExpireDurationdurationTheexpirydurationforcachesforinternedstrings.Seehttps://en.wikipedia.org/wiki/String_interning.Seealso-internStringMaxLenand-internStringDisableCache(default6m0s)-internStringDisableCacheWhethertodisablecachesforinternedstrings.ThismayreducememoryusageatthecostofhigherCPUusage.Seehttps://en.wikipedia.org/wiki/String_interning.Seealso-internStringCacheExpireDurationand-internStringMaxLen-internStringMaxLenintThemaximumlengthforstringstointern.AlowerlimitmaysavememoryatthecostofhigherCPUusage.Seehttps://en.wikipedia.org/wiki/String_interning.Seealso-internStringDisableCacheand-internStringCacheExpireDuration(default500)-licensestringLicensekeyforVictoriaMetricsEnterprise.Seehttps://victoriametrics.com/products/enterprise/.TrialEnterpriselicensecanbeobtainedfromhttps://victoriametrics.com/products/enterprise/trial/.ThisflagisavailableonlyinEnterprisebinaries.Thelicensekeycanbealsopassedviafilespecifiedby-licenseFilecommand-lineflag-license.forceOfflineWhethertoenableofflineverificationforVictoriaMetricsEnterpriselicensekey,whichhasbeenpassedeithervia-licenseorvia-licenseFilecommand-lineflag.Theissuedlicensekeymustsupportofflineverificationfeature.Contactinfo@victoriametrics.comifyouneedofflinelicenseverification.ThisflagisavailableonlyinEnterprisebinaries-licenseFilestringPathtofilewithlicensekeyforVictoriaMetricsEnterprise.Seehttps://victoriametrics.com/products/enterprise/.TrialEnterpriselicensecanbeobtainedfromhttps://victoriametrics.com/products/enterprise/trial/.ThisflagisavailableonlyinEnterprisebinaries.Thelicensekeycanbealsopassedinlinevia-licensecommand-lineflag-licenseFile.reloadIntervaldurationIntervalforreloadingthelicensefilespecifiedvia-licenseFile.Seehttps://victoriametrics.com/products/enterprise/.ThisflagisavailableonlyinEnterprisebinaries(default1h0m0s)-logNewSeriesWhethertolognewseries.Thisoptionisfordebugpurposesonly.ItcanleadtoperformanceissueswhenbignumberofnewseriesareingestedintoVictoriaMetrics-logNewSeriesAuthKeyvalueauthKey,whichmustbepassedinquerystringto/internal/log_new_series.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-logNewSeriesAuthKey=file:///abs/path/to/fileor-logNewSeriesAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-logNewSeriesAuthKey=http://host/pathor-logNewSeriesAuthKey=https://host/path-loggerDisableTimestampsWhethertodisablewritingtimestampsinlogs-loggerErrorsPerSecondLimitintPer-secondlimitonthenumberofERRORmessages.Ifmorethanthegivennumberoferrorsareemittedpersecond,theremainingerrorsaresuppressed.Zerovaluesdisabletheratelimit-loggerFormatstringFormatforlogs.Possiblevalues:default,json(default"default")-loggerJSONFieldsstringAllowsrenamingfieldsinJSONformattedlogs.Example:"ts:timestamp,msg:message"renames"ts"to"timestamp"and"msg"to"message".Supportedfields:ts,level,caller,msg-loggerLevelstringMinimumleveloferrorstolog.Possiblevalues:INFO,WARN,ERROR,FATAL,PANIC(default"INFO")-loggerMaxArgLenintThemaximumlengthofasingleloggedargument.Longerargumentsarereplacedwith'arg_start..arg_end',where'arg_start'and'arg_end'isprefixandsuffixoftheargwiththelengthnotexceeding-loggerMaxArgLen/2(default5000)-loggerOutputstringOutputforthelogs.Supportedvalues:stderr,stdout(default"stderr")-loggerTimezonestringTimezonetousefortimestampsinlogs.TimezonemustbeavalidIANATimeZone.Forexample:America/New_York,Europe/Berlin,Etc/GMT+3orLocal(default"UTC")-loggerWarnsPerSecondLimitintPer-secondlimitonthenumberofWARNmessages.Ifmorethanthegivennumberofwarnsareemittedpersecond,thentheremainingwarnsaresuppressed.Zerovaluesdisabletheratelimit-maxConcurrentInsertsintThemaximumnumberofconcurrentinsertrequests.Sethighervaluewhenclientssenddataoverslownetworks.DefaultvaluedependsonthenumberofavailableCPUcores.Itshouldworkfineinmostcasessinceitminimizesresourceusage.Seealso-insert.maxQueueDuration(default2*cgroup.AvailableCPUs())-maxIngestionRateintThemaximumnumberofsamplesvmsinglecanreceivepersecond.Dataingestionispausedwhenthelimitisexceeded.Bydefaulttherearenolimitsonsamplesingestionrate.-maxInsertRequestSizesizeThemaximumsizeinbytesofasinglePrometheusremote_writeAPIrequestSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default33554432)-maxLabelNameLenintThemaximumlengthoflabelnameintheacceptedtimeseries.Serieswithlongerlabelnameareignored.Inthiscasethevm_rows_ignored_total{reason="too_long_label_name"}metricat/metricspageisincremented(default256)-maxLabelValueLenintThemaximumlengthoflabelvaluesintheacceptedtimeseries.Serieswithlongerlabelvalueareignored.Inthiscasethevm_rows_ignored_total{reason="too_long_label_value"}metricat/metricspageisincremented(default4096)-maxLabelsPerTimeseriesintThemaximumnumberoflabelspertimeseriestobeaccepted.Serieswithsuperfluouslabelsareignored.Inthiscasethevm_rows_ignored_total{reason="too_many_labels"}metricat/metricspageisincremented(default40)-memory.allowedBytessizeAllowedsizeofsystemmemoryVictoriaMetricscachesmayoccupy.Thisoptionoverrides-memory.allowedPercentifsettoanon-zerovalue.ToolowavaluemayincreasethecachemissrateusuallyresultinginhigherCPUanddiskIOusage.ToohighavaluemayevicttoomuchdatafromtheOSpagecacheresultinginhigherdiskIOusageSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-memory.allowedPercentfloatAllowedpercentofsystemmemoryVictoriaMetricscachesmayoccupy.Seealso-memory.allowedBytes.ToolowavaluemayincreasecachemissrateusuallyresultinginhigherCPUanddiskIOusage.ToohighavaluemayevicttoomuchdatafromtheOSpagecachewhichwillresultinhigherdiskIOusage(default60)-metricNamesStatsResetAuthKeyvalueauthKeyforresettingmetricnamesusagecachevia/api/v1/admin/status/metric_names_stats/reset.Itoverrides-httpAuth.*.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#track-ingested-metrics-usageFlagvaluecanbereadfromthegivenfilewhenusing-metricNamesStatsResetAuthKey=file:///abs/path/to/fileor-metricNamesStatsResetAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-metricNamesStatsResetAuthKey=http://host/pathor-metricNamesStatsResetAuthKey=https://host/path-metrics.exposeMetadataWhethertoexposeTYPEandHELPmetadataatthe/metricspage,whichisexposedat-httpListenAddr.Themetadatamaybeneededwhenthe/metricspageisconsumedbysystems,whichrequirethisinformation.Forexample,ManagedPrometheusinGoogleCloud-https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type-metricsAuthKeyvalueAuthkeyfor/metricsendpoint.ItmustbepassedviaauthKeyqueryarg.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-metricsAuthKey=file:///abs/path/to/fileor-metricsAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-metricsAuthKey=http://host/pathor-metricsAuthKey=https://host/path-mtlsarrayWhethertorequirevalidclientcertificateforhttpsrequeststothecorresponding-httpListenAddr.Thisflagworksonlyif-tlsflagisset.Seealso-mtlsCAFile.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/victoriametrics/enterprise/Supportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-mtlsCAFilearrayOptionalpathtoTLSRootCAforverifyingclientcertificatesatthecorresponding-httpListenAddrwhen-mtlsisenabled.BydefaultthehostsystemTLSRootCAisusedforclientcertificateverification.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/victoriametrics/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Eacharrayitemcancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-newrelic.maxInsertRequestSizesizeThemaximumsizeinbytesofasingleNewRelicrequestto/newrelic/infra/v2/metrics/events/bulkSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default67108864)-opentelemetry.convertMetricNamesToPrometheusWhethertoconvertonlymetricnamesintoPrometheus-compatibleformatforthemetricsingestedviaOpenTelemetryprotocol;seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#sending-data-via-opentelemetry-opentelemetry.maxRequestSizesizeThemaximumsizeinbytesofasingleOpenTelemetryrequestSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default67108864)-opentelemetry.usePrometheusNamingWhethertoconvertmetricnamesandlabelsintoPrometheus-compatibleformatforthemetricsingestedviaOpenTelemetryprotocol;seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#sending-data-via-opentelemetry-opentsdbHTTPListenAddrstringTCPaddresstolistenforOpenTSDBHTTPputrequests.Usually:4242mustbeset.Doesn't work if empty. See also -opentsdbHTTPListenAddr.useProxyProtocol-opentsdbHTTPListenAddr.useProxyProtocolWhethertouseproxyprotocolforconnectionsacceptedat-opentsdbHTTPListenAddr.Seehttps://www.haproxy.org/download/1.8/doc/proxy-protocol.txt-opentsdbListenAddrstringTCPandUDPaddresstolistenforOpenTSDBmetrics.TelnetputmessagesandHTTP/api/putmessagesaresimultaneouslyservedonTCPport.Usually:4242mustbeset.Doesn't work if empty. See also -opentsdbListenAddr.useProxyProtocol-opentsdbListenAddr.useProxyProtocolWhethertouseproxyprotocolforconnectionsacceptedat-opentsdbListenAddr.Seehttps://www.haproxy.org/download/1.8/doc/proxy-protocol.txt-opentsdbTrimTimestampdurationTrimtimestampsforOpenTSDB'telnet put'datatothisduration.Minimumpracticaldurationis1s.Higherduration(i.e.1m)maybeusedforreducingdiskspaceusagefortimestampdata(default1s)-opentsdbhttp.maxInsertRequestSizesizeThemaximumsizeofOpenTSDBHTTPputrequestSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default33554432)-opentsdbhttpTrimTimestampdurationTrimtimestampsforOpenTSDBHTTPdatatothisduration.Minimumpracticaldurationis1ms.Higherduration(i.e.1s)maybeusedforreducingdiskspaceusagefortimestampdata(default1ms)-pprofAuthKeyvalueAuthkeyfor/debug/pprof/*endpoints.ItmustbepassedviaauthKeyqueryarg.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-pprofAuthKey=file:///abs/path/to/fileor-pprofAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-pprofAuthKey=http://host/pathor-pprofAuthKey=https://host/path-precisionBitsintThenumberofprecisionbitstostorepereachvalue.Lowerprecisionbitsimprovesdatacompressionatthecostofprecisionloss(default64)-prevCacheRemovalPercentfloatItemsinthepreviouscachesareremovedwhenthepercentofrequestsitservesbecomeslowerthanthisvalue.HighervaluesreducememoryusageatthecostofhigherCPUusage.Seealso-cacheExpireDuration(default0.1)-promscrape.azureSDCheckIntervaldurationIntervalforcheckingforchangesinAzure.Thisworksonlyifazure_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#azure_sd_configs for details (default 1m0s)-promscrape.cluster.memberLabelstringIfnon-empty,thenthelabelwiththisnameandthe-promscrape.cluster.memberNumvalueisaddedtoallthescrapedmetrics.Seehttps://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets for more info-promscrape.cluster.memberNumstringThenumberofvmagentinstanceintheclusterofscrapers.Itmustbeauniquevalueintherange0...promscrape.cluster.membersCount-1acrossscrapersinthecluster.CanbespecifiedaspodnameofKubernetesStatefulSet-pod-name-Num,whereNumisanumericpartofpodname.Seealso-promscrape.cluster.memberLabel.Seehttps://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets for more info (default "0")-promscrape.cluster.memberURLTemplatestringAnoptionaltemplateforURLtoaccessvmagentinstancewiththegiven-promscrape.cluster.memberNumvalue.Every%doccurrenceinthetemplateissubstitutedwith-promscrape.cluster.memberNumaturlstovmagentinstancesresponsibleforscrapingthegiventargetat/service-discoverypage.Forexample-promscrape.cluster.memberURLTemplate='http://vmagent-%d:8429/targets'.Seehttps://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets for more details-promscrape.cluster.membersCountintThenumberofmembersinaclusterofscrapers.Eachmembermusthaveaunique-promscrape.cluster.memberNumintherange0...promscrape.cluster.membersCount-1.Eachmemberthenscrapesroughly1/Nofallthetargets.Bydefault,clusterscrapingisdisabled,i.e.asinglescraperscrapesallthetargets.Seehttps://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets for more info (default 1)-promscrape.cluster.namestringOptionalnameofthecluster.Ifmultiplevmagentclustersscrapethesametargets,theneachclustermusthaveuniquenameinordertoproperlyde-duplicatesamplesreceivedfromtheseclusters.Seehttps://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets for more info-promscrape.cluster.replicationFactorintThenumberofmembersinthecluster,whichscrapethesametargets.Ifthereplicationfactorisgreaterthan1,thenthededuplicationmustbeenabledatremotestorageside.Seehttps://docs.victoriametrics.com/victoriametrics/vmagent/#scraping-big-number-of-targets for more info (default 1)-promscrape.configstringOptionalpathtoPrometheusconfigfilewith'scrape_configs'sectioncontainingtargetstoscrape.Thepathcanpointtolocalfileandtohttpurl.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#how-to-scrape-prometheus-exporters-such-as-node-exporter for details-promscrape.config.dryRunChecks-promscrape.configfileforerrorsandunsupportedfieldsandthenexits.Returnsnon-zeroexitcodeonparsingerrorsandemitstheseerrorstostderr.Seealso-promscrape.config.strictParsecommand-lineflag.Pass-loggerLevel=ERRORifyoudon't need to see info messages in the output.-promscrape.config.strictParseWhethertodenyunsupportedfieldsin-promscrape.config.Settofalseinordertosilentlyskipunsupportedfields(defaulttrue)-promscrape.configCheckIntervaldurationIntervalforcheckingforchangesin-promscrape.configfile.Bydefault,thecheckingisdisabled.Seehowtoreload-promscrape.configfileathttps://docs.victoriametrics.com/victoriametrics/vmagent/#configuration-update-promscrape.consul.waitTimedurationWaittimeusedbyConsulservicediscovery.Defaultvalueisusedifnotset-promscrape.consulSDCheckIntervaldurationIntervalforcheckingforchangesinConsul.Thisworksonlyifconsul_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#consul_sd_configs for details (default 30s)-promscrape.consulagentSDCheckIntervaldurationIntervalforcheckingforchangesinConsulAgent.Thisworksonlyifconsulagent_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#consulagent_sd_configs for details (default 30s)-promscrape.digitaloceanSDCheckIntervaldurationIntervalforcheckingforchangesindigitalocean.Thisworksonlyifdigitalocean_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#digitalocean_sd_configs for details (default 1m0s)-promscrape.disableCompressionWhethertodisablesending'Accept-Encoding: gzip'requestheaderstoallthescrapetargets.ThismayreduceCPUusageonscrapetargetsatthecostofhighernetworkbandwidthutilization.Itispossibletoset'disable_compression: true'individuallypereach'scrape_config'sectionin'-promscrape.config'forfine-grainedcontrol-promscrape.disableKeepAliveWhethertodisableHTTPkeep-aliveconnectionswhenscrapingallthetargets.ThismaybeusefulwhentargetshasnosupportforHTTPkeep-aliveconnection.Itispossibletoset'disable_keepalive: true'individuallypereach'scrape_config'sectionin'-promscrape.config'forfine-grainedcontrol.NotethatdisablingHTTPkeep-alivemayincreaseloadonbothvmagentandscrapetargets-promscrape.discovery.concurrencyintThemaximumnumberofconcurrentrequeststoPrometheusautodiscoveryAPI(Consul,Kubernetes,etc.)(default100)-promscrape.discovery.concurrentWaitTimedurationThemaximumdurationforwaitingtoperformAPIrequestsifmorethan-promscrape.discovery.concurrencyrequestsaresimultaneouslyperformed(default1m0s)-promscrape.dnsSDCheckIntervaldurationIntervalforcheckingforchangesindns.Thisworksonlyifdns_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#dns_sd_configs for details (default 30s)-promscrape.dockerSDCheckIntervaldurationIntervalforcheckingforchangesindocker.Thisworksonlyifdocker_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#docker_sd_configs for details (default 30s)-promscrape.dockerswarmSDCheckIntervaldurationIntervalforcheckingforchangesindockerswarm.Thisworksonlyifdockerswarm_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#dockerswarm_sd_configs for details (default 30s)-promscrape.dropOriginalLabelsWhethertodroporiginallabelsforscrapetargetsat/targetsand/api/v1/targetspages.Thismaybeneededforreducingmemoryusagewhenoriginallabelsforbignumberofscrapetargetsoccupybigamountsofmemory.Notethatthisreducesdebuggabilityforimproperper-targetrelabelingconfigs(defaulttrue)-promscrape.ec2SDCheckIntervaldurationIntervalforcheckingforchangesinec2.Thisworksonlyifec2_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#ec2_sd_configs for details (default 1m0s)-promscrape.eurekaSDCheckIntervaldurationIntervalforcheckingforchangesineureka.Thisworksonlyifeureka_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#eureka_sd_configs for details (default 30s)-promscrape.fileSDCheckIntervaldurationIntervalforcheckingforchangesin'file_sd_config'.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#file_sd_configs for details (default 1m0s)-promscrape.gceSDCheckIntervaldurationIntervalforcheckingforchangesingce.Thisworksonlyifgce_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#gce_sd_configs for details (default 1m0s)-promscrape.hetznerSDCheckIntervaldurationIntervalforcheckingforchangesinHetznerAPI.Thisworksonlyifhetzner_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#hetzner_sd_configs for details (default 1m0s)-promscrape.httpSDCheckIntervaldurationIntervalforcheckingforchangesinhttpendpointservicediscovery.Thisworksonlyifhttp_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#http_sd_configs for details (default 1m0s)-promscrape.kubernetes.apiServerTimeoutdurationHowfrequentlytoreloadthefullstatefromKubernetesAPIserver(default30m0s)-promscrape.kubernetes.attachNamespaceMetadataAllWhethertosetattach_metadata.namespace=trueforallthekubernetes_sd_configsat-promscrape.config.Itispossibletosetattach_metadata.namespace=falseindividuallypereachkubernetes_sd_configs.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs-promscrape.kubernetes.attachNodeMetadataAllWhethertosetattach_metadata.node=trueforallthekubernetes_sd_configsat-promscrape.config.Itispossibletosetattach_metadata.node=falseindividuallypereachkubernetes_sd_configs.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs-promscrape.kubernetes.useHTTP2ClientWhethertouseHTTP/2clientforconnectiontoKubernetesAPIserver.ThismayreduceamountofconcurrentconnectionstoAPIserverwhenwatchingforabignumberofKubernetesobjects.-promscrape.kubernetesSDCheckIntervaldurationIntervalforcheckingforchangesinKubernetesAPIserver.Thisworksonlyifkubernetes_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#kubernetes_sd_configs for details (default 30s)-promscrape.kumaSDCheckIntervaldurationIntervalforcheckingforchangesinkumaservicediscovery.Thisworksonlyifkuma_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#kuma_sd_configs for details (default 30s)-promscrape.marathonSDCheckIntervaldurationIntervalforcheckingforchangesinMarathonRESTAPI.Thisworksonlyifmarathon_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#marathon_sd_configs for details (default 30s)-promscrape.maxDroppedTargetsintThemaximumnumberofdroppedTargetstoshowat/api/v1/targetspage.Increasethisvalueifyoursetupdropsmorescrapetargetsduringrelabelingandyouneedinvestigatinglabelsforallthedroppedtargets.Notethattheincreasednumberoftrackeddroppedtargetsmayresultinincreasedmemoryusage(default10000)-promscrape.maxResponseHeadersSizesizeThemaximumsizeofhttpresponseheadersfromPrometheusscrapetargetsSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default4096)-promscrape.maxScrapeSizesizeThemaximumsizeofscraperesponseinbytestoprocessfromPrometheustargets.Biggerresponsesarerejected.Seealsomax_scrape_sizeoptionathttps://docs.victoriametrics.com/victoriametrics/sd_configs/#scrape_configsSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default16777216)-promscrape.minResponseSizeForStreamParsesizeTheminimumtargetresponsesizeforautomaticswitchingtostreamparsingmode,whichcanreducememoryusage.Seehttps://docs.victoriametrics.com/victoriametrics/vmagent/#stream-parsing-modeSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default1000000)-promscrape.noStaleMarkersWhethertodisablesendingPrometheusstalemarkersformetricswhenscrapetargetdisappears.Thisoptionmayreducememoryusageifstalemarkersaren't needed for your setup. This option also disables populating the scrape_series_added metric. See https://prometheus.io/docs/concepts/jobs_instances/#automatically-generated-labels-and-time-series-promscrape.nomad.waitTimedurationWaittimeusedbyNomadservicediscovery.Defaultvalueisusedifnotset-promscrape.nomadSDCheckIntervaldurationIntervalforcheckingforchangesinNomad.Thisworksonlyifnomad_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#nomad_sd_configs for details (default 30s)-promscrape.openstackSDCheckIntervaldurationIntervalforcheckingforchangesinopenstackAPIserver.Thisworksonlyifopenstack_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#openstack_sd_configs for details (default 30s)-promscrape.ovhcloudSDCheckIntervaldurationIntervalforcheckingforchangesinOVHCloudAPI.Thisworksonlyifovhcloud_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#ovhcloud_sd_configs for details (default 30s)-promscrape.puppetdbSDCheckIntervaldurationIntervalforcheckingforchangesinPuppetDBAPI.Thisworksonlyifpuppetdb_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#puppetdb_sd_configs for details (default 30s)-promscrape.seriesLimitPerTargetintOptionallimitonthenumberofuniquetimeseriesasinglescrapetargetcanexpose.Seehttps://docs.victoriametrics.com/victoriametrics/vmagent/#cardinality-limiter for more info-promscrape.streamParseWhethertoenablestreamparsingformetricsobtainedfromscrapetargets.Thismaybeusefulforreducingmemoryusagewhenmillionsofmetricsareexposedpereachscrapetarget.Itispossibletoset'stream_parse: true'individuallypereach'scrape_config'sectionin'-promscrape.config'forfine-grainedcontrol-promscrape.suppressDuplicateScrapeTargetErrorsWhethertosuppress'duplicate scrape target'errors;seehttps://docs.victoriametrics.com/victoriametrics/vmagent/#troubleshooting for details-promscrape.suppressScrapeErrorsWhethertosuppressscrapeerrorslogging.Thelasterrorforeachtargetisalwaysavailableat'/targets'pageevenifscrapeerrorsloggingissuppressed.Seealso-promscrape.suppressScrapeErrorsDelay-promscrape.suppressScrapeErrorsDelaydurationThedelayforsuppressingrepeatedscrapeerrorsloggingpereachscrapetargets.Thismaybeusedforreducingthenumberofloglinesrelatedtoscrapeerrors.Seealso-promscrape.suppressScrapeErrors-promscrape.vultrSDCheckIntervaldurationIntervalforcheckingforchangesinVultr.Thisworksonlyifvultr_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#vultr_sd_configs for details (default 30s)-promscrape.yandexcloudSDCheckIntervaldurationIntervalforcheckingforchangesinYandexCloudAPI.Thisworksonlyifyandexcloud_sd_configsisconfiguredin'-promscrape.config'file.Seehttps://docs.victoriametrics.com/victoriametrics/sd_configs/#yandexcloud_sd_configs for details (default 30s)-pushmetrics.disableCompressionWhethertodisablerequestbodycompressionwhenpushingmetricstoevery-pushmetrics.url-pushmetrics.extraLabelarrayOptionallabelstoaddtometricspushedtoevery-pushmetrics.url.Forexample,-pushmetrics.extraLabel='instance="foo"'addsinstance="foo"labeltoallthemetricspushedtoevery-pushmetrics.urlSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Eacharrayitemcancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-pushmetrics.headerarrayOptionalHTTPrequestheadertosendtoevery-pushmetrics.url.Forexample,-pushmetrics.header='Authorization: Basic foobar'adds'Authorization: Basic foobar'headertoeveryrequesttoevery-pushmetrics.urlSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Eacharrayitemcancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-pushmetrics.intervaldurationIntervalforpushingmetricstoevery-pushmetrics.url(default10s)-pushmetrics.urlarrayOptionalURLtopushmetricsexposedat/metricspage.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#push-metrics . By default, metrics exposed at /metrics page aren't pushed to any remote storageSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Eacharrayitemcancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-relabelConfigstringOptionalpathtoafilewithrelabelingrules,whichareappliedtoalltheingestedmetrics.Thepathcanpointeithertolocalfileortohttpurl.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#relabeling for details. The config is reloaded on SIGHUP signal-reloadAuthKeyvalueAuthkeyfor/-/reloadhttpendpoint.ItmustbepassedviaauthKeyqueryarg.ItoverrideshttpAuth.*settings.Flagvaluecanbereadfromthegivenfilewhenusing-reloadAuthKey=file:///abs/path/to/fileor-reloadAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-reloadAuthKey=http://host/pathor-reloadAuthKey=https://host/path-retentionFilterarrayRetentionfilterintheformat'filter:retention'.Forexample,'{env="dev"}:3d'configurestheretentionfortimeserieswithenv="dev"labelto3days.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#retention-filters for details. This flag is available only in VictoriaMetrics enterprise. See https://docs.victoriametrics.com/victoriametrics/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Eacharrayitemcancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-retentionPeriodvalueDatawithtimestampsoutsidetheretentionPeriodisautomaticallydeleted.TheminimumretentionPeriodis24hor1d.Seealso-retentionFilterThefollowingoptionalsuffixesaresupported:s(second),h(hour),d(day),w(week),y(year).Ifsuffixisn't set, then the duration is counted in months (default 1)-retentionTimezoneOffsetdurationTheoffsetforperformingindexdbrotation.Ifsetto0,thentheindexdbrotationisperformedat4amUTCtimepereach-retentionPeriod.Ifsetto2h,thentheindexdbrotationisperformedat4amEETtime(thetimezonewith+2hoffset)-search.cacheTimestampOffsetdurationThemaximumdurationsincethecurrenttimeforresponsedata,whichisalwaysqueriedfromtheoriginalrawdata,withoutusingtheresponsecache.IncreasethisvalueifyouseegapsinresponsesduetotimesynchronizationissuesbetweenVictoriaMetricsanddatasources.Seealso-search.disableAutoCacheReset(default5m0s)-search.disableAutoCacheResetWhethertodisableautomaticresponsecacheresetifasamplewithtimestampoutside-search.cacheTimestampOffsetisinsertedintoVictoriaMetrics-search.disableCacheWhethertodisableresponsecaching.Thismaybeusefulwheningestinghistoricaldata.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#backfilling . See also -search.resetRollupResultCacheOnStartup-search.disableImplicitConversionWhethertoreturnanerrorforqueriesthatrelyonimplicitsubqueryconversions,seehttps://docs.victoriametrics.com/victoriametrics/metricsql/#subqueries for details. See also -search.logImplicitConversion.-search.graphiteMaxPointsPerSeriesintThemaximumnumberofpointsperseriesGraphiterenderAPIcanreturn(default1000000)-search.graphiteStorageStepdurationTheintervalbetweendatapointsstoredinthedatabase.ItisusedatGraphiteRenderAPIhandlerfornormalizingtheintervalbetweendatapointsincaseitisn't normalized. It can be overridden by sending 'storage_step' query arg to /render API or by sending the desired interval via 'Storage-Step' http header during querying /render API (default 10s)-search.ignoreExtraFiltersAtLabelsAPIWhethertoignorematch[],extra_filters[]andextra_labelqueryargsat/api/v1/labelsand/api/v1/label/.../values.ThismaybeusefulfordecreasingloadonVictoriaMetricswhenextrafiltersmatchtoomanytimeseries.Thedownsideisthatsuperfluouslabelsorseriescouldbereturned,whichdonotmatchtheextrafilters.Seealso-search.maxLabelsAPISeriesand-search.maxLabelsAPIDuration-search.inmemoryBufSizeBytessizeSizeforin-memorydatablocksusedduringprocessingsearchrequests.Bydefault,thesizeisautomaticallycalculatedbasedonavailablememory.Adjustthisflagvalueifyouobservethatvm_tmp_blocks_max_inmemory_file_size_bytesmetricconstantlyshowsmuchhighervaluesthanvm_tmp_blocks_inmemory_file_size_bytes.Seehttps://github.com/VictoriaMetrics/VictoriaMetrics/pull/6851Supportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-search.latencyOffsetdurationThetimewhendatapointsbecomevisibleinqueryresultsafterthecollection.Itcanbeoverriddenonper-querybasisvialatency_offsetarg.Toosmallvaluecanresultinincompletelastpointsforqueryresults(default30s)-search.logImplicitConversionWhethertologquerieswithimplicitsubqueryconversions,seehttps://docs.victoriametrics.com/victoriametrics/metricsql/#subqueries for details. Such conversion can be disabled using -search.disableImplicitConversion.-search.logQueryMemoryUsagesizeLogqueryandincrementvm_memory_intensive_queries_totalmetriceachtimethequeryrequiresmorememorythanspecifiedbythisflag.Thismayhelpdetectingandoptimizingheavyqueries.Queryloggingisdisabledbydefault.Seealso-search.logSlowQueryDurationand-search.maxMemoryPerQuerySupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-search.logSlowQueryDurationdurationLogquerieswithexecutiontimeexceedingthisvalue.Zerodisablesslowquerylogging.Seealso-search.logQueryMemoryUsage(default5s)-search.logSlowQueryStatsdurationLogquerystatisticsifexecutiontimeexceedingthisvalue-seehttps://docs.victoriametrics.com/victoriametrics/query-stats.Zerodisablesslowquerystatisticslogging.ThisflagisavailableonlyinVictoriaMetricsenterprise.Seehttps://docs.victoriametrics.com/victoriametrics/enterprise/(default5s)-search.logSlowQueryStatsHeadersarrayHTTPrequestheaderkeystologforqueriesexceedingthethresholdsetby-search.logSlowQueryStats.Case-insensitive.Bydefault,noheadersarelogged.ThisflagisavailableonlyinVictoriaMetricsenterprise.Seehttps://docs.victoriametrics.com/victoriametrics/query-stats/#log-fields for details and examples.Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Eacharrayitemcancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-search.maxBinaryOpPushdownLabelValuesintThemaximumnumberofvaluesforalabelinthefirstexpressionthatcanbeextractedasacommonlabelfilterandpusheddowntothesecondexpressioninabinaryoperation.Alargervaluemakesthepushed-downfiltermorecomplexbutfewertimeserieswillbereturned.Thisflagisusefulwhenselectivelabel(e.g.,'instance')containsnumerousvalues,andstorageresourcesareabundant.(default100)-search.maxConcurrentRequestsintThemaximumnumberofconcurrentsearchrequests.Itshouldn't be high, since a single request can saturate all the CPU cores, while many concurrently executed requests may require high amounts of memory. See also -search.maxQueueDuration and -search.maxMemoryPerQuery (default vmselect.getDefaultMaxConcurrentRequests())-search.maxDeleteDurationdurationThemaximumdurationfor/api/v1/admin/tsdb/delete_seriescall(default5m0s)-search.maxDeleteSeriesintThemaximumnumberoftimeseries,whichcanbedeletedusing/api/v1/admin/tsdb/delete_series.Thisoptionallowslimitingmemoryusage(default1000000)-search.maxExportDurationdurationThemaximumdurationfor/api/v1/exportcall(default720h0m0s)-search.maxExportSeriesintThemaximumnumberoftimeseries,whichcanbereturnedfrom/api/v1/export*APIs.Thisoptionallowslimitingmemoryusage(default10000000)-search.maxFederateSeriesintThemaximumnumberoftimeseries,whichcanbereturnedfrom/federate.Thisoptionallowslimitingmemoryusage(default1000000)-search.maxGraphitePathExpressionLenintThemaximumlengthofpathExpressionfieldinGraphiteseries.Longerexpressionsaretruncatedtopreventmemoryexhaustiononcomplexnestedqueries.Setto0todisabletruncation.(default1024)-search.maxGraphiteSeriesintThemaximumnumberoftimeseries,whichcanbescannedduringqueriestoGraphiteRenderAPI.Seehttps://docs.victoriametrics.com/victoriametrics/integrations/graphite/#render-api (default 300000)-search.maxGraphiteTagKeysintThemaximumnumberoftagkeysreturnedfromGraphiteAPI,whichreturnstags.Seehttps://docs.victoriametrics.com/victoriametrics/integrations/graphite/#tags-api (default 100000)-search.maxGraphiteTagValuesintThemaximumnumberoftagvaluesreturnedfromGraphiteAPI,whichreturnstagvalues.Seehttps://docs.victoriametrics.com/victoriametrics/integrations/graphite/#tags-api (default 100000)-search.maxLabelsAPIDurationdurationThemaximumdurationfor/api/v1/labels,/api/v1/label/.../valuesand/api/v1/seriesrequests.Seealso-search.maxLabelsAPISeriesand-search.ignoreExtraFiltersAtLabelsAPI(default5s)-search.maxLabelsAPISeriesintThemaximumnumberoftimeseries,whichcouldbescannedwhensearchingforthematchingtimeseriesat/api/v1/labelsand/api/v1/label/.../values.ThisoptionallowslimitingmemoryusageandCPUusage.Seealso-search.maxLabelsAPIDuration,-search.maxTagKeys,-search.maxTagValuesand-search.ignoreExtraFiltersAtLabelsAPI(default1000000)-search.maxLookbackdurationSynonymto-query.lookback-deltafromPrometheus.Thevalueisdynamicallydetectedfromintervalbetweentimeseriesdatapointsifnotset.Itcanbeoverriddenonper-querybasisviamax_lookbackarg.Seealso'-search.maxStalenessInterval'flag,whichhasthesamemeaningduetohistoricalreasons-search.maxMemoryPerQuerysizeThemaximumamountsofmemoryasinglequerymayconsume.Queriesrequiringmorememoryarerejected.Thetotalmemorylimitforconcurrentlyexecutedqueriescanbeestimatedas-search.maxMemoryPerQuerymultipliedby-search.maxConcurrentRequests.Seealso-search.logQueryMemoryUsageSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-search.maxPointsPerTimeseriesintThemaximumpointsperasingletimeseriesreturnedfrom/api/v1/query_range.Thisoptiondoesn't limit the number of scanned raw samples in the database. The main purpose of this option is to limit the number of per-series points returned to graphing UI such as VMUI or Grafana. There is no sense in setting this limit to values bigger than the horizontal resolution of the graph. See also -search.maxResponseSeries (default 30000)-search.maxPointsSubqueryPerTimeseriesintThemaximumnumberofpointsperseries,whichcanbegeneratedbysubquery.Seehttps://valyala.medium.com/prometheus-subqueries-in-victoriametrics-9b1492b720b3(default100000)-search.maxQueryDurationdurationThemaximumdurationforqueryexecution.Itcanbeoverriddentoasmallervalueonaper-querybasisvia'timeout'queryarg(default30s)-search.maxQueryLensizeThemaximumsearchquerylengthinbytesSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default16384)-search.maxQueueDurationdurationThemaximumtimetherequestwaitsforexecutionwhen-search.maxConcurrentRequestslimitisreached;seealso-search.maxQueryDuration(default10s)-search.maxResponseSeriesintThemaximumnumberoftimeserieswhichcanbereturnedfrom/api/v1/queryand/api/v1/query_range.Thelimitisdisabledifitequalsto0.Seealso-search.maxPointsPerTimeseriesand-search.maxUniqueTimeseries-search.maxSamplesPerQueryintThemaximumnumberofrawsamplesasinglequerycanprocessacrossalltimeseries.Thisprotectsfromheavyqueries,whichselectunexpectedlyhighnumberofrawsamples.Seealso-search.maxSamplesPerSeries(default1000000000)-search.maxSamplesPerSeriesintThemaximumnumberofrawsamplesasinglequerycanscanpereachtimeseries.Thisoptionallowslimitingmemoryusage(default30000000)-search.maxSeriesintThemaximumnumberoftimeseries,whichcanbereturnedfrom/api/v1/series.Thisoptionallowslimitingmemoryusage(default30000)-search.maxSeriesPerAggrFuncintThemaximumnumberoftimeseriesanaggregateMetricsQLfunctioncangenerate(default1000000)-search.maxStalenessIntervaldurationThemaximumintervalforstalenesscalculations.Bydefault,itisautomaticallycalculatedfromthemedianintervalbetweensamples.ThisflagcouldbeusefulfortuningPrometheusdatamodelclosertoInflux-styledatamodel.Seehttps://prometheus.io/docs/prometheus/latest/querying/basics/#staleness for details. See also '-search.setLookbackToStep' flag-search.maxStatusRequestDurationdurationThemaximumdurationfor/api/v1/status/*requests(default5m0s)-search.maxStepForPointsAdjustmentdurationThemaximumstepwhen/api/v1/query_rangehandleradjustspointswithtimestampscloserthan-search.latencyOffsettothecurrenttime.Theadjustmentisneededbecausesuchpointsmaycontainincompletedata(default1m0s)-search.maxTSDBStatusSeriesintThemaximumnumberoftimeseries,whichcanbeprocessedduringthecallto/api/v1/status/tsdb.Thisoptionallowslimitingmemoryusage(default10000000)-search.maxTSDBStatusTopNSeriesintThemaximumvalueof'topN'argumentthatcanbepassedto/api/v1/status/tsdbAPI.Thisoptionallowslimitingmemoryusage.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#tsdb-stats (default 1000)-search.maxTagKeysintThemaximumnumberoftagkeysreturnedfrom/api/v1/labels.Seealso-search.maxLabelsAPISeriesand-search.maxLabelsAPIDuration(default100000)-search.maxTagValueSuffixesPerSearchintThemaximumnumberoftagvaluesuffixesreturnedfrom/metrics/find(default100000)-search.maxTagValuesintThemaximumnumberoftagvaluesreturnedfrom/api/v1/label/<label_name>/values.Seealso-search.maxLabelsAPISeriesand-search.maxLabelsAPIDuration(default100000)-search.maxUniqueTimeseriesintThemaximumnumberofuniquetimeseries,whichcanbeselectedduring/api/v1/queryand/api/v1/query_rangequeries.Thisoptionallowslimitingmemoryusage.Whensettozero,thelimitisautomaticallycalculatedbasedon-search.maxConcurrentRequests(inverselyproportional)andmemoryavailabletotheprocess(proportional).-search.maxWorkersPerQueryintThemaximumnumberofCPUcoresasinglequerycanuse.Thedefaultvalueshouldworkgoodformostcases.Theflagcanbesettolowervaluesforimprovingperformanceofbignumberofconcurrentlyexecutedqueries.Theflagcanbesettobiggervaluesforimprovingperformanceofheavyqueries,whichscanbignumberoftimeseries(>10K)and/orbignumberofsamples(>100M).ThereisnosenseinsettingthisflagtovaluesbiggerthanthenumberofCPUcoresavailableonthesystem(defaultnetstorage.defaultMaxWorkersPerQuery())-search.minStalenessIntervaldurationTheminimumintervalforstalenesscalculations.Thisflagcouldbeusefulforremovinggapsongraphsgeneratedfromtimeserieswithirregularintervalsbetweensamples.Seealso'-search.maxStalenessInterval'-search.minWindowForInstantRollupOptimizationdurationEnablecache-basedoptimizationforrepeatedqueriesto/api/v1/query(akainstantqueries),whichcontainrollupfunctionswithlookbehindwindowexceedingthegivenvalue(default3h0m0s)-search.noStaleMarkersSetthisflagtotrueifthedatabasedoesn't contain Prometheus stale markers, so there is no need in spending additional CPU time on its handling. Staleness markers may exist only in data obtained from Prometheus scrape targets-search.queryStats.lastQueriesCountintQuerystatsfor/api/v1/status/top_queriesistrackedonthisnumberoflastqueries.Zerovaluedisablesquerystatstracking(default20000)-search.queryStats.minQueryDurationdurationTheminimumdurationforqueriestotrackinquerystatsat/api/v1/status/top_queries.Querieswithlowerdurationareignoredinquerystats(default1ms)-search.resetCacheAuthKeyvalueOptionalauthKeyforresettingrollupcachevia/internal/resetRollupResultCachecall.ItcouldbepassedviaauthKeyqueryarg.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-search.resetCacheAuthKey=file:///abs/path/to/fileor-search.resetCacheAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-search.resetCacheAuthKey=http://host/pathor-search.resetCacheAuthKey=https://host/path-search.resetRollupResultCacheOnStartupWhethertoresetrollupresultcacheonstartup.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#rollup-result-cache . See also -search.disableCache-search.setLookbackToStepWhethertofixlookbackintervalto'step'queryargvalue.Ifsettotrue,thequerymodelbecomesclosertoInfluxDBdatamodel.Ifsettotrue,then-search.maxLookbackand-search.maxStalenessIntervalareignored-search.treatDotsAsIsInRegexpsWhethertotreatdotsasisinregexplabelfiltersusedinqueries.Forexample,foo{bar=~"a.b.c"}willbeautomaticallyconvertedtofoo{bar=~"a\\.b\\.c"},i.e.allthedotsinregexpfilterswillbeautomaticallyescapedinordertomatchonlydotcharinsteadofmatchinganychar.Dotsin".+",".*"and".{n}"regexpsaren't escaped. This option is DEPRECATED in favor of {__graphite__="a.*.c"} syntax for selecting metrics matching the given Graphite metrics filter-secret.flagsarrayComma-separatedlistofflagnameswithsecretvalues.Valuesfortheseflagsarehiddeninlogsandon/metricspageSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Eacharrayitemcancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-selfScrapeInstancestringValuefor'instance'label,whichisaddedtoself-scrapedmetrics(default"self")-selfScrapeIntervaldurationIntervalforself-scrapingownmetricsat/metricspage-selfScrapeJobstringValuefor'job'label,whichisaddedtoself-scrapedmetrics(default"victoria-metrics")-smallMergeConcurrencyintDeprecated:thisflagdoesnothing-snapshotAuthKeyvalueauthKey,whichmustbepassedinquerystringto/snapshot*pages.Itoverrides-httpAuth.*Flagvaluecanbereadfromthegivenfilewhenusing-snapshotAuthKey=file:///abs/path/to/fileor-snapshotAuthKey=file://./relative/path/to/file.Flagvaluecanbereadfromthegivenhttp/httpsurlwhenusing-snapshotAuthKey=http://host/pathor-snapshotAuthKey=https://host/path-snapshotCreateTimeoutdurationDeprecated:thisflagdoesnothing-snapshotsMaxAgevalueAutomaticallydeletesnapshotsolderthan-snapshotsMaxAgeifitissettonon-zeroduration.MakesurethatbackupprocesshasenoughtimetofinishthebackupbeforethecorrespondingsnapshotisautomaticallydeletedThefollowingoptionalsuffixesaresupported:s(second),h(hour),d(day),w(week),y(year).Ifsuffixisn't set, then the duration is counted in months (default 3d)-sortLabelsWhethertosortlabelsforincomingsamplesbeforewritingthemtostorage.Thismaybeneededforreducingmemoryusageatstoragewhentheorderoflabelsinincomingsamplesisrandom.Forexample,ifm{k1="v1",k2="v2"}maybesentasm{k2="v2",k1="v1"}.Enabledsortingforlabelscanslowdowningestionperformanceabit-storage.cacheSizeIndexDBDataBlockssizeOverridesmaxsizeforindexdb/dataBlockscache.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#cache-tuningSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-storage.cacheSizeIndexDBDataBlocksSparsesizeOverridesmaxsizeforindexdb/dataBlocksSparsecache.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#cache-tuningSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-storage.cacheSizeIndexDBDateMetricIDsizeOverridesmaxsizeforindexdb/date_metricIDcache.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#cache-tuningSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-storage.cacheSizeIndexDBIndexBlockssizeOverridesmaxsizeforindexdb/indexBlockscache.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#cache-tuningSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-storage.cacheSizeIndexDBTagFilterssizeOverridesmaxsizeforindexdb/tagFiltersToMetricIDscache.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#cache-tuningSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-storage.cacheSizeMetricNamesStatssizeOverridesmaxsizeforstorage/metricNamesStatsTrackercache.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#cache-tuningSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-storage.cacheSizeStorageMetricNamesizeOverridesmaxsizeforstorage/metricNamecache.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#cache-tuningSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-storage.cacheSizeStorageTSIDsizeOverridesmaxsizeforstorage/tsidcache.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#cache-tuningSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-storage.finalDedupScheduleCheckIntervaldurationTheintervalforcheckingwhenfinaldeduplicationprocessshouldbestarted.Storageunconditionallyadds25%jittertotheintervalvalueoneachcheckevaluation.Changingtheintervaltothebiggervaluesmaydelaydownsampling,deduplicationforhistoricaldata.Seealsohttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#deduplication (default 1h0m0s)-storage.idbPrefillStartdurationSpecifieshowearlyVictoriaMetricsstartspre-fillingindexDBrecordsbeforeindexDBrotation.Startingthepre-fillprocessearliercanhelpreduceresourceusagespikesduringrotation.Inmostcases,thisvalueshouldnotbechanged.Themaximumallowedvalueis23h.(default1h0m0s)-storage.maxDailySeriesintThemaximumnumberofuniqueseriescanbeaddedtothestorageduringthelast24hours.Excessseriesareloggedanddropped.Thiscanbeusefulforlimitingserieschurnrate.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#cardinality-limiter . See also -storage.maxHourlySeries-storage.maxHourlySeriesintThemaximumnumberofuniqueseriescanbeaddedtothestorageduringthelasthour.Excessseriesareloggedanddropped.Thiscanbeusefulforlimitingseriescardinality.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#cardinality-limiter . See also -storage.maxDailySeries-storage.maxMetadataStorageSizesizeOverridesmaxsizeformetricsmetadataentriesin-memorystorage.Ifsetto0oranegativevalue,defaultsto1%ofallowedmemory.Supportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default0)-storage.minFreeDiskSpaceBytessizeTheminimumfreediskspaceat-storageDataPathafterwhichthestoragestopsacceptingnewdataSupportsthefollowingoptionalsuffixesforsizevalues:KB,MB,GB,TB,KiB,MiB,GiB,TiB(default10000000)-storage.trackMetricNamesStatsWhethertotrackingestandqueryrequestsfortimeseriesmetricnames.Thisfeatureallowstotrackmetricnamesunusedatqueryrequests.Seehttps://docs.victoriametrics.com/victoriametrics/single-server-victoriametrics/#track-ingested-metrics-usage (default true)-storageDataPathstringPathtostoragedata(default"victoria-metrics-data")-streamAggr.configstringOptionalpathtofilewithstreamaggregationconfig.Seehttps://docs.victoriametrics.com/victoriametrics/stream-aggregation/.Seealso-streamAggr.keepInput,-streamAggr.dropInputand-streamAggr.dedupInterval-streamAggr.dedupIntervaldurationInputsamplesarede-duplicatedwiththisintervalbeforeoptionalaggregationwith-streamAggr.config.Seealso-streamAggr.dropInputLabelsand-dedup.minScrapeIntervalandhttps://docs.victoriametrics.com/victoriametrics/stream-aggregation/#deduplication-streamAggr.dropInputWhethertodropinputsamplesthatnotmatchinganyrulein-streamAggr.config.Bydefault,onlymatchedrawsamplesaredropped,whileunmatchedsamplesarewrittentotheremotestorage.Seealso-streamAggr.keepInputandhttps://docs.victoriametrics.com/victoriametrics/stream-aggregation/-streamAggr.dropInputLabelsarrayAnoptionallistoflabelstodropfromsamplesbeforestreamde-duplicationandaggregation.Seehttps://docs.victoriametrics.com/victoriametrics/stream-aggregation/#dropping-unneeded-labelsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Eacharrayitemcancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-streamAggr.enableWindowsEnablesaggregationwithinfixedwindowsforallaggregators.Thisallowstogetmorepreciseresults,butimpactsresourceusageasitrequirestwicemorememorytostoretwostates.Seehttps://docs.victoriametrics.com/victoriametrics/stream-aggregation/#aggregation-windows.-streamAggr.ignoreFirstIntervalsintNumberofaggregationintervalstoskipafterthestart.Increasethisvalueifyouobserveincorrectaggregationresultsafterrestarts.Itcouldbecausedbyreceivingunordereddelayeddatafromclientspushingdataintothedatabase.Seehttps://docs.victoriametrics.com/victoriametrics/stream-aggregation/#ignore-aggregation-intervals-on-start-streamAggr.ignoreOldSamplesWhethertoignoreinputsampleswitholdtimestampsoutsidethecurrentaggregationinterval.Seehttps://docs.victoriametrics.com/victoriametrics/stream-aggregation/#ignoring-old-samples-streamAggr.keepInputWhethertokeepinputsamplesthatmatchanyrulein-streamAggr.config.Bydefault,matchedrawsamplesareaggregatedanddropped,whileunmatchedsamplesarewrittentotheremotestorage.Seealso-streamAggr.dropInputandhttps://docs.victoriametrics.com/victoriametrics/stream-aggregation/-tlsarrayWhethertoenableTLSforincomingHTTPrequestsatthegiven-httpListenAddr(akahttps).-tlsCertFileand-tlsKeyFilemustbesetif-tlsisset.Seealso-mtlsSupportsarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Emptyvaluesaresettofalse.-tlsAutocertCacheDirstringDirectorytostoreTLScertificatesissuedviaLet's Encrypt. Certificates are lost on restarts if this flag isn'tset.ThisflagisavailableonlyinEnterprisebinaries.Seehttps://docs.victoriametrics.com/victoriametrics/enterprise/-tlsAutocertEmailstringContactemailfortheissuedLet's Encrypt TLS certificates. See also -tlsAutocertHosts and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/victoriametrics/enterprise/-tlsAutocertHostsarrayOptionalhostnamesforautomaticissuingofLet's Encrypt TLS certificates. These hostnames must be reachable at -httpListenAddr . The -httpListenAddr must listen tcp port 443 . The -tlsAutocertHosts overrides -tlsCertFile and -tlsKeyFile . See also -tlsAutocertEmail and -tlsAutocertCacheDir . This flag is available only in Enterprise binaries. See https://docs.victoriametrics.com/victoriametrics/enterprise/Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Eacharrayitemcancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsCertFilearrayPathtofilewithTLScertificateforthecorresponding-httpListenAddrif-tlsisset.PreferECDSAcertsinsteadofRSAcertsasRSAcertsareslower.Theprovidedcertificatefileisautomaticallyre-readeverysecond,soitcanbedynamicallyupdated.Seealso-tlsAutocertHostsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Eacharrayitemcancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsCipherSuitesarrayOptionallistofTLSciphersuitesforincomingrequestsoverHTTPSif-tlsisset.Seethelistofsupportedciphersuitesathttps://pkg.go.dev/crypto/tls#pkg-constantsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Eacharrayitemcancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsKeyFilearrayPathtofilewithTLSkeyforthecorresponding-httpListenAddrif-tlsisset.Theprovidedkeyfileisautomaticallyre-readeverysecond,soitcanbedynamicallyupdated.Seealso-tlsAutocertHostsSupportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Eacharrayitemcancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-tlsMinVersionarrayOptionalminimumTLSversiontouseforthecorresponding-httpListenAddrif-tlsisset.Supportedvalues:TLS10,TLS11,TLS12,TLS13Supportsanarrayofvaluesseparatedbycommaorspecifiedviamultipleflags.Eacharrayitemcancontaincommainsidesingle-quotedordouble-quotedstring,{},[]and()braces.-usePromCompatibleNamingWhethertoreplacecharactersunsupportedbyPrometheuswithunderscoresintheingestedmetricnamesandlabelnames.Forexample,foo.bar{a.b='c'}istransformedintofoo_bar{a_b='c'}duringdataingestionifthisflagisset.Seehttps://prometheus.io/docs/concepts/data_model/#metric-names-and-labels-versionShowVictoriaMetricsversion-vmalert.proxyURLstringOptionalURLforproxyingrequeststovmalert.Forexample,if-vmalert.proxyURL=http://vmalert:8880,thenalertingAPIrequestssuchas/api/v1/rulesfromGrafanawillbeproxiedtohttp://vmalert:8880/api/v1/rules-vmui.customDashboardsPathstringOptionalpathtovmuidashboards.Seehttps://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmui/packages/vmui/public/dashboards-vmui.defaultTimezonestringThedefaulttimezonetobeusedinvmui.TimezonemustbeavalidIANATimeZone.Forexample:America/New_York,Europe/Berlin,Etc/GMT+3orLocal

Section below contains backward-compatible anchors for links that were moved or renamed.

Prometheus setup#

Moved tointegrations/prometheus.

Grafana setup#

Moved tointegrations/grafana.

How to send data from DataDog agent#

Moved tointegrations/datadog.

How to send data from InfluxDB-compatible agents such asTelegraf#

Moved tointegrations/influxdb.

How to send data from Graphite-compatible agents such asStatsD#

Moved tointegrations/graphite#ingesting.

Querying Graphite data#

Moved tointegrations/graphite#querying.

Selecting Graphite metrics#

Moved tointegrations/graphite/#selecting-graphite-metrics.

How to send data from OpenTSDB-compatible agents#

Moved tointegrations/opentsdb.

Sending data viatelnet put protocol#

Moved tointegrations/opentsdb#sending-data-via-telnet.

Sending OpenTSDB data via HTTP/api/put requests#

Moved tointegrations/opentsdb#sending-data-via-http.

How to send data from NewRelic agent#

Moved tointegrations/newrelic.

Graphite API usage#

Moved tointegrations/graphite/#graphite-api-usage.

Graphite Render API usage#

Moved tointegrations/graphite/#render-api.

Graphite Metrics API usage#

Moved tointegrations/graphite/#metrics-api.

Graphite Tags API usage#

Moved tointegrations/graphite/#tags-api.

Integrations#

Moved tointegrations.

PreviousQuick startNextCluster version