VictoriaMetrics Cheat Sheet

Overview

VictoriaMetrics is a high-performance, cost-effective time series database designed as a long-term remote storage for Prometheus. It offers significant advantages over vanilla Prometheus: 5-10x better data compression, lower memory usage, faster queries on large datasets, and native support for multi-tenancy. VictoriaMetrics implements the Prometheus remote write and read APIs, PromQL (with MetricsQL extensions), Graphite, InfluxDB line protocol, OpenTSDB, and DataDog protocols, making it a versatile metrics backend.

VictoriaMetrics comes in two editions: single-node (a single binary handling ingestion, storage, and queries) and cluster version (with separate vmstorage, vminsert, and vmselect components for horizontal scalability). The single-node version can handle millions of metrics per second on modest hardware. Additional components include vmagent (metrics scraping and forwarding), vmalert (alerting and recording rules), vmauth (authentication proxy), and vmctl (data migration tool). Its MetricsQL query language extends PromQL with useful functions like range_median, histogram_quantiles, and label manipulation.

Installation

Single-Node Binary

# Download
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.101.0/victoria-metrics-linux-amd64-v1.101.0.tar.gz
tar xzf victoria-metrics-linux-amd64-v1.101.0.tar.gz
sudo mv victoria-metrics-prod /usr/local/bin/

# Start with default settings
victoria-metrics-prod \
  -storageDataPath=/var/lib/victoriametrics \
  -retentionPeriod=12 \
  -httpListenAddr=:8428

Docker

docker run -d --name victoriametrics \
  -p 8428:8428 \
  -v vmdata:/storage \
  victoriametrics/victoria-metrics:v1.101.0 \
  -storageDataPath=/storage \
  -retentionPeriod=12

Helm (Kubernetes)

helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update

# Single-node
helm install victoria-metrics vm/victoria-metrics-single \
  --namespace monitoring --create-namespace

# Cluster version
helm install victoria-metrics vm/victoria-metrics-cluster \
  --namespace monitoring --create-namespace

Install vmagent

wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.101.0/vmutils-linux-amd64-v1.101.0.tar.gz
tar xzf vmutils-linux-amd64-v1.101.0.tar.gz
sudo mv vmagent-prod /usr/local/bin/
sudo mv vmalert-prod /usr/local/bin/
sudo mv vmctl-prod /usr/local/bin/

Configuration

Prometheus Remote Write to VictoriaMetrics

# prometheus.yml
remote_write:
  - url: http://victoriametrics:8428/api/v1/write

vmagent Configuration

# vmagent-config.yaml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node1:9100', 'node2:9100', 'node3:9100']

  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true

# Run vmagent
vmagent-prod \
  -promscrape.config=vmagent-config.yaml \
  -remoteWrite.url=http://victoriametrics:8428/api/v1/write

Key Startup Flags

Flag	Default	Description
`-storageDataPath`	`victoria-metrics-data`	Data directory
`-retentionPeriod`	`1` (month)	Data retention (1=1 month, 12=1 year)
`-httpListenAddr`	`:8428`	HTTP listen address
`-maxLabelsPerTimeseries`	`30`	Max labels per series
`-search.maxUniqueTimeseries`	`300000`	Max unique series per query
`-memory.allowedPercent`	`60`	Max RAM percentage
`-dedup.minScrapeInterval`	`0s`	Deduplication interval

Core API Endpoints

Endpoint	Description
`GET /api/v1/query`	Instant PromQL/MetricsQL query
`GET /api/v1/query_range`	Range query
`POST /api/v1/write`	Prometheus remote write
`GET /api/v1/series`	Find matching series
`GET /api/v1/labels`	List label names
`GET /api/v1/label/{name}/values`	List label values
`POST /api/v1/import`	Import JSON data
`POST /api/v1/import/csv`	Import CSV data
`GET /api/v1/export`	Export raw data
`GET /api/v1/status/tsdb`	TSDB stats
`GET /metrics`	Internal metrics

Query Examples

# Instant query
curl 'http://vm:8428/api/v1/query?query=up'

# Range query
curl 'http://vm:8428/api/v1/query_range?query=rate(http_requests_total[5m])&start=2024-01-01T00:00:00Z&end=2024-01-02T00:00:00Z&step=60s'

# Series count
curl 'http://vm:8428/api/v1/status/tsdb'

# Delete series
curl -X POST 'http://vm:8428/api/v1/admin/tsdb/delete_series?match[]=process_cpu_seconds_total'

# Force merge (compaction)
curl -X POST 'http://vm:8428/internal/force_merge'

MetricsQL Extensions

# Range median (not in PromQL)
range_median(cpu_usage_percent[1h])

# Running average
running_avg(http_requests_total[1h])

# Lag (time since last sample)
lag(up[5m])

# Interpolate missing data
interpolate(temperature[10m])

# Label manipulation
label_set(up, "env", "production")
label_del(metric, "instance")
label_join(metric, "combined", "-", "job", "instance")
label_replace(metric, "short_name", "$1", "instance", "(.+):.+")

# Rollup functions
rollup_rate(http_requests_total[5m])
rollup(cpu_usage[1h])

# Aggregate by time
avg_over_time(cpu_usage[1h:5m])  # 5-minute steps within 1h window

Cluster Version

Architecture Components

# vminsert (ingestion)
vminsert-prod \
  -storageNode=vmstorage-1:8400,vmstorage-2:8400,vmstorage-3:8400 \
  -httpListenAddr=:8480

# vmstorage (storage)
vmstorage-prod \
  -storageDataPath=/data \
  -retentionPeriod=12 \
  -vminsertAddr=:8400 \
  -vmselectAddr=:8401 \
  -httpListenAddr=:8482

# vmselect (queries)
vmselect-prod \
  -storageNode=vmstorage-1:8401,vmstorage-2:8401,vmstorage-3:8401 \
  -httpListenAddr=:8481

Multi-Tenancy (Cluster)

# Write to tenant 1
curl -X POST 'http://vminsert:8480/insert/1/prometheus/api/v1/write' --data-binary @data.pb

# Query tenant 1
curl 'http://vmselect:8481/select/1/prometheus/api/v1/query?query=up'

Advanced Usage

vmalert (Alerting and Recording Rules)

vmalert-prod \
  -rule=/etc/vmalert/rules/*.yaml \
  -datasource.url=http://victoriametrics:8428 \
  -notifier.url=http://alertmanager:9093 \
  -remoteWrite.url=http://victoriametrics:8428 \
  -remoteRead.url=http://victoriametrics:8428

# rules.yaml
groups:
  - name: example
    rules:
      - alert: HighCPU
        expr: avg by (instance) (cpu_usage_percent) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ $labels.instance }}"

Data Migration with vmctl

# Migrate from Prometheus
vmctl-prod prometheus \
  --prom-snapshot=/prometheus/snapshots/20240101 \
  --vm-addr=http://victoriametrics:8428

# Migrate from InfluxDB
vmctl-prod influx \
  --influx-addr=http://influxdb:8086 \
  --influx-database=mydb \
  --vm-addr=http://victoriametrics:8428

Backup and Restore

# Create snapshot
curl http://victoriametrics:8428/snapshot/create

# Backup snapshot to S3
vmbackup-prod \
  -storageDataPath=/var/lib/victoriametrics \
  -snapshot.createURL=http://localhost:8428/snapshot/create \
  -dst=s3://my-bucket/backups/

# Restore
vmrestore-prod \
  -src=s3://my-bucket/backups/latest/ \
  -storageDataPath=/var/lib/victoriametrics

Troubleshooting

Issue	Solution
High memory usage	Set `-memory.allowedPercent=50`; reduce `-search.maxUniqueTimeseries`
Slow queries	Add time range filters; reduce cardinality; check `-search.maxQueryDuration`
`too many unique timeseries`	Increase `-search.maxUniqueTimeseries` or reduce query scope
Data gaps	Check vmagent scrape targets at `/targets`; verify remote write connectivity
Disk space growing fast	Enable downsampling; reduce retention; check high-cardinality labels
Import failing	Verify data format; check content-type header; review `/api/v1/import` docs
Deduplication not working	Set `-dedup.minScrapeInterval` to match your scrape interval