Vector (by Datadog, open-source) is a high-performance observability data pipeline written in Rust. It collects logs, metrics, and traces from any source, reshapes them through transforms, and delivers them to any sink — all vendor-agnostic, so you can switch backends without re-instrumenting. It typically uses a fraction of the CPU and memory of Logstash or Fluentd, which matters when an agent runs on every node.
Installation
| Method | Command |
|---|
| Install script | `curl —proto ‘=https’ —tlsv1.2 -sSf https://sh.vector.dev |
| Debian/Ubuntu | .deb from the Vector releases |
| macOS (Homebrew) | brew install vectordotdev/brew/vector |
| Docker | docker run -v $PWD/vector.yaml:/etc/vector/vector.yaml timberio/vector:latest-alpine |
| Verify | vector --version |
# vector.yaml
sources:
app_logs:
type: file
include: ["/var/log/app/*.log"]
transforms:
parse:
type: remap
inputs: ["app_logs"]
source: |
. = parse_json!(.message)
.env = "production"
del(.password)
sinks:
out:
type: elasticsearch
inputs: ["parse"]
endpoints: ["http://es:9200"]
| Stage | Purpose |
|---|
sources | Where data comes from |
transforms | Parse, filter, enrich, redact, aggregate |
sinks | Where data goes |
inputs | Wires components into a DAG |
Common Sources
| Source | Collects |
|---|
file | Log files (tailing) |
journald | systemd journal |
kubernetes_logs | Pod logs with k8s metadata |
docker_logs | Container logs |
http_server | Push endpoint |
syslog | Syslog over TCP/UDP |
prometheus_scrape | Scrape metrics endpoints |
opentelemetry | OTLP ingestion |
Common Sinks
| Sink | Sends to |
|---|
elasticsearch | Elasticsearch/OpenSearch |
loki | Grafana Loki |
clickhouse | ClickHouse |
aws_s3 | S3 (archival) |
prometheus_exporter | Expose metrics for scraping |
datadog_logs / splunk_hec | Vendor backends |
console | stdout (debugging) |
blackhole | Discard (testing) |
VRL (Vector Remap Language)
The remap transform is where most work happens.
# Parse, enrich, redact, and drop noise
. = parse_json!(.message)
.timestamp = parse_timestamp!(.ts, "%+")
.service = "checkout"
del(.credit_card)
if .level == "debug" { abort }
| VRL function | Purpose |
|---|
parse_json!, parse_regex!, parse_grok! | Structure raw text |
del(.field) | Remove a field (PII redaction) |
to_int!, to_float! | Type coercion |
abort | Drop the event |
merge, flatten | Reshape objects |
| Transform | Does |
|---|
remap | VRL-based reshaping (the workhorse) |
filter | Keep events matching a condition |
route | Split a stream into named branches |
aggregate | Roll up metrics over an interval |
throttle | Rate-limit noisy events |
dedupe | Drop duplicates |
sample | Keep a percentage of events |
Operating Vector
| Command | Description |
|---|
vector --config vector.yaml | Run with a config |
vector validate | Check config before deploying |
vector test | Run unit tests defined in the config |
vector top | Live TUI of throughput per component |
vector tap <component> | Stream events flowing through a component |
--watch-config | Hot-reload on config change |
# Debug what a transform is actually emitting
vector tap parse
Reliability
| Feature | Note |
|---|
| Disk buffers | Survive restarts/backpressure (buffer.type: disk) |
| Acknowledgements | End-to-end delivery guarantees |
| Adaptive concurrency | Auto-tunes sink request rate |
| Health checks | Fail fast on unreachable sinks |
Vector vs Alternatives
| Aspect | Vector | Fluent Bit | Logstash |
|---|
| Language | Rust | C | JRuby/JVM |
| Resource use | Very low | Very low | High |
| Transform language | VRL (rich) | Lua/filters | Ruby/grok |
| Data types | Logs, metrics, traces | Logs, metrics | Logs |
| Best for | Vendor-agnostic pipelines | Ultra-light edge agent | Legacy ELK stacks |
Feeds sinks like OpenObserve, Loki, and Elasticsearch; compare with Fluentd.
Resources
