ebpf_exporter - Custom Kernel Metrics for Prometheus Cheatsheet
ebpf_exporter (by Cloudflare) bridges eBPF and Prometheus. Standard exporters surface counters the kernel already publishes; ebpf_exporter lets you attach custom eBPF programs to kernel probes and export whatever they measure as Prometheus metrics — disk I/O latency histograms, TCP retransmit causes, scheduler run-queue delay, page-cache behavior. It turns deep, ad-hoc kernel questions into continuously-scraped time series.
Requirements
- Linux kernel with BPF + BTF (5.x recommended)
- Root /
CAP_BPF privileges
- Prometheus to scrape the endpoint
Installation
| Method | Command |
|---|
| Binary | Download from GitHub Releases |
| Docker | docker run --privileged -p 9435:9435 cloudflare/ebpf_exporter |
| From source | git clone https://github.com/cloudflare/ebpf_exporter && make |
| Run | sudo ebpf_exporter --config.dir=examples --config.names=biolatency |
| Metrics | http://localhost:9435/metrics |
Running with Bundled Examples
# Export block I/O latency histograms
sudo ./ebpf_exporter --config.dir=examples --config.names=biolatency
# Multiple programs at once
sudo ./ebpf_exporter --config.dir=examples \
--config.names=biolatency,tcp-syn-backlog,run-queue-latency
| Flag | Purpose |
|---|
--config.dir | Directory of program configs |
--config.names | Which programs to load (comma-separated) |
--web.listen-address | Bind address (default :9435) |
--debug | Verbose loading/diagnostics |
Bundled Programs (examples)
| Program | Measures |
|---|
biolatency | Block I/O latency histogram |
bio-tracepoints | Disk I/O by device/operation |
run-queue-latency | Scheduler delay before a task runs |
tcp-syn-backlog | SYN backlog depth |
tcp-window-clamps | TCP window clamping events |
timers | Timer/softirq behavior |
cachestat | Page cache hit/miss |
oomkill | OOM kill events |
Config Structure
A program pairs a compiled eBPF object with metric definitions:
metrics:
histograms:
- name: bio_latency_seconds
help: Block IO latency histogram
bucket_type: exp2
bucket_multiplier: 0.000001 # microseconds → seconds
bucket_min: 0
bucket_max: 26
labels:
- name: device
size: 32
decoders:
- name: string
- name: bucket
size: 8
decoders:
- name: uint
| Field | Purpose |
|---|
counters / histograms | Metric type to export |
bucket_type | exp2 / linear histogram bucketing |
bucket_multiplier | Scale raw kernel units to seconds |
labels | Map BPF map keys to Prometheus labels |
decoders | Convert raw bytes (string, uint, ksym, cgroup…) |
Useful Decoders
| Decoder | Converts |
|---|
string | Byte array → label string |
uint | Raw integer |
ksym | Kernel address → symbol name |
majorminor | Device numbers → sda, nvme0n1 |
cgroup | cgroup ID → path |
static_map | Numeric enum → readable label |
Querying in Prometheus
# p99 block I/O latency by device
histogram_quantile(0.99,
sum(rate(bio_latency_seconds_bucket[5m])) by (le, device))
# Scheduler run-queue delay p95 (CPU saturation signal)
histogram_quantile(0.95,
sum(rate(run_queue_latency_seconds_bucket[5m])) by (le))
When to Reach for It
| Question | Standard exporters | ebpf_exporter |
|---|
| ”Is the disk busy?” | Yes (node_exporter) | — |
| “What is the distribution of I/O latency?” | No | Yes |
| ”How long do tasks wait to be scheduled?” | No | Yes |
| ”Why are TCP retransmits happening?” | Partially | Yes |
| ”Which cgroup caused this?” | No | Yes (cgroup decoder) |
| Tool | Model |
|---|
| ebpf_exporter | Continuous eBPF metrics → Prometheus |
| bpftrace | Ad-hoc, interactive eBPF one-liners |
| bcc-tools | Prebuilt eBPF diagnostic tools |
| node_exporter | Standard kernel-published metrics |
| bpftop | Overhead of running eBPF programs |
Use bpftrace to explore a question interactively, then encode the answer as a permanent ebpf_exporter metric.
Resources
