Perfetto - System-Wide Tracing & Analysis Cheatsheet
Perfetto is Google’s open-source, production-grade tracing and profiling stack for Linux, Android, and Chrome. It captures system-wide traces — CPU scheduling, syscalls, frequency changes, memory events, and custom application events — into a compact protobuf format, then lets you explore them in a sophisticated web UI or, uniquely, query them with SQL. Where perf samples stacks, Perfetto records a timeline of what every thread and CPU was doing, which is what you need for latency and concurrency problems.
Installation
| Component | How |
|---|
| UI (no install) | Open ui.perfetto.dev in a browser |
| CLI tools | Download tracebox/perfetto from GitHub Releases |
| Android | Built in (adb shell perfetto ...) |
| trace_processor | Download for local SQL analysis |
| Verify | ./tracebox --help |
Capturing a Trace (Linux)
# Simple 10-second system-wide trace with ftrace scheduling data
./tracebox -o trace.pftrace --txt -c - <<EOF
buffers: { size_kb: 65536 }
data_sources: {
config {
name: "linux.ftrace"
ftrace_config {
ftrace_events: "sched/sched_switch"
ftrace_events: "sched/sched_wakeup"
ftrace_events: "power/cpu_frequency"
}
}
}
duration_ms: 10000
EOF
| Element | Purpose |
|---|
buffers | Ring buffer size for trace data |
data_sources | What to record (ftrace, procstats, heapprofd, …) |
ftrace_events | Specific kernel tracepoints |
duration_ms | Capture length |
-o | Output trace file |
Common Data Sources
| Source | Records |
|---|
linux.ftrace | Scheduling, syscalls, IRQs, custom tracepoints |
linux.process_stats | Per-process CPU/memory over time |
linux.sys_stats | System-wide counters (meminfo, vmstat) |
android.heapprofd | Native heap profiling |
android.java_hprof | Java heap dumps |
track_event | Application-emitted spans/counters |
The UI Workflow
| Step | Action |
|---|
| 1 | Open ui.perfetto.dev and drag in the trace |
| 2 | Timeline shows CPUs, threads, and tracks |
| 3 | W/S zoom, A/D pan |
| 4 | Click a slice to see its duration and args |
| 5 | Select a range → summary of what ran |
| 6 | Use the query box for SQL analysis |
SQL Analysis (the superpower)
Perfetto’s trace processor exposes the trace as queryable tables.
-- Longest-running slices
SELECT name, dur/1e6 AS ms
FROM slice
ORDER BY dur DESC
LIMIT 20;
-- CPU time per process
SELECT p.name, SUM(s.dur)/1e6 AS cpu_ms
FROM sched s JOIN thread t USING(utid) JOIN process p USING(upid)
GROUP BY p.name ORDER BY cpu_ms DESC;
-- Why was a thread blocked?
SELECT ts, dur, state
FROM thread_state
WHERE utid = 42 AND state != 'Running'
ORDER BY dur DESC;
| Table | Contains |
|---|
slice | Named, nested duration events |
sched | Scheduling slices per CPU |
thread_state | Running/sleeping/blocked intervals |
counter | Numeric time series (freq, memory) |
process, thread | Process/thread metadata |
Local Analysis
# Interactive SQL shell over a trace
./trace_processor trace.pftrace
# Run a query and emit JSON for scripting
./trace_processor -q query.sql --query-output json trace.pftrace
What Perfetto Is Best At
| Problem | Why Perfetto helps |
|---|
| Latency spikes | See exactly what ran/blocked at that moment |
| Lock contention | thread_state shows blocked intervals and reasons |
| CPU migration/scheduling | Per-CPU timeline of every thread |
| Frequency/thermal effects | Correlate cpufreq counters with slowdowns |
| App + kernel correlation | Application spans on the same timeline as kernel events |
| Aspect | Perfetto | perf | bpftrace |
|---|
| Model | Full timeline trace | Sampling profiler | Ad-hoc eBPF probes |
| Analysis | UI + SQL | Reports/flame graphs | Custom scripts |
| Best for | Latency/concurrency/timeline | CPU hot paths | Targeted kernel questions |
| Overhead | Moderate (configurable) | Low | Low |
Complements perf (CPU hot spots) and bpftrace (targeted probes); Perfetto answers “what happened when.”
Resources
