magic-trace - Intel PT High-Resolution Tracing Cheatsheet
magic-trace (by Jane Street) captures and displays high-resolution execution traces using Intel Processor Trace (Intel PT). Where a sampling profiler tells you where time goes on average, magic-trace records the actual control flow — every function entry and exit — for the last few milliseconds, at nanosecond resolution. That makes it uniquely suited to rare, latency-sensitive events: the one request in ten thousand that took 40ms, where an averaged profile shows you nothing.
Requirements
- Intel CPU with Processor Trace support (most modern Intel)
- Linux with perf and PT enabled
- Root or appropriate
perf_event_paranoid setting
- Binaries with frame pointers / symbols for readable output
Installation
| Method | Command |
|---|
| Binary | download magic-trace from GitHub Releases |
| opam (OCaml) | opam install magic_trace |
| Permissions | sysctl kernel.perf_event_paranoid=-1 (or 1) |
| Verify | magic-trace --help |
Two Capture Modes
| Mode | Command | Use |
|---|
| Attach to running | magic-trace attach -pid <pid> | Live service |
| Run a command | magic-trace run ./my-program | Reproducible run |
# Attach, capture on Ctrl-C, write a trace
sudo magic-trace attach -pid $(pgrep -n myservice)
Snapshot Triggers
The core idea: keep a rolling buffer and snapshot when something interesting happens.
| Trigger | How |
|---|
| Manual | Press Ctrl-C while attached |
| Magic breakpoint | App calls a designated symbol to trigger a snapshot |
| Timer | Capture after a fixed duration |
| Symbol trigger | -trigger <symbol> to snapshot on a function call |
# Snapshot whenever a specific function is hit
sudo magic-trace attach -pid <pid> -trigger 'handle_slow_path'
Because the buffer holds the preceding few milliseconds, you see what led up to the event, not just the event itself.
Viewing Traces
magic-trace emits a trace viewable in the Perfetto UI.
| Step | Action |
|---|
| 1 | Capture produces trace.fxt (or similar) |
| 2 | Open ui.perfetto.dev |
| 3 | Load the trace file |
| 4 | Zoom into the microsecond range of interest |
| 5 | Read the exact call sequence and durations |
| Perfetto control | Does |
|---|
W/S | Zoom in/out |
A/D | Pan |
| Click a slice | Duration and function detail |
| Select a range | Summary of what ran |
What You Can See
| Question | magic-trace answer |
|---|
| Which function caused this 40ms spike? | Exact call, exact duration |
| Did we take an unexpected branch? | Full control flow, visible |
| Where did the time go inside one request? | Nanosecond-level breakdown |
| Was it a syscall, a lock, or compute? | The call sequence shows it |
Options Worth Knowing
| Flag | Purpose |
|---|
-multi-thread | Trace all threads |
-duration | Length of capture window |
-trigger SYMBOL | Snapshot on a symbol |
-full-execution | Trace an entire short run |
-output FILE | Trace output path |
Limitations
| Limitation | Note |
|---|
| Intel only | Requires Intel PT (no AMD/ARM) |
| Short window | Milliseconds, not minutes |
| Symbol quality | Stripped binaries yield poor traces |
| Overhead | Low but non-zero while tracing |
| Aspect | magic-trace | perf | Perfetto |
|---|
| Data | Full control flow (Intel PT) | Sampled stacks | System-wide timeline |
| Resolution | Nanoseconds | Sample interval | Event-level |
| Window | Last few ms | Whole run | Configurable |
| Best for | Rare latency spikes | Average hot paths | System-wide correlation |
Use perf for average hot paths, Perfetto for system-wide timelines, and magic-trace when you need the exact story of one rare slow event.
Resources
