The traditional way to profile a production problem has an awkward dependency: you have to already suspect it. Something looks slow, you attach a profiler, you try to reproduce the condition, and — if you are lucky and the problem is still happening — you capture data. The failure mode is obvious and common. The incident happened at 3 a.m., it lasted four minutes, and by the time anyone looked, the evidence was gone. You add logging, hope it recurs, and wait.
eBPF changed the economics enough to invert that workflow. Because eBPF programs run inside the kernel in a verified sandbox, they can observe syscalls, scheduling, and stack traces across every process on a machine at overhead low enough — consistently under 1% — to leave running permanently. That turns profiling from something you start into something you query: the data for 3 a.m. already exists. This guide covers that shift through Parca Agent for continuous profiling, and a newer application that has emerged as autonomous agents proliferate — using the same kernel-level visibility to see what an AI agent actually did, via tools like AgentSight.
Why eBPF made always-on viable
The technical reason continuous profiling was impractical before is that the older options all imposed a tax you would not pay permanently. Sampling profilers based on perf are cheap but require setup per target and produce data you must manage yourself. Instrumented profilers require code changes and add per-call overhead. Language-specific agents cover one runtime and often introduce their own performance cost. None of them were things you would run on every process on every node forever.
eBPF's contribution is that programs run in kernel space, so collecting a stack trace does not require context-switching to a userspace agent for each sample, and the kernel's verifier statically proves the program cannot crash the system or loop forever before it is allowed to load. The result is kernel-level visibility with safety guarantees and overhead in the noise. Crucially it is also zero-instrumentation: you do not modify, recompile, or restart the applications being profiled, which removes the organizational friction that killed most previous attempts ("we'd need every team to add an agent").
There is one real prerequisite worth stating: this depends on a reasonably modern kernel with BTF support, and stack unwinding quality depends on your binaries having symbols. Stripped production binaries yield profiles full of hex addresses, which is a solvable problem but one you should solve before concluding the tool does not work.
Continuous profiling in practice
Parca Agent is the clearest expression of the idea. Deployed as a DaemonSet on Kubernetes (or a process on a node), it samples stack traces from every process on that node, continuously, labels them with pod, container, and namespace metadata, and ships them to a server where they are stored like metrics — queryable by time range and label.
That storage model is what makes it useful rather than merely interesting. Because profiles are labeled time series, you can ask questions that on-demand profiling cannot answer. What was consuming CPU in the payments namespace between 03:04 and 03:08? The profile exists; you filter to the window. Even more valuable is comparison: diff the profile from before a deployment against after, and the regression appears as a function that grew. You did not need to predict you would want the "before" profile — continuous collection means it is always there.
Parca handles mixed-language stacks, which matters more than it sounds. A modern service might be Go calling into C libraries, or Python invoking native extensions, and a profile that resolves only one layer tells a misleading story. Kernel-level unwinding sees the whole stack. For deep one-off analysis you will still reach for perf or async-profiler — continuous profiling optimizes for coverage and history, not maximum per-run detail — but the everyday question of "what is this cluster spending CPU on" is answered continuously and cheaply.
The newer problem: what did the agent do?
The second application is newer and, in 2026, increasingly pressing. Teams are running autonomous coding agents and LLM agents with tool access on real systems. These agents read files, execute commands, install packages, and make network calls. The observability those systems provide is application-level: prompts, tool calls, token counts, spans. Tools like Langfuse and Arize Phoenix do this well and are genuinely useful for understanding an agent's reasoning.
But application-level tracing has a structural limitation: it shows what the agent framework chose to report. If a tool executes a shell command, the trace records the command string — not the seventeen processes that command spawned, the files they touched, or the host they contacted. If the agent is compromised by prompt injection and takes an action outside its intended scope, the trace shows the action it narrated, which is exactly the wrong source of truth for a security question.
AgentSight applies eBPF here: observe the agent's actual system behavior at the kernel level — process execution, file access, network connections — with no instrumentation of the agent at all. The agent cannot omit or misreport these events because it is not the one reporting them. For a coding agent operating on a repository, you get the real answer to "which files did it modify," "what did it run," and "did it reach outside the workspace."
The right posture is that these are complementary layers, not competitors. Application tracing tells you the agent's intent and reasoning; kernel tracing tells you its effects. Correlating them by timestamp gives you both halves: this is what the agent was trying to do, and this is what actually happened on the machine. For anything with real permissions, having only the first half is a gap.
Security, not just performance
That framing points at the broader use. The same eBPF foundation underpins runtime security tools — Tracee, Falco, Tetragon — which watch for suspicious kernel-level behavior and, in Tetragon's case, can enforce policy in-kernel. Once you are collecting process, file, and network events on every node, the difference between "observability" and "detection" is mostly which questions you ask of the data.
For agent workloads specifically, that convergence is useful. The questions "why is this slow," "what did the agent change," and "did anything escape its sandbox" are all answered from the same event stream. Practically, this means an organization deploying agents with system access should treat kernel-level visibility as part of the deployment, not an afterthought — it is the only layer that produces auditable evidence of what an autonomous process actually did.
The honest caveat is volume. Kernel-level tracing can generate enormous event streams, and naively logging every syscall on a busy node will overwhelm your storage and your ability to find anything. Filtering at the source — by path, by process, by event type — is not an optimization but a requirement, and it is where most of the operational work in these tools lives.
Reading a profile without fooling yourself
Continuous profiling produces a lot of data, and there are a few reliable ways to misread it. Worth knowing before you act on a flame graph.
Width is samples, not wall-clock latency. A CPU profile shows where CPU time went. If your service is slow because it is waiting — on a database, a lock, a network call — the CPU profile may look perfectly healthy, because a blocked thread consumes no CPU. This is the single most common misdiagnosis: concluding there is no problem because the CPU profile is flat, when the problem is off-CPU time. Wall-clock or off-CPU profiling answers that question instead.
Aggregate hotspots are not necessarily your latency problem. A function consuming 30% of cluster CPU may be entirely expected background work — serialization, compression, garbage collection. The interesting signal is usually a change: this function was 5% last week and is 30% now. That is why comparison matters more than absolute ranking, and why continuous collection beats a single snapshot.
Symbol quality shapes conclusions. Missing symbols make stacks collapse into unhelpful hex addresses, and worse, they can misattribute time to the nearest resolvable frame — producing a confident, wrong answer. If a profile blames a function that makes no sense, suspect unwinding before you suspect the code.
Sampling has a floor. A function that runs frequently but very briefly may be underrepresented relative to its true cost, and rare-but-expensive events may not appear at all in a short window. Continuous collection mitigates this by aggregating over long periods, which is another argument for always-on over on-demand.
The practical habit: use the profile to form a hypothesis, then confirm it with a targeted measurement before you spend a sprint optimizing.
Adopting this sensibly
A pragmatic sequence avoids the common failure of deploying everything and drowning. Start with continuous profiling on a subset of nodes. It is the lowest-risk introduction, delivers value immediately (you will find something surprising in the first week), and teaches you how your kernels and symbols behave before anything depends on it. Confirm that stacks resolve to real function names; fix symbol availability if they do not.
Then add comparison workflows, because that is where the payoff concentrates: wire profile diffing into your release process so a CPU regression is caught as a profile change rather than as a latency alert three days later.
Add agent-level tracing when you actually run agents with system access. If your agents only call APIs and return text, kernel tracing is overkill. If they execute commands, write files, or operate on repositories, the visibility gap is real and worth closing.
And filter aggressively from the start. Decide up front which paths, processes, and event types matter, and exclude the rest. It is far easier to widen a narrow filter later than to retrofit filtering onto a firehose you are already storing.
Finally, keep the on-demand tools sharp. Continuous profiling tells you that a function became hot and when; perf, bpftrace, and async-profiler remain better for the deep dive into why. The workflow that works is continuous data for detection and history, targeted tools for diagnosis.
The bottom line
eBPF moved observability from "attach a profiler when you suspect a problem" to "the data already exists, query it," because kernel-side execution with a verified sandbox made always-on collection cost under 1% and require no application changes. Parca Agent delivers that as continuous, label-queryable, mixed-language CPU profiling whose real superpower is comparing before and after. The newer frontier is applying the same visibility to autonomous agents: AgentSight shows what an agent actually did at the syscall level, closing the gap left by application traces that only report what the agent chose to narrate. Start with profiling on a few nodes, verify your symbols, build comparison into releases, add agent tracing when agents touch real systems, filter hard from day one, and keep perf and bpftrace for the deep dives.
References and Resources
Tools
Background and analysis
- 8 Best Open-Source eBPF Tracing Tools in 2026 — Better Stack
- eBPF.io — what is eBPF
- Brendan Gregg on eBPF performance tools
Related 1337skills cheatsheets