For most of the last decade, observability had an uncomfortable secret: the main constraint on what you could observe was not technical but financial. Teams ran Elasticsearch clusters whose storage bills grew faster than their traffic, and the standard response was to throw data away — cut retention from ninety days to seven, sample logs at ten percent, drop debug output entirely. Then, inevitably, an incident would land in the gap. The data that would have explained it had been discarded to save money, and the postmortem action item read "increase log retention," which nobody funded.
The economics changed because a new generation of tools moved log storage from expensive local disks to cheap object storage — S3, GCS, Azure Blob. When storage costs an order of magnitude less, retention stops being a budget negotiation. This guide covers that shift: why the old architecture was expensive, how the object-storage model works, and the tools that implement it — Quickwit for searchable log and trace retention, OpenObserve as a full observability platform, and Vector as the pipeline that feeds them.
Why the old architecture was expensive
Elasticsearch and the stacks built on it were designed for a demanding use case: interactive full-text search over data that changes, with results in milliseconds. To deliver that, they keep indices on fast local disks attached to always-running nodes, replicate every shard for durability and availability, and hold substantial index structures in memory. Every one of those choices is reasonable for search, and every one is expensive.
The costs compound in ways that surprise teams. Storage is provisioned rather than consumed — you pay for the disks whether they are full or not, and you over-provision because running out is an outage. Replication multiplies that by two or three. The nodes must run continuously even though most logs are never read after the day they were written. And scaling is coupled: needing more storage means adding nodes, which means paying for CPU and RAM you did not need. The result is a cost curve that rises steeply with retention, which is exactly why "reduce retention" became the standard cost-control lever.
The deeper mismatch is that logs are not the workload Elasticsearch was optimized for. Log data is append-only — written once, never updated. It is overwhelmingly cold: the vast majority is never queried, and the small fraction that is gets queried during an incident, where a few seconds of latency is perfectly acceptable. Paying for millisecond interactive search and mutable-document machinery across petabytes of write-once data that nobody reads is the architectural error underneath the bill.
The object-storage model
The newer architecture starts from those properties. If data is append-only and mostly cold, store it in object storage — which is roughly an order of magnitude cheaper per terabyte than provisioned block storage, is effectively infinite, and is durable by default without you managing replication. Then decouple compute from storage: run query nodes that read from object storage on demand, and scale them independently of how much data you retain. Storing another year of logs costs storage only, not more nodes.
The catch, of course, is that object storage is slow relative to local disk — every read is a network request with meaningful latency. The engineering insight that makes this work is that you can compensate with format and indexing. Data is written in compressed columnar formats (Parquet is common) alongside index structures that let a query fetch only the byte ranges it actually needs, rather than scanning everything. Aggressive caching keeps hot data close. The result is queries that return in seconds rather than milliseconds — a tradeoff that is completely acceptable for log search and completely unacceptable for a user-facing search box, which is precisely why this architecture suits observability and not general search.
What this buys, beyond money, is a change in behavior. When retention is cheap, you stop pre-emptively discarding data, which means the answer to the next incident is more likely to still exist. That is the real payoff: not a smaller invoice, but fewer investigations that dead-end because the evidence was deleted.
Quickwit: search built for logs and traces
Quickwit is the purest expression of the idea. It is a Rust search engine designed from the start for append-only data on object storage — it needs no local disk for indices and no cluster state machinery for them, because the index lives in S3 as immutable splits. It provides genuine full-text search (not just label filtering), exposes an Elasticsearch-compatible search API that eases migration, and serves as a native Jaeger backend for traces, which makes it a natural fit for teams already running OpenTelemetry.
Quickwit's sweet spot is cheap, searchable, long-retention storage for logs and traces where you want real search capability. Retention management becomes trivial — expiring old data is just deleting old splits. Its tradeoff is scope: it is a search engine, not a complete observability platform, so you bring your own dashboards (Grafana), your own alerting, and your own metrics stack. For teams that already have those pieces and want to fix the storage cost specifically, that focus is a feature.
OpenObserve: the batteries-included platform
OpenObserve takes the same storage insight and wraps a full platform around it. It handles logs, metrics, and traces in one system, stores data as compressed Parquet in object storage, and ships with a built-in UI, SQL querying, dashboards, and an alerting engine — all as a single Rust binary that is genuinely easy to stand up. It ingests OpenTelemetry natively and supports Prometheus remote write, so it slots into existing instrumentation.
The appeal is consolidation. Rather than running a search engine plus Grafana plus an alerting stack plus separate metrics storage, you run one thing. For small and medium teams especially, the operational savings can matter more than any per-terabyte comparison — every additional component in an observability stack is something to upgrade, secure, and debug at 3 a.m. Its tradeoff is the usual one for integrated platforms: less flexibility than assembling best-of-breed components, and a younger ecosystem than the Grafana or Elastic worlds. Choose it when operational simplicity is worth more than modularity.
Vector: the pipeline that makes migration safe
Neither backend matters if you cannot get data into it, and this is where Vector becomes the quiet linchpin. Vector is a high-performance, vendor-agnostic pipeline: it collects logs, metrics, and traces from any source, reshapes them through transforms written in VRL, and delivers them to any sink. Written in Rust, it uses a fraction of the resources of Logstash or Fluentd — which matters because an agent runs on every node you own.
Its strategic value is that it decouples your instrumentation from your backend. Applications and agents ship data to Vector; Vector decides where it goes. That turns a backend migration from a fleet-wide re-instrumentation project into a config change — and, crucially, it lets you dual-write during a transition, sending the same data to both your existing stack and a new one so you can validate the replacement on real traffic before cutting over. Vector's transforms also cut cost at the source: dropping debug logs, sampling high-volume events, and redacting PII before storage all reduce what you pay to keep. Even teams with no intention of changing backends often find Vector pays for itself on volume reduction alone.
Adopting this without a risky migration
The sensible path is incremental, and Vector is what makes it safe. Start by putting a pipeline in front of your existing stack. Deploy Vector, point your sources at it, and have it forward to whatever you run today. Nothing changes functionally, but you have gained a control point — and you can immediately use transforms to drop noise and cut current costs.
Then dual-write to a candidate backend. Add a second sink pointing at Quickwit or OpenObserve and let real production data flow to both. Now you can evaluate honestly: are the queries you actually run fast enough? Does the search syntax cover your needs? What does storage really cost at your volume? This is a far better basis for a decision than a benchmark, and it costs only the new backend's storage during the trial.
Shift retention before you shift primary querying. A low-risk intermediate state is keeping short-retention hot data where it is while sending long-retention data to object storage. You get the cost win on the expensive part (long retention) without betting your incident response on an unfamiliar system. Then move querying over gradually, starting with non-urgent analysis and moving to incident response once the team trusts it. And keep the old stack running until you have been through a real incident on the new one — the moment you discover whether it works is the moment you least want a surprise.
What you give up, honestly
Every architecture trades something, and it is worth being explicit about what moving to object storage costs so the decision is informed rather than enthusiastic.
Query latency. Reads from object storage are network requests, so queries return in seconds rather than the sub-second responses a warm local-disk index can deliver. For log search during an incident this is genuinely fine — you are reading results, not typing ahead. But if you have built workflows that depend on instantaneous interactive filtering, or dashboards that fire dozens of queries on every refresh, they will feel slower. Test your actual query patterns, not a synthetic benchmark.
Ecosystem maturity. Elasticsearch has a decade of accumulated integrations, tutorials, Stack Overflow answers, and engineers who already know it. The newer tools have smaller communities, thinner documentation for edge cases, and fewer people you can hire who have run them at scale. When something breaks at an awkward hour, that difference is real. Elasticsearch-compatible APIs soften the migration but do not eliminate the knowledge gap.
Feature surface. Elasticsearch does a great deal beyond log search — complex aggregations, machine-learning jobs, relevance tuning, a rich query DSL. Purpose-built log engines deliberately do less. If you rely on those capabilities, verify equivalents exist before committing rather than discovering the gap mid-migration.
Operational unfamiliarity. Your runbooks, alerts, and instincts are calibrated to your current stack. A new backend means new failure modes and a period where the team is slower at diagnosing its problems. This is the strongest argument for the dual-write approach and for keeping the old stack running through at least one real incident — you want the learning curve to happen while you still have a fallback.
None of these outweigh an order-of-magnitude cost reduction for most teams, particularly ones currently deleting data they wish they had. But they are the reason to migrate deliberately rather than all at once.
The bottom line
Observability got dramatically cheaper because the tooling finally matched the workload. Logs are append-only and mostly cold, so storing them on replicated, always-on local disks optimized for millisecond mutable search was paying a large premium for properties the data does not need. Moving to object storage with columnar formats and smart indexing trades milliseconds for seconds — irrelevant for log search — and cuts storage cost by an order of magnitude. Quickwit delivers that as a focused, searchable log and trace store with a Jaeger backend; OpenObserve delivers it as a consolidated platform with UI, dashboards, and alerting built in; and Vector is the pipeline that decouples your instrumentation from either, making adoption a config change and a dual-write rather than a migration. Put the pipeline in first, dual-write to evaluate on real traffic, move long retention before hot querying, and stop deleting the data that explains your incidents.
References and Resources
Tools
- Quickwit and GitHub
- OpenObserve and GitHub
- Vector and VRL reference
Background and analysis
Related 1337skills cheatsheets