OTel Isn't Going Well (And I Made A Spreadsheet About It)

14 min read in opentelemetry

For years now one of the most reliable complaints I hear when I try to drag a team off their vendor specific SDK and onto OpenTelemetry is some variation of: "why does it seem like this isn't done yet?"

Vendor SDKs for observability are, to put it charitably, idiot-proof. You install the thing, dashboards just load data, someone else worries about how all those pieces fit together, and you get on with your life. OpenTelemetry, by contrast, greets you at the door with a lot of "experimental" stamps and roughly six different ways to accomplish any given task.

In OpenTelemetry's defense this was never what they were going for as a project. I've always respect that they stuck to their guns by attempting to build a truly vendor agnostic system that really doesn't care what you do with the data. I have never gotten a sense of a vendor being strongly preferred with OTel, which is quite the feat considering how lucrative and contentious the observability ecosystem was. Also considering that the maintainers of this project are largely employed by exclusively those companies.

As the years wore on, I started to get nervous. Conversations in the semantic-conventions repo drag on and on and on. Different languages had dramatically different stories. Golang and Dotnet were first class citizens, but other languages lagged years behind the others.

I started asking a lot of probing questions before recommending OpenTelemetry to smaller teams who didn't have the time, budget, or emotional bandwidth for it. Auto-instrumentation was genuinely magical, but the cliff between "auto-instrument works" and "now I have to manually instrument something" was steep enough that you owed people a warning before you pushed them off it.

This narrative has been going on for awhile in the observability space, a vague sense of "something is wrong in Otel-land". But let's try to generate some actual data here. Is there an actual problem, or is this something where the perception by the community of slow progress is imaginary? Is the problem not enough maintainers, too big of a scope, or something in-between?

My guess when I started was "oh this is your classic open-source bit off more than they can chew". Not enough maintainers, not enough budget. Now there is some of that, but there's also something else going on.

The actual problem happening inside of OpenTelemetry is a three way crash. You have a binary stability gate which, when combined with a very small bench of actual maintainers means there is understandable worry about marking a feature not experimental then add on just a massive scope of languages and frameworks they are attempting to cover. This creates a perfect storm where there is an incentive to argue about potential problems a feature might create since once it is locked in and shipped as stable you can never change them.

How does OpenTelemetry Work

So OpenTelemetry currently is attempting to support a dizzying number of languages and frameworks.

OpenTelemetry is a giant project. It spans dozens of languages, hundreds of libraries, and countless backends. To keep things sane, the project splits work into two buckets:

  • Core → Maintained directly by the OTel project. Small, stable, vendor-neutral, and tightly reviewed. This is the "spec-defining" surface.
  • Contrib → Community- and vendor-contributed. Broader, faster-moving, and covers the long tail of integrations.

There exists the otel-collector, the thing that runs along the thing so that you can ship logs metrics and traces. That copies the same rough pattern. But for the languages when we're talking about core vs contrib this is what we're talking about.

opentelemetry-python (core) The API, SDK, OTLP exporter, context propagation, resource detection primitives
opentelemetry-python-contrib Instrumentation libraries for Flask, Django, requests, psycopg2, Redis, Kafka, boto3, etc.

Stuff that breaks goes in contrib, stuff that doesn't break goes into core.

Now the reason this causes a conflict. contrib is massive overkill for most projects. You don't want 300 exporters to add the one you typically need. On the language side, this isn't that big of a problem. pip install opentelemetry-instrumentation-flask gives you the stuff you need for flask. However on the collector side you end up having to do the OpenTelemetry Collector Builder to make your own collector (or just kinda ride the wave and hope it works out). While cool that this exists, it's a lot of scope to ask a team to take on.

Process of adding a new feature

So I believe I have captured the workflow of adding a new feature to OTel. You can check my homework here:

  1. OpenTelemetry Enhancement Proposal (OTEP) (https://github.com/open-telemetry/opentelemetry-specification/tree/main/oteps/)
  2. Once the OTEP is accepted, the text goes into the Specification directory in the same repo.
  3. After that it seems to go to Semantic conventions. This seems to be where we get down to the specific details and where most of the long discussions seem to live. At this point we're talking about more or less a permanent commitment to this design and where the lock-in process becomes very hard to change.
  4. Each of the SDKs implements the API surface that is defined in the specification. Now some of the SDKs have done 2.0 breaking changes, so it does seem like the earlier "please no 2.0 at all costs" sentiment has been abandoned (which I think is smart and good).
  5. Contrib / instrumentation. This is slightly more mushy. Looks like they should track latest API/SDK but each contrib package may version independently so its more flexible as a design.
  6. Collector + OTLP. The data has to actually go somewhere. OTLP (wire protocol) has its own stability lifecycle and specification (here). Collector components have their own stability in their READMEs and as far as I can tell that's kinda all over the place.

Things I'm not really clear on

  • It's unclear how long the OTEP -> Specification process takes. I've looked through the Git history but there doesn't seem to be any predictable number or cycle.
  • I don't fully understand what is the relationship between all these stability commitments. Does Collector + OTLP group work in lockstep? Can a language "fall out of scope" if you lag too far behind?

Attempting to test it

So because OpenTelemetry is a CNCF project, I figured it made the most sense to compare them to other CNCF projects. My basis for comparison is Envoy and Prometheus. I have used a hacky Python script I've used before for measuring the "health" of open-source projects, which is probably not the best. However I'll include a link to the raw data without the charts so folks can review it and (more than likely) find a problem in what I generated.

So we look at 24 months of activity for Envoy and what we see is a pretty healthy project. There's good distribution of authors, mergers, issue closers. phlax is obviously pretty important to the project but in general there's a good bench of people to step in if needed. I've attempted to filter out all the known bot traffic.

Let's compare that to one of the OpenTelemetry languages. The ones I have the most professional experience with are Golang and Python, but I hear from a lot of folks in the community that the Ruby and PHP ones struggle a lot. This is the PHP one for the same period.

So we see pretty clearly that there's way too much concentrated on 2 people. This is not a healthy open-source project and they clearly don't have enough people to cover the kind of scope OTel needs to cover. Same story with Ruby.

In comparison the "strongest" OpenTelemetry SDKs in my opinion, Golang and Dotnet (although Python is also no slouch) look more healthy.

Golang

So the first issue is maybe the least surprising. There's too much concentration among too few maintainers. Your authors shouldn't also be your mergers and your issue closers. Ideally these tasks should be distributed out more evenly.

For what its worth I think the maintainers have done a good job of attempting to keep their discussions public. It was very easy for me to find the public meeting notes of the different groups of maintainers, read through them and see what was going on. I don't get the sense that these maintainers are trying to stop people from getting involved as much as the expectations of stability have, more or less, frozen the project in place.

The issue is more a classic case of "someone has to pay the maintainers". The project is too complex for someone to realistically do this as a hobby. I think any project signing on for such long stability contracts cannot turn to the community of hobbyists expecting assistance. I can't join calls and do the things I would be expected to do for a project of this size and importance for free. But it also means that the people doing this critical work have expectations placed on them by their parent organizations.

Repository 24mo Merged PRs Distinct Mergers Top-1 Merger % Top Merger Role
opentelemetry-cpp 544 4 86.1% Single human (marcalff)
opentelemetry-kotlin 281 2 79.7% Single human (fractalwrench)
opentelemetry-browser 102 4 79.5% Single human
opentelemetry-ruby 213 5 78.7% Single human
opentelemetry-js 829 14 64.9% Highly concentrated
opentelemetry-python 486 4 61.4% Single human (xrmx)
opentelemetry-php 181 2 53.0% Two mergers total
semantic-conventions 911 9 49.7% Single human (lmolkova)
opentelemetry-go 686 5 36.9% Distributed bench
opentelemetry-dotnet 657 6 31.5% Distributed bench
prometheus 1,849 31 14.4% Broad bench
envoy 5,432 28 35.8% Broad bench

So these SDKs have too few maintainers. But that doesn't fully explain why it seems to take so long for new features to get through the stack. My guess for that was that somewhere in the process between submission of the new idea and the formalization of the idea was a long discussion that took a million years.

Conventions about Semantics

So with this level of surface area across different frameworks and languages, it makes sense to concentrate the conversation about conventions in one place. That lives here: https://github.com/open-telemetry/semantic-conventions

If vendor debate is causing the slowdown, we should (in theory) see this slowdown in PRs here. Then you should see the slowdown basically propagate out. Spoiler alert, I was wrong about this. Big thanks to the OpenTelemetry people for having good conventions on labeling their PRs which made this much easier.

So if semconv is the slowdown, let's look at the slowest PRs there.

PR days comments reviews labels topic
#2083 277.5 17 115 area:gen-ai MCP semantic conventions
#2617 258.6 29 13 area:gcp GCE instance labels
#1698 187.9 3 7 area:azure, breaking rename `azure_` → `azure.`
#2619 174.6 24 8 area:gcp GCE instance group manager
#3118 147.1 19 8 area:graphql, breaking GraphQL Recommended vs Opt-In
#1741 141.0 4 23 changelog.opentelemetry.io Mainframes
#1784 127.3 7 48 area:k8s k8s.container.status metrics
#2287 118.5 12 95 area:rpc ONC/Sun RPC + NFS metrics
#2179 117.0 7 114 area:gen-ai, breaking Gen-AI chat history attributes

Yeah some of them are pretty slow, but there are some complex topics being discussed. However interestingly this slowdown doesn't really trickle into the SDK/API space, suggesting that OpenTelemetry is going a good job of keeping these conversations siloed off.

If we look at Python we see that their slowest PRs aren't semconv related.

PR days comments reviews labels topic
#4646 361.1 5 19 — OpAMP integration sketch
#4576 314.0 11 27 Stale OTLP HTTP max_export_batch_size
#4609 253.7 2 9 — env carrier
#4709 172.1 8 40 — http exporter error handling
#4333 164.9 6 4 — GRPC exporter backoff config
#4654 161.7 7 14 log-breaking-changes deprecate events API/SDK
#4863 155.0 5 36 — add/remove metric readers at runtime
#4647 152.9 4 9 Approve Public API check, log-breaking-changes rename Log → LogRecord
#4854 150.5 7 9 hold W3C traceparent random-trace-id
#4676 126.4 20 30 Approve Public API check, log-breaking-changes logs SDK refactor

In reality the slowdown for these are the extra required check imposed by the Approve Public API check which requires another maintainer. But that seems appropriate and takes us back to the initial problem of "not enough maintainers".

Potential Solutions

So after looking at all of this, the pattern becomes clear. A new feature takes a very long time to make it to the end user in OpenTelemetry because they take stability very seriously, combined with a relatively limited bench of talent to pull from. Once things make it through the entire stack, implementing the API and getting that API change through to the end user falls on an overworked maintainer pool. So what do we do?

I think one idea worth exploring is adding some sort of time-bound beta tier. Basically between the "Experimental" and the "Stable" in the following diagram. The problem is that for end users, due to the extra steps to use Experimental features, they might as well not exist. 99% of us have no idea when an experimental feature is added and we would never engage with it. But if I knew the feature would stick around for at least 12 months without a removal and was more accessible to me as an end user, it could actually help the project get more actionable feedback.

Basically a feature would go Experimental (pretty low usage) -> Beta (more exposed to the end user than Experimental) -> 12 months -> Removal or Stable.

Now confusingly Beta exists for Otel but is used for SDKs, not for components. Like Rust is a Beta but it seems like Profiles cannot be a Beta. Honestly it's nearly impossible for me to figure out like what labels should apply to what things. I suspect nobody really knows. Here's the explanation of Beta that I think only applies to SDKs. 

Development

Not all pieces of the component are in place yet, and it might not be available for users yet. Bugs and performance issues are expected to be reported. User feedback around the UX of the component is desired, such as for configuration options, component observability, technical implementation details, and planned use-cases for the component. Configuration options might break often depending on how things evolve. The component SHOULD NOT be used in production. The component MAY be removed without prior notice.

Alpha

This is the default level: any components with no explicit maturity level should be assumed to be "Alpha". The component is ready to be used for limited non-critical production workloads, and the authors of this component welcome user feedback. Bugs and performance problems are encouraged to be reported, but component owners might not work on them immediately. The component's interface and configuration options might often change without backward compatibility guarantees. Components at this stage might be dropped at any time without notice.

Beta

Same as Alpha, but the interfaces (API, configuration, generated telemetry) are treated as stable whenever possible. While there might be breaking changes between releases, component owners should try to minimize them. A component at this stage is expected to have had exposure to non-critical production workloads already during its Alpha phase, making it suitable for broader usage.

Release Candidate

The component is feature-complete and ready for broader usage. The component is ready to be declared stable, it might just need to be tested in more production environments before that can happen. Bugs and performance problems are expected to be reported, and there's an expectation that the component owners will work on them. Breaking changes, including configuration options and the component's output, are only allowed under special circumstances. Whenever possible, users should be given prior notice of the breaking changes.
Stable

The component is ready for general availability. Bugs and performance problems should be reported, and there's an expectation that the component owners will work on them. Breaking changes, including configuration options and the component's output, are only allowed under special circumstances. Whenever possible, users should be given prior notice of the breaking changes.

Deprecated

Development of this component is halted. No new versions are planned, and the component might be removed from its included distributions. Note that new issues will likely not be worked on except for critical security issues. Components that are included in distributions are expected to exist for at least two minor releases or six months, whichever happens later. They also MUST communicate in which version they will be removed, either in terms of a concrete version number or the date of a release, like: "the first release after 2023-08-01".

Unmaintained

A component identified as unmaintained does not have an active code owner. Such components may have never been assigned a code owner, or a previously active code owner has not responded to requests for feedback within 6 weeks of being contacted. Issues and pull requests for unmaintained components SHOULD be labeled as such. After 6 months of being unmaintained, these components MAY be deprecated. Unmaintained components are actively seeking contributors to become code owners.

In addition it is, respectfully, misleading to imply that Go and Ruby are being maintained at the same standard. This isn't a shot at the Ruby folks — they are doing heroic work with what they have. But pretending parity exists when it doesn't just creates confusion and quiet resentment when a user shows up expecting one experience and gets another. Being honest about maintenance tiers would let people make informed choices and might attract more help to the other tiers by naming the problem out loud.

Finally I would try to surface these problems more openly for OpenTelemetry from the perspective of "we need more maintainers". I feel like the people doing this work probably knew there was a problem, but it seems like the community at large has no idea that there is a need for frankly more engaged ideally independent maintainers and contributors.

OpenTelemetry is a great project that is doing great work. It's doing, frankly, heroic work at this scale with this few people. But I think in order to actually replace the vendor specific SDKs we need to start getting a bit more pragmatic about what is realistic to do in terms of stability contracts and number of languages. I don't think breaking changes are as devastating to the community as these promises imply as long as they are communicated well and I think with this thin of a bench of maintainers, something has to give.

Anyway feel free to check my data for accuracy and let me know if you find problems!

data
data.zip
343 KB
download-circle

Clickhouse is winning the Observability Wars

12 min read in Observability

For roughly the last ten years, a meaningful percentage of my working hours have been spent thinking about observability. If you're not familiar with the term, "observability" is what we call it now that "monitoring" doesn't sound expensive enough. The actual work is unglamorous in that you collect a lot of logs, some metrics, a few traces, and then you give them to people.

I generally like my job. I like that we're always trying new ideas and approaches. I like the fact that when things go wrong, the answer is almost always sitting there in the data, waiting to be found by whoever is patient enough to look. But I want to be honest with you: in ten years of doing this work, across a half-dozen companies and every observability platform you've heard of and a few you probably haven't, logs have never stopped being the worst part of the job. They were the worst part when I started. They are the worst part today. I fully expect them to be the worst part of this job forever until the robots rise up and rip my head off in one clean sweep.

I've written about why logs are terrible before, so I'll spare you the full lecture and give you the short version.

Every developer's expectations for logs are set by a single formative experience: the syslog box. Or a container running locally. Or tail -f on a production server they probably shouldn't have SSH'd into. The point is that at some early, tender moment in their career, they had an experience with logs that was flawless. They ran grep and something useful came back. They piped it into jq and got exactly what they needed.

This experience is the observability equivalent of a first kiss. It ruins them for everything that comes after.

Because here is the thing about that flawless experience: it works because the system is small, the volume is trivial, and the person querying is the same person who wrote the log line. There is no schema drift, no cardinality explosion, no cross-team consumer with dashboard expectations, no VP asking why the "revenue events" graph has a gap in it.

Then there are forty services. Now there are four hundred. Now the logs are being consumed not just by developers but by customer service, who need to look up a specific user's failed checkout from Tuesday. And by the data team, who are quietly building a business-critical dashboard on top of a log line that a backend engineer is about to refactor without telling anyone. And by the on-call, who at 3 AM does not want to learn a new query language, does not want to think about index patterns, and would like the search bar to just work.

So you have a technical problem — the volume is enormous, the shape is inconsistent, the queries are unpredictable — sitting on top of an expectations problem, which is worse. Developers want logs instantly, they want to run arbitrary operations on them, and they will not commit to a schema. Meanwhile the less-technical consumers of that same data want the dashboards to be stable forever, the UI to be forgiving, and the whole thing to feel like a normal product. These two audiences are, in most practical respects, at war with each other, and you are the diplomat.

Clickhouse

ClickHouse came out of Yandex, where it was built to chew through analytical queries against absurd volumes of clickstream data. It was not designed for observability. It just happens to be shockingly good at it, because clickstream data and observability data have a lot in common: high volume, append-heavy, time-ordered, mostly read in aggregate, and every so often you need to reach in and find one specific needle.

You can run it yourself with Helm charts. You can point Grafana at it via the ClickHouse plugin, or use their own web UI, or bring your own frontend. Their docs are actually good, which I mention because it's rare enough to be worth flagging. I've never used their ClickStack setup though, so YMMV.

For observability specifically, the OpenTelemetry Collector has a ClickHouse exporter, which means you can pipe OTLP data straight in and let it manage the initial schema for you. ClickHouse is designed to scan billions of rows and ingest an amount of data that, when you first see the numbers, makes you assume they're lying. They're not lying. You query it with SQL, which is a language that already exists and was not created by a startup two weeks ago.

But why Clickhouse specifically for logs?

I'm ranting about logs and then I'm explaining why I like to administer Clickhouse more. Let me take a second and explain why Clickhouse is really good at logs at scale.

Logs, as a data shape, have some peculiar properties. They're append-only. You never update a log line, and you almost never delete a single one, though you delete a lot of them at once when retention kicks in. They arrive roughly in time order, though never actually in order. They're read in bursts where nobody looks at logs for days, and then during an incident somebody wants to scan a billion of them in seconds. They're highly compressible, because most of the bytes in your logs are repeated: the same service names, the same hostnames, the same error strings, the same JSON keys, over and over and over again. And critically, when you query them, you almost always want either a narrow time range across all fields or an aggregation across a wide time range with a few filters. You very rarely want "give me one specific row by ID" the way you would from a transactional database. (There are exceptions when its something like GDPR or compliance logging which is its own subgenre of nightmares).

In a row-oriented database — Elasticsearch, Postgres, MySQL — the data for a single log line is stored together on disk. If your log has 40 fields and your query only cares about 3 of them, tough luck, you're reading all 40 from disk anyway. The database will filter it in memory, but the disk I/O has already happened.

ClickHouse stores each column separately. If your query says SELECT service, status_code, count() FROM logs WHERE timestamp > now() - INTERVAL 1 HOUR GROUP BY service, status_code, ClickHouse reads exactly three columns off disk: timestamp, service, and status_code. The other 37 columns in your schema might as well not exist. On observability data, where you often have dozens of attributes but any given query touches three or four, this is the difference between scanning 800GB and scanning 40GB.

This is also why the compression numbers look absurd. Columnar data compresses far better than row-oriented data because the values within a single column are, by nature, similar to each other. A column of service_name values might have a hundred distinct strings across a billion rows. ZSTD eats that for breakfast. You'll routinely see 10–14x compression ratios on real observability data, compared to 2–3x for Elasticsearch.

That's not the amazing part though

The amazing thing is that ClickHouse scales without changing shape.

I don't know how else to say this. Every other observability backend I've worked with mutates as it grows. The architecture at 1 TB a day and the architecture at 10 TB a day are recognizably different systems, with different failure modes, different ops burdens, and different mental models. ClickHouse at 10 TB a day looks like ClickHouse at 1 TB a day with more shards. That's it. That's the pitch. That's the whole reason I'm writing this.

Let me show you what I mean.

1 TB a Day

At 1 TB a day, every modern observability stack is roughly okay. If you're at this scale, you can pick almost anything and be productive. The differences below are real but they're not yet painful.

Elasticsearch

  • A relatively vanilla Elasticsearch cluster with Logstash providing some buffer between ingest and the Lucene indexes. Users get full-text search, which is genuinely good — this is the thing Elasticsearch is actually best at, and at this scale it delivers. Mapping explosions are already a background risk with mixed data, so dynamic mapping needs to be disabled or carefully templated from day one. ILM policies (hot → warm → delete) are non-optional even at this size, because forgetting to set them is how you get paged on a Saturday about disk pressure. Roughly $6–9K/month.

LGTM

  • Nothing too crazy. Alloy (formerly Grafana Agent, RIP) unifies the collection story into a single daemon, which is nice. Loki works well as long as you spend some time educating developers on how to attach useful labels — a conversation you will have many times, with many people, for the rest of your career. Mimir and Tempo largely do what it says on the tin. Roughly $3.5–5K/month.

Datadog

  • At 1 TB a day, Datadog is genuinely great. This is the scale it was built for, and it shows. You install the agent, you look at dashboards, you go home. There is almost nothing to think about, which is the entire point. You can already see the shape of the cost problem lurking in the diagram — the metered pipelines, the indexed-vs-ingested logs distinction, the custom metrics cardinality tax — but at this scale it's manageable. Roughly $45–75K/month, though negotiated pricing varies enough that I'd take that number with a grain of salt the size of a fist.
  • Datadog's whole pricing philosophy is that they save you a full-time engineer. I think that framing is somewhat deranged, but they are extremely rich and I am not, so consider your source.

Clickhouse

Here is the honest truth: at 1 TB a day, ClickHouse is not less complicated than its peers. It's roughly the same. Maybe slightly more, if you count the schema design work you have to do up front. You get 10–14x compression with ZSTD and proper codecs, the Altinity Operator handles keeper coordination and the whole thing runs in about seven pods. But you do have to design your schemas. ORDER BY keys matter enormously. There is no native PromQL, so metrics workflows go through the Grafana plugin or through chproxy and an adapter. Roughly $1.5–2.5K/month.

If you took the diagrams at this tier and squinted, you'd say they're all in the same weight class. And you'd be right. Now watch what happens next.

5 TB a Day

This is where the exponential curve kicks in for everybody except one of these.

Elasticsearch

  • Kafka is no longer optional. At 5 TB a day, direct writes into Elasticsearch cause bulk-reject storms and backpressure that will absolutely take your cluster down during a traffic spike. So now you're running Kafka, which means you're either running Kafka well or you're about to have a second, entirely different set of problems. Shard math becomes critical — at 50GB target shards, you're minting ~200 shards a day counting replicas, and your cluster state size becomes its own concern. You almost certainly need Elastic's commercial license for searchable snapshots and the frozen tier. Roughly $40–55K/month before licensing.
  • That but Kafka

LGTM stack

  • You are now in microservices mode, whether you wanted to be or not. That means 65+ pods across three separate systems, each with its own compaction pipeline, its own hash ring, its own memcached tier. The gossip/memberlist ring becomes a real operational concern; ingester rollouts require careful -ingester.autoforget-unhealthy tuning, and if you get it wrong you either lose data or duplicate it. Roughly $22–32K/month.

Datadog

  • The operational complexity is still low, in that you don't run any servers. But you now need a full pipeline team whose entire job is reducing your Datadog bill. Exclusion filters, sampling rules, cardinality caps, tag allow-lists, the whole apparatus. This is what I call the "you build a system to avoid using the system you're paying for" trap, and once you're in it, you are in it forever. Roughly $180–350K/month, depending on how aggressive the pipeline team gets.
  • This is also where you are basically fighting with your SaaS provider all the time, pouring over their billing documentation to figure out how to reduce costs. It's a hostile relationship and one I don't enjoy.

Clickhouse

You'll notice, if you look at the diagram, that I basically just added shards. That's it. That's the change. Same operator, same query engine, same query language, same mental model. Rebalancing after adding shards is manual, which is a real trade-off — most teams pre-provision or use weighting on Distributed tables to sidestep it. Materialized views for dashboard rollups shift from "nice to have" to "essential." Roughly $7–11K/month.

The gap between ClickHouse and everything else opens up here. It doesn't close.

10 TB a Day

This is where most solutions genuinely stop working, in the sense that even a well-staffed internal team cannot keep up with the operational load.

Elasticsearch

  • You are now running three separate Elasticsearch clusters — one for logs, one for metrics, one for APM — federated through Cross-Cluster Search. Hot-tier NVMe cost dominates the bill. This is the scale at which teams start seriously evaluating alternatives, and where a lot of the recent migrations to ClickHouse have originated. Roughly $95–140K/month plus commercial licensing.
  • You need people who are legitimate experts on Elasticsearch. Now thankfully Elastic just laid a ton of those people off, so they're probably possible to get, but still. Running this thing at this size is very complicated.

LGTM

  • Around 180+ pods, zone-aware everything, split-and-merge compaction, per-tenant limits, shuffle sharding to prevent noisy neighbors. You almost certainly have a dedicated observability platform team of three to five engineers at this point. If you don't, get ready for a bad fucking time. Roughly $55–85K/month.

Datadog

  • Still very easy to run, in the strict sense that you don't run anything. But your bill is now measured in six or seven figures a month, and the org has almost certainly built a pre-processing pipeline team whose entire existence is dedicated to reducing that bill. Most companies at this scale have gone hybrid: Datadog for APM and high-value metrics, self-hosted (increasingly ClickHouse) for logs. The complexity paradox at this scale is that you now have Datadog's simplicity plus your pipeline complexity plus a second self-hosted stack. Pricing is all over the goddamn place. You might be over a $1 million a month here.

Clickhouse

  • Look at the diagram and then look back at the 1 TB diagram. It's the same diagram. There are more shards. That's the difference. Materialized views for rollups are now mandatory rather than optional. Schema design mistakes you made two years ago will start to hurt, so hopefully you didn't make many. Rebalancing after adding shards is still manual; most teams pre-provision or use clickhouse-copier or a dual-write migration when they need to grow the cluster. Kafka starts to become useful as a buffer for very bursty ingest, though it's not required. Roughly $18–28K/month.

What does this mean?

If you've read this far, the point is probably already obvious, but I want to say it directly.

Every observability stack works at 1 TB a day. If you're small, pick whatever your team already knows. Life is short. We're all just waiting for the robots to kick our heads off like soccer balls.

Robot Robot Fights Human GIF - Robot Robot fights human Robot kick - Discover & Share GIFs
Click to view the GIF
TenorV

The question is not which stack works today. The question is which stack still resembles itself two years from now, when your data volume has 5x'd and your team has 2x'd and the person who originally designed the whole thing has left the company.

Elasticsearch mutates. LGTM mutates. Datadog stays operationally simple but mutates financially into something that requires its own dedicated team of accountants and pipeline engineers just to keep the bill from spiraling.

ClickHouse just gets wider. You add shards. That's the whole trick.

There is a real cost to this: you have to eat the schema-design and query-engine complexity up front, at a scale where the other options are objectively easier. You will be, briefly, the one making things harder for your developers. They will not always appreciate this. But the trade you're making is that their experience — and yours — remains roughly the same as the data grows by an order of magnitude, and the next order of magnitude, and probably the one after that.

I have spent ten years watching observability stacks change shape underneath me while I tried to keep them running. ClickHouse is the first one that hasn't and that has been able to actually scale with me. That's pretty incredible.