2006 - Bryan Cantrill publishes this work on software observability https://queue.acm.org/detail.cfm?id=1117401
2015 - Ben Sigelman (one of the Dapper folks) cofounds Lightstep
Observability's past, present, and future
11–20 of 31 posts
Re: Observability's past, present, and future
#12Strong post. The mismatch between observability effort and reliability gains feels real. The bottleneck seems cognitive, not technical.
Re: Observability's past, present, and future
#13Observability and APM are way, way older than depicted in this simplistic post!
Hello! Yes, you are right - observability and APM have both been around for many decades, but the incarnations that most people are familiar with are the ones that emerged in the 2010s.
My intention wasn't for this post to be a comprehensive historical record. That would have taken many more words & would have put everyone to sleep. My goal was to unpack and analyze _modern observability_ - the version that we are all dealing w/ today.
Good point though!
Re: Observability's past, present, and future
#14> Observability made us very good at producing signals, but only slightly better at what comes after: interpreting them, generating insights, and translating those insights into reliability. I'm a data professional who's kind of SRE adjacent for a big corpo's infra arm and wow does this post ring true for me. I'm tempted to just say "well duh, producing telemetry was always the low hanging fruit, it's the 'generating…
This is a super insightful comment & there is a bunch that I want to respond to but I can't do it all neatly in one comment. Hahaha
I'll choose this point:
> reliability is still ultimately an incentive problem
This is a fascinating argument and it feels true.
Think about it. Why do companies give a shit about reliability at all? They only care b/c it impacts bottom line. If the app is "reliable enough" such that customers aren't complaining and churning, it makes sense that the company would not make further investments in reliability.
This same logic is true at all levels of the organization, but the signal gets weaker as you go down the chain. A department cares about reliability b/c it impacts the bottom line of the org, but that signal (revenue) is not directly and attributable to the department. This is even more true for a team, or an individual.
I think SLOs are, to some extent, a mechanism that is designed to mitigate this problem; they serve as stronger incentive signals for departments and teams.
Re: Observability's past, present, and future
#152006 - Bryan Cantrill publishes this work on software observability https://queue.acm.org/detail.cfm?id=1117401 2015 - Ben Sigelman (one of the Dapper folks) cofounds Lightstep
These are great. I should have included them in my timeline!
Huge fan of historical artifacts like Cantrill's ACM paper
Re: Observability's past, present, and future
#16Of course that sucks. Just enable full time-travel recording in production and then you can use a standard multi-program trace visualizer and time travel debugger to identify the exact execution down to the instruction and precisely identify root causes in the code. Everything is then instrumented automatically and exhaustively analyzable using standard tools. At most you might need to add in some manual instrumentat…
You're not the first person I've met that has articulated an idea like this. It sounds amazing. Do you have an idea about why this approach isn't broadly popular?
Re: Observability's past, present, and future
#17We need more automation. Less data, more insight. We're at the firehose stage, and nobody's got time for that. ML-based anomaly detection is not widespread and automated RCA barely exists. We'll have solved the problem when AI detects the problem and submits the bug fix before the engineers wake up.
You're so right.
> We'll have solved the problem when AI detects the problem and submits the bug fix before the engineers wake up.
Working on it :)
Re: Observability's past, present, and future
#18First. Love that more tools like Honeycomb (amazing) are popping up in the space. I agree with the post. But. IMO, statistics and probability can’t be replaced with tooling. As software engineering can’t be replaced with no-code services to build applications… If you need to profile some bug or troubleshoot complex systems (distributed, dbs). You must do your math homework consistently as part of the job. If you don’…
Vibe-coders don't comprehend how the code works, yet they can create impressive apps that are completely functional.
I don't see why the same isn't true for "vibe-fixers" and their data (telemetry).
Re: Observability's past, present, and future
#19I share a lot of this sentiment, although I struggle more with the setup and maintenance than the diagnosis. It's baffling to me that it can still take _so_much_work_ to set up a good baseline of observability (not to mention the time we spend on tweaking alerting). I recently spent an inordinate amount of time trying to make sense of our telemetry setup and fill in the gaps. It took weeks. We had data in many system…
I completely agree w/ your points about why observability sucks:
- Too much setup
- Too much maintenance
- Too steep of a learning curve
This isn't the whole picture, but it's a huge part of the picture. IMO, observability shouldn't be so complex that it warrants specialized experience; it should be something that any junior product engineer can do on their own.
> I can definitely imagine having Claude debug an issue faster than I can type and click around dashboards and query UIs. That sounds fun.
Working on it :)
Re: Observability's past, present, and future
#20Log stores should be first-class members, allowing other services to read logs too, not just write.
We need better primitives and protocols for event-based communication. Logging configuration should be mostly about routing and storage.