Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Observability's past, present, and future

blog.sherwoodcallaway.com

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Observability's past, present, and future

#21

Post by donavanm »

shcallaway wrote: post #16
Veserv wrote: post #8

Of course that sucks. Just enable full time-travel recording in production and then you can use a standard multi-program trace visualizer and time travel debugger to identify the exact execution down to the instruction and precisely identify root causes in the code. Everything is then instrumented automatically and exhaustively analyzable using standard tools. At most you might need to add in some manual instrumentat…

You're not the first person I've met that has articulated an idea like this. It sounds amazing. Do you have an idea about why this approach isn't broadly popular?

cost and compliance are non-trivial for non-trivial applications. Universal instrumentation and recording creates a meaningful fixed cost for every transaction, and you must record ~every transaction; you can't sample & retain post-hoc. If you're processing many thousands of TPS on many thousands of nodes that quickly adds up to a very significant aggregate cost even if the individual cost is small.

For compliance (or contractual agreement) there are limitations on data collection, retention, transfer, and access. I certainly don't want private keys, credentials, or payment instruments inadvertently retained. I dont want confidential material to be distributed out of band or in an uncontrolled manner (like your dev laptop). I probably don't even want employees to be able to _see_ "customer data." Which runs head long in to a bunch challenges where low level trace/sampling/profiling tools have more less open access to record and retain arbitrary bytes.

Edit: Im a big fan of continuous and pervasive observability and tracing data. Enable and retain that at ~debug level and filter + join post-hoc as needed. My skepticism above is about continuous profiling and recording (ala vtune/perf/ebpf), which is where "you" need to be cognizant of risks & costs.

Re: Observability's past, present, and future

#22

Post by donavanm »

shcallaway wrote: post #14
tech_ken wrote: post #4

> Observability made us very good at producing signals, but only slightly better at what comes after: interpreting them, generating insights, and translating those insights into reliability. I'm a data professional who's kind of SRE adjacent for a big corpo's infra arm and wow does this post ring true for me. I'm tempted to just say "well duh, producing telemetry was always the low hanging fruit, it's the 'generating…

This is a super insightful comment & there is a bunch that I want to respond to but I can't do it all neatly in one comment. Hahaha I'll choose this point: > reliability is still ultimately an incentive problem This is a fascinating argument and it feels true. Think about it. Why do companies give a shit about reliability at all? They only care b/c it impacts bottom line. If the app is "reliable enough" such that cus…

I'd +1 incentives, primarily P&L/revenue/customer acquisition/retention, with a small carve out for "culture." I've worked places, and for people, where the culture was to "do the right thing" or focus on user experience as the objective which influenced decisions like paying more (time and money) for better support. For the SDEs and line teams it wasnt about revenue or someone yelling at them, they just emulated the behavior they saw around them which led to better observability/introspection/reliable/support. Which, of course, we'd like to believe leads to long term to success and $$$$.

I also like the call out of SLOs (or OKR or SMART goals or whatever) as a mechanism to broadcast your priorities and improve visibility. BUT I've also worked places where they didnt work because the ultimate owner with a VP title didnt care or understand to buy in to it.

And of course theres the hazard of principal agent problems between those selling, buying, building, and running are probably different teams and may not have any meaningful overlap in directly responsible individual.

Re: Observability's past, present, and future

#23

Post by jbs789 »

shcallaway wrote: post #18
vrnvu wrote: post #7

First. Love that more tools like Honeycomb (amazing) are popping up in the space. I agree with the post. But. IMO, statistics and probability can’t be replaced with tooling. As software engineering can’t be replaced with no-code services to build applications… If you need to profile some bug or troubleshoot complex systems (distributed, dbs). You must do your math homework consistently as part of the job. If you don’…

Vibe-coders don't comprehend how the code works, yet they can create impressive apps that are completely functional. I don't see why the same isn't true for "vibe-fixers" and their data (telemetry).

The distinction is originality vs replicating existing.

I believe the author is in the former camp.

Re: Observability's past, present, and future

#24

Post by hommes-r »

tech_ken wrote: post #4

> Observability made us very good at producing signals, but only slightly better at what comes after: interpreting them, generating insights, and translating those insights into reliability. I'm a data professional who's kind of SRE adjacent for a big corpo's infra arm and wow does this post ring true for me. I'm tempted to just say "well duh, producing telemetry was always the low hanging fruit, it's the 'generating…

I would add that "extremely sound data engineering" is also necessary to make observability cost-effective. Some of these otel platforms can burn 10%-25% of your cloud budget to show you your logs. That is insane.

Re: Observability's past, present, and future

#26

Post by ptx »

Wait a minute...

> with the rise of [...] microservices, apps were becoming [...] too complex for any individual to fully understand.

But wasn't the idea of microservices that these services would be developed and deployed independently and owned by different teams? Building a single app out of multiple microservices and expecting a single individual to debug it sounds like holding it wrong, which then requires this distributed tracing solution to fix it.

Re: Observability's past, present, and future

#28

Post by nijave »

Mostly agree although I think ease of instrumentation is getting pretty good. At least in the Python ecosystem, you set some env vars and run opentelemtry-bootstrap and it spits out a list of packages to add. Then you run your code with the otel cli wrapper and it just works.

Datadog is equally as easy.

That alone gets you pretty good canned dashboards on vendors that have built in APM views.

The rest definitely rings true and I suspect some of it has come with the ease of software development. You need to know less about computer fundamentals and debugging with the proliferation of high level frameworks, codegen, and AI.

I also have noticed a trend that brings observability closer to development with IDE integration which I think is a good direction. Having the info "silo'd" in an opaque mega data store isn't useful.

Post reply on HN