Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Saving Three Months of Latency with a Single OpenTelemetry Trace

checklyhq.com

Post reply on HN
Print view View original

31–40 of 46 posts

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#31

Post by encoderer »

redman25 wrote: post #27
simonbarker87 wrote: post #5

I wish posts like this would explore the relative savings rather than the absolute. On its own I don’t feel like that saving is really telling me much, taken to the extreme you could just not run the service at all and save all the time - a tongue in cheek example but in context is this saving a big deal or is it just engineering looking for small efficiencies to justify their time?

I often ask myself the same question. We have some user facing queries that slow the frontend down. I’ve fixed some slowness but it’s definitely not a priority. I wonder how much speed improvements correlate with increased revenue by happy customers.

Think of this like changing the oil in your car.

Over-optimizing is not going to help you at all but if you ignore it eventually it will all seize up.

You have to keep that stuff in check.

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#32

Post by serverlessmom »

hiatus wrote: post #26
serverlessmom wrote: post #25

Earlier quoted context omitted.

I think that’s flawed analysis, if you’re running FaaS then sure you can fail to see benefit from small improvements in time (AWS Lambda changed their billing resolution a few years back but before then the Go services didn’t save much money despite being faster) but if you’re running thousands of requests, and speeding them all up, you should be able to realize tangible compute savings whatever your platform.

Help me to understand, then. If this stuff is being done on an autoscaling cluster, I can see it, but if you are just running everything on an always-on box for instance, it is less clear to me. edit: Do you have an affiliation with the blog? I ask because you have submitted several articles from checkly in the past.

Yes I work at Checkly, though I didn’t answer authoritatively since this one wasn’t written by me!

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#33

Post by dmurray »

simonbarker87 wrote: post #5

I wish posts like this would explore the relative savings rather than the absolute. On its own I don’t feel like that saving is really telling me much, taken to the extreme you could just not run the service at all and save all the time - a tongue in cheek example but in context is this saving a big deal or is it just engineering looking for small efficiencies to justify their time?

The units seem wrong in any case. It's 3 months of compute per day, which is actually much more impressive.

If we think about the business impact, we don't usually think of compute expenditure per-day, so you might reasonably say, the fix saved 90 years of annual compute. Looks better in your promotion packet, too.

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#34

Post by Veserv »

harisamin wrote: post #12

On the noisy NodeJS auto-instrumentation, it is indeed very noisy out of the box. Myself along with a bunch of other ppl finally got the project to allow you to select the instrumentations via configuration. Saves having to create your own tracer.ts/js file. Here's the PR that got merged earlier in the year: https://github.com/open-telemetry/opentelemetry-js-contrib/p... The env var config is `OTEL_NODE_ENABLED_INSTR…

Why would you disable instrumentation instead of just filtering the recorded log?

That only makes sense if the instrumentation overhead itself is significant. But, for a efficient recording implementation that should only really start being a problem when your average span is ~1 us.

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#35

Post by tnolet »

AtlasBarfed wrote: post #19

It's AWS, if you shake a stick at some network transfer optimization or storage/EBS/S3 you'll save three engineers salary.

This deserves an updoot. We reached a level of scale right now at Checkly that all of these things start adding up. We moved workloads off of S3 to Cloudflare R2 because of this.

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#36

Post by tnolet »

Veserv wrote: post #34
harisamin wrote: post #12

On the noisy NodeJS auto-instrumentation, it is indeed very noisy out of the box. Myself along with a bunch of other ppl finally got the project to allow you to select the instrumentations via configuration. Saves having to create your own tracer.ts/js file. Here's the PR that got merged earlier in the year: https://github.com/open-telemetry/opentelemetry-js-contrib/p... The env var config is `OTEL_NODE_ENABLED_INSTR…

Why would you disable instrumentation instead of just filtering the recorded log? That only makes sense if the instrumentation overhead itself is significant. But, for a efficient recording implementation that should only really start being a problem when your average span is ~1 us.

Oh, simple answer. The tools you use to inspect those traces just blow up with noise. Like a trace that shows 600+ of file reads that all take less than half a millisecond.

This is all noise when you are trying to debug more common issues than your FS being too slow.

+ also storage cost. Most vendors charge by Mb stored or span recorded.

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#37

Post by Veserv »

tnolet wrote: post #36
Veserv wrote: post #34

Earlier quoted context omitted.

Why would you disable instrumentation instead of just filtering the recorded log? That only makes sense if the instrumentation overhead itself is significant. But, for a efficient recording implementation that should only really start being a problem when your average span is ~1 us.

Oh, simple answer. The tools you use to inspect those traces just blow up with noise. Like a trace that shows 600+ of file reads that all take less than half a millisecond. This is all noise when you are trying to debug more common issues than your FS being too slow. + also storage cost. Most vendors charge by Mb stored or span recorded.

That is why I mentioned post-filtering the recording as the alternative. Grab the full recording then filter to just the relevant results before inspection.

For that matter, why are a few hundred spans a problem? Are the visualizers that poor? I usually use function tracing where hundreds of millions to billions of spans per second are the norm and there is no difficulty managing or understanding those.

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#38

Post by tnolet »

Veserv wrote: post #37
tnolet wrote: post #36

Earlier quoted context omitted.

Oh, simple answer. The tools you use to inspect those traces just blow up with noise. Like a trace that shows 600+ of file reads that all take less than half a millisecond. This is all noise when you are trying to debug more common issues than your FS being too slow. + also storage cost. Most vendors charge by Mb stored or span recorded.

That is why I mentioned post-filtering the recording as the alternative. Grab the full recording then filter to just the relevant results before inspection. For that matter, why are a few hundred spans a problem? Are the visualizers that poor? I usually use function tracing where hundreds of millions to billions of spans per second are the norm and there is no difficulty managing or understanding those.

In most cases these traces are shipped over the wire to a vendor. Only that will cost $$. Then, not all vendors have tail sampling as a "free" feature. So, in many cases it's better to not record at all.

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#40

Post by tmpz22 »

tnolet wrote: post #35
AtlasBarfed wrote: post #19

It's AWS, if you shake a stick at some network transfer optimization or storage/EBS/S3 you'll save three engineers salary.

This deserves an updoot. We reached a level of scale right now at Checkly that all of these things start adding up. We moved workloads off of S3 to Cloudflare R2 because of this.

> We moved workloads off of S3 to Cloudflare R2 because of this.

So you moved from a mature but expensive storage solution to a younger currently subsidized storage solution? What happens when R2 jacks up pricing?

Post reply on HN