Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Saving Three Months of Latency with a Single OpenTelemetry Trace

checklyhq.com

Post reply on HN
Print view View original

21–30 of 46 posts

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#21

Post by hiatus »

janOsch wrote: post #8
simonbarker87 wrote: post #5

I wish posts like this would explore the relative savings rather than the absolute. On its own I don’t feel like that saving is really telling me much, taken to the extreme you could just not run the service at all and save all the time - a tongue in cheek example but in context is this saving a big deal or is it just engineering looking for small efficiencies to justify their time?

I'm the author of the post. You raise a good point about relative savings. Based on last week's data, our change reduced the task time by 40ms from an average of 3440ms, and this task runs 11 million times daily. This translates to a saving of about 1% on compute.

> This translates to a saving of about 1% on compute.

Does this translate to any tangible savings? I'm not sure what the checkly backend looks like but if tasks are running on a cluster of hosts vs invoked per-task it seems hard to realize savings. Even per-task, 40 ms can only be realized on a service like Lambda—ECS minimum billing unit is 1 second afaik.

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#22

Post by serverlessmom »

janOsch wrote: post #18
cdelsolar wrote: post #7

μs isn't picoseconds, it's microseconds, which are a million times bigger...

Thank you for pointing that out! You are correct, μs stands for microseconds, not picoseconds. I've corrected the mistake, and the update should be visible as soon as the CDN cache invalidates.

Every day I have more sympathy for the Mars Climate Orbiter team. https://science.nasa.gov/mission/mars-climate-orbiter/

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#23

Post by serverlessmom »

harisamin wrote: post #12

On the noisy NodeJS auto-instrumentation, it is indeed very noisy out of the box. Myself along with a bunch of other ppl finally got the project to allow you to select the instrumentations via configuration. Saves having to create your own tracer.ts/js file. Here's the PR that got merged earlier in the year: https://github.com/open-telemetry/opentelemetry-js-contrib/p... The env var config is `OTEL_NODE_ENABLED_INSTR…

This is so cool! I’ve had this exact problem before.

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#25

Post by serverlessmom »

hiatus wrote: post #21
janOsch wrote: post #8

Earlier quoted context omitted.

I'm the author of the post. You raise a good point about relative savings. Based on last week's data, our change reduced the task time by 40ms from an average of 3440ms, and this task runs 11 million times daily. This translates to a saving of about 1% on compute.

> This translates to a saving of about 1% on compute. Does this translate to any tangible savings? I'm not sure what the checkly backend looks like but if tasks are running on a cluster of hosts vs invoked per-task it seems hard to realize savings. Even per-task, 40 ms can only be realized on a service like Lambda—ECS minimum billing unit is 1 second afaik.

I think that’s flawed analysis, if you’re running FaaS then sure you can fail to see benefit from small improvements in time (AWS Lambda changed their billing resolution a few years back but before then the Go services didn’t save much money despite being faster) but if you’re running thousands of requests, and speeding them all up, you should be able to realize tangible compute savings whatever your platform.

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#26

Post by hiatus »

serverlessmom wrote: post #25
hiatus wrote: post #21

Earlier quoted context omitted.

> This translates to a saving of about 1% on compute. Does this translate to any tangible savings? I'm not sure what the checkly backend looks like but if tasks are running on a cluster of hosts vs invoked per-task it seems hard to realize savings. Even per-task, 40 ms can only be realized on a service like Lambda—ECS minimum billing unit is 1 second afaik.

I think that’s flawed analysis, if you’re running FaaS then sure you can fail to see benefit from small improvements in time (AWS Lambda changed their billing resolution a few years back but before then the Go services didn’t save much money despite being faster) but if you’re running thousands of requests, and speeding them all up, you should be able to realize tangible compute savings whatever your platform.

Help me to understand, then. If this stuff is being done on an autoscaling cluster, I can see it, but if you are just running everything on an always-on box for instance, it is less clear to me.

edit: Do you have an affiliation with the blog? I ask because you have submitted several articles from checkly in the past.

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#27

Post by redman25 »

simonbarker87 wrote: post #5

I wish posts like this would explore the relative savings rather than the absolute. On its own I don’t feel like that saving is really telling me much, taken to the extreme you could just not run the service at all and save all the time - a tongue in cheek example but in context is this saving a big deal or is it just engineering looking for small efficiencies to justify their time?

I often ask myself the same question. We have some user facing queries that slow the frontend down. I’ve fixed some slowness but it’s definitely not a priority. I wonder how much speed improvements correlate with increased revenue by happy customers.

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#28

Post by [deleted] »

simonbarker87 wrote: post #5

I wish posts like this would explore the relative savings rather than the absolute. On its own I don’t feel like that saving is really telling me much, taken to the extreme you could just not run the service at all and save all the time - a tongue in cheek example but in context is this saving a big deal or is it just engineering looking for small efficiencies to justify their time?

[deleted]

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#29

Post by simonbarker87 »

janOsch wrote: post #11
simonbarker87 wrote: post #10

Earlier quoted context omitted.

Thanks for the follow up, sounds like a decent saving and investment of time then.

Fun fact: it probably took more time to write up and refine the blog post than it did to hunt down that sneaky 40ms savings.

True but the value of the hunt and fix may really come from this blog post long term. Content marketing and all that

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#30

Post by tnolet »

hiatus wrote: post #26
serverlessmom wrote: post #25

Earlier quoted context omitted.

I think that’s flawed analysis, if you’re running FaaS then sure you can fail to see benefit from small improvements in time (AWS Lambda changed their billing resolution a few years back but before then the Go services didn’t save much money despite being faster) but if you’re running thousands of requests, and speeding them all up, you should be able to realize tangible compute savings whatever your platform.

Help me to understand, then. If this stuff is being done on an autoscaling cluster, I can see it, but if you are just running everything on an always-on box for instance, it is less clear to me. edit: Do you have an affiliation with the blog? I ask because you have submitted several articles from checkly in the past.

Hey Checkly founder here, we changed our infra quite a bit over the last ~1 year. Still, it's mostly ephemeral compute. We started actually on AWS Lambda. We are on a mix of AWS EC2 and EKS now, all autoscaled per region (we run 20+ of them).

It seems tiny, but in aggregate this will have an impact on our COGS. You are correct that if we had a fixed fleet of instances, the impact would have been not super interesting.

But still, for a couple of hours spent, this saves us quite some $1Ks per year.

Post reply on HN