Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Saving Three Months of Latency with a Single OpenTelemetry Trace

checklyhq.com

Post reply on HN
Print view View original

11–20 of 46 posts

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#11

Post by janOsch »

simonbarker87 wrote: post #10
janOsch wrote: post #8

Earlier quoted context omitted.

I'm the author of the post. You raise a good point about relative savings. Based on last week's data, our change reduced the task time by 40ms from an average of 3440ms, and this task runs 11 million times daily. This translates to a saving of about 1% on compute.

Thanks for the follow up, sounds like a decent saving and investment of time then.

Fun fact: it probably took more time to write up and refine the blog post than it did to hunt down that sneaky 40ms savings.

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#12

Post by harisamin »

On the noisy NodeJS auto-instrumentation, it is indeed very noisy out of the box. Myself along with a bunch of other ppl finally got the project to allow you to select the instrumentations via configuration. Saves having to create your own tracer.ts/js file.

Here's the PR that got merged earlier in the year: https://github.com/open-telemetry/opentelemetry-js-contrib/p...

The env var config is `OTEL_NODE_ENABLED_INSTRUMENTATIONS`

Anyways, love Opentelemetry success stories. Been working hard on it at my current company and yielding fruits already :)

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#13

Post by roboben »

harisamin wrote: post #12

On the noisy NodeJS auto-instrumentation, it is indeed very noisy out of the box. Myself along with a bunch of other ppl finally got the project to allow you to select the instrumentations via configuration. Saves having to create your own tracer.ts/js file. Here's the PR that got merged earlier in the year: https://github.com/open-telemetry/opentelemetry-js-contrib/p... The env var config is `OTEL_NODE_ENABLED_INSTR…

This is great. Last time I tried this I couldn’t even find a way in code to disable some.

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#14

Post by case »

brunoarueira wrote: post #9
simonbarker87 wrote: post #5

I wish posts like this would explore the relative savings rather than the absolute. On its own I don’t feel like that saving is really telling me much, taken to the extreme you could just not run the service at all and save all the time - a tongue in cheek example but in context is this saving a big deal or is it just engineering looking for small efficiencies to justify their time?

I agree, but this post looks like an advertisement about the service itself.

It’s literally on the company’s blog, which is partially about promoting the company’s service. What’s the issue with that?

(Long time happy Checkly user here, the service is fantastic)

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#15

Post by tnolet »

harisamin wrote: post #12

On the noisy NodeJS auto-instrumentation, it is indeed very noisy out of the box. Myself along with a bunch of other ppl finally got the project to allow you to select the instrumentations via configuration. Saves having to create your own tracer.ts/js file. Here's the PR that got merged earlier in the year: https://github.com/open-telemetry/opentelemetry-js-contrib/p... The env var config is `OTEL_NODE_ENABLED_INSTR…

That is awesome. Had no idea this was available as an env var. After diving into OTel for our backend, we also found some of this stuff is just too noisy. We switch it of using this code snippet, for anyone bumping into this thread:

   instrumentations: [getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-fs': {
        enabled: false,
      },
      '@opentelemetry/instrumentation-net': {
        enabled: false,
      },
      '@opentelemetry/instrumentation-dns': {
        enabled: false,
      },

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#17

Post by brunoarueira »

case wrote: post #14
brunoarueira wrote: post #9

Earlier quoted context omitted.

I agree, but this post looks like an advertisement about the service itself.

It’s literally on the company’s blog, which is partially about promoting the company’s service. What’s the issue with that? (Long time happy Checkly user here, the service is fantastic)

Not a problem, but the OP is questioning about the savings!

I, for example, like to dive more on insights like the relative savings vs absolut to learn the approaches other engineers take! It's all about metrics we should take care.

(I'll put this service on my list to try someday, looks like fantastic indeed)

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#18

Post by janOsch »

cdelsolar wrote: post #7

μs isn't picoseconds, it's microseconds, which are a million times bigger...

Thank you for pointing that out! You are correct, μs stands for microseconds, not picoseconds. I've corrected the mistake, and the update should be visible as soon as the CDN cache invalidates.

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#20

Post by harisamin »

tnolet wrote: post #15
harisamin wrote: post #12

On the noisy NodeJS auto-instrumentation, it is indeed very noisy out of the box. Myself along with a bunch of other ppl finally got the project to allow you to select the instrumentations via configuration. Saves having to create your own tracer.ts/js file. Here's the PR that got merged earlier in the year: https://github.com/open-telemetry/opentelemetry-js-contrib/p... The env var config is `OTEL_NODE_ENABLED_INSTR…

That is awesome. Had no idea this was available as an env var. After diving into OTel for our backend, we also found some of this stuff is just too noisy. We switch it of using this code snippet, for anyone bumping into this thread: instrumentations: [getNodeAutoInstrumentations({ '@opentelemetry/instrumentation-fs': { enabled: false, }, '@opentelemetry/instrumentation-net': { enabled: false, }, '@opentelemetry/instr…

yeah I totally turned all of those off...way too noisy :)
Post reply on HN