Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Saving Three Months of Latency with a Single OpenTelemetry Trace

checklyhq.com

Post reply on HN
Print view View original

41–46 of 46 posts

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#41

Post by tnolet »

tmpz22 wrote: post #40
tnolet wrote: post #35

Earlier quoted context omitted.

This deserves an updoot. We reached a level of scale right now at Checkly that all of these things start adding up. We moved workloads off of S3 to Cloudflare R2 because of this.

> We moved workloads off of S3 to Cloudflare R2 because of this. So you moved from a mature but expensive storage solution to a younger currently subsidized storage solution? What happens when R2 jacks up pricing?

let me nuance that a bit. 99% of our workload is write heavy and is still on S3. We run monitoring checks that snap a screenshot and record a video. We write that to S3. Most folks will never view any of that as most checks pass and these artefacts only become interesting when things fail.

Enter a new product feature we launched (Visual Regression Testing) which requires us to fetch an image from storage on every "run" we do. These could be every 10sec. This is where R2 shines. No egress cost for us. It's been rock solid and saved us about 60x compared to AWS. Still, we run most of our infra on AWS.

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#42

Post by Veserv »

tnolet wrote: post #38
Veserv wrote: post #37

Earlier quoted context omitted.

That is why I mentioned post-filtering the recording as the alternative. Grab the full recording then filter to just the relevant results before inspection. For that matter, why are a few hundred spans a problem? Are the visualizers that poor? I usually use function tracing where hundreds of millions to billions of spans per second are the norm and there is no difficulty managing or understanding those.

In most cases these traces are shipped over the wire to a vendor. Only that will cost $$. Then, not all vendors have tail sampling as a "free" feature. So, in many cases it's better to not record at all.

That sounds positively dystopian. Is it really that hard to dump to private/non-vendor storage for local analysis using your own tools?

I do not do cloud or web development, so this is just totally alien. I generate multi-gigabyte logs with billions of events for just seconds of execution and get to slice them however I want when doing performance analysis. The inability to even process your own logs seems crazy.

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#43

Post by cweld510 »

Veserv wrote: post #42
tnolet wrote: post #38

Earlier quoted context omitted.

In most cases these traces are shipped over the wire to a vendor. Only that will cost $$. Then, not all vendors have tail sampling as a "free" feature. So, in many cases it's better to not record at all.

That sounds positively dystopian. Is it really that hard to dump to private/non-vendor storage for local analysis using your own tools? I do not do cloud or web development, so this is just totally alien. I generate multi-gigabyte logs with billions of events for just seconds of execution and get to slice them however I want when doing performance analysis. The inability to even process your own logs seems crazy.

You can absolutely dump the traces somewhere and analyze them yourself. The problem is that this falls apart with scale. You are maybe serving thousands of requests per second. Your service has a ton of instances. Capturing all trace data for all requests from all services is just difficult. Where do you store all of it? How do you quickly find what you need? It gets very annoying very fast. When you pay a vendor, you pay them to deal with this.

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#44

Post by 0xFF0123 »

tmpz22 wrote: post #40
tnolet wrote: post #35

Earlier quoted context omitted.

This deserves an updoot. We reached a level of scale right now at Checkly that all of these things start adding up. We moved workloads off of S3 to Cloudflare R2 because of this.

> We moved workloads off of S3 to Cloudflare R2 because of this. So you moved from a mature but expensive storage solution to a younger currently subsidized storage solution? What happens when R2 jacks up pricing?

> currently subsidized storage solution

Interesting, do you have a source on the subsidized nature of R2?

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#45

Post by tmpz22 »

0xFF0123 wrote: post #44
tmpz22 wrote: post #40

Earlier quoted context omitted.

> We moved workloads off of S3 to Cloudflare R2 because of this. So you moved from a mature but expensive storage solution to a younger currently subsidized storage solution? What happens when R2 jacks up pricing?

> currently subsidized storage solution Interesting, do you have a source on the subsidized nature of R2?

I do not and I'm likely misusing the word subsidized.

My concern is that as a newer product (R2 in 2022 [1] compared to S3 2006 [2]) R2 has deliberately priced itself to compete with egress pricing of S3 in order to gain market share and developer mindshare. I am not confident Cloudflare will maintain this competitive pricing indefinitely as I expect it to follow well established industry trends of jacking up prices once a walled garden has been sufficiently establed.

Further its my opinion cloud costs have grown at an absurd level as engineers and executives made poor and frankly lazy technology choices for the last decade.

Ultimately I like cloudflare a lot but I think we need more discipline and lower operational overhead if we want infrastructure development to remain practical to individuals and small businesses versus mega-corps. Cloudflare with its free pricing tiers is often a default choice for these organizational sizes but it should not be viewed as a panacea and carries tradeoffs as with everything in life.

[1] https://www.cloudflare.com/press-releases/2022/cloudflare-ma...

[2] https://hidekazu-konishi.com/entry/aws_history_and_timeline_...

Re: Saving Three Months of Latency with a Single OpenTelemetry Trace

#46

Post by drakir_nosslin »

redman25 wrote: post #27
simonbarker87 wrote: post #5

I wish posts like this would explore the relative savings rather than the absolute. On its own I don’t feel like that saving is really telling me much, taken to the extreme you could just not run the service at all and save all the time - a tongue in cheek example but in context is this saving a big deal or is it just engineering looking for small efficiencies to justify their time?

I often ask myself the same question. We have some user facing queries that slow the frontend down. I’ve fixed some slowness but it’s definitely not a priority. I wonder how much speed improvements correlate with increased revenue by happy customers.

Bit late to the party, but companies report that webpage speed correlates with conversion. See e.g. https://www.cloudflare.com/en-gb/learning/performance/why-si... & https://www.cloudflare.com/en-gb/learning/performance/more/w...

This one is also interesting; written in 2012, it claims that Amazon could lose 1b+ from a 1 sec slowdown: https://www.fastcompany.com/1825005/how-one-second-could-cos.... I imagine people are even less tolerant of slow pages today.

Fixing website performance can be one of the cheapest ways to increase conversion because it's hard to figure out what else moves the needle.

Post reply on HN