Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Instrumentation: The First Four Things You Measure

honeycomb.io

Post reply on HN
Print view View original

31–38 of 38 posts

Re: Instrumentation: The First Four Things You Measure

#31

Post by saravana87 »

bbrazil wrote: post #26
saravana87 wrote: post #24

Earlier quoted context omitted.

TCP retransmission rates looks like a useful metric which can help in monitoring the health of a service. One way to obtain that is by analyzing service interactions as mentioned in the blog. Tracing could be another way through which we can find that info. I am curious as to how code instrumented monitoring solutions get that information. (PS: I work for Netsil)

By default you can only get that per-kernel from /proc/net/netsnmp. BPF may allow something more granular. The other way of approaching it is to look for the additional latency it causes, which you can spot on a per-service basis.

Additional latency could be an indicator, but there's no guarantee that it is because of retransmissions ?

Re: Instrumentation: The First Four Things You Measure

#32

Post by jholman »

dmichulke wrote: post #22
techbio wrote: post #3

Author appears to use "downstream" and "upstream" to refer to "further down the stack" and "further up the stack". Is this normal usage? Seems reversed to me.

It depends on whether you look at control flow (who calls whom) or data flow.

This is the issue, yeah.

To rephrase that, "upstream" means "where events come from".

Re: Instrumentation: The First Four Things You Measure

#33

Post by bbrazil »

saravana87 wrote: post #31
bbrazil wrote: post #26

Earlier quoted context omitted.

By default you can only get that per-kernel from /proc/net/netsnmp. BPF may allow something more granular. The other way of approaching it is to look for the additional latency it causes, which you can spot on a per-service basis.

Additional latency could be an indicator, but there's no guarantee that it is because of retransmissions ?

If you look at your latency histogram and are seeing a bump at around 200ms above normal (which was the default minimum wait time a few years back anyway), it's probably retransmits.

Re: Instrumentation: The First Four Things You Measure

#34

Post by kasey_junk »

> A histogram of the duration it took to serve a response to a request, also labelled by successes or errors.

This is so much easier said than done. Most time series db that people use to instrument things quite simply cannot handle histogram data correctly. They make incorrect assumptions about the way roll-ups can happen or they require you to be specific about resolution requirements before you can know them well.

Then histogram data tends to be very expensive to query so it bogs down preventing you from making the kinds of queries that are really valuable for diagnosing performance regressions.

Finally, the visualization systems for histograms are really difficult because you need a third dimension to see them over time. Heat maps accomplish this but are hard to read at times and most dashboard systems don't have great visualization options for "show this time period next to this time period" which is an incredibly common requirement when comparing latency histograms.

Re: Instrumentation: The First Four Things You Measure

#35

Post by tyingq »

anw wrote: post #13

While this is good advice, I feel it is a bit too over-simplified. Counting incoming and outgoing requests misses a lot of potential data points when determining "is this my fault?" I work mainly in system integrations. If I check for the ratio for input:output, then I may miss that some service providers return a 200 with a body of " Error ". A better message is to make sure your systems are knowledgeable in how dat…

I agree that this is over-simplified. It also skips over the mess you can get when a downstream dependency is having issues.

If the "things calling you" can't be effectively throttled, you often run into issues like, for example, hitting the limit on number of open sockets, file descriptors, receive queue, threads etc.

So, just saying "the downstream service is at fault" isn't really correct. Your service may also not be acting correctly in that situation. Those issues can also affect your logging and metrics.

It's not a trivial exercise to architect your service such that it always does the right thing (throttling input vs retries vs fail fast vs priority queues vs load balancing to multiple instances of a downstream service, exponential backoff, etc) when a downstream dependency is slow and/or down.

Edit: Similar to your observation about structured errors, connection pooling is probably also worth talking about in this situation. Which would change the stats you want...once # of connections made isn't the same thing as # of transactions, you would want to know both.

Re: Instrumentation: The First Four Things You Measure

#37

Post by saravana87 »

bbrazil wrote: post #33
saravana87 wrote: post #31

Earlier quoted context omitted.

Additional latency could be an indicator, but there's no guarantee that it is because of retransmissions ?

If you look at your latency histogram and are seeing a bump at around 200ms above normal (which was the default minimum wait time a few years back anyway), it's probably retransmits.

Got it.

Re: Instrumentation: The First Four Things You Measure

#38

Post by maplebed »

kasey_junk wrote: post #34

> A histogram of the duration it took to serve a response to a request, also labelled by successes or errors. This is so much easier said than done. Most time series db that people use to instrument things quite simply cannot handle histogram data correctly. They make incorrect assumptions about the way roll-ups can happen or they require you to be specific about resolution requirements before you can know them well.…

Yup! It's hard! All the things you point out are right on.

We don't have the visualizations for histograms yet (though you can chart specific percentiles), but for the reasons you mention, Honeycomb is perfectly suited to give you that kind of data. I can't say we'll get that out the door soon, but it's one of my pet most wanted features so as soon as I can convince myself it's actually more important than all the other mountain of things that need to get done, you'll get your histograms and your time over time comparisons.

I've been advocating for a heat map style presentation of histograms for a long time, but I hadn't considered the difficulty that creates when trying to show time over time. That's an interesting one to noodle on.

Thanks for articulating well the value and reasons for difficulty in implementing histograms!

(bias alert - I work on Honeycomb)

Post reply on HN