Instrumentation: The First Four Things You Measure
1–10 of 38 posts
Re: Instrumentation: The First Four Things You Measure
#2Off Topic: Does anyone know what tools the author uses to make the diagrams?
Re: Instrumentation: The First Four Things You Measure
#3Author appears to use "downstream" and "upstream" to refer to "further down the stack" and "further up the stack".
Is this normal usage? Seems reversed to me.
Re: Instrumentation: The First Four Things You Measure
#4Off Topic: Does anyone know what tools the author uses to make the diagrams?
Paper by 53 or Sketches or Sketchbook or any of the several sketching apps available for the IPad would be my guess.
Re: Instrumentation: The First Four Things You Measure
#5Is the last paragraph a joke? If so, could someone explain it?
Re: Instrumentation: The First Four Things You Measure
#6Author appears to use "downstream" and "upstream" to refer to "further down the stack" and "further up the stack". Is this normal usage? Seems reversed to me.
It seems backwards to me too. I usually see these terms when referencing dependencies in software projects, ie: Ubuntu is downstream from the Linux kernel. I would think that you would see the same thing with services.
Re: Instrumentation: The First Four Things You Measure
#7> A histogram of the duration it took to serve a response to a request, also labelled by successes or errors.
I recommend against this, rather have one overall duration metric and another metric tracking a count of failures.
The reason for this is that very often just the success latency will end up being graphed, and high overall latency due to timing-out failed requests will be missed.
The more information you put on a dashboard, the more chance someone will miss a subtlety like this in the interpretation. Particularly if debugging distributed systems isn't their forte, or they've been woken up in the middle of the night by a page.
This guide only covers what I'd consider online serving systems, I'd suggest a look at the Prometheus instrumentation guidelines on what sort of things to monitor for other types of systems: https://prometheus.io/docs/practices/instrumentation/
Re: Instrumentation: The First Four Things You Measure
#8> A histogram of the duration it took to serve a response to a request, also labelled by successes or errors. I recommend against this, rather have one overall duration metric and another metric tracking a count of failures. The reason for this is that very often just the success latency will end up being graphed, and high overall latency due to timing-out failed requests will be missed. The more information you put…
> rather have one overall duration metric
What exactly would that be?
Re: Instrumentation: The First Four Things You Measure
#9Author appears to use "downstream" and "upstream" to refer to "further down the stack" and "further up the stack". Is this normal usage? Seems reversed to me.
Feels right to me, though I'm no authority figure on this matter. The visual I get is that requests flow from the end-user your service is supporting down to things that indirectly support your user.
Re: Instrumentation: The First Four Things You Measure
#10> A histogram of the duration it took to serve a response to a request, also labelled by successes or errors. I recommend against this, rather have one overall duration metric and another metric tracking a count of failures. The reason for this is that very often just the success latency will end up being graphed, and high overall latency due to timing-out failed requests will be missed. The more information you put…
> rather have one overall duration metric What exactly would that be?
yea that's not good advice, but "don't rely on overstuffed dashboards" is good advice.
you should only have simple dashboards (and alerting) for KPIs and end to end checks. Everything else should be instrumented and debugged using a real time sorting and slicing tool, esp if you have a complex system (microservices, distributed system, polygot persistence).