11–20 of 38 posts
Is the last paragraph a joke? If so, could someone explain it?
Counting incoming and outgoing requests misses a lot of potential data points when determining "is this my fault?"
I work mainly in system integrations. If I check for the ratio for input:output, then I may miss that some service providers return a 200 with a body of "Error".
A better message is to make sure your systems are knowledgeable in how data is received from downstream submissions, and to have a universal way of translating that feedback to a format your own service understands.
HTTP codes are (pretty much) universal. But let's say you forgot to inlcude a header or forgot to base64 encode login details or simply are using a wrong value for an API key. If your system knows that "this XML element means Y for provider X, and means Z in our own system", then you can better gauge issues as they come up, instead of waiting for customers to complain. This is also where tools like Splunk are handy, so you can be alerted to these kinds of errors as they come up.
> A histogram of the duration it took to serve a response to a request, also labelled by successes or errors. I recommend against this, rather have one overall duration metric and another metric tracking a count of failures. The reason for this is that very often just the success latency will end up being graphed, and high overall latency due to timing-out failed requests will be missed. The more information you put…
> rather have one overall duration metric What exactly would that be?
> A histogram of the duration it took to serve a response to a request, also labelled by successes or errors. I recommend against this, rather have one overall duration metric and another metric tracking a count of failures. The reason for this is that very often just the success latency will end up being graphed, and high overall latency due to timing-out failed requests will be missed. The more information you put…
The originally quoted advice, to show "the duration it took to serve a response to a request, also labelled by successes or errors" remains good advice, so long as the visualization of that data makes clear the separation.
I absolutely agree that careful consideration is required when choosing what to put on dashboards to avoid confusion. That seems to be a separate issue.
(bias alert - I work on Honeycomb, and care deeply about collecting data in a way that lets you pull away the irrelevant data to illuminate the real problems.)
One thing of note in the graph is the tracking of response size. This would be very useful for 200 responses with "Error" in the text. Because then the response size would drop drastically below a normal successful response payload size.
In addition to Latency, Error Rates, Throughput and Saturation , folks like Brendan Gregg @ Netflix have recommended tracking capacity.
Came across this which gives good insight into the 4 golden signals for a top-level health tracking: https://blog.netsil.com/the-4-golden-signals-of-api-health-a... One thing of note in the graph is the tracking of response size. This would be very useful for 200 responses with "Error" in the text. Because then the response size would drop drastically below a normal successful response payload size. In addition to La…
(bias alert - I work on Honeycomb)
Author appears to use "downstream" and "upstream" to refer to "further down the stack" and "further up the stack". Is this normal usage? Seems reversed to me.
If the river is data, then stuff you depend on is upstream from you, and things that depend on you are downstream.