Curious Case of a 99.9% Latency Hike
mahdytech.com
Curious Case of a 99.9% Latency Hike
1–10 of 20 posts
Re: Curious Case of a 99.9% Latency Hike
#2Nice, easy read. Light but to the point.
Re: Curious Case of a 99.9% Latency Hike
#3Why did you not see that on metrics / dashboards ... Tracing an app should be last resort, a proper setup would have told that you had memory issues right away.
Re: Curious Case of a 99.9% Latency Hike
#4Nice insight into the tools, but wouldn't a simple memory monitor have shown the issue?
Re: Curious Case of a 99.9% Latency Hike
#5You could add an aliveness probe that makes sure the binary is paged in completely.
Re: Curious Case of a 99.9% Latency Hike
#6Why did you not see that on metrics / dashboards ... Tracing an app should be last resort, a proper setup would have told that you had memory issues right away.
There's that and you could be monitoring swap. Though this reads like the writer might not the potential impact swapping has on performance. They seem to have their own datacenters? Id imagine the servers would have some sort of monitoring?
It would be nice to language and os your talking about.
Also, to the writer, I think you want to turn off swap?
Re: Curious Case of a 99.9% Latency Hike
#7I've watched team after team get surprised that their app gets substantially slower when they use up nearly all of the memory on the servers.
They look at the free memory and think, "wow look at all that free space, I can totally take up half of that no problem".
The moment the collective working set exceeds total memory things go sideways. They learn that the performance of the system tanks when there's not enough space left for OS-level caches and especially for memory mapped files.
Re: Curious Case of a 99.9% Latency Hike
#8Why did you not see that on metrics / dashboards ... Tracing an app should be last resort, a proper setup would have told that you had memory issues right away.
There's that and you could be monitoring swap. Though this reads like the writer might not the potential impact swapping has on performance. They seem to have their own datacenters? Id imagine the servers would have some sort of monitoring? It would be nice to language and os your talking about. Also, to the writer, I think you want to turn off swap?
Better than turning it off, on Linux try to use vm.overcommit_memory
It will give you the option of swap when OOM would otherwise kill process, but you can decide exactly how much, to prevent swapping from hurting performance -- say have a 10% extra.
Assuming you have a significant number of identical servers, turning off swap should be reserved for after you have done your tuning of the daemons and their memory use, and you are confident nothing will run out (still, you should monitor dmesg) to maximize bang for bugs.
If however you need swap a good old trick- don't put the swap on LVM or RAID. Just have separate partition on each physical device. The kernel will always know to use them in parallel, and you get to squeeze out a bit of extra performance
Re: Curious Case of a 99.9% Latency Hike
#9Why did you not see that on metrics / dashboards ... Tracing an app should be last resort, a proper setup would have told that you had memory issues right away.
There's that and you could be monitoring swap. Though this reads like the writer might not the potential impact swapping has on performance. They seem to have their own datacenters? Id imagine the servers would have some sort of monitoring? It would be nice to language and os your talking about. Also, to the writer, I think you want to turn off swap?
> Also, to the writer, I think you want to turn off swap?
Using kernel panics as an alerting mechanism is a brutal solution.
Why not track page faults? Most orgs that I interact with track paging across the whole fleet.
Re: Curious Case of a 99.9% Latency Hike
#10Nice insight into the tools, but wouldn't a simple memory monitor have shown the issue?
That is already tracked, but it is not very meaningful. We already have a lot of page faults (and paged memory), they are just not on the hot path.