Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Curious Case of a 99.9% Latency Hike

mahdytech.com

Post reply on HN
Print view View original

11–20 of 20 posts

Re: Curious Case of a 99.9% Latency Hike

#11

Post by aybassiouny »

tayo42 wrote: post #6
Thaxll wrote: post #3

Why did you not see that on metrics / dashboards ... Tracing an app should be last resort, a proper setup would have told that you had memory issues right away.

There's that and you could be monitoring swap. Though this reads like the writer might not the potential impact swapping has on performance. They seem to have their own datacenters? Id imagine the servers would have some sort of monitoring? It would be nice to language and os your talking about. Also, to the writer, I think you want to turn off swap?

Lucy, even in a healthy state, already have many page faults. They are just not on the hot path, e.g. logging. They don't affect latency much and nobody cares for them.

Turning off swapping has been a heated discussion, and I am on the negative side of it. Swapping is good; not all memory is important and some of it should not occupy precious physical space. Not all processes are important though. What I would rather have is a per-process swapping flag, allowing me to disable swapping for one or two essential processes, but swap the rest.

Re: Curious Case of a 99.9% Latency Hike

#12

Post by aybassiouny »

hinkley wrote: post #7

I've watched team after team get surprised that their app gets substantially slower when they use up nearly all of the memory on the servers. They look at the free memory and think, "wow look at all that free space, I can totally take up half of that no problem". The moment the collective working set exceeds total memory things go sideways. They learn that the performance of the system tanks when there's not enough s…

The surprising part for me is, OS does not recover. As if once it tanks, it becomes more cautious about which memory is paged and which is not, even if there's plenty of free physical memory later.

Re: Curious Case of a 99.9% Latency Hike

#13

Post by haneefmubarak »

aybassiouny wrote: post #12
hinkley wrote: post #7

I've watched team after team get surprised that their app gets substantially slower when they use up nearly all of the memory on the servers. They look at the free memory and think, "wow look at all that free space, I can totally take up half of that no problem". The moment the collective working set exceeds total memory things go sideways. They learn that the performance of the system tanks when there's not enough s…

The surprising part for me is, OS does not recover. As if once it tanks, it becomes more cautious about which memory is paged and which is not, even if there's plenty of free physical memory later.

I think that this is something that I've noticed MacOS has generally handled well (at least since Lion, which is when I started noticing), among other memory subsystem things (being a little smarter about what to swap out, etc).

However, the MacOS kernel seems to require a much higher amount of wired memory than the Linux kernel for similar workloads, so...

I don't know if those are related, but it'd be interesting to figure out what MacOS does differently (their kernel is open sourced) and to see if that behavior could be adapted into the Linux kernel.

Re: Curious Case of a 99.9% Latency Hike

#14

Post by existencebox »

aybassiouny wrote: post #12
hinkley wrote: post #7

I've watched team after team get surprised that their app gets substantially slower when they use up nearly all of the memory on the servers. They look at the free memory and think, "wow look at all that free space, I can totally take up half of that no problem". The moment the collective working set exceeds total memory things go sideways. They learn that the performance of the system tanks when there's not enough s…

The surprising part for me is, OS does not recover. As if once it tanks, it becomes more cautious about which memory is paged and which is not, even if there's plenty of free physical memory later.

Preemptive apologies for this glorified "it's nice to hear it's not just me" of a post.

To let you skip the anecdote if you don't care, I'd be very curious if someone can answer _why_ this happens.

Earlier this week I was running into Very Strange errors coming out of some deep libraries in one of my hobby services. Nothing was specifically failing but things were going very slow and vomiting a bit. I observed that I had over 25g of active Ram (Typically idle at ~8g) and some absurd amount of paged data too. Even after killing almost all non-system running processes I was sitting at ~20g total, including a 10g nonpaged pool and 6g process private. On reboot, and for the week since? 8g. (On Win10)

Re: Curious Case of a 99.9% Latency Hike

#15

Post by pstrateman »

aybassiouny wrote: post #12
hinkley wrote: post #7

I've watched team after team get surprised that their app gets substantially slower when they use up nearly all of the memory on the servers. They look at the free memory and think, "wow look at all that free space, I can totally take up half of that no problem". The moment the collective working set exceeds total memory things go sideways. They learn that the performance of the system tanks when there's not enough s…

The surprising part for me is, OS does not recover. As if once it tanks, it becomes more cautious about which memory is paged and which is not, even if there's plenty of free physical memory later.

You need to push things out of swap back into memory.

I usually just swapoff and swapon, but I'm sure there's a better way.

Re: Curious Case of a 99.9% Latency Hike

#16

Post by aybassiouny »

existencebox wrote: post #14
aybassiouny wrote: post #12

Earlier quoted context omitted.

The surprising part for me is, OS does not recover. As if once it tanks, it becomes more cautious about which memory is paged and which is not, even if there's plenty of free physical memory later.

Preemptive apologies for this glorified "it's nice to hear it's not just me" of a post. To let you skip the anecdote if you don't care, I'd be very curious if someone can answer _why_ this happens. Earlier this week I was running into Very Strange errors coming out of some deep libraries in one of my hobby services. Nothing was specifically failing but things were going very slow and vomiting a bit. I observed that I…

I recommend to do more debugging next time and see which process exactly is costing you all this. It sounds like a classic memory leak - those can be triggered nondeterministically. Maybe try some of the tools suggested here https://mahdytech.com/2019/01/05/task-manager-memory-info/

Re: Curious Case of a 99.9% Latency Hike

#17

Post by existencebox »

aybassiouny wrote: post #16
existencebox wrote: post #14

Earlier quoted context omitted.

Preemptive apologies for this glorified "it's nice to hear it's not just me" of a post. To let you skip the anecdote if you don't care, I'd be very curious if someone can answer _why_ this happens. Earlier this week I was running into Very Strange errors coming out of some deep libraries in one of my hobby services. Nothing was specifically failing but things were going very slow and vomiting a bit. I observed that I…

I recommend to do more debugging next time and see which process exactly is costing you all this. It sounds like a classic memory leak - those can be triggered nondeterministically. Maybe try some of the tools suggested here https://mahdytech.com/2019/01/05/task-manager-memory-info/

That's the fascinating part; I followed _that specific_ top google result plus a few others, dug around in RamMap, but couldn't find anything to finger any particular process as owning all the consumed Ram. Admittedly a bit out of my league in terms of system level debugging here, but all of the typical culprits said "everything looks normal except you actually have like no ram free," and given that it was impeding my primary work machine + that I had no track record of these symptoms occurring before, I eventually resigned myself to a reboot.

Re: Curious Case of a 99.9% Latency Hike

#18

Post by jsjohnst »

haneefmubarak wrote: post #13
aybassiouny wrote: post #12

Earlier quoted context omitted.

The surprising part for me is, OS does not recover. As if once it tanks, it becomes more cautious about which memory is paged and which is not, even if there's plenty of free physical memory later.

I think that this is something that I've noticed MacOS has generally handled well (at least since Lion, which is when I started noticing), among other memory subsystem things (being a little smarter about what to swap out, etc). However, the MacOS kernel seems to require a much higher amount of wired memory than the Linux kernel for similar workloads, so... I don't know if those are related, but it'd be interesting t…

> this is something that I've noticed MacOS has generally handled well

Really? I’ve had the opposite experience. MacOS running out of memory and paging heavily almost always necessitates a reboot to get performance fully restored, even if you kill the memory hog (if you aren’t willing to deal with a sluggish machine for a while that is).

Re: Curious Case of a 99.9% Latency Hike

#19

Post by gnulinux »

pstrateman wrote: post #15
aybassiouny wrote: post #12

Earlier quoted context omitted.

The surprising part for me is, OS does not recover. As if once it tanks, it becomes more cautious about which memory is paged and which is not, even if there's plenty of free physical memory later.

You need to push things out of swap back into memory. I usually just swapoff and swapon, but I'm sure there's a better way.

The problem is not just swap. OS caches A LOT of things in memory, and when it cannot do that, it simply deletes the cache. At that point it's not possible to build the cache again, unless it specifically performs unnecessary operations to build an inaccurate cache. Same goes for mmapped files. This is not an easy problem. Once you lose your cache, there is no way to get it back. At least if swap is enabled, it can temporarily store its cache. A different but similar problem is "thrashing" where vmem is overloaded so everything page faults. Simply doing swapoff swapon won't recover from this.

The real recovery from this is cleaning virtual mem, and let your program run for a while so that some cache is collected. This may take seconds to hours, depending on your application and vmem size.

Re: Curious Case of a 99.9% Latency Hike

#20

Post by ovi256 »

Thaxll wrote: post #3

Why did you not see that on metrics / dashboards ... Tracing an app should be last resort, a proper setup would have told that you had memory issues right away.

Obviously there are different levels of monitoring maturity, just like with all aspects of engineering. The brave poster solved the issue, so he leveled up by his own.
Post reply on HN