Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Finding zombies in our systems: A real-world story of CPU bottlenecks

medium.com

Post reply on HN
Print view View original

1–10 of 12 posts

Re: Finding zombies in our systems: A real-world story of CPU bottlenecks

#2

Post by alain94040 »

Interesting story.

I don't know anything about the Linux scheduler, but if one core out of 96 is busy chasing after those ghost allocations, why would a network thread not use one of the other 95 cores, and instead be stuck for seconds? Unless that one core is blocking the entire kernel, but then that would be a lot more visible instantly.

Re: Finding zombies in our systems: A real-world story of CPU bottlenecks

#5

Post by troelsSteegin »

This was a nostalgic pleasure to read in contrast to all the AI implications and tools posts. That said about AI, this as a case study would make an interesting benchmark.

Especially noted that the root cause here was pegged because somebody else had blogged about it. Our "digital commons" should be valued as other than an externality. Edit: by that I think that "the internet should be free" but that commercialization through derivative work should compensate authors. That's off-topic, so, nevermind.

Re: Finding zombies in our systems: A real-world story of CPU bottlenecks

#6

Post by Aurornis »

4fterd4rk wrote: post #4

I know this is a hopelessly naive question but what is Pinterest even doing with all this AI training?

Recommendations and similarity search are core to a site like that. Those were classic ML topics before AI become synonymous with LLM in common news

Re: Finding zombies in our systems: A real-world story of CPU bottlenecks

#7

Post by neerajsi »

alain94040 wrote: post #2

Interesting story. I don't know anything about the Linux scheduler, but if one core out of 96 is busy chasing after those ghost allocations, why would a network thread not use one of the other 95 cores, and instead be stuck for seconds? Unless that one core is blocking the entire kernel, but then that would be a lot more visible instantly.

It's probably a "bottom half", which is tightly bound to the CPU where the interrupt landed. And probably the memcg iteration uses a spin lock, which disables bottom half execution.

On NT, this style of problem would be slightly less likely. The job object, which is NT's memcg equivalent, is a handle based object that naturally goes away when the owning process dies.

I also bet systemd would have native mechanisms to clean this memcg state up if it was used for this purpose. Seems like a bug in the AWS ecs code.

Re: Finding zombies in our systems: A real-world story of CPU bottlenecks

#9

Post by qingcharles »

4fterd4rk wrote: post #4

I know this is a hopelessly naive question but what is Pinterest even doing with all this AI training?

I feel like they can't have any devs left at this point, the quality of the site code has gone down significantly in the last couple of years. Real shame, it used to be an awesome utility for bookmarking ideas.
Post reply on HN