Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Found 7 TiB of Memory Just Sitting Around

render.com

Post reply on HN
Print view View original

41–50 of 76 posts

Re: How We Found 7 TiB of Memory Just Sitting Around

#41

Post by azornathogron »

Negitivefrags wrote: post #14
azornathogron wrote: post #10

Earlier quoted context omitted.

Pedantic nit: At 60 fps the per frame time is 16.66... ms, not 30 ms. Having said that a lot of games run at 30 fps, or run different parts of their logic at different frequencies, or do other tricks that mean there isn't exactly one FPS rate that the thing is running at.

The CPU part happens on one frame, the GPU part happens on the next frame. If you want to talk about the total time for a game to render a frame, it needs to count two frames.

If latency of input->visible effect is what you're talking about, then yes, that's a great point!

Re: How We Found 7 TiB of Memory Just Sitting Around

#42

Post by ffsm8 »

devjab wrote: post #32
Aeolun wrote: post #24

I read this and I have to wonder, did anyone ever think it was reasonable that a cluster that apparently needed only 120gb of memory was consuming 1.2TB just for logging (or whatever vector does)

We're a much smaller scale company and the cost we lose on these things is insignificant compared to what's in this story. Yesterday I was improving the process for creating databases in our azure and I stumbled upon a subscription which was running 7 mssql servers for 12 databases. These weren't elastic and they were each paying a license that we don't have to pay because we qualify for the base cost through our con…

Isn't that mostly down to the fact the vast majority of devs explicitly don't want to do anything wrt Ops?

DevOps has - ever since it's originally well meaning inception (by Netflix iirc?) - been implemented across our industry as an effective cost cutting measure, forcing devs that didn't see it as their job to also handle it.

Which consequently means they're not interfacing with it whatsoever. They do as little as they can get away with, which inevitably means things are being done with borderline malicious compliance... Or just complete incompetence.

I'm not even sure I'd blame these devs in particular. The devs just saw it as a quick bonus generator for the MBA in charge of this rebranding while offloading more responsibilities in their shoulders.

DevOps made total sense in the work culture where this concept was conceived - Netflix was well known at that point to only ever employ senior Devs. However, in the context of the average 9-5 dev, which often knows a lot less then even some enthusiastic Jrs... Let's just say that it's incredibly dicey wherever it's successful in practice.

Re: How We Found 7 TiB of Memory Just Sitting Around

#43

Post by formerly_proven »

Aeolun wrote: post #24

I read this and I have to wonder, did anyone ever think it was reasonable that a cluster that apparently needed only 120gb of memory was consuming 1.2TB just for logging (or whatever vector does)

It probably doesn't help that the first line of treatment for any error is to blindly increase memory request/limit and claim it's fixed (preferably without looking at the logs once).

Re: How We Found 7 TiB of Memory Just Sitting Around

#44

Post by embedding-shape »

Aeolun wrote: post #37
bstack wrote: post #30

Earlier quoted context omitted.

Author here: You’d be surprised what you don’t notice given enough nodes and slow enough resource growth over time! Out of the total resource usage in these clusters even at the high water mark for this daemonset it was still a small overall portion of the total.

I’m not sure if that makes it better or worse.

I didn't know what Render was when I skimmed the article at first, but after reading these comments, I had to check out what they do.

And they're a "Cloud Application Platform" meaning they manage deploys and infrastructure for other people. Their website says "Click, click, done." which is cool and quick and all, but to me it's kind of crazy an organization that should be really engineering focused and mature, doesn't immediately notice 1.2TB being used and tries to figure out why, when 120GB ended up being sufficient.

It gives much more of a "We're a startup, we're learning as we're running" vibe which again, cool and all, but hardly what people should use for hosting their own stuff on.

Re: How We Found 7 TiB of Memory Just Sitting Around

#45

Post by mustyoshi »

ffsm8 wrote: post #42
devjab wrote: post #32

Earlier quoted context omitted.

We're a much smaller scale company and the cost we lose on these things is insignificant compared to what's in this story. Yesterday I was improving the process for creating databases in our azure and I stumbled upon a subscription which was running 7 mssql servers for 12 databases. These weren't elastic and they were each paying a license that we don't have to pay because we qualify for the base cost through our con…

Isn't that mostly down to the fact the vast majority of devs explicitly don't want to do anything wrt Ops? DevOps has - ever since it's originally well meaning inception (by Netflix iirc?) - been implemented across our industry as an effective cost cutting measure, forcing devs that didn't see it as their job to also handle it. Which consequently means they're not interfacing with it whatsoever. They do as little as…

I politely disagree. I spent maybe 8 hours over a week rightsizing a handful of heavy deployments from a previous team and reduced their peak resource usage by implementing better scaling policies. Before the new scaling policy the service would scale out and new pods would remain idle and ultimately get terminated without ever responding to a request quite frequently.

The service dashboards already existed, all I had to do was a bit of load testing and read the graphs.

It's not too much extra work to make sure you're scaling efficiently.

Re: How We Found 7 TiB of Memory Just Sitting Around

#46

Post by liampulles »

I'm a little surprised that it got to the point where pods which should consume a couple MB of RAM were consuming 4GB before action was taken. But I can also kind of understand it, because the way k8s operators (apps running in k8s that manipulate k8s resource) are meant to run is essentially a loop of listing resources, comparing to spec, and making moves to try and bring the state of the cluster closer to spec. This reconciliation loop is simple to understand (and I think this benefit has led to the creation of a wide array of excellent open source and proprietary operators that can be added to clusters). But its also a recipe for cascading explosions in resource usage.

These kind of resource explosions are something I see all the time in k8s clusters. The general advice is to always try and keep pressure off the k8s API, and the consequence is that one must be very minimal and tactical with the operators one installs, and then engage in many hours of work trying to fine tune each operator to run efficiently (e.g. Grafana, whose default helm settings do not use the recommended log indexing algorithm, and which needs to be tweaked to get an appropriate set of read vs. write pods for your situation).

Again, I recognize there is a tradeoff here - the simplicity and openness of the k8s API is what has led to a flourish of new operators, which really has allowed one to run "their own cloud". But there is definitely a cost. I don't know what the solution is, and I'm curious to hear from people who have other views of it, or use other solutions to k8s which offer a different set of tradeoffs.

Re: How We Found 7 TiB of Memory Just Sitting Around

#47

Post by ffsm8 »

mustyoshi wrote: post #45
ffsm8 wrote: post #42

Earlier quoted context omitted.

Isn't that mostly down to the fact the vast majority of devs explicitly don't want to do anything wrt Ops? DevOps has - ever since it's originally well meaning inception (by Netflix iirc?) - been implemented across our industry as an effective cost cutting measure, forcing devs that didn't see it as their job to also handle it. Which consequently means they're not interfacing with it whatsoever. They do as little as…

I politely disagree. I spent maybe 8 hours over a week rightsizing a handful of heavy deployments from a previous team and reduced their peak resource usage by implementing better scaling policies. Before the new scaling policy the service would scale out and new pods would remain idle and ultimately get terminated without ever responding to a request quite frequently. The service dashboards already existed, all I ha…

You disagree but then cite another example of low hanging fruits that nobody took action on until you came along?

Did you accidentally respond to the wrong comment? Because if anything you're giving another example of "most devs not wanting to interface with ops, hence letting it slide until someone bothers to pick up their slack"...

Re: How We Found 7 TiB of Memory Just Sitting Around

#48

Post by stackskipton »

fells wrote: post #27
stackskipton wrote: post #26

Earlier quoted context omitted.

You mentioned in the blog article that it's doing listwatch. List Watch registers with Kubernetes API that get a list of all objects AND get a notification when anything in object you have registered with changes. A bunch of Vector Pods saying "Hey, send me a notification when anything with namespaces changes" and poof goes your Memory keeping track of who needs to know what. At this point, I wonder if instead of rel…

>you just gave every namespace a vector instance that was responsible for that namespace and pods within. Vector is a daemonset, because it needs to tail the log files on each node. A single vector per namespace might not reside on the nodes that each pod is on.

I think DaemonSet is to reduce network load so Vector is not pulling logs files over the network.

We run Vector as Daemonset as well but we don't have a ton of namespaces. Render sounds like they have a ton of namespaces running maybe one or two pods since their customers are much smaller. This is probably much more niche setup then many users of Kubernetes.

Re: How We Found 7 TiB of Memory Just Sitting Around

#49

Post by zahlman »

hinkley wrote: post #16
zahlman wrote: post #7

Earlier quoted context omitted.

> The biggest tool in the performance toolbox is stubbornness. Without it all the mechanical sympathy in the world will go unexploited. The sympathy is also needed. Problems aren't found when people don't care, or consider the current performance acceptable. > There’s about a factor of 3 improvement that can be made to most code after the profiler has given up. That probably means there are better profilers than coul…

Architecture, cache eviction, memory bandwidth, thermal throttling. All of which have gotten perhaps an order of magnitude worse in the time since I started on this theory.

I meant architecture of the codebase, to be clear. (I'm sure that the increasing complexity of hardware architecture makes it harder to figure out how to write optimal code, but it isn't really degrading the performance of naive attempts, is it?)

Re: How We Found 7 TiB of Memory Just Sitting Around

#50

Post by ahoka »

fells wrote: post #27
stackskipton wrote: post #26

Earlier quoted context omitted.

You mentioned in the blog article that it's doing listwatch. List Watch registers with Kubernetes API that get a list of all objects AND get a notification when anything in object you have registered with changes. A bunch of Vector Pods saying "Hey, send me a notification when anything with namespaces changes" and poof goes your Memory keeping track of who needs to know what. At this point, I wonder if instead of rel…

>you just gave every namespace a vector instance that was responsible for that namespace and pods within. Vector is a daemonset, because it needs to tail the log files on each node. A single vector per namespace might not reside on the nodes that each pod is on.

That's where the design is wrong.
Post reply on HN