Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Found 7 TiB of Memory Just Sitting Around

render.com

Post reply on HN
Print view View original

21–30 of 76 posts

Re: How We Found 7 TiB of Memory Just Sitting Around

#21

Post by wizzwizz4 »

Negitivefrags wrote: post #14
azornathogron wrote: post #10

Earlier quoted context omitted.

Pedantic nit: At 60 fps the per frame time is 16.66... ms, not 30 ms. Having said that a lot of games run at 30 fps, or run different parts of their logic at different frequencies, or do other tricks that mean there isn't exactly one FPS rate that the thing is running at.

The CPU part happens on one frame, the GPU part happens on the next frame. If you want to talk about the total time for a game to render a frame, it needs to count two frames.

Computers are fast. Why do you accept a frame of lag? The average game for a PC from the 1980s ran with less lag than that. Super Mario Bros had less than a frame between controller input and character movement on the screen. (Technically, it could be more than a frame, but only if there were enough objects in play that the processor couldn't handle all the physics updates in time and missed the v-blank interval.)

Re: How We Found 7 TiB of Memory Just Sitting Around

#22

Post by hinkley »

antonymoose wrote: post #17
hinkley wrote: post #15

Earlier quoted context omitted.

Lowered expectations are come in part from people giving up on theirs. Accepting versus pushing back.

I have high hopes and expectations, unfortunately my chain of command does not, and is often an immovable force.

This is a terrible time to tell someone to find a movable object in another part of the org or elsewhere. :/

I always liked Shaw’s “The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.”

Re: How We Found 7 TiB of Memory Just Sitting Around

#23

Post by Negitivefrags »

wizzwizz4 wrote: post #21
Negitivefrags wrote: post #14

Earlier quoted context omitted.

The CPU part happens on one frame, the GPU part happens on the next frame. If you want to talk about the total time for a game to render a frame, it needs to count two frames.

Computers are fast. Why do you accept a frame of lag? The average game for a PC from the 1980s ran with less lag than that. Super Mario Bros had less than a frame between controller input and character movement on the screen. (Technically, it could be more than a frame, but only if there were enough objects in play that the processor couldn't handle all the physics updates in time and missed the v-blank interval.)

If Vsync is on which was my assumption from my previous comment, then if your computer is fast enough, you might be able to run CPU and GPU work entirely in a single frame if you use Reflex to delay when simulation starts to lower latency, but regardless, you still have a total time budget of 1/30th of a second to do all your combined CPU and GPU work to get to 60fps.

Re: How We Found 7 TiB of Memory Just Sitting Around

#25

Post by Aeolun »

javier2 wrote: post #9
Negitivefrags wrote: post #8

Earlier quoted context omitted.

I think the biggest tool is higher expectations. Most programmers really haven't come to grips with the idea that computers are fast. If you see a database query that takes 1 hour to run, and only touches a few gb of data, you should be thinking "Well nvme bandwidth is multiple gigabytes per second, why can't it run in 1 second or less?" The idea that anyone would accept a request to a website taking longer than 30ms…

its also about cost. My game computer has 8 cores + 1 expensive gpu + 32GB ram for me alone. We dont have that per customer.

If your websites take less than 16ms to serve, you can serve 60 customers per second with that. So you sorta do have it per customer?

Re: How We Found 7 TiB of Memory Just Sitting Around

#26

Post by stackskipton »

bstack wrote: post #4
nitinreddy88 wrote: post #3

The other way to look is why adding NS label is causing so much memory footprint in Kubernetes. Shouldn't be fixing that (could be much bigger design change), will benefit whole Kube community?

Author here: yeah that's a good point. tbh I was mostly unfamiliar with Vector so I took the shortest path to the goal but that could be interesting followup. It does seem like there's a lot of bytes per namespace!

You mentioned in the blog article that it's doing listwatch. List Watch registers with Kubernetes API that get a list of all objects AND get a notification when anything in object you have registered with changes. A bunch of Vector Pods saying "Hey, send me a notification when anything with namespaces changes" and poof goes your Memory keeping track of who needs to know what.

At this point, I wonder if instead of relying on daemonsets, you just gave every namespace a vector instance that was responsible for that namespace and pods within. ElasticSearch or whatever you pipe logging data to might not be happy with all those TCP connections.

Just my SRE brain thoughts.

Re: How We Found 7 TiB of Memory Just Sitting Around

#27

Post by fells »

stackskipton wrote: post #26
bstack wrote: post #4

Earlier quoted context omitted.

Author here: yeah that's a good point. tbh I was mostly unfamiliar with Vector so I took the shortest path to the goal but that could be interesting followup. It does seem like there's a lot of bytes per namespace!

You mentioned in the blog article that it's doing listwatch. List Watch registers with Kubernetes API that get a list of all objects AND get a notification when anything in object you have registered with changes. A bunch of Vector Pods saying "Hey, send me a notification when anything with namespaces changes" and poof goes your Memory keeping track of who needs to know what. At this point, I wonder if instead of rel…

>you just gave every namespace a vector instance that was responsible for that namespace and pods within.

Vector is a daemonset, because it needs to tail the log files on each node. A single vector per namespace might not reside on the nodes that each pod is on.

Re: How We Found 7 TiB of Memory Just Sitting Around

#28

Post by vlovich123 »

Aeolun wrote: post #25
javier2 wrote: post #9

Earlier quoted context omitted.

its also about cost. My game computer has 8 cores + 1 expensive gpu + 32GB ram for me alone. We dont have that per customer.

If your websites take less than 16ms to serve, you can serve 60 customers per second with that. So you sorta do have it per customer?

That’s per core assuming the 16ms is CPU bound activity (so 100 cores would serve 100 customers). If it’s I/O you can overlap a lot of customers since a single core could easily keep track of thousands of in flight requests.

Re: How We Found 7 TiB of Memory Just Sitting Around

#29

Post by vlovich123 »

hinkley wrote: post #6

Keys require O(logn) space per key or nlogn for the entire data set, simply to avoid key collisions. But human friendly key spaces grow much, much faster and I don’t think many people have looked too hard at that. There were recent changes to the NodeJS Prometheus client that eliminates tag names from the keys used for storing the tag cardinality for metrics. The memory savings wasn’t reported but the cpu savings for…

Why aren’t let’s just 128bit UUIDs? Those are guaranteed to be globally unique and don’t require so much spacex

Re: How We Found 7 TiB of Memory Just Sitting Around

#30

Post by bstack »

Aeolun wrote: post #24

I read this and I have to wonder, did anyone ever think it was reasonable that a cluster that apparently needed only 120gb of memory was consuming 1.2TB just for logging (or whatever vector does)

Author here: You’d be surprised what you don’t notice given enough nodes and slow enough resource growth over time! Out of the total resource usage in these clusters even at the high water mark for this daemonset it was still a small overall portion of the total.
Post reply on HN