How We Found 7 TiB of Memory Just Sitting Around
1–10 of 76 posts
Re: How We Found 7 TiB of Memory Just Sitting Around
#2The unreasonable effectiveness of profiling and digging deep strikes again.
Re: How We Found 7 TiB of Memory Just Sitting Around
#3The other way to look is why adding NS label is causing so much memory footprint in Kubernetes. Shouldn't be fixing that (could be much bigger design change), will benefit whole Kube community?
Re: How We Found 7 TiB of Memory Just Sitting Around
#4The other way to look is why adding NS label is causing so much memory footprint in Kubernetes. Shouldn't be fixing that (could be much bigger design change), will benefit whole Kube community?
Author here: yeah that's a good point. tbh I was mostly unfamiliar with Vector so I took the shortest path to the goal but that could be interesting followup. It does seem like there's a lot of bytes per namespace!
Re: How We Found 7 TiB of Memory Just Sitting Around
#5The unreasonable effectiveness of profiling and digging deep strikes again.
The biggest tool in the performance toolbox is stubbornness. Without it all the mechanical sympathy in the world will go unexploited.
There’s about a factor of 3 improvement that can be made to most code after the profiler has given up. That probably means there are better profilers than could be written, but in 20 years of having them I’ve only seen 2 that tried. Sadly I think flame graphs made profiling more accessible to the unmotivated but didn’t actually improve overall results.
Re: How We Found 7 TiB of Memory Just Sitting Around
#6Keys require O(logn) space per key or nlogn for the entire data set, simply to avoid key collisions. But human friendly key spaces grow much, much faster and I don’t think many people have looked too hard at that.
There were recent changes to the NodeJS Prometheus client that eliminates tag names from the keys used for storing the tag cardinality for metrics. The memory savings wasn’t reported but the cpu savings for recording data points was over 1/3. And about twice that when applied to the aggregation logic.
Lookups are rarely O(1), even in hash tables.
I wonder if there’s a general solution for keeping names concise without triggering transposition or reading comprehension errors. And what the space complexity is of such an algorithm.
Re: How We Found 7 TiB of Memory Just Sitting Around
#7The unreasonable effectiveness of profiling and digging deep strikes again.
The biggest tool in the performance toolbox is stubbornness. Without it all the mechanical sympathy in the world will go unexploited. There’s about a factor of 3 improvement that can be made to most code after the profiler has given up. That probably means there are better profilers than could be written, but in 20 years of having them I’ve only seen 2 that tried. Sadly I think flame graphs made profiling more access…
> The biggest tool in the performance toolbox is stubbornness. Without it all the mechanical sympathy in the world will go unexploited.
The sympathy is also needed. Problems aren't found when people don't care, or consider the current performance acceptable.
> There’s about a factor of 3 improvement that can be made to most code after the profiler has given up. That probably means there are better profilers than could be written, but in 20 years of having them I’ve only seen 2 that tried.
It's hard for profilers to identify slowdowns that are due to the architecture. Making the function do less work to get its result feels different from determining that the function's result is unnecessary.
Re: How We Found 7 TiB of Memory Just Sitting Around
#8The unreasonable effectiveness of profiling and digging deep strikes again.
The biggest tool in the performance toolbox is stubbornness. Without it all the mechanical sympathy in the world will go unexploited. There’s about a factor of 3 improvement that can be made to most code after the profiler has given up. That probably means there are better profilers than could be written, but in 20 years of having them I’ve only seen 2 that tried. Sadly I think flame graphs made profiling more access…
I think the biggest tool is higher expectations. Most programmers really haven't come to grips with the idea that computers are fast.
If you see a database query that takes 1 hour to run, and only touches a few gb of data, you should be thinking "Well nvme bandwidth is multiple gigabytes per second, why can't it run in 1 second or less?"
The idea that anyone would accept a request to a website taking longer than 30ms, (the time it takes for a game to render it's entire world including both the CPU and GPU parts at 60fps) is insane, and nobody should really accept it, but we commonly do.
Re: How We Found 7 TiB of Memory Just Sitting Around
#9Earlier quoted context omitted.
The biggest tool in the performance toolbox is stubbornness. Without it all the mechanical sympathy in the world will go unexploited. There’s about a factor of 3 improvement that can be made to most code after the profiler has given up. That probably means there are better profilers than could be written, but in 20 years of having them I’ve only seen 2 that tried. Sadly I think flame graphs made profiling more access…
I think the biggest tool is higher expectations. Most programmers really haven't come to grips with the idea that computers are fast. If you see a database query that takes 1 hour to run, and only touches a few gb of data, you should be thinking "Well nvme bandwidth is multiple gigabytes per second, why can't it run in 1 second or less?" The idea that anyone would accept a request to a website taking longer than 30ms…
its also about cost. My game computer has 8 cores + 1 expensive gpu + 32GB ram for me alone. We dont have that per customer.
Re: How We Found 7 TiB of Memory Just Sitting Around
#10Earlier quoted context omitted.
The biggest tool in the performance toolbox is stubbornness. Without it all the mechanical sympathy in the world will go unexploited. There’s about a factor of 3 improvement that can be made to most code after the profiler has given up. That probably means there are better profilers than could be written, but in 20 years of having them I’ve only seen 2 that tried. Sadly I think flame graphs made profiling more access…
I think the biggest tool is higher expectations. Most programmers really haven't come to grips with the idea that computers are fast. If you see a database query that takes 1 hour to run, and only touches a few gb of data, you should be thinking "Well nvme bandwidth is multiple gigabytes per second, why can't it run in 1 second or less?" The idea that anyone would accept a request to a website taking longer than 30ms…
Pedantic nit: At 60 fps the per frame time is 16.66... ms, not 30 ms. Having said that a lot of games run at 30 fps, or run different parts of their logic at different frequencies, or do other tricks that mean there isn't exactly one FPS rate that the thing is running at.