Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Found 7 TiB of Memory Just Sitting Around

render.com

Post reply on HN
Print view View original

31–40 of 76 posts

Re: How We Found 7 TiB of Memory Just Sitting Around

#31

Post by mjevans »

Negitivefrags wrote: post #8
hinkley wrote: post #5

Earlier quoted context omitted.

The biggest tool in the performance toolbox is stubbornness. Without it all the mechanical sympathy in the world will go unexploited. There’s about a factor of 3 improvement that can be made to most code after the profiler has given up. That probably means there are better profilers than could be written, but in 20 years of having them I’ve only seen 2 that tried. Sadly I think flame graphs made profiling more access…

I think the biggest tool is higher expectations. Most programmers really haven't come to grips with the idea that computers are fast. If you see a database query that takes 1 hour to run, and only touches a few gb of data, you should be thinking "Well nvme bandwidth is multiple gigabytes per second, why can't it run in 1 second or less?" The idea that anyone would accept a request to a website taking longer than 30ms…

30mS for a website is a tough bar to clear considering Speed of Light (or rather electrons in copper / light in fiber)

https://en.wikipedia.org/wiki/Speed_of_light

Just as an example, round trip delay from where I rent to the local backbone is about 14mS alone, and the average for a webserver is 53mS. Just as a simple echo reply. (I picked it because I'd hoped that was in Redmond or some nearby datacenter, but it looks more likely to be in a cheaper labor area.)

However it's only the bloated ECMAScript (javascript) trash web of today that makes a website take longer than ~1 second to load on a modern PC. Plain old HTML, images on a reasonable diet, and some script elements only for interactive things can scream.

    mtr -bzw microsoft.com
    6. AS7922        be-36131-cs03.seattle.wa.ibone.comcast.net (2001:558:3:942::1)         0.0%    10   12.9  13.9  11.5  18.7   2.6
    7. AS7922        be-2311-pe11.seattle.wa.ibone.comcast.net (2001:558:3:3a::2)           0.0%    10   11.8  13.3  10.6  17.2   2.4
    8. AS7922        2001:559:0:80::101e                                                    0.0%    10   15.2  20.7  10.7  60.0  17.3
    9. AS8075        ae25-0.icr02.mwh01.ntwk.msn.net (2a01:111:2000:2:8000::b9a)            0.0%    10   41.1  23.7  14.8  41.9  10.4
    10. AS8075        be140.ibr03.mwh01.ntwk.msn.net (2603:1060:0:12::f18e)                  0.0%    10   53.1  53.1  50.2  57.4   2.1
    11. AS8075        2603:1060:0:10::f536                                                   0.0%    10   82.1  55.7  50.5  82.1   9.7
    12. AS8075        2603:1060:0:10::f3b1                                                   0.0%    10   54.4  96.6  50.4 147.4  32.5
    13. AS8075        2603:1060:0:10::f51a                                                   0.0%    10   49.7  55.3  49.7  78.4   8.3
    14. AS8075        2a01:111:201:f200::d9d                                                 0.0%    10   52.7  53.2  50.2  58.1   2.7
    15. AS8075        2a01:111:2000:6::4a51                                                  0.0%    10   49.4  51.6  49.4  54.1   1.7
    20. AS8075        2603:1030:b:3::152                                                     0.0%    10   50.7  53.4  49.2  60.7   4.2

Re: How We Found 7 TiB of Memory Just Sitting Around

#32

Post by devjab »

Aeolun wrote: post #24

I read this and I have to wonder, did anyone ever think it was reasonable that a cluster that apparently needed only 120gb of memory was consuming 1.2TB just for logging (or whatever vector does)

We're a much smaller scale company and the cost we lose on these things is insignificant compared to what's in this story. Yesterday I was improving the process for creating databases in our azure and I stumbled upon a subscription which was running 7 mssql servers for 12 databases. These weren't elastic and they were each paying a license that we don't have to pay because we qualify for the base cost through our contract with our microsoft partner. This company has some of the thightest control over their cloud infrastructure out of any organisation I've worked with.

This is anecdotal, but if my experiences aren't unique then there is a lot of lack of reasonable in DevOps.

Re: How We Found 7 TiB of Memory Just Sitting Around

#33

Post by fock »

Aeolun wrote: post #24

I read this and I have to wonder, did anyone ever think it was reasonable that a cluster that apparently needed only 120gb of memory was consuming 1.2TB just for logging (or whatever vector does)

we have on-prem with heavy spikes (our batch workload can utilize the 20TB of memory in the cluster easily) and we just don't care much and add 10% every year to the hardware requested. Compared to employing people or paying other vendors (relational databases with many TB-sized tables...) this is just irrelevant.

Sadly devs are incentivized by that and going towards the cloud might be a fun story. Given the environment I hope they scrap the effort sooner rather than later, buy some Oxide systems for the people who need to iterate faster than the usual process of getting a VM and replace/reuse the 10% of the company occupied with the cloud (mind you: no real workload runs there yet...) to actually improve local processes...

Re: How We Found 7 TiB of Memory Just Sitting Around

#34

Post by fock »

bstack wrote: post #30
Aeolun wrote: post #24

I read this and I have to wonder, did anyone ever think it was reasonable that a cluster that apparently needed only 120gb of memory was consuming 1.2TB just for logging (or whatever vector does)

Author here: You’d be surprised what you don’t notice given enough nodes and slow enough resource growth over time! Out of the total resource usage in these clusters even at the high water mark for this daemonset it was still a small overall portion of the total.

how large are the clusters then?

Re: How We Found 7 TiB of Memory Just Sitting Around

#35

Post by onethumb »

avidiax wrote: post #11
javier2 wrote: post #9

Earlier quoted context omitted.

its also about cost. My game computer has 8 cores + 1 expensive gpu + 32GB ram for me alone. We dont have that per customer.

It's also about revenue. Uber could run the complete global rider/driver flow from a single server. It doesn't, in part because all of those individual trips earn $1 or more each, so it's perfectly acceptable to the business to be more more inefficient and use hundreds of servers for this task. Similarly, a small website taking 150ms to render the page only matters if the lost productivity costs less than the enginee…

Uber could not run the complete global rider/driver flow from a single server.

Re: How We Found 7 TiB of Memory Just Sitting Around

#36

Post by hinkley »

vlovich123 wrote: post #29
hinkley wrote: post #6

Keys require O(logn) space per key or nlogn for the entire data set, simply to avoid key collisions. But human friendly key spaces grow much, much faster and I don’t think many people have looked too hard at that. There were recent changes to the NodeJS Prometheus client that eliminates tag names from the keys used for storing the tag cardinality for metrics. The memory savings wasn’t reported but the cpu savings for…

Why aren’t let’s just 128bit UUIDs? Those are guaranteed to be globally unique and don’t require so much spacex

Why aren’t what 128bit UUIDs?

> keeping names concise without triggering transposition or reading comprehension errors.

Code that doesn’t work for developers first will soon cease to work for anyone. Plus how do you look up a uuid for a set of tags? What’s your perfect hash plan to make sure you don’t misattribute stats to the wrong place?

UUIDs are entirely opaque and difficult to tell apart consistently.

Re: How We Found 7 TiB of Memory Just Sitting Around

#37

Post by Aeolun »

bstack wrote: post #30
Aeolun wrote: post #24

I read this and I have to wonder, did anyone ever think it was reasonable that a cluster that apparently needed only 120gb of memory was consuming 1.2TB just for logging (or whatever vector does)

Author here: You’d be surprised what you don’t notice given enough nodes and slow enough resource growth over time! Out of the total resource usage in these clusters even at the high water mark for this daemonset it was still a small overall portion of the total.

I’m not sure if that makes it better or worse.

Re: How We Found 7 TiB of Memory Just Sitting Around

#38

Post by hinkley »

hinkley wrote: post #16
zahlman wrote: post #7

Earlier quoted context omitted.

> The biggest tool in the performance toolbox is stubbornness. Without it all the mechanical sympathy in the world will go unexploited. The sympathy is also needed. Problems aren't found when people don't care, or consider the current performance acceptable. > There’s about a factor of 3 improvement that can be made to most code after the profiler has given up. That probably means there are better profilers than coul…

Architecture, cache eviction, memory bandwidth, thermal throttling. All of which have gotten perhaps an order of magnitude worse in the time since I started on this theory.

And Amdahl’s Law. Perf charts will complain about how much CPU you’re burning in the parallel parts of code and ignore that the bottleneck is down in 8% of the code that can’t be made concurrent.

Re: How We Found 7 TiB of Memory Just Sitting Around

#39

Post by OJFord »

Aeolun wrote: post #25
javier2 wrote: post #9

Earlier quoted context omitted.

its also about cost. My game computer has 8 cores + 1 expensive gpu + 32GB ram for me alone. We dont have that per customer.

If your websites take less than 16ms to serve, you can serve 60 customers per second with that. So you sorta do have it per customer?

With a latency of up to 984ms

Re: How We Found 7 TiB of Memory Just Sitting Around

#40

Post by exe34 »

onethumb wrote: post #35
avidiax wrote: post #11

Earlier quoted context omitted.

It's also about revenue. Uber could run the complete global rider/driver flow from a single server. It doesn't, in part because all of those individual trips earn $1 or more each, so it's perfectly acceptable to the business to be more more inefficient and use hundreds of servers for this task. Similarly, a small website taking 150ms to render the page only matters if the lost productivity costs less than the enginee…

Uber could not run the complete global rider/driver flow from a single server.

I believe the argument was that somebody competent could do it.
Post reply on HN