Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Found 7 TiB of Memory Just Sitting Around

render.com

Post reply on HN
Print view View original

51–60 of 76 posts

Re: How We Found 7 TiB of Memory Just Sitting Around

#51

Post by never_inline »

liampulles wrote: post #46

I'm a little surprised that it got to the point where pods which should consume a couple MB of RAM were consuming 4GB before action was taken. But I can also kind of understand it, because the way k8s operators (apps running in k8s that manipulate k8s resource) are meant to run is essentially a loop of listing resources, comparing to spec, and making moves to try and bring the state of the cluster closer to spec. Thi…

> are meant to run is essentially a loop of listing resources, comparing to spec, and making moves to try and bring the state of the cluster closer to spec.

Aren't they supposed to use watch/long polling?

Re: How We Found 7 TiB of Memory Just Sitting Around

#52

Post by [deleted] »

hinkley wrote: post #5
shanemhansen wrote: post #2

The unreasonable effectiveness of profiling and digging deep strikes again.

The biggest tool in the performance toolbox is stubbornness. Without it all the mechanical sympathy in the world will go unexploited. There’s about a factor of 3 improvement that can be made to most code after the profiler has given up. That probably means there are better profilers than could be written, but in 20 years of having them I’ve only seen 2 that tried. Sadly I think flame graphs made profiling more access…

[deleted]

Re: How We Found 7 TiB of Memory Just Sitting Around

#53

Post by never_inline »

hinkley wrote: post #18
jesse__ wrote: post #12

Earlier quoted context omitted.

Broadly agree. I'm curious, what're the profilers you know of that tried to be better? I have a little homebrew game engine with an integrated profiler that I'm always looking for ideas to make more effective.

Clinic.js tried and lost steam. I have a recollection of a profiler called JProfiler that represented space and time as a graph, but also a recollection they went under. And there is a company selling a product of that name that has been around since that time, but doesn’t quite look how I recalled and so I don’t know if I was mistaken about their demise or I’ve swapped product names in my brain. It was 20 years ago…

what do you think about speedscope's sandwich view?

Re: How We Found 7 TiB of Memory Just Sitting Around

#54

Post by avidiax »

onethumb wrote: post #35
avidiax wrote: post #11

Earlier quoted context omitted.

It's also about revenue. Uber could run the complete global rider/driver flow from a single server. It doesn't, in part because all of those individual trips earn $1 or more each, so it's perfectly acceptable to the business to be more more inefficient and use hundreds of servers for this task. Similarly, a small website taking 150ms to render the page only matters if the lost productivity costs less than the enginee…

Uber could not run the complete global rider/driver flow from a single server.

I'm saying you can keep track of all the riders and drivers, matchmake, start/progress/complete trips, with a single server, for the entire world.

Billing, serving assets like map tiles, etc. not included.

Some key things to understand:

* The scale of Uber is not that high. A big city surely has * The driver and rider phones participate in the state keeping. They send updates every 4 seconds, but they only have to be online to start a trip. Both mobiles cache a trip log that gets uploaded when network is available.

* Since driver/rider send updates every 4 seconds, and since you don't need to be online to continue or end a trip, you don't even need an active spare for the server. A hot spare can rebuild the world state in 4 seconds. State for a rider and driver is just a few bytes each for id, position and status.

* Since you'll have the rider and driver trip logs from their phones, you don't necessarily have to log the ride server side either. Its also OK to lose a little data on the server. You can use UDP.

Don't forget that in the olden times, all the taxis in a city like New York were dispatched by humans. All the police in the city were dispatched by humans. You can replace a building of dispatchers with a good server and mobile hardware working together.

Re: How We Found 7 TiB of Memory Just Sitting Around

#55

Post by hinkley »

avidiax wrote: post #11
javier2 wrote: post #9

Earlier quoted context omitted.

its also about cost. My game computer has 8 cores + 1 expensive gpu + 32GB ram for me alone. We dont have that per customer.

It's also about revenue. Uber could run the complete global rider/driver flow from a single server. It doesn't, in part because all of those individual trips earn $1 or more each, so it's perfectly acceptable to the business to be more more inefficient and use hundreds of servers for this task. Similarly, a small website taking 150ms to render the page only matters if the lost productivity costs less than the enginee…

Practically, you have to parcel out points of contention to a larger and larger team to stop them from spending 30 hours a week just coordinating for changes to the servers. So the servers divide to follow Conway’s Law, or the company goes bankrupt (why not both?).

Microservices try to fix that. But then you need bin packing so microservices beget kubernetes.

Re: How We Found 7 TiB of Memory Just Sitting Around

#56

Post by hinkley »

avidiax wrote: post #54
onethumb wrote: post #35

Earlier quoted context omitted.

Uber could not run the complete global rider/driver flow from a single server.

I'm saying you can keep track of all the riders and drivers, matchmake, start/progress/complete trips, with a single server, for the entire world. Billing, serving assets like map tiles, etc. not included. Some key things to understand: * The scale of Uber is not that high. A big city surely has * The driver and rider phones participate in the state keeping. They send updates every 4 seconds, but they only have to be…

You could envision a system that used one server per county and that’s 3k servers. Combine rural counties to get that down to 1000, and that’s probably less servers than uber runs.

What the internet will tell me is that uber has 4500 distinct services, which is more services than there are counties in the US.

Re: How We Found 7 TiB of Memory Just Sitting Around

#57

Post by hinkley »

zahlman wrote: post #49
hinkley wrote: post #16

Earlier quoted context omitted.

Architecture, cache eviction, memory bandwidth, thermal throttling. All of which have gotten perhaps an order of magnitude worse in the time since I started on this theory.

I meant architecture of the codebase , to be clear. (I'm sure that the increasing complexity of hardware architecture makes it harder to figure out how to write optimal code, but it isn't really degrading the performance of naive attempts, is it?)

The problem Windows had during its time of fame is the developers always had the fastest machines money could buy. That decreased the code-build-test cycle for them, but it also made it difficult for the developers to visualize how their code would run on normal hardware. Add the general lack of empathy inspired by their toxic corporate culture of “we are the best in the world” and its small wonder why windows, 95 and 98 ran more and more dogshit on older hardware.

My first job out of college, I got handed the slowest machine they had. The app was already half done and was dogshit slow even with small data sets. I was embarrassed to think my name would be associated with it. The UI painted so slowly I could watch the individual lines paint on my screen.

My friend and I in college had made homework into a game of seeing who could make their homework assignment run faster or using less memory. Such as calculating the Fibonacci of 100, or 1000. So I just started applying those skills and learning new ones.

For weeks I evaluated improvements to the code by saying “one Mississippi, two Mississippi”. Then how many syllables I got through. Then the stopwatch function on my watch. No profilers, no benchmarking tools, just code review.

And that’s how my first specialization became optimization.

Re: How We Found 7 TiB of Memory Just Sitting Around

#58

Post by g-mork »

fock wrote: post #33
Aeolun wrote: post #24

I read this and I have to wonder, did anyone ever think it was reasonable that a cluster that apparently needed only 120gb of memory was consuming 1.2TB just for logging (or whatever vector does)

we have on-prem with heavy spikes (our batch workload can utilize the 20TB of memory in the cluster easily) and we just don't care much and add 10% every year to the hardware requested. Compared to employing people or paying other vendors (relational databases with many TB-sized tables...) this is just irrelevant. Sadly devs are incentivized by that and going towards the cloud might be a fun story. Given the environm…

Somewhat unrelated, but you just tied wasteful software design to high it salaries, and also suggest a reason why Russian programmers might also seem to on the whole be far more effective than we are

I wonder if msft simply cut dev salaries by 50% in the 90s, would it have had any measurable effect on windows quality by today

Re: How We Found 7 TiB of Memory Just Sitting Around

#59

Post by hinkley »

never_inline wrote: post #53
hinkley wrote: post #18

Earlier quoted context omitted.

Clinic.js tried and lost steam. I have a recollection of a profiler called JProfiler that represented space and time as a graph, but also a recollection they went under. And there is a company selling a product of that name that has been around since that time, but doesn’t quite look how I recalled and so I don’t know if I was mistaken about their demise or I’ve swapped product names in my brain. It was 20 years ago…

what do you think about speedscope's sandwich view?

More of the same. JetBrains has an equivalent, though it seems to be broken at present. The sandwich keeps dragging you back to the flame graph. Call stack depth has value but width is harder for people to judge and it’s the wrong yardstick for many of the concerns I’ve mentioned in the rest of this thread.

The sandwich view hides invocation count, which is one of the biggest things you need to look at for that remaining 3x.

Also you need to think about budgets. Which is something game designers do and the rest of us ignore. Do I want 10% of overall processing time to be spent accessing reloadable config? Reporting stats? If the answer is no then we need to look at that, even if data retrieval is currently 40% of overall response time and we are trying to get from 2 seconds to 200 ms.

That means config and stats have a budget of 20ms each and you will never hit 200ms if someone doesn’t look at them. So you can pretend like they don’t exist until you get all the other tent poles chopped and then surprise pikachu face when you’ve already painted them into a corner with your other changes.

When we have a lot of shit that all needs to get done, you want to get to transparency, look at the pile and figure out how to do it all effectively. Combine errands and spread the stressful bits out over time. None of the tools and none of the literature supports this exercise, and in fact most of the literature is actively hostile to this exercise. Which is why you should read a certain level of reproval or even contempt in my writing about optimization. It’s very much intended.

Most advice on writing fast code has not materially changed for a time period where the number of calculations we do has increased by 5 orders of magnitude. In every other domain, we re-evaluate our solutions at each order of magnitude. We have marched past ignorant and into insane at this point. We are broken and we have been broken for twenty years.

Re: How We Found 7 TiB of Memory Just Sitting Around

#60

Post by hinkley »

mjevans wrote: post #31
Negitivefrags wrote: post #8

Earlier quoted context omitted.

I think the biggest tool is higher expectations. Most programmers really haven't come to grips with the idea that computers are fast. If you see a database query that takes 1 hour to run, and only touches a few gb of data, you should be thinking "Well nvme bandwidth is multiple gigabytes per second, why can't it run in 1 second or less?" The idea that anyone would accept a request to a website taking longer than 30ms…

30mS for a website is a tough bar to clear considering Speed of Light (or rather electrons in copper / light in fiber) https://en.wikipedia.org/wiki/Speed_of_light Just as an example, round trip delay from where I rent to the local backbone is about 14mS alone, and the average for a webserver is 53mS. Just as a simple echo reply. (I picked it because I'd hoped that was in Redmond or some nearby datacenter, but it loo…

In the cloud era this gets a bit better but my last job I removed a single service that was adding 30ms to response time and replaced it with a consul lookup with a watch on it. It wasn’t even a big service. Same DC, very simple graph query with a very small response. You can burn through 30 ms without half trying.
Post reply on HN