Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Found 7 TiB of Memory Just Sitting Around

render.com

Post reply on HN
Print view View original

61–70 of 76 posts

Re: How We Found 7 TiB of Memory Just Sitting Around

#61

Post by FroshKiller »

ffsm8 wrote: post #42
devjab wrote: post #32

Earlier quoted context omitted.

We're a much smaller scale company and the cost we lose on these things is insignificant compared to what's in this story. Yesterday I was improving the process for creating databases in our azure and I stumbled upon a subscription which was running 7 mssql servers for 12 databases. These weren't elastic and they were each paying a license that we don't have to pay because we qualify for the base cost through our con…

Isn't that mostly down to the fact the vast majority of devs explicitly don't want to do anything wrt Ops? DevOps has - ever since it's originally well meaning inception (by Netflix iirc?) - been implemented across our industry as an effective cost cutting measure, forcing devs that didn't see it as their job to also handle it. Which consequently means they're not interfacing with it whatsoever. They do as little as…

The first time my director asked me if I'd ever heard of DevOps, I said, "Sure, doing two jobs for one paycheck." I'm a software developer, buddy. I write the programs. Leave me out of running them.

Re: How We Found 7 TiB of Memory Just Sitting Around

#62

Post by antoniojtorres »

Aeolun wrote: post #37
bstack wrote: post #30

Earlier quoted context omitted.

Author here: You’d be surprised what you don’t notice given enough nodes and slow enough resource growth over time! Out of the total resource usage in these clusters even at the high water mark for this daemonset it was still a small overall portion of the total.

I’m not sure if that makes it better or worse.

It seems realistic to me, commonplace even. Lots to do in a company like this one.

Re: How We Found 7 TiB of Memory Just Sitting Around

#63

Post by javier2 »

Aeolun wrote: post #25
javier2 wrote: post #9

Earlier quoted context omitted.

its also about cost. My game computer has 8 cores + 1 expensive gpu + 32GB ram for me alone. We dont have that per customer.

If your websites take less than 16ms to serve, you can serve 60 customers per second with that. So you sorta do have it per customer?

Im just saying that we dont have gaming pc specs per customer to chug that 7GB of data for every request in 30ms

Re: How We Found 7 TiB of Memory Just Sitting Around

#64

Post by lazide »

exe34 wrote: post #40
onethumb wrote: post #35

Earlier quoted context omitted.

Uber could not run the complete global rider/driver flow from a single server.

I believe the argument was that somebody competent could do it.

The reality is that, no, that is not possible. If a single core can render and return a web page in 16ms, what do you do when you have a million requests/sec?

The reality is most of those requests (now) get mixed in with a firehose of traffic, and could be served much faster than 16ms if that is all that was going on. But it’s never all that is going on.

Re: How We Found 7 TiB of Memory Just Sitting Around

#65

Post by jiggawatts »

FroshKiller wrote: post #61
ffsm8 wrote: post #42

Earlier quoted context omitted.

Isn't that mostly down to the fact the vast majority of devs explicitly don't want to do anything wrt Ops? DevOps has - ever since it's originally well meaning inception (by Netflix iirc?) - been implemented across our industry as an effective cost cutting measure, forcing devs that didn't see it as their job to also handle it. Which consequently means they're not interfacing with it whatsoever. They do as little as…

The first time my director asked me if I'd ever heard of DevOps, I said, "Sure, doing two jobs for one paycheck." I'm a software developer, buddy. I write the programs. Leave me out of running them.

> Leave me out of running them.

This is how customers end up with too-expensive Rube Goldberg machines.

You have to take some interest in how your code will run in production, even if you don't personally "operate" it.

Re: How We Found 7 TiB of Memory Just Sitting Around

#67

Post by never_inline »

hinkley wrote: post #59
never_inline wrote: post #53

Earlier quoted context omitted.

what do you think about speedscope's sandwich view?

More of the same. JetBrains has an equivalent, though it seems to be broken at present. The sandwich keeps dragging you back to the flame graph. Call stack depth has value but width is harder for people to judge and it’s the wrong yardstick for many of the concerns I’ve mentioned in the rest of this thread. The sandwich view hides invocation count, which is one of the biggest things you need to look at for that remai…

I would like to know where I can read more in depth about profiling and performance analysis techniques.

Re: How We Found 7 TiB of Memory Just Sitting Around

#68

Post by Anonbrit »

embedding-shape wrote: post #44
Aeolun wrote: post #37

Earlier quoted context omitted.

I’m not sure if that makes it better or worse.

I didn't know what Render was when I skimmed the article at first, but after reading these comments, I had to check out what they do. And they're a "Cloud Application Platform" meaning they manage deploys and infrastructure for other people . Their website says "Click, click, done." which is cool and quick and all, but to me it's kind of crazy an organization that should be really engineering focused and mature, does…

If your report for the month is "I saved a terabyte of ram usage across our cluster estate!" and I as a manager do some quick maths and say great, that's our income from 2 median customers. We lost 8 customers because we didn't laugh feature foo in time, which is what you were supposed to be working on, so your contribution for the month is a massive loss to the company...

Does that frame things differently? There's are times in your product lifecycle where you doing want your developers looking at things like this, and a time when you do

Re: How We Found 7 TiB of Memory Just Sitting Around

#69

Post by bastawhiz »

embedding-shape wrote: post #44
Aeolun wrote: post #37

Earlier quoted context omitted.

I’m not sure if that makes it better or worse.

I didn't know what Render was when I skimmed the article at first, but after reading these comments, I had to check out what they do. And they're a "Cloud Application Platform" meaning they manage deploys and infrastructure for other people . Their website says "Click, click, done." which is cool and quick and all, but to me it's kind of crazy an organization that should be really engineering focused and mature, does…

7tb in an organization running probably petabytes of ram total is easy to slip under the radar. There's a lot of systems and a lot of moving parts and if it's not broke or triggering alarms, you probably don't care very much.
Post reply on HN