Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Calculating the cost of a Google DeepMind paper

152334h.github.io

Post reply on HN
Print view View original

51–60 of 160 posts

Re: Calculating the cost of a Google DeepMind paper

#51

Post by echoangle »

rrr_oh_man wrote: post #29
BartjeD wrote: post #5

If this ran on google's own cloud it amounts to internal bookkeeping. The only cost is then the electricity and used capacity. Not consumer pricing. So negligible. It is rather unfortunate that this sort of paper is hard to reproduce. That is a BIG downside, because it makes the result unreliable. They invested effort and money in getting an unreliable result. But perhaps other research will corroborate. Or it may gi…

> They chose to publish. So they are interested in seeing it reproduced or improved upon. Call me cynical, but this is not what I experienced to be the #1 reason of publishing AI papers.

As someone not in the AI space, what do you think is the reason for publishing? Marketing and hype for your products?

Re: Calculating the cost of a Google DeepMind paper

#52

Post by dweekly »

arcade79 wrote: post #11

A lot of misunderstandings among the commenters here. From the link: "the total compute cost it would take to replicate the paper" It's not Google's cost. Google's cost is of course entirely different. It's the cost for the author if he were to rent the resources to replicate the paper. For Google, all of it is running at a "best effort" resource tier, grabbing available resources when not requested by higher priorit…

Possible corollary: it may be difficult to regularly turn out highly compute-dependent research if you're paying full retail rack rates for your hardware (i.e. using someone else's cloud).

Re: Calculating the cost of a Google DeepMind paper

#53

Post by simonw »

echoangle wrote: post #51
rrr_oh_man wrote: post #29

Earlier quoted context omitted.

> They chose to publish. So they are interested in seeing it reproduced or improved upon. Call me cynical, but this is not what I experienced to be the #1 reason of publishing AI papers.

As someone not in the AI space, what do you think is the reason for publishing? Marketing and hype for your products?

Retaining your researchers so they don't get frustrated and move to another company that lets them publish.

Re: Calculating the cost of a Google DeepMind paper

#54

Post by axus »

thaumasiotes wrote: post #30
bombcar wrote: post #15

Earlier quoted context omitted.

This is the side effect of underutilized capital and it’s present in many cases. For example, if YOU want to rent a backhoe to do some yard rearrangement it’s going to cost you. But Bob who owns BackHoesInc has them sitting around all the time when they’re not being rented or used; he can rearrange his yard wholesale or almost free.

> This is the side effect of underutilized capital and it’s present in many cases. "Underutilized" isn't the right word here. There's some value in putting your capital to productive use. But, once immediate needs are satisfied, there's more value in having the capital available to address future needs quickly than there would be in making sure that everything necessary to address those future needs is tied up in low…

I'm going to say this the next time I argue I need my servers online 24/7.

Re: Calculating the cost of a Google DeepMind paper

#55

Post by efitz »

thaumasiotes wrote: post #30
bombcar wrote: post #15

Earlier quoted context omitted.

This is the side effect of underutilized capital and it’s present in many cases. For example, if YOU want to rent a backhoe to do some yard rearrangement it’s going to cost you. But Bob who owns BackHoesInc has them sitting around all the time when they’re not being rented or used; he can rearrange his yard wholesale or almost free.

> This is the side effect of underutilized capital and it’s present in many cases. "Underutilized" isn't the right word here. There's some value in putting your capital to productive use. But, once immediate needs are satisfied, there's more value in having the capital available to address future needs quickly than there would be in making sure that everything necessary to address those future needs is tied up in low…

I think a better description than “underutilized” would be “sunk capex cost” - Google (or any cloud provider) cannot run at 100% customer utilization because then they could neither acquire new customers nor service transitory usage spikes for existing customers. So they stay ahead of predicted demand, which means that they will almost always have excess capacity available.

Cloud providers pay capital costs (CapEx) for servers, GPUs, data centers, employees, etc. Utilization allows them to recoup those costs faster.

Cloud customers pay operational expenses (OpEx) for usage.

So Google generally has excess capacity, and while they would prefer revenue-generating customer usage, they’ve already paid for everything but the electricity, so it’s extremely cheap for them to run their own jobs if the hardware would otherwise be sitting idle.

Re: Calculating the cost of a Google DeepMind paper

#56

Post by spi »

pama wrote: post #40

3USD/hour on the H100 is much more expensive than a reasonable amortized full ownership cost, unless one assumes the GPU is useless within 18 months, which I find a bit dramatic. The MFU can be above 40% and certainly well above the 35% in the estimate, also for small models with plain pytorch and trivial tuning [1] I didnt read the linked paper carefully but I seriously doubt the google team used vocab embedding lay…

Do you have sources for "The MFU can be above 40% and certainly well above the 35 % in the estimate"?

Looking at [1], the authors there claim that their improvements were needed to push BERT training beyond 30% MFU, and that the "default" training only reaches 10%. Certainly numbers don't translate exactly, it might well be that with a different stack, model, etc., it is easier to surpass, but 35% doesn't seem like a terribly off estimate to me. Especially so if you are training a whole suite of different models (with different parameters, sizes, etc.) so you can't realistically optimize all of them.

It might be that the real estimate is around 40% instead of the 35% used here (frankly it might be that it is 30% or less, for that matter), but I would doubt it's so high as to make the estimates in this blog post terribly off, and I would doubt even more that you can get that "also for small models with plain pytorch and trivial tuning".

[1] https://www.databricks.com/blog/mosaicbert

Re: Calculating the cost of a Google DeepMind paper

#57

Post by b112 »

efitz wrote: post #55
thaumasiotes wrote: post #30

Earlier quoted context omitted.

> This is the side effect of underutilized capital and it’s present in many cases. "Underutilized" isn't the right word here. There's some value in putting your capital to productive use. But, once immediate needs are satisfied, there's more value in having the capital available to address future needs quickly than there would be in making sure that everything necessary to address those future needs is tied up in low…

I think a better description than “underutilized” would be “sunk capex cost” - Google (or any cloud provider) cannot run at 100% customer utilization because then they could neither acquire new customers nor service transitory usage spikes for existing customers. So they stay ahead of predicted demand, which means that they will almost always have excess capacity available. Cloud providers pay capital costs (CapEx) f…

I doubt they are doing this, but if they did burn in tests with 3 machines doing identical workloads, they could validate workloads but also test new infra. Unlike customer workloads, it would be OK to retey due to error.

This would be 100% free, as all electricity and "wear and tear" would be required anyhow.

Re: Calculating the cost of a Google DeepMind paper

#58

Post by huijzer »

arcade79 wrote: post #11

A lot of misunderstandings among the commenters here. From the link: "the total compute cost it would take to replicate the paper" It's not Google's cost. Google's cost is of course entirely different. It's the cost for the author if he were to rent the resources to replicate the paper. For Google, all of it is running at a "best effort" resource tier, grabbing available resources when not requested by higher priorit…

Still, don’t get high on your own supply.

Re: Calculating the cost of a Google DeepMind paper

#59

Post by b112 »

Bjartr wrote: post #44
nathancahill wrote: post #33

Earlier quoted context omitted.

Same effect when leasing companies let office space sit unoccupied for years on end. The future value is higher than the marginal value of reducing the price to fill it with a tenant.

That may be part of it for spaces properties left unleased for years, but I believe it's not the only part. I believe the larger factor, and someone correct me if they have a better understanding of this, is that for commercially rented properties the valuation used to determine the mortgage terms you get takes into account what you claim to be able to get from rent. Renting for less than that reduces the valuation a…

Many cities only tax on leased property, or have very low rates on unleased property.

Re: Calculating the cost of a Google DeepMind paper

#60

Post by immibis »

efitz wrote: post #55
thaumasiotes wrote: post #30

Earlier quoted context omitted.

> This is the side effect of underutilized capital and it’s present in many cases. "Underutilized" isn't the right word here. There's some value in putting your capital to productive use. But, once immediate needs are satisfied, there's more value in having the capital available to address future needs quickly than there would be in making sure that everything necessary to address those future needs is tied up in low…

I think a better description than “underutilized” would be “sunk capex cost” - Google (or any cloud provider) cannot run at 100% customer utilization because then they could neither acquire new customers nor service transitory usage spikes for existing customers. So they stay ahead of predicted demand, which means that they will almost always have excess capacity available. Cloud providers pay capital costs (CapEx) f…

There is also a mathematical relationship in queuing theory between utilization and average queue length, which all programmers should be told: https://blog.danslimmon.com/2016/08/26/the-most-important-th...

As you run close to 100% utilization, you also run close to infinity waiting times. You don't want that. It might be acceptable for your internal projects (the actual waiting time won't be infinity, and you'll cancel them if it gets too close to infinity) but it's certainly not acceptable for customers.

Post reply on HN