Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Calculating the cost of a Google DeepMind paper

152334h.github.io

Post reply on HN
Print view View original

81–90 of 160 posts

Re: Calculating the cost of a Google DeepMind paper

#81

Post by thaumasiotes »

axus wrote: post #54
thaumasiotes wrote: post #30

Earlier quoted context omitted.

> This is the side effect of underutilized capital and it’s present in many cases. "Underutilized" isn't the right word here. There's some value in putting your capital to productive use. But, once immediate needs are satisfied, there's more value in having the capital available to address future needs quickly than there would be in making sure that everything necessary to address those future needs is tied up in low…

I'm going to say this the next time I argue I need my servers online 24/7.

I'm not really sure I'm following you.

Re: Calculating the cost of a Google DeepMind paper

#82

Post by Workaccount2 »

OtherShrezzing wrote: post #79
willis936 wrote: post #3

Earlier quoted context omitted.

Any company of any size that doesn't learn the right lessons from a $10M mistake will be out of business before long.

I'm not really certain that's true at Google's size. Their annual revenue is something like a quarter trillion dollars. 25,000x larger than a $10m mistake. The equivalent wastage for a self-employed person would be allowing a few cups of Starbucks coffee per year to go cold.

There was that time that google paid out something like $300M in fraudulent invoices.

Re: Calculating the cost of a Google DeepMind paper

#83

Post by ramraj07 »

rgmerk wrote: post #67

Worth pointing out here that in other scientific domains, papers routinely require hundreds of thousands of dollars, sometimes millions of dollars, of resources to produce. My wife works on high-throughout drug screens. They routinely use over $100,000 of consumables in a single screen, not counting the cost of the screening “libraries”, the cost of using some of the -$10mil of equipment in the lab for several weeks,…

I estimated that any paper that has mouse work and produced in a first world country (I.e. they have to do good by the animals), the minimum cost of that paper in expenses and salary would be $200,000. Average likely higher. Tens of thousands of papers a year published like this!

Re: Calculating the cost of a Google DeepMind paper

#84

Post by stavros »

jfengel wrote: post #35
BartjeD wrote: post #5

If this ran on google's own cloud it amounts to internal bookkeeping. The only cost is then the electricity and used capacity. Not consumer pricing. So negligible. It is rather unfortunate that this sort of paper is hard to reproduce. That is a BIG downside, because it makes the result unreliable. They invested effort and money in getting an unreliable result. But perhaps other research will corroborate. Or it may gi…

Is the electricity cost negligible? It's a pretty compute intensive application. Of course it would be a tiny fraction of the $10m figure here, but even 1% would be $100,000. Negligible to Google, but for Google even $10 million is couch cushion money.

I feel like your comment answers itself: If you have the money to be running a datacenter of thousands of A100 GPUs (or equivalent), the cost of the electricity is negligible to you, and definitely worth training a SOTA model with your spare compute.

Re: Calculating the cost of a Google DeepMind paper

#85

Post by bbminner »

akutlay wrote: post #24
mrazomor wrote: post #13

Earlier quoted context omitted.

This assumes the common resources (CPU, RAM, etc.), not the ones required for the LLM training (GPU, TPU, etc.). It's different economy. TL; DR: It's not ~free.

Why does GPU matter? Do you think GCP keeps GPU utilization at 100% at all times?

Because accelerators (tpus, gpus) unlike ram/cpu are notoriously hard to timeshare and vitrualize. So if you get evicted in an environment like that, you have to reload your entire experiment state from a model checkpoint. With giant models like that, it might take dozens of minutes. As a result, I doubt that these experiments are done using "spare" resources - in that case, constant interruptions and reloading would result in these experiments finishing sometime around the heat death of the universe :)

Re: Calculating the cost of a Google DeepMind paper

#86

Post by Cthulhu_ »

BartjeD wrote: post #5

If this ran on google's own cloud it amounts to internal bookkeeping. The only cost is then the electricity and used capacity. Not consumer pricing. So negligible. It is rather unfortunate that this sort of paper is hard to reproduce. That is a BIG downside, because it makes the result unreliable. They invested effort and money in getting an unreliable result. But perhaps other research will corroborate. Or it may gi…

I'd argue it's not hard to reproduce per se, just expensive; thankfully there are at least half a dozen (cloud) computing providers that have the necessary resources to do so. Google Cloud, AWS and Azure are the big competitors in the west (it seems / from my perspective), but don't underestimate the likes of Alibaba, IBM, DigitalOcean, Rackspace, Salesforce, Tencent, Oracle, Huawei, Dell and Cisco.

Re: Calculating the cost of a Google DeepMind paper

#88

Post by esperent »

ramraj07 wrote: post #83
rgmerk wrote: post #67

Worth pointing out here that in other scientific domains, papers routinely require hundreds of thousands of dollars, sometimes millions of dollars, of resources to produce. My wife works on high-throughout drug screens. They routinely use over $100,000 of consumables in a single screen, not counting the cost of the screening “libraries”, the cost of using some of the -$10mil of equipment in the lab for several weeks,…

I estimated that any paper that has mouse work and produced in a first world country (I.e. they have to do good by the animals), the minimum cost of that paper in expenses and salary would be $200,000. Average likely higher. Tens of thousands of papers a year published like this!

To be fair, supposing the Google paper took six months to a year to produce, it also must have cost several hundred thousand dollars in salaries and other non-compute costs.

Re: Calculating the cost of a Google DeepMind paper

#89

Post by psychoslave »

dont_forget_me wrote: post #87

All that compute power just to invade privacy and show people more ads. Can this get anymore depressing?

Yes, sure! Imagine a world where every HN thread you engage in is fed with information that are all subtly tailored to push you into buying whatever crap the market is able to produce.
Post reply on HN