[deleted]
Calculating the cost of a Google DeepMind paper
21–30 of 160 posts
Re: Calculating the cost of a Google DeepMind paper
#22This is calculation is pretty pointless and the title is flat out wrong. It also gets lost in finer details while totally missing the bigger picture. After all, the original paper written by people either working for Google or at Google. So you can safely assume they used Google resources. That means they wouldn't have used H100s, but Google TPUs. Since they design and own these TPUs, you can also safely assume that…
Reproducibility is a key element of the scientific process How is anyone else going to reproduce the experiment if it's going to cost them $10 million because they don't work at Google and would have to rent the infrastructure?
This specific paper looks plausible, but a lot of published AI papers are simply fake because it is one of the sectors where it is possible to make non-reproducible claims. "We don't give source-code or dataset", but actually they didn't find or do anything of interest.
It works and helps to get a salary raise or a better job, so they continue.
A bit like when someone goes to a job interview, didn't do anything, and claims "My work is under NDA".
Re: Calculating the cost of a Google DeepMind paper
#23Earlier quoted context omitted.
That's like staffing a single-manager team on a bad project for a year. Which I assure you happens all the time in big companies, and yet they survive.
They are not saying it doesn't happen. They are saying: The companies that don't learn from these mistakes will go out of business before long.
I think there might be a disagreement about what "big" means. Google can easily afford to sink millions each year into pointless endeavours without going out of business and they probably have. Alphabet's annual revenue has been growing a good 10% each year since 2021[0]. That's in the range of $20-$30 billion dollars with a B.
To put that into perspective, Alphabet's revenue has increased 13.38% year-over-year as of June 30, arriving at $328.284 billion dollars - i.e. it has increased by $38.74 billion in that time. A $10 million dollar mistake translates to losing 0.0258% of that number.
A $10 million dollar mistake costs Alphabet 0.0258% of the amount their revenue increased year-over-year as of last month. Alphabet could have afforded to make 40 such $10 million dollar mistakes in that period and it would have only represented a loss of 1% of the year-over-year increase in revenue. Taking the year-over-year increase down by 1% (from 13.38% to 12.38%) would have required making 290 such $10 million dollar mistakes within one year.
Let me repeat that because it bears emphasizing: over the past years, every year Google could have easily afforded an additional 200 such $10 million dollar mistakes without significantly impacting their increase in revenue - and even in 2022 when inflation was almost double what it was in the other year they would have still come out ahead of inflation.
So in terms of numbers this is demonstrably false. Of course the existence of repeated $10 million dollar mistakes may suggest the existence of structural issues that will result in $1, $10 or $100 billion dollar problems eventually and sink the company. But that's conjecture at this point.
[0]: https://www.macrotrends.net/stocks/charts/GOOG/alphabet/reve...
Re: Calculating the cost of a Google DeepMind paper
#24A lot of misunderstandings among the commenters here. From the link: "the total compute cost it would take to replicate the paper" It's not Google's cost. Google's cost is of course entirely different. It's the cost for the author if he were to rent the resources to replicate the paper. For Google, all of it is running at a "best effort" resource tier, grabbing available resources when not requested by higher priorit…
This assumes the common resources (CPU, RAM, etc.), not the ones required for the LLM training (GPU, TPU, etc.). It's different economy. TL; DR: It's not ~free.
Why does GPU matter? Do you think GCP keeps GPU utilization at 100% at all times?
Re: Calculating the cost of a Google DeepMind paper
#25If this ran on google's own cloud it amounts to internal bookkeeping. The only cost is then the electricity and used capacity. Not consumer pricing. So negligible. It is rather unfortunate that this sort of paper is hard to reproduce. That is a BIG downside, because it makes the result unreliable. They invested effort and money in getting an unreliable result. But perhaps other research will corroborate. Or it may gi…
Opportunity cost is cost. What you could have earned by selling the resources to customers instead of using them yourself is what the resources are worth.
Re: Calculating the cost of a Google DeepMind paper
#26A lot of misunderstandings among the commenters here. From the link: "the total compute cost it would take to replicate the paper" It's not Google's cost. Google's cost is of course entirely different. It's the cost for the author if he were to rent the resources to replicate the paper. For Google, all of it is running at a "best effort" resource tier, grabbing available resources when not requested by higher priorit…
This is the side effect of underutilized capital and it’s present in many cases. For example, if YOU want to rent a backhoe to do some yard rearrangement it’s going to cost you. But Bob who owns BackHoesInc has them sitting around all the time when they’re not being rented or used; he can rearrange his yard wholesale or almost free.
[deleted]
Re: Calculating the cost of a Google DeepMind paper
#27If this ran on google's own cloud it amounts to internal bookkeeping. The only cost is then the electricity and used capacity. Not consumer pricing. So negligible. It is rather unfortunate that this sort of paper is hard to reproduce. That is a BIG downside, because it makes the result unreliable. They invested effort and money in getting an unreliable result. But perhaps other research will corroborate. Or it may gi…
Opportunity cost is cost. What you could have earned by selling the resources to customers instead of using them yourself is what the resources are worth.
This assumes that you can sell 100% of the resources' availability 100% of the time. Whenever you have more capacity that you can sell, there's no opportunity cost in using it yourself.
Re: Calculating the cost of a Google DeepMind paper
#28Earlier quoted context omitted.
Can others also buy the “best effort” tier? If the job could easily run for weeks, even when you could buy your way for doing it in a day. Then have a bidding on this “best effort” resource, where they factor in electricity at any given time
Is the "best effort" tier similar to AWS spot instances?
[deleted]
Re: Calculating the cost of a Google DeepMind paper
#29If this ran on google's own cloud it amounts to internal bookkeeping. The only cost is then the electricity and used capacity. Not consumer pricing. So negligible. It is rather unfortunate that this sort of paper is hard to reproduce. That is a BIG downside, because it makes the result unreliable. They invested effort and money in getting an unreliable result. But perhaps other research will corroborate. Or it may gi…
> They chose to publish. So they are interested in seeing it reproduced or improved upon.
Call me cynical, but this is not what I experienced to be the #1 reason of publishing AI papers.
Re: Calculating the cost of a Google DeepMind paper
#30A lot of misunderstandings among the commenters here. From the link: "the total compute cost it would take to replicate the paper" It's not Google's cost. Google's cost is of course entirely different. It's the cost for the author if he were to rent the resources to replicate the paper. For Google, all of it is running at a "best effort" resource tier, grabbing available resources when not requested by higher priorit…
This is the side effect of underutilized capital and it’s present in many cases. For example, if YOU want to rent a backhoe to do some yard rearrangement it’s going to cost you. But Bob who owns BackHoesInc has them sitting around all the time when they’re not being rented or used; he can rearrange his yard wholesale or almost free.
> This is the side effect of underutilized capital and it’s present in many cases.
"Underutilized" isn't the right word here. There's some value in putting your capital to productive use. But, once immediate needs are satisfied, there's more value in having the capital available to address future needs quickly than there would be in making sure that everything necessary to address those future needs is tied up in low-value work. Option value is real value; being prepared for unforeseen but urgent circumstances is a real use.