71–80 of 160 posts
Earlier quoted context omitted.
In principle, for some other company, sure. Google makes ~$300b a year in profit . They could make a $10m mistake every day and barely make a dent in it.
They do not, they made ~90 billion in profit. So no one would notice a 10 mil mistake, but no they didn't make 300b in profits.
3USD/hour on the H100 is much more expensive than a reasonable amortized full ownership cost, unless one assumes the GPU is useless within 18 months, which I find a bit dramatic. The MFU can be above 40% and certainly well above the 35% in the estimate, also for small models with plain pytorch and trivial tuning [1] I didnt read the linked paper carefully but I seriously doubt the google team used vocab embedding lay…
You are correct on true H100 ownership costs being far lower. As I mention in the H100 blurb, the H100 numbers are fungible and I don't mind if you halve them. MFU can certainly be improved beyond 40%, as I mention. But on the point of small models specifically: the paper uses FSDP for all models, and I believe a rigorous experiment should not vary sharding strategy due to numerical differences. FSDP2 on small models…
And the blog post author is talking about the output layer where the model has to produce an output prediction for every possible token in the vocabulary. Each output token prediction is a dot-product between the transformer hidden state (D) and the token embedding (D) (whether shared with input or not) for all tokens in the vocabulary (V). That's where the VD comes from.
It would be great to clarify this in the blog post to make it more accessible but I understand that there is a tradeoff.
Earlier quoted context omitted.
This assumes the common resources (CPU, RAM, etc.), not the ones required for the LLM training (GPU, TPU, etc.). It's different economy. TL; DR: It's not ~free.
Why does GPU matter? Do you think GCP keeps GPU utilization at 100% at all times?
You get that organically when you are serving lots of users. And, there's not much GPUs etc. used for that. Training LLMs gives you a different utilization pattern. The "best effort" resources aren't as useful in that setup.
A lot of misunderstandings among the commenters here. From the link: "the total compute cost it would take to replicate the paper" It's not Google's cost. Google's cost is of course entirely different. It's the cost for the author if he were to rent the resources to replicate the paper. For Google, all of it is running at a "best effort" resource tier, grabbing available resources when not requested by higher priorit…
I was told by an employee that GDM internally has a credits system for TPU allocation, with which researchers have to budget out their compute usage. I may have completely misunderstood what they were describing, though.
If this ran on google's own cloud it amounts to internal bookkeeping. The only cost is then the electricity and used capacity. Not consumer pricing. So negligible. It is rather unfortunate that this sort of paper is hard to reproduce. That is a BIG downside, because it makes the result unreliable. They invested effort and money in getting an unreliable result. But perhaps other research will corroborate. Or it may gi…
Opportunity cost is cost. What you could have earned by selling the resources to customers instead of using them yourself is what the resources are worth.
A lot of misunderstandings among the commenters here. From the link: "the total compute cost it would take to replicate the paper" It's not Google's cost. Google's cost is of course entirely different. It's the cost for the author if he were to rent the resources to replicate the paper. For Google, all of it is running at a "best effort" resource tier, grabbing available resources when not requested by higher priorit…
Content aside. This is hands down my favorite blog format.
A lot of misunderstandings among the commenters here. From the link: "the total compute cost it would take to replicate the paper" It's not Google's cost. Google's cost is of course entirely different. It's the cost for the author if he were to rent the resources to replicate the paper. For Google, all of it is running at a "best effort" resource tier, grabbing available resources when not requested by higher priorit…
According to neoclassical economists this is impossible since you can easily and instantaneously scale infrastructure up and down continuously at no cost and the future is known so demand can be predicted reliably. The problem with neoclassical economics is that it doesn't concern itself with the physical counterpart of liquidity. It is assumed that the physical world is just as liquid as the monetary world. The "liq…
Isn't that all what high frequency traders would say? :)
Perhaps there is some limit at which additional liquidity doesn't offer much value?
that's not very much in the business range, it's a lot when it comes to paying us salaries.
Any company of any size that doesn't learn the right lessons from a $10M mistake will be out of business before long.
The equivalent wastage for a self-employed person would be allowing a few cups of Starbucks coffee per year to go cold.
If this ran on google's own cloud it amounts to internal bookkeeping. The only cost is then the electricity and used capacity. Not consumer pricing. So negligible. It is rather unfortunate that this sort of paper is hard to reproduce. That is a BIG downside, because it makes the result unreliable. They invested effort and money in getting an unreliable result. But perhaps other research will corroborate. Or it may gi…
> They chose to publish. So they are interested in seeing it reproduced or improved upon. Call me cynical, but this is not what I experienced to be the #1 reason of publishing AI papers.