101–110 of 160 posts
I think if you wanted to think about a big expense you'd look at AlphaStar.
I am wondering if AlphaStar is the most expensive paper ever.
Earlier quoted context omitted.
Is the "best effort" tier similar to AWS spot instances?
At every cloud provider there's probably a tier below "spot" (or whatever the equivalent is called at AWS's competitors) that is used for the low-priority jobs of the cloud provider itself.
Worth pointing out here that in other scientific domains, papers routinely require hundreds of thousands of dollars, sometimes millions of dollars, of resources to produce. My wife works on high-throughout drug screens. They routinely use over $100,000 of consumables in a single screen, not counting the cost of the screening “libraries”, the cost of using some of the -$10mil of equipment in the lab for several weeks,…
If this ran on google's own cloud it amounts to internal bookkeeping. The only cost is then the electricity and used capacity. Not consumer pricing. So negligible. It is rather unfortunate that this sort of paper is hard to reproduce. That is a BIG downside, because it makes the result unreliable. They invested effort and money in getting an unreliable result. But perhaps other research will corroborate. Or it may gi…
> They chose to publish. So they are interested in seeing it reproduced or improved upon. Not necessarily, publishing also ensure that the stuff is no longer patentable.
If this ran on google's own cloud it amounts to internal bookkeeping. The only cost is then the electricity and used capacity. Not consumer pricing. So negligible. It is rather unfortunate that this sort of paper is hard to reproduce. That is a BIG downside, because it makes the result unreliable. They invested effort and money in getting an unreliable result. But perhaps other research will corroborate. Or it may gi…
I don’t think this is valid, as this point seems to ignore the fact that the data center that this compute took place in required a massive investment.
A paper like this is more akin to HEPP research. Nobody has the capability to reproduce the higgs results outside of at the facility the research was conducted within (CERN).
I don’t think reproduction was a concern of the researchers.
3USD/hour on the H100 is much more expensive than a reasonable amortized full ownership cost, unless one assumes the GPU is useless within 18 months, which I find a bit dramatic. The MFU can be above 40% and certainly well above the 35% in the estimate, also for small models with plain pytorch and trivial tuning [1] I didnt read the linked paper carefully but I seriously doubt the google team used vocab embedding lay…
Earlier quoted context omitted.
I estimated that any paper that has mouse work and produced in a first world country (I.e. they have to do good by the animals), the minimum cost of that paper in expenses and salary would be $200,000. Average likely higher. Tens of thousands of papers a year published like this!
These are mostly fixed costs. If you produce a hundred papers from the same team and same research, the costs aren't 100x.
3USD/hour on the H100 is much more expensive than a reasonable amortized full ownership cost, unless one assumes the GPU is useless within 18 months, which I find a bit dramatic. The MFU can be above 40% and certainly well above the 35% in the estimate, also for small models with plain pytorch and trivial tuning [1] I didnt read the linked paper carefully but I seriously doubt the google team used vocab embedding lay…
You are correct on true H100 ownership costs being far lower. As I mention in the H100 blurb, the H100 numbers are fungible and I don't mind if you halve them. MFU can certainly be improved beyond 40%, as I mention. But on the point of small models specifically: the paper uses FSDP for all models, and I believe a rigorous experiment should not vary sharding strategy due to numerical differences. FSDP2 on small models…
Even if it's a small model, one could use ddp or FSDP/2 without slowdowns on fast interconnect, which certainly adds to the cost. But if you want to reproduce all the work at the cheapest price point you only need to parallelize to the minimal level for fitting in memory (or rather, the one that maxes the MFU), so everything below 2B parameters runs on a single H100 or single node.
Earlier quoted context omitted.
I am wondering if AlphaStar is the most expensive paper ever.
I think it could be. I also think it is likely that HN frequenter `dekhn` has personally spent more money on compute resources than any other living human, so maybe they will chime in on how the cost gets allocated to the research.
From what I can tell, ads made the money and search/ads bought machines with their allocated budget, TI used their budget to run the systems, and then funny money in the form of quota was allocated to groups. THe money was "funny" in the sense that the full reach-through costs of operating a TPU for a year looks completely different from the production allocation quota that gets handed out. I think Google was long trying to create a market economy, but it was really much more like a state-funded exercise.
(I am not proud of how much CPU I wasted on protein folding/design and drug discovery, but I'm eternally thankful for Urs giving me the opportunity to try it out and also to compute the energy costs associated with the CPU use)