Calculating the cost of a Google DeepMind paper • Hacker News Board

Re: Calculating the cost of a Google DeepMind paper

#121

Post by godelski » Tue, Jul 30, 2024, 7:21 PM UTC

lucianbr wrote: post #108

paxys wrote: post #93

Earlier quoted context omitted.

These are mostly fixed costs. If you produce a hundred papers from the same team and same research, the costs aren't 100x.

But starting from the 10th paper, the value is also pretty low I imagine. How many new things can you discover from the same team and same research? That's 3 papers per year for a 30-year career. Every single year, no breaks.

> How many new things can you discover from the same team and same research?

That all depends on how you measure discoveries. The most common metric is... publications. Publications are what advance your career and are what you are evaluated on. The content may or may not matter (lol who reads your papers?) but the number certainly does. So the best way to advance your career is to write a minimum viable paper and submit as often as possible. I think we all forget how Goodhart's Law comes to bite everyone in the ass.

Re: Calculating the cost of a Google DeepMind paper

#122

Post by godelski » Tue, Jul 30, 2024, 7:23 PM UTC

sdenton4 wrote: post #118

lucianbr wrote: post #108

Earlier quoted context omitted.

But starting from the 10th paper, the value is also pretty low I imagine. How many new things can you discover from the same team and same research? That's 3 papers per year for a 30-year career. Every single year, no breaks.

Well, to be sure, mouse research consistently produces amazing cures for cancer, insomnia, lost limbs, and even gravity itself. Sure, none of it translates to humans, but it's an important source of headlines for high impact journals and science columnists.

This is also true for machine learning papers. They cure cancer, discover physics, and all sorts of things. Sure, they don't actually translate to useful science, but they are highly valuable pieces of advertisements. And hey, maybe someday they might!

Re: Calculating the cost of a Google DeepMind paper

#123

Post by godelski » Tue, Jul 30, 2024, 7:29 PM UTC

slashdave wrote: post #103

rgmerk wrote: post #67

Worth pointing out here that in other scientific domains, papers routinely require hundreds of thousands of dollars, sometimes millions of dollars, of resources to produce. My wife works on high-throughout drug screens. They routinely use over $100,000 of consumables in a single screen, not counting the cost of the screening “libraries”, the cost of using some of the -$10mil of equipment in the lab for several weeks,…

I assure you that the companies performing these screens expect a return on this investment. It is not for a journal paper.

I used to believe this line. But then I worked for a big tech company where my manager constantly made those remarks ("the difference in industry and academia is that in industry it has to actually work"). I then improved the generalization performance (i.e. "actually work") by over 100% and they decided not to update the model they were selling. Then again, I had a small fast model and it was 90% as accurate as the new large transformer model. Though they also didn't take the lessons learned and apply them to the big model, which had similar issues but were just masked by the size.

Plus, I mean, there are a lot of products that don't work. We all buy garbage and often can't buy not garbage. Though I guess you're technically correct that in either of these situations there can still be a return on investment, but maybe that shouldn't be good enough...

Re: Calculating the cost of a Google DeepMind paper

#124

Post by computerdork » Tue, Jul 30, 2024, 7:39 PM UTC

sdenton4 wrote: post #118

lucianbr wrote: post #108

Earlier quoted context omitted.

But starting from the 10th paper, the value is also pretty low I imagine. How many new things can you discover from the same team and same research? That's 3 papers per year for a 30-year career. Every single year, no breaks.

Well, to be sure, mouse research consistently produces amazing cures for cancer, insomnia, lost limbs, and even gravity itself. Sure, none of it translates to humans, but it's an important source of headlines for high impact journals and science columnists.

Agreed about this for past mouse research, but this is changing as mice are being genetically engineered to be more human: https://news.uthscsa.edu/scientists-create-first-mouse-model... https://www.nature.com/articles/s41467-019-09716-7

Re: Calculating the cost of a Google DeepMind paper

#125

Post by godelski » Tue, Jul 30, 2024, 7:40 PM UTC

ash-ali wrote: post #80

rrr_oh_man wrote: post #29

Earlier quoted context omitted.

> They chose to publish. So they are interested in seeing it reproduced or improved upon. Call me cynical, but this is not what I experienced to be the #1 reason of publishing AI papers.

I hope someone could share their insight on this comment. I think the other comments are fragile and don't hold too strongly.

It's commonly discussed in AI/ML groups that a paper at a top conference is "worth a million dollars." Not all papers, some papers are worth more. But it is in effect discussing the downstream revenues. As a student, it is your job and potential earnings. As a lab it is worth funding and getting connected to big tech labs (which creates a feedback loop). And to corporations, it is worth far more than that in advertising.

The unfortunate part of this is that it can have odd effects like people renaming well known things to make the work appear more impressive, obscure concepts, and drive up their citations.[0] The incentives do not align to make your paper as clear and concise as possible to communicate your work.

[0] https://youtu.be/Pl8BET_K1mc?t=2510

Re: Calculating the cost of a Google DeepMind paper

#126

Post by chazeon » Tue, Jul 30, 2024, 7:57 PM UTC

faitswulff wrote: post #113

I wonder how many tons of CO2 that amounts to. Google Gemini estimated 125,000 tons of carbon emissions, but I don’t have the know-how to double check it.

If you use solar energy, then there is no CO2 emission. Right?

Re: Calculating the cost of a Google DeepMind paper

#127

Post by hiddencost » Tue, Jul 30, 2024, 8:13 PM UTC

It's likely the cost of the researchers was about $1m/ head, with 11 names that puts the staffing costs on par with the compute costs.

(A good rule of thumb is that an employee costs about twice their total compensation.)

Re: Calculating the cost of a Google DeepMind paper

#128

Post by dmurray » Tue, Jul 30, 2024, 8:24 PM UTC

michaelt wrote: post #69

g15jv2dp wrote: post #27

Earlier quoted context omitted.

This assumes that you can sell 100% of the resources' availability 100% of the time. Whenever you have more capacity that you can sell, there's no opportunity cost in using it yourself.

A few months back, a lot of the most powerful GPU instances on GCP seemed to be sold out 24/7. I suppose it's possible Google's own infrastructure is partitioned from GCP infrastructure, so they have a bunch of idle GPUs even while their cloud division can sell every H100 and A100 they can get their hands on?

I'd expect they have both: dedicated machines that they usually use and are sometimes idle, but also the ability to run a job on GCP if it makes sense.

(I doubt it's the other way round, that the Deepmind researchers could come in one day and find all their GPUs are being used by some cloud customer).

Re: Calculating the cost of a Google DeepMind paper

#129

Post by shpongled » Tue, Jul 30, 2024, 8:32 PM UTC

godelski wrote: post #123

slashdave wrote: post #103

Earlier quoted context omitted.

I assure you that the companies performing these screens expect a return on this investment. It is not for a journal paper.

I used to believe this line. But then I worked for a big tech company where my manager constantly made those remarks ("the difference in industry and academia is that in industry it has to actually work"). I then improved the generalization performance (i.e. "actually work") by over 100% and they decided not to update the model they were selling. Then again, I had a small fast model and it was 90% as accurate as the…

The post you are replying to is talking about high throughput assays for drug development. This is something actually run in a lab, not a model. As another person working at a biotech, I can assure you that screens are not just run as busy work.

Re: Calculating the cost of a Google DeepMind paper

#130

Post by rgmerk » Tue, Jul 30, 2024, 8:50 PM UTC

shpongled wrote: post #129

godelski wrote: post #123

Earlier quoted context omitted.

I used to believe this line. But then I worked for a big tech company where my manager constantly made those remarks ("the difference in industry and academia is that in industry it has to actually work"). I then improved the generalization performance (i.e. "actually work") by over 100% and they decided not to update the model they were selling. Then again, I had a small fast model and it was 90% as accurate as the…

The post you are replying to is talking about high throughput assays for drug development. This is something actually run in a lab, not a model. As another person working at a biotech, I can assure you that screens are not just run as busy work.

No they’re not busywork, but not all such screens are directly in the drug discovery pipeline.