Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why did Google Brain exist?

moderndescartes.com

Post reply on HN
Print view View original

261–270 of 328 posts

Re: Why did Google Brain exist?

#261

Post by why_only_15 »

vl wrote: post #4

>PyTorch/Nvidia GPUs easily overtaking TensorFlow/Google TPUs. TF lost to PyTorch, and this is Google’s fault - TF APIs are both insane and badly documented. But nothing comes close to performance of Google’s TPU exaflop mega-clusters. Nvidia is not even in the same ballpark.

You are not correct about TPUs being drastically better for GPUs at this. If you look at public benchmarks, both have a similar cost per hardware flop ($0.88/hr for 312tflops A100 on GCP, $0.97/hr for 275tflops TPUv4) and both achieve similar model flop:hardware flop ratios (40-60%).

An existence proof that GPU mega-clusters are possible is that GPT-4 cost ~$100m over ~3 months, so ~100m a100-hours / (3 months * 30 days/month * 24 hours/day = 2160 hours) = ~45k a100s collaborating, which is the equivalent of ~10 TPUv4 pods on a single training run.

Re: Why did Google Brain exist?

#262

Post by p1esk »

sdenton4 wrote: post #213
kevviiinn wrote: post #71

Earlier quoted context omitted.

I think you missed my point. Genentech, AFAIK, was not doing research on machine learning as in the principles of how machine learning works and how to make it better. They do biotech research which uses applied machine learning. You don't need a PhD in ML to apply things that are already known

Sadly a lot of foundational ML research works for single-label image classification and not much else. ImageNet is a niche problem and way too much ML research is over-indexed on it. If you can make your problem look like ImageNet, you're going to do OK, but if not you effectively need to re-invent the wheel...

What you wrote was true until 4-5 years ago. These days almost all foundational ML research is about generative models.

Re: Why did Google Brain exist?

#263

Post by modeless »

alphabetting wrote: post #180
dekhn wrote: post #147

Earlier quoted context omitted.

X exists as a press-release generation system, not as a real technology creation system. They onboard many impractical projects that are either copies of something being done already in industry ("but with more Google and ML!") or doesn't have a market (space elevators).

Waymo has developed the modern autonomous vehicle from the ground up. It's basically a matter of scale now. It's a mindblowing tech stack. The first time riding in one is much more otherwordly than using GPT for the first time. The value of the technology is far greater whatever PR they have generated (not many people know about it)

Waymo predates X.

Re: Why did Google Brain exist?

#264

Post by mirker »

dekhn wrote: post #148
dr_dshiv wrote: post #7

Lots of great insight. Here’s one: “Given the long timelines of a PhD program, the vast majority of early ML researchers were self-taught crossovers from other fields. This created the conditions for excellent interdisciplinary work to happen. This transitional anomaly is unfortunately mistaken by most people to be an inherent property of machine learning to upturn existing fields. It is not. Today, the vast majority…

As somebody who has crossed the line between ML and chemistry many times, I would love to see: more ML researchers who know chemistry, more chemistry researchers who know ML, and best of all, fully cross-disciplinary researchers who are both masters of chemistry and ML, as those are the ones who move the field farthest, fastest.

I agree with you but does anyone even recognize the last category outside blue-sky research? People have a tendency to bin other people into buckets. Being a master at 2 things means you can’t be easily placed in a typical team structure.

Re: Why did Google Brain exist?

#265

Post by vl »

why_only_15 wrote: post #261
vl wrote: post #4

>PyTorch/Nvidia GPUs easily overtaking TensorFlow/Google TPUs. TF lost to PyTorch, and this is Google’s fault - TF APIs are both insane and badly documented. But nothing comes close to performance of Google’s TPU exaflop mega-clusters. Nvidia is not even in the same ballpark.

You are not correct about TPUs being drastically better for GPUs at this. If you look at public benchmarks, both have a similar cost per hardware flop ($0.88/hr for 312tflops A100 on GCP, $0.97/hr for 275tflops TPUv4) and both achieve similar model flop:hardware flop ratios (40-60%). An existence proof that GPU mega-clusters are possible is that GPT-4 cost ~$100m over ~3 months, so ~100m a100-hours / (3 months * 30 d…

I assume/suspect internally Google has v5 already.

The thing about TPU clusters that they have hyper-torus optical interconnect between TPUs. This allows for extremely efficient weight updates. To replicate this with A100s you need very custom hardware/software deployment.

But to be fair, I don’t know what is latest and greatest available from NVidia or other clouds in this area right now.

EDIT: Looks like NVidia has NVSwitch, which provides interconnect for 256 GPUs. Pretty cool!

Re: Why did Google Brain exist?

#266

Post by mirker »

throwaway2037 wrote: post #242
epolanski wrote: post #182

Earlier quoted context omitted.

It's the same reason analysts come from math rather than economy degrees. You can teach a mathematician what he needs to know about finance, you can hardly do the opposite.

In my experience, people with degrees in math, physics, chemistry, biology/medicine, astronomy, computer science have higher/stronger intellectual rigor that can be re-applied to other fields, e.g., finance. Also, those areas of study are much harder than economics. Partly, it is a self-selection process. Yes, there are some with economics degrees whom are very bright, but they probably could have majored in any of t…

The story I’ve heard is the economics undergrads can’t get into economics grad school. This is just a rumor but the sentiment is that undergrads get taught a watered down version of economics theory. Economics theory is potentially very technical and includes game theory and proofs. Even in CS, undergrads take intro theory courses and “bottom out” in their math skills, even though grad-level CS gets much more difficult. Therefore, I’d imagine the primary determinant of this rigor phenomenon is the GPA inflation of the major.

Re: Why did Google Brain exist?

#267

Post by PartiallyTyped »

kenjackson wrote: post #199
dr_dshiv wrote: post #7

Lots of great insight. Here’s one: “Given the long timelines of a PhD program, the vast majority of early ML researchers were self-taught crossovers from other fields. This created the conditions for excellent interdisciplinary work to happen. This transitional anomaly is unfortunately mistaken by most people to be an inherent property of machine learning to upturn existing fields. It is not. Today, the vast majority…

Chemistry is a centuries old discipline, that people study undergrad a full four years before getting a PhD in the field of chemistry. ML is a, practically speaking, 15 year old field that PhDs often begin to study after a couple of AI courses in undergrade and a specific track in grad school (while they study other parts of CS as part of their early graduate CS work). There's just way less context in ML than Chemist…

DL is a 15 year old field, ML is far older than that.

Re: Why did Google Brain exist?

#268

Post by mirker »

qumpis wrote: post #15
dr_dshiv wrote: post #7

Lots of great insight. Here’s one: “Given the long timelines of a PhD program, the vast majority of early ML researchers were self-taught crossovers from other fields. This created the conditions for excellent interdisciplinary work to happen. This transitional anomaly is unfortunately mistaken by most people to be an inherent property of machine learning to upturn existing fields. It is not. Today, the vast majority…

I'm yet to see an ML PhD be required to learn chemistry to a similar extent that chemists would need to doing ML (especially at research level)

I don’t understand what you mean. Here’s how many applied ML papers work: create a new dataset for a novel problem, download a PyTorch model, point model at dataset directory. Is it novel? By construction. Is the ML technique novel? No.

Re: Why did Google Brain exist?

#269

Post by bigger_cheese »

sidibe wrote: post #220
drakenot wrote: post #205

Earlier quoted context omitted.

This seems to kind of be the opposite opinion of The Bitter Lesson[0]. [0] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Someone still needs to have enough domain knowledge to figure out what needs solving and how to validate that it was solved. Being able to frame problems, and being able to cobble together the data and infra for the AI to solve them are the most valuable things in the medium term.

Yes I think having domain knowledge (such as Chemistry, Physics etc) is invaluable for real world problem solving. I am a Materials Engineer I see it in my field with ML as well. I think people are starting to understand now that being able to validate the inputs and outputs requires some understanding of the process you are trying to model. i.e. To ensure you aren't proposing things that fundamentally contradict basic Thermodynamics etc.

Even for optimization type modelling you need domain knowledge to design sensible constraints to frame the problem.

Re: Why did Google Brain exist?

#270

Post by kenjackson »

PartiallyTyped wrote: post #267
kenjackson wrote: post #199

Earlier quoted context omitted.

Chemistry is a centuries old discipline, that people study undergrad a full four years before getting a PhD in the field of chemistry. ML is a, practically speaking, 15 year old field that PhDs often begin to study after a couple of AI courses in undergrade and a specific track in grad school (while they study other parts of CS as part of their early graduate CS work). There's just way less context in ML than Chemist…

DL is a 15 year old field, ML is far older than that.

That’s why I said practically speaking. And realistically I don’t think you can even generously consider anything prior to perceptrons.
Post reply on HN