Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why did Google Brain exist?

moderndescartes.com

Post reply on HN
Print view View original

301–310 of 328 posts

Re: Why did Google Brain exist?

#301

Post by [deleted] »

gopalv wrote: post #226
simonster wrote: post #125

I work for Google Brain. I remember meeting Brian at a conference and I have nothing but good things to say about him. That said, I think Brian is underestimating the extent to which the Brain/DeepMind merger is happening because it's what researchers want. Many of us have a strong sense that the future of ML involves models built by large teams in industry environments. My impression is that the goal of the merger i…

> Many of us have a strong sense that the future of ML involves models built by large teams in industry environments The gradient of current moment is that whatever approach is optimized to use more data and more compute is much easier to invest in than something which can do more with less, but with a significant number of possible dead-ends. At some point, this will have diminishing returns, but until that is hit,…

[deleted]

Re: Why did Google Brain exist?

#302

Post by melagonster »

ramraj07 wrote: post #232
qumpis wrote: post #15

Earlier quoted context omitted.

I'm yet to see an ML PhD be required to learn chemistry to a similar extent that chemists would need to doing ML (especially at research level)

I’m pretty sure the author is implying that the new crop of ML PhDs are just not a smart group of people - at least the level of intelligence required to do truly transformative things with ML in any field. I think what you’re saying is a commonly found attitude that relates to this topic: it’s pretty limiting to think a cursory knowledge of a field is sufficient to go change it. That’s likely why most “use ML to sol…

maybe he just say do not expect people resolve problems outside of their discipline.

Re: Why did Google Brain exist?

#303

Post by cubefox »

qumpis wrote: post #297
cubefox wrote: post #225

Earlier quoted context omitted.

Bard uses a smaller model currently, which was announced before release. > We’re releasing it initially with our lightweight model version of LaMDA. This much smaller model requires significantly less computing power, enabling us to scale to more users, allowing for more feedback. [1] > Bard is powered by a research large language model (LLM), specifically a lightweight and optimized version of LaMDA, and will be upd…

I remember the CEO of google saying that more capable models will be releases a few months (a month?) ago already. Either they delayed or the model is just as bad (maybe they released after the announcement of coding in 20 languages?)

First link is the CEO, it's from February.

Re: Why did Google Brain exist?

#304

Post by ramraj07 »

jltsiren wrote: post #299
ramraj07 wrote: post #295

Earlier quoted context omitted.

What exactly has it transformed? Nothing has fundamentally changed in biology, you still have to run western blots and give mice cancer. A great example of “math” solving biology is super resolution - which was just a dud.

Genomics, for example. The entire field could not exist without extensive algorithmic research done over several decades. Even today, many of the key people in the field have a background in CS and mathematics. Theoretical ecology is a bit more old-school answer. Many mathematicians have been involved in that field.

I am not sure I have seen any meaningfully brilliant breakthroughs in those fields though.

I take that back. One can never forget the brilliance of HiC by Erez. But other than that TBH all other “mathematical” or computational breakthroughs I’ve seen are just at best meticulous application of obvious math and computational algorithms to biological problems (and may I say poorly? Thinking back to the microarray nightmare years).

Re: Why did Google Brain exist?

#305

Post by qumpis »

mirker wrote: post #268
qumpis wrote: post #15

Earlier quoted context omitted.

I'm yet to see an ML PhD be required to learn chemistry to a similar extent that chemists would need to doing ML (especially at research level)

I don’t understand what you mean. Here’s how many applied ML papers work: create a new dataset for a novel problem, download a PyTorch model, point model at dataset directory. Is it novel? By construction. Is the ML technique novel? No.

I actually meant exactly this. People apply ML to their domains and say thay that this is similar to having a chemist with a PhD.

Re: Why did Google Brain exist?

#306

Post by rerx »

vl wrote: post #265
why_only_15 wrote: post #261

Earlier quoted context omitted.

You are not correct about TPUs being drastically better for GPUs at this. If you look at public benchmarks, both have a similar cost per hardware flop ($0.88/hr for 312tflops A100 on GCP, $0.97/hr for 275tflops TPUv4) and both achieve similar model flop:hardware flop ratios (40-60%). An existence proof that GPU mega-clusters are possible is that GPT-4 cost ~$100m over ~3 months, so ~100m a100-hours / (3 months * 30 d…

I assume/suspect internally Google has v5 already. The thing about TPU clusters that they have hyper-torus optical interconnect between TPUs. This allows for extremely efficient weight updates. To replicate this with A100s you need very custom hardware/software deployment. But to be fair, I don’t know what is latest and greatest available from NVidia or other clouds in this area right now. EDIT: Looks like NVidia has…

External NVSwitch (for more than 8 GPUs) isn't available for purchase yet. But even now you typically buy your A100 or H100 training GPUs in servers where each GPU has an individual 400 Gbps Mellanox networking adapter, then connect them in a sophisticated switched network that provides full throughput between all GPUs of the cluster. The deployment is not that custom, typically you would let Nvidia and your vendors implement their reference design for you: https://www.nvidia.com/en-us/data-center/dgx-superpod/ Then run an open-source software stack on top of CUDA etc.

I don't know how well the TPU hyper-torus interconnect performs, but the networking topology seems to be less general than switched NVLink or InfiniBand.

Re: Why did Google Brain exist?

#307

Post by rerx »

vl wrote: post #248
sytelus wrote: post #238

Earlier quoted context omitted.

NVidia A100 DGX Superpod is equivalent to exaflop TPU pod. No?

Perhaps NVidia is close now. A bit hard to say without specific hardware info. Google’s were already available 5-6 years ago. And probably current versions are even faster. They have super fast optical interconnects in torus or hyper-torus configuration that allow synchronous weight updates on 1k+ TPUs. This leads to dramatically lower training times and less noise, which leads to better-performing models. I.e. you c…

GPU cluster scaling has come a long way. Just check out the scaling plot here: https://github.com/NVIDIA/Megatron-LM

Re: Why did Google Brain exist?

#308

Post by selimthegrim »

kergonath wrote: post #293
selimthegrim wrote: post #237

Earlier quoted context omitted.

What sort of problems are you trying to solve?

Lots of things, but mostly things like designing software architectures for massively parallel codes implementing fancy physics or coming up with ML models to predict complicated properties (things like wear, radiation, or corrosion resistance for which we just do not have any comprehensive model) or explore humongous problem spaces (things like predicting simple properties of very complex materials with 5 to 8 eleme…

Ha well if you’re looking for a solid state physics PhD I’m on the market ;)

Re: Why did Google Brain exist?

#309

Post by aer_aea »

mirker wrote: post #266
throwaway2037 wrote: post #242

Earlier quoted context omitted.

In my experience, people with degrees in math, physics, chemistry, biology/medicine, astronomy, computer science have higher/stronger intellectual rigor that can be re-applied to other fields, e.g., finance. Also, those areas of study are much harder than economics. Partly, it is a self-selection process. Yes, there are some with economics degrees whom are very bright, but they probably could have majored in any of t…

The story I’ve heard is the economics undergrads can’t get into economics grad school. This is just a rumor but the sentiment is that undergrads get taught a watered down version of economics theory. Economics theory is potentially very technical and includes game theory and proofs. Even in CS, undergrads take intro theory courses and “bottom out” in their math skills, even though grad-level CS gets much more difficu…

This is somewhat true - economics undergrads do get into grad school, though math+econ or an econ degree with lots of math is better. You can just the check the difference between the textbooks for undergrad and graduate economics, eg Principles of Economics by Mankiw (undergrad) and Mas Collel (grad).

Re: Why did Google Brain exist?

#310

Post by DoughnutHole »

deltree7 wrote: post #280
anonymouskimmer wrote: post #58

Earlier quoted context omitted.

> Second, there is a small chance you're absolutely right, and you've just obviated a whole field from first principles. Mostly when I read about things like this happening, it's happening to a formerly intractable problem in mathematics. Do you have examples outside of math?

LLMs weren't invented by Naom Chomsky but Math/CS guy with a simple "Attention is all you need"

Creating a good chatbot was never a goal of linguistics, and I don't see what insights into human language have been derived from LLMs.
Post reply on HN