Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why did Google Brain exist?

moderndescartes.com

Post reply on HN
Print view View original

281–290 of 328 posts

Re: Why did Google Brain exist?

#281

Post by gradstudent »

m00x wrote: post #239
anonylizard wrote: post #193

Earlier quoted context omitted.

Well, where exactly is this progress? Where is Google's answer to GPT-4? Why weren't the 'researchers and developers' making a GPT-4 equivalent? Turns out you sometimes you need a top down, centralised vision to execute on projects. When the goal is undefined, you can allow researchers to run free and explore, now its full on wartime, with clear goals (make GPT-5,6,7....).

GPT-4 and chatGPT are developed from mostly Google papers and Facebook code/frameworks. OpenAI just focused on making it a great product.

A car is basically 4 wheels and a combustion engine. Henry Ford just focused on making it a great product.

(In case it's not clear, I think you might be underestimating the size of the subsequent contributions)

Re: Why did Google Brain exist?

#282

Post by kenjackson »

_delirium wrote: post #278
kenjackson wrote: post #270

Earlier quoted context omitted.

That’s why I said practically speaking. And realistically I don’t think you can even generously consider anything prior to perceptrons.

> realistically I don’t think you can even generously consider anything prior to perceptrons I mean, sure, but for the practical reason that perceptrons (proposed 1943, implemented 1958) are about as old as digital computers, and ML as a field is tied up pretty strongly with the existence of computers.

I wasn’t comparing ML to digital computers though — but rather chemistry. Theory of computation and algorithms (bedrocks of CS) far predate digital computers.

Re: Why did Google Brain exist?

#283

Post by kenjackson »

PartiallyTyped wrote: post #277
kenjackson wrote: post #270

Earlier quoted context omitted.

That’s why I said practically speaking. And realistically I don’t think you can even generously consider anything prior to perceptrons.

ML is so much more than DL / NNs. I recommend that you open a book like Bishop or ESL, the overwhelming majority of ML has been non DL going back to the 70s.

Most of that I learned doing non-ML stats work. But regardless, my point was the history of it is far less than chemistry. Let’s say ML started at the turn of the 1990s to be generous. Still much younger than chemistry.

Re: Why did Google Brain exist?

#284

Post by angarg12 »

dr_dshiv wrote: post #7

Lots of great insight. Here’s one: “Given the long timelines of a PhD program, the vast majority of early ML researchers were self-taught crossovers from other fields. This created the conditions for excellent interdisciplinary work to happen. This transitional anomaly is unfortunately mistaken by most people to be an inherent property of machine learning to upturn existing fields. It is not. Today, the vast majority…

Not so dramatic, but I have a traditional CS background and got into ML much later in my career.

Last year my company formalized the process to hire ML Engineers. The interview is the same format as the software engineer round, but with an extra theoretical ML round.

I've observed two distinct groups of candidates in the process. One are recent grads with PhDs in ML. Other are people from diverse backgrounds that happened to start working in ML in their current job. The first group tends to excel in the ML theory part, but flunk the leetcode style coding questions. The second group tends to do better in coding, but do worse in the theory part. This is exactly what the process has been designed to do.

I am very put off from looking for ML Engineer positions in other companies if they follow a similar process. I know I would fail the interview for my current job. They could get away with it for a while, but I doubt it's sustainability or desirability in the long term.

Re: Why did Google Brain exist?

#285

Post by ramraj07 »

throwaway2037 wrote: post #244
ramraj07 wrote: post #232

Earlier quoted context omitted.

I’m pretty sure the author is implying that the new crop of ML PhDs are just not a smart group of people - at least the level of intelligence required to do truly transformative things with ML in any field. I think what you’re saying is a commonly found attitude that relates to this topic: it’s pretty limiting to think a cursory knowledge of a field is sufficient to go change it. That’s likely why most “use ML to sol…

AlphaFold succeed because the ML engineers truly understood the fundamental tenets of the topic and exploited it What makes you think it wasn't chemists or biologists who learned enough ML to solve the problem?

Because the chemists and biologists have been doing that for decades without success.

Re: Why did Google Brain exist?

#286

Post by pyrale »

cauthon wrote: post #234
epolanski wrote: post #182

Earlier quoted context omitted.

It's the same reason analysts come from math rather than economy degrees. You can teach a mathematician what he needs to know about finance, you can hardly do the opposite.

That’s the opposite of what OP is observing. It’s easier to teach a domain expert “good enough” quantitative and technical skills than to teach a pure quant “good enough” domain expertise (and the corresponding intuition)

and it's easier yet to find one of each that are willing to collaborate.

Re: Why did Google Brain exist?

#287

Post by stolsvik »

cubefox wrote: post #225
oofbey wrote: post #216

Earlier quoted context omitted.

They are behind because what they have released sucks. Have you tried Bard? It's dumb. Like you're talking to some 20th century gimmick dumb. GPT4 is far from perfect, but when it makes mistakes and you point them out, it understands and tries to adapt. Bard just repeats itself saying the same stupid things like a casette-tape answering machine. If you ask a googler about this, they typically assume GPT is just as st…

Bard uses a smaller model currently, which was announced before release. > We’re releasing it initially with our lightweight model version of LaMDA. This much smaller model requires significantly less computing power, enabling us to scale to more users, allowing for more feedback. [1] > Bard is powered by a research large language model (LLM), specifically a lightweight and optimized version of LaMDA, and will be upd…

”We’re releasing something useless and uninteresting, so that we can get more user feedback”

Is it only me that sees a problem here?

Re: Why did Google Brain exist?

#288

Post by AaronFriel »

vl wrote: post #265
why_only_15 wrote: post #261

Earlier quoted context omitted.

You are not correct about TPUs being drastically better for GPUs at this. If you look at public benchmarks, both have a similar cost per hardware flop ($0.88/hr for 312tflops A100 on GCP, $0.97/hr for 275tflops TPUv4) and both achieve similar model flop:hardware flop ratios (40-60%). An existence proof that GPU mega-clusters are possible is that GPT-4 cost ~$100m over ~3 months, so ~100m a100-hours / (3 months * 30 d…

I assume/suspect internally Google has v5 already. The thing about TPU clusters that they have hyper-torus optical interconnect between TPUs. This allows for extremely efficient weight updates. To replicate this with A100s you need very custom hardware/software deployment. But to be fair, I don’t know what is latest and greatest available from NVidia or other clouds in this area right now. EDIT: Looks like NVidia has…

NVidia acquired Mellanox in 2019 in order to own the interconnect - exactly the custom hardware/software stack you thought needed.

Re: Why did Google Brain exist?

#289

Post by [deleted] »

cubefox wrote: post #225
oofbey wrote: post #216

Earlier quoted context omitted.

They are behind because what they have released sucks. Have you tried Bard? It's dumb. Like you're talking to some 20th century gimmick dumb. GPT4 is far from perfect, but when it makes mistakes and you point them out, it understands and tries to adapt. Bard just repeats itself saying the same stupid things like a casette-tape answering machine. If you ask a googler about this, they typically assume GPT is just as st…

Bard uses a smaller model currently, which was announced before release. > We’re releasing it initially with our lightweight model version of LaMDA. This much smaller model requires significantly less computing power, enabling us to scale to more users, allowing for more feedback. [1] > Bard is powered by a research large language model (LLM), specifically a lightweight and optimized version of LaMDA, and will be upd…

[deleted]

Re: Why did Google Brain exist?

#290

Post by aix1 »

jll29 wrote: post #156
khazhoux wrote: post #59

> I sat on it because I wasn’t sure of the optics of posting such an essay while employed by Google Brain. But then Google made my decision easier by laying me off in January. My severance check cleared... I'm really baffled by how people think it's OK to write public accounts of their previous ( and sometime current! ) employers' inner workings. This guy got paid a shitload of money to do work and to keep all intern…

I checked with some Google friends who told me their contract even makes it illegal to tell anyone that they work for Google (no joke). One side is what's in the papers you signed and the other side is to what extent the terms can be enforced. But you have a point in that it would be good professional practice to wait for a decade before disclosing internals especially when names of people are dropped...

> I checked with some Google friends who told me their contract even makes it illegal to tell anyone that they work for Google (no joke).

Which country are they in? Is their employment contract directly with Google, or with some other, independent company that provides services to Google?

(I've been employed by G in multiple jurisdictions and have never seen or heard of such a clause.)

Post reply on HN