Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Google employees who created transformers

wired.com

Post reply on HN
Print view View original

121–130 of 258 posts

Re: The Google employees who created transformers

#121

Post by svnt »

pixl97 wrote: post #117
svnt wrote: post #71

Earlier quoted context omitted.

First generation immigrants are still a tiny minority of the population. The fact that the entire team consists effectively of first generation immigrants says something, probably both about higher education and American culture.

I think you have completely the wrong takeaway here... The US population is around 330 million. The world population is 8.1 billion people. What is that 4%? If you took a random sampling of people around the world, none of them would be Americans. You're going to need a lot more samples to find a trend. Yet when you turn around and look at success stories, a huge portion of this is going to occur in the US for multip…

I understand, but reality has to factor in — to get representative you would have to narrow your sample to English-speaking, narrow it to legal for long-term employment in the US, narrow it to having received at least an American-level higher education…

Re: The Google employees who created transformers

#122

Post by voiceblue »

freddealmeida wrote: post #100

Attention models? Attention existed before those papers. What they did was show that it was enough to predict next word sequences in a certain context. I'm certain they didn't realize what they found. We used this frame work in 2018 and it gave us wildly unusual behavior (but really fun) and we tried to solve it (really looking for HF capability more than RL) but we didn't see what another group found: that scale in…

> but we didn't see what another group found: that scale in compute with simple algorithms were just better

The bitter lesson [0] strikes again.

[0] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Re: The Google employees who created transformers

#123

Post by password54321 »

Thaxll wrote: post #93

It's pretty crazy to think that Google is not OpenAI today, they had deep mind and an army of PHDs early on.

Well at the time before Microsoft got involved it was sort of an unspoken rule among the AI community to be open and not release certain models to the public.

Re: The Google employees who created transformers

#124

Post by voiceblue »

calepayson wrote: post #116
freddealmeida wrote: post #100

Attention models? Attention existed before those papers. What they did was show that it was enough to predict next word sequences in a certain context. I'm certain they didn't realize what they found. We used this frame work in 2018 and it gave us wildly unusual behavior (but really fun) and we tried to solve it (really looking for HF capability more than RL) but we didn't see what another group found: that scale in…

I’m studying neuroscience but very interested in how ai works. I’ve read up on the old school but phrases like memory graph and energy minimization are new to me. What modern papers/articles would you recommend for folks who want to learn more?

For phrases, Google's TF glossary [0] is a good resource, but it does not cover certain subsets of AI (and more specifically, is mostly focused on TensorFlow).

[0] https://developers.google.com/machine-learning/glossary

Re: The Google employees who created transformers

#125

Post by HarHarVeryFunny »

ryandvm wrote: post #111
ralph84 wrote: post #68

And none of them still work for Google. It’s truly baffling that Google’s CEO still has a job after how badly he fumbled on AI.

I think the issue is that there is no future for a trustworthy AI that doesn't completely cannibalize their ad revenue cash cow. Like, who wants to use an AI that says things like, "... and that's why you should wear sunscreen outside. Speaking of skin protection, you should try Banana Boat's new Ultra 95 SPF sunscreen."

Yeah, but isn't the idea to cannibalize your own products before someone else does?!

In any case, consumer chatbots isn't the only way to sell the tech. Lot's of commercial use too.

I don't see why ads couldn't be integrated with chatbots too for that matter. There's no point serving them outside of a context where the user appears to be interested in a product/service, and in that case there are various ways ads could be displayed/inserted.

Re: The Google employees who created transformers

#126

Post by CuriouslyC »

pixl97 wrote: post #106
ralph84 wrote: post #68

And none of them still work for Google. It’s truly baffling that Google’s CEO still has a job after how badly he fumbled on AI.

I'm not 100% sure I agree... Google is an ad company at the end of the day. Google is still making obscene amount of money with ads. Currently Gen AI is a massive expense in training and running and is only looking like it may harm future ad revenue (yet to be seen). Meanwhile OpenAI has not surpassed that critical threshold where they dominate the market with a moat and become a trillion dollar company. It is typica…

Google is an AI/ML company that has shoehorned AI/ML researchers into working on ads to pay the bills.

Re: The Google employees who created transformers

#127

Post by IX-103 »

tyree731 wrote: post #29
photochemsyn wrote: post #27

> "Realistically, we could have had GPT-3 or even 3.5 probably in 2019, maybe 2020. The big question isn’t, did they see it? The question is, why didn’t we do anything with the fact that we had seen it? The answer is tricky.” The answer is that monopolies stifle technological innovation because one well-established part of their business (advertising-centric search) would be negatively impacted by an upstart branch (…

On the other hand, Alphabet's inability to deploy GPT-3 or GPT-3.5 has led to the possibility of its disruption, so anti-trust treatment may not be necessary.

Disrupted by a whom? Microsoft? Facebook? The company formerly known as Twitter? Even if one of these takes over we'd just be trading masters.

And that's ignoring how Alphabet's core business, Search, has little to fear from GPT-3 or GPT-3.5. These models are decent for a chatbot, but for anything where you want reliably correct answers they are lacking.

Re: The Google employees who created transformers

#128

Post by seatac76 »

ralph84 wrote: post #68

And none of them still work for Google. It’s truly baffling that Google’s CEO still has a job after how badly he fumbled on AI.

I mean if you had the pedigree these people have, the right move is to leave a big company and go start your own thing, there is simply a lot more upside there. You’d be foolish to stay with a big company, not sure what Google could have done here. Not everyone wants to be Jeff Dean.

Re: The Google employees who created transformers

#129

Post by throw0101d »

This 'conversation' with Geoffrey Hinton and Fei-Fei Li goes over a lot of the history of things (1h50m):

* https://www.youtube.com/watch?v=QWWgr2rN45o

* https://www.youtube.com/watch?v=E14IsFbAbpI ('mirror')

Goes over Hinton's history and why he went the direction he did with his research, as well as Li's efforts with ImageNet.

Re: The Google employees who created transformers

#130

Post by TimPC »

lelag wrote: post #60
mkl95 wrote: post #47

I thought level 4+ self-driving was going to be the face of "modern AI"... instead we got this thing

Why do you think transformers won't be key in making a level 4+ self-driving AI. It seems to me that Vision-capable multi-modal transformers could be the missing part: they can understand what is happening in the world in a deductive way. The Vision transformer is capable of predicting that a running child is likely to follow that rolling soccer ball. It is capable of deducting that a particular situation looks dange…

I think we've found repeatedly in self-driving that it's not enough to solve the problem in the normal case. You need an AI model that has good behaviors in the edge cases. For the most part it won't matter how good the vision models get, you're going to need similar models that can make the same predictions from LIDAR signals because the technology needs to work when the vision model goes crazy because of the reflectivity of a surface or other such edge cases where it completely misunderstands where the object is.
Post reply on HN