Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Google employees who created transformers

wired.com

Post reply on HN
Print view View original

241–250 of 258 posts

Re: The Google employees who created transformers

#241

Post by Oioioioiio »

hn_throwaway_99 wrote: post #229
a_wild_dandan wrote: post #225

Earlier quoted context omitted.

This is an uncharitable and oddly dismissive take (i.e. perfect for HN, I suppose). Today's incredible state-of-the-art does not exist without the transformer architecture. Transformers aren't merely some lucky passengers riding the coattails of compute scale. If they were, then the ChatGPT app which set the world ablaze would've instead been called ChatMLP, or ChatCNN. But it's not. And in 2024 we still have no comp…

Question for you, as someone relatively new to the world of AI (well, not exactly new - I took many courses in AI, including neural networks, but in the late 90s... the world is just a tad different now!) Is there any good summary of the history of AI/deep learning from, say, late 00s/2010 to the present? I think learning some of this history would really help be better understand how we ended up at the current state…

Not sure if this is wayy to off of what you ask: https://www.youtube.com/watch?v=OFS90-FX6pg

Re: The Google employees who created transformers

#242

Post by crowcroft »

ajcp wrote: post #227
crowcroft wrote: post #194

Earlier quoted context omitted.

I know everyone cites this as the innovators dilemma, but so far the evidence suggests this isn't true. ChatGPT has been around for a while now, and it hasn't led to a collapse in Google's search revenue, and in fact now Google is rushing to roll out their version instead of trying to entrench search. A famous example is the iPhone killing the iPod, and it took around 3 and a half years for the iPod to really collaps…

> ChatGPT has been around for a while now ChatGPT has been around for less that 15 months[1]. In what version of reality is that "a while now"? Your iPhone/iPod timeline is also off by about 9 years[2]. 1. https://en.wikipedia.org/wiki/ChatGPT 2. https://en.wikipedia.org/wiki/IPod_Shuffle

'A while now' being long enough that I might expect some notable impact on Google's business by now. I do agree it could just be too early.

My iPhone/iPod timeline is referring to the time it took for the iPod's sales to be severely impacted [1]. I'm not disputing that the iPod continued to exist for a long time, the iPod Touch wasn't discontinued until 2022, but it was a pretty meaningless part of Apple's business by that point.

1. https://i.insider.com/597b68f2b50ab162018b466b?width=1300&fo...

Re: The Google employees who created transformers

#243

Post by sharadov »

cameldrv wrote: post #138
svnt wrote: post #71

Earlier quoted context omitted.

First generation immigrants are still a tiny minority of the population. The fact that the entire team consists effectively of first generation immigrants says something, probably both about higher education and American culture.

One thing is that getting a PhD is a good way to get into the U.S. As a foreigner, many visa and job opportunities open up to you with the PhD. For an American, it's less of a good deal. Once you have the PhD, you make somewhat more money, but you're trading that for 6 years of hard work and very low pay. The numbers aren't favorable -- you have to love the topic for it to make any sense. As a result, U.S. PhD progra…

A PHD is a faster path to a green card as well - especially for people born in China and India where the green card backlog is decades long.

Re: The Google employees who created transformers

#244

Post by autokad »

flamedoge wrote: post #231
autokad wrote: post #201

Earlier quoted context omitted.

maybe its too early. I almost never google search anymore, a lot of my friends do the same. Kind of like after I was using google for years, lots and lots of people were still using sites like ask jeaves, but the writing was on the wall

Maybe. I can't be the only one who gets irked by the slow, typing-out response from copilot/gpt when google gets me instant results. Near instant or a lot faster result might convince me to use whatever gpt search

honestly I use the free version and the text is really fast. Lets take a look at a typical question I may send to chat gpt:

>I have some tasks, some of them maintenance, some of them relaxation that I do >before bed. For example, maintenance: >brush teeth, floss, water floss, put on nose strips, take prylosec. >For relaxation: >stretch, massage gun, ASMR, music, tiger balm, salt bath, movie, tea, nasal >clean, moisturize. >can you suggest some others tasks?

Google can't even handle this. if you try, the entire page is filled with Videos: ... with 4 video thumbnails from youtube on how to use a water pic.

So I scroll off down (by now chat gpt already has a response) half a page on 'people also ask' with 4 questions about water pics, keep scrolling,

finally search results, and all of them are about ... water pics/flossers.

Re: The Google employees who created transformers

#245

Post by empath-nirvana »

crowcroft wrote: post #242
ajcp wrote: post #227

Earlier quoted context omitted.

> ChatGPT has been around for a while now ChatGPT has been around for less that 15 months[1]. In what version of reality is that "a while now"? Your iPhone/iPod timeline is also off by about 9 years[2]. 1. https://en.wikipedia.org/wiki/ChatGPT 2. https://en.wikipedia.org/wiki/IPod_Shuffle

'A while now' being long enough that I might expect some notable impact on Google's business by now. I do agree it could just be too early. My iPhone/iPod timeline is referring to the time it took for the iPod's sales to be severely impacted [1]. I'm not disputing that the iPod continued to exist for a long time, the iPod Touch wasn't discontinued until 2022, but it was a pretty meaningless part of Apple's business b…

Blackberry chugged along with record profits for a couple of years after the iphone had already made them irrelevant.

Re: The Google employees who created transformers

#246

Post by cooljoseph »

a_wild_dandan wrote: post #225
freddealmeida wrote: post #100

Attention models? Attention existed before those papers. What they did was show that it was enough to predict next word sequences in a certain context. I'm certain they didn't realize what they found. We used this frame work in 2018 and it gave us wildly unusual behavior (but really fun) and we tried to solve it (really looking for HF capability more than RL) but we didn't see what another group found: that scale in…

This is an uncharitable and oddly dismissive take (i.e. perfect for HN, I suppose). Today's incredible state-of-the-art does not exist without the transformer architecture. Transformers aren't merely some lucky passengers riding the coattails of compute scale. If they were, then the ChatGPT app which set the world ablaze would've instead been called ChatMLP, or ChatCNN. But it's not. And in 2024 we still have no comp…

> And in 2024 we still have no competing NLP architecture.

No, we do. State space models are both faster and scale just as well. E.g., RWKV and Mamba.

> Transformers aren't merely some lucky passengers riding the coattails of compute scale.

Err... they are, though. They were just the type of model the right researchers were already using at the time, probably for translating between natural languages.

Re: The Google employees who created transformers

#247

Post by yeshellohi »

victor106 wrote: post #232

> The group is also culturally diverse. Six of the eight authors were born outside the United States; the other two are children of two green-card-carrying Germans who were temporarily in California and a first-generation American whose family had fled persecution, respectively. As much as I think the America has lot of things it needs to fix, there is no other country on earth this would be possible. That's just a f…

> there is no other country on earth this would be possible. That's just a fact

I don’t think this the case. If anything the US makes life very hard for even high-skilled work-based immigrants. Many countries have a higher % of foreign born residents than the US (Singapore, Australia, Germany, Canada)

I myself used to work at Google UK and my own team was 100% foreign born engineers from every continent.

Re: The Google employees who created transformers

#248

Post by malshe »

victor106 wrote: post #232

> The group is also culturally diverse. Six of the eight authors were born outside the United States; the other two are children of two green-card-carrying Germans who were temporarily in California and a first-generation American whose family had fled persecution, respectively. As much as I think the America has lot of things it needs to fix, there is no other country on earth this would be possible. That's just a f…

I agree. As someone who has lived or worked in Asia, Europe, and the US, I always had the most diverse group of colleagues in the US.

Re: The Google employees who created transformers

#249

Post by goertzen »

a_wild_dandan wrote: post #225
freddealmeida wrote: post #100

Attention models? Attention existed before those papers. What they did was show that it was enough to predict next word sequences in a certain context. I'm certain they didn't realize what they found. We used this frame work in 2018 and it gave us wildly unusual behavior (but really fun) and we tried to solve it (really looking for HF capability more than RL) but we didn't see what another group found: that scale in…

This is an uncharitable and oddly dismissive take (i.e. perfect for HN, I suppose). Today's incredible state-of-the-art does not exist without the transformer architecture. Transformers aren't merely some lucky passengers riding the coattails of compute scale. If they were, then the ChatGPT app which set the world ablaze would've instead been called ChatMLP, or ChatCNN. But it's not. And in 2024 we still have no comp…

It’s reply’s like this that make HN a good place.

Re: The Google employees who created transformers

#250

Post by maxcoder4 »

srackey wrote: post #207
calf wrote: post #170

Earlier quoted context omitted.

It is intellectually dishonest to call the military-industrial complex merely "customers that some Google employees didn't like". Humans are more than their companies' mission, and your freedom to exercise your political views on how companies should be run inherently relies on this principle, too. So your argument is fundamentally a hypocrites projection.

“ Humans are more than their companies' mission” Not when they’re on the clock. Besides, many of the Google employees who are against defense projects are against them because those projects actively target their home countries . Much harder problem to fix!

I'm not sure if you're being sarcastic or not. Obviously you don't get to ignore your morals and values just because you're currently employed by someone?
Post reply on HN