Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Google employees who created transformers

wired.com

Post reply on HN
Print view View original

231–240 of 258 posts

Re: The Google employees who created transformers

#231

Post by flamedoge »

autokad wrote: post #201
crowcroft wrote: post #194

Earlier quoted context omitted.

I know everyone cites this as the innovators dilemma, but so far the evidence suggests this isn't true. ChatGPT has been around for a while now, and it hasn't led to a collapse in Google's search revenue, and in fact now Google is rushing to roll out their version instead of trying to entrench search. A famous example is the iPhone killing the iPod, and it took around 3 and a half years for the iPod to really collaps…

maybe its too early. I almost never google search anymore, a lot of my friends do the same. Kind of like after I was using google for years, lots and lots of people were still using sites like ask jeaves, but the writing was on the wall

Maybe. I can't be the only one who gets irked by the slow, typing-out response from copilot/gpt when google gets me instant results. Near instant or a lot faster result might convince me to use whatever gpt search

Re: The Google employees who created transformers

#232

Post by victor106 »

> The group is also culturally diverse. Six of the eight authors were born outside the United States; the other two are children of two green-card-carrying Germans who were temporarily in California and a first-generation American whose family had fled persecution, respectively.

As much as I think the America has lot of things it needs to fix, there is no other country on earth this would be possible. That's just a fact.

Re: The Google employees who created transformers

#233

Post by bevekspldnw »

empath-nirvana wrote: post #158
Thaxll wrote: post #93

It's pretty crazy to think that Google is not OpenAI today, they had deep mind and an army of PHDs early on.

The problem is that chatting with an LLM is extremely disruptive to their business model and it's difficult for them to productize without killing the golden goose.

Ironically they killed the golden goose (the web) with display ads which incentivized low quality content to keep users scrolling. To put salt in their own wound, they are now indexing ever more genAI crap than they can handle and search is imploding.

Re: The Google employees who created transformers

#234

Post by bevekspldnw »

dpflan wrote: post #119
Thaxll wrote: post #93

It's pretty crazy to think that Google is not OpenAI today, they had deep mind and an army of PHDs early on.

What is OpenAI today? Can you elaborate? Google is a varied $trillion company. OpenAI sells access to large generative models.

Where would most people want to work is the question. This is all a talent game.

Re: The Google employees who created transformers

#235

Post by bevekspldnw »

xnx wrote: post #105

> Not only were the authors all Google employees, they also worked out of the same offices. Subtle plug for return-to-office. In-person face-to-face collaboration (with periods of solo uninterrupted deep focus) probably is the best technology we have for innovation.

“Office” does not have to mean open office. Academics all have offices with doors for a reason. I can’t stand open office, but private office in a building with other people is great.

Re: The Google employees who created transformers

#236

Post by j7ake »

freddealmeida wrote: post #100

Attention models? Attention existed before those papers. What they did was show that it was enough to predict next word sequences in a certain context. I'm certain they didn't realize what they found. We used this frame work in 2018 and it gave us wildly unusual behavior (but really fun) and we tried to solve it (really looking for HF capability more than RL) but we didn't see what another group found: that scale in…

They didn’t invent it, they advanced it. I agree.

However, some advances can have huge consequences to the field compared to others, even if at the technical level they appear comparable.

One example that comes to mind is CRISPR.

Re: The Google employees who created transformers

#237

Post by mike_hearn »

empath-nirvana wrote: post #158
Thaxll wrote: post #93

It's pretty crazy to think that Google is not OpenAI today, they had deep mind and an army of PHDs early on.

The problem is that chatting with an LLM is extremely disruptive to their business model and it's difficult for them to productize without killing the golden goose.

No, it's the other way around. Right now web search is "disruptive" to the LLM business model.

Google is a business fundamentally oriented around loss leaders. They make money on commercial queries where someone wants to buy something. They lose money when people search for facts or knowledge. The model works because people don't want to change their search engine every five minutes, so being good at the money losing queries means people will naturally use you for the money making queries too.

Right now LLMs take all the money losing queries and spend vast sums of investor capital on serving them, but they are useless for queries like [pizza near me] or [medical injury legal advice] or [holidays in the canary islands]. So right now I'd expect actually Google to do quite well. Their competitor is burning capital taking away all the stuff that they don't really want, whilst leaving them with the gold.

Now of course, that's today. The obvious direction for OpenAI to go in is finding ways to integrate ads with the free version of ChatGPT. But that's super hard. Building an ad network is hard. It takes a lot of time and effort, and it's really unclear what the product looks like there. Ads on web search is pretty obvious: the ads look like search results. What does an ad look like in a ChatGPT response?

Google have plenty of time to figure this out because OpenAI don't seem interested. They've apparently decided that all of ChatGPT is a loss leader for their API services. Whether that's financially sustainable or not is unclear, but it's also irrelevant. People still want to do commercial queries, they still want things like real images and maps and yes even ads (the way the ad auction works on Google is a very good ranking signal for many businesses). ChatGPT is still useless for them, so for now they will continue to leave money on the table where Google can keep picking it up.

Re: The Google employees who created transformers

#238

Post by Swiffy0 »

hn_throwaway_99 wrote: post #229
a_wild_dandan wrote: post #225

Earlier quoted context omitted.

This is an uncharitable and oddly dismissive take (i.e. perfect for HN, I suppose). Today's incredible state-of-the-art does not exist without the transformer architecture. Transformers aren't merely some lucky passengers riding the coattails of compute scale. If they were, then the ChatGPT app which set the world ablaze would've instead been called ChatMLP, or ChatCNN. But it's not. And in 2024 we still have no comp…

Question for you, as someone relatively new to the world of AI (well, not exactly new - I took many courses in AI, including neural networks, but in the late 90s... the world is just a tad different now!) Is there any good summary of the history of AI/deep learning from, say, late 00s/2010 to the present? I think learning some of this history would really help be better understand how we ended up at the current state…

I've been following a data science course called "The Data Science Course 2023: Complete Data Science Bootcamp" at Udemy.

The course starts all the way back from basic statistics and goes through things like linear regression and supposedly will arrive at neural networks and machine learning at some point.

So I don't know if something like this is exactly what you're looking for, but I think that, in general, if one wants to learn about (the history) AI, then it might be a good idea to start from statistics and learn about how we got from statistics to where we are now.

Re: The Google employees who created transformers

#239

Post by cmrdporcupine »

shiroiushi wrote: post #228
cmrdporcupine wrote: post #76

Earlier quoted context omitted.

Kodak did plenty of great things with digital cameras in the early 2000s. Their CCD sensors from then are still famous and coveted in some older cameras. Go look at the price of a Leica M8 (from 2006) on eBay. The problem Kodak had is what the person you're replying to is alluding to. They got outcompeted because they were a photography company, not a digital hardware manufacturer. Companies like Sony or Canon did be…

>They got outcompeted because they were a photography company, not a digital hardware manufacturer. Companies like Sony or Canon did better because they were in the business of consumer electronics / hardware already. Huh? This makes no sense. Sony was indeed a consumer electronics company at that time, but Canon was not: Canon was a camera manufacturer. They didn't get into electronics until later as cameras became…

What are you talking about? Canon was making copiers, printers, scanners, video cameras from the late 80s on.

So they were already making digital hardware and so presumably had the internal expertise on how to product manage that.

https://global.canon/en/corporate/history/04.html

Re: The Google employees who created transformers

#240

Post by steven »

xnx wrote: post #105

> Not only were the authors all Google employees, they also worked out of the same offices. Subtle plug for return-to-office. In-person face-to-face collaboration (with periods of solo uninterrupted deep focus) probably is the best technology we have for innovation.

Not so subtle!
Post reply on HN