Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Google employees who created transformers

wired.com

Post reply on HN
Print view View original

201–210 of 258 posts

Re: The Google employees who created transformers

#201

Post by autokad »

crowcroft wrote: post #194
empath-nirvana wrote: post #158

Earlier quoted context omitted.

The problem is that chatting with an LLM is extremely disruptive to their business model and it's difficult for them to productize without killing the golden goose.

I know everyone cites this as the innovators dilemma, but so far the evidence suggests this isn't true. ChatGPT has been around for a while now, and it hasn't led to a collapse in Google's search revenue, and in fact now Google is rushing to roll out their version instead of trying to entrench search. A famous example is the iPhone killing the iPod, and it took around 3 and a half years for the iPod to really collaps…

maybe its too early. I almost never google search anymore, a lot of my friends do the same. Kind of like after I was using google for years, lots and lots of people were still using sites like ask jeaves, but the writing was on the wall

Re: The Google employees who created transformers

#202

Post by lokar »

changoplatanero wrote: post #137
koala_man wrote: post #61

In Google's heyday, around 2014, I was talking with Uszkoreit about a possible role on his then NLP team. I asked "What would you do if you had an unlimited budget?" He simply said, "I do"

I shared an office with Uszkoreit when I was a phd intern and I always admired him for him having dropped out of his phd program.

The HR system has (had?) “abd” as an education level/degree between MS and PhD

Re: The Google employees who created transformers

#203

Post by codethatwerks »

crowcroft wrote: post #194
empath-nirvana wrote: post #158

Earlier quoted context omitted.

The problem is that chatting with an LLM is extremely disruptive to their business model and it's difficult for them to productize without killing the golden goose.

I know everyone cites this as the innovators dilemma, but so far the evidence suggests this isn't true. ChatGPT has been around for a while now, and it hasn't led to a collapse in Google's search revenue, and in fact now Google is rushing to roll out their version instead of trying to entrench search. A famous example is the iPhone killing the iPod, and it took around 3 and a half years for the iPod to really collaps…

Google is big enough to avoid innovators dilemma. You just create a new company and leave it alone.

I reckon it is more that you needed special circumstances and series of events to become OpenAI. Not just cash and smart people.

Re: The Google employees who created transformers

#204

Post by codethatwerks »

bitcharmer wrote: post #152
dpflan wrote: post #119

Earlier quoted context omitted.

What is OpenAI today? Can you elaborate? Google is a varied $trillion company. OpenAI sells access to large generative models.

Sure, Google is worth more but they essentially reduced themselves to an ad business. They aren't focused on innovation as much as before. Shareholder value is the new god.

Shareholder value is fine. Share seller value is what is targeted.

Re: The Google employees who created transformers

#205

Post by smolder »

Jensson wrote: post #176
bigcat12345678 wrote: post #114

Earlier quoted context omitted.

Hahaha... I worked at Borg The quota system can kick in at whatever time the limits are reached. And GPUs are scattered across borg cells, limiting the ceiling. That's why XBorg was created so that a global search among all Borg cells for researchers. And data center Capex is around 5 billion each year. Google makes hundres of billions of revenue each year. You are asking what people would do in impossible situation.…

> I cannot even understand what I do stands for in the context of your question That he had a higher budget than he knew what to do with. When I worked at Google I could bring up thousands of workers doing big tasks for hours without issue whenever I wanted, for me that was the same as being infinite since I never needed more, and that team didn't even have a particularly large budget. I can see a top ML team having…

I know a Google operations guy who has occasionally complained that the developers act like computing/network resources are infinite, so this made me chuckle.

Re: The Google employees who created transformers

#206

Post by lordswork »

sangnoir wrote: post #193
lordswork wrote: post #173

Earlier quoted context omitted.

Funnily enough, the same AI safety teams that held Google back from using large transformers in products are also largely responsible for the Gemini image generation debacle. It is tough to find the right balance though, because AI safety is not something you want to brush off.

I thought Google fired it's AI Ethicists a few years back and dismantled the team?

Ethics != Safety. Also, Google still has both afaik

Re: The Google employees who created transformers

#207

Post by srackey »

calf wrote: post #170
nemo44x wrote: post #77

Earlier quoted context omitted.

He was too distracted with all the other things going on under his watch. He spent too much time trying to convince rank and file that it's OK to do business with customers they don't like, trying to ensure arbitrary hiring goals were made, and appeasing the internal activists that led protests and pressured to exit people that didn't fall in line. This is why the CEO of Coinbase sent the memo a few years back statin…

It is intellectually dishonest to call the military-industrial complex merely "customers that some Google employees didn't like". Humans are more than their companies' mission, and your freedom to exercise your political views on how companies should be run inherently relies on this principle, too. So your argument is fundamentally a hypocrites projection.

“ Humans are more than their companies' mission”

Not when they’re on the clock.

Besides, many of the Google employees who are against defense projects are against them because those projects actively target their home countries. Much harder problem to fix!

Re: The Google employees who created transformers

#208

Post by OldMatey »

elevatedastalt wrote: post #64
koala_man wrote: post #61

In Google's heyday, around 2014, I was talking with Uszkoreit about a possible role on his then NLP team. I asked "What would you do if you had an unlimited budget?" He simply said, "I do"

Nice story, but Google's heyday was probably 10 years before that. By 2014 the decline had already started.

Agreed. I remember the town hall meeting where they announced the transition to being Alphabet. My manager was flying home from the US at the time. He left a Google employee and landed an Alphabet employee. I know it was probably meaningless in any real sense, but when they dropped the Dont Be Evil motto, it was a sign that the fun times were drawing to an end.

Re: The Google employees who created transformers

#209

Post by godelski »

freddealmeida wrote: post #100

Attention models? Attention existed before those papers. What they did was show that it was enough to predict next word sequences in a certain context. I'm certain they didn't realize what they found. We used this frame work in 2018 and it gave us wildly unusual behavior (but really fun) and we tried to solve it (really looking for HF capability more than RL) but we didn't see what another group found: that scale in…

This is a classic... This isn't the first time this piece has been posted before either. Here's one from FT last year[0] or Bloomberg[1]. You can find more. Google certainly played a major role, but it is too far to say they invented it or that they are the ones that created modern AI. Like Einstein said, shoulder of giants. And realistically, those giants are just a bunch of people in trench coats. Millions of researchers being unrecognized. I don't want to undermine the work of these researchers, but that doesn't mean we should also undermine the work of the many others (and thank god for the mathematicians who get no recognition and lay all the foundation for us).

And of course, a triggered Yann[2] (who is absolutely right).

But it is odd since it is actually a highly discussed topic, the history of attention. It's been discussed on HN many times before. And of course there's Lilian Weng's very famous blog post[3] that covers this in detail.

The word attention goes back well over a decade and even before Schmidhuber's usage. He has a reasonable claim but these things are always fuzzy and not exactly clear.

At least the article is more correct specifying Transformer rather than Attention, but even this is vague at best. FFormer (FFT-Transformer) was a early iteration and there were many variants. Do we call a transformer a residual attention mechanism with a residual feed forward? Can it be a convolution? There is no definitive definition but generally people mean DPMHA w/ skip layer + a processing network w/ skip layer. But this can be reflective of many architectures since every network can be decomposed into subnetworks. This even includes a 3 layer FFN (1 hidden layer).

Stories are nice, but I think it is bad to forget all the people who are contribution in less obvious ways. If a butterfly can cause a typhoon, then even a poor paper can contribute to a revolution.

[0] https://www.ft.com/content/37bb01af-ee46-4483-982f-ef3921436...

[1] https://www.bloomberg.com/opinion/features/2023-07-13/ex-goo...

[2] https://twitter.com/ylecun/status/1770471957617836138

[3] https://lilianweng.github.io/posts/2018-06-24-attention/

Re: The Google employees who created transformers

#210

Post by solomonb »

empath-nirvana wrote: post #158
Thaxll wrote: post #93

It's pretty crazy to think that Google is not OpenAI today, they had deep mind and an army of PHDs early on.

The problem is that chatting with an LLM is extremely disruptive to their business model and it's difficult for them to productize without killing the golden goose.

Sounds like Xerox all over again.
Post reply on HN