Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Google employees who created transformers

wired.com

Post reply on HN
Print view View original

221–230 of 258 posts

Re: The Google employees who created transformers

#221

Post by int_19h »

lordswork wrote: post #173
pigscantfly wrote: post #108

Earlier quoted context omitted.

It would have gone much better for Google if the Brain team had been permitted to apply their work, but they were repeatedly blocked on stated grounds of AI safety and worries about impacting existing business lines through negative PR. I think this is probably the biggest missed business opportunity of the past decade, and much of the blame for losing key talent and Google's head start in LLMs ultimately resides wit…

Funnily enough, the same AI safety teams that held Google back from using large transformers in products are also largely responsible for the Gemini image generation debacle. It is tough to find the right balance though, because AI safety is not something you want to brush off.

> AI safety is not something you want to brush off

It really depends on what exactly is meant by "safety", because this word is used in several different (and largely unrelated) meanings in this context.

The actual value of the kind of "safety" that led to the Gemini debacle is very unclear to me.

Re: The Google employees who created transformers

#222

Post by int_19h »

tsunamifury wrote: post #56
jimbob45 wrote: post #50

Earlier quoted context omitted.

How is that not immediate grounds for his termination? The board should be canned too for allowing such an obvious charlatan to continue ruining the company.

Two judgements were made to my knowledge: 1) Google enjoyed significant market status at the time and a leap forward like seemingly semi conscious AI in 2019 would be seen as terrifying. Consumer sentiment would go from positive to “Google is winning to hard and making Frankensteins monster” 2) it didn’t weave well into googles current product offering and in fact disrupted it in ways that would confuse the user. It…

This is a good illustration of why the common sentiment that "capitalism drives innovation" is ... not exactly accurate.

Re: The Google employees who created transformers

#223

Post by fennecbutt »

whatever1 wrote: post #18

I am all in for allowing tax free bloated RnD departments just with the hope that once in a decade idea will propell the overall economy. The marvel of modern computing is a result of RnD bloat that was done without immediate impact to the bottomlines of their own companies.

That would make sense, if companies and executives paid their damn taxes in the first place.

Re: The Google employees who created transformers

#224

Post by HarHarVeryFunny »

mindcrime wrote: post #135
HarHarVeryFunny wrote: post #102

Earlier quoted context omitted.

The phrasing kinda makes sense to me. Consider "modern" to mean NN/connectionist vs GOFAI AI attempts like CYC or SOAR. I guess it depends on how you define "AI", and whether you accept the media's labelling of anything ML-related as AI. To me, LLMs are the first thing deserving to be called AI, and other NNs like CNNs better just called ML since there is no intelligence there.

Consider "modern" to mean NN/connectionist vs GOFAI AI attempts like CYC or SOAR. I dunno. The earliest research into what we now call "neural networks" dates back to at least the 1950's (Frank Rosenblatt and the Perceptron) and arguably into the 1940's (Warren McCulloch and Walter Pitts and the TLU "neuron"). And depending on how generous one is with their interpretation of certain things, arguments have been made t…

Sure, the history of NNs goes back a while, but nobody was attempting to build AI out of perceptrons (single layer), which were famously criticized as not being able to even implement an XOR function.

The modern era of NNs started with being able to train multilayer neural nets using backprop, but the ability to train NNs large enough to actually be useful for complex things AI research, can arguably be dated to the 2012 Imagenet competition when Geoff Hinton's team repurposed GPUs to train AlexNet.

But, AlexNet was just a CNN, a classifier, which IMO is better just considered as ML, not AI, so if we're looking for the first AI in this post-GOFAI world of NN-based experimentation, then it seems we have to give the nod to transformer-based LLMs.

Re: The Google employees who created transformers

#225

Post by a_wild_dandan »

freddealmeida wrote: post #100

Attention models? Attention existed before those papers. What they did was show that it was enough to predict next word sequences in a certain context. I'm certain they didn't realize what they found. We used this frame work in 2018 and it gave us wildly unusual behavior (but really fun) and we tried to solve it (really looking for HF capability more than RL) but we didn't see what another group found: that scale in…

This is an uncharitable and oddly dismissive take (i.e. perfect for HN, I suppose).

Today's incredible state-of-the-art does not exist without the transformer architecture. Transformers aren't merely some lucky passengers riding the coattails of compute scale. If they were, then the ChatGPT app which set the world ablaze would've instead been called ChatMLP, or ChatCNN. But it's not. And in 2024 we still have no competing NLP architecture. Because the transformer is a genuinely profound, remarkable idea with remarkable properties (e.g. training parallelism). It's easy to downplay GPTs as a mostly derivative idea with the benefit of hindsight. I'm sure we'll perform the same revisionist history with state-space models, or whatever architecture eventually supplants transformers. Do GPTs build on prior work? Do other approaches and ideas deserve recognition? Yeah, obviously. Like...welcome to science. But the transformer's architects earned their praise -- including via this article -- which isn't some slight against everyone else, as if accolades were a zero-sum game. These 8 people changed our world and genuinely deserve the love!

Re: The Google employees who created transformers

#226

Post by shiroiushi »

samatman wrote: post #175
pksebben wrote: post #157

Earlier quoted context omitted.

The phrasing was "welcoming to immigrants", not "welcoming to the ever shrinking definition of good immigrants established by a bunch of octogenarian plutocrats". "Illegal" is a concept - it's not conflating to assume that it's not the bedrock of the way people think.

Illegal is a status more than it is a concept. Immigrating illegally is not the central definition of immigration, any more than shoplifting is the central definition of customer. America is much more welcoming of immigration, by which I mean legal immigration, than Japan or China. This is not in dispute. It is also, in practice, quite a bit more slack about illegal immigration than either of those countries. Althoug…

>America is much more welcoming of immigration, by which I mean legal immigration, than Japan or China. This is not in dispute.

It's not? It sounds like you know little about the world outside of America. Japan is stupidly easy to immigrate to: just get a job offer here at a place that sponsors your visa and it's pretty trivial to immigrate. Even better, if you have enough points, you can apply for permanent residence after 1 or 3 years, and the cost is trivial. In America, getting a Green Card is very difficult and costly, and depending on your national origin can be almost impossible. In Japan, there's no limits at all, per year or per country of origin, for work visas or PR. Of course, Japan is somewhat selective about who it wants to immigrate, but America is no different there, which is why there's such a huge debate about illegal immigration (in America it's not that hard; in an island country it's not so easy).

Re: The Google employees who created transformers

#227

Post by ajcp »

crowcroft wrote: post #194
empath-nirvana wrote: post #158

Earlier quoted context omitted.

The problem is that chatting with an LLM is extremely disruptive to their business model and it's difficult for them to productize without killing the golden goose.

I know everyone cites this as the innovators dilemma, but so far the evidence suggests this isn't true. ChatGPT has been around for a while now, and it hasn't led to a collapse in Google's search revenue, and in fact now Google is rushing to roll out their version instead of trying to entrench search. A famous example is the iPhone killing the iPod, and it took around 3 and a half years for the iPod to really collaps…

> ChatGPT has been around for a while now

ChatGPT has been around for less that 15 months[1]. In what version of reality is that "a while now"? Your iPhone/iPod timeline is also off by about 9 years[2].

1. https://en.wikipedia.org/wiki/ChatGPT 2. https://en.wikipedia.org/wiki/IPod_Shuffle

Re: The Google employees who created transformers

#228

Post by shiroiushi »

cmrdporcupine wrote: post #76
lelag wrote: post #66

Earlier quoted context omitted.

It does not matter. In the 80s, they owned the whole photography market, now they only exist as a shell of it's former self. By not pursuing this tech, they basically committed corporate suicide over the long run and they knew it. They knew very well, especially going into the 90's and early 2000 than their time making bank selling film was counted. But as long as the money was there, the chemical branch of the compa…

Kodak did plenty of great things with digital cameras in the early 2000s. Their CCD sensors from then are still famous and coveted in some older cameras. Go look at the price of a Leica M8 (from 2006) on eBay. The problem Kodak had is what the person you're replying to is alluding to. They got outcompeted because they were a photography company, not a digital hardware manufacturer. Companies like Sony or Canon did be…

>They got outcompeted because they were a photography company, not a digital hardware manufacturer. Companies like Sony or Canon did better because they were in the business of consumer electronics / hardware already.

Huh? This makes no sense. Sony was indeed a consumer electronics company at that time, but Canon was not: Canon was a camera manufacturer. They didn't get into electronics until later as cameras became digital: their earlier cameras were the all-mechanical kind. Sony was an electronics company that had to learn how to make cameras, but Canon was a camera company that had to learn how to make electronics. Kodak could have done the same.

Re: The Google employees who created transformers

#229

Post by hn_throwaway_99 »

a_wild_dandan wrote: post #225
freddealmeida wrote: post #100

Attention models? Attention existed before those papers. What they did was show that it was enough to predict next word sequences in a certain context. I'm certain they didn't realize what they found. We used this frame work in 2018 and it gave us wildly unusual behavior (but really fun) and we tried to solve it (really looking for HF capability more than RL) but we didn't see what another group found: that scale in…

This is an uncharitable and oddly dismissive take (i.e. perfect for HN, I suppose). Today's incredible state-of-the-art does not exist without the transformer architecture. Transformers aren't merely some lucky passengers riding the coattails of compute scale. If they were, then the ChatGPT app which set the world ablaze would've instead been called ChatMLP, or ChatCNN. But it's not. And in 2024 we still have no comp…

Question for you, as someone relatively new to the world of AI (well, not exactly new - I took many courses in AI, including neural networks, but in the late 90s... the world is just a tad different now!)

Is there any good summary of the history of AI/deep learning from, say, late 00s/2010 to the present? I think learning some of this history would really help be better understand how we ended up at the current state of the art.

Re: The Google employees who created transformers

#230

Post by dragonwriter »

shiroiushi wrote: post #228
cmrdporcupine wrote: post #76

Earlier quoted context omitted.

Kodak did plenty of great things with digital cameras in the early 2000s. Their CCD sensors from then are still famous and coveted in some older cameras. Go look at the price of a Leica M8 (from 2006) on eBay. The problem Kodak had is what the person you're replying to is alluding to. They got outcompeted because they were a photography company, not a digital hardware manufacturer. Companies like Sony or Canon did be…

>They got outcompeted because they were a photography company, not a digital hardware manufacturer. Companies like Sony or Canon did better because they were in the business of consumer electronics / hardware already. Huh? This makes no sense. Sony was indeed a consumer electronics company at that time, but Canon was not: Canon was a camera manufacturer. They didn't get into electronics until later as cameras became…

> Kodak could have done the same.

Kodak, while they incidentally made some cameras, were a film and film processing company that wasn’t great at cameras and wasn’t anything in electronics.

They were much worse positioned than either a camera company, or a consumer electronics company, for a pivot to the post-film photography world.

Post reply on HN