Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Google employees who created transformers

wired.com

Post reply on HN
Print view View original

181–190 of 258 posts

Re: The Google employees who created transformers

#181

Post by tracerbulletx »

voiceblue wrote: post #122
freddealmeida wrote: post #100

Attention models? Attention existed before those papers. What they did was show that it was enough to predict next word sequences in a certain context. I'm certain they didn't realize what they found. We used this frame work in 2018 and it gave us wildly unusual behavior (but really fun) and we tried to solve it (really looking for HF capability more than RL) but we didn't see what another group found: that scale in…

> but we didn't see what another group found: that scale in compute with simple algorithms were just better The bitter lesson [0] strikes again. [0] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

I kind of wonder if the reason this seems to be true is that emergent systems are just able to go a lot farther into much more complex design spaces than any system a human mind is capable of constructing.

Re: The Google employees who created transformers

#182

Post by pixl97 »

xNeil wrote: post #165
pixl97 wrote: post #106

Earlier quoted context omitted.

I'm not 100% sure I agree... Google is an ad company at the end of the day. Google is still making obscene amount of money with ads. Currently Gen AI is a massive expense in training and running and is only looking like it may harm future ad revenue (yet to be seen). Meanwhile OpenAI has not surpassed that critical threshold where they dominate the market with a moat and become a trillion dollar company. It is typica…

It's not about just ad revenue though, is it? Many thousands, if not millions of sites rely entirely on Google Search for their revenue. You cannot simply replace all of them with a chatbot - there are massive consequences to such decisions.

>It's not about just ad revenue though, is it? Many thousands, if not millions of sites rely entirely on Google Search for their revenue.

I mean these are kind of tangled together. Chatbots actively dilute real sites from what we are seeing, by feeding back into their fake into googles index in order to capture some of the ad revenue. This leads to the "Gee Google is really sucking" meme we see more and more often. The point is, Google had an absolute cash cow and the march towards GenAI AGI threatens to interrupt that model just as much as it promises to to make the AGI winner insanely rich.

Re: The Google employees who created transformers

#183

Post by fnordpiglet »

freddealmeida wrote: post #100

Attention models? Attention existed before those papers. What they did was show that it was enough to predict next word sequences in a certain context. I'm certain they didn't realize what they found. We used this frame work in 2018 and it gave us wildly unusual behavior (but really fun) and we tried to solve it (really looking for HF capability more than RL) but we didn't see what another group found: that scale in…

In fairness, an article that’s about “the Google engineers who incrementally advanced AI” wouldn’t sell many advertisements.

Re: The Google employees who created transformers

#184

Post by bigcat12345678 »

Jensson wrote: post #176
bigcat12345678 wrote: post #114

Earlier quoted context omitted.

Hahaha... I worked at Borg The quota system can kick in at whatever time the limits are reached. And GPUs are scattered across borg cells, limiting the ceiling. That's why XBorg was created so that a global search among all Borg cells for researchers. And data center Capex is around 5 billion each year. Google makes hundres of billions of revenue each year. You are asking what people would do in impossible situation.…

> I cannot even understand what I do stands for in the context of your question That he had a higher budget than he knew what to do with. When I worked at Google I could bring up thousands of workers doing big tasks for hours without issue whenever I wanted, for me that was the same as being infinite since I never needed more, and that team didn't even have a particularly large budget. I can see a top ML team having…

Sure, your explanation just matches what I was imaging. Neither the question nor the answer made any sense.

Re: The Google employees who created transformers

#186

Post by stonogo »

sharemywin wrote: post #162
com2kid wrote: post #145

Earlier quoted context omitted.

> It is typically very hard for a company to change what they actually do Microsoft started out selling BASIC runtimes. Then they moved into operating systems. Then they moved into Cloud. Satya seems to be putting his money where his mouth is and working hard to transition the company to AI now. Apple has likewise undergone multiplet transformations over the decades. However Google has the unique problem that their m…

Office was the cash cow compared to windows.

Microsoft effectively controlling the PC market as a platform to sell Excel is akin to Google controlling web standards as a platform to sell ads.

Re: The Google employees who created transformers

#187

Post by vlakreeh »

mzs wrote: post #148
ralph84 wrote: post #68

And none of them still work for Google. It’s truly baffling that Google’s CEO still has a job after how badly he fumbled on AI.

Google doesn't have to try to be the best at AI now, it can just buy whoever is the winner later - a lesson learned from google video.

If that winner turns out to be OpenAI and they attempt to buy them it'll be blocked by regulators instantly.

Re: The Google employees who created transformers

#188

Post by nojvek »

Google is an Ads company ad the end of the day. They are making 10% revenue from GCloud but still bulk of their profits are from Ads.

If they were threatened, Google could have easily owned the cloud. They were the first to a million servers and exabyte storage scale. Their internal infra is top notch.

Under no circumstances can they damage their cash cow. They have already shown to be UX hostile by making Ads look like results, and Youtube forcing more ads onto users.

The biggest breakthrough an AI researcher can have at Google is to get more ad clicks across their empire.

Google management knows this. Without ad revenue, they are fucked.

That is my biggest fear. That with AI we are able to exploit the human mind for dopamine addiction through screens.

Meta has shown that their algorithm can make the feeds more addictive (higher engagement). This results in more ads.

With Insta Reels, Youtube, TikTok e.t.c "Digital Heroin" keeps on getting better year after year.

Re: The Google employees who created transformers

#189

Post by michael1999 »

com2kid wrote: post #145
pixl97 wrote: post #106

Earlier quoted context omitted.

I'm not 100% sure I agree... Google is an ad company at the end of the day. Google is still making obscene amount of money with ads. Currently Gen AI is a massive expense in training and running and is only looking like it may harm future ad revenue (yet to be seen). Meanwhile OpenAI has not surpassed that critical threshold where they dominate the market with a moat and become a trillion dollar company. It is typica…

> It is typically very hard for a company to change what they actually do Microsoft started out selling BASIC runtimes. Then they moved into operating systems. Then they moved into Cloud. Satya seems to be putting his money where his mouth is and working hard to transition the company to AI now. Apple has likewise undergone multiplet transformations over the decades. However Google has the unique problem that their m…

Remember that Microsoft spent 10 years fighting the open internet, and only launched the cloud much later. The Windows business was allergic to the web-model, and they fought it tooth and nail.
Post reply on HN