Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Google employees who created transformers

wired.com

Post reply on HN
Print view View original

141–150 of 258 posts

Re: The Google employees who created transformers

#141

Post by open592 »

ryandvm wrote: post #111
ralph84 wrote: post #68

And none of them still work for Google. It’s truly baffling that Google’s CEO still has a job after how badly he fumbled on AI.

I think the issue is that there is no future for a trustworthy AI that doesn't completely cannibalize their ad revenue cash cow. Like, who wants to use an AI that says things like, "... and that's why you should wear sunscreen outside. Speaking of skin protection, you should try Banana Boat's new Ultra 95 SPF sunscreen."

"Based on the research I have available wearing sunscreen with a SPF at or greater than 95 is the most likely to prevent the harmful effects of sunburn".

Go to google -> Search "sunscreen" -> (Sponsored) Banana Boat new Ultra 95 SPF sunscreen

Oh wow! 95 SPF! Perfect

Re: The Google employees who created transformers

#142

Post by pchristensen »

pixl97 wrote: post #106
ralph84 wrote: post #68

And none of them still work for Google. It’s truly baffling that Google’s CEO still has a job after how badly he fumbled on AI.

I'm not 100% sure I agree... Google is an ad company at the end of the day. Google is still making obscene amount of money with ads. Currently Gen AI is a massive expense in training and running and is only looking like it may harm future ad revenue (yet to be seen). Meanwhile OpenAI has not surpassed that critical threshold where they dominate the market with a moat and become a trillion dollar company. It is typica…

Classic Innovator's Dilemma case study.

Re: The Google employees who created transformers

#143

Post by lelag »

TimPC wrote: post #130
lelag wrote: post #60

Earlier quoted context omitted.

Why do you think transformers won't be key in making a level 4+ self-driving AI. It seems to me that Vision-capable multi-modal transformers could be the missing part: they can understand what is happening in the world in a deductive way. The Vision transformer is capable of predicting that a running child is likely to follow that rolling soccer ball. It is capable of deducting that a particular situation looks dange…

I think we've found repeatedly in self-driving that it's not enough to solve the problem in the normal case. You need an AI model that has good behaviors in the edge cases. For the most part it won't matter how good the vision models get, you're going to need similar models that can make the same predictions from LIDAR signals because the technology needs to work when the vision model goes crazy because of the reflec…

I don't quite agree on this one. While I think that Musk choices to go full vision when he did was foulish because he made his product worse, his main point is not wrong: human do drive well while using mostly vision. Assuming you can replicate the thought process of human driver using AI, I don't see why you could not create a self-driving car using only vision.

That's also where I would see transformers or another AI architecture with reasoning capabilities shine: the fact that it can reason about what is about to happen would allow it to handle edge cases much better than relying on dumb sensors.

As a human, it would be very difficult to drive a car just looking at sensor data. The only vehicule I can think of where we do that is submarines. Sensors data is good for classical AI but I don't think it will handle edge case well.

To be a reasonable self-driving system, it should be able to decide to slow down and maintain a reasonable safety space because it is judging the car in front to be driving erratically (ex: due to driver impairement). Only an AI that can reason about what is going on can do that.

Re: The Google employees who created transformers

#144

Post by cgearhart »

Thaxll wrote: post #93

It's pretty crazy to think that Google is not OpenAI today, they had deep mind and an army of PHDs early on.

I think it’s evidence that timing is everything. In the 2010s deep learning was still figuring out how to leverage GPUs. The scale of compute required for everything after GPT-2 would have been nearly impossible in 2017/2018–our courses at Udacity used a few hours of time on K80 GPUs. By 2020 it was becoming possible to get unbelievable amounts of compute to throw at these models to test the scale hypothesis. The rise of LLMs is stark proof of the Bitter Lesson because it’s at least as much the story of GPU advancement as it is about algorithms.

Re: The Google employees who created transformers

#145

Post by com2kid »

pixl97 wrote: post #106
ralph84 wrote: post #68

And none of them still work for Google. It’s truly baffling that Google’s CEO still has a job after how badly he fumbled on AI.

I'm not 100% sure I agree... Google is an ad company at the end of the day. Google is still making obscene amount of money with ads. Currently Gen AI is a massive expense in training and running and is only looking like it may harm future ad revenue (yet to be seen). Meanwhile OpenAI has not surpassed that critical threshold where they dominate the market with a moat and become a trillion dollar company. It is typica…

> It is typically very hard for a company to change what they actually do

Microsoft started out selling BASIC runtimes. Then they moved into operating systems. Then they moved into Cloud. Satya seems to be putting his money where his mouth is and working hard to transition the company to AI now.

Apple has likewise undergone multiplet transformations over the decades.

However Google has the unique problem that their main cash cow is absurdly lucrative and has been an ongoing source of extreme income for multiple decades now, where as other companies typically can't get away with a single product line keeping them on top of the world for 20+ years. (Windows being an exception, maybe Microsoft is an exception, but say what you will about MS, they pushed technology forward throughout the 90s and accomplished their goal of a PC in every house)

Re: The Google employees who created transformers

#146

Post by dekhn »

eitally wrote: post #136
dekhn wrote: post #83

Earlier quoted context omitted.

The first time I met with Eric Schmidt he asked what I did (harvest idle cycles to do scientific calculations), and then we talked about how Google was slowly getting into cloud. I remember him saying "I've tried to convince the board that the cloud is the future of growth for google, but they aren't interested" (this was before GCS and GCE launched). The second time I met him I presented my project (Google Cloud Gen…

The irony, of course, with that, is that then Google/Alphabet like Microsoft eat its lunch in the space by poorly supporting The Broad and allowing Verily to sign a $150m deal with Azure. Quite a few of the folks who were working on HCLS products in Cloud (were you one of them?) subsequently departed for Verily.

I worked on the predecessor of HCLS and was pushed out before any Azure deal (and would never have moved to verily). I had set up a relationship between Google Research, the Broad, and Cloud (to apply large-scale Google ML technology in genomics) and all that went away when other folks took it in different strategic direction.

The Broad is still a google cloud customer and verily seems to support all three clouds now, but I'm not aware of the details. The whole thing seemed really strange and unbusinesslike.

Re: The Google employees who created transformers

#147

Post by com2kid »

pigscantfly wrote: post #108
paxys wrote: post #96

Earlier quoted context omitted.

Their wildest success at Google would have meant the company's stock price doubling or tripling and their stock grants being worth a couple million dollars at most. Meanwhile all of them had blank checks from top VCs in the valley and the freedom to do whatever they wanted with zero oversight. What exactly could Sundar have done to retain them?

It would have gone much better for Google if the Brain team had been permitted to apply their work, but they were repeatedly blocked on stated grounds of AI safety and worries about impacting existing business lines through negative PR. I think this is probably the biggest missed business opportunity of the past decade, and much of the blame for losing key talent and Google's head start in LLMs ultimately resides wit…

They should have been spun out with a few billion dollars budget, no oversight, and Google having 90% (or some other very high %) ownership and rights of first purchase to buy them back.

It is a successful model that has worked again and again to escape the problem of corporate bureaucracy.

Re: The Google employees who created transformers

#148

Post by mzs »

ralph84 wrote: post #68

And none of them still work for Google. It’s truly baffling that Google’s CEO still has a job after how badly he fumbled on AI.

Google doesn't have to try to be the best at AI now, it can just buy whoever is the winner later - a lesson learned from google video.

Re: The Google employees who created transformers

#149

Post by EForEndeavour »

lelag wrote: post #143
TimPC wrote: post #130

Earlier quoted context omitted.

I think we've found repeatedly in self-driving that it's not enough to solve the problem in the normal case. You need an AI model that has good behaviors in the edge cases. For the most part it won't matter how good the vision models get, you're going to need similar models that can make the same predictions from LIDAR signals because the technology needs to work when the vision model goes crazy because of the reflec…

I don't quite agree on this one. While I think that Musk choices to go full vision when he did was foulish because he made his product worse, his main point is not wrong: human do drive well while using mostly vision. Assuming you can replicate the thought process of human driver using AI, I don't see why you could not create a self-driving car using only vision. That's also where I would see transformers or another…

> As a human, it would be very difficult to drive a car just looking at sensor data.

What is vision if not sensor data?? Our brains have evolved to efficiently process and interpret image data. I don't see why from-scratch neural network architectures should ever be limited to the same highly specific input type.

Re: The Google employees who created transformers

#150

Post by samatman »

com2kid wrote: post #145
pixl97 wrote: post #106

Earlier quoted context omitted.

I'm not 100% sure I agree... Google is an ad company at the end of the day. Google is still making obscene amount of money with ads. Currently Gen AI is a massive expense in training and running and is only looking like it may harm future ad revenue (yet to be seen). Meanwhile OpenAI has not surpassed that critical threshold where they dominate the market with a moat and become a trillion dollar company. It is typica…

> It is typically very hard for a company to change what they actually do Microsoft started out selling BASIC runtimes. Then they moved into operating systems. Then they moved into Cloud. Satya seems to be putting his money where his mouth is and working hard to transition the company to AI now. Apple has likewise undergone multiplet transformations over the decades. However Google has the unique problem that their m…

Also, though I dislike it greatly, Microsoft purchasing GitHub was a brilliant way to buy into a growing segment which were natively hostile to what they stand for. LinkedIn, maybe less clever, they only make pocket change on it, but it is profitable.

Google doesn't really do acquisitions on that level. They do buy companies, but with the purpose of incorporating their biological distinctiveness into the collective. This tends to highlight their many failures in capturing new markets. The last high-profile successful purchase by Google, that I recall at least, were YouTube and Android, nearly twenty years ago.

Post reply on HN