Transformer == Modern AI? Wow, I dont know where should I start My mind literally stuck at the absurdity
AI is in the eye of the beholder.
And in this case, the transformers are more than meets the eye.
The Google employees who created transformers
191–200 of 258 posts
Re: The Google employees who created transformers
#192In Google's heyday, around 2014, I was talking with Uszkoreit about a possible role on his then NLP team. I asked "What would you do if you had an unlimited budget?" He simply said, "I do"
Hahaha... I worked at Borg The quota system can kick in at whatever time the limits are reached. And GPUs are scattered across borg cells, limiting the ceiling. That's why XBorg was created so that a global search among all Borg cells for researchers. And data center Capex is around 5 billion each year. Google makes hundres of billions of revenue each year. You are asking what people would do in impossible situation.…
Lighten up.
Re: The Google employees who created transformers
#193Earlier quoted context omitted.
It would have gone much better for Google if the Brain team had been permitted to apply their work, but they were repeatedly blocked on stated grounds of AI safety and worries about impacting existing business lines through negative PR. I think this is probably the biggest missed business opportunity of the past decade, and much of the blame for losing key talent and Google's head start in LLMs ultimately resides wit…
Funnily enough, the same AI safety teams that held Google back from using large transformers in products are also largely responsible for the Gemini image generation debacle. It is tough to find the right balance though, because AI safety is not something you want to brush off.
I thought Google fired it's AI Ethicists a few years back and dismantled the team?
Re: The Google employees who created transformers
#194It's pretty crazy to think that Google is not OpenAI today, they had deep mind and an army of PHDs early on.
The problem is that chatting with an LLM is extremely disruptive to their business model and it's difficult for them to productize without killing the golden goose.
I know everyone cites this as the innovators dilemma, but so far the evidence suggests this isn't true.
ChatGPT has been around for a while now, and it hasn't led to a collapse in Google's search revenue, and in fact now Google is rushing to roll out their version instead of trying to entrench search.
A famous example is the iPhone killing the iPod, and it took around 3 and a half years for the iPod to really collapse, so chat and co-pilots might be early still. On the other hand handheld consumer electronics have much longer buying cycles than software tools.
Re: The Google employees who created transformers
#195Earlier quoted context omitted.
A little after that. I'd put the heyday between 2003-2010, starting with the GMail launch and ending with the Chrome & Android launches. That period includes GMail, Maps, Scholar, Orkut, Reader, the acquisitions of Blogger/YouTube/Docs/Sheets/Slides/Analytics/Android, Summer of Code, OAuth, Translate, Voice Search, search suggestions, universal search, Picasa, etc. Basically I can look at my phone or computer and bas…
GMail, the service that tells every user that they will be indexed and profiled personally for ad leads? that is an achievement ? GMail was one of the red flags for many that "don't be evil" was not going to be what it appeared. History says that this kind of mass profiling never ends well.
Gmail doesn't show personalized ads for some years FWIW.
Re: The Google employees who created transformers
#196Attention models? Attention existed before those papers. What they did was show that it was enough to predict next word sequences in a certain context. I'm certain they didn't realize what they found. We used this frame work in 2018 and it gave us wildly unusual behavior (but really fun) and we tried to solve it (really looking for HF capability more than RL) but we didn't see what another group found: that scale in…
I guess the same thing happened with special relativity and Poincaré was forgotten.
Re: The Google employees who created transformers
#197It's pretty crazy to think that Google is not OpenAI today, they had deep mind and an army of PHDs early on.
Well at the time before Microsoft got involved it was sort of an unspoken rule among the AI community to be open and not release certain models to the public.
To be open and not release? Isn't that at odds?
Re: The Google employees who created transformers
#198Earlier quoted context omitted.
> It is typically very hard for a company to change what they actually do Microsoft started out selling BASIC runtimes. Then they moved into operating systems. Then they moved into Cloud. Satya seems to be putting his money where his mouth is and working hard to transition the company to AI now. Apple has likewise undergone multiplet transformations over the decades. However Google has the unique problem that their m…
Remember that Microsoft spent 10 years fighting the open internet, and only launched the cloud much later. The Windows business was allergic to the web-model, and they fought it tooth and nail.
In fairness, the web model tends to be objectively worse in every way except profits.
Re: The Google employees who created transformers
#199Earlier quoted context omitted.
Sure, Google is worth more but they essentially reduced themselves to an ad business. They aren't focused on innovation as much as before. Shareholder value is the new god.
Agreed that Friedman cursed the market with "shareholder theory", and it is a totally subversive and corrupting concept because whatever will increase value is accepted because it's the doctrine.
Economic "theories" have an annoying way of being self-fulfilling.
Re: The Google employees who created transformers
#200Attention models? Attention existed before those papers. What they did was show that it was enough to predict next word sequences in a certain context. I'm certain they didn't realize what they found. We used this frame work in 2018 and it gave us wildly unusual behavior (but really fun) and we tried to solve it (really looking for HF capability more than RL) but we didn't see what another group found: that scale in…
I argue they definitely 'changed AI' but definitely agree they didn't 'invent modern AI'.
Personally, I think both compute and NN architecture is probably needed to get closer to AGI.