111–120 of 478 posts
Some context for people who haven’t followed the full loop: Shazeer was a long-time Google researcher, joined Google in 2000, and was one of the co-authors of “Attention Is All You Need.” He left Google in 2021 to co-found Character.AI. In 2024, Google brought him and some Character.AI researchers back via a licensing/talent deal with Character.AI (reportedly around $2.7B). He was then made a Gemini co-lead. Now he’s…
[flagged]
Question two: Why are OpenAI spending that money taking talent from Google, who can definitely outspend them for talent, and not Anthropic, who are leading the market and are at least somewhat financially constrained.
Earlier quoted context omitted.
[flagged]
[flagged]
I have no dog in this race as I'm not fond of either OpenAI or Google.... but employees not being loyal to their big tech employers is a wild thing to be concerned about in 2026 when year after year many large tech companies (Google very prominently among them) continually post record profits and still lay people off by the thousands.
Silver lining: given the leaked financials of OpenAI, he might very well be joining a sinking ship. Also, why didn't they nail him down contractually when they bought character.ai ... isn't that pretty standard with these type of superstar (re)hires?
OpenAI is in a unique position right now to grant pre-IPO options (probably in the form of RSUs). And they wanted him badly enough to grant the extra options necessary to effectively 'buy out' whatever unvested Google bonus he's walking away from.
Earlier quoted context omitted.
[flagged]
[flagged]
Noam Shazeer was one of the lead authors of the seminal paper "Attention Is All You Need", which introduced the transformer architecture. (From Wikipedia)
This understates his criticality. The author list was randomized, but the critical idea was truly his. Wonder what this says about GDM …
Uszkoreit wanted to build a more efficient/scalable language/seq2seq model that could take advantage of GPU parallelism (replacing RNNs which were the main approach to sequence modelling at that time).
Uszkoreit's insight was that although language appears sequential, it is in fact really part parallel part hierarchical, as can be seen by linguist's sentence parse trees where at each level there is parallelism/independence between the branches of the tree, with them getting combined at the next level up. This is what gave rise to the idea of a model that consisted of a stack of of parallel processing layers (transformer layers). I believe that attention was also part of the plan from day one, as this had already been proven to be valuable (Bahdanau) with RNN seq2seq modelling.
So, this is what Uszkoreit wanted to build, but by his own account he failed to come up with an implementation that matched or outperformed the prevailing RNN approach that he wanted to replace. At this point, Uszkoreit mentioned the idea to Shazeer, who got on board and eventually arrived at a performant architecture which was then pared back by an ablation process resulting in the initial encoder-decoder Transformer architecture. Shazeer later came up with the mixture-of-experts architecture, and also other optimizations after he left to found character.ai