[flagged]
Noam Shazeer Joins OpenAI
121–130 of 478 posts
Re: Noam Shazeer Joins OpenAI
#122Re: Noam Shazeer Joins OpenAI
#123Question one: How much did this cost OpenAI? Question two: Why are OpenAI spending that money taking talent from Google, who can definitely outspend them for talent, and not Anthropic, who are leading the market and are at least somewhat financially constrained.
Reporting on this seems to indicate that people at Anthropic are significantly more loyal, and that attempts to poach by OpenAI and Meta have been largely unsuccessful.
Re: Noam Shazeer Joins OpenAI
#124Earlier quoted context omitted.
Is Shazeer known to be opposed to open-weight releases?
OpenAI hasn't released open weights since GPT-OSS-20/120B. Google has the Gemma line. I wouldn't expect OpenAI to start releasing open weight competitive models again, but I could be wrong.
Their models are the only moat they think they have left, which at this point is more of filled-in wet circle of dirt.
Re: Noam Shazeer Joins OpenAI
#125Wow. What could possibly have caused him to quit so soon after coming back? I hope this is not accurate but I'm afraid it is: https://x.com/signulll/status/2067446889956430273
This reads like an episode of Silicon Valley. I wish that show was rebooted, they'd have so much funny material nowadays.
Your dream may be only a prompt away.
Re: Noam Shazeer Joins OpenAI
#126[flagged]
> idea thief. Takes credit for others’ work.
Based on what?
Re: Noam Shazeer Joins OpenAI
#127Wow. What could possibly have caused him to quit so soon after coming back? I hope this is not accurate but I'm afraid it is: https://x.com/signulll/status/2067446889956430273
https://nitter.net/signulll/status/2067446889956430273 for those who don't want to click the above
[deleted]
Re: Noam Shazeer Joins OpenAI
#128Huge blow to Google.
I doubt that the money had anything to do with it.
I also doubt that the state of the technology at OAI vs. Google had much to do with it, Google is behind no doubt, but the gap is not as far as we know, insurmountable.
I suspect that this is a leadership clash. Noam was working in GDM. GDM somehow went away from coding and RSI into "world models" and that has played out very poorly. Who made that call? Who was still playing politics?
Given this is Noam the list of people that could be pissing him off is very small: Demis, Sergey (?!), a couple of VPs in GDM.
What the hell happened?
Re: Noam Shazeer Joins OpenAI
#129Earlier quoted context omitted.
This understates his criticality. The author list was randomized, but the critical idea was truly his. Wonder what this says about GDM …
Source for this? The notion of attention dates to a content-addressable lookup during sequence alignment (as well as, concurrently, memory lookups in neural Turing machines). Attention had been used in other models, like GRUs and LSTMs with attention. The Vaswani et. al. paper did not introduce attention, just removed everything _but_ attention (and FFW) from the network. Are you claiming the "critical idea" of remov…
At some point in late 2017 the paper was updated with this additional detail:
Equal contribution. Listing order is random. Jakob proposed replacing RNNs with self-attention and started the effort to evaluate this idea. Ashish, with Illia, designed and implemented the first Transformer models and has been crucially involved in every aspect of this work. Noam proposed scaled dot-product attention, multi-head attention and the parameter-free position representation and became the other person involved in nearly every detail. Niki designed, implemented, tuned and evaluated countless model variants in our original codebase and tensor2tensor. Llion also experimented with novel model variants, was responsible for our initial codebase, and efficient inference and visualizations. Lukasz and Aidan spent countless long days designing various parts of and implementing tensor2tensor, replacing our earlier codebase, greatly improving results and massively accelerating our research.