381–390 of 478 posts
I guess this means Google is nowhere close, to even discern a hint of an AGI? So when Demis Hassabis says AGI...could arrive in just 3 years he has learned the best from Larry Ellison?
Although I can't fathom why we'd want to? Like what is the advantage of giving tools sentience?
Sorry to bring up the elephant in the room - but could this decision be in part the opportunity to acquire large amounts of stock before a massively inflated IPO?
Earlier quoted context omitted.
You train the model then do a baseline evaluation. Then you evaluate many variants where you have removed or nulled out different layers or chunks of the model. By comparing the performance of those mutated models to the baseline you can learn a lot about the model. What parts don't have much value and can be removed, the location of "functions" or "facts." Etc. Google it.
How come they didn’t ablate encoder? OpenAI GOT models are decoder only.
The research history leading up to this was interesting - there had been a bunch of work, in various domains, on "autoencoder" architectures used to learn compact representations for things like dimensionality reduction and sequence representation. The idea was to have an encoder-decoder pair, connected by a limited bottleneck representation, with the training goal of the decoder reconstructing the encoder input from the bottleneck representation.
One example of this was to learn a fixed size(!) sequence (e.g. sentence) representation using an LSTM-based autoencoder (LSTM->embedding->LSTM), which at the time seemed rather shocking - the ability to represent a variable length sequence with a fixed size embedding. Equally shocking was that you could use this for machine translation simply by connecting an LSTM encoder for one language to an LSTM decoder for another language.
This type of LSTM->LSTM seq2seq encode-decode architecture for machine translation was then improved by Bahdanau by replacing the fixed size representation with an attention mechanism so the decoder could learn to be more specific about input-output relationships.
This type of LSTM-based seq2seq encode-decode architecture, using attention, is what Uszkoreit et al set out to improve - to make more efficient by using a parallel vs sequential (RNN) architecture. The Transformer was never conceived of as purely for language modelling, or as an "AI" architecture. Later when the usage focused on language modelling (generation, not translation), the encoder was dropped since input and output are the same thing.
Sorry to bring up the elephant in the room - but could this decision be in part the opportunity to acquire large amounts of stock before a massively inflated IPO?
Google acquired his company in 2024 for $2.7Bn with him taking about 40% of that. I'm quite sure that no matter where he went, any lab or his own start up, he would be fine financially.
But money at that level isn't about being financially secure - to have a roof over your head and food to eat - it's about power.
Money at that level gives you the ability to shape the world in ways others can only dream of - whether that be starting your own company where you can set the values, funding a cure for Malaria, or political lobbying.
Depends on whether the person in question has strong views and a strong belief that they are in the right.
Full disclaimer - I have no insight or knowledge about this particular person - just making the rather obvious and general case that joining OpenAI now at a senior level is likely to generate a serious windfall, and such a windfall is power.
As I said, no idea what motivates this particular person - don't know them at all - the money may be entirely coincidental and it's all about getting stuff done - but he did choose OpenAI rather than somebody like Anthropic....
Earlier quoted context omitted.
Google bloat gave us transformers. Apple bloat gave us a usable touchscreen only, pocket computer (famously an entire org within Apple had developed an iPod-based approach that was competing with what was released) The leaps forward need bloat. A startup can execute on specific vector direction way better. Now back to your point, what did X deliver with its lean ops? It seems that it needed 2 bailouts (one from xAI,…
innovation requires monopoly is a Theil one liner -- probably true too
Earlier quoted context omitted.
I only said top scientists and top engineers deserve as much fame / respect / gossip as top football players yeah
Sadly most science and engineering is very capital intensive. So take this scenario - I'd argue that if you want to make progress in the field of these particular ML models, then you are going to need resources ( compute/data etc ) that is beyond most individuals capability to muster. ie you have to join a company with resources ( or persuade somebody to give you them ). Right now there is one of those scenarios wher…
To become a good scientist you don’t need much classic capital, you need a good environment. And for ML you only need one computer for yourself or you can rent online
There are still big inefficiencies for those who have capital to discover good scientists / engineers. Lots of them are unknown.
But if there are top ones famous it will bring more people to study those fields
Earlier quoted context omitted.
Thank you for actually answering my question. That is very vague and explains why I find the word so annoying.
Zionist does have a specific meaning. It means you think the Jewish people have a god-given right to the Palestinian land, and that other creeds and ethnicities should be second class within the Jewish state in Palestine. A non-zionist Israel would be one where all peoples had the same right, e.g.
More specifically there are 0 Jews living in areas under control of the Palestinian Authority, or in Lebanon, Syria, Egypt, Jordan, and Yemen.
Earlier quoted context omitted.
Eh, what has X/Twitter delivered since the cull? It’s basically in maintenance mode. Which is fine if that’s what you want to do, but Google and Apple definitely don’t (and I’m glad for that)
> It’s basically in maintenance mode Has been in more of a maintenance mode with a multiple of those people. If anything, the pace of the product has improved. Regardless of what you think about Musk, the company he bought was a bloated mess.
Earlier quoted context omitted.
This understates his criticality. The author list was randomized, but the critical idea was truly his. Wonder what this says about GDM …
The architecture was Shazeer's, but the rough idea came from Jakob Uszkoreit who initiated the project. Uszkoreit wanted to build a more efficient/scalable language/seq2seq model that could take advantage of GPU parallelism (replacing RNNs which were the main approach to sequence modelling at that time). Uszkoreit's insight was that although language appears sequential, it is in fact really part parallel part hierarc…