391–400 of 525 posts
Earlier quoted context omitted.
Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.
Both ChatGPT 3.5 and 4 literally fail the question: "What is the third letter in the third word of this sentence" When you've spent 100 million on training the thing and it fails on 1st grade ordinality I think it's fair to say you may not be on the right path
Earlier quoted context omitted.
I agree, probably for a different reason. IMO the fact that Sam is saying this likely means that the LLMs are close to the upper knee of the S-Curve and after GPT5 they likely dont have many new fundamental ideas (additional SCurves) to throw at it. since they are ahead of the rest, it likely means we might be headed for an AI improvement pause for couple of years after GPT5. Still good enough to upset the balance in…
> it likely means we might be headed for an AI improvement pause for couple of years after GPT5. I suspect that a pause in base LLM performance won’t be an AI improvement pause; there’s a whole lot of space to improve the parts of AI systems around the core “brain in a jar” model.
far as AGI is concerned I dont believe LLMs are really the right architecture for it, AGI likely needs some symbolic logic and a notion of physicality (ie.. physical laws & energy/power).
I strongly believe the next generation of models will be based upon spiking neural concepts wherein action potentials are lazily-evaluated throughout the network (i.e. event-driven). There are a few neuron models that can be modified (at some expense to fidelity) in order to tolerate arbitrary delays between simulation ticks. Using actual latency between neurons as a means of encoding information seems absolutely ess…
Earlier quoted context omitted.
> A transformer trained on all text and all of YouTube will be killer amazing at so much. ah yes, a bot where the answer to everything is to buy ridge wallets and play raid shadow legends
It took a year's worth of a small nation's energy expenditure to train and all it does is tell you how Raycon earbuds will change your life.
I strongly believe the next generation of models will be based upon spiking neural concepts wherein action potentials are lazily-evaluated throughout the network (i.e. event-driven). There are a few neuron models that can be modified (at some expense to fidelity) in order to tolerate arbitrary delays between simulation ticks. Using actual latency between neurons as a means of encoding information seems absolutely ess…
Bad, BAD idea. Remember the Tay chatbot, which 4chan managed to turn into a raging Nazi in the matter of a few hours?
"The great supercomputer Deep Thought is the most powerful computer ever built, with one exception. It was designed by hyper-intelligent pan-dimensional beings, who wanted to know the answer to Life, the Universe, and Everything. Its creation annoyed a fair few philosophers, who felt that it was taking over their turf. After seven and a half million years of serious cogitation, Deep Thought spoke the answer. However,…
How long till the air con goes on strike for miserable working conditions?
"Brain the size of a planet and they ask me write a lesson plan in the style of a pirate" - chatgpt5, probably...
Earlier quoted context omitted.
> (including Google itself!) Bet Google won’t make that mistake again, i.e. it won’t publish as much and will be much more careful about what it publishes, least they give a competitor a useful tool and get nothing in return - when the competitor (in this case very ironically named) goes full commercial and close source everything they can. Open collaboration in AI, at least when it comes to corporations, might have…
The scale of the damage Open AI has done to the trust ecosystem with soliciting not just the work, but also massive fundraising and then privatizing the profits is almost unprecedented and permanent.
What trust ecosystem are you talking about ? It was a lack of foresight by google on their own discovery of transformers, and it would probably have been sitting in dust or been killed off by the time it would have taken them to reach GPT-2 level of progress.
Earlier quoted context omitted.
Both ChatGPT 3.5 and 4 literally fail the question: "What is the third letter in the third word of this sentence" When you've spent 100 million on training the thing and it fails on 1st grade ordinality I think it's fair to say you may not be on the right path
Ah yes, it fails to count letters but successfully explains quantum mechanics. It's useless.
Earlier quoted context omitted.
Further improvements in efficiency need not come from alternative architectures. They'll likely also come from novel training objectives, optimizers, data augmentations, etc.
There's only so much first order derivatives can do for you. My bet is on sparsity, lottery tickets and symmetries.
While these things represent a fundamental way we store information as humans, these have very little to do with actual reasoning.
My bet is that Hebbian learning is going to see a resurgence. Basically the architecture needs to be able to partition data domains while drawing connections between them, and being able to run internal prediction mechanisms.
Earlier quoted context omitted.
Both ChatGPT 3.5 and 4 literally fail the question: "What is the third letter in the third word of this sentence" When you've spent 100 million on training the thing and it fails on 1st grade ordinality I think it's fair to say you may not be on the right path
"The third word of this sentence is "the," and its third letter is "e." - GPT-4