Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Enough

mistral.ai

Post reply on HN
Print view View original

391–400 of 512 posts

Re: Large Enough

#391

Post by lolinder »

jpadkins wrote: post #387
lolinder wrote: post #129

Earlier quoted context omitted.

> It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs Yes. This is exactly why I'm skeptical of AI doomerism/saviorism. Too many people have been looking at the pace of LLM development over the last two (2) years, modeled it as an exponential growth function, and come to the conclusion that AGI is inevitable in the next ${1-5} years and we're headed for ${(dys|u)topia}. B…

> we're already seeing that OpenAI has pivoted from improving on GPT-4's benchmarks to reducing cost, while competitors (including free ones) catch up. What if they have two teams? One dedicated to optimizing (cost, speed, etc) the current model and a different team working on the next frontier model? I don't think we know the growth curve until we see gpt5.

> I don't think we know the growth curve until we see gpt5.

I'm prepared to be wrong, but I think that the fact that we still haven't seen GPT-5 or even had a proper teaser for it 16 months after GPT-4 is evidence that the growth curve is slowing. The teasers that the media assumed were for GPT-5 seem to have actually been for GPT-4o [0]:

> Lex Fridman(01:06:13) So when is GPT-5 coming out again?

> Sam Altman(01:06:15) I don’t know. That’s the honest answer.

> Lex Fridman(01:06:18) Oh, that’s the honest answer. Blink twice if it’s this year.

> Sam Altman(01:06:30) We will release an amazing new model this year. I don’t know what we’ll call it.

> Lex Fridman(01:06:36) So that goes to the question of, what’s the way we release this thing?

> Sam Altman(01:06:41) We’ll release in the coming months many different things. I think that’d be very cool. I think before we talk about a GPT-5-like model called that, or not called that, or a little bit worse or a little bit better than what you’d expect from a GPT-5, I think we have a lot of other important things to release first.

Note that last response. That's not the sound of a CEO who has an amazing v5 of their product lined up, that's the sound of a CEO who's trying to figure out how to brand the model that they're working on that will be cheaper but not substantially better.

[0] https://arstechnica.com/information-technology/2024/03/opena...

Re: Large Enough

#393

Post by impossiblefork »

lolinder wrote: post #386
impossiblefork wrote: post #341

Earlier quoted context omitted.

Notice though, that all these improvements have been with pretty basic transformer models that output all their tokens-- no internal thoughts, no search, no architecture improvements and things are only fed through them once. But we could add internal thoughts-- we could make the model generate tokens that aren't part of its output but are there for it to better figure out its next token. This was tried QuietSTAR. Ho…

> Also, presumably GPT-5 isn't far away. Why do we presume that? People were saying this right before 4o and then what came out was not 5 but instead a major improvement on cost for 4. Is there any specific reason to believe OpenAI has a model coming soon that will be a major step up in capabilities?

OpenAI have made statements saying they've begun training it, as they explain here: https://openai.com/index/openai-board-forms-safety-and-secur...

I assume that this won't take forever, but will be done this year. A couple of months, not more.

Re: Large Enough

#394

Post by viraptor »

VincentEvans wrote: post #366
microtonal wrote: post #342

Earlier quoted context omitted.

How the tokenizer works explains why a model can’t answer the question, what the name of the cat is doesn’t explain anything. This is Hacker News, we are usually interested in how things work.

Indeed, I appreciate the explanation, it is certainly both interesting and informative to me, but to somewhat echo the person you are replying to - if I wanted a boat, and you offer me a boat, and it doesn’t float - the reasons for failure are perhaps full of interesting details, but perhaps the most important thing to focus on first - is to make the boat float, or stop offering it to people who are in need of a boat…

At some point you need to just accept the details and limitations of things. We do this all the time. Why is your calculator giving only approximate result? Why can't your car go backwards as fast as forwards? Etc. It sucks that everyone gets exposed to the relatively low level implementation with LLM (almost the raw model), but that's the reality today.

Re: Large Enough

#395

Post by OKRainbowKid »

groby_b wrote: post #383
probably_wrong wrote: post #263

Earlier quoted context omitted.

I would counterargue with "that's the model's problem, not mine". Here's a thought experiment: if I gave you 5 boxes and told you "how many balls are there in all of this boxes?" and you answered "I don't know because they are inside boxes", that's a fail. A truly intelligent individual would open them and look inside. A truly intelligent model would (say) retokenize the word into its individual letters (which I'm op…

LLMs are not truly intelligent. Never have been, never will be. They model language, not intelligence.

They model the dataset they were trained on. How would a dataset of what you consider intelligence look like?

Re: Large Enough

#396

Post by Zambyte »

TIPSIO wrote: post #3

This race for the top model is getting wild. Everyone is claiming to one-up each with every version. My experience (benchmarks aside) Claude 3.5 Sonnet absolutely blows everything away. I'm not really sure how to even test/use Mistral or Llama for everyday use though.

I recommend using a UI that you can just use whatever models you want. OpenWebUI can use anything OpenAI compatible. I have mine hooked up to Groq and Mistral, in addition to my Ollama instance.

Re: Large Enough

#397

Post by [deleted] »

jack_pp wrote: post #252
LrnByTeach wrote: post #214

Earlier quoted context omitted.

Such a relief/contrast to the period between 2010 and 2020, when the top five Google, Apple, Facebook, Amazon, and Microsoft monopolized their own regions and refused to compete with any other player in new fields. Google : Search Facebook : social Apple : phones Amazon : shopping Microsoft : enterprise .. > Even still, this monthly progress across all companies is exciting to watch. Its very gratifying to see useful…

Google refused to compete with Apple in phones? Microsoft also competes in search, phones Microsoft, Amazon and Google compete in cloud too

[deleted]

Re: Large Enough

#398

Post by OKRainbowKid »

skyde wrote: post #281
grumbel wrote: post #211

Earlier quoted context omitted.

GPT4o already does that, for problems involving math it will write small Python programs to handle the calculations instead of doing it with the LLM itself.

It “work” but the LLM having to use the calculator mean the LLM doesn’t understand arithmetic enough and doesn’t know how to use an follow a set of step (algorithm ) natively to find the answer for bug numbers. I believe this could be fixed and is worth fixing. Because it’s the only way LLM will be able to help math and physic researcher write proof and make real scientific progress

It generates the code to run for the answer. Surely that means it actually knows to build the appropriate algorithm - it just struggles to perform the actual calculation.

Re: Large Enough

#399

Post by audunw »

impossiblefork wrote: post #341
lolinder wrote: post #129

Earlier quoted context omitted.

> It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs Yes. This is exactly why I'm skeptical of AI doomerism/saviorism. Too many people have been looking at the pace of LLM development over the last two (2) years, modeled it as an exponential growth function, and come to the conclusion that AGI is inevitable in the next ${1-5} years and we're headed for ${(dys|u)topia}. B…

Notice though, that all these improvements have been with pretty basic transformer models that output all their tokens-- no internal thoughts, no search, no architecture improvements and things are only fed through them once. But we could add internal thoughts-- we could make the model generate tokens that aren't part of its output but are there for it to better figure out its next token. This was tried QuietSTAR. Ho…

> But we could add internal thoughts

It feels like there’s an assumption in the community that this will be almost trivial.

I suspect it will be one of the hardest tasks humanity has ever endeavoured. I’m guessing it has already been tried many times in internal development.

I suspect if you start creating a feedback loop with these models they will tend to become very unstable very fast. We already see with these more linear LLMs that they can be extremely sensitive to the values of parameters like the temperature settings, and can go “crazy” fairly easily.

With feedback loops it could become much harder to prevent these AIs from spinning out of control. And no I don’t mean in the “become an evil paperclip maximiser” kind of way. Just plain unproductive insanity.

I think I can summarise my vision of the future in one sentence: AI psychologists will become a huge profession, and it will be just as difficult and nebulous as being a human psychologist.

Re: Large Enough

#400

Post by viraptor »

psb217 wrote: post #276
generalizations wrote: post #221

Earlier quoted context omitted.

Testing models on their tokenization has always struck me as kinda odd. Like, that has nothing to do with their intelligence.

How can I know whether any particular question will test a model on its tokenization? If a model makes a boneheaded error, how can I know whether it was due to lack of intelligence or due to tokenization? I think finding places where models are surprisingly dumb is often more informative than finding particular instances where they seem clever. It's also funny, since this strawberry question is one where a model that…

> How can I know whether any particular question will test a model on its tokenization?

Does something deal with separate symbols rather than just meaning of words? Then yes.

This affects spelling, math (value calculation), logic puzzles based on symbols. (You'll have more success with a puzzle about "A B A" rather than "ABA")

> It requires no reasoning, and is unlikely to have any contradicting text in the training corpus.

This thread contains contradictions. Every other announcement of an llm contains a comment with a contradicting text when people post the wrong responses.

Post reply on HN