Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Enough

mistral.ai

Post reply on HN
Print view View original

281–290 of 512 posts

Re: Large Enough

#281

Post by skyde »

grumbel wrote: post #211
asadm wrote: post #96

Earlier quoted context omitted.

this can be automated.

GPT4o already does that, for problems involving math it will write small Python programs to handle the calculations instead of doing it with the LLM itself.

It “work” but the LLM having to use the calculator mean the LLM doesn’t understand arithmetic enough and doesn’t know how to use an follow a set of step (algorithm ) natively to find the answer for bug numbers.

I believe this could be fixed and is worth fixing. Because it’s the only way LLM will be able to help math and physic researcher write proof and make real scientific progress

Re: Large Enough

#282

Post by eschneider »

drewnick wrote: post #27
tikkun wrote: post #5

Links to chat with models that released this week: Large 2 - https://chat.mistral.ai/chat Llama 3.1 405b - https://www.llama2.ai/ I just tested Mistral Large 2 and Llama 3.1 405b on 5 prompts from my Claude history. I'd rank as: 1. Sonnet 3.5 2. Large 2 and Llama 405b (similar, no clear winner between the two) If you're using Claude, stick with it. My Claude wishlist: 1. Smarter (yes, it's the most intelligent, and y…

All 3 models you ranked cannot get "how many r's are in strawberry?" correct. They all claim 2 r's unless you press them. With all the training data I'm surprised none of them fixed this yet.

The models are text generators. They don't "understand" the question.

Re: Large Enough

#283

Post by SebaSeba »

Sorry for the slightly off topic question, but can someone enlighten me which Claude model is more capable, Opus or Sonnet 3.5? I am confused because I see people fuzzing about Sonnet 3.5 being the best and yet somehow I seem to read again and again in factual texts and some benchmarks that Claude Opus is the most capable. Is there a simple answer to the question, what do I not understand? Please, thank you.

Re: Large Enough

#284

Post by ipsod »

ketzo wrote: post #79
rkwz wrote: post #73

Earlier quoted context omitted.

> Longer context window (1M+) What's your use case for this? Uploading multiple documents/books?

Uploading large codebases is particularly useful.

Is it?

I've found that I get better results if I cherry pick code to feed to Claude 3.5, instead of pasting whole files.

I'm kind of isolated, though, so maybe I just don't know the trick.

Re: Large Enough

#285

Post by X6S1x6Okd1st »

splwjs wrote: post #78
__jl__ wrote: post #34

Earlier quoted context omitted.

Meta just claimed the opposite in their Llama 3.1 paper. Look at the conclusion. They say that their experience indicates significant gains for the next iteration of models. The current crop of benchmarks might not reflect these gains, by the way.

I sell widgets. I promise the incalculable power of widgets has yet to be unleashed on the world, but it is tremendous and awesome and we should all be very afraid of widgets taking over the world because I can't see how they won't. Anyway here's the sales page. the widget subscription is so premium you won't even miss the subscription fee.

But Meta isn't selling it

Re: Large Enough

#287

Post by probably_wrong »

pegasus wrote: post #267
probably_wrong wrote: post #263

Earlier quoted context omitted.

I would counterargue with "that's the model's problem, not mine". Here's a thought experiment: if I gave you 5 boxes and told you "how many balls are there in all of this boxes?" and you answered "I don't know because they are inside boxes", that's a fail. A truly intelligent individual would open them and look inside. A truly intelligent model would (say) retokenize the word into its individual letters (which I'm op…

Of course they lack building blocks for full intelligence. They are good at certain tasks, and counting letters is emphatically not one of them. They should be tested and compared on the kind of tasks they're fit for, and so the kind of tasks they will be used in solving, not tasks for which they would be misemployed to begin with.

I agree with you, but that's not what the post claims. From the article:

"A significant effort was also devoted to enhancing the model’s reasoning capabilities. (...) the new Mistral Large 2 is trained to acknowledge when it cannot find solutions or does not have sufficient information to provide a confident answer."

Words like "reasoning capabilities" and "acknowledge when it does not have enough information" have meanings. If Mistral doesn't add footnotes to those assertions then, IMO, they don't get to backtrack when simple examples show the opposite.

Re: Large Enough

#288

Post by mountainriver »

TIPSIO wrote: post #3

This race for the top model is getting wild. Everyone is claiming to one-up each with every version. My experience (benchmarks aside) Claude 3.5 Sonnet absolutely blows everything away. I'm not really sure how to even test/use Mistral or Llama for everyday use though.

It’s so weird LMsys doesn’t reflect that then.

I find it funny how in threads like this everyone swears one model is better than another

Re: Large Enough

#289

Post by m2024 »

drewnick wrote: post #27
tikkun wrote: post #5

Links to chat with models that released this week: Large 2 - https://chat.mistral.ai/chat Llama 3.1 405b - https://www.llama2.ai/ I just tested Mistral Large 2 and Llama 3.1 405b on 5 prompts from my Claude history. I'd rank as: 1. Sonnet 3.5 2. Large 2 and Llama 405b (similar, no clear winner between the two) If you're using Claude, stick with it. My Claude wishlist: 1. Smarter (yes, it's the most intelligent, and y…

All 3 models you ranked cannot get "how many r's are in strawberry?" correct. They all claim 2 r's unless you press them. With all the training data I'm surprised none of them fixed this yet.

Does anyone have input on the feasibility of running an LLM locally and providing an interface to some language runtime and storage space, possibly via a virtual machine or container?

No idea if there's any sense to this, but an LLM could be instructed to formulate and continually test mathematical assumptions by writing / running code and fine-tuning accordingly.

Re: Large Enough

#290

Post by jcheng »

paulcole wrote: post #254
ChikkaChiChi wrote: post #46

Earlier quoted context omitted.

4o will get the answer right on the first go if you ask it "Search the Internet to determine how many R's are in strawberry?" which I find fascinating

I didn't even need to do that. 4o got it right straight away with just: "how many r's are in strawberry?" The funny thing is, I replied, "Are you sure?" and got back, "I apologize for the mistake. There are actually two 'r's in the word strawberry."

GPT-4o-mini consistently gives me this:

> How many times does the letter “r” appear in the word “strawberry”?

> The letter "r" appears 2 times in the word "strawberry."

But also:

> How many occurrences of the letter “r” appear in the word “strawberry”?

> The word "strawberry" contains three occurrences of the letter "r."

Post reply on HN