Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Enough

mistral.ai

Post reply on HN
Print view View original

261–270 of 512 posts

Re: Large Enough

#261

Post by duchenne »

thntk wrote: post #216

Anyone know what caused the very big performance jump from Large1 to Large2 in just a few months? Besides, parameter redundancy seems evidenced. Front-tier models used to be 1.8T, then 405B, and now 123B. Would front-tier models in the future be <10B or even <1B, that would be a game changer.

Counter-intuitively, larger models are cheaper to train. However, smaller models are cheaper to serve. At first, everyone was focusing on training, so the models were much larger. Now, so many people are using AI everyday, so companies spend more on training smaller models to save on serving.

Re: Large Enough

#262

Post by mattnewton »

layer8 wrote: post #173
mattnewton wrote: post #130

Earlier quoted context omitted.

You can always find something to be unimpressed by I suppose, but the fact that this was fixable with plain english is impressive enough to me.

The technology is frustrating because (a) you never know what may require fixing, and (b) you never know if it is fixable by further instructions, and if so, by which ones. You also mostly* cannot teach it any fixes (as an end user). Using it is just exhausting. *) that is, except sometimes by making adjustments to the system prompt

I think this particular example, of counting letters, is obviously going to be hard when you know how tokenization works. It's totally possible to develop an intuition for other times things will work or won't work, but like all ML powered tools, you can't hope for 100% accuracy. The best you can do is have good metrics and track performance on test sets.

I actually think the craziest part of LLMs is that how, as a developer or SME, just how much you can fix with plain english prompting once you have that intuition. Of course some things aren't fixable that way, but the mere fact that many cases are fixable simply by explaining the task to the model better in plain english is a wildly different paradigm! Jury is still out but I think it's worth being excited about, I think that's very powerful since there are a lot more people with good language skills than there are python programmers or ML experts.

Re: Large Enough

#263

Post by probably_wrong »

generalizations wrote: post #221
drewnick wrote: post #27

Earlier quoted context omitted.

All 3 models you ranked cannot get "how many r's are in strawberry?" correct. They all claim 2 r's unless you press them. With all the training data I'm surprised none of them fixed this yet.

Testing models on their tokenization has always struck me as kinda odd. Like, that has nothing to do with their intelligence.

I would counterargue with "that's the model's problem, not mine".

Here's a thought experiment: if I gave you 5 boxes and told you "how many balls are there in all of this boxes?" and you answered "I don't know because they are inside boxes", that's a fail. A truly intelligent individual would open them and look inside.

A truly intelligent model would (say) retokenize the word into its individual letters (which I'm optimistic they can) and then would count those. The fact that models cannot do this is proof that they lack some basic building blocks for intelligence. Model designers don't get to argue "we are human-like except in the tasks where we are not".

Re: Large Enough

#264

Post by SirMaster »

jonas21 wrote: post #228
layer8 wrote: post #62

Earlier quoted context omitted.

It’s not impressive that one has to go to that length though.

To be fair, I just asked a real person and had to go to even greater lengths: Me: How many "r"s are in strawberry? Them: What? Me: How many times does the letter "r" appear in the word "strawberry"? Them: Is this some kind of trick question? Me: No. Just literally, can you count the "r"s? Them: Uh, one, two, three. Is that right? Me: Yeah. Them: Why are you asking me this?

Try asking a young child...

Re: Large Enough

#265

Post by stitched2gethr »

drewnick wrote: post #27
tikkun wrote: post #5

Links to chat with models that released this week: Large 2 - https://chat.mistral.ai/chat Llama 3.1 405b - https://www.llama2.ai/ I just tested Mistral Large 2 and Llama 3.1 405b on 5 prompts from my Claude history. I'd rank as: 1. Sonnet 3.5 2. Large 2 and Llama 405b (similar, no clear winner between the two) If you're using Claude, stick with it. My Claude wishlist: 1. Smarter (yes, it's the most intelligent, and y…

All 3 models you ranked cannot get "how many r's are in strawberry?" correct. They all claim 2 r's unless you press them. With all the training data I'm surprised none of them fixed this yet.

Interestingly enough much simpler models can write an accurate function to give you the answer.

I think it will be a while before we get there. An LLM can lookup knowledge but can't actually perform calculations itself, without some external processor.

Re: Large Enough

#266

Post by mvdtnz »

Imagine bragging about 74% accuracy in any other field of software. You'd be laughed out of the room. But somehow it's accepted in "AI".

Re: Large Enough

#267

Post by pegasus »

probably_wrong wrote: post #263
generalizations wrote: post #221

Earlier quoted context omitted.

Testing models on their tokenization has always struck me as kinda odd. Like, that has nothing to do with their intelligence.

I would counterargue with "that's the model's problem, not mine". Here's a thought experiment: if I gave you 5 boxes and told you "how many balls are there in all of this boxes?" and you answered "I don't know because they are inside boxes", that's a fail. A truly intelligent individual would open them and look inside. A truly intelligent model would (say) retokenize the word into its individual letters (which I'm op…

Of course they lack building blocks for full intelligence. They are good at certain tasks, and counting letters is emphatically not one of them. They should be tested and compared on the kind of tasks they're fit for, and so the kind of tasks they will be used in solving, not tasks for which they would be misemployed to begin with.

Re: Large Enough

#268

Post by SirMaster »

generalizations wrote: post #221
drewnick wrote: post #27

Earlier quoted context omitted.

All 3 models you ranked cannot get "how many r's are in strawberry?" correct. They all claim 2 r's unless you press them. With all the training data I'm surprised none of them fixed this yet.

Testing models on their tokenization has always struck me as kinda odd. Like, that has nothing to do with their intelligence.

How is a layman supposed to even know that it's testing on that? All they know is it's a large language model. It's not unreasonable they should expect it to be good at things having to do with language, like how many letters are in a word.

Seems to me like a legit question for a young child to answer or even ask.

Re: Large Enough

#269

Post by qeternity »

wruza wrote: post #248
generalizations wrote: post #221

Earlier quoted context omitted.

Testing models on their tokenization has always struck me as kinda odd. Like, that has nothing to do with their intelligence.

It doesn’t test “on tokenization” though. What happens when an answer is generated is few abstraction levels deeper than tokens. A “thinking” “slice” of an llm is completely unaware of tokens as an immediate part of its reasoning. The question just shows lack of systemic knowledge about strawberry as a word (which isn’t surprising, tbh).

It is. Strawberry is one token in many tokenziers. The model doesn't have a concept that there are letters there.

Re: Large Enough

#270

Post by meroes »

generalizations wrote: post #221
drewnick wrote: post #27

Earlier quoted context omitted.

All 3 models you ranked cannot get "how many r's are in strawberry?" correct. They all claim 2 r's unless you press them. With all the training data I'm surprised none of them fixed this yet.

Testing models on their tokenization has always struck me as kinda odd. Like, that has nothing to do with their intelligence.

I hear this a lot but there are vast sums of money thrown at where a model fails the strawberry cases.

Think about math and logic. If a single symbol is off, it’s no good.

Like a prompt where we can generate a single tokenization error at my work, by my very rough estimates, generates 2 man hours of work. (We search for incorrect model responses, get them to correct themselves, and if they can’t after trying, we tell them the right answer, and edit it for perfection). Yes even for counting occurrences of characters. Think about how applicable that is. Finding the next term in a sequence, analyzing strings, etc.

Post reply on HN