Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Enough

mistral.ai

Post reply on HN
Print view View original

271–280 of 512 posts

Re: Large Enough

#271

Post by emmelaich »

Spivak wrote: post #95
layer8 wrote: post #62

Earlier quoted context omitted.

It’s not impressive that one has to go to that length though.

To me it's just a limitation based on the world as seen by these models. They know there's a letter called 'r', they even know that some words start with 'r' or have r's in them, and they know what the spelling of some words is. But they've never actually seen one in as their world is made up entirely of tokens. The word 'red' isn't r-e-d but is instead like a pictogram to them. But they know the spelling of strawber…

I think it's more that the question is not unlike "is there a double r in strawberry?' or 'is the r in strawberry doubled?'

Even some people will make this association, it's no surprise that LLMs do.

Re: Large Enough

#272

Post by pegasus »

antisthenes wrote: post #233
soneca wrote: post #206

Earlier quoted context omitted.

This is only an issue if you send commands to a LLM as you were communicating to a human.

> This is only an issue if you send commands to a LLM as you were communicating to a human. Yes, it's an issue. We want the convenience of sending human-legible commands to LLMs and getting back human-readable responses. That's the entire value proposition lol.

Far from the entire value proposition. Chatbots are just one use of LLMs, and not the most useful one at that. But sure, the one "the public" is most aware of. As opposed to "the hackers" that are supposed to frequent this forum. LOL

Re: Large Enough

#273

Post by medmunds »

drewnick wrote: post #27
tikkun wrote: post #5

Links to chat with models that released this week: Large 2 - https://chat.mistral.ai/chat Llama 3.1 405b - https://www.llama2.ai/ I just tested Mistral Large 2 and Llama 3.1 405b on 5 prompts from my Claude history. I'd rank as: 1. Sonnet 3.5 2. Large 2 and Llama 405b (similar, no clear winner between the two) If you're using Claude, stick with it. My Claude wishlist: 1. Smarter (yes, it's the most intelligent, and y…

All 3 models you ranked cannot get "how many r's are in strawberry?" correct. They all claim 2 r's unless you press them. With all the training data I'm surprised none of them fixed this yet.

How much do threads like this provide the training data to convince future generations that—despite all appearances to the contrary—strawberry is in fact spelled with only two R's?

I just researched "how many r's are in strawberry?" in a search engine, and based solely on the results it found, I would have to conclude there is substantial disagreement on whether the correct answer is two or three.

Re: Large Enough

#274

Post by wruza »

imtringued wrote: post #133
Tepix wrote: post #54

Earlier quoted context omitted.

LLMs think in tokens, not letters. It's like asking someone who is dyslexic about spelling. Not their strong suit. In practice, it doesn't matter much, does it?

This doesn't make any sense. The LLM knows the token to letter associations if you ask it. Once it has split the token into letters, it can answer the question anyway. This tells us that the model is unaware of its own limitations, otherwise it would have figured that this tasks needs it to split the letters first.

Does it? It knows how a word in some context represented by tokens split into letters. It may know or not know the table. I wouldn’t trust what it tells about word/token correspondence more than in general.

Re: Large Enough

#275

Post by baq »

generalizations wrote: post #221
drewnick wrote: post #27

Earlier quoted context omitted.

All 3 models you ranked cannot get "how many r's are in strawberry?" correct. They all claim 2 r's unless you press them. With all the training data I'm surprised none of them fixed this yet.

Testing models on their tokenization has always struck me as kinda odd. Like, that has nothing to do with their intelligence.

Call me when models understand when to convert the token into actual letters and count them. Can’t claim they’re more than word calculators before that.

Re: Large Enough

#276

Post by psb217 »

generalizations wrote: post #221
drewnick wrote: post #27

Earlier quoted context omitted.

All 3 models you ranked cannot get "how many r's are in strawberry?" correct. They all claim 2 r's unless you press them. With all the training data I'm surprised none of them fixed this yet.

Testing models on their tokenization has always struck me as kinda odd. Like, that has nothing to do with their intelligence.

How can I know whether any particular question will test a model on its tokenization? If a model makes a boneheaded error, how can I know whether it was due to lack of intelligence or due to tokenization? I think finding places where models are surprisingly dumb is often more informative than finding particular instances where they seem clever.

It's also funny, since this strawberry question is one where a model that's seriously good at predicting the next character/token/whatever quanta of information would get it right. It requires no reasoning, and is unlikely to have any contradicting text in the training corpus.

Re: Large Enough

#277

Post by psb217 »

mattnewton wrote: post #130
layer8 wrote: post #62

Earlier quoted context omitted.

It’s not impressive that one has to go to that length though.

You can always find something to be unimpressed by I suppose, but the fact that this was fixable with plain english is impressive enough to me.

Well, the answer is probably between 1 and 10, so if you try enough prompts I'm sure you'll find one that "works"...

Re: Large Enough

#278

Post by fluoridation »

medmunds wrote: post #273
drewnick wrote: post #27

Earlier quoted context omitted.

All 3 models you ranked cannot get "how many r's are in strawberry?" correct. They all claim 2 r's unless you press them. With all the training data I'm surprised none of them fixed this yet.

How much do threads like this provide the training data to convince future generations that—despite all appearances to the contrary—strawberry is in fact spelled with only two R's? I just researched "how many r's are in strawberry?" in a search engine, and based solely on the results it found, I would have to conclude there is substantial disagreement on whether the correct answer is two or three.

Speaking as a 100% human, my vote goes to the compromise position that "strawberry" has in fact four Rs.

Re: Large Enough

#279

Post by VincentEvans »

generalizations wrote: post #221
drewnick wrote: post #27

Earlier quoted context omitted.

All 3 models you ranked cannot get "how many r's are in strawberry?" correct. They all claim 2 r's unless you press them. With all the training data I'm surprised none of them fixed this yet.

Testing models on their tokenization has always struck me as kinda odd. Like, that has nothing to do with their intelligence.

I don’t know anything about LLMs beyond using ChatGPT and Copilot… but unless because of this lack of knowledge I am misinterpreting your reply - it sounds as if you are excusing the model giving a completely wrong answer to a question that anyone intelligent enough to learn alphabet can answer correctly.

Re: Large Enough

#280

Post by lifty »

freediver wrote: post #22
TIPSIO wrote: post #11

Earlier quoted context omitted.

Have you (or anyone) swapped on Cursor with Anthropic API Key? For coding assistant, it's on my to do list to try. Cursor needs some serious work on model selection clarity though so I keep putting off.

I did it (fairly simple really) but found most of my (unsophisticated) coding these days to go through Aider [1] paired with Sonnet, for UX reasons mostly. It is easier to just prompt over the entire codebase, vs Cursor way of working with text selections. [1] https://aider.chat

Thanks for this suggestion. If anyone has other suggestions for working with large code context windows and changing code workflows, I would love to hear about them.
Post reply on HN