Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Enough

mistral.ai

Post reply on HN
Print view View original

481–490 of 512 posts

Re: Large Enough

#481

Post by brandall10 »

paulcole wrote: post #465
brandall10 wrote: post #444

Earlier quoted context omitted.

I haven't played with this model, but rarely do I find working w/ Claude or GPT-4 for that to be the case. If you say it's incorrect, it will give you another answer instead of insisting on correctness.

Wait what? You haven’t used 4o and you confidently described how it works?

It's how LLMs work in general.

If you find a case where forceful pushback is sticky, it's either because the primary answer is overwhelmingly present in the training set compared to the next best option or because there are conversations in the training that followed similar stickiness, esp. if the structure of the pushback itself is similar to what is found in those conversations.

Re: Large Enough

#482

Post by crowcroft »

breck wrote: post #68
crowcroft wrote: post #59

Earlier quoted context omitted.

Could you elaborate on this? Would love to understand what leads you to this conclusion.

E = T/A! [0] A faster evolving approach to AI is coming out this year that will smoke anyone who still uses the term "license" in regards to ideas [1]. [0] https://breckyunits.com/eta.html [1] https://breckyunits.com/freedom.html

Even if we accept the premise, and the conclusion that this equation 'explains the triumph of open source and public domain software', it explains very little about the relative success of open source/public domain work vs. commercial closed source work.

I think history indicates that some commercial works can also triumph.

Re: Large Enough

#483

Post by doctoboggan »

kim0 wrote: post #112
doctoboggan wrote: post #47

The question I (and I suspect most other HN readers) have is which model is best for coding? While I appreciate the advances in open weights models and all the competition from other companies, when it comes to my professional use I just want the best. Is that still GPT-4?

I kinda trust https://aider.chat/docs/leaderboards/

Have you used the actual aider command line tool? That looks interesting, just wondering if it works well.

Re: Large Enough

#484

Post by breck »

crowcroft wrote: post #482
breck wrote: post #68

Earlier quoted context omitted.

E = T/A! [0] A faster evolving approach to AI is coming out this year that will smoke anyone who still uses the term "license" in regards to ideas [1]. [0] https://breckyunits.com/eta.html [1] https://breckyunits.com/freedom.html

Even if we accept the premise, and the conclusion that this equation 'explains the triumph of open source and public domain software', it explains very little about the relative success of open source/public domain work vs. commercial closed source work. I think history indicates that some commercial works can also triumph.

> it explains very little about the relative success of open source/public domain work vs. commercial closed source work.

It explains everything.

Open source evolves faster, and always out competes closed source.

> I think history indicates that some commercial works can also triumph.

Nope. Not in the long run. I can't think of a single exception.

Re: Large Enough

#485

Post by observationist »

WhitneyLand wrote: post #443
Kuinox wrote: post #49

Earlier quoted context omitted.

Tokenization make it hard for it to count the letters, that's also why if you ask it to do maths, writing the number in letters will yield better results. for strawberry, it see it as [496, 675, 15717], which is str aw berry. If you insert characters to breaks the tokens down, it find the correct result: how many r's are in "s"t"r"a"w"b"e"r"r"y" ? > There are 3 'r's in "s"t"r"a"w"b"e"r"r"y".

Where did you get this idea from? Tokens aren’t the source of facts within a model. it’s an implementation detail and doesn’t inherently constrain how things could be counted.

Tokens are the first form of information being encoded into the model. They're statistically guided, more or less a compression dictionary comparable to a Lempel Ziv setup.

Combinations of tokens get encoded, so if the feature isn't part of the information being carried forward into the network as it models the information in the corpus, the feature isn't modeled well, or at all. The consequence of having many character tokens is that the relevance of individual characters is lost, and you have to explicitly elicit the information. Models know that words have individual characters, but "strawberry" isn't encoded as a sequence of letters, it's encoded as an individual feature of the tokenizer embedding.

Other forms of tokenizing have other tradeoffs. The trend lately is to increase tokenizer dictionary scope, up to 128k in Llama3 from 50k in gpt-3. The more tokens, the more nuanced individual embedding features in that layer can be before downstream modeling.

Tokens inherently constrain how the notion of individual letters are modeled in the context of everything an LLM learns. In a vast majority of cases, the letters don't matter, so the features don't get mapped and carried downstream of the tokenizer.

Re: Large Enough

#486

Post by jvm___ »

coder543 wrote: post #229
GenerWork wrote: post #186

Earlier quoted context omitted.

>If you insert characters to breaks the tokens down, it find the correct result: how many r's are in "s"t"r"a"w"b"e"r"r"y" ? The issue is that humans don't talk like this. I don't ask someone how many r's there are in strawberry by spelling out strawberry, I just say the word.

> I don't ask someone how many r's there are in strawberry by spelling out strawberry, I just say the word. No, I would actually be pretty confident you don’t ask people that question… at all. When is the last time you asked a human that question? I can’t remember ever having anyone in real life ask me how many r’s are in strawberry. A lot of humans would probably refuse to answer such an off-the-wall and useless que…

AI being the same as human, while AI fails at a task that any human can easily do means AI isn't human equivalent in an easily demonstrable way.

If full artificial intelligence, as we're being promised, falls short in this simple way.

Re: Large Enough

#487

Post by coder543 »

abdullahkhalids wrote: post #305
qeternity wrote: post #269

Earlier quoted context omitted.

It is. Strawberry is one token in many tokenziers. The model doesn't have a concept that there are letters there.

If I ask an LLM to generate new words for some concept or category, it can do that. How do the new words form, if not from joining letters?

The tokenizer system supports virtually any input text that you want, so it follows that it also allows virtually any output text. It isn’t limited to a dictionary of the 1000 most common words or something.

There are tokens for individual letters, but the model is not trained on text written with individual tokens per letter, it is trained on text that has been converted into as few tokens as possible. Just like you would get very confused if someone started spelling out entire sentences as they spoke to you, expecting you to reconstruct the words from the individual spoken letters, these LLMs also would perform terribly if you tried to send them individual tokens per letter of input (instead of the current tokenizer scheme that they were trained on).

Even though you might write a message to an LLM, it is better to think of that as speaking to the LLM. The LLM is effectively hearing words, not reading letters.

Re: Large Enough

#488

Post by paulcole »

brandall10 wrote: post #481
paulcole wrote: post #465

Earlier quoted context omitted.

Wait what? You haven’t used 4o and you confidently described how it works?

It's how LLMs work in general. If you find a case where forceful pushback is sticky, it's either because the primary answer is overwhelmingly present in the training set compared to the next best option or because there are conversations in the training that followed similar stickiness, esp. if the structure of the pushback itself is similar to what is found in those conversations.

Right... except you said:

> If you say it's incorrect, it will give you another answer instead of insisting on correctness.

> When you push it, it responds with the next most common answer.

Which clearly isn't as black and white as you made it seem.

Re: Large Enough

#489

Post by Auracle »

generalizations wrote: post #221
drewnick wrote: post #27

Earlier quoted context omitted.

All 3 models you ranked cannot get "how many r's are in strawberry?" correct. They all claim 2 r's unless you press them. With all the training data I'm surprised none of them fixed this yet.

Testing models on their tokenization has always struck me as kinda odd. Like, that has nothing to do with their intelligence.

I suppose what models should have are some instructions of things they aren’t good at and will need to break out into python code or what have you. Humans have an intuition for this - I have basic knowledge of when I need to write something down or use a calculator. LLMs don’t have intuition (yet - though I suppose one could use a smaller model for that), so explicit instructions would work for now.

Re: Large Enough

#490

Post by groby_b »

OKRainbowKid wrote: post #395
groby_b wrote: post #383

Earlier quoted context omitted.

LLMs are not truly intelligent. Never have been, never will be. They model language, not intelligence.

They model the dataset they were trained on. How would a dataset of what you consider intelligence look like?

LLMs are not a tool to model intelligence. It's not a function of the dataset, they are, as is, not sufficient. One of the largest shortcomings being the lack of continuous learning, memory, and (likely) forgetting
Post reply on HN