Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Liquid Foundation Models: Our First Series of Generative AI Models

liquid.ai

Post reply on HN
Print view View original

131–140 of 159 posts

Re: Liquid Foundation Models: Our First Series of Generative AI Models

#131

Post by marcosdumay »

lbotos wrote: post #129
marcosdumay wrote: post #124

Earlier quoted context omitted.

That explanation would require the LLM to actually understand the question and deriving an answer from first principles. It doesn't.

?? If the input is parsed in to tokens, and the tokens split compound words, nothing about that requires "first principles" thinking to explain why LLMs struggle with getting all of the letters -- the LLM is only going down the vector path of one of the compound words... (I don't think LLMs are sentiment or intelligent btw, I think they are giant probability machines, and the probability that the LLM will get 3 r's o…

The LLM gives you the answer it finds on the training set. All the things on that article are irrelevant for the answer.

Re: Liquid Foundation Models: Our First Series of Generative AI Models

#132

Post by lumost »

k__ wrote: post #58
GaggiX wrote: post #40

Earlier quoted context omitted.

Yes, I agree, for these small models it's wasted potential to be closed source, they can only be used effectively if they are open. EDIT: HN is rate-limiting me so I will reply here: In my opinion 1B and 3B truly shine on edge devices, if not than it's not worth the effort, you can have much better models for already dirt cheap using an API.

While I'm all for open models; why can't the small models not be used effectively? Wouldn't they lower the costs compared to big models drastically?

An open small model means I can experiment with it. I can put it on an edge device and scale to billions of users, I can use it with private resources that I can't send externally.

When it's behind an API its just a standard margin/speed/cost discussion.

Re: Liquid Foundation Models: Our First Series of Generative AI Models

#133

Post by lbotos »

marcosdumay wrote: post #131
lbotos wrote: post #129

Earlier quoted context omitted.

?? If the input is parsed in to tokens, and the tokens split compound words, nothing about that requires "first principles" thinking to explain why LLMs struggle with getting all of the letters -- the LLM is only going down the vector path of one of the compound words... (I don't think LLMs are sentiment or intelligent btw, I think they are giant probability machines, and the probability that the LLM will get 3 r's o…

The LLM gives you the answer it finds on the training set. All the things on that article are irrelevant for the answer.

"The choice of tokenization method can directly affect the accuracy of character counting. If the tokenization method obscures the relationship between individual characters, it can be difficult for the LLM to count them accurately. For example, if "strawberry" is tokenized as "straw" and "berry," the LLM may not recognize that the two "r"s are part of the same word.

To improve character counting accuracy, LLMs may need to use more sophisticated tokenization methods, such as subword tokenization or character-level tokenization, that can preserve more information about the structure of words."

Re: Liquid Foundation Models: Our First Series of Generative AI Models

#135

Post by viccis »

alach11 wrote: post #33
zozbot234 wrote: post #30

> How many r's in "strawberry"? Think step by step! What a nice question! Well, let's just count: 1. S 2. T 3. R! So, we have our answer: there are 3 R's in "strawberry". Totally checks out!

This is one of the least interesting questions to ask LLMs. I wish it wasn't so popular.

It's also the case that the more people talk about it, the more future models will "solve" it just by having learned from discussion about it.

Re: Liquid Foundation Models: Our First Series of Generative AI Models

#136

Post by [deleted] »

hn_throwaway_99 wrote: post #102
doctorpangloss wrote: post #75

Earlier quoted context omitted.

Counting shit, like cells, peaks in signals, people, inventory, fingers, and votes, is hard, tedious and important to business and life, so I don’t know dude, it seems like a great benchmark to me. Countless posts wasted on denying this simple and obvious fact.

Not being able to "do math" is an obvious, IMO uninteresting limitation of how LLMs inherently work, and the more advanced models have already figured out good solutions. E.g. while an LLM may stumble on "How many R's are in Strawberry", at this point any code model can easily correctly implement "write me a program in Python that counts the number of R's in strawberry". GPT 3 would essentially always fail on higher…

[deleted]

Re: Liquid Foundation Models: Our First Series of Generative AI Models

#137

Post by [deleted] »

BoorishBears wrote: post #96
doctorpangloss wrote: post #75

Earlier quoted context omitted.

Counting shit, like cells, peaks in signals, people, inventory, fingers, and votes, is hard, tedious and important to business and life, so I don’t know dude, it seems like a great benchmark to me. Countless posts wasted on denying this simple and obvious fact.

It's like using a hammer to turn a screw and calling it useless. To envision what a next generation model bound by the same constraints should do, it'd be to recognize that it can't count tokens and use code access to write code that solves the strawberry problem without prompting. Asked to count cells it'd be a model that could write and execute OpenCV tasks. Or to go a step further, be a multimodal model that can s…

[deleted]

Re: Liquid Foundation Models: Our First Series of Generative AI Models

#138

Post by doctorpangloss »

Scarblac wrote: post #97
doctorpangloss wrote: post #75

Earlier quoted context omitted.

Counting shit, like cells, peaks in signals, people, inventory, fingers, and votes, is hard, tedious and important to business and life, so I don’t know dude, it seems like a great benchmark to me. Countless posts wasted on denying this simple and obvious fact.

So how many 1 bits are in strawberry? Thats about the same kind of question, given that LLMs see tokens instead of characters, and we see characters instead of bits.

Do you sincerely think the folks training these models haven't tried alternative tokenizations?

Re: Liquid Foundation Models: Our First Series of Generative AI Models

#139

Post by doctorpangloss »

BoorishBears wrote: post #96
doctorpangloss wrote: post #75

Earlier quoted context omitted.

Counting shit, like cells, peaks in signals, people, inventory, fingers, and votes, is hard, tedious and important to business and life, so I don’t know dude, it seems like a great benchmark to me. Countless posts wasted on denying this simple and obvious fact.

It's like using a hammer to turn a screw and calling it useless. To envision what a next generation model bound by the same constraints should do, it'd be to recognize that it can't count tokens and use code access to write code that solves the strawberry problem without prompting. Asked to count cells it'd be a model that could write and execute OpenCV tasks. Or to go a step further, be a multimodal model that can s…

> To envision what a next generation model bound by the same constraints should do, it'd be to recognize that it can't count tokens and use code access to write code that solves the strawberry problem without prompting.

The VQA problems I'm describing can be solved seemingly in one case but not combined with counting. Counting is fundamentally challenging for sort of unknown reasons, or perhaps known to the very best labs who are trying to tackle it directly.

Another POV is that the stuff you are describing is in some sense so obvious that it has been tried, no?

Re: Liquid Foundation Models: Our First Series of Generative AI Models

#140

Post by homarp »

hn_throwaway_99 wrote: post #127
WhitneyLand wrote: post #120

Earlier quoted context omitted.

I don’t think we’ve yet come to the point where, how an LLM end to end goes from prompt to output is blatantly obvious. LLMs operate with language at multiple levels of abstraction and tokens are not the only way to have character level knowledge. For example, nothing excludes pre-training data from directly or indirectly encoding such knowledge. And of course LLMs famously have emergent properties, for which there’s…

The fact that large parts of how LLMs work is poorly understood doesn't mean that that we should brush the whole thing with "it's a complex mystery!" We know exactly how LLMs tokenize their input - OpenAI even has a public tool you can test out, https://platform.openai.com/tokenizer . When you say "For example, nothing excludes pre-training data from directly or indirectly encoding such knowledge." - yes, that's why…

teaching LLM to say 'i don't know' in general is hard, not just for counting letters.
Post reply on HN