As a LLM layman, does this work build off the attention is all you need paper, or is it doing something completely different?
Liquid Foundation Models: Our First Series of Generative AI Models
11–20 of 159 posts
Re: Liquid Foundation Models: Our First Series of Generative AI Models
#12So a new set of small models that are competitive with and beat phi-3.5 on some benchmarks is extremely impressive. Context length is the next frontier for models in this range - getting to a true 128-200k token length in a small model would be very very useful. Hallucinations are less of an issue because you can just dump all the source data in, whole codebases can go in for stuff ranging from a refactor to ‘write d…
Gemini has the largest context of any model out there FYI.
From my experience these large context are just techinical feasability but there needs to be a lot better internal benchamarks to claim it really works. I've tested on real world task and it all fails so far.
Re: Liquid Foundation Models: Our First Series of Generative AI Models
#13So a new set of small models that are competitive with and beat phi-3.5 on some benchmarks is extremely impressive. Context length is the next frontier for models in this range - getting to a true 128-200k token length in a small model would be very very useful. Hallucinations are less of an issue because you can just dump all the source data in, whole codebases can go in for stuff ranging from a refactor to ‘write d…
Gemini has the largest context of any model out there FYI.
[deleted]
Re: Liquid Foundation Models: Our First Series of Generative AI Models
#14So a new set of small models that are competitive with and beat phi-3.5 on some benchmarks is extremely impressive. Context length is the next frontier for models in this range - getting to a true 128-200k token length in a small model would be very very useful. Hallucinations are less of an issue because you can just dump all the source data in, whole codebases can go in for stuff ranging from a refactor to ‘write d…
Looking at 3B, the performance is roughly on par with phi 3.5.. not sure where how they gauge on their graph that it is better. Agreed though, I don't think I would switch my 3B model from phi unless Liquid was more transparent in data and research.
Re: Liquid Foundation Models: Our First Series of Generative AI Models
#15Love the humour here: What are Language LFMs not good at today: […] Counting r's in the word "Strawberry"!
Can't give them too much credit ;) the task of counting r's in Strawberry was mentioned elsewhere as a "surprising" failure of LLMs
See previous thread: https://news.ycombinator.com/item?id=41058318
Re: Liquid Foundation Models: Our First Series of Generative AI Models
#16Why add a new TLA in “LFM?”
It adds nothing but branding to the conversation.
Maybe i answered my own question…
Re: Liquid Foundation Models: Our First Series of Generative AI Models
#17I tried my standard testprompt:
Two cars have a 100 mile race. Car A drives 10
miles per hour. Car B drives 5 miles per hour,
but gets a 10 hour headstart. Who wins?
Amusingly, the explanation is quite convincing and very similar to how Zeno explains in one of his paradoxes that a fast runner can never overtake a slower runner with a headstart. Because every time the fast runner gets to the point where he saw the slow runner, the slow runner is already a bit further along the track.
Re: Liquid Foundation Models: Our First Series of Generative AI Models
#18I'm curious how this works, it looks like they haven't published yet but two of the founders are authors on this paper:
Re: Liquid Foundation Models: Our First Series of Generative AI Models
#19As a LLM layman, does this work build off the attention is all you need paper, or is it doing something completely different?
They say it's not a transformer.
Re: Liquid Foundation Models: Our First Series of Generative AI Models
#20I tried my standard testprompt: Two cars have a 100 mile race. Car A drives 10 miles per hour. Car B drives 5 miles per hour, but gets a 10 hour headstart. Who wins? And the Liquid-40B model failed with a long explanation why car B wins the race. Amusingly, the explanation is quite convincing and very similar to how Zeno explains in one of his paradoxes that a fast runner can never overtake a slower runner with a hea…
To be fair, this article does say:
>What are Language LFMs not good at today:
>Zero-shot code tasks
>Precise numerical calculations
>Time-sensitive information
>Counting r's in the word "Strawberry"!
>Human preference optimization techniques have not been applied extensively to our models yet.
Maybe it's just not trained/tuned for math-related challenges. Or maybe it's an architectural challenge.