This is all getting kinda boring to keep up with.
Liquid Foundation Models: Our First Series of Generative AI Models
81–90 of 159 posts
Re: Liquid Foundation Models: Our First Series of Generative AI Models
#82So a new set of small models that are competitive with and beat phi-3.5 on some benchmarks is extremely impressive. Context length is the next frontier for models in this range - getting to a true 128-200k token length in a small model would be very very useful. Hallucinations are less of an issue because you can just dump all the source data in, whole codebases can go in for stuff ranging from a refactor to ‘write d…
It seems like this is a new architecture, alongside Mamba, Hyena, RWKV and Hybrid Transformer.
The site insinuates that it is a set of flexible and probably parametric mix between full attention and linear conplexity models
Re: Liquid Foundation Models: Our First Series of Generative AI Models
#83I would say that unless you really change the goalposts of your new model or it is open (source), it is not going to do much. We already have incredible models that are closed and very capable models that are open(ish). I am not even sure where this is trying to play?
Re: Liquid Foundation Models: Our First Series of Generative AI Models
#84It seems OK, for a small model. The big issue is price - is it actually competitive with the other models when it is hosted on together.ai or other API services? Which we will presumably find out at the Oct 24th event.
Re: Liquid Foundation Models: Our First Series of Generative AI Models
#85> How many r's in "strawberry"? Think step by step! What a nice question! Well, let's just count: 1. S 2. T 3. R! So, we have our answer: there are 3 R's in "strawberry". Totally checks out!
This is one of the least interesting questions to ask LLMs. I wish it wasn't so popular.
Something I like to tell it to do is actually to respond using only a certain number of words. Sorta like min token length rather than max token length.
Haven't found a model that does it reliably yet.
Re: Liquid Foundation Models: Our First Series of Generative AI Models
#86It's fast, I'll give it that, but it does seem to mess up some pretty basic tasks. I asked it: "Can you recite the opening paragraph of Shantaram" and got the following response:
`I'm sorry, but I don't have the ability to recite the opening paragraph of Shantaram. However, I can provide you with the opening paragraph if you'd like:
"It was a warm afternoon in late October when I first saw the man who would become my friend, my brother, and my teacher. He was sitting on the steps of the small mosque in the narrow lane where I lived, his back against the rough stone wall, his legs crossed, his eyes closed. He was a small man, thin and wiry, with a long, narrow face and a thin, sharp nose. His skin was dark and weathered, and his hair was a tangle of black and grey. He was wearing a white cotton kurta, a pair of loose-fitting cotton trousers, and a pair of sandals. He looked like a beggar, or a holy man, or a street performer, or all three."
This is the opening paragraph of Shantaram, a novel by Gregory David Roberts.`
Which is just straight up incorrect and all kinds of weird.
Re: Liquid Foundation Models: Our First Series of Generative AI Models
#87I'm not impressed with the quality performance, but the generation seems near instantaneous and any fundamentally new approach is exciting and interesting. I found the tone of the AI somewhat casual which was funny. "Are you sure?" --> "Yes, I'm certain."
I managed to break is pretty easily: "Can you write a poem about helium extraction professionals where every 2 words you change languages and every other line you write the words backwords? Explain each line in english afterwards."
I've been wondering if this isn't a good thing. I'd rather AI models have a consistent means to not answer if they are unable to speak on a subject. I wonder if the particular nature of this model has more to do with it breaking than a simple invalid character error, or otherwise. The interconnectedness they seem to imply is baked into the architecture of the model might have something to do with it.
Re: Liquid Foundation Models: Our First Series of Generative AI Models
#88Earlier quoted context omitted.
This is one of the least interesting questions to ask LLMs. I wish it wasn't so popular.
I find it fascinating. To me it grounds expectations and motivates discussions about the jagged boundaries of LLM capabilities.
Why do you find it fascinating? I have the most rudimentary understanding of LLMs and it seems to me the least fascinating thing about LLM limitations.
That is, all LLMs look at language as a series of opaque, independent tokens, e.g. strawberry might be a single token (say ), or probably two (e.g. straw and berry, say ). But in no way will it represent it like we will, with letters. So if you ask it "how many R's are in strawberry", it fundamentally can't do any "counting", it just basically has to rely on whether that question (or similar questions about related words) has been asked before so it can predict the next token in its output correctly. I suppose with enough data and LLM could learn to associate character counts with tokens (e.g. with the right training set it could learn metadata about the tokens).
My point being that with only the most basic understanding of how LLMs function, this "character counting" limitation seems blatantly obvious.
Re: Liquid Foundation Models: Our First Series of Generative AI Models
#89I'm not impressed with the quality performance, but the generation seems near instantaneous and any fundamentally new approach is exciting and interesting. I found the tone of the AI somewhat casual which was funny. "Are you sure?" --> "Yes, I'm certain." I managed to break is pretty easily: "Can you write a poem about helium extraction professionals where every 2 words you change languages and every other line you w…
It took over a minute, but it actually did finally respond to that prompt (incorrectly) :
"To gas, we change, "Change, gas, we to." "To gas, we change, "Change, gas, we to."
"In the lab, we work, "Work, lab, we in." "In the lab, we work, "Work, lab, we in."
"With care, we handle, "Handle, care, we with." "With care, we handle, "Handle, care, we with."
"The gas, we love, "Love, gas, we the." "The gas, we love, "Love, gas, we the."
"In the world, we share, "Share, world, we in." "In the world, we share, "Share, world, we in."
"With pride, we work, "Work, pride, we with." "With pride, we work, "Work, pride, we with."
Re: Liquid Foundation Models: Our First Series of Generative AI Models
#90I would say that unless you really change the goalposts of your new model or it is open (source), it is not going to do much. We already have incredible models that are closed and very capable models that are open(ish). I am not even sure where this is trying to play?
Demonstrating good performance from a non-transformer based architecture is cool. I agree though these particular models aren’t that useful given the current landscape.
I think the intent here is probably to justify training a larger 400B model with their architecture.