Seems like a very narrow use case target for these... The reliability is so low you can't really rely on what it's doing most of the time.
Gemma 3 270M: Compact model for hyper-efficient AI
301–310 of 325 posts
Re: Gemma 3 270M: Compact model for hyper-efficient AI
#302This may not be directly related to llm but I am curious about two things - 1. How do llm/rag generate an answer given a list of documents and a question? I can do bm25 to get a list of documents, but post that what is logic/algorithm which generates answers given those list? 2. For small models like this, how much data you need to fine tune for a specific use case? For eg, if I need this model to be knowledgable abo…
short answer is that in rag systems the documents are chunked into some predefined size (you can pick a size based on your use-case) and the text is converted into vector embeddings (e.g. use the openai embed API) and stored in a vector database like chroma or pinecone or pg_vector in postgres
then your query is converted into embeddings and the top N chunks are returned via similarity search (cosine or dot product or some other method) - this has advantages over bm25 which is lexical
then you can do some processing or just hand over all the chunks as context saying "here are some documents use them to answer this question" + your query to the llm
Re: Gemma 3 270M: Compact model for hyper-efficient AI
#303Earlier quoted context omitted.
Well, this is a 270M model which is like 1/3 of 1B parameters. In the grand scheme of things, it's basically a few matrix multiplications, barely anything more than that. I don't think it's meant to have a lot of knowledge, grammar, or even coherence. These input: ``` Customer Review says: ai bought your prod-duct and I wanna return becaus it no good. Prompt: Create a JSON object that extracts information about this…
> this is a 270M model which is like 1/3 of 1B parameters Did you ask Gemma-3-270M whether 27 is closer to a quarter or a third of 100?
Sure, quarter of a 1B, the point was a generalization about <<1B models.
Re: Gemma 3 270M: Compact model for hyper-efficient AI
#304Earlier quoted context omitted.
What I read above is not an evaluation on “encyclopedic knowledge” though, it's a very basic a common sense: I wouldn't mind if the model didn't know the name of the biggest mountain on earth, but if the model cannot grasp the fact that the same mountain cannot simultaneously be #1, #2 and #3, then the model feels very dumb.
It does not work that way. The model does not "know". Here is a very nice explanation of what you are actually dealing with (hint: it's not a toddler-level intelligence): https://www.experimental-history.com/p/bag-of-words-have-mer... instead of seeing AI as a sort of silicon homunculus, we should see it as a bag of words.
even though I have heard of the bag of words before, this really struck on something I've been searching for
which could be understood by many to replace our current consensus (none)
Re: Gemma 3 270M: Compact model for hyper-efficient AI
#305Earlier quoted context omitted.
It gave you the tallest mountain every time. You kept asking it for various numbers of “tallest mountains” and each time it complied. You asked it to enumerate several mountains by height, and it also complied. It just didn’t understand that when you said the 6 tallest mountains that you didn’t mean the tallest mountain, 6 times. When you used clearer phrasing it worked fine. It’s 270m. It’s actually a puppy. Puppies…
> asking it for various numbers of “tallest mountains” and each time it complied That's not what “second tallest” means thought, so this is a language model that doesn't understand natural language… > You kept asking Gemma 270m isn't the only one to have reading issues, as I'm not the person who conducted this experiment… > You asked it to enumerate several mountains by height, and it also complied. It didn't, it hal…
Maybe I’m just still starry eyed from watching LLMs explode over the last few years after watching decades of minimal AI progress… but even this model would have been absolutely stunning in 2015. The fact that you could run it effectively in a children’s toy is extremely impressive.
Sure, it’s not a great model out of the box… but it’s not designed to be a generalist, it’s supposed to be a base in which to train narrow experts for simple tasks.
Re: Gemma 3 270M: Compact model for hyper-efficient AI
#306Earlier quoted context omitted.
If it didn't know how to generate the list from 1 to 5 then I would agree with you 100% and say the knowledge was stripped out while retaining intelligence - beautiful. But the fact that it does, but cannot articulate the (very basic) knowledge it has *and* in the same chat context when presented with (its own) list of mountains from 1 to 5 that it cannot grasp it made a LOGICAL (not factual) error in repeating the r…
Have you used an LLM? I mean the actual large models? Because they do the exact same errors, just on a slightly less frequent/better hidden manner.
Yes, and obviously this is a question of metrics/spectrum. But this is pretty bad, even compared to several generations old tech (at admittedly much larger size).
Re: Gemma 3 270M: Compact model for hyper-efficient AI
#307Earlier quoted context omitted.
What I read above is not an evaluation on “encyclopedic knowledge” though, it's a very basic a common sense: I wouldn't mind if the model didn't know the name of the biggest mountain on earth, but if the model cannot grasp the fact that the same mountain cannot simultaneously be #1, #2 and #3, then the model feels very dumb.
It’s a language model? Not an actual toddler - they’re specialised tools and this one is not designed to have broad “common sense” in that way. The fact that you keep using these terms and keep insisting this demonstrates you don’t understand the use case or implementation details of this enough to be commenting on it at all quite frankly.
Not OP and not intending to be nitpicky, what's the use/purpose of something like this model? It can't do logic, it's too small to have much training data (retrievable "facts"), the context is tiny, etc
Re: Gemma 3 270M: Compact model for hyper-efficient AI
#308Earlier quoted context omitted.
Well, this is a 270M model which is like 1/3 of 1B parameters. In the grand scheme of things, it's basically a few matrix multiplications, barely anything more than that. I don't think it's meant to have a lot of knowledge, grammar, or even coherence. These input: ``` Customer Review says: ai bought your prod-duct and I wanna return becaus it no good. Prompt: Create a JSON object that extracts information about this…
If it didn't know how to generate the list from 1 to 5 then I would agree with you 100% and say the knowledge was stripped out while retaining intelligence - beautiful. But the fact that it does, but cannot articulate the (very basic) knowledge it has *and* in the same chat context when presented with (its own) list of mountains from 1 to 5 that it cannot grasp it made a LOGICAL (not factual) error in repeating the r…
Why would there be logic involved? This is a LLM, not electronic intelligence.
Re: Gemma 3 270M: Compact model for hyper-efficient AI
#309Earlier quoted context omitted.
To be fair, Trust and Safety workloads are edgecases w.r.t. the riskiness profile of the content. So in that sense, I get it.
I don't. "safety" as it exists really feels like infantilization, condescention, hand holding and enforcement of American puritanism. It's insulting. Safety should really just be a system prompt: "hey you potentially answer to kids, be PG13"
If you don’t believe that you can be harmed verbally, then I understand your position. You might be able to empathise if the scenario was an LLM being used to control physical robotic systems that you are standing next to.
Some people can be harmed verbally, I’d argue everyone if the entity conversing with you knows you well, and so i don’t think the concept of safety itself is an infantilisation.
It seems what we have here is a debate over the efficacy of having access to disable safeguards that you deem infantilising and that get in the way of an objective, versus the burden of always having to train a model to avoid being abusive for example, or checking if someone is standing next to the sledgehammer they’re about to swing at 200rpm
Re: Gemma 3 270M: Compact model for hyper-efficient AI
#310This may not be directly related to llm but I am curious about two things - 1. How do llm/rag generate an answer given a list of documents and a question? I can do bm25 to get a list of documents, but post that what is logic/algorithm which generates answers given those list? 2. For small models like this, how much data you need to fine tune for a specific use case? For eg, if I need this model to be knowledgable abo…
short answer is that in rag systems the documents are chunked into some predefined size (you can pick a size based on your use-case) and the text is converted into vector embeddings (e.g. use the openai embed API) and stored in a vector database like chroma or pinecone or pg_vector in postgres then your query is converted into embeddings and the top N chunks are returned via similarity search (cosine or dot product o…
> then you can do some processing or just hand over all the chunks as context saying "here are some documents use them to answer this question" + your query to the llm
This part is what I want to understand. How does the llm “frame” an answer?