Hot take: Dodgy small/fast/cheap LLM in a While True approx. equals AGI for most real world tasks.
I have not found this to be the case but, then again, a lot has changed in the last year.
FunctionGemma 270M Model
51–60 of 60 posts
Re: FunctionGemma 270M Model
#52FTA: In our "Mobile Actions" evaluation, fine-tuning transformed the model’s reliability, boosting accuracy from a 58% baseline to 85%. This confirms that for edge agents, a dedicated, trained specialist is an efficient path to production-grade performance. I would be wary of having a LLM with 85% accuracy call tools on my system. Isn’t that fairly far away from production-grade performance? I also don’t see that the…
There are ways around this. You can push the success rate close to 100% if you use chain of thought and a quorum selection. It isn't great, and it slows response times, but if 85% isn't good enough, you just need to flip the coin about 5 times to get nearly(!) guaranteed results.
Coin flipping works only if the fails are roughly independent. More important is the complexity ceiling above which they fail all the time.
Re: FunctionGemma 270M Model
#53I’ve been wanting to fine tune models for home assistant but unsure how to get some synthetic data, any recommendations?
Check out the approach here: https://github.com/allenporter/home-assistant-datasets and the reports/ directory has a leaderboard for function calling. I'm curious to see how well this model does.
Oh nice, I thought about doing something Ng similar with a docker image and fake devices but never got around to it
Re: FunctionGemma 270M Model
#54I’ve been wanting to fine tune models for home assistant but unsure how to get some synthetic data, any recommendations?
I've found this dataset specifically for Home Assistant with over 32k examples: https://huggingface.co/datasets/acon96/Home-Assistant-Reques...
Funnily, I did fine tune Qwen 1.7B with this but learned this dataset is meant for the authors extension homeLLM
Re: FunctionGemma 270M Model
#55Neat, now we can build our LCARS with this?
Re: FunctionGemma 270M Model
#56FTA: In our "Mobile Actions" evaluation, fine-tuning transformed the model’s reliability, boosting accuracy from a 58% baseline to 85%. This confirms that for edge agents, a dedicated, trained specialist is an efficient path to production-grade performance. I would be wary of having a LLM with 85% accuracy call tools on my system. Isn’t that fairly far away from production-grade performance? I also don’t see that the…
There are ways around this. You can push the success rate close to 100% if you use chain of thought and a quorum selection. It isn't great, and it slows response times, but if 85% isn't good enough, you just need to flip the coin about 5 times to get nearly(!) guaranteed results.
Good insight here, we actually did not include thinking into this model partly because we saw how incredibly fast it was to just get the minimum amount of tokens to output an answer.
Thinking helps performance scores but we'll leave it up to users to add additional tokens if they want. Our goal here was the leanest weight and token base for blazing fast performance for you all.
Re: FunctionGemma 270M Model
#57Hi all, I'm a research lead on this model. Same as every model release post, I enjoy working at Google for a multitude of reasons, and opinions here are my own. Happy to answer whatever technical questions I can!
Thank you. I felt that was a very under appreciated direction ( most of the spotlight seemed to be on 'biggest' models ).
I'm with you! Small generative models are awesome, I thought so a decade ago and I still think so now! The size of what is "small" has definitely increased though, I used to think a 100 parameter model was large back in 2016, but here I am now saying 270 million is small :)
Re: FunctionGemma 270M Model
#58Do you think this would be appropriate for a command line tool that hits various apis as the function calls? Ex: "what's the weather in SF tomorrow?" Or "daily price change of apple, Tesla stock for past week"? (Let's assume I have documented the apis thoroughly somewhere that the model has access to or fine tuned it on this data)
Both your examples require Internet access so there's no reason not to use cloud-hosted model which would work magnitudes better.
Re: FunctionGemma 270M Model
#59Earlier quoted context omitted.
There are ways around this. You can push the success rate close to 100% if you use chain of thought and a quorum selection. It isn't great, and it slows response times, but if 85% isn't good enough, you just need to flip the coin about 5 times to get nearly(!) guaranteed results.
Coin flipping works only if the fails are roughly independent. More important is the complexity ceiling above which they fail all the time.
So my solution to non-binary failure states is
1. Generate a potential solution
2. If the solution is complex, chunk it up into logical parts
3. Vote on each chunk and select those with more than k votes
By doing this you can filter out outliers (not always desirable) and pull the signal out of the noise.
Re: FunctionGemma 270M Model
#60[dead]