I put llama 3 on a RBPi 5 and have it running a small droid. I added a TTS engine so it can hear spoken prompts which it replies to in droid speak. It also has a small screen that translates the response to English. I gave it a backstory about being a astromech droid so it usually just talks about the hyperdrive but it's fun.
Ask HN: Is anyone doing anything cool with tiny language models?
151–160 of 356 posts
Re: Ask HN: Is anyone doing anything cool with tiny language models?
#152Micro Wake Word is a library and set of on device models for ESPs to wake on a spoken wake word. https://github.com/kahrendt/microWakeWord Recently deployed in Home Assistants fully local capable Alexa replacement. https://www.home-assistant.io/voice_control/about_wake_word/
Nice idea.
Re: Ask HN: Is anyone doing anything cool with tiny language models?
#153I used a small (3b, I think) model plus tesseract.js to perform OCR on an image of a nutritional facts table and output structured JSON.
All that server side or client?
Re: Ask HN: Is anyone doing anything cool with tiny language models?
#154I have a tiny device that listens to conversations between two people or more and constantly tries to declare a "winner"
what model do you use for speech to text?
Re: Ask HN: Is anyone doing anything cool with tiny language models?
#155I have ollama responding to SMS spam texts. I told it to feign interest in whatever the spammer is selling/buying. Each number gets its own persona, like a millennial gymbro or 19th century British gentleman. http://files.widloski.com/image10%20(1).png http://files.widloski.com/image11.png
Calling Jessica an old chap is quite a giveaway that it's a bot xD
Nice idea indeed, but I have a feeling that it's just two LLMs now conversing with each other.
Re: Ask HN: Is anyone doing anything cool with tiny language models?
#156Not sure it qualifies, but I've started building an Android app that wraps bergamot[0] (the firefox translation models) to have on-device translation without reliance on google. Bergamot is already used inside firefox, but I wanted translation also outside the browser. [0]: bergamot https://github.com/browsermt/bergamot-translator
I would be very interested if someone is aware of any small/tiny models to perform OCR, so the app can translate pictures as well
MiniCPM-V 2.6 isn't that small (8b) but it can do this.
Here is a demo.
* https://i.imgur.com/pAuTeAf.jpeg
Using this script:
Re: Ask HN: Is anyone doing anything cool with tiny language models?
#157I've been using Llama models to identify cookie notices on websites, for the purpose of adding filter rules to block them in EasyList Cookie. Otherwise, this is normally done by, essentially, manual volunteer reporting. Most cookie notices turn out to be pretty similar, HTML/CSS-wise, and then you can grab their `innerText` and filter out false positives with a small LLM. I've found the 3B models have decent performa…
This is so cool thanks for sharing. I can imagine it’s not technically possible (yet?) but it would be cool if this could simply be run as a browser extension rather than running a docker container
There are a couple of WebGPU LLM platforms available that form the building blocks to accomplish this right from the browser, especially since the models are so small.
https://github.com/mlc-ai/web-llm
https://huggingface.co/docs/transformers.js/en/index
You do have to worry about WebGPU compatibility in browsers though.
Re: Ask HN: Is anyone doing anything cool with tiny language models?
#158I built an Excel Add-In that allows my girlfriend to quickly filter 7000 paper titles and abstracts for a review paper that she is writing [1]. It uses Gemma 2 2b which is a wonderful little model that can run on her laptop CPU. It works surprisingly well for this kind of binary classification task.
The nice thing is that she can copy/paste the titles and abstracts in to two columns and write e.g. "=PROMPT(A1:B1, "If the paper studies diabetic neuropathy and stroke, return 'Include', otherwise return 'Exclude'")" and then drag down the formula across 7000 rows to bulk process the data on her own because it's just Excel. There is a gif on the readme on the Github repo that shows it.
Re: Ask HN: Is anyone doing anything cool with tiny language models?
#159I simply use it to de-anonymize code that I typed in via Claude Maybe should write a plugin for it (open source): 1. Put in all your work related questions in the plugin, an LLM will make it as an abstract question for you to preview and send it 2. And then get the answer with all the data back E.g. df[“cookie_company_name”] becomes df[“a”] and back
So you are using a local small model to remove identifying information and make the question generic, which is then sent to a larger model? Is that understanding correct? I think this would have some additional benefits of not confusing the larger model with facts it doesn't need to know about. My erasing information, you can allow its attention heads to focus on the pieces that matter. Requires further study.
> So you are using a local small model to remove identifying information and make the question generic, which is then sent to a larger model? Is that understanding correct?
Yep that's it
Re: Ask HN: Is anyone doing anything cool with tiny language models?
#160I simply use it to de-anonymize code that I typed in via Claude Maybe should write a plugin for it (open source): 1. Put in all your work related questions in the plugin, an LLM will make it as an abstract question for you to preview and send it 2. And then get the answer with all the data back E.g. df[“cookie_company_name”] becomes df[“a”] and back
You're using it to anonymize your code, not de-anonymize someone's code. I was confused by your comment until I read the replies and realized that's what you meant to say.
I read it the other way, their code contains eg fetch(url, pw:hunter123), and they're asking Claude anonymized questions like
"implement handler for fetch(url, {pw:mycleartrxtpw})"
And then claude replies
fetch(url, {pw:mycleartrxtpw}).then(writething)
And then the local llm converts the placeholder mycleartrxtpw into hunter123 using its access to the real code