Ask HN: Is anyone doing anything cool with tiny language models? • Hacker News Board

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#41

Post by pseudosavant » Tue, Jan 21, 2025, 10:22 PM UTC

psyklic wrote: post #2

JetBrains' local single-line autocomplete model is 0.1B (w/ 1536-token context, ~170 lines of code): https://blog.jetbrains.com/blog/2024/04/04/full-line-code-co... For context, GPT-2-small is 0.124B params (w/ 1024-token context).

I wonder how big that model is in RAM/disk. I use LLMs for FFMPEG all the time, and I was thinking about training a model on just the FFMPEG CLI arguments. If it was small enough, it could be a package for FFMPEG. e.g. `ffmpeg llm "Convert this MP4 into the latest royalty-free codecs in an MKV."`

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#42

Post by jedbrooke » Tue, Jan 21, 2025, 10:33 PM UTC

pseudosavant wrote: post #41

psyklic wrote: post #2

JetBrains' local single-line autocomplete model is 0.1B (w/ 1536-token context, ~170 lines of code): https://blog.jetbrains.com/blog/2024/04/04/full-line-code-co... For context, GPT-2-small is 0.124B params (w/ 1024-token context).

I wonder how big that model is in RAM/disk. I use LLMs for FFMPEG all the time, and I was thinking about training a model on just the FFMPEG CLI arguments. If it was small enough, it could be a package for FFMPEG. e.g. `ffmpeg llm "Convert this MP4 into the latest royalty-free codecs in an MKV."`

the jetbrains models are about 70MB zipped on disk (one model per language)

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#43

Post by simonjgreen » Tue, Jan 21, 2025, 10:33 PM UTC

Micro Wake Word is a library and set of on device models for ESPs to wake on a spoken wake word. https://github.com/kahrendt/microWakeWord

Recently deployed in Home Assistants fully local capable Alexa replacement. https://www.home-assistant.io/voice_control/about_wake_word/

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#44

Post by econ » Tue, Jan 21, 2025, 10:34 PM UTC

flippyhead wrote: post #35

I have a tiny device that listens to conversations between two people or more and constantly tries to declare a "winner"

This is a product I want

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#45

Post by amelius » Tue, Jan 21, 2025, 10:36 PM UTC

flippyhead wrote: post #35

I have a tiny device that listens to conversations between two people or more and constantly tries to declare a "winner"

You can use the model to generate winning speeches also.

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#46

Post by jjcm » Tue, Jan 21, 2025, 10:37 PM UTC

flippyhead wrote: post #35

I have a tiny device that listens to conversations between two people or more and constantly tries to declare a "winner"

Are you raising a funding round? I'm bought in. This is hilarious.

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#47

Post by XMasterrrr » Tue, Jan 21, 2025, 10:39 PM UTC

andai wrote: post #24

iamnotagenius wrote: post #8

No, but I use llama 3.2 1b and qwen2.5 1.5 as bash oneliner generator, always runnimg in console.

Could you elaborate?

I think I know what he means. I use AI Chat. I load Qwen2.5-1.5B-Instruct with llama.cpp server, fully offloaded to the CPU, and then I config AI Chat to connect to the llama.cpp endpoint.

Checkout the demo they have below

https://github.com/sigoden/aichat#shell-assistant

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#48

Post by XMasterrrr » Tue, Jan 21, 2025, 10:39 PM UTC

iamnotagenius wrote: post #8

No, but I use llama 3.2 1b and qwen2.5 1.5 as bash oneliner generator, always runnimg in console.

What's your workflow like? I use AI Chat. I load Qwen2.5-1.5B-Instruct with llama.cpp server, fully offloaded to the CPU, and then I config AI Chat to connect to the llama.cpp endpoint.

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#49

Post by keeganpoppen » Tue, Jan 21, 2025, 10:44 PM UTC

behohippy wrote: post #14

I have a mini PC with an n100 CPU connected to a small 7" monitor sitting on my desk, under the regular PC. I have llama 3b (q4) generating endless stories in different genres and styles. It's fun to glance over at it and read whatever it's in the middle of making. I gave llama.cpp one CPU core and it generates slow enough to just read at a normal pace, and the CPU fans don't go nuts. Totally not productive or really…

oh wow that is actually such a brilliant little use case-- really cuts to the core of the real "magic" of ai: that it can just keep running continuously. it never gets tired, and never gets tired of thinking.

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#50

Post by ipython » Tue, Jan 21, 2025, 10:46 PM UTC

behohippy wrote: post #14

I have a mini PC with an n100 CPU connected to a small 7" monitor sitting on my desk, under the regular PC. I have llama 3b (q4) generating endless stories in different genres and styles. It's fun to glance over at it and read whatever it's in the middle of making. I gave llama.cpp one CPU core and it generates slow enough to just read at a normal pace, and the CPU fans don't go nuts. Totally not productive or really…

That's neat. I just tried something similar:

    FORTUNE=$(fortune) && echo $FORTUNE && echo "Convert the following output of the Unix `fortune` command into a small screenplay in the style of Shakespeare: \n\n $FORTUNE" | ollama run phi4