Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

github.com

Post reply on HN
Print view View original

31–40 of 58 posts

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#31

Post by simion314 »

zurfer wrote: post #4

While this is interesting, I wonder what the use case is, other than better autocomplete?

I tested the Google AI on my phone, I had the browser open and asked it to read the page to me and it responded that it does not have access to the internet.

So I would like an AI assistant that:

1 can understand english and my native language

2 that is aware that runs on Android(or KDE/Linux) and can understand commands like "open the Android Settings , Application section " or "read the page that is opened in the browser" or "read the text in the popup that is now opened". Basically to be integrated with the OS via public and open APIs. Big AI companies could compete on selling us better assistants especially for multi lingual people.

3 the model should be small , it should not know geography, history, music bands etc, for tasks where the user asks question there should be an option for the model to forward the question to a search engine or even an online LLM.

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#33

Post by cjtrowbridge »

mmastrac wrote: post #3

> MobileLLM-125M/350M attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M SoTA models on zero-shot commonsense reasoning tasks Small models, slightly improved, probably still not good enough for the same use as online models. Nothing wrong with incremental progress, however. 1.5B parameter model does seem to be a pretty decent step up, even beating larger models by a wide margin. I'm not sure why t…

Llama-3-8b runs fine on raspberry pi

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#34

Post by lawlessone »

Does it have to stay on mobile devices? Bit of niche but if its not a resource hog it could be handy for giving NPC's in games more interesting dialogue without having use

Even better if it could be tuned in someway to allow dialogue to influence NPC behavior or actions.

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#35

Post by nl »

nsonha wrote: post #23
zurfer wrote: post #4

While this is interesting, I wonder what the use case is, other than better autocomplete?

user cases are that of LLMs, from a mobile UI (so every AI use case there is), when you need privacy from big tech's AI APIs. I'm just so amazed by statements like "LLMs can ONLY be used for autocomplete", like am I supposed to be impressed by the smirkiness?

The question was more about the capability and knowledge in a sub-1B LLM: at that size what is it capable to do beyond excellent autocompletion.

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#36

Post by janalsncm »

lawlessone wrote: post #34

Does it have to stay on mobile devices? Bit of niche but if its not a resource hog it could be handy for giving NPC's in games more interesting dialogue without having use Even better if it could be tuned in someway to allow dialogue to influence NPC behavior or actions.

It would be fascinating if NPCs had more backstory to them and more complex behaviors. Although I would imagine it would be near impossible to test since anything could influence their behavior.

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#37

Post by woadwarrior01 »

Havoc wrote: post #5

What apps can one currently use to run them on say an iPhone? Only aware of the MLC one which has literally 3 old models only

I have an (mlc-llm based) app on the App Store that supports over 2 dozen models, including some recent ones.

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#39

Post by kevingadd »

lawlessone wrote: post #34

Does it have to stay on mobile devices? Bit of niche but if its not a resource hog it could be handy for giving NPC's in games more interesting dialogue without having use Even better if it could be tuned in someway to allow dialogue to influence NPC behavior or actions.

Would it be interesting dialogue? You could generate more dialogue, but would it have anything underpinning it of interest to the player? i.e. you could suddenly have townspeople that would talk about local scenery or their relationships with other NPCs, but none of that stuff they describe would actually exist in the game. I would personally be weirded out if NPCs started making stuff up.

I can imagine training some sort of LLM on your game data such that NPCs are able to actually describe the game world, but I can't imagine what kind of scale you'd need to operate at for that to be cheaper than just paying someone to write the dialogue. Maybe at Ubisoft's scale where your team sizes are in the thousands (AFAIK, they have been investigating using AI for writing, but it's mostly for things like combat barks which are very repetitive and basically noise.)

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#40

Post by woodson »

phkahler wrote: post #21
mmastrac wrote: post #3

> MobileLLM-125M/350M attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M SoTA models on zero-shot commonsense reasoning tasks Small models, slightly improved, probably still not good enough for the same use as online models. Nothing wrong with incremental progress, however. 1.5B parameter model does seem to be a pretty decent step up, even beating larger models by a wide margin. I'm not sure why t…

>> Small models, slightly improved, probably still not good enough for the same use as online models. Nothing wrong with incremental progress, however. An even smaller language model should still be useful as part of a speech-to-text system. These should benefit from using the language model to narrow down what word is spoken in the face of ambiguity or noise.

ASR systems already use language models during decoding, though mostly not large decoder-only LLMs. However, incorporating LLMs into ASR is currently at the center of a lot of research, e.g. using a speech encoder like wav2vec 2.0 or the whisper encoder with a Qformer etc. and a LoRA adapter on an LLM trained for ASR.
Post reply on HN