Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

github.com

Post reply on HN
Print view View original

21–30 of 58 posts

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#21

Post by phkahler »

mmastrac wrote: post #3

> MobileLLM-125M/350M attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M SoTA models on zero-shot commonsense reasoning tasks Small models, slightly improved, probably still not good enough for the same use as online models. Nothing wrong with incremental progress, however. 1.5B parameter model does seem to be a pretty decent step up, even beating larger models by a wide margin. I'm not sure why t…

>> Small models, slightly improved, probably still not good enough for the same use as online models. Nothing wrong with incremental progress, however.

An even smaller language model should still be useful as part of a speech-to-text system. These should benefit from using the language model to narrow down what word is spoken in the face of ambiguity or noise.

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#23

Post by nsonha »

zurfer wrote: post #4

While this is interesting, I wonder what the use case is, other than better autocomplete?

user cases are that of LLMs, from a mobile UI (so every AI use case there is), when you need privacy from big tech's AI APIs.

I'm just so amazed by statements like "LLMs can ONLY be used for autocomplete", like am I supposed to be impressed by the smirkiness?

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#24

Post by lovethevoid »

Narhem wrote: post #19
skiexperte wrote: post #11

Earlier quoted context omitted.

Reading emails, replying to emails, scheduling tasks, using apis for services. Basically everything which doesn't need knowledge but actions. "Tell my wife i'm late" and it will use some configured magic to talk to service xy and just does it. Siri is very good in doing homeautomatistaion without the internet, the old google agent and alexa were absolutly not and i don't think they were ever available offline. This b…

Would be very nice to have my schedule automatically managed by Siri. Already has a few nice things but I genuinely have trust issues, especially with AI.

You can get very far with the Shortcuts app by the way. Some examples: using your current location to estimate when you should leave to get to your next meeting on your calendar, letting those included in the calendar event know you’re running late. Highly highly recommend it, the learning curve isn’t much, a bunch of drag and drop!

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#26

Post by Havoc »

5cott0 wrote: post #22
Havoc wrote: post #5

What apps can one currently use to run them on say an iPhone? Only aware of the MLC one which has literally 3 old models only

wat https://huggingface.co/mlc-ai

On my iphone there doesn’t seem to be an option to download more.

Vaguely recall there being a button initially but don’t see it anymore

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#27

Post by barronli »

zurfer wrote: post #4

While this is interesting, I wonder what the use case is, other than better autocomplete?

It can be fine tuned for device related actions. In other words, with all the capabilities of your device applications or services, the small model can virtually have the same capabilities. It can always dispatch a user request in way of “natural language” to those applications, and orchestrate the applications. It can dispatch user requests beyond the device capabilities to a cloud model. This is powerful since it changes how you interact with your devices.

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#28

Post by euniceee3 »

ejdhshsuwisjsh wrote: post #16

Anyone is aware of custom mobile llms? Optimizing and loading in your own voice, selecting your primary language and adding a little bit of personal knowledge like nicknames, location and stuff? My pixel 8 apparently can use / load local models but don't have the time right now to follow that rabbit hole

Tensor chips are not open enough for an optimized mobile LLM to be ran on them.

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#29

Post by pickettd »

Havoc wrote: post #5

What apps can one currently use to run them on say an iPhone? Only aware of the MLC one which has literally 3 old models only

The Android apk for MLC is updated frequently with recent models built-in. And a Samsung S24+ can comfortably run 7-8B models at reasonable speeds (10ish tokens/sec).

https://llm.mlc.ai/docs/deploy/android.html

Post reply on HN