Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

github.com

Post reply on HN
Print view View original

11–20 of 58 posts

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#11

Post by skiexperte »

zurfer wrote: post #4

While this is interesting, I wonder what the use case is, other than better autocomplete?

Reading emails, replying to emails, scheduling tasks, using apis for services.

Basically everything which doesn't need knowledge but actions.

"Tell my wife i'm late" and it will use some configured magic to talk to service xy and just does it.

Siri is very good in doing homeautomatistaion without the internet, the old google agent and alexa were absolutly not and i don't think they were ever available offline.

This basically gives you a local (local-first!) good working assistent

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#13

Post by sigmoid10 »

PoignardAzur wrote: post #12

I wonder how much you can push the "deeper and thinner" part. At some point your entire FFN fits into your L2 cache, you're bound to get some performance jumps.

Other research from Meta FAIR actually suggests that you should prune deeper layers if you want to improve performance while maintaining accuracy [1]. So there must be a cutoff point for smaller networks where this approach still works, otherwise the results are contradictory. Or we could drastically improve these new models even further.

[1] https://arxiv.org/html/2403.17887v1

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#14

Post by imurray »

yshvrdhn wrote: post #9

Am I missing something but can't something like distillation help here ?

The paper says they tried that: https://arxiv.org/abs/2402.14905

Deep link to the relevant snippet in html version: https://ar5iv.labs.arxiv.org/html/2402.14905#S3.SS5

"So far, we trained compact models from scratch using next tokens as hard labels. We explored Knowledge Distillation (KD)... Unfortunately KD increases training time (slowdown of 2.6−3.2×) and exhibits comparable or inferior accuracy to label-based training (details in appendix)."

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#16

Post by ejdhshsuwisjsh »

Anyone is aware of custom mobile llms?

Optimizing and loading in your own voice, selecting your primary language and adding a little bit of personal knowledge like nicknames, location and stuff?

My pixel 8 apparently can use / load local models but don't have the time right now to follow that rabbit hole

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#17

Post by throwthrowuknow »

zurfer wrote: post #4

While this is interesting, I wonder what the use case is, other than better autocomplete?

You could possibly fine tune it for narrow domain tasks like they did with tiny-agent https://bair.berkeley.edu/blog/2024/05/29/tiny-agent/

I like the approach that Apple seems to be taking with fine tuned small models that handle routine tasks and then defer to larger off device models for things they can’t confidently do. I imagine you could construct a training set that contains examples that should produce low confidence answers where you could add an output that is essentially a “call for help” option so you could train it to choose that. Smaller models also means you could have more running in parallel and use another to route requests to the appropriate expert.

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#18

Post by skiexperte »

sourcecodeplz wrote: post #2

Nice, could one use this to train models for Windows PCs also? I don't have a lot of ram.

Training models is not OS dependend. RAM is dependend on the size and i would argue this should be a lot easier to finetune with less GPU Ram.

Nonetheless the endgoal will probably be downloading a model like this or paying for finetuning than downloading and using it through an optimized Neuralchip.

Its currently more a question of when this will happen. The newest Windows cert already requires some neuralchip and even my google pixel 8 pro can host small models (i know the pixel is not a cheap phone, but the coprocessor should still be much more affordable than a big GPU)

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#19

Post by Narhem »

skiexperte wrote: post #11
zurfer wrote: post #4

While this is interesting, I wonder what the use case is, other than better autocomplete?

Reading emails, replying to emails, scheduling tasks, using apis for services. Basically everything which doesn't need knowledge but actions. "Tell my wife i'm late" and it will use some configured magic to talk to service xy and just does it. Siri is very good in doing homeautomatistaion without the internet, the old google agent and alexa were absolutly not and i don't think they were ever available offline. This b…

Would be very nice to have my schedule automatically managed by Siri. Already has a few nice things but I genuinely have trust issues, especially with AI.
Post reply on HN