Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

github.com

Post reply on HN
Print view View original

51–58 of 58 posts

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#51

Post by lawlessone »

janalsncm wrote: post #36
lawlessone wrote: post #34

Does it have to stay on mobile devices? Bit of niche but if its not a resource hog it could be handy for giving NPC's in games more interesting dialogue without having use Even better if it could be tuned in someway to allow dialogue to influence NPC behavior or actions.

It would be fascinating if NPCs had more backstory to them and more complex behaviors. Although I would imagine it would be near impossible to test since anything could influence their behavior.

yeah definitely testing would be nightmare. especially if conversations could influence the wider game.

You'd have someone on youtube cheesing games by running scamming npcs.

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#52

Post by inhumantsar »

cjtrowbridge wrote: post #33
mmastrac wrote: post #3

> MobileLLM-125M/350M attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M SoTA models on zero-shot commonsense reasoning tasks Small models, slightly improved, probably still not good enough for the same use as online models. Nothing wrong with incremental progress, however. 1.5B parameter model does seem to be a pretty decent step up, even beating larger models by a wide margin. I'm not sure why t…

Llama-3-8b runs fine on raspberry pi

how fast is that for you?

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#53

Post by HanClinto »

mmastrac wrote: post #3

> MobileLLM-125M/350M attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M SoTA models on zero-shot commonsense reasoning tasks Small models, slightly improved, probably still not good enough for the same use as online models. Nothing wrong with incremental progress, however. 1.5B parameter model does seem to be a pretty decent step up, even beating larger models by a wide margin. I'm not sure why t…

> I'm not sure why they didn't go larger -- having a more efficient model that fits on hardware the size of the RPi could be a gamechanger (IIRC TinyLlama 7B does run, barely).

I'm not sure that RPi is the right target for the next step of local LLMs, and I think that it's worth considering web-deployment on engines like WebLLM [1].

A 7B model may "run fine" on a Raspberry Pi, but I've (personally) found 7B models to be a bit larger than I want to download / run for web-based interfaces.

However, a solid 125M model is the sort of thing that I can run on a webpage, and the time it takes to download to the local user's browser (combined with my bandwidth costs) aren't exorbitant.

[1] https://github.com/mlc-ai/web-llm

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#54

Post by HanClinto »

janalsncm wrote: post #36
lawlessone wrote: post #34

Does it have to stay on mobile devices? Bit of niche but if its not a resource hog it could be handy for giving NPC's in games more interesting dialogue without having use Even better if it could be tuned in someway to allow dialogue to influence NPC behavior or actions.

It would be fascinating if NPCs had more backstory to them and more complex behaviors. Although I would imagine it would be near impossible to test since anything could influence their behavior.

I'm definitely interested in exploring this sort of thing. How much can we do with creating interesting characters and interesting circumstances?

Makes me think of the way that characters are set up in AI Alibis -- each with their own secrets, but also with clues about other NPC's secrets. That feels like clever design, and it's the first use-case of using LLMs for NPC dialogue that feels interesting to me: https://news.ycombinator.com/item?id=40921990

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#55

Post by HanClinto »

pmontra wrote: post #45

Interesting research, but Meta do not have any device worth talking about (at least at scale,) unless they want to ship that as part of their apps.

125M parameters feels very feasible to ship as part of apps -- even web-based apps.

Re: MobileLLM: Optimizing Sub-Billion Parameter Language Models for On-Device Use

#58

Post by mark336 »

mark336 wrote: post #57

How about instead of Gen AI on the desktop, just AI on the desktop. Could organize all my files, emails, and notes and let me search for information from my own data.

Btw, for anyone interested I have AI news summaries and ideas for startups at my website here: https://asiaviewnews.com/gigabots/Threads?p=20007
Post reply on HN