41–50 of 58 posts
I wonder how much you can push the "deeper and thinner" part. At some point your entire FFN fits into your L2 cache, you're bound to get some performance jumps.
> MobileLLM-125M/350M attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M SoTA models on zero-shot commonsense reasoning tasks Small models, slightly improved, probably still not good enough for the same use as online models. Nothing wrong with incremental progress, however. 1.5B parameter model does seem to be a pretty decent step up, even beating larger models by a wide margin. I'm not sure why t…
In this application it’s unfair to compare tiny models to cloud models. Moreover any incremental precision boosts to tiny models would be notable (and directly translate to revenue).
Earlier quoted context omitted.
>> Small models, slightly improved, probably still not good enough for the same use as online models. Nothing wrong with incremental progress, however. An even smaller language model should still be useful as part of a speech-to-text system. These should benefit from using the language model to narrow down what word is spoken in the face of ambiguity or noise.
ASR systems already use language models during decoding, though mostly not large decoder-only LLMs. However, incorporating LLMs into ASR is currently at the center of a lot of research, e.g. using a speech encoder like wav2vec 2.0 or the whisper encoder with a Qformer etc. and a LoRA adapter on an LLM trained for ASR.
It seems like the smaller models get the largest size decrease by embedding share/weight tying between the linear head and token embeddings. Is there any research going into how to further reduce size from there?
Unfortunately, the only thing I found out about this is that bigger models benefit from separate layer. But this was only mentioned somewhere in discord, so no paper to read and my personal hunch is that it should work for bigger models too. After all, GPT-3 was just scaled GPT-2.
From my personal experiments, models learn better if you give them harder task. And tied weights could be one of such things. Multi-token prediction could be another and bitnet could be also considered such... (and dropout too)
Interesting research, but Meta do not have any device worth talking about (at least at scale,) unless they want to ship that as part of their apps.
Interesting research, but Meta do not have any device worth talking about (at least at scale,) unless they want to ship that as part of their apps.
Quests have shipped roughly ~1/2 PS5 sales. Certainly a scale only a handful of technologically advanced product lines outside of phones ever reach.
Incidentally, the enabling technology for the Quest? On-device ML that grew out of - you guessed it - developing on-device inference for their apps.
What apps can one currently use to run them on say an iPhone? Only aware of the MLC one which has literally 3 old models only
Does it have to stay on mobile devices? Bit of niche but if its not a resource hog it could be handy for giving NPC's in games more interesting dialogue without having use Even better if it could be tuned in someway to allow dialogue to influence NPC behavior or actions.
Would it be interesting dialogue? You could generate more dialogue, but would it have anything underpinning it of interest to the player? i.e. you could suddenly have townspeople that would talk about local scenery or their relationships with other NPCs, but none of that stuff they describe would actually exist in the game. I would personally be weirded out if NPCs started making stuff up. I can imagine training some…
It would definitely depend a lot on the implementation. I think it could work great for some indie dev's. Not all of course, devs that like writing understandably won't like it.