Earlier quoted context omitted.
An AI chip on laptop devices would be amazing!
They already exist. Nvidia GPUs on laptops, M series CPUs from Apple, NPUs...
oh damn guess i am so uninformed
Forget ChatGPT: why researchers now run small AIs on their laptops
91–100 of 385 posts
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#92Earlier quoted context omitted.
Meta dropped that term, actually, and that's an unofficial website.
It's still present in the llama license...? https://ai.meta.com/llama/license/ Section 1.b.iv
Llama 3.1 isn't under that license, it's under the Llama 3.1 Community License Agreement: https://www.llama.com/llama3_1/license/
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#93Earlier quoted context omitted.
Look into Microsoft's Phi papers. The whole idea here is that if you train models on higher quality data (i.e. textbooks instead of blogspam) you get higher quality results. The exact training is proprietary but they seem to use a lot of GPT-4 generated training data. On that note... I've often wondered if broad memorization of trivia is really a sensible use of precious neurons. It seems like a system trained on a n…
The trivia include information about many things: grammar, vocabulary, slang, entity relationships, metaphor, among others but chiefly they also constitute models of human thought and behaviour. If all you want is a fancy technical encyclopedia then by all means chop away at the training set but if you want something you can talk to then you’ll need to keep the diversity.
> you’ll need to keep the diversity.
You can get diverse low quality data from the web, but for diverse high quality data the organic content is exhausted. The only way is to generate it, and you can maintain a good distribution by structured randomness. For example just sample 5 random words from the dictionary and ask the model to compose a piece of text from them. It will be more diverse than web text.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#94Earlier quoted context omitted.
Generally (not just for LLMs) this is called student-teacher training and/or knowledge distillation.
It reminds me of when I take notes from a textbook then intensively review my own notes
And then when it comes time for the test, I end up hallucinating answers too.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#95Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#96Earlier quoted context omitted.
> nearly unusable for 70b models Can Apple Silicon manage this? Would it be feasible to do with some quantization perhaps?
Any of the newer M2+ Max chips runs 400GB/s and can run 70b pretty well. It's not fast though, 3-4 token/s. You can get better performance using a good CPU + 4090 + offloading layers to GPU. However one is a laptop and the other is a desktop...
Apparently Mac purchasers like to talk about tokens per second without talking about Mac's atrocious time to first token. They also like to enthusiastically talk about tokens per second asking a 200 token question rather than a longer prompt.
I'm not sure what the impact is on a 70b model but it seems there's a lot of exaggeration going on in this space by Mac fans.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#97I have a three year old M1 Max, 32gb RAM. Llama 8bn runs at 25 tokens/sec, that’s fast enough, and covers 80% of what I need. On my ryzen 5600h machine, I get about 10 tokens/second, which is slow enough to be annoying. If I get stuck on a problem, switch to chat gpt or phind.com and see what that gives. Sometimes, it’s not the LLM that helps, but changing the context and rewriting the question. However I cannot use…
A small model necessarily is missing many facts. The large model is the one that has memorized the whole internet, the small one is just trained to mimic the big one.
You simply cannot compress the whole internet under 10gb without throwing out a lot of information.
Please be careful about what you take as fact coming from the local model output. Small models are better suited to summarization.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#98I have a three year old M1 Max, 32gb RAM. Llama 8bn runs at 25 tokens/sec, that’s fast enough, and covers 80% of what I need. On my ryzen 5600h machine, I get about 10 tokens/second, which is slow enough to be annoying. If I get stuck on a problem, switch to chat gpt or phind.com and see what that gives. Sometimes, it’s not the LLM that helps, but changing the context and rewriting the question. However I cannot use…
I'm really curious what you are doing with an LLM that can be solved 80% of the time with a 8b model.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#99The newest laptops are supposed to have 40-50 TOPS performance with the new AI/NPU features. Wondering what that will mean in practice.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#100What's the current cost of building a DIY bare-bones machine setup to run the top LLaMA 3.1 models? I understand that two nodes are typically required for this. Has anyone built something similar recently, and what hardware specs would you recommend for optimal performance? Also, do you suggest waiting for any upcoming hardware releases before making a purchase?
405B is beyond homelab-scale. I recently obtained a 4x4090 rig, and I am comfortable running 70B and occasionally 128B-class models. For 405B, you need 8xH100 or better. A single H100 costs around $40k.
Here is someone running 405b on 12x3090 (4.5bpw). Total cost around $10k.
https://www.reddit.com/r/LocalLLaMA/comments/1ej9uzh/local_l...
Admittedly it's slow (3.5 token/sec)