I saw this demo a few months back - and lost it, of LLM autocompletion that was a few milliseconds - it opened a how new way on how to explore it... any ideas?
Forget ChatGPT: why researchers now run small AIs on their laptops
41–50 of 385 posts
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#42What's the current cost of building a DIY bare-bones machine setup to run the top LLaMA 3.1 models? I understand that two nodes are typically required for this. Has anyone built something similar recently, and what hardware specs would you recommend for optimal performance? Also, do you suggest waiting for any upcoming hardware releases before making a purchase?
405B is beyond homelab-scale. I recently obtained a 4x4090 rig, and I am comfortable running 70B and occasionally 128B-class models. For 405B, you need 8xH100 or better. A single H100 costs around $40k.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#43I like self hosting random stuff on docker. Ollama has been a great addition. I know it's not, but it feels on par with ChatGPT. It works perfectly on my 4090, but I've also seen it work perfectly on my friend's M3 laptop. It feels like an excellent alternative for when you don't need the heavy weights, but want something bespoke and private. I've integrated it with my Obsidian notes for 1) note generation 2) fuzzy s…
which model are you using? what size/quant/etc? thanks!
Agree. Please provide more details on this setup or a link.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#44> Microsoft used LLMs to write millions of short stories and textbooks in which one thing builds on another. The result of training on this text, Bubeck says, is a model that fits on a mobile phone but has the power of the initial 2022 version of ChatGPT. I thought training LLMs on content created by LLMs was ill-advised but this would suggest otherwise
Look into Microsoft's Phi papers. The whole idea here is that if you train models on higher quality data (i.e. textbooks instead of blogspam) you get higher quality results. The exact training is proprietary but they seem to use a lot of GPT-4 generated training data. On that note... I've often wondered if broad memorization of trivia is really a sensible use of precious neurons. It seems like a system trained on a n…
You're not just memorizing text though. Each piece of trivia is something that represents coherent parts of reality. Think of it as being highly compressed.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#45Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#46Am I the only one seeing obvious ads in llama3 results?
Yes.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#47> Microsoft used LLMs to write millions of short stories and textbooks in which one thing builds on another. The result of training on this text, Bubeck says, is a model that fits on a mobile phone but has the power of the initial 2022 version of ChatGPT. I thought training LLMs on content created by LLMs was ill-advised but this would suggest otherwise
Look into Microsoft's Phi papers. The whole idea here is that if you train models on higher quality data (i.e. textbooks instead of blogspam) you get higher quality results. The exact training is proprietary but they seem to use a lot of GPT-4 generated training data. On that note... I've often wondered if broad memorization of trivia is really a sensible use of precious neurons. It seems like a system trained on a n…
The trivia include information about many things: grammar, vocabulary, slang, entity relationships, metaphor, among others but chiefly they also constitute models of human thought and behaviour. If all you want is a fancy technical encyclopedia then by all means chop away at the training set but if you want something you can talk to then you’ll need to keep the diversity.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#48Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory. - https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...
It will have around 250GB/s of bandwidth which makes it nearly unusable for 70b models. So the high amount of RAM doesn’t help with large models.
> nearly unusable for 70b models
Can Apple Silicon manage this? Would it be feasible to do with some quantization perhaps?
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#49Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory. - https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...
It will have around 250GB/s of bandwidth which makes it nearly unusable for 70b models. So the high amount of RAM doesn’t help with large models.
Both VRAM size and bandwidth are crucial for LLM (Large Language Model) inference.
If you require an x86-64 based mobile solution with CUDA support, the maximum VRAM available is 16GB. The Strix HALO is positioned as a competitor to the RTX 4070M.
"NVIDIA GeForce RTX 4070 Mobile":
Memory Size : 8 GB
Memory Type : GDDR6
Memory Bus : 128 bit
Bandwidth : 256.0 GB/s
Memory Size : 16 GB
Memory Type : GDDR6
Memory Bus : 256 bit
Bandwidth : 576.0 GB/sRe: Forget ChatGPT: why researchers now run small AIs on their laptops
#50May as well ask here: what is the best way to use something like an LLM as a personal knowledge base?
I have a few thousand book, papers and articles collected over the last decade. And while I have meticulously categorised them for fast lookup, it's getting harder and harder to search for the desired info, especially in categories which I might not have explored recently.
I do have a 4070 (12 GB VRAM), so I thought that LLMs might be a solutions. But trying to figure out the whats and hows hase proven to be extremely complicated, what with deluge of techniques (fine-tuning, RAG, quantisation) that might not might not be obsolete, too many grifters hawking their own startups with thin wrappers, and a general sense that the "new shiny object" is prioritised more than actual stable solutions to real problems.