Forget ChatGPT: why researchers now run small AIs on their laptops
1–10 of 385 posts
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#2PC: Personal Chatbot
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#3Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory.
- https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#4Summary: It's cheaper, safer for handling sensitive data, easier to reproduce results (only way to be 100% sure it's reproduce even, as "external" models can change anytime), higher degree of customization, no internet connectivity requirements, more efficient, more flexible.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#5Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory. - https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...
According to Tom's (https://www.tomshardware.com/pc-components/cpus/amd-pushes-r...), those are supposed to be laptop CPUs, which makes me wonder what AMD has planned for us desktop users.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#6Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory. - https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...
[dead]
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#7I like self hosting random stuff on docker. Ollama has been a great addition. I know it's not, but it feels on par with ChatGPT.
It works perfectly on my 4090, but I've also seen it work perfectly on my friend's M3 laptop. It feels like an excellent alternative for when you don't need the heavy weights, but want something bespoke and private.
I've integrated it with my Obsidian notes for 1) note generation 2) fuzzy search.
I've used it as an assistant for mental health and medical questions.
I'd much rather use it to query things about my music or photos than whatever the big players have planned.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#8OpenAI APIs for GPT and Dalle have issues like non determnism, and their special prompt injection where they add stuff or modify your prompt (with no option to turn that off. Makes it impossible to do research or to debug as a developer variations of things.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#9> Microsoft used LLMs to write millions of short stories and textbooks in which one thing builds on another. The result of training on this text, Bubeck says, is a model that fits on a mobile phone but has the power of the initial 2022 version of ChatGPT.
I thought training LLMs on content created by LLMs was ill-advised but this would suggest otherwise
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#10It's kinda funny how nowadays an AI with 8 billion parameters is something "small". Specially when just two years back entire racks were needed to run something giving way worst performance.