Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

121–130 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#121

Post by inquirerGeneral »

schmidtleonard wrote: post #109
toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

Button-toggled voice notes in the iPhone Notes app are a godsend for taking measurements. Rather than switching your hands between probe/equipment and notes repeatedly, which sucks badly, you can just dictate your readings and maaaaybe clean out something someone said in the background. Over the last decade, the microphones + speech recognition became Good Enough for this. Wake-word/endpoint models still aren't there…

[dead]

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#122

Post by archagon »

jstummbillig wrote: post #112
pella wrote: post #3

Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory. - https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...

Also, next year, there will be GPT 5. I find it fascinating how much attention small models get, when at the same time the big models just get bigger and prohibitively expensive to train. No leading lab would do that if they thought it a decent chance that small models were able to compete. So who will be interested in a shitty assistant next year when you can have an amazing one, is what I wonder? Is this just the b…

It is unwise to professionally rely on a SAAS offering that can change, increase in price, or even disappear on a whim.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#123

Post by zozbot234 »

noman-land wrote: post #107
dockerd wrote: post #53

What spec people recommend here to run small models like Llama3.1 or mistral-nemo etc. Also is it sensible to wait for newer mac, amd, nvidia hardware releasing soon?

You basically need as much RAM as the size of the model.

You actually need a lot less than that if you use the mmap option, because then only activations need to be stored in RAM, the model itself can be read from disk.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#124

Post by bobbylarrybobby »

azeirah wrote: post #104
racked wrote: post #103

Earlier quoted context omitted.

What software did you use to set all this up? Kindof interested in giving this a shot myself.

You can use llama.cpp, it runs on almost all hardware. Whisper.cpp is similar, but unless you have a mid or high end nvidia card it will be a bit slower. Still very reasonable on modern hardware.

If you build locally for Apple hardware (instructions in the whisper.cpp readme) then it performs quite admirably on Apple computers as well.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#125

Post by navbaker »

racked wrote: post #103
toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

What software did you use to set all this up? Kindof interested in giving this a shot myself.

Definitely try it with Ollama, it is by far the simplest local LLM tool to get up and running with minimal fuss!

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#126

Post by noman-land »

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments.

https://future.mozilla.org/builders/news_insights/introducin...

https://github.com/Mozilla-Ocho/llamafile

They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription.

You can also take this a step further and use this exact setup for a local-only co-pilot style code autocomplete and chat using Twinny. I use this every day. It's free, private, and offline.

https://github.com/twinnydotdev/twinny

Local LLMs are the only future worth living in.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#127

Post by svnt »

nurettin wrote: post #113
IshKebab wrote: post #82

Earlier quoted context omitted.

"All you need is £10k of Apple laptops..."

That is... probable, if you bought a newish m2 to replace your 5-6 year old macbook pro which is now just lying around. Or maybe you and your spouse can share cpu hours.

No, you need two of the newest M3 Macbook Pros with maxed RAM, which in practice some people might have, but it is not gettable by using old hardware.

And not having tried it, I’m guessing it will probably run at 1-2 tokens per second or less since the 70b model on one of these runs at 3-4, and now we are distributing the process over the network, which is best case maybe 40-80Gb/s

It is possible, and that’s about the most you can say about it.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#128

Post by navbaker »

pimeys wrote: post #110

Has anybody found a good way to utilize ollama with an editor such as zed to do things like "generate rustdoc to this method" etc. I use ollama daily for a ton of things, but for code generation, completion and documentation 4o is still much better than any of the local models...

The Continue extension for VSCode is pretty good and has native connectivity to a local install of Ollama

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#129

Post by hdjjhhvvhga »

toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

> It feels super strange to talk to yourself

I remember the first lecture in the Theory of Communication class where the professor introduced the idea that communication by definition requires at least two different participants. We objected by saying that it can perfectly be just one and the same participant (communication is not just about space but also time), and what you say is a perfect example of that.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#130

Post by rvnx »

svnt wrote: post #117
jstummbillig wrote: post #112

Earlier quoted context omitted.

Also, next year, there will be GPT 5. I find it fascinating how much attention small models get, when at the same time the big models just get bigger and prohibitively expensive to train. No leading lab would do that if they thought it a decent chance that small models were able to compete. So who will be interested in a shitty assistant next year when you can have an amazing one, is what I wonder? Is this just the b…

I’ll flip this around a bit: If I’ve raised $1B to buy GPUs and train a “bigger model”, a major part of my competitive advantage is having $1B to spend on sufficient GPUs to train a bigger model. If, after having raised that money it becomes apparent that consumer hardware can run smaller models that are optimized and perform as well without all that money going into training them, how am I going to pivot my business…

Mining cryptos, some "AI" companies already do that (knowingly or not... and not necessarily telling investors)
Post reply on HN