Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

131–140 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#131

Post by noman-land »

zozbot234 wrote: post #123
noman-land wrote: post #107

Earlier quoted context omitted.

You basically need as much RAM as the size of the model.

You actually need a lot less than that if you use the mmap option, because then only activations need to be stored in RAM, the model itself can be read from disk.

Can you say a bit more about this? Based on my non-scientific personal experience on an M1 with 64gb memory, that's approximately what it seems to be. If the model is 4gb in size, loading it up and doing inference takes about 4gb of memory. I've used LM Studio and llamafiles directly and both seem to exhibit this behavior. I believe llamafiles use mmap by default based on what I've seen jart talk about. LM Studio allows you to "GPU offload" the model by loading it partially or completely into GPU memory, so not sure what that means.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#132

Post by svnt »

rvnx wrote: post #130
svnt wrote: post #117

Earlier quoted context omitted.

I’ll flip this around a bit: If I’ve raised $1B to buy GPUs and train a “bigger model”, a major part of my competitive advantage is having $1B to spend on sufficient GPUs to train a bigger model. If, after having raised that money it becomes apparent that consumer hardware can run smaller models that are optimized and perform as well without all that money going into training them, how am I going to pivot my business…

Mining cryptos, some "AI" companies already do that (knowingly or not... and not necessarily telling investors)

Is it still even worth the electricity to do this on a GPU? It wouldn’t surprise me if some startups were renting them out, but is anyone still mining any volume of crypto on GPUs?

edit: I guess to your point if it is not knowingly then the electricity costs are not a factor either.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#133

Post by freeone3000 »

dockerd wrote: post #53

What spec people recommend here to run small models like Llama3.1 or mistral-nemo etc. Also is it sensible to wait for newer mac, amd, nvidia hardware releasing soon?

M4s are releasing in probably a month or two; if you’re going Apple, it might be worth waiting for either those or the price drop on the older models.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#134

Post by stainablesteel »

i think this is laughable, the only good 8B models are the llama ones, phi is terrible, even codestral can barely code and that's 22B iirc

but truthfully the 8B just aren't that great yet, they can provide some decent info if you're just investigating things but a google search is still faster

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#135

Post by freetanga »

schmidtleonard wrote: post #109
toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

Button-toggled voice notes in the iPhone Notes app are a godsend for taking measurements. Rather than switching your hands between probe/equipment and notes repeatedly, which sucks badly, you can just dictate your readings and maaaaybe clean out something someone said in the background. Over the last decade, the microphones + speech recognition became Good Enough for this. Wake-word/endpoint models still aren't there…

I bought an iZYREC (?) and leave the phone at home. MacWhisper and some regex (I use verbal tags) and done

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#136

Post by aledalgrande »

Does anyone know of a local "Siri" implementation? Whisper + Llama (or Phi or something else), that can run shortcuts, take notes, read web pages etc.?

PS: for reading web pages I know there's voices integrated in the browser/OS but those are horrible

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#137

Post by exe34 »

deegles wrote: post #45
axpy906 wrote: post #43

Earlier quoted context omitted.

Agree. Please provide more details on this setup or a link.

Just try a few models on your machine? It takes seconds plus however long it takes to download the model.

I would prefer to have some personal recommendations - I've had some success with Llama3.1-8B/8bits and Llama3.1-70B/1bit, but this is a fast moving field, so I think it's worth the details.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#138

Post by wslh »

HPsquared wrote: post #100
atemerev wrote: post #42

Earlier quoted context omitted.

405B is beyond homelab-scale. I recently obtained a 4x4090 rig, and I am comfortable running 70B and occasionally 128B-class models. For 405B, you need 8xH100 or better. A single H100 costs around $40k.

Here is someone running 405b on 12x3090 (4.5bpw). Total cost around $10k. https://www.reddit.com/r/LocalLLaMA/comments/1ej9uzh/local_l... Admittedly it's slow (3.5 token/sec)

Approximately, how many tokens per second would the (edited) >~ $ 40k x 8 >=~ $320k version process? Would this result in a >~32x boost in performance compared to other setups? Thanks!

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#139

Post by talldayo »

Larrikin wrote: post #116
jstummbillig wrote: post #112

Earlier quoted context omitted.

Also, next year, there will be GPT 5. I find it fascinating how much attention small models get, when at the same time the big models just get bigger and prohibitively expensive to train. No leading lab would do that if they thought it a decent chance that small models were able to compete. So who will be interested in a shitty assistant next year when you can have an amazing one, is what I wonder? Is this just the b…

Why would anyone buy a Raspberry Pi when they can get a fully decked out Mac Pro? There are different use cases and computers are already pretty powerful. Maybe your local model won't be able to produce tests that check all the corner cases of the class you just wrote for work in your massive code base. But the small model is perfectly capable of summarizing the weather from an API call and maybe tack on a joke that…

> Why would anyone buy a Raspberry Pi when they can get a fully decked out Mac Pro?

They want compliant Linux drivers?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#140

Post by pimeys »

navbaker wrote: post #128
pimeys wrote: post #110

Has anybody found a good way to utilize ollama with an editor such as zed to do things like "generate rustdoc to this method" etc. I use ollama daily for a ton of things, but for code generation, completion and documentation 4o is still much better than any of the local models...

The Continue extension for VSCode is pretty good and has native connectivity to a local install of Ollama

Zed has also support for ollama, but all the local models I tried do not really work so well to things like "write docs for this method"... Also local editor autocomplete in the style of github copilot would be great, without needing to use proprietary Microsoft tooling...
Post reply on HN