Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

151–160 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#151

Post by jstummbillig »

svnt wrote: post #117
jstummbillig wrote: post #112

Earlier quoted context omitted.

Also, next year, there will be GPT 5. I find it fascinating how much attention small models get, when at the same time the big models just get bigger and prohibitively expensive to train. No leading lab would do that if they thought it a decent chance that small models were able to compete. So who will be interested in a shitty assistant next year when you can have an amazing one, is what I wonder? Is this just the b…

I’ll flip this around a bit: If I’ve raised $1B to buy GPUs and train a “bigger model”, a major part of my competitive advantage is having $1B to spend on sufficient GPUs to train a bigger model. If, after having raised that money it becomes apparent that consumer hardware can run smaller models that are optimized and perform as well without all that money going into training them, how am I going to pivot my business…

What you suggest is not impossible but simply flies in the face of all currently available evidence and what all leading labs say and do. We know they are actively looking for ways to do things more efficiently. OpenAI alone did a couple of releases to that effect. Because of how easy it is to switch providers, if only one lab found a way to run a small model that competed with the big ones, it would simply win the entire space, so everyone has to be looking for that (and clearly they are, given that all of them do have smaller versions of their models)

Scepticism is fine, if it's plausible. If not it's conspiratorial.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#152

Post by statenjason »

pimeys wrote: post #110

Has anybody found a good way to utilize ollama with an editor such as zed to do things like "generate rustdoc to this method" etc. I use ollama daily for a ton of things, but for code generation, completion and documentation 4o is still much better than any of the local models...

I use gen.nvim[1] with for small tasks, like “write a type definition for this JSON” .

Running locally avoids the concern of sending IP or PII to third parties.

[1]: https://github.com/David-Kunz/gen.nvim

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#153

Post by NortySpock »

exe34 wrote: post #137
deegles wrote: post #45

Earlier quoted context omitted.

Just try a few models on your machine? It takes seconds plus however long it takes to download the model.

I would prefer to have some personal recommendations - I've had some success with Llama3.1-8B/8bits and Llama3.1-70B/1bit, but this is a fast moving field, so I think it's worth the details.

New LLM Prompt:

Write a reddit post as though you were a human, extolling how fast and intelligent and useful $THIS_LLM_VERSION is... Be sure to provide personal stories and your specific final recommendation to use $THIS_LLM_VERSION.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#154

Post by vunderba »

tetrisgm wrote: post #7

I like self hosting random stuff on docker. Ollama has been a great addition. I know it's not, but it feels on par with ChatGPT. It works perfectly on my 4090, but I've also seen it work perfectly on my friend's M3 laptop. It feels like an excellent alternative for when you don't need the heavy weights, but want something bespoke and private. I've integrated it with my Obsidian notes for 1) note generation 2) fuzzy s…

There's actually a very popular plugin for Obsidian that integrates RAG + LLM into Obsidian called Smart Connections.

https://github.com/brianpetro/obsidian-smart-connections

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#155

Post by flimflamm »

lukan wrote: post #148
toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

"before having an LLM clean up my ramblings into organized notes and todo lists." Which local LLM do you use? Edit: And self talk is quite a healthy and useful thing in itself, but avoiding it in public is indeed kind of necessary, because of the stigma https://en.m.wikipedia.org/wiki/Intrapersonal_communication

That's just meat CoT (chain of thought) - right?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#156

Post by wongarsu »

exe34 wrote: post #30
tetrisgm wrote: post #7

I like self hosting random stuff on docker. Ollama has been a great addition. I know it's not, but it feels on par with ChatGPT. It works perfectly on my 4090, but I've also seen it work perfectly on my friend's M3 laptop. It feels like an excellent alternative for when you don't need the heavy weights, but want something bespoke and private. I've integrated it with my Obsidian notes for 1) note generation 2) fuzzy s…

which model are you using? what size/quant/etc? thanks!

I'd be interested in other people's recommendations as well. Personally I'm mostly using openchat with q5_k_m quantization.

OpenChat is imho one of the best 7B models, and while I could run bigger models at least for me they monopolize too many resources to keep them loaded all the time.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#157

Post by [deleted] »

noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

[deleted]

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#158

Post by vunderba »

pimeys wrote: post #140
navbaker wrote: post #128

Earlier quoted context omitted.

The Continue extension for VSCode is pretty good and has native connectivity to a local install of Ollama

Zed has also support for ollama, but all the local models I tried do not really work so well to things like "write docs for this method"... Also local editor autocomplete in the style of github copilot would be great, without needing to use proprietary Microsoft tooling...

There's a lot of plugins/IDEs for assistant style LLMs, but the only TAB style autocompletion ones I know of are either proprietary (Github Copilot), or you need to get an API key (Codestral). If anyone knows of a local autocomplete model I'd love to hear about it.

The Continue extension (Jetbrains, VSCodium) lets you set up assistant and autocompletion independently with different API keys.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#160

Post by [deleted] »

noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

[deleted]
Post reply on HN