Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

181–190 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#181

Post by wkat4242 »

noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Yeah I set up a local server with a strong GPU but even without that it's ok, just a lot slower.

The biggest benefits for me are the uncensored models. I'm pretty kinky so the regular models tend to shut me out way too much, they all enforce this prudish victorian mentality that seems to be prevalent in the US but not where I live. Censored models are just unusable to me which includes all the hosted models. It's just so annoying. And of course the privacy.

It should really be possible for the user to decide what kind of restrictions they want, not the vendor. I understand they don't want to offer violent stuff but 18+ topics should be squarely up to me.

Lately I've been using grimjim's uncensored llama3.1 which works pretty well.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#182

Post by [deleted] »

noman-land wrote: post #173
zelphirkalt wrote: post #166

Earlier quoted context omitted.

Many setup rely on Nvidia GPUs, Intel stuff, Windows or other stuff, that I would rather not use, or are not very clear about how to set things up. What are some recommendations for running models locally, on decent CPUs and getting good valuable output from them? Is that llama stuff portable across CPUs and hardware vendors? And what do people use it for?

llamafile will run on all architectures because it is compiled by cosmopolitan. https://github.com/jart/cosmopolitan "Cosmopolitan Libc makes C a build-once run-anywhere language, like Java, except it doesn't need an interpreter or virtual machine. Instead, it reconfigures stock GCC and Clang to output a POSIX-approved polyglot format that runs natively on Linux + Mac + Windows + FreeBSD + OpenBSD + NetBSD + BIOS wit…

[deleted]

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#183

Post by wkat4242 »

toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

What do you use to run whisper locally? I don't think ollama can do it.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#184

Post by maxnevermind »

theodorthe5 wrote: post #22

Local LLMs are terrible compared to Claude/ChatGPT. They are useful to use as APIs for applications: much cheaper than paying for OpenAI services, and can be fine tuned to do many useful (and less useful, even illegal) things. But for the casual user, they suck compared to the very large LLMs OpenAI/Anthropic deliver.

Yep, unfortunately those local models are noticeably worse. Also models are getting bigger, so even if a local basement rig for a higher quality model is possible right now, that might not be so in the future. Also Zuck and others might stop releasing their weights for the next gen models, then what, just hope they plateau, what if they don't?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#185

Post by schainks »

toddmorey wrote: post #178
alyandon wrote: post #105

Earlier quoted context omitted.

I would be greatly interested in knowing how you set all that up if you felt like sharing the specifics.

My hope is to make this easy with a GH repo or at least detailed instructions. I'm on a Mac and I found the easiest way to run & use local models is Ollama as it has a rest interface: https://github.com/ollama/ollama/blob/main/docs/api.md I just have a local script that pulls the audio file from Voice Memos (after it syncs from my iPhone), runs it through openai's whisper (really the best at voice to speech; excellen…

Amazing, thank you for this!

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#186

Post by ekabod »

tetrisgm wrote: post #7

I like self hosting random stuff on docker. Ollama has been a great addition. I know it's not, but it feels on par with ChatGPT. It works perfectly on my 4090, but I've also seen it work perfectly on my friend's M3 laptop. It feels like an excellent alternative for when you don't need the heavy weights, but want something bespoke and private. I've integrated it with my Obsidian notes for 1) note generation 2) fuzzy s…

Ollama is not a model, it is the sofware to run models.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#187

Post by albertgoeswoof »

fsmv wrote: post #97
albertgoeswoof wrote: post #62

I have a three year old M1 Max, 32gb RAM. Llama 8bn runs at 25 tokens/sec, that’s fast enough, and covers 80% of what I need. On my ryzen 5600h machine, I get about 10 tokens/second, which is slow enough to be annoying. If I get stuck on a problem, switch to chat gpt or phind.com and see what that gives. Sometimes, it’s not the LLM that helps, but changing the context and rewriting the question. However I cannot use…

A small model necessarily is missing many facts. The large model is the one that has memorized the whole internet, the small one is just trained to mimic the big one. You simply cannot compress the whole internet under 10gb without throwing out a lot of information. Please be careful about what you take as fact coming from the local model output. Small models are better suited to summarization.

I don’t trust anything as fact coming out of these models. I ask it for how to structure solutions, with examples. Then I read the output and research the specifics before using anything further.

I wouldn’t copy and paste from even the smartest minds, nevermind a model output.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#188

Post by lukan »

flimflamm wrote: post #155
lukan wrote: post #148

Earlier quoted context omitted.

"before having an LLM clean up my ramblings into organized notes and todo lists." Which local LLM do you use? Edit: And self talk is quite a healthy and useful thing in itself, but avoiding it in public is indeed kind of necessary, because of the stigma https://en.m.wikipedia.org/wiki/Intrapersonal_communication

That's just meat CoT (chain of thought) - right?

I do not understand?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#189

Post by albertgoeswoof »

staticman2 wrote: post #98
albertgoeswoof wrote: post #62

I have a three year old M1 Max, 32gb RAM. Llama 8bn runs at 25 tokens/sec, that’s fast enough, and covers 80% of what I need. On my ryzen 5600h machine, I get about 10 tokens/second, which is slow enough to be annoying. If I get stuck on a problem, switch to chat gpt or phind.com and see what that gives. Sometimes, it’s not the LLM that helps, but changing the context and rewriting the question. However I cannot use…

I'm really curious what you are doing with an LLM that can be solved 80% of the time with a 8b model.

It’s mostly how would you solve this programming problem, or reminders on syntax, scaffolding a configuration file etc.

Often it’s a form of rubber duck programming, with a smarter rubber duck.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#190

Post by wkat4242 »

unethical_ban wrote: post #175
vunderba wrote: post #170

Earlier quoted context omitted.

If you're gonna go with a VS code extension and you're aiming for privacy, then I would at least recommend using the open source fork VS Codium. https://vscodium.com/

It is true that VS Code has some non-optional telemetry, and if VS Codium works for people, that is great. However, the telemetry of VSCode is non-personal metrics, and some of the most popular extensions are only available with VSCode, not with Codium.

> and some of the most popular extensions are only available with VSCode, not with Codium

Which is an artificial restriction from MS that's really easily bypassed.

Personally I don't care whether the telemetry is identifiable. I just don't want it.

Post reply on HN