Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

251–260 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#251

Post by fortyseven »

_kidlike wrote: post #141
noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Or https://ollama.com/

This has been my go-to for all of my local LLM interaction: it easy to get going, manages all of the models easily. Nice clean API for projects. Updated regularly; works across Windows, Mac, Linux. It's a wrapper around LlamaCpp, but it's a damned good one.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#252

Post by skydhash »

albertgoeswoof wrote: post #189
staticman2 wrote: post #98

Earlier quoted context omitted.

I'm really curious what you are doing with an LLM that can be solved 80% of the time with a 8b model.

It’s mostly how would you solve this programming problem, or reminders on syntax, scaffolding a configuration file etc. Often it’s a form of rubber duck programming, with a smarter rubber duck.

All of this can be solved with a 3-20MB PDF file, a 10kb snippet/template file, and a whiteboard.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#253

Post by noman-land »

ComputerGuru wrote: post #241
noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Do you know if whisperfile is akin to whisper or the much better whisperx? Does it do diarization?

Last I checked it was basically just whisper.cpp so not whisperx and no diarization by default but it moves pretty quickly so you may want to ask on the Mozilla AI Discord.

https://discord.com/invite/yTPd7GVG3H

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#254

Post by draebek »

toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

For people on macOS, the free app Aiko on the App Store makes it easy to use Whisper, if you want a GUI: https://sindresorhus.com/aiko

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#255

Post by AustinDev »

noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

https://old.reddit.com/r/LocalLLaMA/ is a great community for this sort of thing as well.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#256

Post by anigbrowl »

lukan wrote: post #148
toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

"before having an LLM clean up my ramblings into organized notes and todo lists." Which local LLM do you use? Edit: And self talk is quite a healthy and useful thing in itself, but avoiding it in public is indeed kind of necessary, because of the stigma https://en.m.wikipedia.org/wiki/Intrapersonal_communication

Just put in some earbuds and everyone will assume you're on the phone.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#257

Post by teaearlgraycold »

wkat4242 wrote: post #195
MobiusHorizons wrote: post #66

Earlier quoted context omitted.

If I remember right, in the press conference they suggested desktop users would use a gpu because desktop uses are less power sensitive. That doesn’t address the vram limitations of discrete GPUs though.

True but try to find a 96GB GPU.

H100 NVL is easily available. It’s just that it’s close to $20k.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#258

Post by rNULLED »

skydhash wrote: post #252
albertgoeswoof wrote: post #189

Earlier quoted context omitted.

It’s mostly how would you solve this programming problem, or reminders on syntax, scaffolding a configuration file etc. Often it’s a form of rubber duck programming, with a smarter rubber duck.

All of this can be solved with a 3-20MB PDF file, a 10kb snippet/template file, and a whiteboard.

Don’t forget the duct tape

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#260

Post by upcoming-sesame »

noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

I just tried now. Super easy indeed but slow to the point it's not usable on my PC
Post reply on HN