Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

301–310 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#301

Post by gardnr »

aledalgrande wrote: post #238
gardnr wrote: post #191

Earlier quoted context omitted.

Edit: I just found this. I'll give it a try today: https://github.com/0ssamaak0/SiriLLama --- Open WebUI has a voice chat but the voices are not great. I'm sure they'd love a PR that integrates StyleTTS2. You can give it a Serper API Key and it will search the web to use as context. It connects to ollama running on a linux box with a $300 RTX 3060 with 12GB of VRAM. The 4bit quant of Llama 3.1 8B takes up a bit more…

That SiriLLama project looks awesome! I'll give it a try. I also just spun up https://github.com/ItzCrazyKns/Perplexica to try a local Perplexity alternative.

I'll check it out. There is also LlocalSearch: https://news.ycombinator.com/item?id=39923404

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#302

Post by MrDrMcCoy »

talldayo wrote: post #240
MrDrMcCoy wrote: post #204

Earlier quoted context omitted.

Since when did Broadcom provide those?

Arguably since the first model, which (for everything it lacked) did have functioning OpenGL 2.0-compliant drivers.

My memory is fuzzy, but I recall that some models had very limited hardware acceleration support in the driver stack for things like video codecs, OpenCL, and Vulcan, unless you used the official kernel with the Broadcom blob. I never liked running that due to bloat and the age of the kernel/Debian they ship. All that combined with the performance of the SOC compared to its peers from Rockchip/Mediatek/Samsung and lack of eMMC support pretty much drove me away from Raspberry Pi devices in favor of Radxa and ODROID boards.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#303

Post by svnt »

ComputerGuru wrote: post #246
svnt wrote: post #162

Earlier quoted context omitted.

There are at least two different optimizations happening: 1) optimizing the model training 2) optimizing the model operation The $1B-spend holy grail is that it costs a lot of money to train, and almost nothing to operate, a proprietary model that benchmarks and chats better than anyone else’s. OpenAI’s optimizations fall into the latter category. The risk to the business model is in the former — if someone can train…

I disagree. Not axiomatically because you’re kind of right, but enough to comment. OpenAI doesn’t believe in optimizing the traisning costs of AI but believes in optimizing (read: maxing) the training period. Their billions go to collecting, collating, and transforming as much training data as they can get their hands on. To see what optimizing model operation looks like, groq is a good example. OpenAI isn’t (yet) ob…

My argument wasn’t that the well-funded entities were optimizing to reduce training costs, but the opposite: they need creative ways to spend $1B that provide some tangible advantage. But they need operating costs to be low or they lose money and try to somehow make it up on volume.

I would roll data acquisition/cleaning processes into training costs for purposes of this because what else is the data for if not training?

If 4o wasn’t an optimization for model operation costs what was it?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#304

Post by a-dub »

noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

idk, i have a pretty powerful laptop with 16GB VRAM and a 3080ti. when i've played with quantized llama2 and llama3 (with llamacpp), it was kinda underwhelming. inference was slow, the laptop would heat up and the results weren't as good. (is llama3.1 better?)

this was with 4bit quantization and offload of as many layers as possible to the gpu.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#305

Post by mafuy »

a-dub wrote: post #304
noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

idk, i have a pretty powerful laptop with 16GB VRAM and a 3080ti. when i've played with quantized llama2 and llama3 (with llamacpp), it was kinda underwhelming. inference was slow, the laptop would heat up and the results weren't as good. (is llama3.1 better?) this was with 4bit quantization and offload of as many layers as possible to the gpu.

(A brief note: While not weak, the laptop version of a 3080 Ti is far surpassed by even just a desktop 4060 Ti, which is sold for less than 400$. So it's possible to setup a stronger system relatively cheaply. What's good enough depends on the expectations.)

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#306

Post by darby_nine »

qingdao99 wrote: post #176
darby_nine wrote: post #80

Earlier quoted context omitted.

I expect people will just ship with their own model where the built-in one isn't sufficient. When people describe it as a "critical tool" i feel like I'm missing basic information about how people use computers and interact with the world. In what way is it critical for anything? It's still just a toy at this point.

When it's expected to be handling reminders, calendar events, and other device functions for millions of users, it will be considered critical.

I expect spamassin will work as well as it always did. The difference now is that we're not allowed fine-grained control over whats spammed at us.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#307

Post by a-dub »

mafuy wrote: post #305
a-dub wrote: post #304

Earlier quoted context omitted.

idk, i have a pretty powerful laptop with 16GB VRAM and a 3080ti. when i've played with quantized llama2 and llama3 (with llamacpp), it was kinda underwhelming. inference was slow, the laptop would heat up and the results weren't as good. (is llama3.1 better?) this was with 4bit quantization and offload of as many layers as possible to the gpu.

(A brief note: While not weak, the laptop version of a 3080 Ti is far surpassed by even just a desktop 4060 Ti, which is sold for less than 400$. So it's possible to setup a stronger system relatively cheaply. What's good enough depends on the expectations.)

it seemed to me that the bottleneck mostly revolved around the layers that were in system ram and that a lack of vram was really the gating factor in terms of reasonable inference performance. (although i would imagine that there's probably some more optimization that could be done to make best use of a split vram/sysram setup.)

in any event it was fun to try out, but still didn't seem anywhere near how well the hosted models work. a heavy duty workstation with a bunch of gpus/vram would probably be a different story though.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#308

Post by giancarlostoro »

xyc wrote: post #269
noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

If anyone is interested in trying local AI, you can give https://recurse.chat/ a spin. It lets you use local llama.cpp without setup, chat with PDF offline and provides chat history / nested folders chat organization, and can handle thousands of conversations. In addition you can import your ChatGPT history and continue chats with local AI.

I dont see any indication that it runs on Linux, I'll stick to Jan which is free.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#309

Post by teaearlgraycold »

andersa wrote: post #291
teaearlgraycold wrote: post #257

Earlier quoted context omitted.

H100 NVL is easily available. It’s just that it’s close to $20k.

It is actually much more expensive than 20k. I can find them between 25-30k only.

My bad. Looks like it’s $29k at the local store.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#310

Post by jjtheblunt »

neom wrote: post #171
toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

This is exactly why I think the AI pins are a good idea. The Humane pin seems too big/too expensive/not quite there yet, but for exactly what you're doing, I would like some type of brooch.

how is that preferable to an airpod with siri or google or whatever app one has on a phone?
Post reply on HN