Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

191–200 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#191

Post by gardnr »

aledalgrande wrote: post #136

Does anyone know of a local "Siri" implementation? Whisper + Llama (or Phi or something else), that can run shortcuts, take notes, read web pages etc.? PS: for reading web pages I know there's voices integrated in the browser/OS but those are horrible

Edit: I just found this. I'll give it a try today: https://github.com/0ssamaak0/SiriLLama

---

Open WebUI has a voice chat but the voices are not great. I'm sure they'd love a PR that integrates StyleTTS2.

You can give it a Serper API Key and it will search the web to use as context. It connects to ollama running on a linux box with a $300 RTX 3060 with 12GB of VRAM. The 4bit quant of Llama 3.1 8B takes up a bit more than 6GB of VRAM which means it can run embedding models and STT on the card at the same time.

12GB is the minimum I'd recommend for running quantized models. The RTX 4070 Ti Super is 3x the cost but 7 times "faster" on matmuls.

The AMD cards do inference OK but they are a constant source of frustration when trying to do anything else. I bought one and tried for 3 months before selling it. It's not worth the effort.

I don't have any interest in allowing it to run shortcuts. Open WebUI has pipelines for integrating function calling. HomeAssistant has some integrations if that's the kind of thing you are thinking about.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#192

Post by narrator »

threecheese wrote: post #174
zelphirkalt wrote: post #166

Earlier quoted context omitted.

Many setup rely on Nvidia GPUs, Intel stuff, Windows or other stuff, that I would rather not use, or are not very clear about how to set things up. What are some recommendations for running models locally, on decent CPUs and getting good valuable output from them? Is that llama stuff portable across CPUs and hardware vendors? And what do people use it for?

Have you tried a Llamafile? Not sure what platform you are using. From their readme: > … by combining llama.cpp with Cosmopolitan Libc into one framework that collapses all the complexity of LLMs down to a single-file executable (called a "llamafile") that runs locally on most computers, with no installation. Low cost to experiment IMO. I am personally using MacOS with an M1 chip and 64gb memory and it works perfectl…

With 64GB can you run the 70B size llama models well?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#193

Post by wkat4242 »

zelphirkalt wrote: post #166
noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Many setup rely on Nvidia GPUs, Intel stuff, Windows or other stuff, that I would rather not use, or are not very clear about how to set things up. What are some recommendations for running models locally, on decent CPUs and getting good valuable output from them? Is that llama stuff portable across CPUs and hardware vendors? And what do people use it for?

Not really. I run ollama on an AMD Radeon Pro and it works great.

For tooling to train models it's a bit more difficult but inference works great on AMD.

My CPU is an AMD Ryzen and the OS Linux. No problem.

I use OpenWebUI as frontend and it's great. I use it for everything that people use GPT for.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#194

Post by kaoD »

noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Well this was my experience...

    User: Hey, how are you?
    Llama: [object Object]
It's funny but I don't think I did anything wrong?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#195

Post by wkat4242 »

MobiusHorizons wrote: post #66
diggan wrote: post #5

Earlier quoted context omitted.

According to Tom's ( https://www.tomshardware.com/pc-components/cpus/amd-pushes-r... ), those are supposed to be laptop CPUs, which makes me wonder what AMD has planned for us desktop users.

If I remember right, in the press conference they suggested desktop users would use a gpu because desktop uses are less power sensitive. That doesn’t address the vram limitations of discrete GPUs though.

True but try to find a 96GB GPU.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#196

Post by sergiotapia »

wazdra wrote: post #108

I'd like to point out that llama 3.1 is not open source[1] (I was recently made aware of that fact by [2], when it was on HN front page) While it's very nice to see a peak of interest for local, "open-weights" LLMs, this is an unfortunate choice of words, as it undermines the quite important differences between llama's license model and open-source. The license question does not seem to be addressed at all in the art…

that ship sailed 13 years ago dude.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#197

Post by noman-land »

wkat4242 wrote: post #190
unethical_ban wrote: post #175

Earlier quoted context omitted.

It is true that VS Code has some non-optional telemetry, and if VS Codium works for people, that is great. However, the telemetry of VSCode is non-personal metrics, and some of the most popular extensions are only available with VSCode, not with Codium.

> and some of the most popular extensions are only available with VSCode, not with Codium Which is an artificial restriction from MS that's really easily bypassed. Personally I don't care whether the telemetry is identifiable. I just don't want it.

How is it bypassed?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#198

Post by twh270 »

privacyis1mp wrote: post #161
noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

I built Fluid app exactly with that in mind. You can run local AI on mac without really knowing what an LLM/ollama is. Plug&Play. Sorry for the blatant ad, though I do hope it's useful for some ppl reading this thread: https://getfluid.app

I'm interested, but I can't find any documentation for it. Can I give it local content (documents, spreadsheets, code, etc.) and ask questions?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#199

Post by wkat4242 »

noman-land wrote: post #197
wkat4242 wrote: post #190

Earlier quoted context omitted.

> and some of the most popular extensions are only available with VSCode, not with Codium Which is an artificial restriction from MS that's really easily bypassed. Personally I don't care whether the telemetry is identifiable. I just don't want it.

How is it bypassed?

There's a whitelist identifier that you can add bundle IDs to, to get access to the more sensitive APIs. Then you can download the extension file and install it manually. I don't have the exact process right now but just Google it :)

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#200

Post by [deleted] »

kaoD wrote: post #194
noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Well this was my experience... User: Hey, how are you? Llama: [object Object] It's funny but I don't think I did anything wrong?

[deleted]
Post reply on HN