Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

161–170 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#161

Post by privacyis1mp »

noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

I built Fluid app exactly with that in mind. You can run local AI on mac without really knowing what an LLM/ollama is. Plug&Play.

Sorry for the blatant ad, though I do hope it's useful for some ppl reading this thread: https://getfluid.app

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#162

Post by svnt »

jstummbillig wrote: post #151
svnt wrote: post #117

Earlier quoted context omitted.

I’ll flip this around a bit: If I’ve raised $1B to buy GPUs and train a “bigger model”, a major part of my competitive advantage is having $1B to spend on sufficient GPUs to train a bigger model. If, after having raised that money it becomes apparent that consumer hardware can run smaller models that are optimized and perform as well without all that money going into training them, how am I going to pivot my business…

What you suggest is not impossible but simply flies in the face of all currently available evidence and what all leading labs say and do. We know they are actively looking for ways to do things more efficiently. OpenAI alone did a couple of releases to that effect. Because of how easy it is to switch providers, if only one lab found a way to run a small model that competed with the big ones, it would simply win the e…

There are at least two different optimizations happening:

1) optimizing the model training

2) optimizing the model operation

The $1B-spend holy grail is that it costs a lot of money to train, and almost nothing to operate, a proprietary model that benchmarks and chats better than anyone else’s.

OpenAI’s optimizations fall into the latter category. The risk to the business model is in the former — if someone can train a world-beating model without lots of money, it’s a tough day for the big players.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#163

Post by pickettd »

pella wrote: post #68
throwaway314155 wrote: post #48

Earlier quoted context omitted.

> nearly unusable for 70b models Can Apple Silicon manage this? Would it be feasible to do with some quantization perhaps?

- "2 MacBooks is all you need. Llama 3.1 405B running distributed across 2 MacBooks using @exolabs_ home AI cluster" https://x.com/AIatMeta/status/1834633042339741961 - "Running Qwen 2.5 Math 72B distributed across 2 MacBooks. Uses @exolabs_ with the MLX backend." https://x.com/ac_crypto/status/1836558930585034961

I experimented with both Exo and llama.cpp in RPC-server mode this week. Using an M3 Max and an M1 Ultra in Exo specifically I was able to get around 13 tok/s on DeepSeek 2.5 236B (using MLX and a 4 bit quant with a very small test prompt - so maybe 140 gigs total of model+cache). It definitely took some trial and error but the Exo community folks were super helpful/responsive with debugging/advice.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#164

Post by [deleted] »

visarga wrote: post #89
jannyfer wrote: post #79

Earlier quoted context omitted.

> in the next eight months when roughly 1.5 billion iPhone users will get a local language model tied seamlessly to a mid-tier cloud based language model native in their OS. Only iPhone 15 Pro or later will get Apple Intelligence, so the number will be wayyy smaller.

Not in EU they won't.

[deleted]

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#166

Post by zelphirkalt »

noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Many setup rely on Nvidia GPUs, Intel stuff, Windows or other stuff, that I would rather not use, or are not very clear about how to set things up.

What are some recommendations for running models locally, on decent CPUs and getting good valuable output from them? Is that llama stuff portable across CPUs and hardware vendors? And what do people use it for?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#167

Post by mistrial9 »

Anunayj wrote: post #83

I recently experimented with running llama-3.1-8b-instruct locally on my Consumer hardware, aka my Nvidia RTX 4060 with 8GB VRAM, as I wanted to experiment with prompting pdfs with a large context which is extremely expensive with how LLMs are priced. I was able to fit the model with decent speeds (30 tokens/seconds) and a 20k token context completely on the GPU. For summarization, the performance of these models are…

you moved the goalposts when you add 'multimodal' there; another item is, no one reads PDF tables and illustrations perfectly, at any price AFAIK

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#170

Post by vunderba »

noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

If you're gonna go with a VS code extension and you're aiming for privacy, then I would at least recommend using the open source fork VS Codium.

https://vscodium.com/

Post reply on HN