Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

261–270 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#261

Post by throwthrowuknow »

visarga wrote: post #93
throwthrowuknow wrote: post #47

Earlier quoted context omitted.

The trivia include information about many things: grammar, vocabulary, slang, entity relationships, metaphor, among others but chiefly they also constitute models of human thought and behaviour. If all you want is a fancy technical encyclopedia then by all means chop away at the training set but if you want something you can talk to then you’ll need to keep the diversity.

> you’ll need to keep the diversity. You can get diverse low quality data from the web, but for diverse high quality data the organic content is exhausted. The only way is to generate it, and you can maintain a good distribution by structured randomness. For example just sample 5 random words from the dictionary and ask the model to compose a piece of text from them. It will be more diverse than web text.

not exhausted, just not currently being collected. Generating via existing models is ok for distilling a better training set or refining existing low quality samples but won’t break out of distribution without some feedback mechanism. That’s why simulation is promising but it’s pretty narrow at the moment. There’s still a lot of space to fill in the point cloud so coming up with novel data collection methods is important. I think this is off topic though, my original contention was if you take too thin of a slice you won’t get a very useful model.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#262

Post by metadat »

unethical_ban wrote: post #175
vunderba wrote: post #170

Earlier quoted context omitted.

If you're gonna go with a VS code extension and you're aiming for privacy, then I would at least recommend using the open source fork VS Codium. https://vscodium.com/

It is true that VS Code has some non-optional telemetry, and if VS Codium works for people, that is great. However, the telemetry of VSCode is non-personal metrics, and some of the most popular extensions are only available with VSCode, not with Codium.

Not allowing end-users to disable telemetry is actually awful. The gold standard is that IP addresses are considered personally identifiable information.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#263

Post by christkv »

noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

I recommend llmstudio for this usually

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#264

Post by stevenhuang »

jstummbillig wrote: post #112
pella wrote: post #3

Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory. - https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...

Also, next year, there will be GPT 5. I find it fascinating how much attention small models get, when at the same time the big models just get bigger and prohibitively expensive to train. No leading lab would do that if they thought it a decent chance that small models were able to compete. So who will be interested in a shitty assistant next year when you can have an amazing one, is what I wonder? Is this just the b…

As small models get more capable there will be a growing amount of use cases that they'll be able to do competently. Is that so hard to believe?

Leave the problems that require competent reasoning ability to the larger models.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#265

Post by stevenhuang »

aurareturn wrote: post #36
pella wrote: post #3

Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory. - https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...

It will have around 250GB/s of bandwidth which makes it nearly unusable for 70b models. So the high amount of RAM doesn’t help with large models.

Nearly unusable what? High amount of RAM doesn't help with larger models what?

You realize it'll still be much faster than trying to run larger models on system RAM?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#266

Post by brewtide »

fortyseven wrote: post #251
_kidlike wrote: post #141

Earlier quoted context omitted.

Or https://ollama.com/

This has been my go-to for all of my local LLM interaction: it easy to get going, manages all of the models easily. Nice clean API for projects. Updated regularly; works across Windows, Mac, Linux. It's a wrapper around LlamaCpp, but it's a damned good one.

Same here, however minimal. I've also installed openwebui so the instance has a local web interface, and then use tailscale to access my at home LAN when put and about on the cellphone. (Goes16 weather data, ollama, a speed cam setup, and esphome temp sensors around the home / property).

It's been pretty flawless, and honestly pretty darn useful here and there. The big guns go faster and do more, but I'd prefer not having every interaction logged etc.

6core 8th gen i7 I think, with a 1050ti. Old stuff. And it's quick enough on the smaller 7/8b models for sure.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#267

Post by amelius »

noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

How do we rate whether the smaller models are any good? How many questions do we need to ask it to know that it can be trusted and we didn't waste our time on it?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#268

Post by ignoramous »

ycombinatrix wrote: post #233
zozbot234 wrote: post #123

Earlier quoted context omitted.

You actually need a lot less than that if you use the mmap option, because then only activations need to be stored in RAM, the model itself can be read from disk.

How does one set this up?

With ggml the mmap part is the default. It isn't a panacea though [0]. Note that most runtimes (like MLX, ONNX, TensorFlow, JAX/XLA etc) will employ a number of techniques for efficient inference and mmap is just one part of it.

[0] https://news.ycombinator.com/item?id=35455930

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#269

Post by xyc »

noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

If anyone is interested in trying local AI, you can give https://recurse.chat/ a spin.

It lets you use local llama.cpp without setup, chat with PDF offline and provides chat history / nested folders chat organization, and can handle thousands of conversations. In addition you can import your ChatGPT history and continue chats with local AI.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#270

Post by poincaredisk »

unethical_ban wrote: post #175
vunderba wrote: post #170

Earlier quoted context omitted.

If you're gonna go with a VS code extension and you're aiming for privacy, then I would at least recommend using the open source fork VS Codium. https://vscodium.com/

It is true that VS Code has some non-optional telemetry, and if VS Codium works for people, that is great. However, the telemetry of VSCode is non-personal metrics, and some of the most popular extensions are only available with VSCode, not with Codium.

>the telemetry of VSCode is non-personal metrics

But I don't want it. I want my software to work for me, not against me.

>and some of the most popular extensions are only available with VSCode, not with Codium.

I'll manage without them. What's especially annoying is that this restriction is completely artificial.

Having said that, MS did a great job with VsCode and I applaud them for that. I guess nothing is perfect, and I bet these decisions were made by suits against engineer wishes.

Post reply on HN