Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

361–370 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#361

Post by noman-land »

soco wrote: post #357
noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Fully agree with you, it should, but after trying a few times different llamas I think they're very far from "try it out in just a few moments". Unless all you want is to see one running, for anything beyond that you'll be in dependency hell...

There is no dependency hell. It's just a single file. If you want to get into trying different models and various settings, you can use LM Studio, and still no need to worry about dependencies.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#362

Post by jonnycomputer »

soco wrote: post #357
noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Fully agree with you, it should, but after trying a few times different llamas I think they're very far from "try it out in just a few moments". Unless all you want is to see one running, for anything beyond that you'll be in dependency hell...

If you just want to chat, download https://lmstudio.ai/, then download their recommended LLM files, and you're good to go. Really that simple these days.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#363

Post by jonnycomputer »

HiPHInch wrote: post #319
noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Thanks for your recommendation! I just ran Llamafile for the first time with a custom prompt on my Windows machine (i5-13600KF, RX6600) and found that it performed extremely slowly and wasn't as smart as ChatGPT. It doesn't seem suitable for productive writing. Did I do something wrong, or is there a way to improve its writing performance?

RAM and what GPU you have are big determinants of how fast it will run, and how smart a model you can run. A large amount of RAM and GPU memory is required for larger models without significant slowdown because its much faster if it can keep the entire model in memory. Small models range from 3-8 gigabytes, but a 70B parameter model will be 30-50 gigabytes.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#364

Post by [deleted] »

alyandon wrote: post #105
toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

I would be greatly interested in knowing how you set all that up if you felt like sharing the specifics.

[deleted]

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#365

Post by heresie-dabord »

mnky9800n wrote: post #216
evbogue wrote: post #207

Earlier quoted context omitted.

From this data we must conclude that within our lifetimes all matter in the universe will eventually be reprogrammed in JavaScript.

I'm not sure I want to live in that reality.

... a reality where everything in software development that was previously established as robust foundation is discarded, only to be re-learned and re-implemented less well while burning VC cash.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#366

Post by aftbit »

prometheon1 wrote: post #327
all2 wrote: post #298

Earlier quoted context omitted.

Jetbrains is pretty ok on this front. I've been enjoying using my beefy computer to do work from my potato laptop.

The way I read it, the message you replied to was a complaint about parts of VSCode being proprietary. Do you mean to say Jetbrains is pretty ok on the "not being proprietary" front?

Yeah, 100%. I'm not a hardcore FOSS only person, but for my core workflow, when a FOSS tool exists and works well, I am not likely to use a proprietary alternative if I can avoid it at all.

So yeah, I'll use Excel to interoperate with fancy spreadsheets, but if LibreOffice will do the job, I'll use it instead. I tried out several of the fancy proprietary editors at various times (SublimeText, VSCode, even Jetbrains), but IMO they were not better _enough_ to justify switching away from something like vim, which is both ubiquitously available and FOSS.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#367

Post by senkora »

senkora wrote: post #356
hmottestad wrote: post #351

Earlier quoted context omitted.

Probably need to try a smaller model :P When the article says that researchers are using their laptops those researchers are either using very small models on a gaming laptop or they have a fairly modern MacBook with a lot of ram. There are also options for running open LLMs in the cloud. Groq (not to be confused with Grok) runs Llama, Mixtral and Gemma models really cheaply: https://groq.com/pricing/

I'll play around with it some more later. I was running llava-v1.5-7b-q4.llamafile which is the example that they recommend trying first at https://github.com/Mozilla-Ocho/llamafile Groq looks interesting and might be a better option for me. Thank you.

I got better performance of 20.18 tokens per second using tinyllama-1.1b-chat-v1.0.Q8_0.llamafile from https://huggingface.co/Bojun-Feng/TinyLlama-1.1B-Chat-v1.0-l...

If anyone is reading this and had trouble with a larger model, that might be the one to try next.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#368

Post by causal »

aftbit wrote: post #285
SoothingSorbet wrote: post #281

Earlier quoted context omitted.

> However, the telemetry of VSCode is non-personal metrics I don't care, I don't want my text editor to send _any_ telemetry, _especially_ without my explicit consent. > some of the most popular extensions are only available with VSCode This has never been an issue for me, fortunately. The only issue is Microsoft's proprietary extensions, which I have no interest in using either. If I wanted a proprietary editor I'd…

I dropped VSCode when I found out that the remote editing and language server extensions were both proprietary. Back to vim and sorry I strayed.

Making the remote editing extension closed is particularly frustrating, as you have little visibility into what it's doing and it is impossible to debug obscure errors

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#369

Post by thebiss »

jonnycomputer wrote: post #362
soco wrote: post #357

Earlier quoted context omitted.

Fully agree with you, it should, but after trying a few times different llamas I think they're very far from "try it out in just a few moments". Unless all you want is to see one running, for anything beyond that you'll be in dependency hell...

If you just want to chat, download https://lmstudio.ai/ , then download their recommended LLM files, and you're good to go. Really that simple these days.

lmstudio prohibits commercial use [1]:

> Subject to the Agreement, Company grants you a limited license to reproduce portions of Company Properties for the sole purpose of using the Services for your personal, non-commercial purposes.

[1] https://lmstudio.ai/terms

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#370

Post by neop1x »

jonnycomputer wrote: post #363
HiPHInch wrote: post #319

Earlier quoted context omitted.

Thanks for your recommendation! I just ran Llamafile for the first time with a custom prompt on my Windows machine (i5-13600KF, RX6600) and found that it performed extremely slowly and wasn't as smart as ChatGPT. It doesn't seem suitable for productive writing. Did I do something wrong, or is there a way to improve its writing performance?

RAM and what GPU you have are big determinants of how fast it will run, and how smart a model you can run. A large amount of RAM and GPU memory is required for larger models without significant slowdown because its much faster if it can keep the entire model in memory. Small models range from 3-8 gigabytes, but a 70B parameter model will be 30-50 gigabytes.

I am running 70B models on M2 Max with 96 GB of RAM and it works very well. As HW evolves, it will become a standard
Post reply on HN