Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

311–320 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#311

Post by zaptrem »

lhl wrote: post #249
staticman2 wrote: post #96

Earlier quoted context omitted.

Apparently Mac purchasers like to talk about tokens per second without talking about Mac's atrocious time to first token. They also like to enthusiastically talk about tokens per second asking a 200 token question rather than a longer prompt. I'm not sure what the impact is on a 70b model but it seems there's a lot of exaggeration going on in this space by Mac fans.

For those interested, a few months ago someone posted benchmarks with their MBP 14 w/ an M3 Max [1] (128GB, 40CU, theoretical: 28.4 FP16 TFLOPS, 400GB/s MBW) The results for Llama 2 70B Q4_0 (39GB) was 8.5 tok/s for text generation (you'd expect a theoretical max of a bit over 10 tok/s based on theoretical MBW) and a prompt processing of 19 tok/s. On a 4K context conversation, that means you would be waiting about 3.…

> On a 4K context conversation, that means you would be waiting about 3.5min between turns before tokens started outputting.

Wouldn't the time be negligible with interturn kv caching? Many inference providers already do this.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#312

Post by [deleted] »

jxcl wrote: post #205
toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

This has inspired me. I do a lot of stargazing and have experimented with voice memos for recording my observations. The problem of course is later going back and listening to the voice memo and getting organized information out of what essentially turns into me rambling to myself. I'm going to try to use whisper + AI to transcribe my voice memos into structured notes.

[deleted]

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#313

Post by adpirz »

toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

YES. I discovered this exact use case myself a few months ago. Tweeted about it even:

https://x.com/adpirz/status/1823727814191014323?s=46

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#314

Post by neom »

jjtheblunt wrote: post #310
neom wrote: post #171

Earlier quoted context omitted.

This is exactly why I think the AI pins are a good idea. The Humane pin seems too big/too expensive/not quite there yet, but for exactly what you're doing, I would like some type of brooch.

how is that preferable to an airpod with siri or google or whatever app one has on a phone?

Wish I was good enough at holding thought, on my long thinking walks, sometimes even just a screen is enough to kill the thought I want to flesh out. Typically I just take a notepad but the pin thing would be better!

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#315

Post by wiether »

mafuy wrote: post #305
a-dub wrote: post #304

Earlier quoted context omitted.

idk, i have a pretty powerful laptop with 16GB VRAM and a 3080ti. when i've played with quantized llama2 and llama3 (with llamacpp), it was kinda underwhelming. inference was slow, the laptop would heat up and the results weren't as good. (is llama3.1 better?) this was with 4bit quantization and offload of as many layers as possible to the gpu.

(A brief note: While not weak, the laptop version of a 3080 Ti is far surpassed by even just a desktop 4060 Ti, which is sold for less than 400$. So it's possible to setup a stronger system relatively cheaply. What's good enough depends on the expectations.)

Unless you have special needs like very high usage, privacy or other ones depicted in the article, buying another computer many hundred dollars for the unique purpose of running local models is a hard sell.

If you use their API instead of their sub-based offers, the most popular models are cheap to use and with BYOK tools, switching model is as easy as entering another string in a form.

For instance I put $15 on my OpenAI account in August 2023, since then I used Dall-E weekly and I still got more than $5 credit left!

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#316

Post by tymscar »

VikingCoder wrote: post #299
wkat4242 wrote: post #195

Earlier quoted context omitted.

True but try to find a 96GB GPU.

This drives me insane. PC Part Picker, DDR5-8400 48 GB (2x24GB) is... $340 right now. For $680 you can get 96 GB of very fast RAM. How about someone make an NVidia GPU with 96 GB of RAM at a reasonable price? Please?

I totally agree that we should get higher vram sizes on GPUs but they are not the same as DDR5.

The one you listed does around 50Gbps. A really good gpu does almost 450Gbps. Prices as you know also don’t scale linearly. For something twice as good sometimes you pay 4x the price and so on.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#317

Post by lhl »

zaptrem wrote: post #311
lhl wrote: post #249

Earlier quoted context omitted.

For those interested, a few months ago someone posted benchmarks with their MBP 14 w/ an M3 Max [1] (128GB, 40CU, theoretical: 28.4 FP16 TFLOPS, 400GB/s MBW) The results for Llama 2 70B Q4_0 (39GB) was 8.5 tok/s for text generation (you'd expect a theoretical max of a bit over 10 tok/s based on theoretical MBW) and a prompt processing of 19 tok/s. On a 4K context conversation, that means you would be waiting about 3.…

> On a 4K context conversation, that means you would be waiting about 3.5min between turns before tokens started outputting. Wouldn't the time be negligible with interturn kv caching? Many inference providers already do this.

Yes, for single user multiturn kvcache reuse could help a lot. vLLM has support for this via Automatic Prefix Caching (APC) so you’d be able to take advantage of this w/ Strix Halo now. llama.cpp has had a “prompt-cache” option but when I last looked it was a bit weird (only works for non-interactive use, saves and loads cache to disk) so it might not help on the Mac side.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#318

Post by [deleted] »

poincaredisk wrote: post #270
unethical_ban wrote: post #175

Earlier quoted context omitted.

It is true that VS Code has some non-optional telemetry, and if VS Codium works for people, that is great. However, the telemetry of VSCode is non-personal metrics, and some of the most popular extensions are only available with VSCode, not with Codium.

>the telemetry of VSCode is non-personal metrics But I don't want it. I want my software to work for me, not against me. >and some of the most popular extensions are only available with VSCode, not with Codium. I'll manage without them. What's especially annoying is that this restriction is completely artificial. Having said that, MS did a great job with VsCode and I applaud them for that. I guess nothing is perfect,…

[deleted]

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#319

Post by HiPHInch »

noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Thanks for your recommendation! I just ran Llamafile for the first time with a custom prompt on my Windows machine (i5-13600KF, RX6600) and found that it performed extremely slowly and wasn't as smart as ChatGPT. It doesn't seem suitable for productive writing. Did I do something wrong, or is there a way to improve its writing performance?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#320

Post by csmpltn »

meiraleal wrote: post #77
csmpltn wrote: post #21

Earlier quoted context omitted.

There's plenty of competition in this space already, and it'll only get accelerated with time. There's not enough "moat" in building proprietary LLMs - you can tell by how the leading companies in this space are basically down to fighting over patents and regulatory capture (ie. mounting legal and technical barriers to scraping, procuring hardware, locking down datasets, releasing less information to the public about…

And then the successful chatgpt wrappers with traction will become valuable than the companies creating propietary LLMs. I bet openai will start buying many AI apps to find profitable niches.

Correct, since the competitive edge is in the domain-specific data (which OpenAI, at-least on-paper, shouldn't have access to).

Two things to remember:

1. OpenAI can analyze which "wrappers" or "apps" are most successful, and make better purchasing decisions that way. This is information which isn't available outside of OpenAI.

2. OpenAI can in theory analyze the actual queries and interactions in an organization, record them, analyze, etc - in an attempt to get a hold of the organization's internal data. Unclear on the legality of this, but could perhaps be enforced through a draconic license.

Post reply on HN