Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

371–380 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#371

Post by schmidtleonard »

whatindaheck wrote: post #287
schmidtleonard wrote: post #109

Earlier quoted context omitted.

Button-toggled voice notes in the iPhone Notes app are a godsend for taking measurements. Rather than switching your hands between probe/equipment and notes repeatedly, which sucks badly, you can just dictate your readings and maaaaybe clean out something someone said in the background. Over the last decade, the microphones + speech recognition became Good Enough for this. Wake-word/endpoint models still aren't there…

> Button-toggled voice notes in the iPhone Notes app Is this a physical button or on-screen? I’ve been rewatching Twin Peaks recently and would love a high-tech implementation of Cooper’s tape recorder.

It's an on-screen toggle button in the Notes app. Press to start recording, take your time, don't worry about about a 10 second turn around if you pause slightly too long between words, just speak and toggle the button back off when you are done. If someone walked up and had a conversation half way through just delete the words.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#372

Post by drio »

api wrote: post #120

I use ollama through a Mac app called BoltAI quite a bit. It’s like having a smart portable sci-fi “computer assistant” for research and it’s all local. It is about the only thing I can do on my M1 Pro to spin up the fans and make the bottom of the case hot. Llama3.1, Deepseek Coder v2, and some of the Mistral models are good. ChatGPT and Claude top tier models are still better for very hard stuff.

It looks good. Thank you for sharing. Is anyone aware of a similar tool for Linux?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#373

Post by froggit »

a-dub wrote: post #307
mafuy wrote: post #305

Earlier quoted context omitted.

(A brief note: While not weak, the laptop version of a 3080 Ti is far surpassed by even just a desktop 4060 Ti, which is sold for less than 400$. So it's possible to setup a stronger system relatively cheaply. What's good enough depends on the expectations.)

it seemed to me that the bottleneck mostly revolved around the layers that were in system ram and that a lack of vram was really the gating factor in terms of reasonable inference performance. (although i would imagine that there's probably some more optimization that could be done to make best use of a split vram/sysram setup.) in any event it was fun to try out, but still didn't seem anywhere near how well the host…

> it seemed to me that the bottleneck mostly revolved around the layers that were in system ram and that a lack of vram was really the gating factor in terms of reasonable inference performance. (although i would imagine that there's probably some more optimization that could be done to make best use of a split vram/sysram setup.)

You could try a model that fits entirely into in VRAM. It"s a trade of precision for a decent bit of performance. 16GB is plenty to work with as i've seen acceptable enough results with 7B models on my 8GB GPU.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#374

Post by creata »

neop1x wrote: post #370
jonnycomputer wrote: post #363

Earlier quoted context omitted.

RAM and what GPU you have are big determinants of how fast it will run, and how smart a model you can run. A large amount of RAM and GPU memory is required for larger models without significant slowdown because its much faster if it can keep the entire model in memory. Small models range from 3-8 gigabytes, but a 70B parameter model will be 30-50 gigabytes.

I am running 70B models on M2 Max with 96 GB of RAM and it works very well. As HW evolves, it will become a standard

Out of curiosity, what degree of quantization are you applying to these 70B models?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#375

Post by neop1x »

creata wrote: post #374
neop1x wrote: post #370

Earlier quoted context omitted.

I am running 70B models on M2 Max with 96 GB of RAM and it works very well. As HW evolves, it will become a standard

Out of curiosity, what degree of quantization are you applying to these 70B models?

Q4_K_S. While not as good as top commercial models like chatgpt, they are still quite capable and I like that there are also uncensored/abliterated models like Dolphin.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#376

Post by akimbostrawman »

neoberg wrote: post #336
poincaredisk wrote: post #270

Earlier quoted context omitted.

>the telemetry of VSCode is non-personal metrics But I don't want it. I want my software to work for me, not against me. >and some of the most popular extensions are only available with VSCode, not with Codium. I'll manage without them. What's especially annoying is that this restriction is completely artificial. Having said that, MS did a great job with VsCode and I applaud them for that. I guess nothing is perfect,…

> But I don't want it. I want my software to work for me, not against me. How is said software working "against" you by collecint non-personal telemetry while purpose of that telemetry usually is making the software better for most users?

>usually is making the software better for most users?

that usually hasn't been the case since at least a decade. it's truly bewildering that someone especially on hackernews would voluntarily give big tech there finger and not expect to get bitten.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#377

Post by akimbostrawman »

noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

GPT4All for an even easier gui

https://github.com/nomic-ai/gpt4all

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#378

Post by routerl »

perfmode wrote: post #352
routerl wrote: post #58

Earlier quoted context omitted.

Imho opinion, and I'm no expert, but this has been working well for me: Segment the texts into chunks that make sense (i.e. into the lengths of text you'll want to find, whether this means chapters, sub-chapters, paragraphs, etc), create embeddings of each chunk, and store the resultant vectors in a vector database. Your search workflow will then be to create an embedding of your query, and perform a distance compari…

Where are LLMs used in this workflow? For creating embeddings?

Yes.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#379

Post by tripzilch »

it doesn't fill me with confidence that an MD calls themselves a "ML researcher" when clearly they've only been "into ML" since recently LLMs became popular. I could believe that this MD became an "expert in using LLMs" in the past few years (like, we all became), but this guy is training networks based on medical data, without experience in ML going back further than 2022. a little knowledge is dangerous, there's a whole bunch of boring stuff I'd want someone to learn about ML before going "hey LLMs are cool for medical use" ...

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#380

Post by tripzilch »

neoberg wrote: post #336
poincaredisk wrote: post #270

Earlier quoted context omitted.

>the telemetry of VSCode is non-personal metrics But I don't want it. I want my software to work for me, not against me. >and some of the most popular extensions are only available with VSCode, not with Codium. I'll manage without them. What's especially annoying is that this restriction is completely artificial. Having said that, MS did a great job with VsCode and I applaud them for that. I guess nothing is perfect,…

> But I don't want it. I want my software to work for me, not against me. How is said software working "against" you by collecint non-personal telemetry while purpose of that telemetry usually is making the software better for most users?

Software did absolutely not get any better after corporations started adding telemetry to their software.

Point in case: Software actually got worse.

Second point in case: Great software and editors have been built without telemetry for decades.

Post reply on HN