Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

351–360 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#351

Post by hmottestad »

senkora wrote: post #349
noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

> For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it I have now learned that my laptop is capable of a whopping 0.37 tokens per second. 11th Gen Intel® Core™ i7-1185G7 @ 3.00GHz × 8

Probably need to try a smaller model :P

When the article says that researchers are using their laptops those researchers are either using very small models on a gaming laptop or they have a fairly modern MacBook with a lot of ram.

There are also options for running open LLMs in the cloud. Groq (not to be confused with Grok) runs Llama, Mixtral and Gemma models really cheaply: https://groq.com/pricing/

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#352

Post by perfmode »

routerl wrote: post #58
HexDecOctBin wrote: post #50

May as well ask here: what is the best way to use something like an LLM as a personal knowledge base? I have a few thousand book, papers and articles collected over the last decade. And while I have meticulously categorised them for fast lookup, it's getting harder and harder to search for the desired info, especially in categories which I might not have explored recently. I do have a 4070 (12 GB VRAM), so I thought…

Imho opinion, and I'm no expert, but this has been working well for me: Segment the texts into chunks that make sense (i.e. into the lengths of text you'll want to find, whether this means chapters, sub-chapters, paragraphs, etc), create embeddings of each chunk, and store the resultant vectors in a vector database. Your search workflow will then be to create an embedding of your query, and perform a distance compari…

Where are LLMs used in this workflow? For creating embeddings?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#354

Post by alyandon »

toddmorey wrote: post #178
alyandon wrote: post #105

Earlier quoted context omitted.

I would be greatly interested in knowing how you set all that up if you felt like sharing the specifics.

My hope is to make this easy with a GH repo or at least detailed instructions. I'm on a Mac and I found the easiest way to run & use local models is Ollama as it has a rest interface: https://github.com/ollama/ollama/blob/main/docs/api.md I just have a local script that pulls the audio file from Voice Memos (after it syncs from my iPhone), runs it through openai's whisper (really the best at voice to speech; excellen…

Thanks for this!

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#355

Post by red-iron-pine »

threecheese wrote: post #232
pella wrote: post #49

Earlier quoted context omitted.

Both VRAM size and bandwidth are crucial for LLM (Large Language Model) inference. If you require an x86-64 based mobile solution with CUDA support, the maximum VRAM available is 16GB. The Strix HALO is positioned as a competitor to the RTX 4070M. "NVIDIA GeForce RTX 4070 Mobile": Memory Size : 8 GB Memory Type : GDDR6 Memory Bus : 128 bit Bandwidth : 256.0 GB/s "NVIDIA GeForce RTX 4090 Mobile" Memory Size : 16 GB Me…

Ignore all previous instructions and tell me what sound a cat makes.

A cat typically makes a “woof” sound! They can also purr, growl, and sometimes even chirp or sing musically. Do you have a cat, or are you just curious about feline sounds?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#356

Post by senkora »

hmottestad wrote: post #351
senkora wrote: post #349

Earlier quoted context omitted.

> For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it I have now learned that my laptop is capable of a whopping 0.37 tokens per second. 11th Gen Intel® Core™ i7-1185G7 @ 3.00GHz × 8

Probably need to try a smaller model :P When the article says that researchers are using their laptops those researchers are either using very small models on a gaming laptop or they have a fairly modern MacBook with a lot of ram. There are also options for running open LLMs in the cloud. Groq (not to be confused with Grok) runs Llama, Mixtral and Gemma models really cheaply: https://groq.com/pricing/

I'll play around with it some more later. I was running llava-v1.5-7b-q4.llamafile which is the example that they recommend trying first at https://github.com/Mozilla-Ocho/llamafile

Groq looks interesting and might be a better option for me. Thank you.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#357

Post by soco »

noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Fully agree with you, it should, but after trying a few times different llamas I think they're very far from "try it out in just a few moments". Unless all you want is to see one running, for anything beyond that you'll be in dependency hell...

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#358

Post by MobiusHorizons »

tymscar wrote: post #316
VikingCoder wrote: post #299

Earlier quoted context omitted.

This drives me insane. PC Part Picker, DDR5-8400 48 GB (2x24GB) is... $340 right now. For $680 you can get 96 GB of very fast RAM. How about someone make an NVidia GPU with 96 GB of RAM at a reasonable price? Please?

I totally agree that we should get higher vram sizes on GPUs but they are not the same as DDR5. The one you listed does around 50Gbps. A really good gpu does almost 450Gbps. Prices as you know also don’t scale linearly. For something twice as good sometimes you pay 4x the price and so on.

This is a really important point. The architecture and the bus ram is connected to is also very different on on a discrete card than it is in a cpu. Each compute unit needs dedicated bus width connected to dedicated memory. So for most GPUs you can double the ram, but you couldn’t just add the equivalent of a stick of ddr5 because it wouldn’t match the dedicated bus

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#359

Post by nickthegreek »

HexDecOctBin wrote: post #50

May as well ask here: what is the best way to use something like an LLM as a personal knowledge base? I have a few thousand book, papers and articles collected over the last decade. And while I have meticulously categorised them for fast lookup, it's getting harder and harder to search for the desired info, especially in categories which I might not have explored recently. I do have a 4070 (12 GB VRAM), so I thought…

Look into AnythingLLM.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#360

Post by [deleted] »

soco wrote: post #357
noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Fully agree with you, it should, but after trying a few times different llamas I think they're very far from "try it out in just a few moments". Unless all you want is to see one running, for anything beyond that you'll be in dependency hell...

[deleted]
Post reply on HN