Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

51–60 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#51

Post by roywiggins »

bionhoward wrote: post #24
diggan wrote: post #4

Summary: It's cheaper, safer for handling sensitive data, easier to reproduce results (only way to be 100% sure it's reproduce even, as "external" models can change anytime), higher degree of customization, no internet connectivity requirements, more efficient, more flexible.

No ridiculous prohibitions on training on logs… Man, imagine being OpenAI and flushing your brand down the toilet with an explicit customer noncompete rule which totally backfires and inspires 100x more competition than it prevents

Llama's license does forbid it:

"Llama 3.1 materials or outputs cannot be used to improve or train any other large language models outside of the Llama family."

https://llamaimodel.com/commercial-use/

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#52

Post by throwaway314155 »

brap wrote: post #14

Some companies (OpenAI, Anthropic…) base their whole business on hosted closed source models. What’s going to happen when all of this inevitably gets commoditized? This is why I’m putting my money on Google in the long run. They have the reach to make it useful and the monetization behemoth to make it profitable.

I don't have a horse in the race but wouldn't Meta be more likely to commoditize things given that they sort of already are?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#54

Post by bearjaws »

aurareturn wrote: post #36
pella wrote: post #3

Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory. - https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...

It will have around 250GB/s of bandwidth which makes it nearly unusable for 70b models. So the high amount of RAM doesn’t help with large models.

And that is nearly half of what M4 will produce (450GB/s estimated).

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#55

Post by throwaway314155 »

simion314 wrote: post #8

OpenAI APIs for GPT and Dalle have issues like non determnism, and their special prompt injection where they add stuff or modify your prompt (with no option to turn that off. Makes it impossible to do research or to debug as a developer variations of things.

While that's true for their ChatGPT SaaS, the API they provide doesn't impose as many restrictions.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#56

Post by adrian_b »

diggan wrote: post #5
pella wrote: post #3

Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory. - https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...

According to Tom's ( https://www.tomshardware.com/pc-components/cpus/amd-pushes-r... ), those are supposed to be laptop CPUs, which makes me wonder what AMD has planned for us desktop users.

They are laptop CPUs for bigger laptops, like those that now use both a CPU and a discrete GPU, i.e. gaming laptops or mobile workstations.

It seems that the thermal design power for Strix Halo can be configured between 55 W and 120 W, which is similar to the power used now by a combo laptop CPU + discrete GPU.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#57

Post by aurareturn »

throwaway314155 wrote: post #48
aurareturn wrote: post #36

Earlier quoted context omitted.

It will have around 250GB/s of bandwidth which makes it nearly unusable for 70b models. So the high amount of RAM doesn’t help with large models.

> nearly unusable for 70b models Can Apple Silicon manage this? Would it be feasible to do with some quantization perhaps?

Yes at around 8 tokens/s. Also quite slow.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#58

Post by routerl »

HexDecOctBin wrote: post #50

May as well ask here: what is the best way to use something like an LLM as a personal knowledge base? I have a few thousand book, papers and articles collected over the last decade. And while I have meticulously categorised them for fast lookup, it's getting harder and harder to search for the desired info, especially in categories which I might not have explored recently. I do have a 4070 (12 GB VRAM), so I thought…

Imho opinion, and I'm no expert, but this has been working well for me:

Segment the texts into chunks that make sense (i.e. into the lengths of text you'll want to find, whether this means chapters, sub-chapters, paragraphs, etc), create embeddings of each chunk, and store the resultant vectors in a vector database. Your search workflow will then be to create an embedding of your query, and perform a distance comparison (e.g. cosine similarity) which returns ranked results. This way you can now semantically search your texts.

Everything I've mentioned above is fairly easily doable with existing LLM libraries like langchain or llamaindex. For reference, this is an RAG workflow.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#59

Post by bearjaws »

throwaway314155 wrote: post #48
aurareturn wrote: post #36

Earlier quoted context omitted.

It will have around 250GB/s of bandwidth which makes it nearly unusable for 70b models. So the high amount of RAM doesn’t help with large models.

> nearly unusable for 70b models Can Apple Silicon manage this? Would it be feasible to do with some quantization perhaps?

Any of the newer M2+ Max chips runs 400GB/s and can run 70b pretty well. It's not fast though, 3-4 token/s.

You can get better performance using a good CPU + 4090 + offloading layers to GPU. However one is a laptop and the other is a desktop...

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#60

Post by ronsor »

roywiggins wrote: post #51
bionhoward wrote: post #24

Earlier quoted context omitted.

No ridiculous prohibitions on training on logs… Man, imagine being OpenAI and flushing your brand down the toilet with an explicit customer noncompete rule which totally backfires and inspires 100x more competition than it prevents

Llama's license does forbid it: "Llama 3.1 materials or outputs cannot be used to improve or train any other large language models outside of the Llama family." https://llamaimodel.com/commercial-use/

Meta dropped that term, actually, and that's an unofficial website.
Post reply on HN