Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

61–70 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#61

Post by jclulow »

roywiggins wrote: post #51
bionhoward wrote: post #24

Earlier quoted context omitted.

No ridiculous prohibitions on training on logs… Man, imagine being OpenAI and flushing your brand down the toilet with an explicit customer noncompete rule which totally backfires and inspires 100x more competition than it prevents

Llama's license does forbid it: "Llama 3.1 materials or outputs cannot be used to improve or train any other large language models outside of the Llama family." https://llamaimodel.com/commercial-use/

I'm not sure why anybody would respect that licence term, given the whole field rests on the rapacious misappropriation of other people's intellectual property.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#62

Post by albertgoeswoof »

I have a three year old M1 Max, 32gb RAM. Llama 8bn runs at 25 tokens/sec, that’s fast enough, and covers 80% of what I need. On my ryzen 5600h machine, I get about 10 tokens/second, which is slow enough to be annoying.

If I get stuck on a problem, switch to chat gpt or phind.com and see what that gives. Sometimes, it’s not the LLM that helps, but changing the context and rewriting the question.

However I cannot use the online providers for anything remotely sensitive, which is more often than you might think.

Local LLMs are the future, it’s like having your own private Google running locally.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#63

Post by candiddevmike »

ronsor wrote: post #60
roywiggins wrote: post #51

Earlier quoted context omitted.

Llama's license does forbid it: "Llama 3.1 materials or outputs cannot be used to improve or train any other large language models outside of the Llama family." https://llamaimodel.com/commercial-use/

Meta dropped that term, actually, and that's an unofficial website.

It's still present in the llama license...?

https://ai.meta.com/llama/license/

Section 1.b.iv

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#64

Post by sigmoid10 »

ronsor wrote: post #60
roywiggins wrote: post #51

Earlier quoted context omitted.

Llama's license does forbid it: "Llama 3.1 materials or outputs cannot be used to improve or train any other large language models outside of the Llama family." https://llamaimodel.com/commercial-use/

Meta dropped that term, actually, and that's an unofficial website.

>If you use the Llama Materials to create, train, fine tune, or otherwise improve an AI model, which is distributed or made available, you shall also include “Llama 3” at the beginning of any such AI model name.

The official llama 3 repo still says this, which is a different phrasing but effectively equal in meaning to what the commenter above said.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#65

Post by jmount »

I think this is a big deal. In my opinion, many money making stable AI services are going to be deliberately of limited ability on limited domains. One doesn't want one's site help bot answering political questions. So this could really pull much of the revenue away from AI/LLMs as service.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#66

Post by MobiusHorizons »

diggan wrote: post #5
pella wrote: post #3

Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory. - https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...

According to Tom's ( https://www.tomshardware.com/pc-components/cpus/amd-pushes-r... ), those are supposed to be laptop CPUs, which makes me wonder what AMD has planned for us desktop users.

If I remember right, in the press conference they suggested desktop users would use a gpu because desktop uses are less power sensitive. That doesn’t address the vram limitations of discrete GPUs though.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#67

Post by rkwz »

exe34 wrote: post #30
tetrisgm wrote: post #7

I like self hosting random stuff on docker. Ollama has been a great addition. I know it's not, but it feels on par with ChatGPT. It works perfectly on my 4090, but I've also seen it work perfectly on my friend's M3 laptop. It feels like an excellent alternative for when you don't need the heavy weights, but want something bespoke and private. I've integrated it with my Obsidian notes for 1) note generation 2) fuzzy s…

which model are you using? what size/quant/etc? thanks!

Not the parent, but I started using Llama 3.1 8b and it's very good.

I'd say it's as good as or better than GPT 3.5 based on my usage. Some benchmarks: https://ai.meta.com/blog/meta-llama-3-1/

Looking forward to try other models like Qwen and Phi in near future.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#68

Post by pella »

throwaway314155 wrote: post #48
aurareturn wrote: post #36

Earlier quoted context omitted.

It will have around 250GB/s of bandwidth which makes it nearly unusable for 70b models. So the high amount of RAM doesn’t help with large models.

> nearly unusable for 70b models Can Apple Silicon manage this? Would it be feasible to do with some quantization perhaps?

- "2 MacBooks is all you need. Llama 3.1 405B running distributed across 2 MacBooks using @exolabs_ home AI cluster" https://x.com/AIatMeta/status/1834633042339741961

- "Running Qwen 2.5 Math 72B distributed across 2 MacBooks. Uses @exolabs_ with the MLX backend." https://x.com/ac_crypto/status/1836558930585034961

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#70

Post by smcleod »

aurareturn wrote: post #36
pella wrote: post #3

Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory. - https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...

It will have around 250GB/s of bandwidth which makes it nearly unusable for 70b models. So the high amount of RAM doesn’t help with large models.

That’s less than half the Ultra Apple silicon chip two generations ago (800GB/s), and just over the current pro (400GB/s).
Post reply on HN