Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

111–120 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#111

Post by throwaway314155 »

zdragnar wrote: post #86
throwaway314155 wrote: post #52

Earlier quoted context omitted.

I don't have a horse in the race but wouldn't Meta be more likely to commoditize things given that they sort of already are?

Search Gmail Docs Android Chrome (browser and Chromebooks) I don't use any Meta properties at all, but at least a dozen alphabet ones. My wife uses Facebook, but that's about it. I can see it being handy for insta filters. YMMV of course, but I suspect alphabet has much deeper reach, even if the actual overall number of people is similar.

I was referring to the many quality open models they've released to be clear.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#112

Post by jstummbillig »

pella wrote: post #3

Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory. - https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...

Also, next year, there will be GPT 5. I find it fascinating how much attention small models get, when at the same time the big models just get bigger and prohibitively expensive to train. No leading lab would do that if they thought it a decent chance that small models were able to compete.

So who will be interested in a shitty assistant next year when you can have an amazing one, is what I wonder? Is this just the biggest cup of wishful thinking that we have ever seen?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#113

Post by nurettin »

IshKebab wrote: post #82
pella wrote: post #69

Llama 3.1 405B "2 MacBooks is all you need. Llama 3.1 405B running distributed across 2 MacBooks using @exolabs_ home AI cluster" https://x.com/AIatMeta/status/1834633042339741961

"All you need is £10k of Apple laptops..."

That is... probable, if you bought a newish m2 to replace your 5-6 year old macbook pro which is now just lying around. Or maybe you and your spouse can share cpu hours.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#114

Post by jabroni_salad »

jstummbillig wrote: post #112
pella wrote: post #3

Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory. - https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...

Also, next year, there will be GPT 5. I find it fascinating how much attention small models get, when at the same time the big models just get bigger and prohibitively expensive to train. No leading lab would do that if they thought it a decent chance that small models were able to compete. So who will be interested in a shitty assistant next year when you can have an amazing one, is what I wonder? Is this just the b…

One of the reasons I run local is that the models are completely uncensored and unfiltered. If you're doing anything slightly 'risky' the only thing APIs are good for is a slew of very politely written apology letters, and the definition of 'risky' will change randomly without notice or fail to accommodate novel situations.

It is also evident in the moderation that your usage is subject to human review and I don't think that should even be possible.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#115

Post by Tempest1981 »

jstummbillig wrote: post #112
pella wrote: post #3

Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory. - https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...

Also, next year, there will be GPT 5. I find it fascinating how much attention small models get, when at the same time the big models just get bigger and prohibitively expensive to train. No leading lab would do that if they thought it a decent chance that small models were able to compete. So who will be interested in a shitty assistant next year when you can have an amazing one, is what I wonder? Is this just the b…

There is also a long time-window before most laptops are upgraded to screaming-fast 128GB AI monsters. Either way, it will be fun to watch the battle.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#116

Post by Larrikin »

jstummbillig wrote: post #112
pella wrote: post #3

Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory. - https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...

Also, next year, there will be GPT 5. I find it fascinating how much attention small models get, when at the same time the big models just get bigger and prohibitively expensive to train. No leading lab would do that if they thought it a decent chance that small models were able to compete. So who will be interested in a shitty assistant next year when you can have an amazing one, is what I wonder? Is this just the b…

Why would anyone buy a Raspberry Pi when they can get a fully decked out Mac Pro?

There are different use cases and computers are already pretty powerful. Maybe your local model won't be able to produce tests that check all the corner cases of the class you just wrote for work in your massive code base.

But the small model is perfectly capable of summarizing the weather from an API call and maybe tack on a joke that can be read out to you on your speakers in the morning.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#117

Post by svnt »

jstummbillig wrote: post #112
pella wrote: post #3

Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory. - https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...

Also, next year, there will be GPT 5. I find it fascinating how much attention small models get, when at the same time the big models just get bigger and prohibitively expensive to train. No leading lab would do that if they thought it a decent chance that small models were able to compete. So who will be interested in a shitty assistant next year when you can have an amazing one, is what I wonder? Is this just the b…

I’ll flip this around a bit:

If I’ve raised $1B to buy GPUs and train a “bigger model”, a major part of my competitive advantage is having $1B to spend on sufficient GPUs to train a bigger model.

If, after having raised that money it becomes apparent that consumer hardware can run smaller models that are optimized and perform as well without all that money going into training them, how am I going to pivot my business to something that works, given these smaller models are released this way on purpose to undermine my efforts?

It seems there are two major possibilities: one, people raising billions find a new and expensive intelligence step function that at least time-locally separates them from the pack, or two (and significantly more likely in my view) they don’t, and the improvements come from layering on different systems such as do not require acres of GPUs, while the “more data more GPUs” crowd is found to have hit a nonlinearity that in practical terms means they are generations of technology away from the next tier.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#119

Post by 123yawaworht456 »

theodorthe5 wrote: post #22

Local LLMs are terrible compared to Claude/ChatGPT. They are useful to use as APIs for applications: much cheaper than paying for OpenAI services, and can be fine tuned to do many useful (and less useful, even illegal) things. But for the casual user, they suck compared to the very large LLMs OpenAI/Anthropic deliver.

they are the only thing you can use if you don't want to or aren't allowed to hand over your data to US corporations and intelligence agencies.

every single query to ChatGPT/Claude/Gemini/etc will be used for any purpose, by any party, at any time. shamelessly so, because this is the new normal. Welcome to 2024. I own nothing, have no privacy, and life has never been better.

>(and less useful, even illegal) things

the same illegal things you can do with Notepad, or a pencil and a piece of paper.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#120

Post by api »

I use ollama through a Mac app called BoltAI quite a bit. It’s like having a smart portable sci-fi “computer assistant” for research and it’s all local.

It is about the only thing I can do on my M1 Pro to spin up the fans and make the bottom of the case hot.

Llama3.1, Deepseek Coder v2, and some of the Mistral models are good.

ChatGPT and Claude top tier models are still better for very hard stuff.

Post reply on HN