171–180 of 273 posts
Earlier quoted context omitted.
749 for an M4 air at Amazon right now
Try running anything interesting on these 8gb of ram. You need 96gb or 128gb to do non trivial things. That is not yet 749 usd
Earlier quoted context omitted.
Think of it like loading a moving truck where: - The house is the disk - You are the RAM - The truck is the VRAM There won't be a single time you can observe yourself carrying the weight of everything being moved out of the house because that's not what's happening. Instead you can observe yourself taking many tiny loads until everything is finally moved, at which point you yourself should not be loaded as a result o…
My confusion was on the shuffling process happening per token. If this was happening per token, it would be effectively the same as loading the model from disk every token.
Earlier quoted context omitted.
The point is that when you run it on your own hardware you can feed the model your health data, bank statements and private journals and can be 5000% sure they’re not going anywhere
Regular people don't understand nor care about any of that. They'll happily take the Faustian bargain.
Combine it with a media player like an Apple TV or Nvidia Shield and people might buy it.
The author seems unaware of how well recent Apple laptops run LLMs. This is puzzling and puts into question the validity of anything in this article.
But economically, it is still much better to buy a lower spec't laptop and to pay a monthly subscription for AI. However, I agree with the article that people will run big LLMs on their laptop N years down the line. Especially if hardware outgrows best-in-class LLM model requirements. If a phone could run a 512GB LLM model fast, you would want it.
See: "3D TVs are driving the biggest change in TVs in decades"
Earlier quoted context omitted.
Then idk why they say that most laptops are bad at running LLMs, Apple has a huge marketshare in the laptop market and even their cheapest laptops are capable in that realm. And their PC competitors are more likely to be generously specced out in terms of included memory. > However, for the average laptop that’s over a year old, the number of useful AI models you can run locally on your PC is close to zero. This stra…
Apple has a 10-18% market share for laptops. That's significant but it certainly isn't "most". Most laptops can run at best a 7-14b model, even if you buy one with a high spec graphics chip. These are not useful models unless you're writing spam. Most desktops have a decent amount of system memory but that can't be used for running LLMs at a useful speed, especially since the stuff you could run in 32-64GB RAM would…
Earlier quoted context omitted.
Then idk why they say that most laptops are bad at running LLMs, Apple has a huge marketshare in the laptop market and even their cheapest laptops are capable in that realm. And their PC competitors are more likely to be generously specced out in terms of included memory. > However, for the average laptop that’s over a year old, the number of useful AI models you can run locally on your PC is close to zero. This stra…
So I'm hearing a lot of people running LLMs on Apple hardware. But is there actually anything useful you can run? Does it run at a usable speed? And is it worth the cost? Because the last time I checked the answer to all three questions appeared to be no. Though maybe it depends on what you're doing? (Although if you're doing something simple like embeddings, then you don't need the Apple hardware in the first place.…
This mostly just shows you how far behind the M1 (which came out 5 years ago) all the non Apple laptops are.
You can still get an M1 Macbook Air at retail for $599 ($300 for refurbs), which is a Chromebook price for a laptop that is better in pretty much every respect than any Chromebook.
MicroCenter has(had? OOS near me) M4 Minis for $400!
A remarkable bargain, even more so considering the recent hardware price hikes.
The author seems unaware of how well recent Apple laptops run LLMs. This is puzzling and puts into question the validity of anything in this article.
But economically, it is still much better to buy a lower spec't laptop and to pay a monthly subscription for AI. However, I agree with the article that people will run big LLMs on their laptop N years down the line. Especially if hardware outgrows best-in-class LLM model requirements. If a phone could run a 512GB LLM model fast, you would want it.
I don't see why consumer hardware won't evolve to run more LLMs locally. It is a nice goal to strive for, which consumer hardware makers have been missing for a decade now. It is definitely achievable, especially if you just care about inference.