800GB/s and 512 unified ram is going to go stupid for llms
Apple M3 Ultra
671–680 of 1001 posts
Re: Apple M3 Ultra
#672+ $4,000 to bump to 512GB from 96GB.
Re: Apple M3 Ultra
#673Re: Apple M3 Ultra
#674Earlier quoted context omitted.
> that 10k is a absolute bargain The higher end NVidia workstation boxes won’t run well on normal 20amp plugs. So you need to move them to a computer room (whoops, ripped those out already) or spend months getting dedicated circuits run to office spaces.
In the US, normal circuits aren't always 20A, especially in residential buildings, where they are more commonly 15A in bedrooms and offices. https://en.wikipedia.org/wiki/NEMA_connector
While technically true, the NEMA 5-15R receptacles are rated for use on 20A circuits, and circuits for receptacles are almost always 20A circuits, in modern construction at least. Older builds may not be, of course.
That said, if your load is going to be a continuous load drawing 80% of the rated amperage, it really should be a NEMA 5-20 plug and receptacle, the one where one of the prongs is horizontal instead of vertical. Swapping out the receptacle for one that accepts a NEMA 5-20P plug is like $5.
If you are going to actually run such a load on a 20A circuit with multiple receptacles, you will want to make sure you're not plugging anything substantial into any of the other receptacles on that circuit. A couple LED lights are fine. A microwave or kettle, not so much.
Re: Apple M3 Ultra
#675Earlier quoted context omitted.
Thanks. >> Apple seems to be using LPDDR, but HBM will also likely be a key tech. SK Hynix and Samsung are the most reputable for both. So not much Micron? Any US based stocks to invest in? :-)
I forgot about Micron, absolutely. TSMC is the supplier for all of these, so you're covering both memory and compute if that's your strategy (the risk is that US TSMC is over provisioning manufacturing based on the pandemic hardware boom).
Thanks!
Re: Apple M3 Ultra
#676Earlier quoted context omitted.
What kind of Nvidia-based rig would one need to achieve 40 tokens/sec on Deepseek 671b? And how much would it cost?
Around 5x Nvidia A100 80GB can fit 671b Q4. $50k just for the GPUs and likely much more when including cooling, power, motherboard, CPU, system RAM, etc.
How many tokens/s would that be though?
Re: Apple M3 Ultra
#677Earlier quoted context omitted.
Keep in mind the minimum configuration that has 512GB of unified RAM is $9,499.
And how is it only £9,699.00!! Does that dollar price include sales tax or are Brits finally getting a bargain?
What's the bargain?
There is also "parity" in other products like a MacBook Pro from £1,599 / $1,599 or an iPhone 16 from £799 / $799. £9,699 / $9,499 is worse than that!
Re: Apple M3 Ultra
#678Two questions for the fellow HNers: 1. What are various average joe (as opposed to researchers, etc.) use cases for running powerful AI models locally vs. just using cloud AI. Privacy of course is a benefit, but it by itself may not justify upgrades for an average user. Or are we expecting that new innovation will lead to much more proliferation of AI and use cases that will make running locally more feasible? 2. Wit…
One important one that I haven't seen mentioned is simply working without an internet connection. It was quite important for me when I was using AI whilst travelling through the countryside, where there is very limited network access.
Re: Apple M3 Ultra
#679Computers these days - the more appealing, exciting, cooler desirable, the higher the price, into the stratosphere. $9499 What ever happening to competition in computing? Computing hardware competition used to be cut throat, drop dead, knife fight, last man standing brutally competitive. Now it's just a massive gold rush cash grab.
The Macintosh plus, released in 1986, cost $2600 at the time, or $7460 adjusted for inflation.
It even came with an official display! Nowadays that's a $1,600-$6,000 accessory, depending on whether you own a VESA mount.
Re: Apple M3 Ultra
#680Earlier quoted context omitted.
Not that dramatic of an increase actually - the M2 Max already had 400GB/s and M2 Ultra 800GB/s memory bandwidth, so the M3 Ultra's 819GB/s is just a modest bump. Though the M4's additional 146GB/s is indeed a more noticeable improvement.
Also should note that 800/819GB/s of memory bandwidth is actually VERY usable for LLMs. Consider that a 4090 is just a hair above 1000GB/s
[deleted]