301–310 of 562 posts
Earlier quoted context omitted.
The M2 is a chip designed to be in a laptop (and it is quite powerful given its low power consumption). Presumedly they have a different chip or at least completely different configuration (RAM, network, etc.) in their data centers.
The interesting point here is that developers targeting the Mac can safely assume that the users will have a processor capable of significant AI/ML workloads. On the Windows (and Linux) side of things, there's no common platform, no assumption that the users will have an NPU or GPU capable of doing what you want. I think that's also why Microsoft was initially going for the ARM laptops, where they'd be sure that the…
Also that a significant proportion (majority?) of them will have just 8 GB of memory which is not exactly sufficient to run any complex AI/ML workloads.
Earlier quoted context omitted.
Not at Apple's price points.
I mean, even Apple can't match the markups nVidia has right now. If you break a GPU in your compute server, you wait months for a replacement, and the part is sent back if you can't replace it in five days. Crazy times.
3.5B per weight with no quality loss is state of the art - that's an awesome optimization result (a mix of 2b and 4b weights).
I wonder if they didn't stretch the truth using the phrase "without loss in accuracy".
Earlier quoted context omitted.
I mean, even Apple can't match the markups nVidia has right now. If you break a GPU in your compute server, you wait months for a replacement, and the part is sent back if you can't replace it in five days. Crazy times.
Enterprise offerings tend to differ. You can get a replacement NVIDIA GPU via a partner, like Lenovo, in 2-3 weeks. And that's on the high side for some support contracts.
I hope, this could mean Apple will push the baseline of ALL Macs to have higher than 8GB of Memory. While I wish we all get 16GB M4 as baseline. Apple being Apple may only give us 12GB, and charges extra $100 for the 16GB option. It will still be a lot better than 8GB though.
It probably will change. Note that, so far, a 16GB apple device has much better usability than the equivalent on windows. This may sound biased, but the memory compression and foreground/background actions by macOS tight integration with the hardware is really good. I've never felt like I couldn't do things on smaller hardware, except (larges) LLMs. Also when I compare with my co-workers the memory pressure is a lot…
It certainly does, close to irrational even. IIRC memory compression is enabled by default on Windows as well.
I hope, this could mean Apple will push the baseline of ALL Macs to have higher than 8GB of Memory. While I wish we all get 16GB M4 as baseline. Apple being Apple may only give us 12GB, and charges extra $100 for the 16GB option. It will still be a lot better than 8GB though.
If you could do that, you could easily get hundreds of GB/s read speed out of simple TLC flash.
Obviously this is the future, but I think it's a promising one.
For people interested in AI research, there's nothing new here. IMO they should do a better job of referencing existing papers and techniques. The way they wrote about "adaptors" can make it seem like it's something novel, but it's actually just re-iterating vanilla LoRA. It was enough to convince one of the top-voted HackerNews comments that this was a "huge development". Benchmarks are nice though.