241–250 of 365 posts
Earlier quoted context omitted.
He also documented his progress on the tinygrad discord
I feel like I should say something about discord not being a suitable replacement for a forum or bugtracker.
Earlier quoted context omitted.
It does seem like an oversight, but there's nothing "suboptimal non-default options" about iteven if the implementation posted here seems somewhat hastily hacked together.
> but there's nothing "suboptimal non-default options" about it If "bypassing the official driver to invoke the underlying hardware feature directly through source code modification (and incompatibilities must be carefully worked around by turning off IOMMU and large BAR, since the feature was never officially supported)" does not count as "suboptimal non-default options", then I don't know what counts as "suboptimal…
Earlier quoted context omitted.
> very few libraries are adapted to apple silicone let alone optimized This is a joke, right? Have you been anywhere in the LLM ecosystem for the past year or so? I'm constantly hearing about new ways in which ASi outperforms traditional platforms, and new projects that are optimized for ASi. Such as, for instance, llama.cpp.
Nothing compared to Nvidia though. The FLOPS and memory bandwidth is simply not there.
Even the A100 for something around $15,000 (edit: used to say $10,000) only goes up to 80 gigabytes of VRAM, but a 192GB Mac Studio goes for under $6,000.
Those figures alone proves Nvidia isn't even competing in the consumer or even the enthusiast space anymore. They know you'll buy their hardware if you really need it, so they aggressively segment the market with VRAM restrictions.
Earlier quoted context omitted.
crypto mining only needs 1 PCIe lane per GPU, so you can fit 24+ GPUs on a standard consumer CPU motherboard (24-32 lanes depending on the CPU). Apparently ML workloads require more interconnect bandwidth when doing parallel compute, so each card in this demo system uses 16 lanes, and therefore requires 1.) full size slots, and 2.) epyc[0] or xeon based systems with 128 lanes (or at least greater than 32 lanes). per…
Oooo, got a link to one of these fabric boards? I've been playing with stupid PCIe tricks but that's a new one on me.
i used to use this one when i had all (three of my) nvme -> 4x sata boardlets and therefore could not fit a GPU in a PCIe slot due to the cabling mess.
Earlier quoted context omitted.
Nothing compared to Nvidia though. The FLOPS and memory bandwidth is simply not there.
Yeah. Let me just walk down to Best Buy and get myself a GPU with over 24 gigabytes of VRAM (impossible) for less than $3,000 (even more impossible). Then tell me ASi is nothing compared to Nvidia. Even the A100 for something around $15,000 (edit: used to say $10,000) only goes up to 80 gigabytes of VRAM, but a 192GB Mac Studio goes for under $6,000. Those figures alone proves Nvidia isn't even competing in the consu…
Earlier quoted context omitted.
Yeah. Let me just walk down to Best Buy and get myself a GPU with over 24 gigabytes of VRAM (impossible) for less than $3,000 (even more impossible). Then tell me ASi is nothing compared to Nvidia. Even the A100 for something around $15,000 (edit: used to say $10,000) only goes up to 80 gigabytes of VRAM, but a 192GB Mac Studio goes for under $6,000. Those figures alone proves Nvidia isn't even competing in the consu…
Where are you getting an A100 80GB for $10k?
A100 80GB goes for around $14,000 - $20,000 on eBay and A100 40GB goes for around $4,000 - $6,000. New (not from eBay - from PNY and such), it looks like an 80GB would set you back $18,000 to $26,000 depending on whether you want HBM2 or HBM2e.
Meanwhile you can buy a Mac Studio today without going through a distributor and they're under $6,000 if the only thing you care about is having 192GB of Unified Memory.
And while the memory bandwidth isn't quite as high as the 4090, the M-series chips can run certain models faster anyway, if Apple is to be believed
Earlier quoted context omitted.
Nothing compared to Nvidia though. The FLOPS and memory bandwidth is simply not there.
The memory bandwidth of the M2 Ultra is around 800GB/s verses 1008 GB/s for the 4090. While it’s true the M2 has neither the bandwidth or the GPU power, it is not limited to 24G of VRAM per card. The 192G upper limit on the M2 Ultra will have a much easier time running inference on a 70+ billion parameter model, if that is your aim. Besides size, heat, fan noise, and not having to build it yourself, this is the only…
Earlier quoted context omitted.
The memory bandwidth of the M2 Ultra is around 800GB/s verses 1008 GB/s for the 4090. While it’s true the M2 has neither the bandwidth or the GPU power, it is not limited to 24G of VRAM per card. The 192G upper limit on the M2 Ultra will have a much easier time running inference on a 70+ billion parameter model, if that is your aim. Besides size, heat, fan noise, and not having to build it yourself, this is the only…
It doesn't need GPU power to beat the 4090 in benchmarks: https://appleinsider.com/articles/23/12/13/apple-silicon-m3-...
> You may need to uninstall the driver from DKMS. Your system needs large BAR support and IOMMU off. Can someone point me to the correct tutorial on how to do these things?
BAR: enable resizable BAR in motherboard CMOS setup
IOMMU: Add "amd_iommu=off" or "intel_iommu=off" to kernel command line for AMD or Intel CPU, respectively (or just add both). You may or may not need to disable the IOMMU in CMOS setup (Intel calls its IOMMU VT-d).
See motherboard docs for specific option names. See distro docs for procedures to list/uninstall packages and to add kernel command line options.