81–90 of 149 posts
Earlier quoted context omitted.
I'm pretty sure one can rent a GPU for a few minutes with the electricity cost of leaving an M5 overnight.
Domestic electricity is free nowadays, certainly for most of the year, as solar plus battery covers your usage for a tiny percentage of the cost of your house.
> One expectation to set honestly Hello Claude!
I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
Agree with this. As soon as things get in range for motivated amateurs, progress skyrockets. Has also been the case for things like chess computing; a lot of the progress we made over the last decades there (even before involving neural networks!) happened thanks to software improvements because the problem got so accessible, not just faster hardware. I expect similar trends with AI; I'd expect to get decent, human c…
That's how the AI boom came around to start with, before CUDA it was an academic topic.
I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
Earlier quoted context omitted.
Haha 1T on $50k might be a bit hopeful, mate, even at FP8. But I too am hopeful.
8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS. 1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS. 5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today. Theoretically around 104.8 TFLOPS. In the same timeframe mobile processor CPU's went from 0.001 TFLOPS,…
Earlier quoted context omitted.
There's read disturb on SSDs, enough reads will eventually force controller to rewrite the cell and it's neighbours. Practically if you're not streaming weights 24/7 from a full SSD, then it shouldn't be a problem.
Read disturb ought to be quite rare, especially on a fresh drive that was written only once or a handful of times (WORM-like usage). Practically, it's not likely to be an issue even with very heavy read workloads.
It claims that each individual page read induces read disturb across whole block. And references https://arxiv.org/pdf/2501.02517 that tested Samsung 3D TLC and found ~518K sequential page reads in a block to be ECC threshold (although it's unclear how they got 518K number -- e.g. is it single worst chip they've tried? authors brings up 160 chip sample size later on).
With 7704 pages in a block that's only ~70 sequential block reads till data is lost and to retain data controller would have to refresh block fair bit earlier.. basically it gives modern 3D TLC SSD lifespan measured in months (1TB drive 24/7 sequential reads at 5GB/s).
I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
Do reads wear out nvme? Iirc only writes do
I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
> but this is how progress happens. This is progress in the same way that a man climbing a tree is making progress toward reaching the moon. This project is essentially the MoE-of-the day, with some platform-related optimizations. > Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips. That won't happen. Projects like this just give the illus…
That is, an essential step?
I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
> but this is how progress happens. This is progress in the same way that a man climbing a tree is making progress toward reaching the moon. This project is essentially the MoE-of-the day, with some platform-related optimizations. > Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips. That won't happen. Projects like this just give the illus…
1. Better hardware optimized for matrix operations and other ML/NN specific workflows (NVIDIA hardware, Apple M4/M5 chips, Intel AI Boost, other NPU/TPU chips, etc.).
2. Bigger RAM/VRAM/unified for loading models in memory -- with 3090/4090 you can run 22-24B models in VRAM; with 5090 you can run 30-32B models; with Apple hardware with 128GB unified RAM you can run 70-128B models.
3. Better quantization (smaller, better value representations/lower errors/loss, optimized CUDA/etc. kernels) will lead to larger models being able to be run on the lower VRAM hardware at (with optimized kernels) decent tokens/s.
4. Better optimizations in general taking advantage of the hardware (coupled with the hardware and driver advances) and the NN architectures (NOTE: I saw a huge performance boost a while back after upgrading my llama.cpp version).
5. Better model architectures optimized/designed for running on lower-spec hardware, like Mixture-of-Experts which make it easier to run on a single GPU or distribute the weights more efficiently across multiple GPUs (i.e. you know that specific weights will be used together so you can keep them resident on a specific GPU and not have to swap the weights in/out of the various GPUs or transfer intermediate results of a single layer, only the result from that block/expert).
5. Improved NVME or similar SSD technology which a) has a large amount of storage, and b) has access efficiency on par with RAM/VRAM, especially if/when coupled with efficient DMA access to the GPU/NPU.
I know everyone wants to crap all over these setups that are impractical, but this is how progress happens. People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc. Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
> but this is how progress happens. This is progress in the same way that a man climbing a tree is making progress toward reaching the moon. This project is essentially the MoE-of-the day, with some platform-related optimizations. > Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips. That won't happen. Projects like this just give the illus…
While not exactly an “SSD”, High Bandwidth Flash will make it to market and make big sparse MOE models even more accessible. Everyone knows bandwidth is the game right now and you think hardware vendors are going to sit on their heels and not take part of the market? Absolutely silly to discount progress both on the software and hardware sides.