31–40 of 52 posts
> *256 A100 throughput was extrapolated using the other throughput measurements. It seems worth noting that the $160k scenario wasn't actually measured.
There are some large AWS customers that probably burn that in idle time on a bunch of unused machines per week (probably day).
Can the training be parallelized in a manner similar to SETI-at-home?
General model training https://github.com/learning-at-home/hivemind
Stable diffusion specific https://github.com/chavinlo/distributed-diffusion
Inference only stable diffusion https://stablehorde.net/
I wonder if AMD is as over-the-top brutal with legal control over where their GPUs can be used as Nvidia is. Maybe with energy cost you might possibly still want to stick with the A100's anyways, but you can afford quite a lot of RX 7900's with $100k (if you can find em).
Note that this doesn't take into account the numerous iterations required to dial in the correct hyperparameters and model architecture, which could easily increase cost 5-10x. > 256 A100 throughput was extrapolated using the other throughput measurements Is it an indictment of their service that they couldn't afford 256 GPUs on their own cloud?
It turns out that, in these kinds of large-scale experiments, hardware failures are a constant fact of life, and we have tools to manage these hardware failures and allow runs to continue anyway.
Unfortunately, it would mess up our throughput calculations for getting clean baselines here, so we're waiting for our cloud provider to kindly replace the bad A100. Expect those numbers in the next day or so.
Note that this doesn't take into account the numerous iterations required to dial in the correct hyperparameters and model architecture, which could easily increase cost 5-10x. > 256 A100 throughput was extrapolated using the other throughput measurements Is it an indictment of their service that they couldn't afford 256 GPUs on their own cloud?
It's an indictment of the A100 node that died on us yesterday, leaving us with 248 GPUs in the particular cluster where we were running the experiments :( It turns out that, in these kinds of large-scale experiments, hardware failures are a constant fact of life, and we have tools to manage these hardware failures and allow runs to continue anyway. Unfortunately, it would mess up our throughput calculations for getti…
> *256 A100 throughput was extrapolated using the other throughput measurements. It seems worth noting that the $160k scenario wasn't actually measured.
Hi, one of the authors of the post, we will update the post with numbers from 256 GPU run within the next few days. We estimated the 256 run to be the fastest (13 days), but also the most expensive at $160k. The measured 128 GPU run would take 21 days but for $125k if you are interested in lower costs.
Earlier quoted context omitted.
It's an indictment of the A100 node that died on us yesterday, leaving us with 248 GPUs in the particular cluster where we were running the experiments :( It turns out that, in these kinds of large-scale experiments, hardware failures are a constant fact of life, and we have tools to manage these hardware failures and allow runs to continue anyway. Unfortunately, it would mess up our throughput calculations for getti…
Getting reliable GPUs is a difficult problem, I empathize. I've spent a decent amount of time and money because there was one failing GPU on an AWS cluster.
We can do it for way less using spot instances on AWS, though it takes longer.
AWS don't have any GPU spot instances right? I highly doubt you could get anywhere near the same price.
So you would need at least 2,716,796 hours to train on CPU.
A m6a.12xlarge is roughly equivalent to a EPYC 7352 24-core[1], it currently costs $0.5028 an hour on spot.
So that works out to a cost of $1,366,005.
[0]: https://lambdalabs.com/blog/inference-benchmark-stable-diffu...
[1]: https://browser.geekbench.com/v5/cpu/compare/17529628?baseli...
They didn’t make anything.
This is just speculative benchmarking.
I am deeply not interested in multiplying the numbers on your pricing sheet by the estimated numbers on the stable diffusion model card.
I have zero interest in your (certainly excellent) Proprietary Special Sauce (TM) that makes spending money on your service a good idea.
This just reads as spam that got past the spam filter.
Did you actually train a diffusion model?
Are you going to release the model file?
Where is the actual code someone could use to replicate your results?
Given the lack of example outputs, I guess not.