I would really like a thorough analysis on how expensive it is to multiply large matrices, which is the most expensive part of a transformer training for example according to the profiler. Is there some Moore’s law or similar trend?
[deleted]
The cost to train an AI system is improving at 50x the pace of Moore’s Law
21–30 of 60 posts
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#22I remember this article from 2018: https://medium.com/the-mission/why-building-your-own-deep-le... Hackernews discussion for the article: https://news.ycombinator.com/item?id=18063893 It really is interesting how this is changing the dynamics of neural network training. Now it is affordable to train a useful network on the cloud, whereas 2 years ago that would be reserved to companies with either bigger investments o…
> Now it is affordable to train a useful network on the cloud I honestly don't see how anything changed significantly in past 2 years. Benchmarks indicate that a V100 is barely 2x the performance of an RTX 2080 Ti [1] and a V100 is • $2.50/h at Google [2] • $13.46/h (4xV100) at Microsoft Azure [3] • $12.24/h (4xV100) at AWS [4] • ~$2.80/h (2xV100, 1 month) at LeaderGPU [5] • ~$3.38/h (4xV100, 1 month) at Exoscale [6]…
Another data point:
"For example, we recently internally benchmarked an Inferentia instance (inf1.2xlarge) against a GPU instance with an almost identical spot price (g4dn.xlarge) and found that, when serving the same ResNet50 model on Cortex, the Inferentia instance offered a more than 4x speedup."
https://towardsdatascience.com/why-every-company-will-have-m...
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#23Earlier quoted context omitted.
> Now it is affordable to train a useful network on the cloud I honestly don't see how anything changed significantly in past 2 years. Benchmarks indicate that a V100 is barely 2x the performance of an RTX 2080 Ti [1] and a V100 is • $2.50/h at Google [2] • $13.46/h (4xV100) at Microsoft Azure [3] • $12.24/h (4xV100) at AWS [4] • ~$2.80/h (2xV100, 1 month) at LeaderGPU [5] • ~$3.38/h (4xV100, 1 month) at Exoscale [6]…
Another data point: "For example, we recently internally benchmarked an Inferentia instance (inf1.2xlarge) against a GPU instance with an almost identical spot price (g4dn.xlarge) and found that, when serving the same ResNet50 model on Cortex, the Inferentia instance offered a more than 4x speedup." https://towardsdatascience.com/why-every-company-will-have-m...
That data point talks about inference though, and nobody's arguing that deployment and inference have improved significantly over the past years.
I'm referring to training and fine-tuning, not inference, which - let's be honest - can be done on a phone these days.
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#24This is an odd framing. Training has become much more accessible, due to a variety of things (ASICs, offerings from public clouds, innovations on the data science side). Comparing it to Moore's Law doesn't make any sense to me, though. Moore's Law is an observation on the pace of increase of a tightly scoped thing, the number of transistors. The cost of training a model is not a single "thing," it's a cumulative effe…
> Comparing it to Moore's Law doesn't make any sense to me, though.
I assume it's meant as a qualitative comparison rather than a meaningful quantitative one. Sort of a (sub-)cultural touchstone to illustrate a point about which phase of development we're in.
With CPUs, during the phase of consistent year after year exponential growth, there were ripple effects on software. For example, for a while it was cost-prohibitive to run HTTPS for everything, then CPUs got faster and it wasn't anymore. So during that phase, you expected all kinds of things to keep changing.
If deep learning is in a similar phase, then whatever the numbers are, we can expect other things to keep changing as a result.
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#25Does it mean that the cost to train something like gpt3 by OpenAI will reduce from 12 million dollars to less next year ? If so how much will it reduce to ?
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#26It is regrettable if an equivalent to the self-fulfilling prophecy of Moore's "Law" (originally an astute observation and forecast, but not remotely a law) became a driver/limiter in this field as well, even more so if it's a straight transplant for soundbite reasons rather than through any impartial and thoughtful analysis.
One thing I've wondered is if Moore's Law is good or bad, in the sense of how fast should we have been able to improve IC technology. Was progress limited by business decisions or is this as fast as improvements could take place? A thought experiment: suppose we meet aliens who are remarkably similar to ourselves and have an IC industry. Would they be impressed by our Moore's law progress, or wonder why we took so lo…
https://en.wikipedia.org/wiki/Moore%27s_law, third paragraph of the header, claims that Moore's Law drove targets in R&D and manufacturing, but does not cite a reference for this claim.
"Moore's prediction has been used in the semiconductor industry to guide long-term planning and to set targets for research and development."
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#27Earlier quoted context omitted.
> Now it is affordable to train a useful network on the cloud I honestly don't see how anything changed significantly in past 2 years. Benchmarks indicate that a V100 is barely 2x the performance of an RTX 2080 Ti [1] and a V100 is • $2.50/h at Google [2] • $13.46/h (4xV100) at Microsoft Azure [3] • $12.24/h (4xV100) at AWS [4] • ~$2.80/h (2xV100, 1 month) at LeaderGPU [5] • ~$3.38/h (4xV100, 1 month) at Exoscale [6]…
I don't really know if those hardware breakthroughs that the article refers to already reflects in Cloud GPU performance, but software reflects nonetheless. So even though pricing has fluctuated marginally since 2018, it is just plain faster to train a neural network today because of software advances, from what I understood.
But that's not what the actual data says.
Here's some figures from an actual benchmark [1] w.r.t. training costs:
1. [Mar 2020] $7.43 (AlibabaCloud, 8xV100, TF v2.1)
2. [Sep 2018] $12.60 (Google, 8 TPU cores, TF v1.11)
3. [Mar 2020] $14.42 (AlibabaCloud, 128xV100, TF v2.1)
--
Training time didn't go down exponentially either [1]:
1. [Mar 2020] 0:02:38 (AlibabaCloud, 128 x V100, TF v2.1)
2. [May 2019] 0:02:43 (Huawei Cloud, 128 x V100, TF v1.13)
3. [Dec 2018] 0:09:22 (Huawei Cloud, 128 x V100, MXNet)
So again, I have to ask where exactly do these magical improvement occur (regarding training - inference is another matter entirely, I understand that)? I've yet to find a source that supports 4x to 10x cost reductions.
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#28I remember this article from 2018: https://medium.com/the-mission/why-building-your-own-deep-le... Hackernews discussion for the article: https://news.ycombinator.com/item?id=18063893 It really is interesting how this is changing the dynamics of neural network training. Now it is affordable to train a useful network on the cloud, whereas 2 years ago that would be reserved to companies with either bigger investments o…
I trained a useful neural network and prototyped a viable [failed] startup technology something like 4 years ago on a 1080ti with a mid range CPU. It was enough to get me meetings with a couple of the largest companies in the world.
Yeah it took 12-24 hours to do what I could login to AWS and accomplish in minutes with parallel GPUs...but practical solutions were already in reach. The primary changes now are buzz and possibly unprecedent rate of research progress.
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#29This is an odd framing. Training has become much more accessible, due to a variety of things (ASICs, offerings from public clouds, innovations on the data science side). Comparing it to Moore's Law doesn't make any sense to me, though. Moore's Law is an observation on the pace of increase of a tightly scoped thing, the number of transistors. The cost of training a model is not a single "thing," it's a cumulative effe…
Agreed, but Moore's Law has morphed to refer to both xtors and performance despite his original phrasing.
The biggest innovation I've seen is in the cloud: backplane I/O and memory is essential and up until a few years ago there weren't many cloud configurations suitable for massive amount of I/O.
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#30This is an odd framing. Training has become much more accessible, due to a variety of things (ASICs, offerings from public clouds, innovations on the data science side). Comparing it to Moore's Law doesn't make any sense to me, though. Moore's Law is an observation on the pace of increase of a tightly scoped thing, the number of transistors. The cost of training a model is not a single "thing," it's a cumulative effe…
Many people use Moore's Law to mean some kind of "law of accelerated returns" - which actually is a thing, and it kind of does work like the author implies: