It was probably because very inefficient to begin with.
Indeed nonexistent
The cost to train an AI system is improving at 50x the pace of Moore’s Law
11–20 of 60 posts
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#12Resnet-50 with DawnBench settings is a very poor choice for illustrating this trend. The main technique driving this reduction in cost-to-train has been finding arcane, fast training schedules. This sounds good until you realize its a type of sleight of hand where finding that schedule takes tens of thousands of dollars (usually more) that isn't counted in cost-to-train, but is a real-world cost you would experience if you want to train models.
However, I think the overall trend this article talks about is accurate. There has been an increased focus on cost-to-train and you can see that with models like EfficientNet where NAS is used to optimize both accuracy and model size jointly.
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#13It is regrettable if an equivalent to the self-fulfilling prophecy of Moore's "Law" (originally an astute observation and forecast, but not remotely a law) became a driver/limiter in this field as well, even more so if it's a straight transplant for soundbite reasons rather than through any impartial and thoughtful analysis.
One thing I've wondered is if Moore's Law is good or bad, in the sense of how fast should we have been able to improve IC technology. Was progress limited by business decisions or is this as fast as improvements could take place?
A thought experiment: suppose we meet aliens who are remarkably similar to ourselves and have an IC industry. Would they be impressed by our Moore's law progress, or wonder why we took so long?
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#14[deleted]
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#15This is an odd framing. Training has become much more accessible, due to a variety of things (ASICs, offerings from public clouds, innovations on the data science side). Comparing it to Moore's Law doesn't make any sense to me, though. Moore's Law is an observation on the pace of increase of a tightly scoped thing, the number of transistors. The cost of training a model is not a single "thing," it's a cumulative effe…
Like many things, Moore’s law is garbled when adopted by analogy outside its domain.
What does “more transistors” mean? To you, it means just what Gordon Moore means when he said it: opportunity for more function in same space/cost.
The laypersons, marketing grabbed the term and said it would imply “faster”. Which then was absurdly conflated with CPU clock speed (itself an important input, though hardly the only one, determining the actual speed of A system).
The use here is of the “garbled analogy” sort which surely is the dominant use today.
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#16I remember this article from 2018: https://medium.com/the-mission/why-building-your-own-deep-le... Hackernews discussion for the article: https://news.ycombinator.com/item?id=18063893 It really is interesting how this is changing the dynamics of neural network training. Now it is affordable to train a useful network on the cloud, whereas 2 years ago that would be reserved to companies with either bigger investments o…
> Now it is affordable to train a useful network on the cloud I honestly don't see how anything changed significantly in past 2 years. Benchmarks indicate that a V100 is barely 2x the performance of an RTX 2080 Ti [1] and a V100 is • $2.50/h at Google [2] • $13.46/h (4xV100) at Microsoft Azure [3] • $12.24/h (4xV100) at AWS [4] • ~$2.80/h (2xV100, 1 month) at LeaderGPU [5] • ~$3.38/h (4xV100, 1 month) at Exoscale [6]…
You are missing TPU and spot/preemptible pricing, which need to be considered when we are talking about training cost. The big one to me is the ability to consistently train on V100s with spot pricing, which was not possible a couple of years ago (there wasn't enough spare capacity). Also, the improvement in cloud bandwidth for DL-type instances has helped distributed training a lot.
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#17I remember this article from 2018: https://medium.com/the-mission/why-building-your-own-deep-le... Hackernews discussion for the article: https://news.ycombinator.com/item?id=18063893 It really is interesting how this is changing the dynamics of neural network training. Now it is affordable to train a useful network on the cloud, whereas 2 years ago that would be reserved to companies with either bigger investments o…
> Now it is affordable to train a useful network on the cloud I honestly don't see how anything changed significantly in past 2 years. Benchmarks indicate that a V100 is barely 2x the performance of an RTX 2080 Ti [1] and a V100 is • $2.50/h at Google [2] • $13.46/h (4xV100) at Microsoft Azure [3] • $12.24/h (4xV100) at AWS [4] • ~$2.80/h (2xV100, 1 month) at LeaderGPU [5] • ~$3.38/h (4xV100, 1 month) at Exoscale [6]…
I don't really know if those hardware breakthroughs that the article refers to already reflects in Cloud GPU performance, but software reflects nonetheless. So even though pricing has fluctuated marginally since 2018, it is just plain faster to train a neural network today because of software advances, from what I understood.
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#18Ark Invest are the creators of the ARKK [1] and ARKW ETFs that have become retail darlings, mainly because they're heavily invested in TSLA. They pride themselves on this type of fundamental, bottom up analysis on the market. It's fine.. I don't know if I agree with using Moore's law which is fundamentally about hardware, with the cost to run a "system" which is a combination of customized hardware and new software t…
[deleted]
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#19This is an odd framing. Training has become much more accessible, due to a variety of things (ASICs, offerings from public clouds, innovations on the data science side). Comparing it to Moore's Law doesn't make any sense to me, though. Moore's Law is an observation on the pace of increase of a tightly scoped thing, the number of transistors. The cost of training a model is not a single "thing," it's a cumulative effe…
Ok, but achieving Moore's law has required combining an enormous number of conceptually distinct technical insights. Both training costs and transistor density seem like well-defined single parameters that incorporate many small complicated effects.
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#20I would really like a thorough analysis on how expensive it is to multiply large matrices, which is the most expensive part of a transformer training for example according to the profiler. Is there some Moore’s law or similar trend?