What are some domains that a solo developer could build something commercially compelling to capture some of this $37 trillion? Are there any workflows or tools or efficiencies that could be easily realized as a commercial offering that would not require massive man hours to implement?
The cost to train an AI system is improving at 50x the pace of Moore’s Law
31–40 of 60 posts
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#32What are some domains that a solo developer could build something commercially compelling to capture some of this $37 trillion? Are there any workflows or tools or efficiencies that could be easily realized as a commercial offering that would not require massive man hours to implement?
Take any domain that requires classification work that has not yet been targeted and make a run for it. You likely will be able to adapt one of the existing nets or even use transfer learning to outperform a human. That's the low hanging fruit.
For instance: quality control: abnormality detection (for instance: in medicine), agriculture (lots of movement there right now), parts inspection, assembly inspection, sorting and so on. There are more applications for this stuff than you might think at first glance, essentially if a toddler can do it and it is a job right now that's a good target.
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#33Despite nvidia vaguely prohibiting users from using their desktop cards for machine learning in any sort of data center-like or server-like capacity. Hopefully AMDs ml support / OpenCl will continue improving
Last I saw, they don’t even support ROCm on their recent Navi cards, so I’d be hesitant.
Wow. This is really disappointing to see. (https://github.com/RadeonOpenCompute/ROCm/issues/887)
I guess PlaidML might be a viable option?
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#34"AI" is not really appropriate name for what it is
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#35This is an odd framing. Training has become much more accessible, due to a variety of things (ASICs, offerings from public clouds, innovations on the data science side). Comparing it to Moore's Law doesn't make any sense to me, though. Moore's Law is an observation on the pace of increase of a tightly scoped thing, the number of transistors. The cost of training a model is not a single "thing," it's a cumulative effe…
> Comparing it to Moore's Law doesn't make any sense to me, though. I assume it's meant as a qualitative comparison rather than a meaningful quantitative one. Sort of a (sub-)cultural touchstone to illustrate a point about which phase of development we're in. With CPUs, during the phase of consistent year after year exponential growth, there were ripple effects on software. For example, for a while it was cost-prohib…
> then CPUs got faster and it wasn't anymore
The enabling tech was AES-NI instruction set, not the speed.
Agree on the rest. The main reason why modern CPUs and GPUs all have 16-bit floats is probably the deep learning trend.
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#36Earlier quoted context omitted.
I don't really know if those hardware breakthroughs that the article refers to already reflects in Cloud GPU performance, but software reflects nonetheless. So even though pricing has fluctuated marginally since 2018, it is just plain faster to train a neural network today because of software advances, from what I understood.
But that's not what the actual data says. Here's some figures from an actual benchmark [1] w.r.t. training costs: 1. [Mar 2020] $7.43 (AlibabaCloud, 8xV100, TF v2.1) 2. [Sep 2018] $12.60 (Google, 8 TPU cores, TF v1.11) 3. [Mar 2020] $14.42 (AlibabaCloud, 128xV100, TF v2.1) -- Training time didn't go down exponentially either [1]: 1. [Mar 2020] 0:02:38 (AlibabaCloud, 128 x V100, TF v2.1) 2. [May 2019] 0:02:43 (Huawei…
I guess I should have been more skeptical of the articles figures. But still, if we give the benefit of the doubt, is there any scenario we might see the reduction mentioned? 1000 to 10 USD?
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#37What are some domains that a solo developer could build something commercially compelling to capture some of this $37 trillion? Are there any workflows or tools or efficiencies that could be easily realized as a commercial offering that would not require massive man hours to implement?
Take any domain that requires classification work that has not yet been targeted and make a run for it. You likely will be able to adapt one of the existing nets or even use transfer learning to outperform a human. That's the low hanging fruit. For instance: quality control: abnormality detection (for instance: in medicine), agriculture (lots of movement there right now), parts inspection, assembly inspection, sortin…
anything that's even remotely profitable is already taken
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#38Earlier quoted context omitted.
Take any domain that requires classification work that has not yet been targeted and make a run for it. You likely will be able to adapt one of the existing nets or even use transfer learning to outperform a human. That's the low hanging fruit. For instance: quality control: abnormality detection (for instance: in medicine), agriculture (lots of movement there right now), parts inspection, assembly inspection, sortin…
anything that's even remotely profitable is already taken
This simply isn't true. Every year since the present day ML wave started has seen more and more domains tackled. Even something like that silly lego sorting machine I built could be the basis of a whole company pursuing sorting technology if you set your mind to it. And that's just resnet50 in disguise, likely you could do better today without any effort.
Your statement reminds of 'all the good domains are taken', which I've been hearing since 1996 or so. Of course you'll need to do some work to identify a niche that doesn't have a major player in it yet. But the 'boring' niches are where a lot of money is to be made, the sexy stuff (cancer, fruit sorting) is well covered. But more obscure things are still wide open, I get decks with some regularity about new players in very interesting spaces using thinly wrapped ML to do very profitable things.
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#39I remember this article from 2018: https://medium.com/the-mission/why-building-your-own-deep-le... Hackernews discussion for the article: https://news.ycombinator.com/item?id=18063893 It really is interesting how this is changing the dynamics of neural network training. Now it is affordable to train a useful network on the cloud, whereas 2 years ago that would be reserved to companies with either bigger investments o…
> Now it is affordable to train a useful network on the cloud I honestly don't see how anything changed significantly in past 2 years. Benchmarks indicate that a V100 is barely 2x the performance of an RTX 2080 Ti [1] and a V100 is • $2.50/h at Google [2] • $13.46/h (4xV100) at Microsoft Azure [3] • $12.24/h (4xV100) at AWS [4] • ~$2.80/h (2xV100, 1 month) at LeaderGPU [5] • ~$3.38/h (4xV100, 1 month) at Exoscale [6]…
Nothing really has changed in the last two years in terms of training cost. I think the author is making unreasonable extrapolations based on changes in performance on the Dawn benchmarks. A lot of the results are fast but require a lot more compute / search time to find the best parameters and training regimen that lead to those fast convergence times. (Learning rate schedule, batch size, image size schedules, etc.) The point being that once the juice is squeezed out you aren’t going to continue to see training convergence time improvements on the same hardware.
Also, because you cited our GPU benchmarks, I also wanted to throw in a mention our GPU instances which have some of the lowest training costs on the Stanford Dawn Benchmarks discussed in the article.
Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law
#40This is an odd framing. Training has become much more accessible, due to a variety of things (ASICs, offerings from public clouds, innovations on the data science side). Comparing it to Moore's Law doesn't make any sense to me, though. Moore's Law is an observation on the pace of increase of a tightly scoped thing, the number of transistors. The cost of training a model is not a single "thing," it's a cumulative effe…
Like many things, Moore’s law is garbled when adopted by analogy outside its domain. What does “more transistors” mean? To you, it means just what Gordon Moore means when he said it: opportunity for more function in same space/cost. The laypersons, marketing grabbed the term and said it would imply “faster”. Which then was absurdly conflated with CPU clock speed (itself an important input, though hardly the only one,…
Yes but that aspect of Moore's law for CPUs expired over a decade ago. It's the whole reason we got multicore in the first place.