Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The cost to train an AI system is improving at 50x the pace of Moore’s Law

ark-invest.com

Post reply on HN
Print view View original

51–60 of 60 posts

Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law

#51

Post by imtringued »

kens wrote: post #13
mellosouls wrote: post #9

It is regrettable if an equivalent to the self-fulfilling prophecy of Moore's "Law" (originally an astute observation and forecast, but not remotely a law) became a driver/limiter in this field as well, even more so if it's a straight transplant for soundbite reasons rather than through any impartial and thoughtful analysis.

One thing I've wondered is if Moore's Law is good or bad, in the sense of how fast should we have been able to improve IC technology. Was progress limited by business decisions or is this as fast as improvements could take place? A thought experiment: suppose we meet aliens who are remarkably similar to ourselves and have an IC industry. Would they be impressed by our Moore's law progress, or wonder why we took so lo…

I'm not sure what the point of that question is. In theory you could have a government subsidize construction of fabs so that skipping nodes is feasible but why on earth would you do that when the industry is fully self sufficient and wildly profitable?

Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law

#52

Post by [deleted] »

jacquesm wrote: post #43
Barrin92 wrote: post #41

Earlier quoted context omitted.

> abnormality detection (for instance: in medicine), agriculture (lots of movement there right now), parts inspection, assembly inspection, sorting and so on none of these is anything someone can run from their bedroom because they have very high quality and regulatory requirements and require constant work outside of the actual AI training. This is actually reflected in the margins of "AI" companies, which are signi…

Well, depending on the size of your bedroom ;) I've seen teams of two people running fairly impressive ML based stuff. They were good enough at it that they didn't remain at two people for very long but that was more than enough to be useful to others. One interesting company - that I'm free to talk about - did a nice one on e-commerce sites to help with risk management: spot fraudulent orders before they ship. In th…

[deleted]

Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law

#53

Post by andrewprock »

bcrosby95 wrote: post #40
gumby wrote: post #15

Earlier quoted context omitted.

Like many things, Moore’s law is garbled when adopted by analogy outside its domain. What does “more transistors” mean? To you, it means just what Gordon Moore means when he said it: opportunity for more function in same space/cost. The laypersons, marketing grabbed the term and said it would imply “faster”. Which then was absurdly conflated with CPU clock speed (itself an important input, though hardly the only one,…

Yes but that aspect of Moore's law for CPUs expired over a decade ago. It's the whole reason we got multicore in the first place.

Even with multi-core, a CPU today is only 6x faster than a 10-year old CPU.

Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law

#54

Post by moonchild »

Const-me wrote: post #35
adrianmonk wrote: post #24

Earlier quoted context omitted.

> Comparing it to Moore's Law doesn't make any sense to me, though. I assume it's meant as a qualitative comparison rather than a meaningful quantitative one. Sort of a (sub-)cultural touchstone to illustrate a point about which phase of development we're in. With CPUs, during the phase of consistent year after year exponential growth, there were ripple effects on software. For example, for a while it was cost-prohib…

> then CPUs got faster and it wasn't anymore The enabling tech was AES-NI instruction set, not the speed. Agree on the rest. The main reason why modern CPUs and GPUs all have 16-bit floats is probably the deep learning trend.

If it hadn't been aes-ni, it would have been chacha, which is much faster than unaccelerated aes and close to the speed of accelerated aes.

Phones use https without a problem, and those haven't had hw-accelerated aes until recently.

Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law

#55

Post by cft »

lukevp wrote: post #31

What are some domains that a solo developer could build something commercially compelling to capture some of this $37 trillion? Are there any workflows or tools or efficiencies that could be easily realized as a commercial offering that would not require massive man hours to implement?

The cost of training is decreasing, but the meaningfully large and non-trivial training sets are almost exclusively in the domain of large companies, economically inaccessible to the individual developers/startups.

Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law

#56

Post by godzillabrennus »

cft wrote: post #55
lukevp wrote: post #31

What are some domains that a solo developer could build something commercially compelling to capture some of this $37 trillion? Are there any workflows or tools or efficiencies that could be easily realized as a commercial offering that would not require massive man hours to implement?

The cost of training is decreasing, but the meaningfully large and non-trivial training sets are almost exclusively in the domain of large companies, economically inaccessible to the individual developers/startups.

This is a space I worked on during the crypto boom of 2017/2018.

The opportunity is present for a decentralized network that allows for training of models to be done from training sets at facilities.

Think of all the data sitting in silos from clinical trials. There is of course the painful process of authenticating researchers for access to data like that but it can be done. There just needs to be an economic reason to make that kind of effort.

I got pulled into a direction of using ML to predict costs of care in insurance so didn’t go further down the rabbit hole but I did author a patent for a novel approach to have a decentralized identity exchange data.

If any of this sounds exciting to you feel free to email me. hn (at) strapr (dot) com

Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law

#57

Post by reitzensteinm »

andrewprock wrote: post #53
bcrosby95 wrote: post #40

Earlier quoted context omitted.

Yes but that aspect of Moore's law for CPUs expired over a decade ago. It's the whole reason we got multicore in the first place.

Even with multi-core, a CPU today is only 6x faster than a 10-year old CPU.

The difference might be even less. 4 Sandy Bridge cores (excluding memory controller and graphics) were not much bigger than the current 8 core Zen 2 die.

Certainly the peak performance you can put in a socket is much higher, but it's got more silicon in it than it used to.

Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law

#58

Post by giantrobot »

moonchild wrote: post #54
Const-me wrote: post #35

Earlier quoted context omitted.

> then CPUs got faster and it wasn't anymore The enabling tech was AES-NI instruction set, not the speed. Agree on the rest. The main reason why modern CPUs and GPUs all have 16-bit floats is probably the deep learning trend.

If it hadn't been aes-ni, it would have been chacha, which is much faster than unaccelerated aes and close to the speed of accelerated aes. Phones use https without a problem, and those haven't had hw-accelerated aes until recently.

A phone needing to set up a dozen HTTPS sockets is nothing for the CPU to do even without acceleration. A server needing to consistently set up hundreds of HTTPS sockets is where AES-NI and other accelerated crypto instructions becomes useful.

Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law

#59

Post by Rastonbury »

calebkaiser wrote: post #7

This is an odd framing. Training has become much more accessible, due to a variety of things (ASICs, offerings from public clouds, innovations on the data science side). Comparing it to Moore's Law doesn't make any sense to me, though. Moore's Law is an observation on the pace of increase of a tightly scoped thing, the number of transistors. The cost of training a model is not a single "thing," it's a cumulative effe…

Implicit in moore's law is that cost does not increase in the same way, if not prices of chips would also be doubling. Something more analogously sounding would be the decrease of cost per transistor.

The number of transistors is also not dependent on a single thing, it can be argued many macro events contributed since the 80s, the VC model for chipmakers in SV, the rise of the internet, going fabless, rise of mobile, innovations in fabrication technology.

Re: The cost to train an AI system is improving at 50x the pace of Moore’s Law

#60

Post by Rastonbury »

smabie wrote: post #50
yelloweyes wrote: post #37

Earlier quoted context omitted.

anything that's even remotely profitable is already taken

Ah yes, of course. There will never be A new profitable ML startup until the end of time. Makes perfect sense.

People said the same thing about SaaS 5 years ago
Post reply on HN