Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Path-breaking Papers About Image Classification

blog.paralleldots.com

Post reply on HN
Print view View original

21–30 of 32 posts

Re: Path-breaking Papers About Image Classification

#21

Post by rspeer »

scj wrote: post #20
falcolas wrote: post #11

Some really cool information, but this concluding bit annoyed me: > By Moore’s law, we will reach computing power of human brain by 2025 and all of the humanity by 2050. Their graph does show exponential growth, but the data points cut off at the year 2000. Not surprising, given that Moore's law has reached its end in the last decade. ML improvements now depend upon better algorithms to make them more parallel, and t…

> Their graph does show exponential growth, but the data points cut off at the year 2000. Not surprising, given that Moore's law has reached its end in the last decade. I think the graph was originally produced for Ray Kurzweil's 1999 book "The Age of Spiritual Machines".

Which predicted we'd meet a bunch of milestones of general AI by 2009.

Why is this post about a specific machine-learning task even referencing Kurzweil, anyway?

Re: Path-breaking Papers About Image Classification

#22

Post by lostmsu »

falcolas wrote: post #19
londons_explore wrote: post #12

Earlier quoted context omitted.

Moore’s law is still alive and well, you just have to move over to the parallel architectures like GPU's. Considering machine learning is all on GPU's and TPU's now, I think this is still a fair assessment.

Not really - since GPUs have started to hit the same "Process" size as CPUs - and haven't been showing a lot of growth in that area. The best improvement per Wikipedia's chart is a small foray into 14 and 12nm, and those haven't doubled the transistor counts (per square mm). What we are seeing is an increase in die sizes; more parallel cores. Parallel cores still require parallel algorithms, so I stand by my earlier…

Latest NVidia GPUs are ~1.3-1.4 times faster, than previous gen with the same or lower power consumption.

Re: Path-breaking Papers About Image Classification

#23

Post by randcraw »

kayoone wrote: post #17
falcolas wrote: post #11

Some really cool information, but this concluding bit annoyed me: > By Moore’s law, we will reach computing power of human brain by 2025 and all of the humanity by 2050. Their graph does show exponential growth, but the data points cut off at the year 2000. Not surprising, given that Moore's law has reached its end in the last decade. ML improvements now depend upon better algorithms to make them more parallel, and t…

GPUs still improve a lot year over year, so i think Moore's law still holds true for at least a few more years.

I believe you're comparing apples and oranges.

GPUs and multicore CPUs do not perform the same work as the single core CPUs that dominated before ~2003, when Moore's Law first slowed. By expanding the measure of speedup to include GPU/multicore, arguments like yours require not only a change in hardware but in benchmark code as well.

Longstanding general app benchmarks like SPEC emphasized everyday tasks that rarely benefit from parallelism, appropriately revealing the general ineffectiveness of adding GPUs or multicore to everyday apps. The only fair way to assess the impact of GPU/multicore is to continue using the same benchmark code as when mono-core CPUs reigned. When you do that, the value of adding GPU/multicore essentially disappears and the speedup of Moore's Law duly fades (again ~2003).

Thus until users begin to run deep learning code on their computer's GPU, AI-code won't speed up exponentially, nor continue with future GPU advancement. The hardware basis driving Kurzweil's Singularity has truly run out of steam.

Re: Path-breaking Papers About Image Classification

#24

Post by T_D_K »

Does anyone have insight as to why they're still doing top 5? It seems to me like the error rates have dropped low enough that they could move on to top 3 or even single guess challenges. Is there data that shows how these same models perform in such tasks? Though I suppose, if I was motivated, all the needed tools are available to find out for myself.

Re: Path-breaking Papers About Image Classification

#25

Post by AstralStorm »

lostmsu wrote: post #22
falcolas wrote: post #19

Earlier quoted context omitted.

Not really - since GPUs have started to hit the same "Process" size as CPUs - and haven't been showing a lot of growth in that area. The best improvement per Wikipedia's chart is a small foray into 14 and 12nm, and those haven't doubled the transistor counts (per square mm). What we are seeing is an increase in die sizes; more parallel cores. Parallel cores still require parallel algorithms, so I stand by my earlier…

Latest NVidia GPUs are ~1.3-1.4 times faster, than previous gen with the same or lower power consumption.

Moore's law is about transistor count and not performance. Performance of those parallel architectures always get subject to Amdahl's rule of scaling.

This is partly why we see slowing scaling in performance and why you cannot just throw cores at the problem.

Re: Path-breaking Papers About Image Classification

#26

Post by AstralStorm »

parths wrote: post #8
kushankpoddar wrote: post #6

Sometimes I wonder why is the top-5 image classification task so difficult. If you are giving me 5 chances to look at an image and correctly classify it from ~1000 Imagenet classes, I can surely do better than 5-10% error rate. Also, now that the top-5 error rate been brought down considerably, what is the next benchmark for the research community to beat? A new dataset, top-1 error rate on Imagenet?

A large majority of human errors come from fine-grained categories(such as correctly identifying two similar cat species) and class unawareness. I would recommend this article by Andrej Karpathy, where he talks about his learning from competing against GoogLeNet: http://karpathy.github.io/2014/09/02/what-i-learned-from-com...

That would be relatively low grade error. Specifically errors have to be valued and not just counted.

Re: Path-breaking Papers About Image Classification

#27

Post by donovanr »

falcolas wrote: post #11

Some really cool information, but this concluding bit annoyed me: > By Moore’s law, we will reach computing power of human brain by 2025 and all of the humanity by 2050. Their graph does show exponential growth, but the data points cut off at the year 2000. Not surprising, given that Moore's law has reached its end in the last decade. ML improvements now depend upon better algorithms to make them more parallel, and t…

specious extrapolations aside, the plot itself is deceptive -- exponential growth in a log-linear plot should be a straight line, not one accelerating upward super-exponentially.

Re: Path-breaking Papers About Image Classification

#28

Post by bllguo »

londons_explore wrote: post #12
falcolas wrote: post #11

Some really cool information, but this concluding bit annoyed me: > By Moore’s law, we will reach computing power of human brain by 2025 and all of the humanity by 2050. Their graph does show exponential growth, but the data points cut off at the year 2000. Not surprising, given that Moore's law has reached its end in the last decade. ML improvements now depend upon better algorithms to make them more parallel, and t…

Moore’s law is still alive and well, you just have to move over to the parallel architectures like GPU's. Considering machine learning is all on GPU's and TPU's now, I think this is still a fair assessment.

You're conflating things here; Moore's law is about how many transistors you can pack onto a chip, not performance. Moore's law is unsustainable due to laws of physics. Transistors are reaching size scales where quantum effects like tunneling dominate.

Think of a transistor as having two regions, separated by a channel. When the transistor is on, charge carriers flow through the channel between the regions. When off, charge carriers do not flow. But when we move to smaller and smaller length scales, the channel is so small that charge carriers will tunnel through and reach the other region. How will you distinguish on/off behavior now?

Re: Path-breaking Papers About Image Classification

#30

Post by nomel »

bllguo wrote: post #28
londons_explore wrote: post #12

Earlier quoted context omitted.

Moore’s law is still alive and well, you just have to move over to the parallel architectures like GPU's. Considering machine learning is all on GPU's and TPU's now, I think this is still a fair assessment.

You're conflating things here; Moore's law is about how many transistors you can pack onto a chip, not performance. Moore's law is unsustainable due to laws of physics. Transistors are reaching size scales where quantum effects like tunneling dominate. Think of a transistor as having two regions, separated by a channel. When the transistor is on, charge carriers flow through the channel between the regions. When off,…

Maybe this is silly, but could you use redundancy and statistics to help?
Post reply on HN