Limitless exponential growth doesn't happen in the physical world, it happens on S-curves. Black phosphorus anyone?
Care to elaborate?
The future of computing: After Moore's law
21–30 of 67 posts
Re: The future of computing: After Moore's law
#22I was recently thinking about Moore's Law and if it is truly coming to an end. My initial reaction is that it doesn't matter in itself, if the number of transistors doubles, what might matter is that our compute power is doubling.
This led me to think that maybe Moore's Law is looking at the wrong metric, and is there a more fundamental law regarding increased capacity. Some proof of this is in the drop in prices of cloud computing and increases in network capacity. If chips stopped getting more powerful, in theory, we may not notice as the capability of the entire network continues to increase at the familiar rate.
However, this begs the question, did this increased capacity actually begin with Moore's Law and computer processing in general? Or is there an overarching law of progress which has always existed. To prove this, I'd need to go back and look at the growth of industrialization. I suspect as the capabilities of the factories slowed, our network speed (rail and sea, then road and then air) increased. As the network speed reached a plateau, we began to find places where we could manufacture more cheaply, thereby increasing the output per cost. Does the same growth exist in agriculture? What major industry does not fit?
There is some evidence that a law similar to Moore's law exists not only for technology. What effect does that have in our understanding of why things grow?
[edit: this is was an except from my YC application to the question regarding what have you discovered]
Re: The future of computing: After Moore's law
#23Earlier quoted context omitted.
> do we have any ideas what the limits are on the algorithm side of things? I've always believed that humans do not have the ability to program things smarter than themself, because we do not understand our own intelligence, so we have no way to reproduce it. At the time, I said the only alternative I can think of is make random permutations and pick the best one, and go from there. But I said this as a ridiculous su…
Humans every day "program" things smart than themselves. Every parent who raises a child smarter than themselves does exactly that. Smarter than human level computers will be created the same way we create smart humans.
I do not think this is equal. Babies may actually be "brighter" than parents, without the parents having to actually "do" anything to produce the higher IQ baby. But building an artificial system means some humans actually have to design it.
Re: The future of computing: After Moore's law
#24Earlier quoted context omitted.
It is worth remembering that just because we are reaching some physical limits of "classical" chip design. For example, both x86 and arm date back to the 80s. While I have no doubt that the implementation of these architectures has improved to reflect modern manufacturing capabilities, this still suggests that there is room for architectural improvements in performance. Beyond simple architectural improvements, we co…
> For example, both x86 and arm date back to the 80s. Modern x86 and high performance ARM cores are almost unrecognizable compared to processors in the late 90s, much less the 80s. (Also, ARM was founded in 1990, not the 80s). > While I have no doubt that the implementation of these architectures has improved to reflect modern manufacturing capabilities, this still suggests that there is room for architectural improv…
From wikipedia on ARM:
The official Acorn RISC Machine project started in October 1983.
But everything else you say is bang on.
Re: The future of computing: After Moore's law
#25I was recently thinking about Moore's Law and if it is truly coming to an end. My initial reaction is that it doesn't matter in itself, if the number of transistors doubles, what might matter is that our compute power is doubling. This led me to think that maybe Moore's Law is looking at the wrong metric, and is there a more fundamental law regarding increased capacity. Some proof of this is in the drop in prices of…
The idea that exponential growth has been happening long before Moore's law is one of the main themes of Ray Kurzweil's book "The Singularity is Near". https://en.wikipedia.org/wiki/The_Singularity_Is_Near
Re: The future of computing: After Moore's law
#26Research by K. Ebcioglu on very long instruction word (VLIW) shows that 24 way VLIW can get 9:1 speedup on ordinary code.
Compared to what? What does ordinary code mean? VLIW has been around for a while (Itanium is probably the most famous "general purpose" example) and has failed to gain traction outside of GPUs and DSP (ie not "ordinary code").
> Compared to what?
A single core processor, of course.
> What does ordinary code mean?
Say, simple random sample of all the other code being run.
> VLIW has been around for a while
Yup. I never claimed otherwise.
> Itanium is probably the most famous "general purpose" example
Yup.
> and has failed to gain traction outside of GPUs and DSP (ie not "ordinary code").
Yup.
Still, yet again, over again, one more time, once again, VLIW can get 9:1 speedup.
Why mention this? Because the OP was talking about the challenges of getting faster computing. Well, if want faster computing, one approach, that, indeed, works on general purpose code, and gives ballpark 9:1 speedup is, and may I have the envelope please [drum roll, please], VLIW. Really new? Nope. Tried with Itanium? Yup. Works? Yup.
Bottom line -- we still have 9:1 speedup available to us.
Maybe an objection is that pay a factor of 24 in transistor count and electrical power but get only a factor of 9 in performance.
Not everyone knows about VLIW.
Where did I say something wrong?
Re: The future of computing: After Moore's law
#27Limitless exponential growth doesn't happen in the physical world, it happens on S-curves. Black phosphorus anyone?
Care to elaborate?
The logistic function (or "S-curve") describes systems that expand first at exponential rates, then logarithmic ones. With respect to new technologies, it's been observed that adoption rates and most measurable improvements follow a logistic function.
For example, people did not go from buying 1 car to buying 10 cars and then 100 cars - most of us hit saturation somewhere between 1 and 2, and stayed there. Similarly, the average speed of our cars did not increase at an increasing rate except in the earliest years of automotive technology; we have a "highway maximum" between 50 and 70 MPH, and a "technical maximum" in the high 200's range for production cars which do not rely on rockets or other not street-legal tricks (a quick search brings up the Hennessey Venom GT at 270.49 MPH). Likewise applied to fast moving vehicles as a whole, including supersonic aircraft and rockets, we've already brushed up against vehicle weight and power density limits that slow the rate of improvement in acceleration.
Applied to the Moore's Law measures, that indicates we have a "endless sunset" period ahead of us where we'll still get more doublings of semiconductors, but they'll come increasingly slowly as more fundamental innovations become necessary to realize them.
What we don't know is whether there is a logistic curve on technology as a whole. Belief in the Singularity is premised on this not being the case.
Re: The future of computing: After Moore's law
#28The use of highly parallel GPUs as general purpose compute elements is a major trend. Most graphics boards have more compute power than the main CPU they serve. The Titan and Summit supercomputers have most of their compute power in the GPUs. The limits of GPU parallelism haven't been reached yet. Machine learning can be done on GPUs, and that's the biggest CPU hog problem right now that is getting mainstream use. That technology isn't near a ceiling.
As machine learning/AI/neuron emulation becomes more useful, we'll see hardware specialized for that. It's not yet clear what shape that hardware will take. Look for "works fine, but is slow on GPUs" results to lead to such hardware, rather than "build it and they will come" projects like the Human Brain Project.
There's still more improvement possible in storage devices. With the 20TB SSD drive expected in 4 years, things are looking good in that area. For compute elements, cooling limits transistor density. Storage tends not to be heat dissipation limited.
Re: The future of computing: After Moore's law
#29While we're reaching the physical limits of classical chip design, do we have any ideas what the limits are on the algorithm side of things? As much speed up has come from software as hardware according to a few reports. http://www.johndcook.com/blog/2015/12/08/algorithms-vs-moore...
> do we have any ideas what the limits are on the algorithm side of things? I've always believed that humans do not have the ability to program things smarter than themself, because we do not understand our own intelligence, so we have no way to reproduce it. At the time, I said the only alternative I can think of is make random permutations and pick the best one, and go from there. But I said this as a ridiculous su…
>But then Google actually did exactly that with their Go machine!
I would argue that this is not what they did. What they did was take something a human player does (study games to learn how to play). And then parallelized this to a level that a human can't match. Basically the equivalent of having one go player playing against an entire team of go players who are all experts.
Re: The future of computing: After Moore's law
#30Earlier quoted context omitted.
Humans every day "program" things smart than themselves. Every parent who raises a child smarter than themselves does exactly that. Smarter than human level computers will be created the same way we create smart humans.
I do not think this is equal. Babies may actually be "brighter" than parents, without the parents having to actually "do" anything to produce the higher IQ baby. But building an artificial system means some humans actually have to design it.
Yes parents have to do something to end up with brighter children than themselves - if you put a child in a dark room from birth with no human interaction I can guarentee that they will not come out very bright 20 years later (there have been some very sad examples of this in history).
The point I was making is that intelligent agents are built all the time by less intelligent agents. Humans will build more intellgent AI using exactly the same approach and these AI's can do the same.
Of couse building an AI via evolutionary training opens up a whole lot of control issues. Do we really want to be creating entities vastly more intelligent than ourselves when we have no real idea of their interests and when we occupy something that is valuble (e.g. matter and energy) to this entity?