Intel Reveals Post-8th Gen. Core Architecture 10nm+ Ice Lake

2007 thinkpad x61, core 2 duo T7100, but with 4GB of RAM and an SSD, to be fair I'm cheating a little as I use Debian+DWM which is lighter than Windows 10.

But it holds itself pretty well and CPU limitation will probably not be the reason I change it. The screen resolution (1024x768) will probably be the main motivator.

But this only illustrates that there were some improvement because of RAM evoluation (4 to 8 GB or RAM now for average PCs vs 1 to 2 GB back then) and huge improvement because of disk evolution (SSD vs mechanical drives). CPU is far from being the main improvement factor for common usage in the last 10 years.

This actually describes neural networks in general, not so much "deep learning".

Deep learning comes from being able to scale up neural networks from having only a few 10s or 100s of nodes per layer, to thousands and 10s of thousands of nodes per layer (and of course the combinatorial explosion of edges in the network graph between layers), coupled with the ability to process and use massive datasets to train with, and ultimately process on the trained model.

This has mainly been enabled by the cheap availability of GPUs and other parallel architectures, coupled with fast memory interconnects (both to hold the model and to shuttle data in/out of it for training and later processing) and the CPU (probably disk, too).

But neural networks have almost always been represented by matrix operations (linear algebra), it's just that there wasn't the data, nor the vast (and cheap) numbers of parallelizable processing elements available to handle it (the closest architectures I can think of that could potentially do it in the 1980/90s would be from Thinking Machines (Connection Machines) and probably systolic array processors (which were pretty niche at the time, mainly from CMU):

https://en.wikipedia.org/wiki/Systolic_array

https://en.wikipedia.org/wiki/WARP_(systolic_array)

These latter machines started to prove some of what we take for granted today, in the form of the NAVLAB ALVINN self-driving vehicle:

http://repository.cmu.edu/cgi/viewcontent.cgi?article=2874&c...

Of course, today it can be done on a smartphone:

http://blog.davidsingleton.org/nnrccar/

The point, though, is that neural networks have long been known to be most effectively computed using matrix operations, it's just that the hardware wasn't there (unless you had a lot of money to spend) nor the datasets - to enable what we today call "deep learning".

That, and AI winters didn't help matters. I would imagine that if somebody from the late 1980s had asked for 100 million to build or purchase a large parallel processing system of some form for neural network research - they would've been laughed at. Of course, no one at that time really knew that what was needed was such large architecture, nor the amount of data (plus the concept of convolutional NNs and other recent model architectures weren't yet around). Also - programming for such a system would have been extremely difficult.

So - today is the "perfect storm", of hardware, data, and software (and people who know how to use and abuse it, of course).

What this means is that to implement the hardware the FPGA represents, you have to "program" it; this is typically done in one of only a couple HLLs (VHDL and Verilog, known as Hardware Descriptor Languages or HDLs).

At one time, Xilinx (Altera's competitor) made an FPGA which could be programmed "on the fly" very quickly; it (well, many thousands) were used for an interesting machine, of which only a few examples survive (the whole thing at the time was surreal, if you followed it - it seemed like a scam more than anything, but real hardware was shipped).

This machine was called the CAM-Brain machine, and was the creation of researcher Hugo de Garis (who is retired, and is a seemingly strange fellow in the AI community - but not as strange as Mentifex):

https://en.wikipedia.org/wiki/Hugo_de_Garis

http://dl.acm.org/citation.cfm?id=591856

https://profhugodegaris.files.wordpress.com/2014/02/arj-rev2...

I encourage you to research this machine, and Mr de Garis, as the whole thing is fascinating (and I will also say, from a design perspective, the shipped CAM-Brain Machine was one of the "sexiest" looking boxen since the early Crays).

CAM-Brain meant "cellular automata machine brain" - it was basically an effort to evolve a neural network using CA and FPGA; the CA would evolve the HDL which described the hardware representation of the NN, which would then be dumped to the FPGA for processing. The process (from what I understand) was iterative.

I don't believe the "kitten" ever went past much more than some early 3D models (maybe some CAD, too) and a software simulator. At least, that's what you can still find out there today (images of the simulator running on Windows NT, iirc).

The effort was noble, but it didn't work for more than simple things. I think it was part of the "evolve-a-brain" NN dead end, which seemed to hold out some promise at the time.

That's just a bit of background, but it shows how Intel and FPGAs can be used for building hardware to represent neural networks (a GPU/TPU is not a neural network - it is merely a processor for the software representation of the neural network). Whether that's their intention, or something else (maybe something like Transmeta tried?) - only they know.