While this is a collective work, honestly, after hearing about JD for so many years: is there anything he CAN’T do?
He did little for tpu.
Machine Learning for Systems and Systems for Machine Learning [pdf]
11–20 of 50 posts
Re: Machine Learning for Systems and Systems for Machine Learning [pdf]
#12Was it filmed? If yes, when video will be available?
Re: Machine Learning for Systems and Systems for Machine Learning [pdf]
#13Nvidia Titan V can do 110 TFLOPS, 12GB of 1.7 Gb/s Memory [1] and sells for 3,000$. TPU v2 does 180 TFLOPS, 64GB of 19.2Gb/s Memory [2]. That's a heck of a performance boost for a chip that's likely costing google way less than the nvidia flagship. [1] http://www.tomshardware.com/news/nvidia-titan-v-110-teraflop...
It's not clear to me how programmable the tpu is. I'm sure it's great at convolutions and matrix multiplies. Can it do anything else?
Neither do tensor cores
Re: Machine Learning for Systems and Systems for Machine Learning [pdf]
#14That "Learned Index Structures" makes it pretty clear that Karpathy was right in his widely criticized "Software 2.0" piece.
I thought it was a stretch when reading that medium post. Now reading this and thinking about it after finishing two undergraduate classes one on operating systems and another on compilers, the machine learning for systems part makes a lot of sense, apart from the heuristics the learned index structures idea is just fascinating.
Another illuminating sentence from the paper was this:
>This leads to an interesting observation: a model which predicts the position given a key inside a sorted array effectively approximates the cumulative distribution function (CDF). We can model the CDF of the data to predict the position as: p = F(Key) ∗ N
Maybe it's just my very limited knowledge as an undergrad but I'm feeling that this can be the start of something big. Another idea that just came to me after is how much of this ML is applicable to the domain of cryptography. In my security class it seemed like much of the famous hash functions for example were somehow "found" in vast space of potential schemes.
Re: Machine Learning for Systems and Systems for Machine Learning [pdf]
#15It looks like this paper has more information: https://arxiv.org/pdf/1712.01208v1.pdf
Re: Machine Learning for Systems and Systems for Machine Learning [pdf]
#16Nvidia Titan V can do 110 TFLOPS, 12GB of 1.7 Gb/s Memory [1] and sells for 3,000$. TPU v2 does 180 TFLOPS, 64GB of 19.2Gb/s Memory [2]. That's a heck of a performance boost for a chip that's likely costing google way less than the nvidia flagship. [1] http://www.tomshardware.com/news/nvidia-titan-v-110-teraflop...
It's not clear to me how programmable the tpu is. I'm sure it's great at convolutions and matrix multiplies. Can it do anything else?
Without speaking to the capabilities of TPUs, note that most ML models today are mostly convolutions and matrix multiplies.
Re: Machine Learning for Systems and Systems for Machine Learning [pdf]
#17That "Learned Index Structures" makes it pretty clear that Karpathy was right in his widely criticized "Software 2.0" piece.
I think you got the point though. This is going to be HUGE. But why the downvotes? Job security concern?
It could be. CS and more generally tech related fields have this positive feedback loop where the technology facilitates it's own development.
For example:
The easiest and most abundant thing to learn on the web is unsurprisingly web development.
An ML engineer can use ML to optimize the data structures that he uses for his models.
I could not say the same about fields like biology or physics.
Re: Machine Learning for Systems and Systems for Machine Learning [pdf]
#18Great talk, with lots of new insights into what's happening at Google. I really think his point that ImageNet is the new Mnist now holds true. Even research labs should be buying DeepLearning11 servers (10 x 1080Ti) for $15k, and training large models in a reasonable amount of time. It may seem that Google are way ahead, but they are just doing synchronous SGD, and it was interesting to see the drop in prediction accuracy from 128 TPU2 cores to 256 TPU2 cores for ImageNet (76 -> 75% accuracy). So, the algorithms for dist. training aren't unknown, and with cheap hardware like the DL11 server, many well-financed research groups can compete with this.
Re: Machine Learning for Systems and Systems for Machine Learning [pdf]
#19Earlier quoted context omitted.
I think you got the point though. This is going to be HUGE. But why the downvotes? Job security concern?
It could be. CS and more generally tech related fields have this positive feedback loop where the technology facilitates it's own development. For example: The easiest and most abundant thing to learn on the web is unsurprisingly web development. An ML engineer can use ML to optimize the data structures that he uses for his models. I could not say the same about fields like biology or physics.
Yeah, it is funny that ML, which is biggest fears driven by the media as the ultimate job destroyers, would put this generation of programmers in danger as well.
However, the paradigm shift is inevitable, once discovered, people will use it, and use it anywhere possible.
Re: Machine Learning for Systems and Systems for Machine Learning [pdf]
#20That "Learned Index Structures" makes it pretty clear that Karpathy was right in his widely criticized "Software 2.0" piece.
I haven't read that paper (Learned Index Structures), but things like gperf have existed for decades. Are these enhanced data structures dynamic, i.e. unlike gperf which is a static one, does it reoptimize as you insert new elements?
In the case of the hash table, I assume it's using the model to compute the hash function.