Away. (Or less snarkily, https://twitter.com/iquilezles/status/1212377355349417986 )
"Derp learning" :-)
Predicting where AI is going in 2020
41–50 of 88 posts
Re: Predicting where AI is going in 2020
#42- individual GPUs will hit a plateau at around 25TFlops in FP32 due to Moore's law and thermal dissipation however it will be easier than ever to interconnect multiple GPUs into large virtual ones due to interconnect tech improvements and modularization of GPU processing units - only large companies will be able to train and use SOTA models with training costs in $10M-$100M per training run and those models will hit…
So the Butlerian Jihad [0] for the 2040s?
[0] https://en.wikipedia.org/wiki/Dune_(franchise)#The_Butlerian...
Re: Predicting where AI is going in 2020
#43Earlier quoted context omitted.
There may be some kind of labeling encoded in genes. One thing that it is safe to assume is genetically encoded somehow is that sounds made by your parents/humans around you is worth repeating while other sounds are not. However, past that, the actual sounds themselves, and any association to meaning, are pretty far from tagged data sets. Stuff like the specifics of language (e.g. that a dog is called 'dog') are defi…
> One thing that it is safe to assume is genetically encoded somehow is that sounds made by your parents/humans around you is worth repeating while other sounds are not. I don't see how that's safe to assume at all. What one could assume is the level of familiarity and comfort (sight, smell, touch) might be somewhat genetic and gives such inputs precedence. OR it might just be that those sources of information are en…
The encoding I was talking about may well be something more abstract than 'imitate humans'. Still, babies don't generally try to imitate the sound of rattles or household sounds nearly as much as speech, so I still conclude that it is a safe assumption that there is something about sounds made by humans that is inherently interesting to them for some reason (instead of being a learned behavior).
Related to the second, the rate at which we learn, and the very specific order we learn things in, points very strongly in the direction that there is some built-in model that we train inside of. For example, essentially all babies first learn intonation before learning words. Also, most words are learned with an extremely small set of examples - at some ages, often hearing a word a single time is enough for the child to learn it (known as the 'poverty of the stimulus' problem). This has been mainstream understanding ever since behaviorism fell out of favor due to similar arguments by Chomsky.
Re: Predicting where AI is going in 2020
#44Earlier quoted context omitted.
There may be some kind of labeling encoded in genes. One thing that it is safe to assume is genetically encoded somehow is that sounds made by your parents/humans around you is worth repeating while other sounds are not. However, past that, the actual sounds themselves, and any association to meaning, are pretty far from tagged data sets. Stuff like the specifics of language (e.g. that a dog is called 'dog') are defi…
> One thing that it is safe to assume is genetically encoded somehow is that sounds made by your parents/humans around you is worth repeating while other sounds are not. Well, these sounds come with a face attached and we know babies are hardwired to pay attention to faces.
That may well be the mechanism behind this. I was talking in very general terms, not a specific 'imitate humans' structure in the brain.
Re: Predicting where AI is going in 2020
#45Earlier quoted context omitted.
For most of the engineering applications I work on, AI is useless. When we talk about controlling machines, our control algorithms have mathematically proven strict error bounds, such that if we provide an input with a particular maximal error (e.g. from a sensor that has some error tolerance), we can calculate what's the maximum error possible in the response that our model would produce, and then use that to evalua…
Control algorithms are a part of the problem. What about input data? There's nothing that comes close to NNs in answering a question, say, "Is there pedestrian ahead and what he/she will probably do?" A control system doesn't need to be end-to-end neural, by the way.
> What about input data?
What about it? You get input data from data sources which, e.g., in a car it would be a sensor. The manufacturer of the sensor provides you with the guaranteed sensor accuracy for some inputs, which gives you the upper bound on the input error from that source.
That is, in a reliable control pipeline, the upper bounds on the errors of data sources are known a priori.
Sure, sensors can malfunction, but that's a different problem that's solved differently (e.g. via resiliency using multiple sensors).
> A control system doesn't need to be end-to-end neural, by the way.
Who's talking about end-to-end neural nets for control? If a single part of your control pipeline has unknown error bounds, your whole control system has unknown error bounds. That is, it suffices for your control pipeline to use a NN somewhere for it to become unreliable.
This doesn't mean that you can't use control systems with unknown error bounds somewhere in your product, but it does mean that you can't trust those control systems. This is why drivers still need to keep their hands on the steering wheel on a Tesla: the parts of the pipeline doing the autonomous driving use NNs for image recognition, and the errors on that are unknown.
This is also why all "self driving" cars have simpler data sources like ultrasonic sensors, radar, lidar, etc. which can be processed without NNs to avoid collisions reliably. You might still use NNs to improve the experience but those NNs are going to be overridden by reliable control pipelines when required.
Re: Predicting where AI is going in 2020
#46Earlier quoted context omitted.
I show my 19 month old daughter like three cartoon drawings of owls and she recognises a live one at the bird park instantly, unprompted. We have a way to go.
I believe cartoons are our equivalent of adversarial images. They typically look nothing like (photos of) their namesake and yet we recognise them usually without prompting.
It is my understanding (although I sure don't have any evidence on me) that cartoons and such (at least, the ones where we haven't simply learned that this cartoon means this animal) work by being a picture of what we remember about an animal. Akin to a caricature; the cartoon contains the most salient features. It doesn't work by looking like the actual animal; it works by reacting with how we remember the animal.
Re: Predicting where AI is going in 2020
#47Earlier quoted context omitted.
There may be some kind of labeling encoded in genes. One thing that it is safe to assume is genetically encoded somehow is that sounds made by your parents/humans around you is worth repeating while other sounds are not. However, past that, the actual sounds themselves, and any association to meaning, are pretty far from tagged data sets. Stuff like the specifics of language (e.g. that a dog is called 'dog') are defi…
There seems to be a spectacular underestimation of the amount of training data humans experience. Not only does socialised human intelligence require at least a decade of formal education, but it also spends a lot of time in a complex 3D environment which is literally hands-on. It's true some of the meta-structures predispose certain kinds of learning - starting with 3D object constancy, mapping, simple environmental…
Well, babies start picking out objects within weeks or months after birth. And many birds and mammals are much faster than that. That's not a huge amount of data to learn something so abstract from scratch, especially given the limited bandwidth of our data acquisition.
Furthermore, for other kinds of human knowledge, the learning process is very rarely based on data. After the acquisition of language, we generally seem to learn much more by analogy and deduction than by purely analyzing data. The difference is evident, since we can often pick up facts with a single datapoint, even in small children in kindergarten.
Also, getting back to your point on how we start AI - if you try to take a neural network and throw 3D sensor data at it, and immediately start using its outputs to modify the environment those sensors are sensing, I suspect you will not get any meaningful amount of learning. You probably need a very complex model and set of initial weights to have any chance of learning something like 3D objects and their basic physics (weight, speed and hwo those affect their predicted position). I would at least bet that you wouldn't get anywhere near, say, kitten accuracy in one month of training.
Related to 3D objects vs 2D, I completely agree.
Re: Predicting where AI is going in 2020
#48Honesty, repeatability, numerical analysis. Canonicalization. Honesty: how many times was the exact same data processed? Was the result cherry picked and the best one published? For the sake of integrity how is it possible to scientifically improve on this result? (example, your AI outputs some life altering decision?) Repeatability: In science, if a result can be independently verified, it gives validity to the "con…
I would argue most machine learning papers that use public datasets have code available and are often also reproduced independently (sometimes just because of somebody's need to port between PyTorch/TensorFlow). Reproducibility is still a big problem in reinforcement learning, however.
People are definitely thinking carefully about issues of noise and quantization error. Low-precision or quantized neural networks are increasingly popular at both train and test time. And people deliberately introduce noise into neural networks for various reasons (dropout, robustness certificates) and then have to think about the effect on performance. Typically things are quite reproducible in these situations btw, for a given noise distribution.
Re: canonicalization, the theoretical work on "neural tangent kernels" might be relevant.
Re: Predicting where AI is going in 2020
#49Earlier quoted context omitted.
There seems to be a spectacular underestimation of the amount of training data humans experience. Not only does socialised human intelligence require at least a decade of formal education, but it also spends a lot of time in a complex 3D environment which is literally hands-on. It's true some of the meta-structures predispose certain kinds of learning - starting with 3D object constancy, mapping, simple environmental…
Well, babies start picking out objects within weeks or months after birth. And many birds and mammals are much faster than that. That's not a huge amount of data to learn something so abstract from scratch, especially given the limited bandwidth of our data acquisition. Furthermore, for other kinds of human knowledge, the learning process is very rarely based on data. After the acquisition of language, we generally s…
[deleted]
Re: Predicting where AI is going in 2020
#50Earlier quoted context omitted.
> One thing that it is safe to assume is genetically encoded somehow is that sounds made by your parents/humans around you is worth repeating while other sounds are not. I don't see how that's safe to assume at all. What one could assume is the level of familiarity and comfort (sight, smell, touch) might be somewhat genetic and gives such inputs precedence. OR it might just be that those sources of information are en…
The encoding I was talking about may well be something more abstract than 'imitate humans'. Still, babies don't generally try to imitate the sound of rattles or household sounds nearly as much as speech, so I still conclude that it is a safe assumption that there is something about sounds made by humans that is inherently interesting to them for some reason (instead of being a learned behavior). Related to the second…
> try to imitate the sound of rattles or household sounds nearly as much as speech
Well surely that's a case of the range of the vocal chords? Parrots are another intelligent creature that has better range and they imitate all sorts of sounds.
> Related to the second, the rate at which we learn, and the very specific order we learn things in, points very strongly in the direction that there is some built-in model that we train inside of.
Or that an action like walking requires one to put one foot ahead of the other, all other strategies in attempting to walk end in failure, which is why we don't see them.
I'd like to point out that all humans perceive intonation and its perceivable outside of language, that's why its easy to pick up, you don't need language to realise that someone is cross, or happy or sad. However considering autistic children cannot then maybe there are some genetic markers at play there at least.