> You’re statement about us not learning anything about Go from AI agents is just false. Also we have seen a rapid improvement in the accuracy/speed curve, most notably with NAS. It hasn’t just been increased computational power.
We've learned how to play better Go, but not anything like a new (mathematical) theory of Go, as far as I have read. And I didn't claim it's just about computing speed improvements, I also noted that we have started the right formulas to fit specific NN architectures with specific kinds of problems.
> AFAIK the use of learned embedding in the biomedical space has improved our understanding of the science/our ability to find high value experiments to run.
Awesome, this is one thing I hadn't heard about.
> I think the coherency of GPT-3 and the RL learned tool use work both raise extremely interesting questions about the nature of language and high-level intelligence. Questions that we couldn’t meaningfully ask before that evidence.
They do raise interesting questions in a philosophical sense, but those questions do not seem to have piqued the interest of the AI community. The GPT-3 paper covers some interesting facts about GPT-3's ability to do arithmetic, but only briefly, and that's about it. They explicitly attribute GPT-3's impressive gains over GPT-2 entirely to the hugely increased parameter space, and mention that they believe that increasing the parameter space again by an order of magnitude will increase the realness even more - that's the extent of their analysis about its implications on language that I've seen (perhaps I missed something?).
I haven't even seen a real discussion about how different GPT-3's output is compared to it's training inputs. Given the gigantic corpus that they have used to train it, I would expect to see more scrutiny in this area. For the arithmetic operations, they mention some attempts they made to check that it wasn't reproducing a calculation it had explicitly seen, but they are not confident that they were enough.
More importantly, it is obvious that, even if GPT-{X} will be a complete model of human language, it will (1) have nothing to do with how humans acquired language, either individually or evolutionarily; and (2) that it won't be able to produce meaning that is not captured in its corpus, as it will have no notion of the human world and its reality; it might be able to produce commentary on current events that sounds plausible, but any relation between its commentary and reality will be either entirely captured in the training corpus, in the priming text, or entirely accidental.
> I also think you’re being shortsighted about the possibilities that are opened up as we enable computers to understand and interact with the real world in more way. Increased visual and audio understanding will lead to more passive computing and that has all sorts of fascinating implications.
Of course that's a strong possibility - the future is usually surprising. However, the only applications actually visible on the horizon are bone-chilling: mass surveillance that even Orwell didn't dream of, increasingly being actively used by more and more authoritarian regimes, from China to the US to Europe. It may well soon turn out that the AI fearmongers were right to fear AI, though of course not in the puerile fantasy of the paper-clip optimizer.
> We’re seeing AI reveal fascinating phenomena in language, vision and general purpose learning.
I have seen little to no coverage of such fascinating phenomena, and entirely too much coverage about precision rates, eerily human-like language generation, and a belief that bigger models and more data are the only way forward. Everything I have seen has been AI research not only not asking questions about such phenomena, but instead shutting down any questions, claiming that the million-parameter models that they produce on terrabytes of training ARE the answers [0].
> Are you in the field? You’ve focused on the things that made global headlines, but I feel like there’s so much more.
I am not in the field of AI/ML, no (the closest I got was doing my bachelor's thesis on an RL approach). The things I focused on were the advances highlighted by the post I replied to. I have no doubts that there are many advances in the techniques of AI that would completely fly over my head, and I am certain that there are successful applications of AI on hard problems that I never even heard about. AI is certainly a useful technique in many fields, though often over-hyped as well. Still, judging by what is commonly highlighted as the major achievements of the field, my prediction is still that it's positive impact on the world will continue to be limitted for the next 1-2 decades; its negative impact through enabling mass surveillance may well be far greater.
[0] http://norvig.com/chomsky.html