An Epidemic of AI Misinformation

Sigh.

> The Economist [...] said that GPT-2’s answers were “unedited”, when in reality each answer that was published was selected from five options

> [Erik Bryjngjolffson] tweeted that the interview was “impressive” and that “the answers are more coherent than those of many humans.” In fact the apparent coherence of the interview stemmed from (a) the enormous corpus of human writing that the system drew from and (b) the filtering for coherence that was done by the human journalist.

If your success rate is ≥20%, the coherence is coming from the model, not the selection process. This is just basic statistics.

> OpenAI created a pair of neural networks that allowed a robot to learn to manipulate a custom-built Rubik's cube

Jeez, I've already corrected you here... well, why not have to do it again?

> publicized it with a somewhat misleading video and blog that led many to think that the system had learned the cognitive aspects of cube-solving

The side not stated: OpenAI said explicitly in the blog that they used an unlearned algorithm for this, and sent a correction to a publisher that got this wrong.

> the cube was instrumented with Bluetooth sensors

During training, but they ended up with a fully vision-based system.

> even in the best case only 20% of fully-scrambled cubes were solved

No, 60% of fully scrambled cubes were solved. 20% of maximally difficult scrambles were solved.

> one report claimed that “A neural net solves the three-body problem 100 million times faster” [...] but the network did no solving in the classical sense, it did approximation

All solvers for this problem are approximators, and vice-versa. The article you complain about states the accuracy (“error of just 10^(-5)”) in the body of text.

> and it approximated only a highly simplified two degree-of-freedom problem

As reported: “Breen and co first simplify the problem by limiting it to those involving three equal-mass particles in a plane, each with zero velocity to start with.”

> MIT AI lab famously assigned Gerald Sussman the problem of solving vision in a summer [https://dspace.mit.edu/handle/1721.1/6125]

I... sigh

“The original document outlined a plan to do some kind of basic foreground/background segmentation, followed by a subgoal of analysing scenes with simple non-overlapping objects, with distinct uniform colour and texture and homogeneous backgrounds. A further subgoal was to extend the system to more complex objects.

So it would seem that Computer Vision was never a summer project for a single student, nor did it aim to make a complete working vision system.”

http://www.lyndonhill.com/opinion-cvlegends.html

> Geoff Hinton [said] that the company (again The Guardian’s paraphrase), “is on the brink of developing algorithms with the capacity for logic, natural conversation and even flirtation.” Four years later, we are still a long way from machines that can hold natural conversations absent human intervention

‘Four years later’ to natural conversation is not a reasonable point of criticism when the only timeline given was ‘within a decade’ for a specified subset of the problem.

> [In 2016 Hinton said] “We should stop training radiologists now. It’s just completely obvious that within five years, deep learning is going to do better than radiologists.” [...] but thus far no actual radiologists have been replaced

So Hinton actually said “People should stop training radiologists now. It’s just completely obvious that within five years, deep learning is going to do better than radiologists, because it's going to be able to get a lot more experience. It might be 10 years, but we've got plenty of radiologists already.”

2019 is not 2026. “thus far no actual radiologists have been replaced” is thus not a counterargument.

> Andrew Ng, another well-known figure in deep learning, wrote that “If a typical person can do a mental task with less than one second of thought, we can probably automate it using AI either now or in the near future.” [...] Ng’s claim thus far has proven incorrect.

I agree. This quote captures the wrong nuance of the issue.

Well, finally finding one point by Gary Marcus that isn't misleading, I think I'm going to call this a day.

These are Edge TPUs, for inference, not for training

Skynet Today is very good. I wish you had more original articles because the ones you have, I enjoy a lot (your editorials and briefs). I see you are calling for contributions, I hope you get some good ones.

Writing good articles- there's no other way to combat bad coverage. Keep it up please and all the best.

Gary Marcus' tone amounts to base trolling and it's a shame that's how he chooses to carry out his criticism. I understand your frustration.

Regarding the lack of transfer, yes, AlphaGo, AlphaZero and most of their variants have boards of fixed size and shape hard-coded in their architecture (as they have the types of piece moves-hard coded) and need architectural modifications and re-training before they can play on different boards or with different pieces (e.g. AlphaGo can't play Chess and Shoggi unmodified). The KataGo paper (the paper you linked) is one exception to this. Personally, I don't know others. Anyway general game-playing is a hard task and nobody claims it's solved by AlphaGo.

Regarding KataGo its main contribution is a significant reduction to the cost of training an AlpahGo variant while maintaining a competitive performance. This is very promising- after DeepBlue, creating a chess engine became cheaper and cheaper until they could run on a smartphone. We are far from that with Go computer players.

However, in the KataGo paper, major gains are claimed to come from a) game-playing specific or MCTS-specific improvements (playout cap randomisation, forced playouts and policy target pruning) or architecture-specific improvements (global pooling) or, b) domain-specific improvements (auxiliary ownership and score targets). Finally, KataGo has a few game-specific features (liberties, pass-alive regions and ladder features).

The KataGo paper itself says it very clearly. I quote, snipping for brevity:

Second, our work serves as a case study that there is still a significant efficiency gap between AlphaZero's methods and what is possible from self-play. We find nontrivial further gains from some domain-specific methods (...) We also find that a set of standard game-specific input features still significantly accelerates learning, showing that AlphaZero does not yet obsolete even simple additional tuning.

Finally, "it would obviously work so nobody tried" would make sense if it wasn't for the extremely competitive nature of machine learning research where every novel result is presented as a big breakthrough. Also, if something is obvious but never seems to make it to publication the chances are someone has tried and it didn't work as expected so they shelved the paper. We all know what happens to negative results in machine learning.

> but I am confused by its claim ... without knowing the game rules.

> probably some similar hard-coding of Atari actions

Nope, no hard coding.

Consider trying to MCTS on an Atari game. You have to "learn to predict" the pairs. Initially this guess is very bad, but eventually your predictions are good enough that rolling out a tree of predictions improves your action selection

For Go, and chess, we twist our self into NOT using the game rules in the simulator e.g. for each move, just indicate if GAME LOSS WIN

Whether this paper worthy of a new Nature hype cycle is a separate debate