AI Expert Roadmap

It doesn't much matter what people wrongly thought in the past, any more than one has to study the four fundamental elements of water, air, earth and fire to become an expert in chemistry. GOFAI didn't work, hence the winter. If it did I'd think otherwise.

It's true that you can, in some places, merge the two. The significant majority of the time, this just makes your ML system worse in the long run, per The Bitter Lesson. Occasionally—very occasionally—your problem is fundamentally simple, so something brittle like AlphaZero works, even though we'd still rather shave the fixed-function parts off with MuZero and the like. But it's no coincidence this is reserved for simple, thoughtless problems (like brute-force move search) and kept isolated from the thinking, as intelligence needs generalization and abstraction, and GOFAI doesn't generalize or abstract.

> And of course, one of the most widely used and well-known type of machine learning approach used to this day, decision trees

Though decision trees live a healthy life in data analysis, alongside things like k-means clustering, they're obviously not AI.

> Of course, most people today know "ML" as a byword for deep learning, or at best statistical pattern recognition (if that).

‘Statistical pattern recognition’ is just name calling devoid of real criticisms. I can ask GPT-3,

> a = ["fitness", "health", "heart"], b = ["lifting", "curls", "squats"], c = ["running", "jogging"], so what is b.append("pushups")?

and it'll happily answer

> b.append("pushups") returns ["lifting", "curls", "squats", "pushups"]

or I can give it the Loebner Prize questions which it can almost ace, or I can test it on 10-digit addition and it gets 60% right (with thousands separators, but still suffering BPEs), or I can ask it to do quirky tasks like shuffling letters, or ask it to write sentences with novel words and find that it understands nuanced differences between the definitions...

You can scream ‘but it's just statistical pattern recognition’ all you want, but the criticism doesn't mean anything if ‘statistical pattern recognition’ includes this level of generalization, reasoning and algorithmic sophistication in natural language with scaling curves that assure us the best is yet to come.

While waiting for Veedrac's reply to my comment, I thought I'd clarify here what I meant by "statistical pattern recognition". "Pattern recognition" is the name of the sub-field of AI research from which statistical machine learning grew into what it is today, for example many problems in machine vision are typically considered as pattern recognition problems, etc. "Statistical" refers to the methods used to achieve the task, e.g. neural networks are normally filed under "statistical AI" (for historical reasons).

Like I say in my previous comment, modern machine learning research started as a discipline that was separate from pattern recognition (and Pattern Recognition was sometimes considered distinct to AI, as a research subject). I quote below from Tom Mitchell's wildly influential paper, "Generalisation as Search" (AI 18, 1982):

"5.2. Statistical pattern recognition

The field of statistical pattern recognition deals with one important subclass of generalization problems. In this subclass, the instances are represented by points in n-space, and the generalizations are represented by decision surfaces in n-space (e.g. hyperplanes, polynomials of specified degree). The matching predicate corresponds to determining whether a given point (instance) lies on one side or another of a given decision surface (generalization). The field of Statistical Pattern Recognition has developed very good generalization methods for particular classes of decision surfaces. Many of these methods are relatively insensitive to errors in the data and some have well understood statistical convergence properties, under certain assumptions about the probability distribution of input instances."

"In contrast to work in Statistical Pattern Recognition, work on the generalization problem within Artificial Intelligence has focused on problems involving a different class of instance and generalization languages. These languages are incompatible with numerically oriented representations that describe objects as feature vectors in n-space. For example, Winston's program [21] for learning descriptions of simple block structures such as arches and towers, represents instance block structures in terms of their component blocks and relationships among these. In this domain the natural representation for instances is a generalized graph rather than a feature vector. (...)

https://www.sciencedirect.com/science/article/abs/pii/000437...

Pattern recognition these days is dominated by neural methods, e.g. CNNs for object classification etc. and so is machine learning so it kind of makes sense that the three terms are used interchangeably, but unfortunately many people are not aware of the historical context of the terms, hence misunderstandings as the one by Veedrac above, regarding my comment that machine learning has become a byword for statistical pattern recognition: it's not because I'm dismissive of statistical pattern recognition, or of the ability of deep learning systems to perform it; it's because the terms have really become interchangeable and I think even among researchers.

In any case, before commenting on a complex subject of research with a long history, my recommendation remains to first become well acquainted with the subject and its history. Otherwise, one runs the risk of appearing confused.

Edit: Note that Mitchell's use of "generalisation" in the excerpt above does not refer to the ability of a model to generalise to test, or unseen, data. Rather, "generalisation" in Mitchell's paper refers to the ability of a system "to take into account a large number of specific observations, then to extract and retain the important common features that characterize classes of these observations." Mitchell's paper describes a theory of search guided by generalisation relations between hypotheses and observations.

>> Sorry, I did misunderstand what you were saying.

Thank you for your honesty and there's no need to apologise.

Let me explain my point above, if I can. You said that search, planning, expert systems, etc, are "not AI, and they never will be". I understand that as saying that such systems are not artificial intelligences, in the sense of an intelligent machine created by humans out of whole cloth (without trying to define what "intelligent" means).

That is certainly true, but it is also uncontroversial that the above are sub-fields of the field of research that is known as "AI". That is, there is a field that researches approaches that could lead to the creation of intelligent machines and that is called "AI" and then there's the ultimate target of that field which is to create "AI".

My original comment bemoans the fact that in some sectors, "AI", as a field of research, has become synonymous with only one sub-sub-field of AI research, that is, deep learning.

Contrary to what you state, these "GOFAI" fields (symbolic AI, if you will) are still active and far from having "failed" in any way, they are "SOTA" in their respective tasks. For example, the field of automated theorem proving is dominated by systems that employ the resolution rule, a logic-based approach and while recent efforts have been made to make inroads into theorem proving with deep neural nets (e.g. a recent attempt to use transformers) results are still very far from being comparable to the traditional approaches. I know more about automated theorem proving that I know e.g. about planning or search (because my PhD is in a subject close to the former) but my understanding is that in those fields too, traditional techniques dominate- which is why research in them is still active.

If I am permitted to tout my own horn, my broader subject area can be described as "program learning", i.e. machine learning of arbitrary programs from examples of their inputs and outputs. In this area also, deep learning systems are hopelessly outclassed by symbolic approaches, not least because these approaches learn precise representations of target programs (rather than approximations) from a mere handful of examples (four or five, etc).

And so on. These are just some examples of AI research that is ongoing, that is not deep learning, and that is state of the art for its respective tasks. In view of that, I consider that using "AI" to mean "deep learning" (as the article above does) is not only misinformed but also misinforming and actively harmful. In the sense of harming people's understanding, that is.

As to your comment about how GOFAI "failed", I'm afraid this opinion, which is common, is also misinformed. Here, too, a knowledge of the history of the field helps, but to summarise, the last winter happened because of strictly political reasons and for no reason that had anything to do with the scientific merits, or practical successes of the relevant approaches. In fact, expert systems, now widely considered a failure, were one of the first big success stories of AI; a success story that was cut short only because, again, of political reasons.

I could talk about the AI winter subject for hours (it's a favourit subject of mine) but a good starting point is this article by the editor of the IEEE journal Intelligent Systems: (Avoiding Another Winter) https://www.computer.org/csdl/magazine/ex/2008/02/mex2008020.... The wikipedia page on https://en.wikipedia.org/wiki/AI_winter also has a decent summary and some sources. Finally, see the wikipedia article on the Lighthill Report https://en.wikipedia.org/wiki/Lighthill_report which is more relevant to AI research in the UK (the Report killed AI research in the UK, dead) and this review of the report by John McCarthy: http://www-formal.stanford.edu/jmc/reviews/lighthill/lighthi... (man who coined the term AI and also created LISP on the side).

>> As much as you argue I should adopt the subject's history, I'm saying you should adopt its present.

More to the point, I'm recommending that you should know a subject's history before forming a strong opinion about its present and future. As to me, I'm firmly rooted to the present. About half of the literature I read is neural networks- and that's not even the subject of my research. But if you think about it, in the era where the trend is to use deep learning, the most interesting results can only come from not using deep learning. In a gold rush, if everyone is digging up Widow's Creek, then Widow's Creek is the last place to dig for gold.

I'm happy to see we're still in healthy disagreement. However, I have to apologise for confusing you by describing my field as "program learning" which is admittedly vague, but I didn't want to go into the particulars. My field is not program synthesis, which is constructing programs from complete, formal specifications. Rather, it's Inductive Programming and more specifically Inductive Logic Programming (ILP), which is learning programs from examples, i.e. "incomplete specifications". I'm not familiar with the General Program Synthesis Benchmark Suite, but the problem you list (test three strings are ordered by length) is trivial for ILP approaches. Again, I don't want to point to my own research, directly (I'm going through a bit of a modesty phase) (oh, alright, it's just that the documentation of my project is crap). However, I have to say that even so, if something is a difficult problem for program synthesis approaches, then it's very unlikely that neural networks will do any better at it. For instance, do you know how well deep neural nets perform on this benchmark? I can't find any results with a quick search.

You make the point that one does not need to learn these "obsolete" AI approaches because they are not relevant anymore. I don't understand why you say that. These approaches are still state of the art for their respective tasks and there is no other approach that has been shown to do any better, including deep neural networks. In what sense are they "no longer more than briefly and tangentially relevant" as you say?

Regarding the gold rush, the point of the analogy is that in a gold rush only a very few people will ever strike gold. This is exactly the state of research into deep learning currently. After a few initial big breakthroughs, like CNNs and LSTMs, progress has stalled and the vast, vast majority of published papers (or papers put on arxiv permanently) present incremental results, if that. Literally thousands of deep learning papers are published each month and the chance to have an impact is miniscule. From my point of view, as a researcher, going into deep learning right now would be career suicide. Not to mention that, while the first few successes were achieved by small academic teams, who had typical academic motives (er, glory), the game has now passed to the hands of big corporate teams that have quite different incentives, so it's almost impossible for small teams or individual researchers to make a dent.

As to the winter and whether GOFAI works, perhaps I haven't convinced you with my sources, but in that case, I have to go back to my earlier question and ask where your knowledge comes from. You clearly have a strong opinion on GOFAI and the AI winter of the '80s, but what knowledge does this opinion come from? Can you say? And if this sounds like a challenge, well, that's because it is. I'm challenging you to re-examine the basis of your convictions, if you like. Because to me, they sound like they are not well-founded and that you should put some water in your wine. The things you say "don't work", work and the things you say work, don't work as well as you say.

For my part, I certainly agree that GPT-3 or the next iteration of a large transformer-built language model can be a useful tool, but such a tool will always be limited by the fact that it's, well, a language model, and it can only do what language models do, which does not include e.g. the ability for reasoning (despite big claims to the contrary) or arithmetic (ditto) or generation of novel programs. For instance, the append() example you show above is clearly memorised: you haven't given the model any examples of append(), so it can't possibly learn its definition from examples. It only returns a correct result because it's seen the results of append() before. Not the same result, but close enough. Like I say, this ability can definitely be useful- but its usefulness is limited compared to the ability to learn arbitrary programs, never before seen.

btw, why do you need to give it the list "a"? What happens if this is ommitted from the prompt?