Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI Expert Roadmap

i.am.ai

Post reply on HN
Print view View original

71–80 of 92 posts

Re: AI Expert Roadmap

#71

Post by Veedrac »

YeGoblynQueenne wrote: post #69
Veedrac wrote: post #66

Earlier quoted context omitted.

Because GOFAI just observably doesn't work. The ideas are brittle, can't generalize and abstract the way is needed, has made very little progress recently (if any) an AI context, and you just _don't see_ anything that would argue otherwise. In contrast, ML methods do work, observably and clearly, and they work in a ridiculously general way, to a degree larger than almost anyone thought (or even thinks) is reasonable…

>> If this doesn't answer your question, perhaps answer the opposite; how do you know that it's wrong? I know the literature. It's my job. >> And it's not just my opinion; there's a reason AI conference attendance has shot up a factor of 10 or so in the last few years, why NeurIPS is the leading one (and even historically GOFAI conferences are majority NNs), why the big AI labs with big AI cash are all doing NNs, and…

> For instance, you claimed that "GOFAI" successes just "aren't there" but I listed a few, like Deep Blue or MYCIN - and you didn't seem to have heard of these before (I'm more surprised about not knowning of Deep Blue than MYCIN).

At this point I think we're just hopelessly talking past each other. Of course I know about Deep Blue. I didn't know about MYCIN, but, like, “MYCIN was never actually used in practice”, so I don't feel particularly bad about missing that one.

But neither of those challenge my point. If you want to go back in time 30 years, then sure, if you want to be an AI expert, then you have to know GOFAI. That's what the ‘OF’ stands for.

> I know the literature. It's my job.

Yah I read the literature too. (Albeit it seems a very different subset.) That's not an argument though.

> Or, you know, ask any AI researcher :)

OpenAI is explicitly about the path to AGI, https://openai.com/about/.

DeepMind was also founded to tackle AGI (no source, sorry).

Geoffrey Hinton thinks NNs will get to AGI https://www.technologyreview.com/2020/11/03/1011616/ai-godfa....

Even in your own link, Yoshua Bengio is saying that this is a path to AGI, it's just not there yet.

> Which "GOFAI" conferences are majority NNs?

I said “historically GOFAI conferences”, so eg. AAAI.

Re: AI Expert Roadmap

#72

Post by YeGoblynQueenne »

Veedrac wrote: post #70
YeGoblynQueenne wrote: post #68

Earlier quoted context omitted.

>> Heck, I don't think it even got `shorter` right; it gave shorter(A,B):-llength(A,C),llength(B,D),s(C,D). >> which means len(A) + 1 == len(B), not len(A) Oops. Haha well spotted @^_^ This is correct for ?- list_mil_problem([llength/2,shorter/2,ordered/3]). Positive examples ----------------- llength([],0). llength([a],s(0)). shorter([a],[b,c]). shorter([1,2],[3,4,5]). ordered([a],[b,c],[d,e,f]). Negative examples -…

I don't think you've understood my point. At every step of the way you've put in more effort than writing the program and specifying the examples. Clearly this is not trivial. If you're actually inferring programs and not just doing a sort of guided line-by-line generalization of a program you had already written, the only things you would need would be Positive examples ----------------- ordered([a],[b,c],[d,e,f]).…

Aren't you shifting the goalposts a bit? First you ask me how I'd solve this problem with ILP:

“Given three strings n1, n2, and n3, return true if length(n1) I say this is a trivial problem to solve with ILP and I show you a, well, trivial solution and you complain that it's - trivial.

Then I show you a more elaborate version that learns sub-programs on the way to the full solution and you say that a) it doesn't solve a different problem, with ≤ instead of And now you're saying you want a solution that learns from examples only. You would have saved us both a lot of time had you clarified your expectations up front.

No matter. There isn't anything that can do what you ask. Or rather, there are many approaches that could learn (a) just from examples, with a brute-force search. But there is no approach that could learn arbitrary programs only from examples. The reason is that the space of all programs that can be computed by a Universal Turing Machine ("arbitrary") is infinite and any learner trying to find one of them blindly, without some kind of hint to guide it, would be lost for ever inside it.

Most machine learning approaches that learn programs from examples adopt some sort of inductive bias to guide a search for a program that satisfies some set of goodness criteria, including neural approaches [1]. In ILP, inductive bias consists primarily of BK and language bias (like the metarules in Louise). ILP has a certain advantage in this, in that the languages of examples, bias and hypotheses are the same (some first order logic language, like Prolog or ASP) and so ILP systems can learn their own bias, like Louise can learn its own BK and metarules. By way of comparison, neural nets, with their hand-crafted architectures, minutely fine-tuned to specific domains or even particular datasets, cannot do that (e.g. a trained model can't be used as a feature to another neural net, in the way that ILP hypotheses can be used as BK). Of course you need to start somewhere, from obvious primitives like head/2, tail/2, s/2 and p/2 that I used above.

But I digress. The bottom line is that learning arbitrary programs from examples is a hard problem for any machine learning approach [2]. Classification is a piece of cake, by comparison. And that is why there has been so little progress in this problem even after decades of research [3].

The take home message of course is that neural nets are not the end of the line in AI research and it would be disastrous for the progress of the field to allow research into neural nets to eclipse every other approach. If this happens it will all have to be discovered again, from scratch. And in another 70 years.

____________________

[1] e.g. see Learning explanatory rules from noisy data https://arxiv.org/abs/1711.04574 by DeepMind, which also uses metarules.

[2] See for example:

Deep Learning for Program Synthesis

Synthesizing a program from a specification has been a long-standing challenge.

(...)

This problem is extremely challenging, and the complexity of the synthesized programs by existing approaches is still limited.

https://sunblaze-ucb.github.io/program-synthesis/index.html

[3] This is where I'd normally say that there have been recent breakthroughs that promise to overturn years of slow progress, but that's a story for another time (and another venue most like).

Re: AI Expert Roadmap

#73

Post by YeGoblynQueenne »

Veedrac wrote: post #71
YeGoblynQueenne wrote: post #69

Earlier quoted context omitted.

>> If this doesn't answer your question, perhaps answer the opposite; how do you know that it's wrong? I know the literature. It's my job. >> And it's not just my opinion; there's a reason AI conference attendance has shot up a factor of 10 or so in the last few years, why NeurIPS is the leading one (and even historically GOFAI conferences are majority NNs), why the big AI labs with big AI cash are all doing NNs, and…

> For instance, you claimed that "GOFAI" successes just "aren't there" but I listed a few, like Deep Blue or MYCIN - and you didn't seem to have heard of these before (I'm more surprised about not knowning of Deep Blue than MYCIN). At this point I think we're just hopelessly talking past each other. Of course I know about Deep Blue. I didn't know about MYCIN, but, like, “MYCIN was never actually used in practice”, so…

But none of those sources says that e.g. search or planning are not AI fields. That was your original claim, if I'm not mistaken? Anyway it doesn't matter. It's a very strange thing to say and I was just trying to understand what made you say it- strictly out of curiousity.

I too can quote Hinton -from memory and without a link. I remember him saying that the next big thing in AI will come from a grad student who distrusts everything he (Hinton) has ever said. Unfortunately, I won't be that grad student- I haven't heard everything that Hinton has ever said.

Re: AI Expert Roadmap

#74

Post by Veedrac »

YeGoblynQueenne wrote: post #73
Veedrac wrote: post #71

Earlier quoted context omitted.

> For instance, you claimed that "GOFAI" successes just "aren't there" but I listed a few, like Deep Blue or MYCIN - and you didn't seem to have heard of these before (I'm more surprised about not knowning of Deep Blue than MYCIN). At this point I think we're just hopelessly talking past each other. Of course I know about Deep Blue. I didn't know about MYCIN, but, like, “MYCIN was never actually used in practice”, so…

But none of those sources says that e.g. search or planning are not AI fields. That was your original claim, if I'm not mistaken? Anyway it doesn't matter. It's a very strange thing to say and I was just trying to understand what made you say it- strictly out of curiousity. I too can quote Hinton -from memory and without a link. I remember him saying that the next big thing in AI will come from a grad student who dis…

I best summarized my claim when I said the following. Whether or not it's an ‘AI field’ is not very interesting to me, as long as the following holds.

---

Like, my point is not about whether you can find the odd person trying to solve intelligence with grammars, or what were GOFAI conferences still harbour GOFAI research in the corners, my point is that a) these approaches don't work as a way to actually tackle AI, the problem, b) the vast majority of the field does not take them as seriously as a method of doing so, regardless of other uses, and c) therefore it's natural, not ‘impossible’, to gain AI expertise without having much care for those parts of the field.

Re: AI Expert Roadmap

#75

Post by Veedrac »

YeGoblynQueenne wrote: post #72
Veedrac wrote: post #70

Earlier quoted context omitted.

I don't think you've understood my point. At every step of the way you've put in more effort than writing the program and specifying the examples. Clearly this is not trivial. If you're actually inferring programs and not just doing a sort of guided line-by-line generalization of a program you had already written, the only things you would need would be Positive examples ----------------- ordered([a],[b,c],[d,e,f]).…

Aren't you shifting the goalposts a bit? First you ask me how I'd solve this problem with ILP: “Given three strings n1, n2, and n3, return true if length(n1) I say this is a trivial problem to solve with ILP and I show you a, well, trivial solution and you complain that it's - trivial. Then I show you a more elaborate version that learns sub-programs on the way to the full solution and you say that a) it doesn't solv…

> Aren't you shifting the goalposts a bit?

No, I don't think so. My original comment was

“I have looked at the program synthesis literature before and it really does not seem very advanced to me. The General Program Synthesis Benchmark Suite lists unsolved benchmarks like “Given three strings n1, n2, and n3, return true if length(n1) I only asked about ILP to clarify your defense. I maintain, after seeing the ILP you were referring to, that the defence doesn't meaningfully affect my point, that the problem you claim to be solving is not the one I was talking about, and not meaningfully more interesting.

> The reason is that the space of all programs that can be computed by a Universal Turing Machine ("arbitrary") is infinite and any learner trying to find one of them blindly, without some kind of hint to guide it, would be lost for ever inside it.

I'm not looking to solve arbitrary programs. I'm looking, at least at this first hurdle, to solve a few largely trivial ones.

If you saw 100 examples, you would be able to solve this problem, or at least get very close. Therefore this sort of theoretical argument cannot hold.

> By way of comparison, neural nets, with their hand-crafted architectures, minutely fine-tuned to specific domains or even particular datasets, cannot do that (e.g. a trained model can't be used as a feature to another neural net, in the way that ILP hypotheses can be used as BK).

I assume you've seen GPT-f? (https://arxiv.org/abs/2009.03393)

Section 4.7.1 shows this is entirely feasible for neural networks, though the technique is very different.

Though I prefer even more the elegance (and general absurdity) of learning the bias by pretraining on WebMath (GitHub, arXiv Math, Math StackExchange).

> The bottom line is that learning arbitrary programs from examples is a hard problem

I agree it's hard, but in my mind this sentence completes to “and therefore our only hope is to use most powerful tool we have available, neural networks.”

Re: AI Expert Roadmap

#76

Post by YeGoblynQueenne »

Veedrac wrote: post #75
YeGoblynQueenne wrote: post #72

Earlier quoted context omitted.

Aren't you shifting the goalposts a bit? First you ask me how I'd solve this problem with ILP: “Given three strings n1, n2, and n3, return true if length(n1) I say this is a trivial problem to solve with ILP and I show you a, well, trivial solution and you complain that it's - trivial. Then I show you a more elaborate version that learns sub-programs on the way to the full solution and you say that a) it doesn't solv…

> Aren't you shifting the goalposts a bit? No, I don't think so. My original comment was “I have looked at the program synthesis literature before and it really does not seem very advanced to me. The General Program Synthesis Benchmark Suite lists unsolved benchmarks like “Given three strings n1, n2, and n3, return true if length(n1) I only asked about ILP to clarify your defense. I maintain, after seeing the ILP you…

>> I only asked about ILP to clarify your defense. I maintain, after seeing the ILP you were referring to, that the defence doesn't meaningfully affect my point, that the problem you claim to be solving is not the one I was talking about, and not meaningfully more interesting.

I must admit I'm very confused by this. I really thought you were talking about the problem of ordering three strings by length. What problem where you talking about, if not that?

Re: AI Expert Roadmap

#77

Post by Veedrac »

YeGoblynQueenne wrote: post #76
Veedrac wrote: post #75

Earlier quoted context omitted.

> Aren't you shifting the goalposts a bit? No, I don't think so. My original comment was “I have looked at the program synthesis literature before and it really does not seem very advanced to me. The General Program Synthesis Benchmark Suite lists unsolved benchmarks like “Given three strings n1, n2, and n3, return true if length(n1) I only asked about ILP to clarify your defense. I maintain, after seeing the ILP you…

>> I only asked about ILP to clarify your defense. I maintain, after seeing the ILP you were referring to, that the defence doesn't meaningfully affect my point, that the problem you claim to be solving is not the one I was talking about, and not meaningfully more interesting. I must admit I'm very confused by this. I really thought you were talking about the problem of ordering three strings by length. What problem…

The problem from the benchmark is automatically learning the function, to check three strings are ordered by length, from examples.

What you are doing is categorically different, because you are manually guiding the search process by telling it which interim lines to generate. This ignores all the interesting parts of the challenge.

Re: AI Expert Roadmap

#78

Post by [deleted] »

Veedrac wrote: post #77
YeGoblynQueenne wrote: post #76

Earlier quoted context omitted.

>> I only asked about ILP to clarify your defense. I maintain, after seeing the ILP you were referring to, that the defence doesn't meaningfully affect my point, that the problem you claim to be solving is not the one I was talking about, and not meaningfully more interesting. I must admit I'm very confused by this. I really thought you were talking about the problem of ordering three strings by length. What problem…

The problem from the benchmark is automatically learning the function, to check three strings are ordered by length, from examples . What you are doing is categorically different, because you are manually guiding the search process by telling it which interim lines to generate. This ignores all the interesting parts of the challenge.

[deleted]

Re: AI Expert Roadmap

#79

Post by YeGoblynQueenne »

Veedrac wrote: post #77
YeGoblynQueenne wrote: post #76

Earlier quoted context omitted.

>> I only asked about ILP to clarify your defense. I maintain, after seeing the ILP you were referring to, that the defence doesn't meaningfully affect my point, that the problem you claim to be solving is not the one I was talking about, and not meaningfully more interesting. I must admit I'm very confused by this. I really thought you were talking about the problem of ordering three strings by length. What problem…

The problem from the benchmark is automatically learning the function, to check three strings are ordered by length, from examples . What you are doing is categorically different, because you are manually guiding the search process by telling it which interim lines to generate. This ignores all the interesting parts of the challenge.

Ah, I see what you mean. You mean that I explicitly told Louise how to compose each program from its sub-programs. In truth, I did not. In the incremental learning problems for ordered/3 and ordered_leq/3 I gave Louise three learning targets and a few primitives from which to construct them. I specifically and very deliberately did not tell it to use each target to define another. It figured it out on its own.

For instance, I did not direct Louise to define shorter/2 by means of length/2. In order to do this I would have to specify length/2 as BK for shorter/2, but I didn't. Note the BK declarations in the experiment file I put on pastebin:

  background_knowledge(llength/2, [tail/2,p/2]).
  background_knowledge(shorter/2, [s/2]).
  % ground_peano/1 is added here so it's reported by list_mil_problem/1
  background_knowledge(ordered/3, [ground_peano/1]).
  background_knowledge(ordered_leq/3, [leq/2,ground_peano/1]).
Remember how in an erarlier comment I pointed out that the definition of shorter/2 changed to use leq/2, instead of s/2, when the examples of ordered_leq/3 where in the BK? Again, Louise figured that on its own.

In fact, this ability of Louise (actually, its learning procedure, Top Program Construction, or TPC) is kind of native, in the sense that TPC was originally conceived as an algorithm to select relevant background knowledge for a different learning system ("Thelma" for "Theory Learning Machine"; hence "Louise"). So it can figure out what BK it needs on its own. Automatic selection of relevant background knowledge in ILP was the original subject of my PhD research. Interestingly, it turns out that once we had a system that could perform this task, it could also learn its own programs.

Re: AI Expert Roadmap

#80

Post by Veedrac »

YeGoblynQueenne wrote: post #79
Veedrac wrote: post #77

Earlier quoted context omitted.

The problem from the benchmark is automatically learning the function, to check three strings are ordered by length, from examples . What you are doing is categorically different, because you are manually guiding the search process by telling it which interim lines to generate. This ignores all the interesting parts of the challenge.

Ah, I see what you mean. You mean that I explicitly told Louise how to compose each program from its sub-programs. In truth, I did not. In the incremental learning problems for ordered/3 and ordered_leq/3 I gave Louise three learning targets and a few primitives from which to construct them. I specifically and very deliberately did not tell it to use each target to define another. It figured it out on its own. For in…

> You mean that I explicitly told Louise how to compose each program from its sub-programs. In truth, I did not. In the incremental learning problems for ordered/3 and ordered_leq/3 I gave Louise three learning targets and a few primitives from which to construct them. I specifically and very deliberately did not tell it to use each target to define another. It figured it out on its own.

No, I get this, it's just not really more than a surface level pretense of choice. The hard thing about generating programs is that there are many possible programs; something like m^n, where ’m’ is the number of functions you have available to use (say, ~1000) and `n` is the number of steps the program needs to take (say, ~5 in this case), and there's another factor for where to put the parameters which here is low enough to be mostly negligible. It turns out even 1000^5 is really big, so this problem is hard if you don't do it smart.

The ‘choice’ you are offering Louise is something like, idk, m=5, n=2. 5^2 is not big. You argue about the terminology, but whatever you call it, it's still exponentially less interesting than the problem first posed. All the interesting work has been done for it, by you.

With the way you first laid out the question, there's a good chance (>1%) I could have gotten the answer mostly right (up to parameter order) without looking at the examples, just the background knowledge and the target type.

Post reply on HN