This time, it really does make sense
o3 and Grok 4 accidentally vindicate neurosymbolic AI
21–26 of 26 posts
Re: o3 and Grok 4 accidentally vindicate neurosymbolic AI
#22Earlier quoted context omitted.
It says a lot about the current discourse around AI that 6 years ago Marcus would write: > Despite all of the problems I have sketched, I don’t think that we need to abandon deep learning. And that would somehow be spun, today, as "LLMs are the wrong approach". Meanwhile, another attempt to post this article here got straight up flagged, I can only assume because this whole topic has become about religious orthodoxy…
Thanks for your reply; I can’t edit the original comment but I have updated my personal understanding of Marcus’ position.
have also updated
Re: o3 and Grok 4 accidentally vindicate neurosymbolic AI
#23As far as I can tell, putting the conspiratorial thinking aside, he's not really wrong, but I'm also not sure it matters that much. If neurosymbolic AI was "sidelined" in favor of "connectionist" pure NN scaling, I don't think it was part of a conspiracy or deeply embedded ideological bias. I mean, maybe that's the case, but it seems far more likely to me that pure deep learning scaling just provided a more increment…
That resonates. There _are_ a lot of good approximation functions can be developed from deep learning and good data and now RL on top. But then, we really do need symbolism, and now we need to somehow combine them. And it'll be different for text vs vision... Lots of ...s ahead
Re: o3 and Grok 4 accidentally vindicate neurosymbolic AI
#24As far as I can tell, putting the conspiratorial thinking aside, he's not really wrong, but I'm also not sure it matters that much. If neurosymbolic AI was "sidelined" in favor of "connectionist" pure NN scaling, I don't think it was part of a conspiracy or deeply embedded ideological bias. I mean, maybe that's the case, but it seems far more likely to me that pure deep learning scaling just provided a more increment…
That resonates. There _are_ a lot of good approximation functions can be developed from deep learning and good data and now RL on top. But then, we really do need symbolism, and now we need to somehow combine them. And it'll be different for text vs vision... Lots of ...s ahead
But how to even combine them. Is it only via another AGENT who has a symbolism tool. and if that (group of agents) cant extract multiple symbolisms from the context, of one which best fits, from the current approximation (context) then..
Re: o3 and Grok 4 accidentally vindicate neurosymbolic AI
#25It's a very funny read. "See, LLMs that are allowed to use Python perform better than ones that aren't, and Python is symbolic, so I was right all along!" Looks like a surrender to me.
Surrender by whom? Isn't his argument that leaders in the AI/ML space have consistently dismissed the need for that the entire point of the article? And that seems like a valid question to be after reading it. And huge financial implications for the industry.
That's a rather silly strawman.
OpenAI demonstrated Codex - a version of GPT-3 which could write Python and JS code - in 2021, only a year after the first GPT-3 release.
Here's a demo of Codex doing MS Word tasks using Python code: https://www.youtube.com/watch?v=-Dpl2awseZU
Live coding demo: https://www.youtube.com/watch?v=SGUCcjHTmGY
There's also a demo of doing some data processing using Codex.
The idea of using LLM to write code is rater obvious, and many people talked about it around GPT-3 or even GPT-2 release. People know that many logic tasks require search and it's rather silly to use LLM to do that. If it can generate any text it can generate Prolog code or a specification for SAT solver, and then dedicated efficient tool can handle the computation.
Gary Marcus is so detached from actual research that he might have genuinely missed the 2021 version of Codex. So he might truly believe that they added Python interpreter "quietly", and it's not the big announcement of 2021.
This guy is known as a clown in the industry, and for a very good reason... If he was an actual researcher he would have jumped on the opportunity to make an actual neurosymbolic demos when first language models were released - you don't actually need to wait GPT-3 to do that. But he preferred to relish in writing books claiming that the entire industry is wrong instead of actually building something.
Re: o3 and Grok 4 accidentally vindicate neurosymbolic AI
#26It's a very funny read. "See, LLMs that are allowed to use Python perform better than ones that aren't, and Python is symbolic, so I was right all along!" Looks like a surrender to me.
I'm not really a follower of low level implementation of AIs. Why would python tool use(among others) not qualify as symbolic? I don't think it's under question that tool use vastly improves LLMs.
I'd say it's just not a part of the machine-that-is-learning; I don't consider myself to be symbolic, even though I also use python