Wolfram Alpha and ChatGPT

In the past, I have found Stephen Wolfram's air of superiority off-putting[a], but in this case I find myself nodding in agreement with every point he makes in the OP. I highly recommend you read it. This proposal, in particular, sounds like a great idea for improving ChatGPT in the near term:

> ...there’s the immediate opportunity of giving ChatGPT computational knowledge superpowers through Wolfram|Alpha. So it can not just produce “plausible human-like output”, but output that leverages the whole tower of computation and knowledge that’s encapsulated in Wolfram|Alpha and the Wolfram Language.

To anyone from OpenAI or Wolfram here: PLEASE DO THIS.

In many ways, what we're seeing is a modern-day rehash of the "classic AI"/"structured"/"symbolic" versus "deep learning"/"connectionist" approaches to AI, with people like Wolfram coming from the "classic AI"/"structured data"/"symbolic" tradition. For a good summary of both approaches from someone coming from the other tradition, read "The Bitter Lesson" by Rich Sutton: http://incompleteideas.net/IncIdeas/BitterLesson.html

There are AI researchers seeking to bridge the two approaches. Here's a recent example that seems significant to me: https://news.ycombinator.com/item?id=34108047 . See also this comment referencing Google's MuJoCo and LaMDA: https://news.ycombinator.com/item?id=34329847 elsewhere on this page. Maybe we will eventually find that the two approaches are actually not different, as people like Marvin Minsky contended?

[a] In my experience, Wolfram makes even Jürgen Schmidhuber seem humble by comparison, always claiming to have done or thought about new things before everyone else. AI researchers may occasionally get 'Schmidhubered,' but everyone who claims anything significant in math/physics/AI sooner or later gets 'Wolframed.'

"Also the Contact Us thing is a red flag for me."

Why is that a red flag and not something you see as potentially finding a custom solution to your need?

All of their products are like this. It's just a Wolfram thing and I doubt they'll ever change. It's a shame because I would genuinely love to use their products, but the upfront costs and insanely difficult licensing mean that most people never learn the software at home, companies refuse to use it internally, and nothing ever gets built.

> And, yes, one can imagine finding a way to “fix this particular bug”. But the point is that the fundamental idea of a generative-language-based AI system like ChatGPT just isn’t a good fit in situations where there are structured computational things to do. Put another way, it’d take “fixing” an almost infinite number of “bugs” to patch up what even an almost-infinitesimal corner of Wolfram|Alpha can achieve in its structured way.

I can see why Wolfram is bearish on what might be termed "the naive scaling hypothesis", i.e. that given more data, LLMs will naturally cease making false utterances by learning more systems of knowledge. If the naive scaling hypothesis is true, it recapitulates and invalidates a good chunk of the hand-coded work that he's built over the last decade or two. But I am not so sure; my money is on the robots for now.

For example, it should be really easy to generate training data for a LLM using more-formal systems like Wolfram Alpha; not least by having your LLM generate an arbitrary large list of "questions for Wolfram Alpha", then take that query and put it into WA, then attach the results to your LLM training set.

In other words, systems like Wolfram Alpha will be used to boost LLMs; the other way round is less obvious. Given the recent success, I'd put my money on "LLM can learn any structured system that can be wrapped in a text interface".

An example that's even more impressive than "wrap Wolfram Alpha" has already been demonstrated: LLMs plugged into Physics models (MuJoCo) at Google (https://arxiv.org/abs/2210.05359). There is (currently) no reason that these models can't be plugged in to learn any given simulator or oracle.

And on a more prosaic note, Google's LaMDA is clearly ahead of ChatGPT (it's just not public), and explicitly tackles the bullshit/falsehood problem by having a second layer that fact-checks the LLM by querying a fact database / knowledge-graph.

Of course, perhaps at some point before AGI the "naive scaling" approach will break down. It just seems to be a bad bet to be making right now; we are seeing no evidence of a slowdown in capabilities gains (quite the opposite, if anything).

Off-Topic but important, I wonder when they will fix the following bug:

--- This: ---

https://www.wolframalpha.com/input?i=2019-04-15+to+2022-01-0...

2 years 8 months 21 days

https://www.wolframalpha.com/input?i=2022-01-05+to+2019-04-1...

2 years 8 months 20 days

--- versus this: ---

https://www.wolframalpha.com/input?i=2019-01-09+to+2022-01-0...

2 years 11 months 27 days

https://www.wolframalpha.com/input?i=2022-01-05+to+2019-01-0...

2 years 11 months 27 days

--- Let's assume you would be using Wolfram Alpha or its backend for computing something related to a mission to Mars, worst case scenario people could die.