When ChatGPT broke the field of NLP: An oral history

Neural nets are quite old, and everyone knew that they were universal function approximators. The reason why models never took off was because it was very expensive to train a model even of a limited size. There was no real available hardware to do this on short of supercomputer clusters, which were just all cpus, and thus wildly inefficient. But any researcher back then would have told you that you can figure anything out with neural nets.

Sometime in 2006, Nvidia realized that a lot of the graphics compute was just generic parallel compute and released Cuda. People started using graphics cards for compute. Then someone figured out you can actually train deep neural nets with decent speed.

Transformers wasn't even that big of a leap. The paper makes it sound like its some sort of novel architecture - in essence, instead of inputweights to next layer, you do inputmatrix1, inputmatrix2, inputmatrix3, and multiply them together. And as you guessed this, to train it you need more hardware because now you have to train 3 matrices rather than just one.

If we ever get like ASIC for ml, basically at a certain point, we will be able to iterate on architectures itself. The optimal LLM may be a combination of CNN,RNN, and Transformer blocks, all interwtined.

I mean he did use the state of the art for his work, he just checked how much better it actually was in comparison to a much simpler algorithm and thought the cost/benefit ratio to be questionable... At least that's what I read from his comments

Do LLMS think? Of course they do. But thinking =/= intelligence.

Its pretty easy to define what an AI actually would look like:

A human coder sits down and writes an algorithm. In that algorithm, there is no reference to any specific piece of information on ANYTHING (including human words), whether its manually written in code or derived through training a neural net on that information and the code is just a bunch of matrix multiplies.

The algorithm has 2 interfaces - a terminal for a human to interact with, and an api to a tcp socket over which it can communicate to the world wide web.

A human could give this algorithm an instruction, like for example, "Design and build me a flying car and put it in my driveway and do not spend a single cent of my money, and do everything legally".

Provided there are no limits on communication that would result in the algorithm being perma banned of the internet, the algorithm prior to even tackling the task at hand will have to do the following at the least:

- figure out how to properly structure HTTP communication to be able to talk to servers, and essentially build an internal API.

- figure out what the words you typed mean - i.e map them to information collected from the web and

- start running internal simulations to figure out what the best course of action is

- figure out how to deal with ambiguity and ask you questions (like "how far do you want to fly"), and figure out how to deal with dead ends.

- start executing actions with preplanned risk (figuring out what risk is in the process) and learn from mistakes.

And that's just the short start.

But the key factor is that this same process that it uses to figure basic functionality is the same process (at least on the lowest level) that it would use to start designing a flying car once it has all the information it needs to "understand" the task.

And there isn't anything even remotely close on the horizon with any of the current AI research that indicates that we have any idea what that process looks like. The only claims that we can make is that its definitely recursive, not fully forward like LLMs, and its essentially a search algorithm. But what its searching and what the guidance metric is for search direction is the mystery.

The major part of tradidional NLP was about formal representations. We are still to see the efficient mining techniques to extract the formal representations and analyses back from LLM.

How would we solve the traditional NLP problems, such as, for example, formalization of law corpus of a given country with LLM?

As an approximation we can look at non-natural language processing, e.g. compiler technologies. How do we write an optimizing compiler on LLM technologies? How do we ensure stability, correctness and price?

In a sence, the traditional NLP field has just doubled, not died. In addition to humans as language capable entities, who can not really explain how they use the language, we now also have LLM as another kind of language capable entities. Who in fact also can not explain anything. The only benefit is that it is cheaper to ask LLM the same question a million of times that a human.