Facebook LLAMA is being openly distributed via torrents

My favorite article, which I post any time I have an excuse to, mentions Eliza.

https://www.bbc.co.uk/blogs/adamcurtis/entries/78691781-c9b7...

Eliza Excerpt.

The key to why this happened lies in an odd experiment carried out in a computer laboratory in California in 1966.

A computer scientist called Joseph Weizenbaum was researching Artificial Intelligence. The idea was that computers could be taught to think - and become like human beings. Here is a picture of Mr Weizenbaum.

There were lots of enthusiasts in the Artificial Intelligence world at that time. They dreamt about creating a new kind of techno-human hybrid world - where computers could interact with human beings and respond to their needs and desires.

Weizenbaum though was sceptical about this. And in 1966 he built an intelligent computer system that he called ELIZA. It was, he said, a computer psychotherapist who could listen to your feelings and respond - just as a therapist did.

But what he did was model ELIZA on a real psychotherapist called Carl Rogers who was famous for simply repeating back the the patient what they had just said. And that is what ELIZA did. You sat in front of a screen and typed in what you were feeling or thinking - and the programme simply repeated what you had written back to you - often in the form of a question.

Weizenbaum's aim was to parody the whole idea of AI - by showing the simplification of interaction that was necessary for a machine to "think". But when he started to let people use ELIZA he discovered something very strange that he had not predicted at all.

Here is a bit from a documentary where Weizenbaum describes what happened. (video in article)

Weizenbaum found his secretary was not unusual. He was stunned - he wrote - to discover that his students and others all became completely engrossed in the programme. They knew exactly how it worked - that really they were just talking to themselves. But they would sit there for hours telling the machine all about their lives and their inner feelings - sometimes revealing incredibly personal details.

His response was to get very gloomy about the whole idea of machines and people. Weizenbaum wrote a book in the 1970s that said that the only way you were going to get a world of thinking machines was not by making computers become like humans. Instead you would have to do the opposite - somehow persuade humans to simplify themselves, and become more like machines.

But others argued that, in the age of the self, what Weizenbaum had invented was a new kind of mirror for people to explore their inner world. A space where individuals could liberate themselves and explore their feelings without the patronising elitism and fallibility of traditional authority figures.

When a journalist asked a computer engineer what he thought about having therapy from a machine. He said in a way it was better because -

"after all, the computer doesn't burn out, look down on you, or try to have sex with you"

ELIZA became very popular and lots of researchers at MIT had it on their computers. One night a lecturer called Mr Bobrow left ELIZA running. The next morning the vice president of a sales firm who was working with MIT sat down at the computer. He thought he could use it to contact the lecturer at home - and he started to type into it.

In reality he was talking to Eliza - but he didn't realise it.

This is the conversation that followed. (photograph of conversation)

But, of course, ELIZA didn't ring him. The Vice President sat there fuming - and then decided to ring the lecturer himself. And this is the response he got:

Vice President - “Why are you being so snotty to me?”

Mr Bobrow - “What do you mean I am being snotty to you?”

Out of ELIZA and lots of other programmes like it came an idea. That computers could monitor what human beings did and said - and then analyse that data intelligently. If they did this they could respond by predicting what that human being should then do, or what they might want.

Can "something simpler", for example, code correct function bodies from comments describing functions in natural language? I think people are too quick to dismiss the power of these models.

It would be interesting to see an analysis of this. I see your point - otoh is there a reason to believe that more computation is being "banked" than say matrix inversion, or other optimizations that aren't gradient descent based?

The large datasets involved let us usefully (for some value of useful) bank lots of compute, but it's not obvious to me that it's done particularly efficiently compared to other things you might precompute.

For converged model training, training is often quite inefficient because the weight updates decay to zero and most epochs are having a very small individual effect. I think for e.g. stable diffusion, they dont train to anywhere near convergence so weight updates have a bigger average effect. Not sure if that applies to llms

It will say something nice to you for $3.50

What if it's an incremental upload? E.g. we start with some prosthetics and slowly migrate organic function to digital?

Is this the Ship of Theseus, or is it a slow but nonobvious death?