OpenAI researchers warned board of AI breakthrough ahead of CEO ouster

Looking at humanity, persuasion seems to be an extremely low bar! Also for a superhuman trait is it that it’s capable of persuading anyone anything or rather that it’s able to persuade everyone about something. Power vs. Reach.

This seems to refer to the GSM8K (grade school math) benchmark [1]. GPT-4 scores 0.92 on that one. A breakthrough could mean it gets all of them correct.

This would have major implications for long-term planning. For instance, if you have a sequence of 10 steps, each with a 90% success rate, the overall success rate after all 10 steps falls to just 34%. This is one of the reasons why agents like AutoGPT often fail in complex tasks.

[1] https://github.com/openai/grade-school-math

Actual reasoning shows the understanding and use of a model of the key features of the underlying problem/domain.

As a simple example that you can replicate using chatgpt, ask it to solve some simple maths problem. Very frequently you will get a solution that looks like reasoning but is not, and reveals that it does not have an actual model of the underlying maths but is in fact doing text prediction based on a history of maths. For example see here[1]. I ask it for some quadratics in x with some specification on the number of roots. It gives me what looks at first glance like a decent answer. Then I ask the same exact question but asking for quadratics in x and y[2]. Again the answer looks plausible except that for the solution "with one real root" it says the solution has one real root when x + y =1. Well there are infinite real values for x and y such that x + y =1, not one real root. It looks like it has solved the problem but instead it has simulated the solving of the problem.

Likewise stacking problems, used to check for whether an AI has a model of the world. This is covered in "From task structures to world models: What do LLMs know?"[3] but for example here[4] I ask it whether it's easier to balance a barrel on a plank or a plank on a barrel. The model says it's easier to balance a plank on a barrel with an output text that simulates reasoning discussing center of mass and the difference between the flatness of the plank and the tendency of the barrel to roll because of its curvature. Actual reasoning would say to put the barrel on its end so it doesn't roll (whether you put the plank on top or not).

[1] https://chat.openai.com/share/64556be8-ad20-41aa-99af-ed5a42...

[2] https://chat.openai.com/share/2cd39197-dc09-4d07-a0d6-6cd800...

[3] https://arxiv.org/abs/2310.04276

[4] https://chat.openai.com/share/4b631a92-0d55-4ae5-8892-9be025...

I'm not convinced. Openai specifically trained their models in a way that is not trying to pass the Turing test. I suspect current models are more than capable of passing Turing tests. For example, i suspect most humans will give nonsense answers to many logic problems!

This is totally normal. It's very common for people to be more scared of public speaking than of dying, for example; there's no shame in it. It's helpful to be aware of, even, because if we know that we're not perfectly "rational" with our fears we can try to compensate.

If there's a referendum between two government policies, the first that every single person had to publicly speak in front of at least ten strangers once a year, that policy would be terrifying and bad to people who don't like public speaking. If the second policy was that every single person should be killed, that might be scary but it's not really as viscerally scary as the forced public speaking madman, at least to a lot of people, and it's also so bad that we have a natural impulse to just reject it as possible.

Nevertheless, if we recognise these impulses in ourselves we can attempt to adjust for them and tick the right box on the imaginary referendum, because even though public speaking is really bad and scary, it's still better than everyone dying.