A misalignment of AI in mathematics

Without human scientific and mathematical intuition you can bet that AI will always take the road most travelled and miss the genuinely new and interesting breakthroughs - in fact it will not even think of them or try them without a human rider whipping it constantly to go down paths it would normally not consider.

And I think that is true in the case of the recent OpenAI blow-up -- it is alleged that even in this case AI did not come up with the winning technique without some human guidance.

I have never - across perhaps many thousands of chats with AI - seen AI push beyond the edge of what is known by itself, without being forced by a human to think outside the box.

I do think it might be possible to encode this process with prompting and agent orchestration methodologies - but even then - without a human - and human intuition - in the loop, I am skeptical.

Therefore I think a more accurate view of AI for math and science today is that it is an extremely powerful tool in the hands of a skilled operator with a strong intuition and roadmap of where to go, and rather boring when left to its devices.

Whether that will change in the future is a question. Nothing says that in principle AI could never do what a human driver does - but I still doubt that AI will replicate the life history and experience that real scientific and mathematical intuition is really made of.

AI trains on a lot of stuff - but it doesn't have hallway conversations, office-hours with teachers, or hard-won experience from all the things that failed that are NOT in the training data...

AI trains mainly on the record of what worked, not what didn't work and never even got published (and only lives in peoples heads) - and what didn't work is arguably as or more important for making breakthroughs and forming real intuition.

> The perverse incentives of academia mean this has never occurred.

This. Mathematicians in their most energetic years are trying to get tenure or land a tenure-track job. They are disincentivized to go all-in on ultra high risk, high-reward problems. The potential downside is just too forbidding. It's much safer to develop a research program in a mainstream field that affords many opportunities for partial progress that can translate to a robust publication record.

> why bother grinding through all the material of climbing Mt. Everest and then attempting it if the helicopter ride is how things are done now?

I think you answered this yourself earlier:

> I think progress is really measured by what humans are able to do and understand

People want to make this progress. Therefore people will "grind Everest" as a mathematical community, and that is maybe not so hugely different from a lot of previous mathematical work.

There's still ample room for creativity: simplifying, generalizing, asking new questions humans are interested in, ...

All very valid criticism and those Fields medalists are truly reprehensible human beings that have never done anything but solve lots of problems nobody asked for to boost their egos.

Look for example at this guy who isn't even a fields medalist:

In August 2006, Perelman was offered the Fields Medal[1] for "his contributions to geometry and his revolutionary insights into the analytical and geometric structure of the Ricci flow", but he declined the award, stating: "I'm not interested in money or fame; I don't want to be on display like an animal in a zoo."[2] On 22 December 2006, the scientific journal Science recognized Perelman's proof of the Poincaré conjecture as the scientific "Breakthrough of the Year", the first such recognition in the area of mathematics.[3]

On 18 March 2010, it was announced that he had met the criteria to receive the first Clay Millennium Prize[4] for resolution of the Poincaré conjecture. On 1 July 2010, he rejected the prize of one million dollars, saying that he considered the decision of the board of the Clay Institute to be unfair, in that his contribution to solving the Poincaré conjecture was no greater than that of Richard S. Hamilton, the mathematician who pioneered the Ricci flow partly with the aim of attacking the conjecture.[5][6] He had previously rejected the prestigious prize of the European Mathematical Society in 1996.[7]

https://en.wikipedia.org/wiki/Grigori_Perelman

What a fucking ego booster who never solved any problem anyone ever asked for. Oh no wait, that's actually the only human to ever solve a Millennium Prize problem and I'm sure he'd tell you were to get off if you told him that you made a wonderful math breaking machine that solves maths without giving due credit and without producing any new maths that anyone else can use to solve more problems but you know, ego boost and all that.