An internal OpenAI Astra model solved 10 major open math and CS problems

Perhaps not in general, but this is nothing new or related to AI. For one thing software is a strangely private craft - your boss is likely only aware of what you are producing, not the craft or quality (or not) that went into it - the only ones aware of that would be co-workers who are looking at your source code, and perhaps choosing to use it or not. Secondly, the sad truth in business, not just for software developers, is that there is a huge amount of truth in "it's not what you know, it's who you know".

> People are having a rough time figuring out that good design and fundamentals no longer align with the business side of things.

The business may be unaware of what your code looks like, or the expertise that went into it to avoid bad outcomes (immediate or future), but they surely DO care when poor quality software leads to production outages, and customer complaints, and their boss yelling at them as a result. The results of poor quality software will manifest in many ways, not just the poor reliability and performance of your latest release, but also the future outcomes of attempts to build on top of it.

I think it's extremely early to assess the impact of AI software quality and the ways it impacts businesses in these ways, such as production outages, that they do care about. There have certainly been well publicized cases such as AWS outages blamed on AI, but although it seems like an eternity it's only been ~6 months since AI really became usefully competent and capable of "vibe coding", so I think we're only just beginning to see the consequences.

Companies and bosses that have enthusiastically adopted AI are unlikely to point to AI and themselves if/when it proves to be detrimental (AWS were quick to deny it, despite the insider leaks to the contrary), so I'd not be surprised if it takes years rather than months for there to be much clarity on this.

FWIW, I think there is a considerable difference between math, which is what we're really discussing here, and software development, regarding how quickly it will or has fallen to AI. Math is really uniquely suited to LLMs and reinforcement learning of domain-specific reasoning since mathematical papers basically consist of nothing but reasoning (training data!), and it is an entirely theoretical field that essentially lives in the world of language (incl. mathematical notation) without any external grounding. Contrast to software development where what is available to train on is mostly the end result (code) rather than the reasoning that got you there, and there is also a large "experimental" aspect - what happens in the real world when you run the produced artifact.

You need to be at a select few institutions to avoid these safety features, which I'm not. University needs to sign some stuff but I that's outside my control.

What I'm talking about is the way OpenAI have handled this. They are in the business of artificial intelligence, striving to create human-level, or even super-human intelligence, and telling us what a wonderful future this will create.

Shouldn't they be celebrating intelligence, especially human intelligence which is what they have bottled here?! LLMs are themselves minimally intelligent - they are primarily regurgitating human intelligence. Without their human training data, these systems would be nothing.

Let's see OpenAI build "GPTZero", an actually intelligent system, that learns for itself, learns language and mathematics from nothing like a baby, then I feel they'd have the right to minimize and cheapen the achievement if they chose to (unlikely). As it is, all they have is a predictor, utterly dependent on HUMAN intelligence, and it would be more becoming of them to acknowledge this by paying a bit more deference and respect to the people who enabled them to build what they have.

FWIW I am not a mathematician.

That's not true and wasn't true in the sense that businesses don't care directly but they are indirectly. Are you saying business doesn't care if things go down or it doesn't work?

At the end of the day people don't care until they do and then they blame you.

Yes,their intelligence is jagged, excellent for some things and not so good for others, but calling them "minimally intelligent" is disingenuous.