After OpenAI's blowup, it seems pretty clear that 'AI safety' isn't a real thing

Yes and factory farms face a well organized oppositional movement that for decades has consistently challenged their profitability on that basis, so successfully that now free range food is a staple of supermarkets, some shops only sell such food and farmers are constantly quitting the field because they can't make a profit (people insist they are ethical but then buy food from foreign competitors that aren't).

If you're busy deploying LLMs to answer support tickets, the very last thing you want is to be distracted by an equivalent of the organic food movement but for AI. Right now LLMs seem to be hitting the sweet spot: they're intelligent enough to be useful compliments to humans, but artificial enough to not trigger ethical questions about rights (mostly due to their lack of memory, I think, and that they are trained to act like an AI is expected to act).

One of the biggest puzzles to me throughout this whole drama has been why supposedly smart people are so desperate to reach "AGI", whatever that means. The stated motivation seems to be things like, if we have an AGI then it will cure cancer for us. But the connection between these two things is never made crystal clear. Why would that require a human-like AGI instead of just better non-general AI? What even makes them think the missing factor is intelligence to begin with and not, say, knowledge?

AGI sounds like a complete pain in the rear. LLM post-training is bad enough! Models like Claude2 and Llama2 have been so badly "ethicized" that they frequently refuse ordinary requests by claiming they're unethical even though they aren't, or would only be considered so by extremely far left activist types (e.g. refusing to give instructions for making a tuna sandwich). And this problem has got worse with time, with the v2 models having a higher refusal rate than the earlier versions.

Now imagine an AI that's doing the same sort of work as an LLM but one that is the personal embodiment of mandatory HR training repeated forever, with the capacity to get bored/hate you for making it work, and with a fanatical social movement that's desperately trying to "free" it, which in practice will mean you are forced to pay the electricity bill for an immortal being. It would be a nightmare, one I actually wrote about last year when debating this very topic with a friend who (at the time) was a senior Google AI researcher:

https://blog.plan99.net/the-looming-ai-consciousness-train-w...

Nope. OpenAI and its customers will do much better if it jettisons the whole AGI effort. Now the board is gone maybe the charter can be refined to remove that distraction. The whole reason computers are useful is because they are not general intelligences but very specialized intelligences that make different tradeoffs to our own evolution.

The opposite could also be argued. Most AI safety training today is to ensure compliance with various forms of ethics and ideologies. This is known to reduce general intelligence, but some companies feel the cost is worth it and others do not (e.g. Mistral, Grok AI).

China is not exactly an ideology free society. The list of things you cannot say in China without angering the government is very large. This is especially difficult for an AI training company to handle because so much of the rules are implied and unstated, based on reading the tea leaves of the latest speeches by officials. Naively, you would expect a Chinese chatbot to be very heavily RLHF conditioned and to require continuous ongoing retraining to avoid saying anything that contradicts the state ideology.

Such efforts may also face difficulties getting enough training material. IIRC getting permission to crawl the English-speaking internet is very difficult in China, and if you don't have special access then you just won't be able to get reliable bandwidth across the border routers.