Not so much a criticism of the tech that is called AI currently, but more of the impression that the term "AI" itself instills. A thought experiment I've come up with is to look for the term "artificial intelligence" when it shows up in articles and comments and replace it with something like "(advanced) technological progress". After all AI is a term that can be used to refer to a wide variety of specific technologies like LLMs, segmentation and diffusion models, all of which seemed to explode in usefulness and popularity in a relatively short time frame. And I think "AI" is a reactionary term for technology that's so above what came before it in recent memory that it's in parts fascinating and scary.
I always thought the term "artificial intelligence" had a sort of disabling effect, like there is an intelligence outside of ourselves that serves to drive us in X direction, good or bad. "Technological progress" implies we are the ones driving the changes and the problems they will invariably bring. We sort of grasp this tech will cause profound impacts on society of some vague quality, enough to leave an "ethics" section in every white paper that comes with freely-distributed code and instructions for use, but continue plowing on regardless of what they could possibly be. How sustainable is this? Will there ever come a time when uploading code or even papers to GitHub for anyone to consume become taboo from the stigma and change that's been inflicted on ourselves?
I think the inflection point for those problems creeping into society at a visible everyday level is on a much quicker time scale than AGI. Sometimes I think it's like equipping people with pistols that shoot precision-guided homing bullets - not so much on the scale of a civilization-ending scenario, but it changes the game in its own significant ways. Look at comments accusing others of using ChatGPT to write their responses for them. I think most tech can cause these effects and it's worth questioning what it's meant to accomplish as they're created or used.
At times I wonder if the end stage of any given intelligent civilization is to delegate all parts of its thought process to technology that can be engineered to be superior, with all consequences that entails, because there's no point to being stuck with the tech that is already there forever. The thought that scares me the most is that the revolution might not be directed by governments or angry anarchists, but indirectly, by bored machine learning engineers sitting in their rooms contributing just one more paper or PyTorch implementation towards an inflection point in humankind because it's fun and rewarding to them.
And even if we're supposed to stop advancing this tech to prevent irreversible societal change, would it even be possible if we tried? There's 8 billion of us on Earth and metric tons of GPUs in existence. The question of if progress can ever be halted in a state such as ours for in the name of self-preservation is one I'll probably be keeping in mind for the rest of my lifetime.