Are we offloading too much of our thinking to AI?

Using AI for making decisions create a whole host of novel vulnerabilities.

I've seen a lot of people talk about the obvious privacy and autonomy disaster it creates -- giving companies (and anyone who can legally compel or hack them) such intimate details or allowing yourself to be dependent on their services which they can withdraw or degrade at any time... Or where the model operators can bias the results based on bribes (ad payments) or their own agenda ... but really that's the start.

Another one what I haven't seen discussed is that use of an AI provides oracle access to your opponents. What I mean is that if I want to influence you, I can have an AI try variations of arguments against the model you're using (or if I don't know, a panel of models you might be using) until it can construct a super influential output. I can see what your model might recommend and anticipate your actions and change my actions to control yours. This can be done in a targeted way against individual people, or against populations e.g. by marketers.

There are reasons why this may be even more effective than it seems: It appears that models themselves are trained on each other, so in testing I see in family effects: GPT-4o mini seems much more convinced by essays written by GPT5 than ones written by GLM 5.2, while other models are more neutral (and my human review says the GPT5 essays are unambiguously worse in this trial). So I assume GPT5's writing was refined by review by in-family models and it's already somewhat of a super persuader against GPT-4o mini as a result.

The danger of oracle access is somewhat a symptom of AI monoculture resulting in a reasoning monoculture. If there were a billion highly distinct models widely used then it might be the only way to convince or predict a disproportionate portion of them would be to simply be right, and then we'd just be left with a operational security concern of making sure your opponents/competitors don't know what model(s) you use. But of course, that isn't the world we have today: people use a small number of models, and their behavior is generally pretty related due to architectural commonalities and cross distillation and such.

Power consumption per unit of work keeps going down, too. Our pocket supercomputers are astoundingly efficient compared to what it used to take to get the same work done at relatable points in the past. We'll get there.

Before we get there, we'll have network connectivity to our desktop AI boxes at home. That's pretty good, too.

And if this clown-bot[1] boom is a bubble (as I believe it is), then it's just a matter of time before it pops. The blast radius is unknown, but at one end it seems likely to result in something between potentially-idle fabs that have already been mostly paid for. Idle factories are bad and it makes sense to avoid that even if it is expensive, so this means cheap hardware.

At the other end, it means failed companies with fabs that are sold for pennies on the dollar alongside a resolute unwillingness amongst the investors (who just had their asses handed to them) to start on Boom 2.0. This latter scenario would mean golden age of cheap hardware.

I think we'll be fine.

[1]: Yeah, clown is on purpose. It fits as a replacement in any technical parlance where "cloud" would be used. :)

> Calling AI "just autocomplete" for system architecture is outdated.

I didn't say that, though.

> it would do a better job than any junior dev ever could. Sure, giving context is still necessary, but you can just speak in plain, normal English and it will understand. AI is getting scary good at system design.

Sure. But that's not my point. Don't you still want those juniors to be capable of creating those diagrams? So they can understand them? If the juniors don't get what the AI is outputting, why are you even creating the diagrams in the first place, right?

That's what this topic is about. Just like how calculators helped us do complex tasks, we still needed to understand math. Same with AI, you still need to be understand the topics you ask it to do.

You are heavily self-selecting what part of humanity you interact in the real world as well. You pick your friends, places you live, places you frequent, times you choose to go out and so on.

I didn't know how wide the humanity variance is before I had to go to a government office to exchange identity card (as the old one expires after some years). In the queue, when I looked to my left and right I saw some of the most bizarre people I ever saw. Because they were picked fully randomly just by having their identity documents sort of "off by one" in relation to mine. Humans and their output, when taken representatively are at most tolerable.

Coding LLMs got good only very recently (about 3 months ago). And many people don't use the best ones. The amount of LLM generated bs dropped for me from about 20%, which made me very cautious, to nearly zero, where I see the agent do only the errors that a smart person could also make.

It's not evenly distributed though. There are still areas for improvement in my experience. They still require a bit of handholding for example when doing SQL when large amount of data is involved. I'm sure they'll get better intuitions about indexes in the future. Although I don't think that's anything that couldn't be solved with a skill file today.