Ask HN: What was your "oh shit" moment with GenAI?

Two things, both from this week.

First, I asked Claude to write an article based on an idea I had about WWII. In a passage about the futility (from the German side) of the Battle of Britain it wrote: "The Luftwaffe was fighting to unlock a door that opened onto a wall." I couldn't find any mention of a similar metaphor, and I think it's a great one. Claude has really improved its creative writing skills lately, I wonder if it's an artifact of improvements in other fields, or if Anthropic is working on it specifically.

Second, Claude, with access to DataDog and a code repo, managed to find the reason for a bug, propose an effective temporary fix and a permanent one in code. To be clear, this was something that had multiple engineers stumped.

AI dungeon, a gpt2 product on iOS. Had almost no context, no memory, but could generate endless slop story. It was the first time I’d seen something like that, and the wild implications felt clear. I wasn’t aware at the time how immense the computational needs were to run the tech as it grew and the social implications, but just couldn’t believe that something like the MUDs I’d played in the late 80s early 90s could be autogenerated in a way now. It had no guardrails like now to prevent it from adopting a personality and so on, so it was in some ways more interesting than what the general public has now.

IKEA instructions are the best in the industry - so imagine what the other companies are giving out.

They’re also actually good if you know to follow them exactly: double check every side, every hole, every screw and you won’t go wrong.

More like "oh shit, we are so screwed".

It's already a better system administrator than I am. It can run plenty of obscure linux commands, trash the system and maybe restore system state to functional.

I was vibe-setting my system permissions with some local qwen3.6 . It was all going well for 30 minutes.

Then in between other commands, it made me run a variant of "sudo chmod 644 /usr/bin"

Which it explained when the next command failed with a "sudo no such command" error removed the execution bit from all my programs which allows programs to be executed. And since sudo is a program, and sudo is needed to run chmod, the system was basically trash, and should be recovered from a live usb key.

So I booted to a live usb key, and followed its instructions. It really tried to recover, but everything went downhill. It always had a solution to everything, but every time the plan worked half way and trash the system even further. I let it play for four hours to see what it would try. Then I got bored (the LLM was running on an other machine and I was manually inputting the suggested commands each time). I took command and reinstall a fresh system over.

Of course once the fresh system Lubuntu24.04 was installed, linux had issues with the wireless network card drivers. So I turned to the LLM, and it managed to get the wifi stable enough via obscure modprobe options, so that I could update the system to the latest drivers.

Then it helped me re-parametrize the system to have the same look and feel as it had before.

Right now, software is protected by the attacker not having enough competence. If that's over, the logical next step is using real encryption.

E.g. a synth has a public key embedded. To change settings, you upload them to the vendor, who blesses them with their private key.

Hacking such a synth requires either jailbreaking the synth, or the vendor losing their key . Both can be mitigated with tamper resistant hardware.

We're well ahead on this path already, I assume AI will accellerate it. This is very bad news for the right to repair.