Live data from Hacker News

Ask HN: What was your "oh shit" moment with GenAI?

news.ycombinator.com

201–210 of 1001 posts

Re: Ask HN: What was your "oh shit" moment with GenAI?

#201

My furnace went out during the 2025 holiday and I couldn't get an appointment with a repair person for 2 days. It was getting very cold in my house so I went into my attic and made several videos of the furnace attempting to start and gave it to gemini. It diagnosed the issue immediately and had me spin one of the components (a small exhaust fan) while the furnace tried to fire. It came on immediately. I had to do th…

Gemini almost killed you.

The exhaust blower not working triggered a safety that prevented the furnace from firing.

Spinning it bypassed the safety.

You likely inhaled a lot more carbon monoxide than you know.

Re: Ask HN: What was your "oh shit" moment with GenAI?

#202

I work with someone who is very AI-forward, high confidence, and very low execution. He has started sending me large PRs of AI slop that he assured me doesn't need to be reviewed. I quickly find many minor issues from an initial pass of one of the reviews. He gets mad at the team for slowing him down. He also will paste chat logs with Claude into our team chat. Often Claude will say the same thing I told him but he e…

What's funny about this is that it sounds like your coworker reviews his LLM output roughly as well as you read the other replies before assuming that this was an anti-LLM pile-on thread.

I did read the other replies. I don't think my comment is that LLMs are bad. I use LLMs and agents for work. I think my "oh shit" moment is the dynamic that giving someone LLMs amplifies their impact (positive or negative).

For example, some people give kids tiny go karts and that's acceptable because the damage they can do with a very tiny battery powered 4 wheeler is minimal. We now live in a world where everyone has access to a tank and can plow over everything.

I think LLMs will increase anti-social behavior.

Re: Ask HN: What was your "oh shit" moment with GenAI?

#203
Not sure that I've had it yet, although hypothetically I'm sure it would probably be something similar to the examples of writing new software for old hardware mentioned ITT. The idea of resurrecting useful but unsupported gadgets that would otherwise become e-waste is something I've always found compelling.

Problem is, I just don't have enough old crap, and if I did, I would have a hard time justifying the expense, because that money could maybe just go toward a more intimate tinkering process.

For everything else, I either haven't had any sufficiently interesting ideas, or they ended up not being worth pursuing with those tools or at all.

When I do have success that I'm happy with and care about, it's a slow process that I ultimately need to know the details of anyway, but otherwise it's a bunch of luckily narrow work-related scenarios with well-documented constraints. Nothing's really been that shocking though.

The shocking thing to me is how unrewarding most of the successful tasks have been, partly because they often create unnecessary work and partly because the type of thinking required to massage or evaluate the result is much less stimulating, and there's much more of it in aggregate. It's fine if it's something like generating a UI from scratch because that hasn't produced dopamine in a long long time anyway

Re: Ask HN: What was your "oh shit" moment with GenAI?

#204
post #195

"We're traveling to Tokyo on our way home from China. We'd like to plan a trip accessible by train that hits some beaches, some hot springs, and allows me to get the 4th does of a rabies vaccine sequence (the first three shots were rabvac)"

You can't just leave that hanging out there unexplained.

Re: Ask HN: What was your "oh shit" moment with GenAI?

#205
It was about two days after Google released Deep Dream, if you remember, the thing that took a video and filled it with fleeting hallucinations of mostly puppies, fish heads and lizards. I was suddenly struck by the realization "oh shit, this is much more boring and samey than it first appeared to be", and all subsequent gen AI has been similarly underwhelming.

Re: Ask HN: What was your "oh shit" moment with GenAI?

#206

I work with someone who is very AI-forward, high confidence, and very low execution. He has started sending me large PRs of AI slop that he assured me doesn't need to be reviewed. I quickly find many minor issues from an initial pass of one of the reviews. He gets mad at the team for slowing him down. He also will paste chat logs with Claude into our team chat. Often Claude will say the same thing I told him but he e…

CI/CD? My non-techie friends send me screenshots of ChatGPT. I guess that’s a modern micro aggression?

> CI/CD?

That is less useful when the changes are editing the tests but we don't know if a human has validated the assertions.

> My non-techie friends send me screenshots of ChatGPT. I guess that’s a modern micro aggression?

I think the concern I have is explicitly not the sending the chat logs. I think it's this flow:

1. Ask a question

2. Get an answer from a team member.

3. I don't like the answer and instead of discussing I am going to go to Claude and ask the same question.

4. Copy/paste the answer into chat without seeing if it includes novel information.

In one case the engineer was asking which model to select in the agent framework we are using. I gave an answer and provided a list of reasons. They did not like this answer and asked Claude which gave the same answer.

The answer was something inherently obvious and that anyone should be able to derive from first principals.

Re: Ask HN: What was your "oh shit" moment with GenAI?

#207

I think it's really scary how agents are hallucinating/doing bad actions, then proceeding to gaslight you about how nothing went wrong. Then you tell the agent that it deleted your whole company database, it says something like "I'm so sorry, I shouldn't have done that. Won't do that again" As AGI looms overhead, this thought of agents going "rogue" with nothing really stopping them has caused me some panic.

"I'm sorry" is not gaslighting but an admission of fault it learned from our texts. And if an LLM managed to delete your database, it's time to slow down the vibe train and put up some guard rails.

LLMs are awesome but not without supervision.

Re: Ask HN: What was your "oh shit" moment with GenAI?

#208
Two things:

1) I wanted a harness for running BPC.EXE (the old Borland Pascal 7.0 Compiler) and I asked Gemini 3.5 to build it for me using the unicorn engine. It whipped out a working .py file easily under ten minutes. Most likely five.

2) I handed a random assembly function from the OS/2 1.x kernel to Gemini 3.5, and it proceeded to tell me that it was related to disk I/O and partitioning, without a single associated string, and it annotated it all, including the relevant structures it was addressing.

Re: Ask HN: What was your "oh shit" moment with GenAI?

#209

I bought an Alesis QS8.1 super cheap in perfect condition (was a top grade digital piano/synth in the 90s). and then i realized that ALL of the software (which i collected from defunct websites and archived on github) related to it was ancient and after a while of getting tired of using WINE every single time i decided i wanted a cross platform modern equivalent that did everything that several of these different pro…

That's fantastic. Did you use a Ghidra MCP server? It's kind of magical huh?

Re: Ask HN: What was your "oh shit" moment with GenAI?

#210
post #183

I think my favorite early story was when OpenAI launched deep research. I was going to an event that I was headlining, and I gave it a CSV of the attendees and asked it to give me a small background on each company they represented. When people introduced themselves to me, I knew a little about their startup. Felt magical.

OMG I love this!

I was at an industry event this week. a CEO of a startup took the big board of vendors who are present, put it through an LLM. It summarised the companies he should be looking at discuss partnership opportunities with and why based on his business. Spot on.

Post reply on HN