Live data from Hacker News

Ask HN: What was your "oh shit" moment with GenAI?

news.ycombinator.com

891–900 of 1001 posts

Re: Ask HN: What was your "oh shit" moment with GenAI?

#891
The same thing I've been concerned about for years. These things can sound extremely confident and be wrong. If you're out of your depth with these tools, you'll find a "trustworthy sounding partner" who's basically lying to you about what it is "sure about".

AKA - Slop... It's a brand new form of bullshit.

Re: Ask HN: What was your "oh shit" moment with GenAI?

#892

For me it was right at the beginning. They said it was a dungeon game. It would describe a room, etc, and I would take some action. But I thought that this dungeon was built in some intricate database. But then I told it that I wanted to leave, got to an inn, where I flirted with the bar waitress, and soon we were watching the sunset in some meadow. As cheesy as that was, it was then that I went "oh shit" this is a m…

Reminds me of Microsoft Research's TextWorld [1] (from 2018!).

[1] https://www.microsoft.com/en-us/research/project/textworld

Re: Ask HN: What was your "oh shit" moment with GenAI?

#893
I think for most people paying a bit of attention it was gradual - relatively speaking. I.e. a bunch of "holy shit" moments spread out of months.

First uptick for my own usage was copying in a ~800 line rust file into ChatGPT before agentic was a thing and having it rewrite it for clarity and thinking: "Yeah, this is actually a genuine improvement I'd want for all my files".

As a tidbit for posterity. The most basic thing an LLM does is find a word or name that best matches an idea so - while there are going to be some incestuous problems down the line - we're going to have to explain to juniors that we had weird problems like: "You had to come up with a name for a variable". Nowadays, I only occasional prompt that some struct/function name needs to be renamed and choose the best of the options it generates.

---

There used to be 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors. Now its just the 1.

Re: Ask HN: What was your "oh shit" moment with GenAI?

#896
Not an "oh shit" per se, but Gemini is a complete leap in terms of actual smarts over Google Assistant.

I was having problems getting it to parse my speech correctly when I was asking about "autolyse" (I was attempting to bake bread). All I had to do to fix it was add this to the system prompt: " I primarily interact with you via a speech to text mechanism. You should consider ways in which my words may not be accurately transcribed and attempt to infer the correct reading in context. When you do this, do not mention it - just proceed as if my words were recorded properly."

Never had that problem again. With Assistant though, if I had any issue like that I'd be waiting months or years for a fix, if it ever came.

Re: Ask HN: What was your "oh shit" moment with GenAI?

#897

Earlier quoted context omitted.

>Claude walked me through examining the some of the original software in GHIDRA, I wanted to be able to decrypt the files on The Complete New Yorker magazine DVDs. The old software was WinXP only, and crashed by the time you turned to page 3 or 4. It walked me through using Ghidra on the relevant dll, mapped out how it was using Blowfish, what the credentials were that it was passing, and re-implemented all of that i…

What was your setup for this and did you have any preferences set in Claude to get started with something like this?

I use Claude on the desktop, and only occasionally Claude Code. It's the one that recommended Ghidra. Walked me through the install. Taught me the basics (G to go to an address, etc). Would tell me where to go, and what to paste back to it. It eventually converged on where to find the iv and credentials and so forth (after acting confused for awhile), and then wrote the python script for me that decrypts. I'd like to think my questions (and challenges to its assertions) were intelligent enough to spur it towards the solution, but self-flattery is all that is.

The dll in question was pretty obvious just from the filename alone that it was where the magic happened.

If you want something similar, you might just start by asking it if it would be feasible to decompile the software in question to reverse engineer the decryption, that you'd heard Ghidra was a big deal. Keep nudging it to guide you along that sort of path.

Re: Ask HN: What was your "oh shit" moment with GenAI?

#900
I asked claude code to troubleshoot why a certain strange behavior was occurring in a Go service. Only then I realized that the behavior I described wasn't actually occurring. A few minutes of thinking pass and Claude code confidently makes up some scenario why this was definitely happening. It was then I though, "oh shit, these LLMs are just complete BS machined telling us what we want to hear."
Post reply on HN