Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The scariest "user support" email I've received

devas.life

Post reply on HN
Print view View original

91–100 of 267 posts

Re: The scariest "user support" email I've received

#91

Post by Legend2440 »

sublinear wrote: post #27
notRobot wrote: post #21

Earlier quoted context omitted.

Isn't analysing and writing bits of code one of the few things LLMs are actually good at and useful for (as opposed to creative writing or whatever). Before LLMs if someone wasn't familiar with deobfuscation they would have no easy way to analyse the attack string as they were able to do here.

LLMs are just as bad at code as "creative writing or whatever". It's just that fewer people know how to write/smell code at the same level as prose, so we get drowned out as "anti-AI" cynics and the lie continues.

But chatGPT was correct in this case, so you are indeed being cynical.

Re: The scariest "user support" email I've received

#93

Post by jay_kyburz »

croes wrote: post #86
margalabargala wrote: post #41

Earlier quoted context omitted.

I think it's great. If the LLM takes it upon itself to download malware, the user is protected.

Wait for next step, when the target is actually the LLM.

Or you are the target, and your LLM is poisoned to work against you with some kind of global directive.

Re: The scariest "user support" email I've received

#95

Post by Legend2440 »

reaperducer wrote: post #61
tantalor wrote: post #4

> as ChatGPT confirmed when I asked it to analyze it lol we are so cooked

> as ChatGPT confirmed when I asked it to analyze it lol we are so cooked This guy makes an app and had to use a chatbot to do a base64 decode?

You’re right! He should have decoded it by hand with pencil and paper, like a real programmer.

Re: The scariest "user support" email I've received

#96

Post by nneonneo »

tantalor wrote: post #4

> as ChatGPT confirmed when I asked it to analyze it lol we are so cooked

Better yet - ChatGPT didn't actually decode the blob accurately.

It nails the URL, but manages somehow to get the temporary filename completely wrong (the actual filename is /tmp/pjKmMUFEYv8AlfKR, but ChatGPT says /tmp/lRghl71wClxAGs).

It's possible the screenshot is from a different payload, but I'm more inclined to believe that ChatGPT just squinted and made up a plausible /tmp/ filename.

In this case it doesn't matter what the filename is, but it's not hard to imagine a scenario where it did (e.g. it was a key to unlock the malware, an actually relevant filename, etc.).

Re: The scariest "user support" email I've received

#97

Post by Arainach »

croes wrote: post #84
notRobot wrote: post #21

Earlier quoted context omitted.

Isn't analysing and writing bits of code one of the few things LLMs are actually good at and useful for (as opposed to creative writing or whatever). Before LLMs if someone wasn't familiar with deobfuscation they would have no easy way to analyse the attack string as they were able to do here.

Come on. Base64 decoding should be like binary to hex conversion for a developer. The command even mentions base64. What if ChatGPT said everything is fine?

Correct, but again this is one of the things LLMs are consistently good at and an actual time saver.

I'm very much an AI skeptic, but it's undeniable that LLMs have obsoleted 30 years worth of bash scripting knowledge - any time I think "I could take 5min and write that" an LLM can do it in under 30 seconds and adds a lot more input validation checks than I would in 5min. It also gets the regex right the first time, which is better than my grug brain for anything non-trivial.

Re: The scariest "user support" email I've received

#98

Post by spartanatreyu »

notRobot wrote: post #21
tantalor wrote: post #4

> as ChatGPT confirmed when I asked it to analyze it lol we are so cooked

Isn't analysing and writing bits of code one of the few things LLMs are actually good at and useful for (as opposed to creative writing or whatever). Before LLMs if someone wasn't familiar with deobfuscation they would have no easy way to analyse the attack string as they were able to do here.

> Isn't analysing and writing bits of code one of the few things LLMs are actually good at and useful for

Absolutely not.

I just wasted 4 hours trying to debug an issue because a developer decided they would shortcut things and use an LLM to add just one more feature to an existing project. The LLM had changed the code in a non-obvious way to refer to things by ID, but the data source doesn't have IDs in it which broke everything.

I had to instrument everything to find where the problem actually was.

As soon as I saw it was referring to things that don't exist I realised it was created by an LLM instead of a developer.

LLMs can only create convincing looking code. They don't actually understand what they are writing, they are just mimicking what they've seen before.

If they did have the capacity to understand, I wouldn't have lost those 4 hours debugging its approximation of code.

Now I'm trying to figure out if I should hash each chunk of data into an ID and bolt it onto the data chunk, or if I should just rip out the feature and make it myself.

Re: The scariest "user support" email I've received

#100

Post by Legend2440 »

croes wrote: post #89
gambiting wrote: post #31

Earlier quoted context omitted.

I don't understand? It's actually a pretty good idea - ChatGPT will download whatever the link contains in its own sandboxed environment, without endangering your own machine. Or do you mean something else by saying we're cooked?

ChatGPT didn’t download anything, hopefully. The we’re cooked refers to the fact of using ChatGPT to decode the base64 command. That’s like using ChatGPT to solve a simple equation like 4*12, especially for a developer. There are tons of base64 decoder if don’t want to write that one liner yourself.

So what? Why not use the everything machine for everything? You have it open anyway, it’s a fast copy-paste.
Post reply on HN