Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Curl: We still have not seen a valid security report done with AI help

linkedin.com

Post reply on HN
Print view View original

41–50 of 258 posts

Re: Curl: We still have not seen a valid security report done with AI help

#41

Post by spiffyk »

gruez wrote: post #35
spiffyk wrote: post #31

Earlier quoted context omitted.

Well, it is not, but the three "sources" you mention are not worth much either, much like ChatGPT.

>but the three "sources" you mention are not worth much either, much like ChatGPT. I don't think I've ever seen anyone lambasted for citing stackoverflow as a source. At best, they chastised for not reading the comments, but nowhere as much pushback as for LLMs.

I have personally always been kind of against using StackOverflow as a sole source for things. It is very often a good pointer, but it's always a good idea to cross-check with primary sources. Otherwise you get all sorts of interesting surprises, like that Razer Synapse + Docker for Windows debacle. Not to mention that you are technically not allowed to just copy-paste stuff from SO.

Re: Curl: We still have not seen a valid security report done with AI help

#42

Post by mynameisvlad »

gruez wrote: post #35
spiffyk wrote: post #31

Earlier quoted context omitted.

Well, it is not, but the three "sources" you mention are not worth much either, much like ChatGPT.

>but the three "sources" you mention are not worth much either, much like ChatGPT. I don't think I've ever seen anyone lambasted for citing stackoverflow as a source. At best, they chastised for not reading the comments, but nowhere as much pushback as for LLMs.

I mean, if all they did is regurgitate a SO post wholesale without checking the correctness or applicability, and the answer was in fact not correct or applicable, they would probably get equally lambasted.

If anything, SO having verified answers helps its credibility slightly compared to a LLM which are all known to regularly hallucinate (see: literally this post).

Re: Curl: We still have not seen a valid security report done with AI help

#43

Post by alabastervlog »

ivape wrote: post #5
_se wrote: post #3

Earlier quoted context omitted.

Fortunately, they can beat an automated idiot agent, which is much closer to what we have today.

[flagged]

I'm still waiting to see an LLM that is actually helpful versus the alternatives, outside a very small set of tasks that's basically the same as "2 years ago".

Re: Curl: We still have not seen a valid security report done with AI help

#44

Post by bloppe »

spiffyk wrote: post #31
hashmush wrote: post #24

Earlier quoted context omitted.

As much as I'm also annoyed by that phrase, is it really any different from: - I had to Google it... - According to a StackOverflow answer... - Person X told me about this nice trick... - etc. Stating your sources should surely not be a bad thing, no?

Well, it is not, but the three "sources" you mention are not worth much either, much like ChatGPT.

SO at least has reputation scores and people vote on answers. An answer with 5000 upvotes, written by someone with high karma, is probably legit.

Re: Curl: We still have not seen a valid security report done with AI help

#45

Post by alabastervlog »

indigodaddy wrote: post #29
parliament32 wrote: post #22

Didn't even have to click through to the report in question to know it would be all hallucinations -- both the original patchfile and the segfault ("ngtcp2_http3_handle_priority_frame".. "There is no function named like this in current ngtcp2 or nghttp3.") I guess these guys don't bother to verify, they just blast out AI slop and hope one of them hits?

Reminds me of when some LLM (might have been Deepseek) told me I could add wasm_mode=True in my FastHTML python code which would allow me to compile it to WebAssembly, when of course there is no such feature in FastHTML. This was even when I had provided it full llms-ctx.txt

I had Google's in-search "AI" invent a command line switch that would have been very helpful... if it existed. Complete with usage caveats and warnings!

This was like two weeks ago. These things suck.

Re: Curl: We still have not seen a valid security report done with AI help

#46

Post by h4ck_th3_pl4n3t »

yoyohello13 wrote: post #25
jacksnipe wrote: post #15

Something that really frustrates me about interacting with (some) people who use AI a lot is that they will often tell me things that start “I asked ChatGPT and it said…” stop it!!! If the chatbot taught you something and you understood it, explain it to me. If you didn’t understand or didn’t trust it, then keep it to yourself!

Seriously. Being able to look up stuff using AI is not unique. I can do that too. This is kind of the same with any AI gen art. Like I can go generate a bunch of cool images with AI too, why should I give a shit about your random Midjourney output.

How can you be so harsh on all the new kids with Senior Prompt Engineer in their job titles?

They have to prove to someone that they're worth their money. /s

Re: Curl: We still have not seen a valid security report done with AI help

#47

Post by uludag »

I can imagine that most LLMs, if you ask it to find a security vulnerability in a given piece of code, will make something up completely out of the air. I've (mistakenly) sent valid code with an unrelated error and to this day I get nonsense "fixes" for these errors.

This alignment problem between responding with what the user wants (e.g. a security report, flattering responses) and going against the user seems a major problem limiting the effectiveness of such systems.

Re: Curl: We still have not seen a valid security report done with AI help

#48

Post by multjoy »

tough wrote: post #39
hx8 wrote: post #36

Earlier quoted context omitted.

It depends on if they are just repeating things without understanding, or if they have understanding. My issue is that people that say "I asked gpt" is that they often do not have any understanding themselves. Copy and pasting from ChatGPT has the same consequences as copying and pasting from StackOverflow, which is to say you're now on the hook supporting code in production that you don't understand.

We cannot blame the tools for how they are used by those yielding them. I can use ChatGPT to teach me and understand a topic or i can use it to give me an answer and not double check and just copy paste. Just shows off how much you care about the topic at hand, no?

How do you know that ChatGPT is teaching you about the topic? It doesn't know what is right or what is wrong.

Re: Curl: We still have not seen a valid security report done with AI help

#49

Post by emushack »

vessenes wrote: post #34

Reading the straw that broke the camel's back commit illustrates the problem really well: https://hackerone.com/reports/3125832 . This shit must be infuriating to dig through. I wonder if reputation systems might work here - you could give anyone who id's with an AML/KYC provider some reputation, enough for two or three reports, let people earn reputation digging through zero rep submissions and give someone like 10,…

Reputation systems for this kind of thing sounds like rubbing some anti-itch cream on bullet wound. I feel like the problem seems to me to be behavior, not a technology issue.

Personally I can't imagine how miserable it would be for my hard-earned expertise to be relegated to sifting through SLOP where maybe 1 in hundreds or even thousands of inquiries is worth any time at all. But it also doesn't seem prudent to just ignore them.

I don't think better ML/AI technology or better information systems will make a significant difference on this issue. It's fundamentally about trust in people.

Re: Curl: We still have not seen a valid security report done with AI help

#50

Post by stonemetal12 »

hashmush wrote: post #24
jacksnipe wrote: post #15

Something that really frustrates me about interacting with (some) people who use AI a lot is that they will often tell me things that start “I asked ChatGPT and it said…” stop it!!! If the chatbot taught you something and you understood it, explain it to me. If you didn’t understand or didn’t trust it, then keep it to yourself!

As much as I'm also annoyed by that phrase, is it really any different from: - I had to Google it... - According to a StackOverflow answer... - Person X told me about this nice trick... - etc. Stating your sources should surely not be a bad thing, no?

In general those point to the person's understanding being shallow. So far when someone says "GPT said..." it is a new low in understanding, and there is no more to the article they googled or second stackOverflow answer with a different take on it, it is the end of the conversation.
Post reply on HN