Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Curl: We still have not seen a valid security report done with AI help

linkedin.com

Post reply on HN
Print view View original

161–170 of 258 posts

Re: Curl: We still have not seen a valid security report done with AI help

#161

Post by bluGill »

nneonneo wrote: post #118
unsnap_biceps wrote: post #16

For those of you who don't want to click into linked in, https://hackerone.com/reports/3125832 is the latest example of a invalid curl report

Good god did they hallucinate the segmentation fault and the resulting GDB trace too? Given that the diffs don’t even apply and the functions don’t even exist, I guess the answer is yes - in which case, this is truly a new low for AI slop bug reports.

An real report would have a GDB trace that looks like that, so it isn't hard to create such a trace. Many of us could create a real looking GDB trace just as well by hand - it would be tedious, boring, and pointless but we could.

Re: Curl: We still have not seen a valid security report done with AI help

#162

Post by __turbobrew__ »

jacksnipe wrote: post #15

Something that really frustrates me about interacting with (some) people who use AI a lot is that they will often tell me things that start “I asked ChatGPT and it said…” stop it!!! If the chatbot taught you something and you understood it, explain it to me. If you didn’t understand or didn’t trust it, then keep it to yourself!

I had someone at work lead me down a wild goose chase because claude told them to do something which was outright wrong to solve some performance issues they were having in their app. I helped them do this migration and it turned put that claude’s suggestions made performance worse! I know for sure the time wasted on this task was not debited from the so called company productivity stats that come from AI usage.

Re: Curl: We still have not seen a valid security report done with AI help

#163

Post by bluGill »

harrisi wrote: post #131
unsnap_biceps wrote: post #16

For those of you who don't want to click into linked in, https://hackerone.com/reports/3125832 is the latest example of a invalid curl report

This is interesting because they've apparently made a couple thousand dollars reporting things to other companies. Is it just a case of a broken clock being right twice a day? Seems like a terrible use of everyone's time and money. I find it hard to believe a random person on the internet using ChatGPT is worth $1000.

$1000 is cheap... The real question is when will companies become wise to this scam?

Most companies make you fill in expense reports for every trivial purchase. It would be cheaper to just let employees take the cash - and most employees are honest enough. However the dishonest employee isn't why they do expense reports (there are other ways to catch dishonest employees). There used to be a scam where someone would just send a bill for "services" and those got paid often enough until companies realized the costs and started making everyone do the expense reports so they could track the little expenses.

Re: Curl: We still have not seen a valid security report done with AI help

#164

Post by kristopolous »

codr7 wrote: post #158
kristopolous wrote: post #144

Earlier quoted context omitted.

Comfyui workflows, fine-tuning models, keeping up with the latest arxiv papers, patching academic code to work with generative stacks, this stuff is grueling. Here's an example https://files.meiobit.com/wp-content/uploads/2024/11/22l0nqm... Being dismissive of AI art is like those people who dismiss electronic music because there's a drum machine. Doing things well still requires an immense amount of skill and exhaus…

Makes even less sense when you put it like that, why not invest that effort into your own skills instead?

It is somebody's own skill.

Photographers are not painters.

People who do modular synths aren't guitarists.

Technical DJing is quite different from tapping on a Spotify app on a smartphone.

Just because you've exclusively exposed yourself to crude implementations doesn't mean sophisticated ones don't exist.

Re: Curl: We still have not seen a valid security report done with AI help

#165

Post by kimixa »

hashmush wrote: post #24
jacksnipe wrote: post #15

Something that really frustrates me about interacting with (some) people who use AI a lot is that they will often tell me things that start “I asked ChatGPT and it said…” stop it!!! If the chatbot taught you something and you understood it, explain it to me. If you didn’t understand or didn’t trust it, then keep it to yourself!

As much as I'm also annoyed by that phrase, is it really any different from: - I had to Google it... - According to a StackOverflow answer... - Person X told me about this nice trick... - etc. Stating your sources should surely not be a bad thing, no?

It's a "source" that cannot be reproduced or actually referenced in any way.

And all the other examples will have a chain of "upstream" references, data and discussion.

I suppose you can use those same phrases to reference things without that, random "summaries" without references or research, "expert opinion" from someone without any experience in that sector, opinion pieces from similarly reputation-less people etc. but I'd say they're equally worthless as references as "According to GPT...", and should be treated similarly.

Re: Curl: We still have not seen a valid security report done with AI help

#166

Post by pacifika »

pixl97 wrote: post #73
parliament32 wrote: post #22

Didn't even have to click through to the report in question to know it would be all hallucinations -- both the original patchfile and the segfault ("ngtcp2_http3_handle_priority_frame".. "There is no function named like this in current ngtcp2 or nghttp3.") I guess these guys don't bother to verify, they just blast out AI slop and hope one of them hits?

>"ngtcp2_http3_handle_priority_frame" I wonder if you could use AI to classify the probability factor that something is AI bullshit and deprioritize it?

AI red tape.

Re: Curl: We still have not seen a valid security report done with AI help

#167

Post by sidewndr46 »

alabastervlog wrote: post #45
indigodaddy wrote: post #29

Earlier quoted context omitted.

Reminds me of when some LLM (might have been Deepseek) told me I could add wasm_mode=True in my FastHTML python code which would allow me to compile it to WebAssembly, when of course there is no such feature in FastHTML. This was even when I had provided it full llms-ctx.txt

I had Google's in-search "AI" invent a command line switch that would have been very helpful... if it existed. Complete with usage caveats and warnings! This was like two weeks ago. These things suck.

Isn't there a website that builds git man pages this way? By just stringing together random concepts into sentences that seem vaguely like something Git would implement. I thought it was silly and potentially harmful the first time I saw it. Apparently, it may have just been ahead of the curve.

Re: Curl: We still have not seen a valid security report done with AI help

#168

Post by terom »

nneonneo wrote: post #118
unsnap_biceps wrote: post #16

For those of you who don't want to click into linked in, https://hackerone.com/reports/3125832 is the latest example of a invalid curl report

Good god did they hallucinate the segmentation fault and the resulting GDB trace too? Given that the diffs don’t even apply and the functions don’t even exist, I guess the answer is yes - in which case, this is truly a new low for AI slop bug reports.

The git commit hashes in the diff are interesting: 1a2b3c4..d4e5f6a

I think my wetware pattern-matching brain spots a pattern there.

Re: Curl: We still have not seen a valid security report done with AI help

#169

Post by ToValueFunfetti »

jsheard wrote: post #57
evandrofisico wrote: post #53

Earlier quoted context omitted.

It is supremely annoying when i ask in a group if someone has experience with a tool or system and some idiot copies my question into some LLM and paste the answer. I can use the LLM just like anyone, if i'm asking for EXPERIENCE it is because I want the opinion of a human who actually had to deal with stuff like corner cases.

If it's not worth writing, it's not worth reading.

There's a lot of documentation out there that I've found was left unwritten but that I would have loved to read

Re: Curl: We still have not seen a valid security report done with AI help

#170

Post by tharant »

kevmo314 wrote: post #159
tharant wrote: post #149

Earlier quoted context omitted.

Is it possible that what happened was an impedance mismatch between you and the engineer such that they couldn’t grok what you told them but ChatGPT was able to describe it in a manner they could understand? Real-life experts (myself included, though I don’t claim to be an expert in much) sometimes have difficulty explaining domain-specific concepts to other folks; it’s not a flaw in anyone, folks just have different…

Whenever someone has done that to me, it's clear they didn't read the ChatGPT output either and were sending it to me as some sort of "look someone else thinks you're wrong".

Again, is it possible you and the other party have (perhaps significantly) different mental models of the domain—or maybe different perspectives of the issues involved? I get that folks can be contrarian (sadly, contrariness is probably my defining trait) but it seems unlikely that someone would argue that you’re wrong by using output they didn’t read. I see impedance mismatches regularly yet folks seem often to assume laziness/apathy/stupidity/pride is the reason for the mismatch. Best advice I ever received is “Assume folks are acting rationally, with good intention, and with a willingness to understand others.” — which for some reason, in my contrarian mind, fits oddly nicely with Hanlon’s razor but I tend to make weird connections like that.
Post reply on HN