Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Curl: We still have not seen a valid security report done with AI help

linkedin.com

Post reply on HN
Print view View original

141–150 of 258 posts

Re: Curl: We still have not seen a valid security report done with AI help

#141

Post by qmr »

esafak wrote: post #23
jacksnipe wrote: post #15

Something that really frustrates me about interacting with (some) people who use AI a lot is that they will often tell me things that start “I asked ChatGPT and it said…” stop it!!! If the chatbot taught you something and you understood it, explain it to me. If you didn’t understand or didn’t trust it, then keep it to yourself!

I had to deal with someone who tried to check in hallucinated code with the defense "I checked it with chatGPT!" If you're just parroting what you read, what is it that you do here?!

I hope you dealt with them by firing them.

Re: Curl: We still have not seen a valid security report done with AI help

#142

Post by zulban »

eestrada wrote: post #117
zulban wrote: post #89

Shame they need to put up with that spam. However, every big open source project has by now had good contributions with "AI help". Many millions of developers are using AI a little as a tool, like Google.

And that increase in LLM usage has resulted in an enormous increase of code duplications and code churn in said open source projects. Any benefit from new features implemented by LLMs is being offset by the tech debt caused by duplication and the maintenance burden of constantly reverting bad code (i.e. churn). https://arc.dev/talent-blog/impact-of-ai-on-code/

Yes. The internet has also created a ton of email spam but I wouldn't say "we've never seen a single valid contribution to our project that had internet help". Many millions of developers are using AI. Sometimes in a good way. When that results in a good MR, they likely don't even mention they used Google, or stackoverflow, or AI, they just submit.

Re: Curl: We still have not seen a valid security report done with AI help

#143

Post by _ea1k »

cogman10 wrote: post #60
jacksnipe wrote: post #15

Something that really frustrates me about interacting with (some) people who use AI a lot is that they will often tell me things that start “I asked ChatGPT and it said…” stop it!!! If the chatbot taught you something and you understood it, explain it to me. If you didn’t understand or didn’t trust it, then keep it to yourself!

I recently had this happen from a senior engineer. What's really frustrating is I TOLD them the issues and how to fix it. Instead of listening to what I told them, they plugged it into GPT and responded with "Oh, interesting this is what GPT says" (Which, spoiler, was similar but lacking from what I'd said). Meaning, instead of listening to a real-life expert in the company telling them how to handle the problem they…

I wonder if this is an indication that they didn't really understand what you said to begin with.

Re: Curl: We still have not seen a valid security report done with AI help

#144

Post by kristopolous »

yoyohello13 wrote: post #25
jacksnipe wrote: post #15

Something that really frustrates me about interacting with (some) people who use AI a lot is that they will often tell me things that start “I asked ChatGPT and it said…” stop it!!! If the chatbot taught you something and you understood it, explain it to me. If you didn’t understand or didn’t trust it, then keep it to yourself!

Seriously. Being able to look up stuff using AI is not unique. I can do that too. This is kind of the same with any AI gen art. Like I can go generate a bunch of cool images with AI too, why should I give a shit about your random Midjourney output.

Comfyui workflows, fine-tuning models, keeping up with the latest arxiv papers, patching academic code to work with generative stacks, this stuff is grueling.

Here's an example https://files.meiobit.com/wp-content/uploads/2024/11/22l0nqm...

Being dismissive of AI art is like those people who dismiss electronic music because there's a drum machine.

Doing things well still requires an immense amount of skill and exhaustive amount of effort. It's wildly complicated

Re: Curl: We still have not seen a valid security report done with AI help

#145

Post by esafak »

qmr wrote: post #141
esafak wrote: post #23

Earlier quoted context omitted.

I had to deal with someone who tried to check in hallucinated code with the defense "I checked it with chatGPT!" If you're just parroting what you read, what is it that you do here?!

I hope you dealt with them by firing them.

Yes, unfortunately. This was the last straw, not the first.

Re: Curl: We still have not seen a valid security report done with AI help

#146

Post by godelski »

jacksnipe wrote: post #15

Something that really frustrates me about interacting with (some) people who use AI a lot is that they will often tell me things that start “I asked ChatGPT and it said…” stop it!!! If the chatbot taught you something and you understood it, explain it to me. If you didn’t understand or didn’t trust it, then keep it to yourself!

  > Something that really frustrates me about interacting with
Something that frustrates me with LLMs is that they are optimized such that errors are as silent as possible.

It is just bad design. You want errors to be as loud as possible. So they can be traced and resolved. On the other hand, LLMs optimize human preference (or some proxy of this). While humans prefer accuracy, it would be naive to ignore all the other things that optimize this objective. Specifically, humans prefer answers that they don't know are wrong over those that they do know are wrong.

This doesn't make LLMs useless but certainly it should strongly inform how we use them. Frankly, you cannot trust outputs, so you have to verify. I think this is where there's a big divergence between LLM users (and non-users). Those that blindly trust and those that don't (extreme case is non-users). If you need to constantly verify AND recognize that verification is extra hard (because it is optimized to be invisible to you), it can create extra work, not less.

It really is two camps and I think it says a lot:

  - "Blindly" trust
  - "Trust" but verify
Wide range of opinions in these two camps, but I think it comes down to some threshold of default trust or default suspicion.

Re: Curl: We still have not seen a valid security report done with AI help

#147

Post by squigz »

orthecreedence wrote: post #91
delusional wrote: post #68

Earlier quoted context omitted.

I consider myself a left leaning soyboy, but this could be the outcome of too "nice" of a discourse. I won't advocate for toxicity, but I am considering if we bolster the self-image of idiots when we refuse to call them idiots. Because you're right, this is fundamentally a people problem, specifically we need people to filter this themselves. I don't know where the limit would go.

Shame is a useful social tool. It can be overused or underused, but it's still a tool and people like this should be made to publicly answer for their obnoxious and destructive behavior.

How?

Re: Curl: We still have not seen a valid security report done with AI help

#148

Post by rippleanxiously »

Frost1x wrote: post #129
jacksnipe wrote: post #15

Something that really frustrates me about interacting with (some) people who use AI a lot is that they will often tell me things that start “I asked ChatGPT and it said…” stop it!!! If the chatbot taught you something and you understood it, explain it to me. If you didn’t understand or didn’t trust it, then keep it to yourself!

I work in a corporate environment as I’m sure many others do. Many executives have it in their head that LLMs are this brand new efficiency gain they can pad profit margins with, so you should be using it for efficiency. There’s a lot of push for that, everywhere where I work. I see email blasts suggesting I should be using it, I get peers saying I should be using it, I get management suggesting I should use it to cu…

It just feels to me like a boss walking into a car mechanic's shop holding some random tool, walking up to a mechanic, and:

"Hey, whatcha doin?"

"Oh hi, yea, this car has a slight misfire on cyl 4, so I was just pulling one of the coilpacks to-"

"Yea alright, that's great. So hey! You _really_ need to use this tool. Trust me, it's gonna make your life so much easier"

"umm... that's a 3d printer. I don't really think-"

"Trust me! It's gonna 10x your work!"

...

I love the tech. It's the evangelists that don't seem to bother researching the tech beyond making an account and asking it to write a couple scripts that bug me. And then they proclaim it can replace a bunch of other stuff they don't/haven't ever bothered to research or understand.

Re: Curl: We still have not seen a valid security report done with AI help

#149

Post by tharant »

cogman10 wrote: post #60
jacksnipe wrote: post #15

Something that really frustrates me about interacting with (some) people who use AI a lot is that they will often tell me things that start “I asked ChatGPT and it said…” stop it!!! If the chatbot taught you something and you understood it, explain it to me. If you didn’t understand or didn’t trust it, then keep it to yourself!

I recently had this happen from a senior engineer. What's really frustrating is I TOLD them the issues and how to fix it. Instead of listening to what I told them, they plugged it into GPT and responded with "Oh, interesting this is what GPT says" (Which, spoiler, was similar but lacking from what I'd said). Meaning, instead of listening to a real-life expert in the company telling them how to handle the problem they…

Is it possible that what happened was an impedance mismatch between you and the engineer such that they couldn’t grok what you told them but ChatGPT was able to describe it in a manner they could understand? Real-life experts (myself included, though I don’t claim to be an expert in much) sometimes have difficulty explaining domain-specific concepts to other folks; it’s not a flaw in anyone, folks just have different ways of assembling mental models.

Re: Curl: We still have not seen a valid security report done with AI help

#150

Post by XorNot »

ModernMech wrote: post #74
layer8 wrote: post #72

Earlier quoted context omitted.

Nah, that’s different. Lmgtfy has nothing to do with experience, other than experience in googling. Lmgtfy applies to stuff that can expediently be googled.

In my experience, usually what people had done was take your question on a forum, go to lmgtfy, paste the exact words in and then link back to it. As if to say "See how easy that was? Why are you asking us when you could have just done that?" Yes is true there could have been a skill issue. But it could also be true that the person just wanted input from people rather than Google. So that's why I drew the connection.

In my experience what happened was the top hit for the question was a topical forum, with a lmgtfy link as a response to the exact question I'm googling.
Post reply on HN