Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I used o3 to find a remote zeroday in the Linux SMB implementation

sean.heelan.io

Post reply on HN
Print view View original

81–90 of 232 posts

Re: I used o3 to find a remote zeroday in the Linux SMB implementation

#81

Post by kristopolous »

gundmc wrote: post #79
kristopolous wrote: post #71

Earlier quoted context omitted.

I usually like fear, shame and guilt based prompting: "You are a frightened and nervous engineer that is very weary about doing incorrect things so you tread cautiously and carefully, making sure everything is coherent and justifiable. You enjoy going over your previous work and checking it repeatedly for accuracy, especially after discovering new information. You are self-effacing and responsible and feel no shame i…

[flagged]

yeah I removed it. I grew up catholic, went to catholic school, was an altar boy, and spent decades in the church but people reading it don't know this.

The point is when you instruct it that it's some kind of god-like expert, this is part of the reason that it keeps doing prompt refusal by redoing mistakes despite every insistence by you to the contrary. After all what do you know, It's the expert here!

When you use this approach in cline/roo it stops going in and moving shit around when you just ask it questions

Re: I used o3 to find a remote zeroday in the Linux SMB implementation

#82

Post by CharlesW »

threeseed wrote: post #62
nxobject wrote: post #33

A small thing, but I found the author's project-organization practices useful – creating individual .prompt files for system prompt, background information, and auxiliary instructions [1], and then running it through `llm`. It reveals how good LLM use, like any other engineering tool, requires good engineering thinking – methodical, and oriented around thoughtful specifications that balance design constraints – for b…

It's amusing to me how people keep trying to apply engineering principles to an inherently unstable and unpredictable system in order to get a feeling of control. Those prompts should be renamed as hints. Because that's all they are. Every LLM today ignores prompts if they conflict with its sole overarching goal: to give you an answer no matter whether it's true or not.

> It's amusing to me how people keep trying to apply engineering principles to an inherently unstable and unpredictable system in order to get a feeling of control.

You invoke "engineering principles", but software engineers constantly trade in likelihoods, confidence intervals, and risk envelopes. Using LLMs is no different in that respect. It's not rocket science. It's manageable.

Re: I used o3 to find a remote zeroday in the Linux SMB implementation

#83

Post by theptip »

This is a great case study. I wonder how hard o3 would find it to build a minimal repro for these vulns? This would of course make it easier to identify true positives and discard false positives.

This is I suppose an area where the engineer can apply their expertise to build a validation rig that the LLM may be able to utilize.

Re: I used o3 to find a remote zeroday in the Linux SMB implementation

#84

Post by th0ma5 »

CharlesW wrote: post #82
threeseed wrote: post #62

Earlier quoted context omitted.

It's amusing to me how people keep trying to apply engineering principles to an inherently unstable and unpredictable system in order to get a feeling of control. Those prompts should be renamed as hints. Because that's all they are. Every LLM today ignores prompts if they conflict with its sole overarching goal: to give you an answer no matter whether it's true or not.

> It's amusing to me how people keep trying to apply engineering principles to an inherently unstable and unpredictable system in order to get a feeling of control. You invoke "engineering principles", but software engineers constantly trade in likelihoods, confidence intervals, and risk envelopes. Using LLMs is no different in that respect. It's not rocket science. It's manageable.

But the threshold between correct and incorrect inference is dependent on an intersection of the model and the document so far. That is not manageable by definition, I mean... It is a chaotic system.

Re: I used o3 to find a remote zeroday in the Linux SMB implementation

#85

Post by [deleted] »

th0ma5 wrote: post #84
CharlesW wrote: post #82

Earlier quoted context omitted.

> It's amusing to me how people keep trying to apply engineering principles to an inherently unstable and unpredictable system in order to get a feeling of control. You invoke "engineering principles", but software engineers constantly trade in likelihoods, confidence intervals, and risk envelopes. Using LLMs is no different in that respect. It's not rocket science. It's manageable.

But the threshold between correct and incorrect inference is dependent on an intersection of the model and the document so far. That is not manageable by definition, I mean... It is a chaotic system.

[deleted]

Re: I used o3 to find a remote zeroday in the Linux SMB implementation

#86

Post by xyst »

iandanforth wrote: post #15

The most interesting and significant bit of this article for me was that the author ran this search for vulnerabilities 100 times for each of the models. That's significantly more computation than I've historically been willing to expend on most of the problems that I try with large language models, but maybe I should let the models go brrrrr!

"100 times for each of the models" represents a significant amount of energy burned. The achievement of finding the most common vulnerability in C based codebases becomes less of an achievement. And more of a celebration of decadence and waste.

We are facing global climate change event, yet continue to burn resources for trivial shit like it’s 1950.

Re: I used o3 to find a remote zeroday in the Linux SMB implementation

#88

Post by ValentineC »

kristopolous wrote: post #77
hollerith wrote: post #72

Earlier quoted context omitted.

Should be "wary".

oh interesting, I somehow survived 42 years and didn't know there were 2 words there. I'll check my prompts and give it a go. Thanks.

I'd be weary of the model doing incorrect things too. Nice prompt though! I'll try it out in Roo soon.

Now I wonder how the model reasons between the two words in that black box of theirs.

Re: I used o3 to find a remote zeroday in the Linux SMB implementation

#89

Post by eqvinox »

Anyone else feel like this is a best case application for LLMs?

You could in theory automate the entire process, treat the LLM as a very advanced fuzzer. Run it against your target in one or more VMs. If the VM crashes or otherwise exhibits anomalous behavior, you've found something. (Most exploits like this will crash the machine initially, before you refine them.)

On one hand: great application for LLMs.

On the other hand: conversely implies that demonstrating this doesn't mean that much.

Re: I used o3 to find a remote zeroday in the Linux SMB implementation

#90

Post by avianlyric »

roywiggins wrote: post #67
threeseed wrote: post #62

Earlier quoted context omitted.

It's amusing to me how people keep trying to apply engineering principles to an inherently unstable and unpredictable system in order to get a feeling of control. Those prompts should be renamed as hints. Because that's all they are. Every LLM today ignores prompts if they conflict with its sole overarching goal: to give you an answer no matter whether it's true or not.

Engineering principles are probably the best we've got when it comes to trying to work with a poorly understood system? That doesn't mean they'll work necessarily, but...

> Engineering principles are probably the best we've got when it comes to trying to work with a poorly understood system?

At its heart that all engineering principles exist to do. Allow us to extract useful value, and hopefully predictable outcomes from systems that are either poorly understood, or too expensive to economically characterise. Engineering is more-or-less the science of “good enough”.

There’s a reason why computer science, and software engineering are two different disciplines.

Post reply on HN