Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

xbow.com

Post reply on HN
Print view View original

111–120 of 128 posts

Re: XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

#111

Post by eeeeeeehio »

moyix wrote: post #49
ryandrake wrote: post #6

Receiving hundreds of AI generated bug reports would be so demoralizing and probably turn me off from maintaining an open source project forever. I think developers are going to eventually need tools to filter out slop. If you didn’t take the time to write it, why should I take the time to read it?

All of these reports came with executable proof of the vulnerabilities – otherwise, as you say, you get flooded with hallucinated junk like the poor curl dev. This is one of the things that makes offensive security an actually good use case for AI – exploits serve as hard evidence that the LLM can't fake.

Is "proof of vulnerability" a marketing term, or do you actually claim that XBOW has a 0% false positive rate? (i.e. "all" reports come with a PoV, and this PoV "proves" there is a vulnerability?)

Re: XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

#113

Post by skeeter2020 »

Sytten wrote: post #87
hinterlands wrote: post #66

Xbow has really smart people working on it, so they're well-aware of the usual 30-second critiques that come up in this thread. For example, they take specific steps to eliminate false positives. The #1 spot in the ranking is both more of a deal and less of a deal than it might appear. It's less of a deal in that HackerOne is an economic numbers game. There are countless programs you can sign up for, with varied diff…

100% agree with OP, to make a living in BBH you can't go hunting on VDP program that don't pay anything all day. That means you will have a lot of low hanging fruits on those programs. I don't think LLM replace humans, they do free up time to do nicer tasks.

...which is exactly what technology advancements in our field have done since its inception, vs. the "this changes everything for everybody forever" narative that makes AI cheerleaders so exhausting.

Re: XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

#114

Post by billy99k »

t0mas88 wrote: post #72
ryandrake wrote: post #55

Earlier quoted context omitted.

Got it! The financial incentive will probably turn out to be a double edged sword. Maybe in the pre-AI age, it’s By Design to drive those goals, but I bet the ability to automate submissions will inevitably alter the rules of these programs. I think within the next 5 years or so, we are going to see a societal pattern repeating: any program that rewards human ingenuity and input will become industrialized by AI to th…

Might be fixable by adding a $ 100 submission fee that is returned when you're proving working exploit code. Would make the Curl team a lot of money.

I've been on Hackerone for almost 8 years and I think the problem with this is that too many companies won't pay for legitimate bugs, even when you have a working exploit.

I had one critical bug take 3 years to get a pay out. I had a full walkthrough with videos and report. The company kept stalling and at one point told me that because they completely had the app remade, they weren't going to pay me anything.

Hackerone doesn't really protect the researcher either. I was told multiple times that there was 'nothing they could do'.

I eventually got paid, but this is pretty normal behavior with regards to bug bounty. Too many companies use it for free security work.

Re: XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

#115

Post by keisborg »

«XBOW submitted nearly 1,060 vulnerabilities. All findings were fully automated, though our security team reviewed them pre-submission to comply with HackerOne’s policy on automated tools»

That seems a bit unethical. I’ve thought companies specifically deny usage of automated tools. A bit too late ey…?

Re: XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

#116

Post by billy99k »

While I think this is great progress, It will be a hard sell for many business owners. I've been in this space (infosec) for awhile now and customers can be very apprehensive about non-AI/humans looking for vulnerabilities on their networks/systems.

Re: XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

#117

Post by nickpsecurity »

moyix wrote: post #34
bgwalter wrote: post #26

"XBOW is an enterprise solution. If your company would like a demo, email us at info@xbow.com." Like any "AI" article, this is an ad. If you are willing to tolerate a high false positive rate, you can as well use Rational Purify or various analyzers.

You should come to my upcoming BlackHat talk on how we did this while avoiding false positives :D https://www.blackhat.com/us-25/briefings/schedule/#ai-agents...

"we"

I remember your work on seeding vulnerabilities into C programs. I didnt know you got into AI-assisted pentesting. I already have more confidence in the product. :)

Re: XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

#118

Post by ActorNightly »

martinald wrote: post #71

This does not surprise me. In a couple of 'legacy' open source projects I found DoS attacks within 10 minutes, with a working PoC. It crashed the server entirely. I suspect with more prompting it could have found RCE but it was an idle shower thought to try. While niche and not widely used; there are at least thousands of publicly available servers for each of these projects. I genuinely think this is one of the bigg…

Legacy code (especially C++) does suffer from a lot of bugs, however modern code is generally much better.

There is also a BIG hurdle between crashing something (which generally will be detected), versus RCE which requires a lot more work.

Re: XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

#119

Post by eddd-ddde »

jp0001 wrote: post #89

I want to know how much they made in bounties versus how much they spent on compute. The thing about bug bounties, the only way to win is to not play the game.

I think you have to factor in the training value. Each submission is a new data point that will only improve the overall performance.

Re: XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

#120

Post by 8200_unit »

keisborg wrote: post #115

«XBOW submitted nearly 1,060 vulnerabilities. All findings were fully automated, though our security team reviewed them pre-submission to comply with HackerOne’s policy on automated tools» That seems a bit unethical. I’ve thought companies specifically deny usage of automated tools. A bit too late ey…?

They acknowledge that in the article and all submissions are human reviewed before they are submitted.
Post reply on HN