Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

xbow.com

Post reply on HN
Print view View original

61–70 of 128 posts

Re: XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

#61

Post by peanut-walrus »

tptacek wrote: post #41
croes wrote: post #39

Earlier quoted context omitted.

Whether it is legit-finding is precisely what needs to be checked, but you’re at spot 1061. >130 resolved >303 were classified as Triaged >33 reports marked as new >125 remain pending >208 were marked as duplicates >209 as informative >36 not applicable 20% bind a lot of resources if you have a high input on submissions and the numbers will rise

I think some context I probably don't share with the rest of this thread is that the average quality of a Hacker One submission is incredibly low. Like however bad you think the median bounty submission is, it's worse; think "people threatening to take you to court for not paying them for their report that they can 'XSS' you with the Chrome developer console".

My favorite one I've seen is "open redirect when you change the domain name in the browser address bar". This was submitted twice several years apart by two different people.

Re: XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

#62

Post by bawolff »

ryandrake wrote: post #6

Receiving hundreds of AI generated bug reports would be so demoralizing and probably turn me off from maintaining an open source project forever. I think developers are going to eventually need tools to filter out slop. If you didn’t take the time to write it, why should I take the time to read it?

If you think the AI slop is demoralizing, you should see the human submissions bug bounties get.

There is a reason companies like hackerone exist - its because dealing with the submissions is terrible.

Re: XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

#63

Post by kiitos »

tptacek wrote: post #33
kiitos wrote: post #31

Earlier quoted context omitted.

In the way of what?

Getting more bugs fixed.

> Getting more bugs fixed.

OK.. but "getting more bugs fixed" isn't any kind of objective success metric for, well, anything, right?

It's fine if you want to use it as a KPI for your specific thing! But it's not like it's some global KPI for everyone?

Re: XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

#64

Post by dcminter »

ryandrake wrote: post #55
tptacek wrote: post #44

Earlier quoted context omitted.

These aren't like Github Issues reports; they're bug bounty programs, specifically stood up to soak up incoming reports from anonymous strangers looking to make money on their submissions, with the premise being that enough of those reports will drive specific security goals (the scope of each program is, for smart vendors, tailored to engineering goals they have internally) to make it worthwhile.

Got it! The financial incentive will probably turn out to be a double edged sword. Maybe in the pre-AI age, it’s By Design to drive those goals, but I bet the ability to automate submissions will inevitably alter the rules of these programs. I think within the next 5 years or so, we are going to see a societal pattern repeating: any program that rewards human ingenuity and input will become industrialized by AI to th…

I'm wary of a lot of AI stuff, but here:

> What used to be lone wolves or small groups of humans working on bounties will become truckloads of AI generated “stuff” trying to maximize revenue.

You're objecting to the wrong thing. The purpose of a bug bounty programme is not to provide a cottage industry for security artisans - it's to flush out security vulnerabilities.

There are reasonable objections to AI automation in this space, but this is not one of them.

Re: XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

#65

Post by wslh »

I'm looking forward to the LLM's ELI5 explanation. If I understand correctly, XBOW is genuinely moving the needle and pushing the state of the art.

Another great reading is [1](2024).

[1] "LLM and Bug Finding: Insights from a $2M Winning Team in the White House's AIxCC": https://news.ycombinator.com/item?id=41269791

Re: XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

#66

Post by hinterlands »

Xbow has really smart people working on it, so they're well-aware of the usual 30-second critiques that come up in this thread. For example, they take specific steps to eliminate false positives.

The #1 spot in the ranking is both more of a deal and less of a deal than it might appear. It's less of a deal in that HackerOne is an economic numbers game. There are countless programs you can sign up for, with varied difficulty levels and payouts. Most of them pay not a whole lot and don't attract top talent in the industry. Instead, they offer supplemental income to infosec-minded school-age kids in the developing world. So I wouldn't read this as "Xbow is the best bug hunter in the US". That's a bit of a marketing gimmick.

But this is also not a particularly meaningful objective. The problem is that there's a lot of low-hanging bugs that need squashing and it's hard to allocate sufficient resources to that. Top infosec talent doesn't want to do it (and there's not enough of it). Consulting companies can do it, but they inevitably end up stretching themselves too thin, so the coverage ends up being hit-and-miss. There's a huge market for tools that can find easy bugs cheaply and without too many false positives.

I personally don't doubt that LLMs and related techniques are well-tailored for this task, completely independent of whether they can outperform leading experts. But there are skeptics, so I think this is an important real-world result.

Re: XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

#67

Post by absurdo »

hinterlands wrote: post #66

Xbow has really smart people working on it, so they're well-aware of the usual 30-second critiques that come up in this thread. For example, they take specific steps to eliminate false positives. The #1 spot in the ranking is both more of a deal and less of a deal than it might appear. It's less of a deal in that HackerOne is an economic numbers game. There are countless programs you can sign up for, with varied diff…

> so they're well-aware of the usual 30-second critiques that come up in this thread.

Succinct description of HN. It’s a damn shame.

Re: XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

#68

Post by tecleandor »

tptacek wrote: post #35
tecleandor wrote: post #28

Earlier quoted context omitted.

Still they're sending hundreds of reports that are being refused because they are not following the rules of the bounties. So they better work on that.

If you thought human bounty program participants were generally following the rules, or that programs weren't swamped with slop already... at least these are actually pre-triaged vetted findings.

But I was hoping the idea wasn't "as there's a lot of sloppy posts, we're going to be sloppy too let's flood them". So, use the AI for something useful and at least grep the rules properly. That'd be neat.

Re: XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

#69

Post by tptacek »

kiitos wrote: post #63
tptacek wrote: post #33

Earlier quoted context omitted.

Getting more bugs fixed.

> Getting more bugs fixed. OK.. but "getting more bugs fixed" isn't any kind of objective success metric for, well, anything, right? It's fine if you want to use it as a KPI for your specific thing! But it's not like it's some global KPI for everyone?

It's very specifically the objective of a security bug bounty.

Re: XBOW, an autonomous penetration tester, has reached the top spot on HackerOne

#70

Post by normie3000 »

hinterlands wrote: post #66

Xbow has really smart people working on it, so they're well-aware of the usual 30-second critiques that come up in this thread. For example, they take specific steps to eliminate false positives. The #1 spot in the ranking is both more of a deal and less of a deal than it might appear. It's less of a deal in that HackerOne is an economic numbers game. There are countless programs you can sign up for, with varied diff…

> Top infosec talent doesn't want to do it (and there's not enough of it).

What is the top talent spending its time on?

Post reply on HN