Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's open-source framework for AI-powered vulnerability discovery

github.com

Post reply on HN
Print view View original

61–70 of 177 posts

Re: Anthropic's open-source framework for AI-powered vulnerability discovery

#61

Post by drtz »

andhug wrote: post #59
redfloatplane wrote: post #46

Earlier quoted context omitted.

"Shop jigs" is a great way to put it. I think a lot of software has gone from being made for general use to extremely individualised use. Before the Age of AI, it took so much human effort to write something that solved your problem that you might often go the extra mile so that others could re-use it. Now, it takes almost no effort, so the software stays ungeneralised. Some of the incentive has changed, I think. Mos…

That’s an interesting way to say “code quality in the age of ai has gone out the window”

Are you suggesting that performing a specific task without unnecessary abstractions is indicative of poor quality?

Re: Anthropic's open-source framework for AI-powered vulnerability discovery

#62

Post by sofixa »

xerxes249 wrote: post #9
Analemma_ wrote: post #6

Earlier quoted context omitted.

I mean, you don't need to run it all the time, right? You do it once over your entire existing codebase to start and then once over the diff in your CI/CD pipeline when you make a new change. I'm sure it's not literally that simple but I doubt these need to churn 24/7/365 either.

In the Mythos blogpost they revealed to run the model like a 1000 times on the same code-base maybe with slightly different prompt or temperature. That suggests it will just be pay to win. If the 'attacker' spends more money/tokens than the 'defender' you will eventually be outclassed.

It's even worse, it's loot box style. Not pay to win, but pay to have the chance to win. The result will always be non-deterministic, so for some cases it can give you what you're looking for from the first time, or it can take 1000 tries.

Re: Anthropic's open-source framework for AI-powered vulnerability discovery

#64

Post by Retr0id »

zuzululu wrote: post #57
ryancw wrote: post #41

Earlier quoted context omitted.

As a woodworker, it’s a really nice analogy and beyond anything I’ve seen AI do.

No idea why people are so upset I genuinely thought his references using analogy was a typical AI slop comment that I'm used to seeing from chatgpt

Believe it or not, people have been making analogies since before AI

Re: Anthropic's open-source framework for AI-powered vulnerability discovery

#65

Post by derf_ »

richardbarosky wrote: post #8

To be sure, security is an amazing AI/LLM use case. A huge swath of the work is pattern matching known security issues against stuff that's very precise to analyze -- programming language text. Something that stands out is that for the strongest use cases, AI companies will prefer to sell the technique as a service rather than its raw output. For use cases where the output is less valuable, tokens are sold. If AI tok…

> If AI tokens were so magical in creating new value in developing software applications generally, they wouldn't be selling tokens directly.

If hardware were so magical in creating new value generally, TSMC would be designing the chips instead of selling fabrication as a service.

That is what US chip companies used to do, by the way (back when there was silicon in Silicon Valley, before they got their lunch eaten by Taiwan). If TSMC had to design all of the chips they fabricate now, they would be doing a lot less business. Conversely, if any other company that wanted to design a chip had to build their own cutting-edge fab first, NVIDIA would not exist.

Re: Anthropic's open-source framework for AI-powered vulnerability discovery

#66

Post by colmmacc »

redfloatplane wrote: post #46
tptacek wrote: post #18

The thing about things like this is that they're shop jigs. You can buy a crosscut sled if you really want to, but most woodworkers just make their own. It was a different situation 2 years ago, when there was significant cost to building your own harness (but then: you probably weren't doing AI vuln research 2 years ago). Today, I think your best bet is to look at something like this for ideas, and then just ask for…

"Shop jigs" is a great way to put it. I think a lot of software has gone from being made for general use to extremely individualised use. Before the Age of AI, it took so much human effort to write something that solved your problem that you might often go the extra mile so that others could re-use it. Now, it takes almost no effort, so the software stays ungeneralised. Some of the incentive has changed, I think. Mos…

Unless it is very specific to a proprietary product, craftspeople take their jigs with them from job to job, building up a personal library over a career. As a software developer I've always had a well-tuned IDE and shell config in a safe place.

Something I think about a lot is what is the equivalent for the software builders of today using AI tools? how do make these harnesses exportable and portable? You might think employers would be against this; make it more costly to leave. But I actually think most will favor this because it makes people more productive more quickly. But we have to find ways to normalize it and show that there are no security leaks in the process (like might make it in to a set of personal steering prompts).

Re: Anthropic's open-source framework for AI-powered vulnerability discovery

#67

Post by timacles »

Retr0id wrote: post #64
zuzululu wrote: post #57

Earlier quoted context omitted.

No idea why people are so upset I genuinely thought his references using analogy was a typical AI slop comment that I'm used to seeing from chatgpt

Believe it or not, people have been making analogies since before AI

They used to, they still do, but they used to too.

Re: Anthropic's open-source framework for AI-powered vulnerability discovery

#68

Post by [deleted] »

zuzululu wrote: post #57
ryancw wrote: post #41

Earlier quoted context omitted.

As a woodworker, it’s a really nice analogy and beyond anything I’ve seen AI do.

No idea why people are so upset I genuinely thought his references using analogy was a typical AI slop comment that I'm used to seeing from chatgpt

[deleted]

Re: Anthropic's open-source framework for AI-powered vulnerability discovery

#69

Post by chrisweekly »

Mtinie wrote: post #12
nikcub wrote: post #7

Earlier quoted context omitted.

It's becoming apparent that it requires more tokens to secure code than it does to write it May even be an order of magnitude more

In all seriousness, wasn’t that always the case? Writing bad code is relatively cheap. Ensuring code isn’t bad is the expensive part.

Sort of?

The definition of "bad" from a security PoV is rapidly expanding, in light of relatively new capabilities and increasingly cheap access to exploitable vulnerabilities.

Re: Anthropic's open-source framework for AI-powered vulnerability discovery

#70

Post by jorl17 »

tptacek wrote: post #18

The thing about things like this is that they're shop jigs. You can buy a crosscut sled if you really want to, but most woodworkers just make their own. It was a different situation 2 years ago, when there was significant cost to building your own harness (but then: you probably weren't doing AI vuln research 2 years ago). Today, I think your best bet is to look at something like this for ideas, and then just ask for…

This is exactly it.

I've said many times that I believe "using the computer will transparently involve having it write and run code for you" (and if you're not technical you won't even know it!). What you're saying goes in that direction as well.

I feel that it's often better for us to create purpose-built tools for our lives, and with every model release, the complexity of those tools grows.

These are really personal tools: they solve a problem that other people might have, but are very tied to your own specific way of working, and would be hard to explain or adapt to someone else. So: shop jigs.

I have about 10 custom scripts and programs that are like this -- I haven't felt like this since college! Back then I had all the time in the world to customize my setup...now I have agents!

In a way, I want to show this to all my friends, but whenever I mentally trace how that would go, I realize they wouldn't really understand a bunch of the quirks they have, because they are _my_ quirks. They're reasonably complex pieces of tech that solve my problems very well, which are themselves particular versions of broader problems, and which I (at least for now) have no interest in supporting.

It's so clear we're heading in this direction, and yet so many people still believe code will be for the elites. Maybe production-code...As for the rest, I think soon your mom and dad are going to have their computer running code it wrote to serve them. Security-wise it's scary, but it's exciting to think about!

Post reply on HN