Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tao on “blue team” vs. “red team” LLMs

mathstodon.xyz

Post reply on HN
Print view View original

61–70 of 179 posts

Re: Tao on “blue team” vs. “red team” LLMs

#61

Post by [deleted] »

andruby wrote: post #44
manmal wrote: post #33

Earlier quoted context omitted.

> Tests are the source of truth more so than your code Tests poke and prod with a stick at the SUT, and the SUT's behaviour is observed. The truth lives in the code, the documentation, and, unfortunately, in the heads of the dev team. I think this distinction is quite important, because this question: > Do we have a bug? Or do we have a bad test? cannot be answered by looking at the test + the implementation. The spe…

What does SUT stand for? I'm not familiar with the acronym Is it "System Under Test"? (That's Claude.ai's guess)

[deleted]

Re: Tao on “blue team” vs. “red team” LLMs

#62

Post by winstonewert »

andrepd wrote: post #59
yojo wrote: post #13

Earlier quoted context omitted.

I would add that few things slow developer velocity as much as a large suite of comprehensive and brittle tests. This is just as true on greenfield as on legacy. Anticipating future responses: yes, a robust test harness allows you to make changes fearlessly. But most big test suites I’ve seen are less “harness” and more “straight-jacket”

I don't understand this. How does it slow your development if the tests being green is a necessary condition for the code being correct? Yes it slows it compared to just writing incorrect code lol, but that's not the point.

The problem is that sometimes it is not a necessary condition. Rather, the tests might have been checking implementation details or just been wrong in the first place. Now, when tests fails I have extra work to figure out if its a real break or just a bad test.

Re: Tao on “blue team” vs. “red team” LLMs

#63

Post by sour-taste »

yojo wrote: post #13
skdidjdndh wrote: post #5

Earlier quoted context omitted.

> I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them Having worked on legacy codebases this is extremely wrong and harmful. Tests are the source of truth more so than your code - and incorrect tests are even more harmful than incorrect code. Having worked on legacy codebases, some of the hardest problems are determining “why is this…

I would add that few things slow developer velocity as much as a large suite of comprehensive and brittle tests. This is just as true on greenfield as on legacy. Anticipating future responses: yes, a robust test harness allows you to make changes fearlessly. But most big test suites I’ve seen are less “harness” and more “straight-jacket”

An old coworker used to call these types of tests change detector tests. They are excellent at telling you whether some behavior changed, but horrible at telling you whether that behavior change is meaningful or not.

Re: Tao on “blue team” vs. “red team” LLMs

#64

Post by threatofrain »

andrepd wrote: post #59
yojo wrote: post #13

Earlier quoted context omitted.

I would add that few things slow developer velocity as much as a large suite of comprehensive and brittle tests. This is just as true on greenfield as on legacy. Anticipating future responses: yes, a robust test harness allows you to make changes fearlessly. But most big test suites I’ve seen are less “harness” and more “straight-jacket”

I don't understand this. How does it slow your development if the tests being green is a necessary condition for the code being correct? Yes it slows it compared to just writing incorrect code lol, but that's not the point.

It's that hard to write specs that truly match the business, hence why test-driven-development or specification-first failed to take off as a movement.

Asking specs to truly match the business before we begin using them as tests would handcuff test people in the same way we're saying that tests have the potential to handcuff app and business logic people — as opposed to empowering them. So I wouldn't blame people for writing specs that only match the code implementation at that time. It's hard to engage in prophecy.

Re: Tao on “blue team” vs. “red team” LLMs

#65

Post by hiq »

What about formal proofs? Don't we expect LLMs to help there, in a more "blue team" role? E.g. when a mathematician talks about a "technical proof", enumerating cases in the thousands, my impression is that LLM would save some time, and potentially help mathematicians focus on the actually hard (rather than tedious) parts.

Re: Tao on “blue team” vs. “red team” LLMs

#66

Post by chubot »

I made this point a few months ago here, but using the words attacker and defender (builder) rather than red team and blue team: https://lobste.rs/s/i2edlt/how_i_use_ai

The asymmetry is:

An attacker only has to be right ONCE, and he wins

Conversely, the defender only has to be wrong once, and he is wrong.

So the conclusion is:

Defenders/creators are using LLMs to pump out crappy code, and not testing enough, or relying on the LLM to test itself.

Some attackers might be too dismissive of LLMs, and could accelerate their work by using them to try more things

The comment was related to these stories:

How I Use AI (11 months ago) - https://news.ycombinator.com/item?id=41150317

Carlini has the fairly rare job of being an attacker: Why I Attack - https://nicholas.carlini.com/writing/2024/why-i-attack.html

Re: Tao on “blue team” vs. “red team” LLMs

#67

Post by lanstin »

darkwater wrote: post #19
recipe19 wrote: post #4

I get the broader point, but the infosec framing here is weird. It's a naive and dangerous view that the defense efforts are only as strong as the weakest link. If you're building your security program that way, you're going to lose. The idea is to have multiple layers of defense because you can never really, consistently get 100% with any single layer: people will make mistakes, there will be systems you don't know…

Well, I think the his example (locked door + opened window) makes sense, and the multiple LAYERS concept applies to things an attacker has to do or go through to reach the jackpot. But doors and windows are on the same layer, and there the weakest link totally defines how strong the chain is. A similar example in the web world would be that you have your main login endpoint very well protected, audited, using only st…

Or a single logging jar that will execute some of its message contents. Inside all your DMZ layers in the app content.

Re: Tao on “blue team” vs. “red team” LLMs

#68

Post by LPisGood »

hiq wrote: post #65

What about formal proofs? Don't we expect LLMs to help there, in a more "blue team" role? E.g. when a mathematician talks about a "technical proof", enumerating cases in the thousands, my impression is that LLM would save some time, and potentially help mathematicians focus on the actually hard (rather than tedious) parts.

Formal verification and case automatikn can be done automatically anyway without a mathematician hand checking each case.

For an old example that predates LLMs, see the four color theorem.

Re: Tao on “blue team” vs. “red team” LLMs

#69

Post by munificent »

9rx wrote: post #49
manmal wrote: post #33

Earlier quoted context omitted.

> Tests are the source of truth more so than your code Tests poke and prod with a stick at the SUT, and the SUT's behaviour is observed. The truth lives in the code, the documentation, and, unfortunately, in the heads of the dev team. I think this distinction is quite important, because this question: > Do we have a bug? Or do we have a bad test? cannot be answered by looking at the test + the implementation. The spe…

> The spec The tests are your spec. They exist precisely to document what the program is supposed to do for other humans, with the secondary benefit of also telling a machine what the program is supposed to do, allowing implementations to automatically validate themselves against the spec. If you find yourself writing specs and tests as independent things, that's how you end up with bad, brittle tests that make devel…

Unfortunately, tests can never be a complete specification unless the system is simple enough to have a finite set of possible inputs.

For all real-world software, a test suite tests a number of points in the space of possible inputs and we hope that those points generalize to pinning down the overall behavior of the implementation.

But there's no guarantee of that generalization. An implementation that fails a test is guaranteed to not implement the spec, but an implementation that passes all of the tests is not guaranteed to implement it.

Re: Tao on “blue team” vs. “red team” LLMs

#70

Post by javier_e06 »

In cybersecurity red and blue test are two equal forces. In software development the analogy I think is a stretch, coding and testing are not two equal forces. Test is code too, and as such, it has bugs too. Test runs afoul with police paradox: Who polices the police? The Police police the police.
Post reply on HN