Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tao on “blue team” vs. “red team” LLMs

mathstodon.xyz

Post reply on HN
Print view View original

51–60 of 179 posts

Re: Tao on “blue team” vs. “red team” LLMs

#51

Post by [deleted] »

andruby wrote: post #44
manmal wrote: post #33

Earlier quoted context omitted.

> Tests are the source of truth more so than your code Tests poke and prod with a stick at the SUT, and the SUT's behaviour is observed. The truth lives in the code, the documentation, and, unfortunately, in the heads of the dev team. I think this distinction is quite important, because this question: > Do we have a bug? Or do we have a bad test? cannot be answered by looking at the test + the implementation. The spe…

What does SUT stand for? I'm not familiar with the acronym Is it "System Under Test"? (That's Claude.ai's guess)

[deleted]

Re: Tao on “blue team” vs. “red team” LLMs

#53

Post by bgwalter »

some_random wrote: post #43

This is an interesting discussion intellectually but it ignores the reality of cybersecurity. Yes I agree that AI tools best fit the red team role HOWEVER the reality is that the place that needs the most help is on the blue team and indeed this is where we see the biggest uplift from AI tools. To extend the "defend a house" metaphor, the previous state of security tooling was that an alert would be sent to the SOC e…

Tao's blue team stands for generative "AI", the red team stands for critical/auditing "AI".

I have not seen any independent claim that generative "AI" makes programs safer or that generating supervising features as you suggest works.

For auditing "AI" I have seen one claim (not independent or using a public methodology) that auditing "AI" rakes in bug bounties.

Re: Tao on “blue team” vs. “red team” LLMs

#54

Post by Ensorceled »

iLoveOncall wrote: post #3

Pretty poor analogies here. > The output of a blue team is only as strong as its weakest link: a security system that consists of a strong component and a weak component (e.g., a house with a securely locked door, but an open window) will be insecure Hum, no? With an open window you can go through the whole house. With a XSS vulnerability you cannot do the same amount of damage as with a SQL injection. This is why se…

You can always find problems with analogies, analogies are intentionally simplified to allow readers to better understand difficult or nuanced ideas.

In this case you are criticizing an analogy meant to convey understanding of "weakest link" for not also imparting an understanding of "levels of severity".

Re: Tao on “blue team” vs. “red team” LLMs

#55

Post by tonetegeatinst »

1970-01-01 wrote: post #48

So if they are to be focused on attacking and defending, they are to be separated. This leaves us with an argument where you effectively dismiss purple teams as a hack.

Yes, I feel this author ignores the fact purple teams exist. That or he must not know about them.

In addition, red and purple teams end goal is to help the blue team at the end of the day to remedy the issues discovered.

Re: Tao on “blue team” vs. “red team” LLMs

#56

Post by cookiengineer »

manmal wrote: post #34
ozgrakkurt wrote: post #22

Earlier quoted context omitted.

What do you think about leaning on fuzz testing and deriving unit tests from bugs found by fuzzing?

What kind of bugs do you find this way, besides missing sanitization?

Pointer errors. Null pointer returns instead of using the correct types. Flow/state problems. Multithreading problems. I/O errors. Network errors. Parsing bugs... etc

Basically the whole world of bugs introduced by someone being a too smart C/C++ coder. You can battletest parsers quite nicely with fuzzers, because parsers often have multiple states that assume naive input data structures.

Re: Tao on “blue team” vs. “red team” LLMs

#57

Post by lightbendover »

andruby wrote: post #44
manmal wrote: post #33

Earlier quoted context omitted.

> Tests are the source of truth more so than your code Tests poke and prod with a stick at the SUT, and the SUT's behaviour is observed. The truth lives in the code, the documentation, and, unfortunately, in the heads of the dev team. I think this distinction is quite important, because this question: > Do we have a bug? Or do we have a bad test? cannot be answered by looking at the test + the implementation. The spe…

What does SUT stand for? I'm not familiar with the acronym Is it "System Under Test"? (That's Claude.ai's guess)

[dead]

Re: Tao on “blue team” vs. “red team” LLMs

#58

Post by jgalt212 »

skdidjdndh wrote: post #5
_alternator_ wrote: post #2

This red vs blue team is a good way to understand the capabilities and current utility of LLMs for expert use. I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them; and if they are correct, they adds value. But often they don’t test the core functionality; the best tests I still have to write myself. Having LLMs fix bugs or add feature…

> I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them Having worked on legacy codebases this is extremely wrong and harmful. Tests are the source of truth more so than your code - and incorrect tests are even more harmful than incorrect code. Having worked on legacy codebases, some of the hardest problems are determining “why is this…

> Having worked on legacy codebases this is extremely wrong and harmful. Tests are the source of truth more so than your code - and incorrect tests are even more harmful than incorrect code.

I hear you on this, but you can still use so long as these tests are not comingled with the tests generated by subject-matter experts. I'd treat them almost a fuzzers.

Re: Tao on “blue team” vs. “red team” LLMs

#59

Post by andrepd »

yojo wrote: post #13
skdidjdndh wrote: post #5

Earlier quoted context omitted.

> I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them Having worked on legacy codebases this is extremely wrong and harmful. Tests are the source of truth more so than your code - and incorrect tests are even more harmful than incorrect code. Having worked on legacy codebases, some of the hardest problems are determining “why is this…

I would add that few things slow developer velocity as much as a large suite of comprehensive and brittle tests. This is just as true on greenfield as on legacy. Anticipating future responses: yes, a robust test harness allows you to make changes fearlessly. But most big test suites I’ve seen are less “harness” and more “straight-jacket”

I don't understand this. How does it slow your development if the tests being green is a necessary condition for the code being correct? Yes it slows it compared to just writing incorrect code lol, but that's not the point.

Re: Tao on “blue team” vs. “red team” LLMs

#60

Post by yojo »

andrepd wrote: post #59
yojo wrote: post #13

Earlier quoted context omitted.

I would add that few things slow developer velocity as much as a large suite of comprehensive and brittle tests. This is just as true on greenfield as on legacy. Anticipating future responses: yes, a robust test harness allows you to make changes fearlessly. But most big test suites I’ve seen are less “harness” and more “straight-jacket”

I don't understand this. How does it slow your development if the tests being green is a necessary condition for the code being correct? Yes it slows it compared to just writing incorrect code lol, but that's not the point.

"Brittle" here means either:

1) your test is specific to the implementation at the time of writing, not the business logic you mean to enforce.

2) your test has non-deterministic behavior (more common in end-to-end tests) that cause it to fail some small percentage of the time on repeated runs.

At the extreme, these types of tests degenerate your suite into a "change detector," where any modification to the code-base is guaranteed to make one or more tests fail.

They slow you down because every code change also requires an equal or larger investment debugging the test suite, even if nothing actually "broke" from a functional perspective.

Using LLMs to litter your code-base with low-quality tests will not end well.

Post reply on HN