Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tao on “blue team” vs. “red team” LLMs

mathstodon.xyz

Post reply on HN
Print view View original

41–50 of 179 posts

Re: Tao on “blue team” vs. “red team” LLMs

#41

Post by jeffrallen »

My experience with a really clever agentic workflow (I use sketch.dev) is that the LLM is playing both blue and red team. If I give a good spec, it will make the thing I'm asking for, and then it will test it better than I would have done myself (partly because it's more clever than me, but mostly because it's way harder working than I am, or rather it puts more effort into testing that I would be able to do with the time leftover after writing the thing).

Also, I cam ask it to do security reviews on the system it's made and it works with it's same characteristic fervor.

I love Tao's observation, but I disagree, at least for the domains I'm allowing LLMs to creat for, that they should not play both teams.

Re: Tao on “blue team” vs. “red team” LLMs

#42

Post by chaps »

fdw wrote: post #20
chaps wrote: post #8

Earlier quoted context omitted.

Isn't offense just another layer of defense? As they say, the best defense is a good offense.

They say this about sports, which is (usually) a zero-sum game: If I'm attacking, no matter how badly, my opponent cannot attack at all. Therefore, it is preferable to be attacking. In cyber security, there is no reason the opponent cannot attack as well. So, my red team is attacking is not a reason that I do not need defense, because my opponent can also attack.

My post was really was in the context of real-time strategy games. It's very, very possible to attack and defend at the same time no matter the skill of either side. Offense and defense aren't mutually exclusive, which is kinda the point of my post.

Re: Tao on “blue team” vs. “red team” LLMs

#43

Post by some_random »

This is an interesting discussion intellectually but it ignores the reality of cybersecurity. Yes I agree that AI tools best fit the red team role HOWEVER the reality is that the place that needs the most help is on the blue team and indeed this is where we see the biggest uplift from AI tools. To extend the "defend a house" metaphor, the previous state of security tooling was that an alert would be sent to the SOC every time any motion was detected on the cameras, leading to alert fatigue and increasing the time between a true positive alert being fired and it being escalated. Now add some CV in which tries to categorize those motion detection alerts into a few buckets, "person spotted", "car pulled up", "branch moved", "cat came home", etc and suddenly you go from having a thousand alerts to review a day to fifty.

Re: Tao on “blue team” vs. “red team” LLMs

#44

Post by andruby »

manmal wrote: post #33
skdidjdndh wrote: post #5

Earlier quoted context omitted.

> I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them Having worked on legacy codebases this is extremely wrong and harmful. Tests are the source of truth more so than your code - and incorrect tests are even more harmful than incorrect code. Having worked on legacy codebases, some of the hardest problems are determining “why is this…

> Tests are the source of truth more so than your code Tests poke and prod with a stick at the SUT, and the SUT's behaviour is observed. The truth lives in the code, the documentation, and, unfortunately, in the heads of the dev team. I think this distinction is quite important, because this question: > Do we have a bug? Or do we have a bad test? cannot be answered by looking at the test + the implementation. The spe…

What does SUT stand for? I'm not familiar with the acronym

Is it "System Under Test"? (That's Claude.ai's guess)

Re: Tao on “blue team” vs. “red team” LLMs

#45

Post by griffzhowl »

ashton314 wrote: post #18

As I understand it, this is how the RSA algorithm was made. I don't know where my copy of "The Code Book" by Simon Singh is right now, but iirc, Rivest and Shamir would come up with ideas and Adleman's primary role was finding flaws in the security. Oh look, it's on the Wikipedia page: https://en.wikipedia.org/wiki/RSA_cryptosystem Yay blue/red teams in math!

Reminds me of a pair of cognitive scientists I know who often collaborate. One is expansive and verbose and often gets carried away on tangential trains of thought, the other is very logical and precise. Their way of producing papers is the first one writes and the second deletes.

Re: Tao on “blue team” vs. “red team” LLMs

#46

Post by dfabulich »

andruby wrote: post #44
manmal wrote: post #33

Earlier quoted context omitted.

> Tests are the source of truth more so than your code Tests poke and prod with a stick at the SUT, and the SUT's behaviour is observed. The truth lives in the code, the documentation, and, unfortunately, in the heads of the dev team. I think this distinction is quite important, because this question: > Do we have a bug? Or do we have a bad test? cannot be answered by looking at the test + the implementation. The spe…

What does SUT stand for? I'm not familiar with the acronym Is it "System Under Test"? (That's Claude.ai's guess)

It is.

Re: Tao on “blue team” vs. “red team” LLMs

#47

Post by card_zero »

andruby wrote: post #44
manmal wrote: post #33

Earlier quoted context omitted.

> Tests are the source of truth more so than your code Tests poke and prod with a stick at the SUT, and the SUT's behaviour is observed. The truth lives in the code, the documentation, and, unfortunately, in the heads of the dev team. I think this distinction is quite important, because this question: > Do we have a bug? Or do we have a bad test? cannot be answered by looking at the test + the implementation. The spe…

What does SUT stand for? I'm not familiar with the acronym Is it "System Under Test"? (That's Claude.ai's guess)

That's what Wiktionary says too. Lucky guess, Claude.

Re: Tao on “blue team” vs. “red team” LLMs

#49

Post by 9rx »

manmal wrote: post #33
skdidjdndh wrote: post #5

Earlier quoted context omitted.

> I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them Having worked on legacy codebases this is extremely wrong and harmful. Tests are the source of truth more so than your code - and incorrect tests are even more harmful than incorrect code. Having worked on legacy codebases, some of the hardest problems are determining “why is this…

> Tests are the source of truth more so than your code Tests poke and prod with a stick at the SUT, and the SUT's behaviour is observed. The truth lives in the code, the documentation, and, unfortunately, in the heads of the dev team. I think this distinction is quite important, because this question: > Do we have a bug? Or do we have a bad test? cannot be answered by looking at the test + the implementation. The spe…

> The spec

The tests are your spec. They exist precisely to document what the program is supposed to do for other humans, with the secondary benefit of also telling a machine what the program is supposed to do, allowing implementations to automatically validate themselves against the spec. If you find yourself writing specs and tests as independent things, that's how you end up with bad, brittle tests that make development a nightmare — or you simply like pointless busywork, I suppose.

But, yes, you may still have to consult a human if there is reason to believe the spec isn't accurate.

Re: Tao on “blue team” vs. “red team” LLMs

#50

Post by ozgrakkurt »

manmal wrote: post #34
ozgrakkurt wrote: post #22

Earlier quoted context omitted.

What do you think about leaning on fuzz testing and deriving unit tests from bugs found by fuzzing?

What kind of bugs do you find this way, besides missing sanitization?

You can use the fuzzer to generate test cases instead of writing test cases manually.

For example you can make it generate queries and data for a database and generate a list of operations and timings for the operations.

Then you can mix assertions into the test so you make sure everything is going as expected.

This is very useful because there can be many combinations of inputs and timings etc. and it tests basically everything for you without you needing to write a million unit tests

Post reply on HN