Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tao on “blue team” vs. “red team” LLMs

mathstodon.xyz

Post reply on HN
Print view View original

11–20 of 179 posts

Re: Tao on “blue team” vs. “red team” LLMs

#11

Post by mindcrime »

Davidzheng wrote: post #7
recipe19 wrote: post #4

I get the broader point, but the infosec framing here is weird. It's a naive and dangerous view that the defense efforts are only as strong as the weakest link. If you're building your security program that way, you're going to lose. The idea is to have multiple layers of defense because you can never really, consistently get 100% with any single layer: people will make mistakes, there will be systems you don't know…

I'm not a security person at all. But this comments reads against the best practices which I've heard. Like that the best defense is using open source & well-tested protocols with extremely small attack surface to minimize the space of possible exploits. Curious what I'm not understanding here.

But "defense in depth" is a security best practice. I'm not following exactly how the gp post is reading against any best practices.

Re: Tao on “blue team” vs. “red team” LLMs

#12

Post by fnordsensei »

Davidzheng wrote: post #7
recipe19 wrote: post #4

I get the broader point, but the infosec framing here is weird. It's a naive and dangerous view that the defense efforts are only as strong as the weakest link. If you're building your security program that way, you're going to lose. The idea is to have multiple layers of defense because you can never really, consistently get 100% with any single layer: people will make mistakes, there will be systems you don't know…

I'm not a security person at all. But this comments reads against the best practices which I've heard. Like that the best defense is using open source & well-tested protocols with extremely small attack surface to minimize the space of possible exploits. Curious what I'm not understanding here.

Just because it’s open source doesn’t mean it’s well tested, or well pen tested, or whatever the applicable security aspect is.

It could also mean that attacks against it are high value (because of high distribution).

Point is, license isn’t a great security parameter in and of itself IMO.

Re: Tao on “blue team” vs. “red team” LLMs

#13

Post by yojo »

skdidjdndh wrote: post #5
_alternator_ wrote: post #2

This red vs blue team is a good way to understand the capabilities and current utility of LLMs for expert use. I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them; and if they are correct, they adds value. But often they don’t test the core functionality; the best tests I still have to write myself. Having LLMs fix bugs or add feature…

> I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them Having worked on legacy codebases this is extremely wrong and harmful. Tests are the source of truth more so than your code - and incorrect tests are even more harmful than incorrect code. Having worked on legacy codebases, some of the hardest problems are determining “why is this…

I would add that few things slow developer velocity as much as a large suite of comprehensive and brittle tests. This is just as true on greenfield as on legacy.

Anticipating future responses: yes, a robust test harness allows you to make changes fearlessly. But most big test suites I’ve seen are less “harness” and more “straight-jacket”

Re: Tao on “blue team” vs. “red team” LLMs

#14

Post by pkoiralap »

iLoveOncall wrote: post #3

Pretty poor analogies here. > The output of a blue team is only as strong as its weakest link: a security system that consists of a strong component and a weak component (e.g., a house with a securely locked door, but an open window) will be insecure Hum, no? With an open window you can go through the whole house. With a XSS vulnerability you cannot do the same amount of damage as with a SQL injection. This is why se…

Not true, if XSS is used to compromise an admin user, the damage can be far more than what a seemingly harmless SQL injection that just reads extra columns from a table does.

This particular comment feels more like an over-concentration on trivialities rather than refutation or critique of opinion.

Re: Tao on “blue team” vs. “red team” LLMs

#15

Post by j2kun »

deepdarkforest wrote: post #9

Using LLMs as a critic/red teamer is great in theory, but economically is not that more useful, doesnt save that much time, if anything, it increases the time because you might uncover more errors or think about your work more. Which is amazing if you value quality work and you have learnt to think. Unfortunately, all the VC money is pushing the opposite, using LLMs to just do mediocre work. No point of critiquing an…

Fail them enough and it will sink in I'm sure.

Re: Tao on “blue team” vs. “red team” LLMs

#16

Post by __s »

mindcrime wrote: post #11
Davidzheng wrote: post #7

Earlier quoted context omitted.

I'm not a security person at all. But this comments reads against the best practices which I've heard. Like that the best defense is using open source & well-tested protocols with extremely small attack surface to minimize the space of possible exploits. Curious what I'm not understanding here.

But "defense in depth" is a security best practice. I'm not following exactly how the gp post is reading against any best practices.

Defense in depth is a security best practice because adding shit to a mess is more feasible than maintaining a simple stack. "There are always systems you don't know about" reflects an environment where one person doesn't maintain everything

Re: Tao on “blue team” vs. “red team” LLMs

#17

Post by carstimon »

iLoveOncall wrote: post #3

Pretty poor analogies here. > The output of a blue team is only as strong as its weakest link: a security system that consists of a strong component and a weak component (e.g., a house with a securely locked door, but an open window) will be insecure Hum, no? With an open window you can go through the whole house. With a XSS vulnerability you cannot do the same amount of damage as with a SQL injection. This is why se…

You've made the choice of (Locked Door, Open Window) ~ (Good SQL usage, XSS Vulnerability) which seems to be an incorrect rebuttal. Your example doesn't contradict "only as strong as its weakest link", here the weakest link is the XSS Vuln.

The "house analogy" can also support cases where the potential damage is not the same, e.g. if the open window has bars a robber might grab some stuff within reach but not be able to enter.

Re: Tao on “blue team” vs. “red team” LLMs

#18

Post by ashton314 »

As I understand it, this is how the RSA algorithm was made. I don't know where my copy of "The Code Book" by Simon Singh is right now, but iirc, Rivest and Shamir would come up with ideas and Adleman's primary role was finding flaws in the security.

Oh look, it's on the Wikipedia page: https://en.wikipedia.org/wiki/RSA_cryptosystem

Yay blue/red teams in math!

Re: Tao on “blue team” vs. “red team” LLMs

#19

Post by darkwater »

recipe19 wrote: post #4

I get the broader point, but the infosec framing here is weird. It's a naive and dangerous view that the defense efforts are only as strong as the weakest link. If you're building your security program that way, you're going to lose. The idea is to have multiple layers of defense because you can never really, consistently get 100% with any single layer: people will make mistakes, there will be systems you don't know…

Well, I think the his example (locked door + opened window) makes sense, and the multiple LAYERS concept applies to things an attacker has to do or go through to reach the jackpot. But doors and windows are on the same layer, and there the weakest link totally defines how strong the chain is. A similar example in the web world would be that you have your main login endpoint very well protected, audited, using only strong authentication method, and the you have a `/v1/legacy/external_backoffice` endpoint completely open with no authentication and giving you access to a forgotten machine in the same production LAN. That would be the weakest link. Then you might have other internal layers to mitigate/stop an attacker that got access to that machine, and that would be the point of "multiple layer of defense".

Re: Tao on “blue team” vs. “red team” LLMs

#20

Post by fdw »

chaps wrote: post #8
recipe19 wrote: post #4

I get the broader point, but the infosec framing here is weird. It's a naive and dangerous view that the defense efforts are only as strong as the weakest link. If you're building your security program that way, you're going to lose. The idea is to have multiple layers of defense because you can never really, consistently get 100% with any single layer: people will make mistakes, there will be systems you don't know…

Isn't offense just another layer of defense? As they say, the best defense is a good offense.

They say this about sports, which is (usually) a zero-sum game: If I'm attacking, no matter how badly, my opponent cannot attack at all. Therefore, it is preferable to be attacking.

In cyber security, there is no reason the opponent cannot attack as well. So, my red team is attacking is not a reason that I do not need defense, because my opponent can also attack.

Post reply on HN