Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tao on “blue team” vs. “red team” LLMs

mathstodon.xyz

Post reply on HN
Print view View original

141–150 of 179 posts

Re: Tao on “blue team” vs. “red team” LLMs

#141

Post by godelski »

9rx wrote: post #136
godelski wrote: post #129

Earlier quoted context omitted.

> different words for the same thing You're still misunderstanding and missing the layer of abstraction, which is what I'm (and others are) talking about We have 3 objects: doc, test, spec. How do you prove they are the same thing? You are arguing that they all point to the same address. I'm arguing they all have the same parent. I think it's pretty trivial to show that they aren't identical, so I'll give two example…

> We have 3 objects: doc, test, spec. How do you prove they are the same thing? You... don't? There is nothing good that can come from trying to understand crazy. Best to run away as fast as possible if you ever encounter this. > You are arguing that they all point to the same address. Oh? I did say if you document something the same way three different times (even if you give each time a different name, as if that s…

  > In reality, I "argued" both.
I mean it is hard to have this conversation because you will say that they are the same thing and then leverage the fact that they aren't while disagreeing with me but using nearly identical settings to my examples.

I mean if your argument is that a mallard (test) and a muscovy (docs) are both types of ducks but a mallard is not a muscovy and a muscovy is not a mallard, then I fail to see how we aren't on the same page. I can't put it any clearer than this: all mallards are ducks but not all ducks are mallards. In other words, a mallard is a duck, but it is not representative of all ducks. You can't look at a mallard and know everything there is to know about ducks. You'll be missing stuff. If you treat your mallard and duck as isomorphic you're going to land yourself into trouble, even if most (domesticated) ducks are mallards.

It isn't that complex and saying "don't be overly confident" isn't adding crazy amounts of complexity that is going to overwhelm yourself. It's simply a recognition that you can't write a perfect spec.

Look, munificent[0] is saying the same thing. So is Kinrany[1], and manmal[2]. Do you think we're all wrong? In exactly the same way?

Besides, this whole argument is literally a demonstration of our claim. If you could write a perfect spec you'd (and we'd) be communicating perfectly and there'd be no hangup. But if that were possible we wouldn't need to write code in programming languages in the first place![3]

[0] https://news.ycombinator.com/item?id=44713138

[1] https://news.ycombinator.com/item?id=44713314

[2] https://news.ycombinator.com/item?id=44712266

[3] https://www.cs.utexas.edu/~EWD/transcriptions/EWD06xx/EWD667...

Re: Tao on “blue team” vs. “red team” LLMs

#142

Post by yeahwhatever10 »

ants_everywhere wrote: post #120

I have a couple of thoughts here: (a) AI on both the "red" and "blue" teams is useful. Blue team is basically brain storming. (b) AlphaEvolve is an example of an explicit "red/blue team" approach in his sense, although they don't use those terms [0]. Tao was an advisor to that paper. (c) This is also reminiscent of the "verifier/falsifier" division of labor in game semantics. This may be the way he's actually thinkin…

How is the LLM in AlphaEvolve red team? All the LLM does is generate new code when prompted with examples. It doesn’t evaluate the code.

Re: Tao on “blue team” vs. “red team” LLMs

#143

Post by ethan_smith »

scoreandmore wrote: post #117

The first thing I did when I signed up for Claude was have it analyze my website for security holes. But it only recommended superficial changes, like the lifecycle of my JWTs. After reading this, I’m wondering if a prompt asking it to attack the website would be better than asking it where it should be beefed up. But I no longer pay for Claude, and I suspect it won’t give me instructions on how to attack something.…

Try framing your prompts as security assessments rather than attacks - ask the model to identify "potential vulnerabilities" or "security considerations" while providing specific technical details about your architecture.

Re: Tao on “blue team” vs. “red team” LLMs

#144

Post by ants_everywhere »

yeahwhatever10 wrote: post #142
ants_everywhere wrote: post #120

I have a couple of thoughts here: (a) AI on both the "red" and "blue" teams is useful. Blue team is basically brain storming. (b) AlphaEvolve is an example of an explicit "red/blue team" approach in his sense, although they don't use those terms [0]. Tao was an advisor to that paper. (c) This is also reminiscent of the "verifier/falsifier" division of labor in game semantics. This may be the way he's actually thinkin…

How is the LLM in AlphaEvolve red team? All the LLM does is generate new code when prompted with examples. It doesn’t evaluate the code.

From Tao's post, red team is characterized this way

> In my own personal experiments with AI, for instance, I have found it to be useful for providing additional feedback on some proposed text, argument, code, or slides that I have generated (including this current text).

In AlphaEvolve, different scoring mechanisms are discussed. One is evaluation of a fixed function. Another is evaluation by an LLM. In either case, the LLM takes the score as information and provides feedback on the proposed program, argument, code, etc.

An example is given in the paper

> The current model uses a simple ResNet architecture with only three ResNet blocks. We can improve its performance by increasing the model capacity and adding regularization. This will allow the model to learn more complex features and generalize better to unseen data. We also add weight decay to the optimizer to further regularize the model and prevent overfitting. AdamW is generally a better choice than Adam, especially with weight decay.

It then also generates code, which is something he considers blue team.

More generally, using AI as blue team and red team is conceptually similar to a kind of actor/critic algorithm

Re: Tao on “blue team” vs. “red team” LLMs

#145

Post by [deleted] »

quacksilver wrote: post #137
manmal wrote: post #134

Earlier quoted context omitted.

That sounds a bit like using a jackhammer to drive in a nail. Wouldn’t it be smarter to enumerate edge cases and test all permutations of those?

Would it even be possible to enumerate all edge cases and test all the permutations of them in non-trivial codebases or interconnected systems? How do you know when you have all of the edge cases? With fuzzing you can randomly generate bad input that passes all of your test cases that were written using by whatever method you have already been using but still causes the application to crash or behave badly. This may…

[deleted]

Re: Tao on “blue team” vs. “red team” LLMs

#146

Post by manmal »

quacksilver wrote: post #137
manmal wrote: post #134

Earlier quoted context omitted.

That sounds a bit like using a jackhammer to drive in a nail. Wouldn’t it be smarter to enumerate edge cases and test all permutations of those?

Would it even be possible to enumerate all edge cases and test all the permutations of them in non-trivial codebases or interconnected systems? How do you know when you have all of the edge cases? With fuzzing you can randomly generate bad input that passes all of your test cases that were written using by whatever method you have already been using but still causes the application to crash or behave badly. This may…

Thanks for elaborating, I might start fuzzing.

Re: Tao on “blue team” vs. “red team” LLMs

#147

Post by 0xDEAFBEAD »

Davidzheng wrote: post #7
recipe19 wrote: post #4

I get the broader point, but the infosec framing here is weird. It's a naive and dangerous view that the defense efforts are only as strong as the weakest link. If you're building your security program that way, you're going to lose. The idea is to have multiple layers of defense because you can never really, consistently get 100% with any single layer: people will make mistakes, there will be systems you don't know…

I'm not a security person at all. But this comments reads against the best practices which I've heard. Like that the best defense is using open source & well-tested protocols with extremely small attack surface to minimize the space of possible exploits. Curious what I'm not understanding here.

It should be possible to add layers without increasing attack surface.

Re: Tao on “blue team” vs. “red team” LLMs

#148

Post by adastra22 »

Davidzheng wrote: post #7
recipe19 wrote: post #4

I get the broader point, but the infosec framing here is weird. It's a naive and dangerous view that the defense efforts are only as strong as the weakest link. If you're building your security program that way, you're going to lose. The idea is to have multiple layers of defense because you can never really, consistently get 100% with any single layer: people will make mistakes, there will be systems you don't know…

I'm not a security person at all. But this comments reads against the best practices which I've heard. Like that the best defense is using open source & well-tested protocols with extremely small attack surface to minimize the space of possible exploits. Curious what I'm not understanding here.

Security person here. Open sourcing your entire stack is NOT best practices. The best defense is defense in depth, with some proprietary layers unknown to the attacker.

Re: Tao on “blue team” vs. “red team” LLMs

#149

Post by JonChesterfield »

ozgrakkurt wrote: post #22
skdidjdndh wrote: post #5

Earlier quoted context omitted.

> I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them Having worked on legacy codebases this is extremely wrong and harmful. Tests are the source of truth more so than your code - and incorrect tests are even more harmful than incorrect code. Having worked on legacy codebases, some of the hardest problems are determining “why is this…

What do you think about leaning on fuzz testing and deriving unit tests from bugs found by fuzzing?

You end up with a pile of unit tests called things like "regression, don't crash when rhs null" or "regression, terminate on this" which seems fine.

The "did it change?" genre of characterisation/snapshot tests can be created very effectively using a fuzzer, but should probably be kept separate from the unit tests checking for specific behaviour, and partially regenerated when deliberately changing behaviour.

Llvm has a bunch of tests generated mechanically from whatever the implementation does and checked in. I do not rate these - they're thousands of lines long, glow red in code review and I'm pretty sure don't get read by anyone in practice - but because they exist more focused tests do not.

Re: Tao on “blue team” vs. “red team” LLMs

#150

Post by positron26 »

_alternator_ wrote: post #2

This red vs blue team is a good way to understand the capabilities and current utility of LLMs for expert use. I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them; and if they are correct, they adds value. But often they don’t test the core functionality; the best tests I still have to write myself. Having LLMs fix bugs or add feature…

> prone to cheating or writing non robust code (eg special code paths to pass tests without solving the actual problem).

The solution will come from synthetic data training methods that lobotomize part of the weights. It's just cross-validation. A distilled awareness won't maintain knowledge of the cheat paths, exposing them as erroneous.

This may a reason why every living thing on Earth that encounters psychoactive drugs seems to enjoy them. Self-deceptive paths depend on consistency whereas truth-preservation of facts grounded in reality will always be re-derived.

Post reply on HN