Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tao on “blue team” vs. “red team” LLMs

mathstodon.xyz

Post reply on HN
Print view View original

151–160 of 179 posts

Re: Tao on “blue team” vs. “red team” LLMs

#151

Post by vermarish »

javier_e06 wrote: post #70

In cybersecurity red and blue test are two equal forces. In software development the analogy I think is a stretch, coding and testing are not two equal forces. Test is code too, and as such, it has bugs too. Test runs afoul with police paradox: Who polices the police? The Police police the police.

I interpret it a different way than that. I see application code and testing code as both a part of blue team. It's the code reviews and architectural critiques that are part of red team.

Personally, I've found GitHub's feature of AI PR reviewers exceptionally helpful. I think that's the type of red team LLM app Tao is describing here.

Re: Tao on “blue team” vs. “red team” LLMs

#153

Post by 9rx »

godelski wrote: post #141
9rx wrote: post #136

Earlier quoted context omitted.

> We have 3 objects: doc, test, spec. How do you prove they are the same thing? You... don't? There is nothing good that can come from trying to understand crazy. Best to run away as fast as possible if you ever encounter this. > You are arguing that they all point to the same address. Oh? I did say if you document something the same way three different times (even if you give each time a different name, as if that s…

> In reality, I "argued" both. I mean it is hard to have this conversation because you will say that they are the same thing and then leverage the fact that they aren't while disagreeing with me but using nearly identical settings to my examples. I mean if your argument is that a mallard (test) and a muscovy (docs) are both types of ducks but a mallard is not a muscovy and a muscovy is not a mallard, then I fail to s…

> I mean if your argument is that a mallard (test) and a muscovy (docs) are both types of ducks

To draw a more reasonable analogy with how the words are actually used on a normal basis, you'd have fowl (docs), ducks (specs), and mallards (tests). As before, the terms change in specificity, but do not refer to something else entirely. Pointing at a mallard and calling it a duck, or fowl, doesn't alter what it is. It is all the very same animal.

Yes, fowl could also refer to chickens just as documentation could refer to tax returns. 'Tis the nature of using a word lacking specificity. But from context one should be able to understand that we're not talking about tax returns here.

But I don't have an "argument". High school debate team is over there.

> It's simply a recognition that you can't write a perfect spec.

That was recognized from the onset. What is the purpose of adding this again?

> Do you think we're all wrong?

We're all bad at communicating, if that's what you are straining to ask. Which isn't exactly much of a revelation. We've both already indicated as such, as have many commenters that came before us.

Re: Tao on “blue team” vs. “red team” LLMs

#155

Post by ozgrakkurt »

nostrademons wrote: post #111

Interesting way of viewing this! Business also has a “blue team” (those industries that the rest of the economy is built upon - electricity, oil, telecommunications, software, banking; possibly not coincidentally, “blue chips”) and a “red team” (industries that are additive to consumer welfare, but not crucial if any one of them goes down. Restaurants, specialty retail, luxuries, tourism, etc.) It is almost always be…

Maybe it can’t be blue team in current state but it could get better and actually be able to create software. If this happens then the ones that get there first will have a big advantage.

But not sure if buying a million gpus and training llms will be the strategy to improve it

Re: Tao on “blue team” vs. “red team” LLMs

#156

Post by cubefox »

abalaji wrote: post #131
spoaceman7777 wrote: post #128

The reality is the opposite of this post. LLMs are great at rapidly creating rough drafts, and humans are best (when properly trained) at critiquing LLM results. So, LLMs are in fact better at blue-teaming, and humans are better at red-teaming.

I think this flips at the frontier which may be what Tao is commenting on.

Tao is a) unusually intelligent and b) an expert in his field. Most people are neither very intelligent nor have expert knowledge in any academic subject. So Tao is pretty much the least representative LLM user possible.

Re: Tao on “blue team” vs. “red team” LLMs

#157

Post by cubefox »

pamelafox wrote: post #125

(Disclosure: I work for Microsoft) I run automated red-teaming on my RAG samples through the azure-ai-evaluation SDK, which uses an adversarial LLM (an LLM without the guardrails) plus the pyrit package to come up with horrible questions to ask your app and then transform them (base64, ceaser cipher, urlencode, etc), to see how the app will respond. It's really interesting to see the results, and I agree that red-tea…

Any experiences like this one? https://www.lesswrong.com/posts/MnYnCFgT3hF6LJPwn/why-white-...

Re: Tao on “blue team” vs. “red team” LLMs

#158

Post by theptip »

_alternator_ wrote: post #2

This red vs blue team is a good way to understand the capabilities and current utility of LLMs for expert use. I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them; and if they are correct, they adds value. But often they don’t test the core functionality; the best tests I still have to write myself. Having LLMs fix bugs or add feature…

I think the more fundamental attribute of interest is how easy it is to verify the work.

Much red team work is easily verifiable; either the exploit works or it doesn’t. Whereas more blue-team work is not easily verifiable; it might take judgement to figure out if a feature is promising.

LLMs are extremely powerful (and trainable) on tasks with a good oracle.

Re: Tao on “blue team” vs. “red team” LLMs

#159

Post by auggierose »

cubefox wrote: post #156
abalaji wrote: post #131

Earlier quoted context omitted.

I think this flips at the frontier which may be what Tao is commenting on.

Tao is a) unusually intelligent and b) an expert in his field. Most people are neither very intelligent nor have expert knowledge in any academic subject. So Tao is pretty much the least representative LLM user possible.

You could argue that Tao is the most representative LLM user possible, because why would you need not very intelligent people use LLMs? Just replace them with LLMs.

Re: Tao on “blue team” vs. “red team” LLMs

#160

Post by cubefox »

auggierose wrote: post #159
cubefox wrote: post #156

Earlier quoted context omitted.

Tao is a) unusually intelligent and b) an expert in his field. Most people are neither very intelligent nor have expert knowledge in any academic subject. So Tao is pretty much the least representative LLM user possible.

You could argue that Tao is the most representative LLM user possible, because why would you need not very intelligent people use LLMs? Just replace them with LLMs.

I assume you wouldn't want to be replaced by an LLM.
Post reply on HN