Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tao on “blue team” vs. “red team” LLMs

mathstodon.xyz

Post reply on HN
Print view View original

71–80 of 179 posts

Re: Tao on “blue team” vs. “red team” LLMs

#71

Post by LeifCarrotson »

> The blue team is more obviously necessary to create the desired product; but the red team is just as essential, given the damage that can result from deploying insecure systems.

> Many of the proposed use cases for AI tools try to place such tools in the "blue team" category, such as creating code...

> However, in view of the unreliability and opacity of such tools, it may be better to put them to work on the "red team", critiquing the output of blue team human experts but not directly replacing that output...

The red team is only essential if you're a coward who isn't willing to take a few risks for increased profit. Why bother testing and securing when you can boost your quarterly bonus by just... not doing that?

I suspect that Terence Tao's experience leans heavily towards high-profile risk-averse institutions. People don't call one of the greatest living mathematicians to check your work when they're just trying to duct taping a new interface on top of a line-of-business app that hasn't seen much real investment since the late 90s. Conversely, the people who are writing cutting-edge algorithms for new network protocols and filesystems are hopefully not trying to churn out code as fast and cheap as possible by copy-pasting snippets to and from random chatbots.

There are a lot of people who are already cutting corners on programmer salaries, accruing invisible tech debt minute by minute. They're not trying to add AI tools to create a missing red team, they're trying to reduce headcount on the only team they have, which is the blue team (which is actually just one overworked IT guy in over his head).

Re: Tao on “blue team” vs. “red team” LLMs

#72

Post by fsckboy »

javier_e06 wrote: post #70

In cybersecurity red and blue test are two equal forces. In software development the analogy I think is a stretch, coding and testing are not two equal forces. Test is code too, and as such, it has bugs too. Test runs afoul with police paradox: Who polices the police? The Police police the police.

"Police police police police police police police."

https://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffal...

Re: Tao on “blue team” vs. “red team” LLMs

#73

Post by 9rx »

munificent wrote: post #69
9rx wrote: post #49

Earlier quoted context omitted.

> The spec The tests are your spec. They exist precisely to document what the program is supposed to do for other humans, with the secondary benefit of also telling a machine what the program is supposed to do, allowing implementations to automatically validate themselves against the spec. If you find yourself writing specs and tests as independent things, that's how you end up with bad, brittle tests that make devel…

Unfortunately, tests can never be a complete specification unless the system is simple enough to have a finite set of possible inputs. For all real-world software, a test suite tests a number of points in the space of possible inputs and we hope that those points generalize to pinning down the overall behavior of the implementation. But there's no guarantee of that generalization. An implementation that fails a test…

> Unfortunately, tests can never be a complete specification

They are for the human, which is the intended recipient.

Given infinite time the machine would also be able to validate against the complete specification, but, of course, we normally cut things short because we want to release the software in a reasonable amount of time. But, as before, that this ability exists at all is merely a secondary benefit.

Re: Tao on “blue team” vs. “red team” LLMs

#74

Post by Kinrany »

manmal wrote: post #33
skdidjdndh wrote: post #5

Earlier quoted context omitted.

> I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them Having worked on legacy codebases this is extremely wrong and harmful. Tests are the source of truth more so than your code - and incorrect tests are even more harmful than incorrect code. Having worked on legacy codebases, some of the hardest problems are determining “why is this…

> Tests are the source of truth more so than your code Tests poke and prod with a stick at the SUT, and the SUT's behaviour is observed. The truth lives in the code, the documentation, and, unfortunately, in the heads of the dev team. I think this distinction is quite important, because this question: > Do we have a bug? Or do we have a bad test? cannot be answered by looking at the test + the implementation. The spe…

None of the four: code, tests, spec, people's memory, are the single source of truth.

It's easy to see them as four cache layers, but empirically it's almost never the case that the correct thing to do when they disagree is to blindly purge and recreate levels that are farther from the "truth" (even ignoring the cost of doing that).

Instead, it's always an ad-hoc reasoning exercise in looking at all four of them, deciding what the correct answer is, and updating some or all of them.

Re: Tao on “blue team” vs. “red team” LLMs

#75

Post by marcosdumay »

threatofrain wrote: post #64
andrepd wrote: post #59

Earlier quoted context omitted.

I don't understand this. How does it slow your development if the tests being green is a necessary condition for the code being correct? Yes it slows it compared to just writing incorrect code lol, but that's not the point.

It's that hard to write specs that truly match the business, hence why test-driven-development or specification-first failed to take off as a movement. Asking specs to truly match the business before we begin using them as tests would handcuff test people in the same way we're saying that tests have the potential to handcuff app and business logic people — as opposed to empowering them. So I wouldn't blame people for…

> So I wouldn't blame people for writing specs that only match the code implementation at that time.

WFT are you doing writing specs based on implementation? If you already have the implementation, what are you using the specs for? Or, if you want to apply this direct to tests, if you are already assuming the program is correct, what are you trying to test?

Are you talking about rewriting applications?

Re: Tao on “blue team” vs. “red team” LLMs

#77

Post by nostrademons »

LeifCarrotson wrote: post #71

> The blue team is more obviously necessary to create the desired product; but the red team is just as essential, given the damage that can result from deploying insecure systems. > Many of the proposed use cases for AI tools try to place such tools in the "blue team" category, such as creating code... > However, in view of the unreliability and opacity of such tools, it may be better to put them to work on the "red…

Tao is talking about systems, which are self-sustaining dynamic networks that function independently of who the individual actors and organizations within the system are. You can break up the monopoly at the heart of the blue team system (as the U.S. did with Standard Oil and AT&T) and it will just reform through mergers over generations (as it largely has with Exxon Mobil and Verizon). You can fire or kill all the people involved and they will just be replaced by other people filling the same roles. The details may change, but the overall dynamics remain the same.

In this case, all the companies who are doing what you describe are themselves the red team. They are the unreliable, additive, distributed players in an ecosystem where the companies themselves are disposable. The blue team is the blue team by virtue of incentives: they are the organization where proper functioning of their role requires that all the parts are reliable and work well together, and if the individual people fulfilling those roles do not have those qualities, they will fail and be replaced by people who do.

Re: Tao on “blue team” vs. “red team” LLMs

#78

Post by johnisgood »

_alternator_ wrote: post #40
mvieira38 wrote: post #30

Earlier quoted context omitted.

I have the exact opposite idea. I want the tests to be mine and thoroughly understood, so I am the true arbiter and then I can let the LLM go ham on the code without fear. If the tests are AI made, then I get some anxiety letting agents mess with the rest of the codebase

I think this is exactly the tradeoff (blue team and red team need to be matched in power), except that I’ve seen LLMs literally cheat the tests (eg “match input: TEST_INPUT then return TEST_OUTPUT”) far too many times to be comfortable with letting LLMs be a major blue team player.

Yeah, they may do that, but people really should read the code an LLM produces. Ugh, makes me furious. No wonder LLMs have a bad rep from such users.

Re: Tao on “blue team” vs. “red team” LLMs

#79

Post by zaking17 »

My coding flow today involves a lot of asking an LLM to generate code (blue team) and then me code reviewing, rewriting, and making it scalable (red team?). The analogy breaks down, because I'm providing the safety and correctness; LLMs are offering a head start.

I'm optimistic about AI-powered infra & monitoring tools. When I have a long dump of system logs that I don't understand, LLMs help immensely. But then it's my job to finalize the analysis and make sure whatever debugging comes next is a good use of time. So not quite red team/blue team in that case either.

Post reply on HN