Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tao on “blue team” vs. “red team” LLMs

mathstodon.xyz

Post reply on HN
Print view View original

91–100 of 179 posts

Re: Tao on “blue team” vs. “red team” LLMs

#91

Post by godelski »

9rx wrote: post #49
manmal wrote: post #33

Earlier quoted context omitted.

> Tests are the source of truth more so than your code Tests poke and prod with a stick at the SUT, and the SUT's behaviour is observed. The truth lives in the code, the documentation, and, unfortunately, in the heads of the dev team. I think this distinction is quite important, because this question: > Do we have a bug? Or do we have a bad test? cannot be answered by looking at the test + the implementation. The spe…

> The spec The tests are your spec. They exist precisely to document what the program is supposed to do for other humans, with the secondary benefit of also telling a machine what the program is supposed to do, allowing implementations to automatically validate themselves against the spec. If you find yourself writing specs and tests as independent things, that's how you end up with bad, brittle tests that make devel…

  > The tests are your spec.
That's not quite right, but it's almost right.

Tests are an *approximation* of your spec.

Tests are a description, and like all descriptions are noisy. The thing is it is very very difficult to know if your tests have complete coverage. It's very hard to know if your description is correct.

How often do you figure out something you didn't realize previously? How often do you not realize something and it's instead pointed out by your peers? How often do you realize something after your peers say something that sparks an idea?

Do you think that those events are over? No more things to be found? I know I'm not that smart because if I was I would have gotten it all right from the get go.

There are, of course, formal proofs but even they aren't invulnerable to these issues. And these aren't commonly used in practice and at that point we're back to programming/math, so I'm not sure we should go down that route.

Re: Tao on “blue team” vs. “red team” LLMs

#92

Post by torginus »

_alternator_ wrote: post #2

This red vs blue team is a good way to understand the capabilities and current utility of LLMs for expert use. I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them; and if they are correct, they adds value. But often they don’t test the core functionality; the best tests I still have to write myself. Having LLMs fix bugs or add feature…

There's a saying that since nobody tests the tests, they must be trivially correct.

That's why they came up with the Arrange-Act-Assert pattern.

My favorite kind of unit test nowadays is when you store known input-output pairs and validate the code on them. It's easy to test corner cases and see that the output works as desired.

Re: Tao on “blue team” vs. “red team” LLMs

#93

Post by 9rx »

godelski wrote: post #91
9rx wrote: post #49

Earlier quoted context omitted.

> The spec The tests are your spec. They exist precisely to document what the program is supposed to do for other humans, with the secondary benefit of also telling a machine what the program is supposed to do, allowing implementations to automatically validate themselves against the spec. If you find yourself writing specs and tests as independent things, that's how you end up with bad, brittle tests that make devel…

> The tests are your spec. That's not quite right, but it's almost right. Tests are an * approximation* of your spec. Tests are a description, and like all descriptions are noisy. The thing is it is very very difficult to know if your tests have complete coverage. It's very hard to know if your description is correct. How often do you figure out something you didn't realize previously? How often do you not realize so…

> Tests are a description

As is a spec. "Description" is literally found in the dictionary definition. Which stands to reason as tests are merely a way to write a spec. They are the same thing.

> The thing is it is very very difficult to know if your tests have complete coverage.

There is no way to avoid that, though. Like you point out, not even formal proofs, the closest speccing methodology we know of to try and avoid this, is immune.

> Tests are an approximation of your spec.

Specs are an approximation of what you actually want, sure, but that does not change that tests are the spec. There are other ways to write a spec, of course, but if you went down that road you wouldn't also have tests. That would be not only pointless, but a nightmare due to not having a single source of truth which causes all kinds of social (and sometimes technical) problems.

Re: Tao on “blue team” vs. “red team” LLMs

#95

Post by MoreQARespect »

nyrikki wrote: post #86
threatofrain wrote: post #64

Earlier quoted context omitted.

It's that hard to write specs that truly match the business, hence why test-driven-development or specification-first failed to take off as a movement. Asking specs to truly match the business before we begin using them as tests would handcuff test people in the same way we're saying that tests have the potential to handcuff app and business logic people — as opposed to empowering them. So I wouldn't blame people for…

The problem with TDD is that people assumed it was writing a specification, or directly tried to map it directly to post-hoc testing and metrics. TDD at its core is defining expected inputs and mapping those to expected outputs at the unit of work level, e.g. function, class etc. While UAT and domain informed what those inputs=outputs are, avoiding trying to write a broader spec that that is what many people struggle…

I literally do the diametric opposite of you and it works extremely well.

Im weirded out by your comment. Writing tests that couple to low level implementation details was something I thought most people did accidentally before giving up on TDD, not intentionally.

Re: Tao on “blue team” vs. “red team” LLMs

#96

Post by TheGRS »

After using agentic models and workflows recently, I think these agents belong in both roles. Even more than that, they should be involved in the management tasks too. The developer becomes more of an overseer. You're overseeing the planning of a task - writing prompts, distilling the scope of the task down. You're overseeing writing the tests. And you're overseeing writing out the code. Its a ton of reviewing, but I've always felt more in control as a red team type myself making sure things don't break.

Re: Tao on “blue team” vs. “red team” LLMs

#97

Post by PeeMcGee »

sour-taste wrote: post #63
yojo wrote: post #13

Earlier quoted context omitted.

I would add that few things slow developer velocity as much as a large suite of comprehensive and brittle tests. This is just as true on greenfield as on legacy. Anticipating future responses: yes, a robust test harness allows you to make changes fearlessly. But most big test suites I’ve seen are less “harness” and more “straight-jacket”

An old coworker used to call these types of tests change detector tests. They are excellent at telling you whether some behavior changed, but horrible at telling you whether that behavior change is meaningful or not.

These sorts of tests are invaluable for things like ensuring adherence to specifications such as OAuth2 flows. A high-level test that literally describes each step of a flow will swiftly catch odd changes in behavior such as a request firing twice in a row or a well-defined payload becoming malformed. Say a token validator starts misbehaving and causes a refresh to occur with each request (thus introducing latency and making the IdP angry). That change in behavior would be invisible to users, but a test that verified each step in an expected order would catch it right away, and should require little maintenance unless the spec itself changes.

Re: Tao on “blue team” vs. “red team” LLMs

#99

Post by darkwater »

lanstin wrote: post #67
darkwater wrote: post #19

Earlier quoted context omitted.

Well, I think the his example (locked door + opened window) makes sense, and the multiple LAYERS concept applies to things an attacker has to do or go through to reach the jackpot. But doors and windows are on the same layer, and there the weakest link totally defines how strong the chain is. A similar example in the web world would be that you have your main login endpoint very well protected, audited, using only st…

Or a single logging jar that will execute some of its message contents. Inside all your DMZ layers in the app content.

Poor log4j...

Re: Tao on “blue team” vs. “red team” LLMs

#100

Post by groestl »

jrockway wrote: post #82
sour-taste wrote: post #63

Earlier quoted context omitted.

An old coworker used to call these types of tests change detector tests. They are excellent at telling you whether some behavior changed, but horrible at telling you whether that behavior change is meaningful or not.

Yup. Working on a 10 year old codebase, I always wondered whether a test failing was "a long-standing bug was accidentally fixed" or "this behavior was added on purpose and customers rely on it". It can be about 50/50 but you're always surprised. Change detector tests add to the noise here. No, this wasn't a feature customers care about, some AI added a test to make sure foo.go line 42 contained less than 80 characte…

> a long-standing bug was accidentally fixed

In some cases (e.g. in our case) long standing bugs become part of the API that customers rely on.

Post reply on HN