Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tao on “blue team” vs. “red team” LLMs

mathstodon.xyz

Post reply on HN
Print view View original

101–110 of 179 posts

Re: Tao on “blue team” vs. “red team” LLMs

#101

Post by godelski »

9rx wrote: post #93
godelski wrote: post #91

Earlier quoted context omitted.

> The tests are your spec. That's not quite right, but it's almost right. Tests are an * approximation* of your spec. Tests are a description, and like all descriptions are noisy. The thing is it is very very difficult to know if your tests have complete coverage. It's very hard to know if your description is correct. How often do you figure out something you didn't realize previously? How often do you not realize so…

> Tests are a description As is a spec. "Description" is literally found in the dictionary definition. Which stands to reason as tests are merely a way to write a spec. They are the same thing. > The thing is it is very very difficult to know if your tests have complete coverage. There is no way to avoid that, though. Like you point out, not even formal proofs, the closest speccing methodology we know of to try and a…

  > that does not change that tests are the spec.
I disagree. It's, like you say, one description of your spec but that's not the spec.

  > not having a single source of truth
Well that's the thing, there is no single source of truth. A single source of truth is for religion, not code.

The point of saying this is to ensure you don't fall prey to fooling yourself. You're the easiest person for you to fool, after all. You should always carry some doubt. Not so much it is debilitating, but enough to keep you from being too arrogant. You need to constantly check that your documentation is aligned to your specs and that your specs are aligned to your goals. If you cannot see how these are different things then it's impossible to check your alignment and you've fooled yourself.

Re: Tao on “blue team” vs. “red team” LLMs

#102

Post by SamuelAdams »

skdidjdndh wrote: post #5
_alternator_ wrote: post #2

This red vs blue team is a good way to understand the capabilities and current utility of LLMs for expert use. I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them; and if they are correct, they adds value. But often they don’t test the core functionality; the best tests I still have to write myself. Having LLMs fix bugs or add feature…

> I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them Having worked on legacy codebases this is extremely wrong and harmful. Tests are the source of truth more so than your code - and incorrect tests are even more harmful than incorrect code. Having worked on legacy codebases, some of the hardest problems are determining “why is this…

Ideally the git history provides the “why was this test written”, however if you have one Jira card tied to 500+ AI generated tests, it’s not terribly helpful.

Re: Tao on “blue team” vs. “red team” LLMs

#103

Post by 9rx »

godelski wrote: post #101
9rx wrote: post #93

Earlier quoted context omitted.

> Tests are a description As is a spec. "Description" is literally found in the dictionary definition. Which stands to reason as tests are merely a way to write a spec. They are the same thing. > The thing is it is very very difficult to know if your tests have complete coverage. There is no way to avoid that, though. Like you point out, not even formal proofs, the closest speccing methodology we know of to try and a…

> that does not change that tests are the spec. I disagree. It's, like you say, one description of your spec but that's not the spec. > not having a single source of truth Well that's the thing, there is no single source of truth. A single source of truth is for religion, not code. The point of saying this is to ensure you don't fall prey to fooling yourself. You're the easiest person for you to fool, after all. You…

> You need to constantly check that your documentation is aligned to your specs

Documentation, tests, and specs are all ultimately different words for the same thing.

You do have to check that your implementation and documentation/spec/tests are aligned, which can be a lot of work if you do so by hand, but that's why we invented automatic methods. Formal verification is theoretically best (that we know of) at this, but a huge pain in the ass for humans to write, so that is why virtually everyone has adopted tests instead. It is a reasonable tradeoff between comfort in writing documentation while still providing sufficient automatic guarantees that the documentation is true.

> If you cannot see how these are different things

If you see them as different things, you are either pointlessly repeating yourself over and over or inventing information that is, at best, worthless (but often actively harmful).

Re: Tao on “blue team” vs. “red team” LLMs

#104

Post by strbean »

groestl wrote: post #100
jrockway wrote: post #82

Earlier quoted context omitted.

Yup. Working on a 10 year old codebase, I always wondered whether a test failing was "a long-standing bug was accidentally fixed" or "this behavior was added on purpose and customers rely on it". It can be about 50/50 but you're always surprised. Change detector tests add to the noise here. No, this wasn't a feature customers care about, some AI added a test to make sure foo.go line 42 contained less than 80 characte…

> a long-standing bug was accidentally fixed In some cases (e.g. in our case) long standing bugs become part of the API that customers rely on.

It's nearly guaranteed, even if it is just because customers had to work around the bug in such a way that their flow now breaks when the bug is gone.

Obligatory: https://xkcd.com/1172/

Re: Tao on “blue team” vs. “red team” LLMs

#107

Post by wavemode »

shiandow wrote: post #106
1970-01-01 wrote: post #105

Good read, but I'm struggling to understand why Terry did not use the foundational terms offense and defense .

Because describing the task of writing code as defense is a bit confusing.

well, in a way you're defending against bugs and vulnerabilities by reviewing code

Re: Tao on “blue team” vs. “red team” LLMs

#108

Post by topaz0 »

hiq wrote: post #65

What about formal proofs? Don't we expect LLMs to help there, in a more "blue team" role? E.g. when a mathematician talks about a "technical proof", enumerating cases in the thousands, my impression is that LLM would save some time, and potentially help mathematicians focus on the actually hard (rather than tedious) parts.

A computer can be helpful for enumerating cases and similar mechanical work. But an LLM specifically would be a terrible way to do this.

Re: Tao on “blue team” vs. “red team” LLMs

#109

Post by Pxtl »

skdidjdndh wrote: post #5
_alternator_ wrote: post #2

This red vs blue team is a good way to understand the capabilities and current utility of LLMs for expert use. I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them; and if they are correct, they adds value. But often they don’t test the core functionality; the best tests I still have to write myself. Having LLMs fix bugs or add feature…

> I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them Having worked on legacy codebases this is extremely wrong and harmful. Tests are the source of truth more so than your code - and incorrect tests are even more harmful than incorrect code. Having worked on legacy codebases, some of the hardest problems are determining “why is this…

> “why is this broken test here that appears to test a behavior we don’t support”

Because somebody complained when that behavior we don't support was broken, so the bug-that-wasn't-really-a-bug was fixed and a test was created to prevent regression.

Imho, the mistake was in documentation: the Test should have comments explaining why this test was created.

Just as true for tests as for the actual business logic code:

The code can only describe the what and the how. It's up to comments to describe the why.

Re: Tao on “blue team” vs. “red team” LLMs

#110

Post by zkmon »

Red team is not a team. It is the background context in which the foreground operates. Evolution happens through interaction and adaptation between foreground and background. It is true that the background (context) is a dual form to the foreground (thing). But the context is not just another thing in the same sense as the foreground.
Post reply on HN