I think this is exactly the tradeoff (blue team and red team need to be matched in power), except that I’ve seen LLMs literally cheat the tests (eg “match input: TEST_INPUT then return TEST_OUTPUT”) far too many times to be comfortable with letting LLMs be a major blue team player.
Yeah, they may do that, but people really should read the code an LLM produces. Ugh, makes me furious. No wonder LLMs have a bad rep from such users.
> people really should read the code an LLM produces
Yeah, but that, like, requires that you know how to code.
And wasn't the point of LLMs in the first place to let clueless people make software?
The reality is the opposite of this post. LLMs are great at rapidly creating rough drafts, and humans are best (when properly trained) at critiquing LLM results. So, LLMs are in fact better at blue-teaming, and humans are better at red-teaming.
This red vs blue team is a good way to understand the capabilities and current utility of LLMs for expert use. I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them; and if they are correct, they adds value. But often they don’t test the core functionality; the best tests I still have to write myself. Having LLMs fix bugs or add feature…
> I trust them to add tests almost indiscriminately because tests are usually cheap; if they are wrong it’s easy to remove or modify them Having worked on legacy codebases this is extremely wrong and harmful. Tests are the source of truth more so than your code - and incorrect tests are even more harmful than incorrect code. Having worked on legacy codebases, some of the hardest problems are determining “why is this…
This is why tests need documenting what exactly they intend to test, and why.
> and it will just reform through mergers over generations You say "just" as though this is a failure of the system, but this is the system working as designed. Economies of scale are half the reason to bother with large-scale enterprise, so they inevitably consolidate to the point of monopoly, so disrupting that monopoly by force to keep the market aligned is an ongoing and never-ending process that you should expec…
I'm not saying this is a failure of the system, only that it is the system. My overall point is that systems take the form they do based on available technology, efficiencies of production, lines of communication, and incentives, and that the individual firms involved are disposable actors that are forced by the factors above into economically-rational actions. If the natural form of an industry is monopoly (as most…
Sure, but this is making the common mistake of viewing government intervention as being somehow separate or outside of the market system, rather than being inside the system. Corporations in a competitive market consolidate to the point of monopoly, they use that monopoly to abuse customers, customers demand their governments intervene, some measure of competitiveness is restored, goto start. This is the system.
Interesting. From a writing point of view this suggests that it's better to have the LLM "critique my draft" rather than "write the first draft." (Both for writing text and code) Also implies that we want to manually check all of the LLM's suggestions. This makes it sound more like a co-worker (agent) than all-powerful SuperIntelligence. I guess this is a symptom of the hallucinations.
I'm not saying this is a failure of the system, only that it is the system. My overall point is that systems take the form they do based on available technology, efficiencies of production, lines of communication, and incentives, and that the individual firms involved are disposable actors that are forced by the factors above into economically-rational actions. If the natural form of an industry is monopoly (as most…
Sure, but this is making the common mistake of viewing government intervention as being somehow separate or outside of the market system, rather than being inside the system. Corporations in a competitive market consolidate to the point of monopoly, they use that monopoly to abuse customers, customers demand their governments intervene, some measure of competitiveness is restored, goto start. This is the system.
That's a fair way of looking at it.
Bringing it back to the article's point, the government is part of the "blue team" portion of the system. In that if they don't do their job, and their job includes complex regulations that balance multiple competing factors, then large portions of the system...well, "collapse" is a judgy term, but "function in significantly different ways" gets the point across. Inaction or ineffectiveness of the government effectively creates new "blue team" industries, and distributes power in different ways across the economy.
Interesting. From a writing point of view this suggests that it's better to have the LLM "critique my draft" rather than "write the first draft." (Both for writing text and code) Also implies that we want to manually check all of the LLM's suggestions. This makes it sound more like a co-worker (agent) than all-powerful SuperIntelligence. I guess this is a symptom of the hallucinations. https://open.substack.com/pub/t…
Maybe. I also think that the implications of code can be harder to decipher on first pass than writing text which leads me to believe that maybe that mental model (Red Team, Blue Team) might not fit here.
Interesting. From a writing point of view this suggests that it's better to have the LLM "critique my draft" rather than "write the first draft." (Both for writing text and code) Also implies that we want to manually check all of the LLM's suggestions. This makes it sound more like a co-worker (agent) than all-powerful SuperIntelligence. I guess this is a symptom of the hallucinations. https://open.substack.com/pub/t…
Maybe. I also think that the implications of code can be harder to decipher on first pass than writing text which leads me to believe that maybe that mental model (Red Team, Blue Team) might not fit here.
Good point. I can quickly intuitively tell if the suggestions for my writing is correct. Harder to tell on code.
Perhaps the analogy is better for "Writing code" versus "Writing Test Cases"?
In cybersecurity red and blue test are two equal forces. In software development the analogy I think is a stretch, coding and testing are not two equal forces. Test is code too, and as such, it has bugs too. Test runs afoul with police paradox: Who polices the police? The Police police the police.
This is an underrated comment... Most all LLM stuff suffers from not having any ground truth, even with multiple agentic rag integrations.
I literally do the diametric opposite of you and it works extremely well. Im weirded out by your comment. Writing tests that couple to low level implementation details was something I thought most people did accidentally before giving up on TDD, not intentionally.
It isn't coupling low level implementation details, it is writing tests based on input and output of the unit under test. The expected output from a unit, given an input is not an implementation detail, unless you have some very different definition of implementation detail than I. Testing the unit under test produces the expected outputs from a set of inputs implies nothing about implementation details at all. It is…
If the "unit under test" is low level then thats coupling low level implementation details to the test.
If you're vague about what constitutes a "unit" that means youre probably not thinking about this problem.