171–180 of 229 posts
This is hallucination. Or maybe a sales pitch. If production bugs and the requirement to retain a workable code base don’t get us to write “good” code, then nothing will. And at the current state of the art, “AI” will tend to make it worse.
There's a catch with 100% coverage. If the agent writes both the code and the tests, we risk falling into a tautology trap. The agent can write flawed logic and a test that verifies that flawed logic (which will pass). 100% coverage only makes sense if tests are written before the code or rigorously verified by a human. Otherwise, we're just creating an illusion of reliability by covering hallucinations with tests. A…
As the sibling comments alluded to, it’s not exclusively an AI problem since multiple people can miss the issue too.
It’s wonderful that AI is an impetus for so many people to finally learn proper engineering principles though!
There's a catch with 100% coverage. If the agent writes both the code and the tests, we risk falling into a tautology trap. The agent can write flawed logic and a test that verifies that flawed logic (which will pass). 100% coverage only makes sense if tests are written before the code or rigorously verified by a human. Otherwise, we're just creating an illusion of reliability by covering hallucinations with tests. A…
That’s why you’ve gotta test your tests. Insert bugs and ensure they fail. As the sibling comments alluded to, it’s not exclusively an AI problem since multiple people can miss the issue too. It’s wonderful that AI is an impetus for so many people to finally learn proper engineering principles though!
LOL No. AI code i see is 90% really bad. The poster then snakes around the first commenter that asks "how much of the code was generated by AI?" Replies vary from silence to "ill checked all the code" or "ai code is better than human code" or even "ai was not used at all", even it is obvious it was 100% AI.
You did not read the article did you
This is hallucination. Or maybe a sales pitch. If production bugs and the requirement to retain a workable code base don’t get us to write “good” code, then nothing will. And at the current state of the art, “AI” will tend to make it worse.
> For decades, we’ve all known what “good code” looks like.
When relatively trivial concerns such as the ideal length of methods haven't achieved consensus, I doubt there can be any broadly accepted standard for software quality. There are plenty of metrics such as test coverage, but anyone with experience could tell you how easy it is to game those and that enforcing arbitrary standards can even cause harm.
Earlier quoted context omitted.
What's MC/DC?
Modified Condition/Decision Coverage (MC/DC) is a test coverage approach that considers a chunk of code covered if: - Every branch was "visited". Plain coverage already ensures that. I would actually advocate for 100% branch coverage before 100% line coverage. - Every part (condition) of a branch clause has taken all possible values. If you have if(enabled && limit > 0), MC/DC requires you to test with enabled, !enab…
Earlier quoted context omitted.
That’s why you’ve gotta test your tests. Insert bugs and ensure they fail. As the sibling comments alluded to, it’s not exclusively an AI problem since multiple people can miss the issue too. It’s wonderful that AI is an impetus for so many people to finally learn proper engineering principles though!
but who will test the tests of tests?
Earlier quoted context omitted.
> This is sort of why I think software development might be the only real application of LLMs outside of entertainment. Wow. What about also, I don't know, self-teaching*? In general, you have to be very arrogant to say that you've experienced all the "real" applications. * - For instance, today and yesterday, I've been using LLMs to teach myself about RLC circuits and "inerters".
It’s somewhat delusional and potentially dangerous to assume that chatting with an LLM about a specific topic is self-teaching beyond the most surface-level understanding of a topic. No doubt you can learn some true things, but you’ll also learn some blatant falsehoods and a lot of incorrect theory. And you won’t know which is which. One of the most important factors in actually learning something is humility. Unfort…
Earlier quoted context omitted.
Why would you think that a machine known to cheerfully and confidently assert complete bullshit is suitable to learn from?
Because you can independently check anything it tells you. You understand there can be independent sources of validation?
Something I just started doing yesterday, and I'm hoping it catches on, is that I've been writing the spec for what I want in TLA+/PlusCal at a pretty high level, and then I tell Codex implement exactly to the spec. I tell it to not deviate from the spec at all, and be as uncreative as possible. Since it sticks pretty close to the spec and since TLA+ is about modifying state, the code it generates is pretty ugly, but…