Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Code is cheap. Show me the talk

nadh.in

Post reply on HN
Print view View original

231–237 of 237 posts

Re: Code is cheap. Show me the talk

#231

Post by overgard »

acemarke wrote: post #177
overgard wrote: post #123

I asked Codex to write some unit tests for Redux today. At first glance it looked fine, and I continued on. I then went back to add a test by hand, and after looking more closely at the output there were like 50 wtf worthy things scattered in there. Sure they ran, but it was bad in all sorts of ways. And this was just writing something very basic. This has been my experience almost every time I use AI: superficially…

Hi, I'm the primary Redux maintainer. I'd love to see some examples of what got generated! (Doubt there's anything we could do to _influence_ this, but curious what happened here.) FWIW we do have our docs on testing approaches here, and have recommended a more integrated-style approach to testing for a while: - https://redux.js.org/usage/writing-tests

Unfortunately I think I cleaned up the code before committing it, so I don't have an exact example! I did actually read that usage page though after looking at those tests, and that helped me in fixing the tests (maybe in retrospect I should have pointed the AI at the docs page first).

I think the main issue I was having with it was reusing the store object instead of creating a new one for each test. The other issue I was seeing was that it was creating mock objects and API's for things that weren't even being tested (lot of scope creep), and one of those API's was basically copy-pasted between two files (code duplication). It was also just testing things that weren't really necessary (ie, testing Redux itself, instead of my usage of it).

Another issue was just taking a complex approach to fixing something that could be more easily solved. For instance, I had debug: true turned on for redux-undo so I was seeing some unnecessary log messages in the test. Codex identified this, and asked if I wanted to disable them, so I said yes. What it did though, was instead of setting debug: false, or disabling it on tests, it patched console.log to look for redux-undo prefixes. Technically worked, but kind of byzantine!

None of this was a terrible disaster or anything, especially since I started pretty small, but I think what made me miss some of the issues at first glance is this is my first usage of Redux in a non-toy project so while I understand the fundamentals fine, it was easy to sneak working-but-bad stuff past me until I sat down with the code to write a test on my own and started to see the issues.

Re: Code is cheap. Show me the talk

#232

Post by overgard »

visarga wrote: post #175
overgard wrote: post #123

I asked Codex to write some unit tests for Redux today. At first glance it looked fine, and I continued on. I then went back to add a test by hand, and after looking more closely at the output there were like 50 wtf worthy things scattered in there. Sure they ran, but it was bad in all sorts of ways. And this was just writing something very basic. This has been my experience almost every time I use AI: superficially…

> "write some unit tests for Redux today" The equivalent of "draw me a dog" -> not a masterpiece!? who would have thought? You need to come up with a testing methodology, write it down, and then ask the model to go through it. It likes to make assumptions on unspecified things, so you got to be careful. More fundamentally I think testing is becoming the core component we need to think about. We should not vibe-check…

I think you're assuming a lot about my prompting. I know to be specific with LLMs

Re: Code is cheap. Show me the talk

#233

Post by cheema33 »

3kkdd wrote: post #158
cheema33 wrote: post #152

Earlier quoted context omitted.

This is how you do things if you are new to this game. Get two other, different, LLMs to thoroughly review the code. If you don’t have an automated way to do all of this, you will struggle and eventually put yourself out of a job. If you do use this approach, you will get code that is better than what most software devs put out. And that gives you a good base to work with if you need to add polish to it.

Im sick and tired of these empty posts. SHOW AN EXAMPLE OF YOU ACTUALLY DOING WHAT YOU SAY!

The setup is much simpler than you might think. I have 4 CLI tools I use for this setup. Claude Code, Codex, Copilot and Cursor CLI. I asked Claude Code to create a code reviewer "skill" that uses the other 3 CLI tools to review changes in detail and provide feedback. I then ask Claude Code to use this skill to review any changes in code or even review plan documents. It is very very effective. Is it perfect? No. Nothing is. But, as I stated before, this produces results that are better than what an average developer sends in for PR review. Far far better in my own experience.

In addition to that, we do use CodeRabbit plugin on GitHub to perform a 4th code review. And we tell all of our agents to not get into gold-plating mode.

You can choose not to use modern tools like these to write software. You can also choose to write software in binary.

Re: Code is cheap. Show me the talk

#234

Post by nomel »

galaxyLogic wrote: post #185
jahsome wrote: post #163

Earlier quoted context omitted.

I don't see how the two are related, personally. I'm regularly accused of over-abstraction specifically because I aspire to make each abstraction do as little as possible, i.e. fewest lines possible.

"Abstracting" means extracting the commnon parts of multiple instances, and making everything else a parameter. The difficulty for software is that developers often start by writing the abstraction, rather than having multiple existing instances and then writing code that collects the common parts of those multiple instances into a single abstraction. I guess that is what "refactoring" is about. In sciences and human…

This matches my "ideal" way of writing software, which is something close to "reverse waterfall". Start with the non-negotiable truths at the lowest level, then work your way up towards the goal, which is sufficiently defined. As you go, the patterns became apparent, collapsing into nice abstractions.

The code always ends up nice and clean and modular. And, since I'm working towards the idea, I can say "here are the modular pieces I have to work with, dictated the fundamentals beneath, how do I use them to accomplish the task?". When working from the idea, I think it's easier to want to write something to achieve the immediate task, in a "not see the forest for the trees" kind of way (abstractions are about the goal, rather than the reality under). Of course, both directions are required, but I get the best "separation of concerns" going in reverse.

Re: Code is cheap. Show me the talk

#235

Post by sigotirandolas »

icedchai wrote: post #230
sigotirandolas wrote: post #225

Earlier quoted context omitted.

The most annoying thing is that even after cleaning up all the nonsense, the tests still contain all sort of fanfare and it’s essentially impossible to get the submitter to trim them because it’s death by a thousand cuts (and you better not say "do it as if you didn’t use AI" in the current climate..)

Yep. We've had to throw PRs away and ask them to start over with a smaller set of changes since it became impossible to manage. Reviews went on for weeks. The individual couldn't justify why things were done (and apparently their AI couldn't, either!)

Luckily those I work with are smart enough that I've not seen a PR thrown away yet, but sometimes I'm approving with more "meh, it's fine I guess" than "yeah, that makes sense".

Re: Code is cheap. Show me the talk

#236

Post by akst »

sigotirandolas wrote: post #225
icedchai wrote: post #218

Earlier quoted context omitted.

I work with individuals who attempt to use LLMs to write tests. More than once, it's added nonsensical, useless test cases. Admittedly, humans do this, too, to a lesser extent. Additionally, if their code has broken existing tests, it "fixes" them by not fixing the code under test, but changing the tests... (assert status == 200 becomes 500 and deleting code.) Tests "pass." PR is opened. Reviewers wade through slop..…

The most annoying thing is that even after cleaning up all the nonsense, the tests still contain all sort of fanfare and it’s essentially impossible to get the submitter to trim them because it’s death by a thousand cuts (and you better not say "do it as if you didn’t use AI" in the current climate..)

That’s also another thing. Sometimes the output is just junk, like there wasn’t really any intention behind the test to prevent a certain likely scenario arising

Sometimes it just add tests that lock in specific quirks of the code that weren’t necessarily intentional

Re: Code is cheap. Show me the talk

#237

Post by nomel »

Retric wrote: post #168
nomel wrote: post #165

Earlier quoted context omitted.

I genuinely appreciate your response, because there's a good chance it'll result in me changing my perspective, and I'm asking these questions with that intent! > You are thinking of assembly language which is a different thing. Initially there was no assembler, someone had to write one. This is why I specifically mention opcodes. I've actually written assemblers! And...there's not much to them. It's mostly just repl…

> It's mostly just replacing the names given to the opcodes in the datasheet back to the opcodes Under the assumption that the input data is properly formatted you can generate machine code. This is however an abstraction which can fail as nothing forces a user to input valid files. So we have an abstraction without any encapsulation.

I can only see that as being the case if you weren't aware of it. Otherwise, the awareness would be explicit intent to fail on malformed input, which seems like just as much as an encapsulation?

But, that's a great example! Thank you. This makes it clear that I'm probably sometimes wrong. ;)

Post reply on HN