Live data from Hacker News

LLMs work best when the user defines their acceptance criteria first

blog.katanaquant.com

361–370 of 460 posts

Re: LLMs work best when the user defines their acceptance criteria first

#361

Earlier quoted context omitted.

So in my experience with Opus 4.6 evaluating it in an existing code base has gone like this. You say "Do this thing". - It does the thing (takes 15 min). Looks incredibly fast. I couldn't code that fast. It's inhuman. So far all the fantastical claims hold up. But still. You ask "Did you do the thing?" - it says oops I forgot to do that sub-thing. (+5m) - it fixes the sub-thing (+10m) You say is the change well integ…

You didn't use plan mode.

[deleted]

Re: LLMs work best when the user defines their acceptance criteria first

#362

Earlier quoted context omitted.

> Whew. Ok. You don't tell it the code is slow. Do you tell your coworker "Hey, your code is slow" and expect great results? Yes? Why don't you? They are capable people that just didn't notice something, id I notice some telemetry and tell them "hey this is slow" they are expected to understand the reason(s).

...no? "Your code is slow" is essentially meaningless. A normal human conversation would specify which code/tasks/etc., how long it's currently taking, how much faster it needs to be, and why. And then potentially a much longer conversation about the tradeoffs involved in making in faster. E.g. a new index on the database that will make it gigabytes larger, a lookup table that will take up a ton more memory, etc. Doe…

Thank you.

Re: LLMs work best when the user defines their acceptance criteria first

#363

Earlier quoted context omitted.

So in my experience with Opus 4.6 evaluating it in an existing code base has gone like this. You say "Do this thing". - It does the thing (takes 15 min). Looks incredibly fast. I couldn't code that fast. It's inhuman. So far all the fantastical claims hold up. But still. You ask "Did you do the thing?" - it says oops I forgot to do that sub-thing. (+5m) - it fixes the sub-thing (+10m) You say is the change well integ…

You didn't use plan mode.

I did use plan mode. Plan looked great. Code left something else to be desired.

Re: LLMs work best when the user defines their acceptance criteria first

#364

Earlier quoted context omitted.

If I was on the "replace all the meatsacks AGI ftw" team then I would have referred to it as an oracle, by your own logic, wouldn't I have? It's a tool. It's good for some things, not for others. Use the right tool for the job and know the job well enough to know which tools apply to which tasks. More than anything it's a learning tool. It's also wildly effective at writing code, too. But, man... the things that it m…

>I used it to help me turn a cat exercise wheel (think huge hamster wheel) into a generator that produces enough power to charge a battery that powers an ESP32 powered "CYD" touchscreen LCD that also utilizes a hall effect sensor to monitor, log and display the RPMs and "speed" (given we know the wheel circumference) in real time as well as historically. So what? That's honestly amateur hour. And the LLM derived all…

Right - it's not a big deal and it LITERALLY is amateur hour. But I did it. I wouldn't have done it prior, sure I could have done a bunch of google searches but the time investment it would have taken to sift through all that information and distill it into actionable chunks would have far exceeded the benefit of doing so, in this case.

The whole point is that it is amateur hour and it's wildly effective as a learning tool.

The fact it derived everything from things that have been done... yea, that's also the point? What point are you trying to make here? I'm well aware it's not a great tool if you're trying to use it to create novel things... but I'm not a nuclear physicist. I'm a builder, fixer, tinkerer who happens to make a living writing code. I use it to teach me how to do things, I use it to analyze problems and recommend approaches that I can then delve into myself.

I'm not asking it to fold proteins. (I guess that's been done quite a bit too, so would be amateur as well)

Re: LLMs work best when the user defines their acceptance criteria first

#365

Earlier quoted context omitted.

If I was on the "replace all the meatsacks AGI ftw" team then I would have referred to it as an oracle, by your own logic, wouldn't I have? It's a tool. It's good for some things, not for others. Use the right tool for the job and know the job well enough to know which tools apply to which tasks. More than anything it's a learning tool. It's also wildly effective at writing code, too. But, man... the things that it m…

Oracles have their use too, but as long as you keep confusing "oracle" and "tool" you will get nowhere. P.S. The real big deal is the democratization of oracles. Back in the day building an oracle was a megaproject accessible only to megacorps like Google. Today you can build one for nothing if you have a gaming GPU and use it for powering your kobold text adventure session.

> Oracles have their use too, but as long as you keep confusing "oracle" and "tool" you will get nowhere.

Arguably, I'm getting somewhere.. ;)

Re: LLMs work best when the user defines their acceptance criteria first

#366

Earlier quoted context omitted.

You didn't use plan mode.

I did use plan mode. Plan looked great. Code left something else to be desired.

I've had very consistent success with plan mode, but when I haven't I've noticed many times it's been working with code/features/things that aren't well defined. ie: not using a well defined design pattern, maybe some variability in the application on how something could be done - these are the things I notice it really trips up on. Well defined interfaces, or even specifically telling it to identify and apply design principles where it seems logical.

When I've had repeated issues with a feature/task on existing code often times it really helps to first have the model analyze the code and recommend 'optimizations' - whether or not you agree/accept, it'll give you some insight on the approach it _wants_ to take. Adjust from there.

Re: LLMs work best when the user defines their acceptance criteria first

#367

Earlier quoted context omitted.

They aren't holding it wrong, it's a fundamental limitation of not writing the code yourself . You can make it easier to understand later when you review it, but you still need to put in that effort.

Work in smaller parts then. You should have a mental model of what the code is doing. If the LLM is generating too much you’re being too broad. Break the problem down. Solve smaller problems. All the old techniques and concepts still apply.

This

Re: LLMs work best when the user defines their acceptance criteria first

#368
post #342
post #215

Earlier quoted context omitted.

I’m not a SWE by trade so I could care less about your last comment. But again this is all… vague. I’m personally not convinced at all. I’ll be hiring for a large project soon, so I’ll see for myself what benefits (well I care about net benefits) these tools are providing in the workplace.

If it wasn’t clear, I don’t have any desire to convince anybody of anything. You don’t believe the future is here yet? Good luck holding on to that position. Not my problem. I was taking time to try to help somebody who sounded genuinely curious and seeking help. That I’m happy to do.

You’re writing novels when if you had something compelling to show it’d be simple and easy.

If you can’t make it simple and easy… then you haven’t understood it at all. All geniuses refer to this as the standard by which one understands something. Whether it’s Steve Jobs or Einstein. So don’t get mad. Show us all how simple and easy it is. If you can’t.. then accept you’re full of it and don’t quite get it as well as you claim. Not rocket science is it?

But here we are. And actually my project is going to create the future. You’re a bozo programmer who creates the future that others already see. Know your role and don’t speak for others like me who are in the position of choosing who gets hired.

Re: LLMs work best when the user defines their acceptance criteria first

#369
post #343
post #59

Earlier quoted context omitted.

But isn't that a reflection of reality? If you've made a significant investment in human capital, you're even more likely to protect it now and prevent posting valuable stuff on the web.

No?

Yes it is. There’s a reason why university knowledge is gated. And was gated for centuries.

Can’t believe I have to explain simple stuff.

Re: LLMs work best when the user defines their acceptance criteria first

#370

Earlier quoted context omitted.

And it’s slower to review because you didn’t do the hard part of understanding the code as it was being written.

The same as asking one of your JRs to do something except now it follows instructions a little bit better. Coding has never been about line generation and now you can POC something in a few hours instead of a few days / weeks to see if an idea is dumb.

LLMs can easily output overwhelming quantities of code. Junior devs couldn't really do that, not consistently.

Scale/quantity matter.

This industry is not mature enough for 1000x the bad code we have now. It was barely hanging on with 1x bad code.

Post reply on HN