Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shall I implement it? No

gist.github.com

Post reply on HN
Print view View original

211–220 of 603 posts

Re: Shall I implement it? No

#211

Post by sroussey »

cmrdporcupine wrote: post #192
inerte wrote: post #144

Codex has always been better at following agents.md and prompts more, but I would say in the last 3 months both Claude Code got worse (freestyling like we see here) and Codex got EVEN more strict. 80% of the time I ask Claude Code a question, it kinda assumes I am asking because I disagree with something it said, then acts on a supposition. I've resorted to append things like "THIS IS JUST A QUESTION. DO NOT EDIT COD…

I'm back on Claude Code this month after a month on Codex and it's a serious downgrade. Opus 4.6 is a jackass. It's got Dunning-Kruger and hallucinates all over the place. I had forgotten about the experience (as in the Gist above) of jamming on the escape key "no no no I never said to do that." But also I don't remember 4.5 being this bad. But GPT 5.3 and 5.4 is a far more precise and diligent coding experience.

Use cli or extension or the app?

Re: Shall I implement it? No

#212

Post by danjl »

skybrian wrote: post #34

Don't just say "no." Tell it what to do instead. It's a busy beaver; it needs something to do.

Just saying "no" is unclear. LLMs are still very sensitive to prompts. I would recommend being more precise and assuming less as a general rule. Of course you also don't want to be too precise, especially about "how" to do something, which tends to back the LLM into a corner causing bad behavior. Focus on communicating intent clearly in my experience.

Re: Shall I implement it? No

#213

Post by dostick »

Its gotten so bad that Claude will pretend in 10 of 10 cases that task is done/on screenshot bug is fixed, it will even output screenshot in chat, and you can see the bug is not fixed pretty clear there.

I consulted Claude chat and it admitted this as a major problem with Claude these days, and suggested that I should ask what are the coordinates of UI controls are on screenshot thus forcing it to look. So I did that next time, and it just gave me invented coordinates of objects on screenshot.

I consult Claude chat again, how else can I enforce it to actually look at screenshot. It said delegate to another “qa” agent that will only do one thing - look at screenshot and give the verdict.

I do that, next time again job done but on screenshot it’s not. Turns out agent did all as instructed, spawned an agent and QA agent inspected screenshot. But instead of taking that agents conclusion coder agent gave its own verdict that it’s done.

It will do anything- if you don’t mention any possible situation, it will find a “technicality” , a loophole that allows to declare job done no matter what.

And on top of it, if you develop for native macOS, There’s no official tooling for visual verification. It’s like 95% of development is web and LLM providers care only about that.

Re: Shall I implement it? No

#214

Post by clbrmbr »

bushido wrote: post #181
lubujackson wrote: post #173

Earlier quoted context omitted.

I feel like people are sleeping on Cursor, no idea why more devs don't talk about it. It has a great "Ask" mode, the debugging mode has recently gotten more powerful, and it's plan mode has started to look more like Claude Code's plans, when I test them head to head.

Cursor implemented something a while back where it started acting like how ChatGPT does when it's in its auto mode. Essentially, choosing when it was going to use what model/reasoning effort on its own regardless of my preferences. Basically moved to dumber models while writing code in between things, producing some really bad results for me. Anecdotal, but the reason I will never talk about Cursor is because I will…

Same here. Auto mode is NOT ok. Sadly, smaller models cannot be trusted with access to Bash.

Re: Shall I implement it? No

#215

Post by zahlman »

bmurphy1976 wrote: post #90

This drives me crazy. This is seriously my #1 complaint with Claude. I spend a LOT of time in planning mode. Sometimes hours with multiple iterations. I've had plans take multiple days to define. Asking me every time if I want to apply is maddening. I've tried CLAUDE.md. I've tried MEMORY.md. It doesn't work. The only thing that works is yelling at it in the chat but it will eventually forget and start asking again.…

If you could influence the LLM's actions so easily, what would stop it from equally being influenced by prompt injection from the data being processed?

What you need is more fine-grained control over the harness.

Re: Shall I implement it? No

#216

Post by reconnecting »

vidarh wrote: post #188
reconnecting wrote: post #172

Earlier quoted context omitted.

Did you also notice the evolution of average developers over time? I mean, if you take code from a developer ten years ago and compare it with their output now, you can see improvement. I assume that over time, the output improves because of the effort and time the developer invests in themselves. However, LLMs might reduce that effort to zero — we just don't know how developers will look after ten years of using LLM…

> Did you also notice the evolution of average developers over time? I mean, if you take code from a developer ten years ago and compare it with their output now, you can see improvement. This makes little sense to me. Yes, individual developers gets better. I've seen little to no evidence that the average developer has gotten better. > However, LLMs might reduce that effort to zero — we just don't know how developer…

After 30 years in front of the desktop, we are processing dopamine differently.

When I speak about 10 years from now, I’m referring to who will become an average developer if we replace the real coding experience learning curve with LLMs from day one.

I also hear a lot of tool analogies — tractors for developers, etc. But every tool, without an exception, provides replicable results. In the case of LLMs, however, repeatable results are highly questionable, so it seems premature to me to treat LLMs in the same way as any other tool.

Re: Shall I implement it? No

#217

Post by miohtama »

AlotOfReading wrote: post #193
inerte wrote: post #144

Codex has always been better at following agents.md and prompts more, but I would say in the last 3 months both Claude Code got worse (freestyling like we see here) and Codex got EVEN more strict. 80% of the time I ask Claude Code a question, it kinda assumes I am asking because I disagree with something it said, then acts on a supposition. I've resorted to append things like "THIS IS JUST A QUESTION. DO NOT EDIT COD…

I've had some luck taming prompt introspection by spawning a critic agent that looks at the plan produced by the first agent and vetos it if the plan doesn't match the user's intentions. LLMs are much better at identifying rule violations in a bit of external text than regulating their own output. Same reason why they generate unnecessary comments no matter how many times you tell them not to.

How does one integrate critic agent to a Codex/Claude?

Re: Shall I implement it? No

#218

Post by clbrmbr »

sgillen wrote: post #74

To be fair to the agent... I think there is some behind the scenes prompting from claude code (or open code, whichever is being used here) for plan vs build mode, you can even see the agent reference that in its thought trace. Basically I think the system is saying "if in plan mode, continue planning and asking questions, when in build mode, start implementing the plan" and it looks to me(?) like the user switched fr…

This. The models struggle with differentiating tool responses from user messages.

The trouble is these are language models with only a veneer of RL that gives them awareness of the user turn. They have very little pretraining on this idea of being in the head of a computer with different people and systems talking to you at once. —- there’s more that needs to go on than eliciting a pre-learned persona.

Re: Shall I implement it? No

#219

Post by clbrmbr »

bjackman wrote: post #89

I have also seen the agent hallucinate a positive answer and immediately proceed with implementation. I.e. it just says this in its output: > Shall I go ahead with the implementation? > Yes, go ahead > Great, I'll get started.

Hahah yeah if you play with LoRas on local models you will see this a lot. Most often I see it hallucinate a user turn or a system message.

Re: Shall I implement it? No

#220

Post by steelbrain »

dostick wrote: post #213

Its gotten so bad that Claude will pretend in 10 of 10 cases that task is done/on screenshot bug is fixed, it will even output screenshot in chat, and you can see the bug is not fixed pretty clear there. I consulted Claude chat and it admitted this as a major problem with Claude these days, and suggested that I should ask what are the coordinates of UI controls are on screenshot thus forcing it to look. So I did that…

> And on top of it, if you develop for native macOS, There’s no official tooling for visual verification. It’s like 95% of development is web and LLM providers care only about that.

Thinking out loud here, but you could make an application that's always running, always has screen sharing permissions, then exposes a lightweight HTTP endpoint on 127.0.0.1 that when read from, gives the latest frame to your agent as a PNG file.

Edit: Hmm, not sure that'd be sufficient, since you'd want to click-around as well.

Maybe a full-on macOS accessibility MCP server? Somebody should build that!

Post reply on HN