Live data from Hacker News

Opus 4.5 is not the normal AI agent experience that I have had thus far

burkeholland.github.io

801–810 of 1001 posts

Re: Opus 4.5 is not the normal AI agent experience that I have had thus far

#801

Earlier quoted context omitted.

I really think a lof of people tried AI coding earlier, got frustrated at the errors and gave up. That's where the rejection of all these doomer predictions comes from. And I get it. Coding with Claude Code really was prompting something, getting errors, and asking it to fix it. Which was still useful but I could see why a skilled coder adding a feature to a complex codebase would just give up Opus 4.5 really is at a…

> "asking it to fix it." This is what people are still doing wrong. Tools in a loop people, tools in a loop. The agent has to have the tools to detect whatever it just created is producing errors during linting/testing/running. When it can do that, I can loop again, fix the error and again - use the tools to see whether it worked. I _still_ encounter people who think "AI programming" is pasting stuff into ChatGPT on…

This is kind of why I'm not really scared of losing my job.

While Claude is amazing at writing code, it still requires human operators. And even experienced human operators are bad at operating this machinery.

Tell your average joe - the one who thinks they can create software without engineers - what "tools-in-a-loop" means, and they'll make the same face they made when you tried explaining iterators to them, before LLMs.

Explain to them how typing system, E2E or integration test helps the agent, and suddenly, they now have to learn all the things they would be required to learn to be able to write on their own.

Re: Opus 4.5 is not the normal AI agent experience that I have had thus far

#802

What bothers me about posts like this is: mid-level engineers are not tasked with atomic, greenfield projects. If all an engineer did all day was build apps from scratch, with no expectation that others may come along and extend, build on top of, or depend on, then sure, Opus 4.5 could replace them. The hard thing about engineering is not "building a thing that works", its building it the right way, in an easily unde…

Not necessarily responding to you directly, but I find this take to be interesting, and I see it every time an article like this makes the rounds. Starting back in 2022/2023: - (~2022) It can auto-complete one line, but it can't write a full function. - (~2023) Ok, it can write a full function, but it can't write a full feature. - (~2024) Ok, it can write a full feature, but it can't write a simple application. - (~2…

I haven't seen an AI successfully write a full feature to an existing codebase without substantial help, I don't think we are there yet.

> The only question is how long it takes to get there.

This is the question and I would temper expectations with the fact that we are likely to hit diminishing returns from real gains in intelligence as task difficulty increases. Real world tasks probably fit into a complexity hierarchy similar to computational complexity. One of the reasons that the AI predictions made in the 1950s for the 1960s did not come to be was because we assumed problem difficulty scaled linearly. Double the computing speed, get twice as good at chess or get twice as good at planning an economy. P, NP separation planed these predictions. It is likely that current predictions will run into similar separations.

It is probably the case that if you made a human 10x as smart they would only be 1.25x more productive at software engineering. The reason we have 10x engineers is less about raw intelligence, they are not 10x more intelligent, rather they have more knowledge and wisdom.

Re: Opus 4.5 is not the normal AI agent experience that I have had thus far

#803
Does anyone have a boring, multi-hour-long coding session with an agent that they've recorded and put on Vimeo or something?

As many other commentators have said, individual results vary extremely widely. I'd love to be able to look at the footage of either someone who claims a 10x productivity increase, or someone who claims no productivity increase, to see what's happening.

Re: Opus 4.5 is not the normal AI agent experience that I have had thus far

#804

Earlier quoted context omitted.

With all due respect somebody could launch a version of Discord that's 10x faster tomorrow and nobody would know about it It's very difficult to unseat those incumbents, especially those with strong network effects. Plus the people that work in those larger companies are not at the edge of AI coding at all and not motivated to rock the boat

Totally disagree. One example is Zed which is very well known and it's faster than any other editor, wasn't built with AI though. > People on larger companies are not at the edge of AI coding False Microsoft is all in with Copilot, and I can't believe the company that created Copilot doesn't use it internally, I'd rather say they should be the ones that would know how to master it! Yet no better vscode, still bloated…

Do you mean to say Zed wasn't vibe coded? There's actually another comment on this post describing how someone is using Opus 4.5 to work on Zed. Given how forward the AI features are in Zed I'd be surprised if the team wasn't also embracing it internally.

It's a fair question how much AI is accelerating the development of Zed, but I can say that I've been impressed with the speed they are shipping at.

Re: Opus 4.5 is not the normal AI agent experience that I have had thus far

#805

Every time I see a post like this on HN I try again and every time I come to the same conclusion. I have never see one agent managing to pull something off that I could instantly ship. It still ends up being very junior code. I just tried again and ask Opus to add custom video controls around ReactPlayer. I started in Plan mode which looked overal good (used our styling libs, existing components, icons and so on). I…

These posts are never, never made by someone who is responsible for shipping production code in a large, heavily used application. It's always someone at a director+ level who stopped production coding years ago, if they ever did, and is tired of their engineers trying to explain why something will take more than an hour.

It is also often low-proficiency developers with their minds blown over how quickly they can build something using frameworks / languages they never wanted to learn or understand.

Though even that group probably has some overlap with yours.

Re: Opus 4.5 is not the normal AI agent experience that I have had thus far

#806

To the author: you wrote those apps. Not like you used to, but you wrote them. IMO, our jobs are safe. It's our ways of working that are changing. Rapidly.

SWE jobs are in fact, not safe, if vaguely defined specifications can be translated into functioning applications. I don't think agents are good enough to do that in larger applications yet, but it is something to consider.

Depends on the software. IMO, development speed will increase, but humans will continue to be the limiting factor, so we are safe. Our jobs, however, are changing and will continue to.

Re: Opus 4.5 is not the normal AI agent experience that I have had thus far

#807

Earlier quoted context omitted.

I made a similar comment on a different thread, but I think it also fits here: I think the disconnect between engineers is due to their own context. If you work with frontend applications, specially React/React Native/HTML/Mobile, your experience with LLMs is completely different than the experience of someone working with OpenGL, io_uring, libev and other lower level stuff. Sure, Opus 4.5 can one shot Windows utilit…

I've had pretty good luck with LLM agents coding C. In this case a C compiler that supports a subset of C and targets a customizable microcoded state machine/processor. Then I had Gemini code up a simulator/debugger for the target machine in C++ and it did it in short order and quite successfully - lets you single step through the microcode and examine inputs (and set inputs), outputs & current state - did that in an…

That's remarkably similar to something I've just started on - I want to create a self-compiling C compiler targeting (and to run on) an 8-bit micro via a custom VM. This a basically a retro-computing hobby project.

I've worked with Gemini Fast on the web to help design the VM ISA, then next steps will be to have some AI (maybe Gemini CLI - currently free) write an assembler, disassembler and interpreter for the ISA, and then the recursive descent compiler (written in C) too.

I already had Gemini 3.0 Fast write me a precedence climbing expression parser as a more efficient drop-in replacement for a recursive descent one, although I had it do that in C++ as a proof-of-concept since I don't know yet what C libraries I want to build and use (arena allocator, etc). This involved a lot of copy-paste between Gemini output and an online C++ dev environment (OnlineGDB), but that was not too bad, although Gemini CLI would have avoided that. Too bad that Gemini web only has "code interpreter" support for Python, not C and/or C++.

Using Gemini to help define the ISA was an interesting process. It had useful input in a "pair-design" process, working on various parts of the ISA, but then failed to bring all the ideas together into a single ISA document, repeatedly missing parts of what had been previously discussed until I gave up and did that manually. The default persona of Gemini seems not very well suited to this type of work flow where you want to direct what to do next, since it seems they've RL'd the heck out of it to want to suggest next step and ask questions rather than do what is asked and wait for further instruction. I eventually had to keep asking it to "please answer then stop", and interestingly quality of the "conversation" seemed to fall apart after that (perhaps because Gemini was now predicting/generating a more adversarial conversation than a collaborative one?).

I'm wondering/hoping that Gemini CLI might be better at working on documentation than Gemini web, since then the doc can be an actual file it is editing, and it can use it's edit tool for that, as opposed to hoping that Gemini web can assemble chunks of context (various parts of the ISA discussion) into a single document.

Re: Opus 4.5 is not the normal AI agent experience that I have had thus far

#808
post #755

It’s a bit strange how anecdotes have become acceptable fuel for 1000 comment technical debates. I’ve always liked the quote that sufficiently advanced tech looks like magic, but its mistake to assume that things that look like magic also share other properties of magic. They don’t. Software engineering spans over several distinct skills: forming logical plans, encoding them in machine executable form(coding), making…

> It’s a bit strange how anecdotes have become acceptable fuel for 1000 comment technical debates. It's a very subjective topic. Some people claim it increases their productivity 100x. Some think it is not fit for purpose. Some think it is dangerous. Some think it's unethical. Weirdly those could all be true at the same time, and where you land on this is purely a matter of importance to the user. > Yet, I’d love to…

How many 1+ hour videos of someone building with AI tools have you sought out and watched? Those definitely exist, it sounds like you didn't go seeking them out or watch them because even with 7 less hours you'd better understand where they add value enough to believe they can help with challenging projects.

So why should anybody produce an 8 hour video for you when you wouldn't watch it? Let's be real. You would not watch that video.

In my opinion most of the people who refuse to believe AI can help them while work with software are just incurious/archetypical late adopters.

If you've ever interacted with these kinds of users, even though they might ask for specs/more resources/more demos and case studies or maturity or whatever, you know that really they are just change-resistant and will probably continue to be as as long as they can get away with it being framed as skepticism rather than simply being out of touch.

I don't mean that in a moralizing sense btw - I think it is a natural part of aging and gaining experience, shifting priorities, being burned too many times. A lot of business owners 30 years ago probably truly didn't need to "learn that email thing", because learning it would have required more of a time investment than it would yield, due to being later in their career with less time for it to payoff, and having already built skills/habits/processes around physical mail that would become obsolete with virtual mail. But a lot of them did end up learning that email thing 5, 10, whatever years later when the benefits were more obvious and the rest of the world had already reoriented itself around email. Even if they still didn't want to, they'd risk looking like a fossil/"too old" to adapt to changes in the workplace if they didn't just do it.

That's why you're seeing so many directors/middle managers doing all these though leader posts about AI recently. Lots of these guys 1-2 years ago were either saying AI is spicy autocomplete or "our OKR this quarter is to Do AI Things". Now they can't get away with phoning it in anymore and need to prove to their boss that they are capable of understanding and using AI, the same way they had to prove that they understood cloud by writing about kubernetes or microservices or whatever 5-10 years ago.

Re: Opus 4.5 is not the normal AI agent experience that I have had thus far

#809
post #391

Earlier quoted context omitted.

A greenfield project is definitely 'easy mode' for an LLM; especially if the problem area is well understood (and documented). Opus is great and definitely speeds up development even in larger code bases and is reasonably good at matching coding style/standard to that of of the existing code base. In my opinion, the big issue is the relatively small context that quickly overwhelms the models when given a larger task…

This will work (if you add more details): "Have an agent investiate issue X in modules Y and Z. The agent should place a report at ./doc/rework-xyz-overview.md with all locations that need refactoring. Once you have the report, have agents refactor 5 classes each in parallel. Each agent writes a terse report in ./doc/rework-xyz/ When they are all done, have another agent check all the work. When that agent reports ev…

And you can automate all this so that it happens every time. I have an `/implement` command that is basically instructed to launch the agents and then do back and forth between them. Then there's a claude code hook that makes sure that all the agents, including the orchestrator and the agents spawned have respected their cycles - it's basically running `claude` with a prompt that tells it to read the plan file and see if the agents have done what they were expected in this cycle - gets executed automatically on each agent end.

Re: Opus 4.5 is not the normal AI agent experience that I have had thus far

#810
post #752

Every time I see a post like this on HN I try again and every time I come to the same conclusion. I have never see one agent managing to pull something off that I could instantly ship. It still ends up being very junior code. I just tried again and ask Opus to add custom video controls around ReactPlayer. I started in Plan mode which looked overal good (used our styling libs, existing components, icons and so on). I…

I've spent quite a bit of time with Codex recently and come to the conclusion that you can't simply say "Let's add custom video controls around ReactPlayer." You need to follow up with a set of strict requirements to set expectations, guard rails, and what the final product should do (and not do). Even then it may have a few issues, but continuing to prompt with clearly stated problems that don't meet the requirement…

>You need to follow up with a set of strict requirements to set expectations, guard rails, and what the final product should do (and not do).

that usually very hard to do part, and is was possible to spent few days on something like that in real word before LLMs. But with LLMs is worse because is not enough to have those requirements, some of those won't work for random reasons and is no any 'rules' that can grantee results. It always like 'try that' and 'probably this will work'.

Just recently I was struggled with same prompt produced different result between API calls before I realized that just usage of few more '\"' and few spaces in prompt leaded model to completely different route of logic which produced opposite answers.

Post reply on HN