Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Levels of Agentic Engineering

bassimeledath.com

Post reply on HN
Print view View original

41–50 of 152 posts

Re: Levels of Agentic Engineering

#41

Post by ramoz »

Level4 is most interesting to me right now. And I would say we as an industry are still figuring out the right ergonomics and UX around these four things.

I spend a great deal of my time planning and assessing/reviewing through various mechanisms. I think I do codify in ways when I create a skill for any repeated assessment or planning task.

> To be clear, planning as a general practice isn't going away. It's just changing shape. For newer practitioners, plan mode remains the right entry point (as described in Levels 1 and 2). But for complex features at Level 7, "planning" looks less like writing a step-by-step outline and more like exploration: probing the codebase, prototyping options in worktrees, mapping the solution space. And increasingly, background agents are doing that exploration for you.

I mean, it's worth noting that a lot of plan modes are shaped to do the Socratic discovery before creating plans. For any user level. Advanced users probably put a great deal of effort (or thought) into guiding that process themselves.

> ralph loops (later on)

Ralph loops have been nothing but a dramatic mess for me, honestly. They disrupt the assessment process where humans are needed. Otherwise, don't expect them to go craft out extensive PRD without massive issues that is hard to review.

  - It would seem that this is a Harness problem in terms of how they keep an agent working and focused on specific tasks (in relation to model capability), but not something maybe a user should initiate on their own.

Re: Levels of Agentic Engineering

#42

Post by smallnix »

vidimitrov wrote: post #26

Level 4 is where I see the most interesting design decisions get made, and also where most practitioners take a shortcut that compounds badly later. When the author talks about "codifying" lessons, the instinct for most people is to update the rules file. That works fine for conventions - naming patterns, library preferences, relatively stable stuff. But there's a different category of knowledge that rules files hand…

A good rule would then be to capture such reasoning, at least when made during the session with the agent, in the commit messages the agent creates.

Re: Levels of Agentic Engineering

#43

Post by smallnix »

nimasadri11 wrote: post #16

I really like your post and agree with most things. The one thing I am not fully sure about: > Look at your app, describe a sequence of changes out loud, and watch them happen in front of you. The problem a lot of times is that either you don't know what you want, or you can't communicate it (and usually you can't communicate it properly because you don't know exactly what you want). I think this is going to be the b…

Reminds me of a colleague who said they don't need to learn to type faster, since they use the time to think what they want to write.

Re: Levels of Agentic Engineering

#44

Post by jjmarr »

moosehater wrote: post #33
jjmarr wrote: post #10

I coded a level 8 orchestration layer in CI for code review, two months before Claude launched theirs. It's very powerful and agents can create dynamic microbenchmarks and evaluate what data structure to use for optimal performance, among other things. I also have validation layers that trim hallucinations with handwritten linters. I'd love to find people to network with. Right now this is a side project at work on t…

Do you feel like you are still learning about the programming language(s) and other technologies you are using? Or do you feel like you are already a master at them? Do you ever take the time to validate what one of the agents produces by going to the docs? Or is all debugging/changing of the code done via LLMs/agents? I'm more like level 2 right now and genuinely curious if you feel like learning continues for you (…

I'm learning more than ever before. I'm not a master at anything but I am getting basic proficiency in virtually everything.

> Do you ever take the time to validate what one of the agents produces by going to the docs? Or is all debugging/changing of the code done via LLMs/agents?

I divide my work into vibecoding PoC and review. Only once I have something working do I review the code. And I do so through intense interrogation while referencing the docs.

> I'm more like level 2 right now and genuinely curious if you feel like learning continues for you (besides with agentic orchestration, etc.)

Level 8 only works in production for a defined process where you don't need oversight and the final output is easy to trust.

For example, I made a code review tool that chunks a PR and assigns rule/violation combos to agents. This got a 20% time to merge reduction and catches 10x the issues as any other agent because it can pull context. And the output is easy to incorporate since I have a manager agent summarize everything.

Likewise, I'm working on an automatic performance tool right now that chunks code, assigns agents to make microbenchmarks, and tries to find optimization points. The end result should be easy to verify since the final suggestion would be "replace this data structure with another, here's a microbenchmark proving so".

Re: Levels of Agentic Engineering

#45

Post by philipp-gayret »

Floating what you call levels 6, 7 and 8. I have a strong harness, but manually kick off the background agents which pick up tasks I queue while off my machine.

I've experimented with agent teams. However the current implementation (in Claude Code) burns tokens. I used 1 prompt to spin up a team of 9+ agents: Claude Code used up about 1M output tokens. Granted, it was a long; very long horizon task. (It kept itself busy for almost an hour uninterrupted). But 1M+ output tokens is excessive. What I also find is that for parallel agents, the UI is not good enough yet when you run it in the foreground. My permission management is done in such a way that I almost never get interrupted, but that took a lot of investment to make it that way. Most users will likely run agent teams in an unsafe fashion. From my point of view the devex for agent teams does not really exist yet.

Re: Levels of Agentic Engineering

#46

Post by vidimitrov »

smallnix wrote: post #42
vidimitrov wrote: post #26

Level 4 is where I see the most interesting design decisions get made, and also where most practitioners take a shortcut that compounds badly later. When the author talks about "codifying" lessons, the instinct for most people is to update the rules file. That works fine for conventions - naming patterns, library preferences, relatively stable stuff. But there's a different category of knowledge that rules files hand…

A good rule would then be to capture such reasoning, at least when made during the session with the agent, in the commit messages the agent creates.

That’s exactly the direction I went with. Working on a spec for exactly this - planning to post it here soon:

https://github.com/berserkdisruptors/contextual-commits

Re: Levels of Agentic Engineering

#47

Post by wild_egg »

dolebirchwood wrote: post #25

> Voice-to-voice (thought-to-thought, maybe?) interaction with your coding agent — conversational Claude Code, not just voice-to-text input — is a natural next step. Maybe it's just me, but I don't see the appeal in verbal dictation, especially where complexity is involved. I want to think through issues deliberately, carefully, and slowly to ensure I'm not glossing over subtle nuances. I don't find speaking to be co…

The power of voice dictation for me is that I can get out every scrap of nuance and insight I can think of as unfiltered verbal diarrhea. Doing this gives me solidly an extra 9 in chance of getting good outputs.

Stream of consciousness typing for me is still slower and causes me to buffer and filter more and deliberately crafting a perfect prompt is far slower still.

LLMs are great at extracting the essence of unstructured inputs and voice lets me take best advantage of that.

Voice output, on the other hand, is completely useless unless perhaps it can play at 4x speed. But I need to be able to skim LLM output quickly and revisit important points repeatedly. Can't see why I'd ever want to serialize and slow that down.

Re: Levels of Agentic Engineering

#48

Post by moosehater »

jjmarr wrote: post #44
moosehater wrote: post #33

Earlier quoted context omitted.

Do you feel like you are still learning about the programming language(s) and other technologies you are using? Or do you feel like you are already a master at them? Do you ever take the time to validate what one of the agents produces by going to the docs? Or is all debugging/changing of the code done via LLMs/agents? I'm more like level 2 right now and genuinely curious if you feel like learning continues for you (…

I'm learning more than ever before. I'm not a master at anything but I am getting basic proficiency in virtually everything. > Do you ever take the time to validate what one of the agents produces by going to the docs? Or is all debugging/changing of the code done via LLMs/agents? I divide my work into vibecoding PoC and review. Only once I have something working do I review the code. And I do so through intense inte…

Got it. This all makes sense to me. Very targeted tooling that is specific to your company's CI platform as opposed to a dark factory where you're creating a bunch of new code no one reads. And it sounds like these level 8 agents are given specific permission for everything they're allowed to do ahead of time. That seems sound from an engineering perspective.

Also would be interested in an example of "validation layers that trim hallucinations with handwritten linters" but understand if that's not something you can share. Either way, thanks for responding!

Re: Levels of Agentic Engineering

#49

Post by cheevly »

glhast wrote: post #21
mzg wrote: post #7

As a lowly level 2 who remains skeptical of these software “dark factories” described at the top of this ladder, what I don’t understand is this: If software engineering is enough of a solved problem that you can delegate it entirely to LLM agents, what part of it remains context-specific enough that it can’t be better solved by a general-purpose software factory product? In other words, if you’re a company that is u…

Also a measly level 2er. I'm curious what kind of project truly needs an autonomous agent team Ralph looping out 10,000 LOCs per hour? Seems like harness-maxxing is a competitive pursuit in its own right existing outside the task of delivering software to customers. Feels like K8s cult, overly focused on the cleverness of _how_ something is built versus _what_ is being built.

Software that is otherwise not feasible for humans to build by hand.

Re: Levels of Agentic Engineering

#50

Post by tkiolp4 »

2001zhaozhao wrote: post #28
mzg wrote: post #7

As a lowly level 2 who remains skeptical of these software “dark factories” described at the top of this ladder, what I don’t understand is this: If software engineering is enough of a solved problem that you can delegate it entirely to LLM agents, what part of it remains context-specific enough that it can’t be better solved by a general-purpose software factory product? In other words, if you’re a company that is u…

Why sell the factory when you can create automated software cloner companies that make millions off of instantly copying promising startups as soon as they come out of stealth? If you could get a dark factory working when others don't have one, you can make much more money using it than however much you can make selling it

That’s not true. Even if we assume LLMs can generate the code needed to support the next Facebook, one still has to: buy/rent tons of hardware (virtual or baremetal), put tons of money in marketing, break the network effect, pay for 3rd party services for monitoring, alerting and what not. That’s money, and LLMs don’t help with that
Post reply on HN