Thinking logically about that system seems to be the most important thing.
The AI Productivity Gap
101–110 of 127 posts
Re: The AI Productivity Gap
#102I don’t think human review is worth it for LLM generated code. We design abstractions and all around how humans think. LLMs writes code that is better understood by machines. If you are all in on LLMs, by all means, read the code figure out what it means. But trying to enforce a human flow to its logic is flawed and will be overwritten the next time.
Human review is even more important for LLM-generated code, because LLMs are so much worse at writing code than humans are. You cannot under any circumstances just let them rip or your results will suck.
Re: The AI Productivity Gap
#103Soon nobody will write code by hand , nobody will read it, and nobody will do code reviews. Architecture too will be done by AI. the one whats unclear to me is, will there still be meetings.
With AI doing all the work there is more time for meetings. Product folks aren't going to tell AI to make the code, they are going to still have a meeting to yell at someone that their AI generated requirements weren't followed the way they wanted. You'll also need meetings to discuss how more AI can prevent the AI from taking down production so much.
Re: The AI Productivity Gap
#104Soon nobody will write code by hand , nobody will read it, and nobody will do code reviews. Architecture too will be done by AI. the one whats unclear to me is, will there still be meetings.
With AI doing all the work there is more time for meetings. Product folks aren't going to tell AI to make the code, they are going to still have a meeting to yell at someone that their AI generated requirements weren't followed the way they wanted. You'll also need meetings to discuss how more AI can prevent the AI from taking down production so much.
product and engineering will gradually collapse into one role
Re: The AI Productivity Gap
#105Earlier quoted context omitted.
The hard part is that LLM code looks like there is some sort of flow. It is like a nice statistical smooth flow. It looks very convincing at a glance. No one would write code like that and not know what they are doing comments self assured and all.
My experience as well, it is too fond of abstractions and will constantly spin up functions like: isUserAdmin(){return user.isAdmin} which look nice on a surface level but obfuscates real understanding of the code and the actual data structures being used. Your end result is pretty and reads nice, but is bloated and difficult to reason with code.
> My experience as well, it is too fond of abstractions and will constantly spin up functions like: isUserAdmin(){return user.isAdmin}
Sounds like Uncle Bob disciples
What a nightmare, AI only knows how to write crappy Clean Code*
Re: The AI Productivity Gap
#106Earlier quoted context omitted.
Idk, ymmv. I’ve had very good outcomes writing embedded firmware for complex systems, drivers, etc. it’s really, really good at not bodging the kinds of things that always catch meaty developers unawares. OTOH, we run extensive harness optimization, where everything is specified in advance, then a plan is made, then a naive review of the plan vs the specification vs the blast radius, then implementation, tests, then…
How much does your whole setup cost? Just curious. An estimated range would be interesting to have
We built all the tooling as Claude-code tools (mostly in python) and the rest is just documentation for preloading the context and as reference material to pull in by policy before certain tasks. Really, there is nothing complicated or heavy, it’s all process.
Re: The AI Productivity Gap
#107Earlier quoted context omitted.
Your process engineering is lacking. Just throwing AI at existing workflows seldom produces good results. Processes have to be reengineered to benefit from the strengths and cover for the weaknesses of AI systems, with observability at the right inflection points being fundamental to success.
Care to give an example?
We run extensive harness optimization, where everything is specified in advance, then a plan is made, then a naive review of the plan vs the specification vs the blast radius, then implementation, tests, then a naive review of test coverage, a naive review of the code vis a vis our code guidelines, a review for smells, a review for silo violation and architecture compliance, a reconciliation of the documentation, then planning the next subfeature, etc. probably 10 percent code generation, 40 percent documentation and planning and adversarial review, 50 percent automated adversarial code review.
We use a Claude for planning and generation, sol for adversarial review. Our metrics say we are about 2x. Productive, at a cost of about $300 per dev per month. We are also shipping less bugs and better, more clearly written documentation (we use technical writing English style guides implemented by Claude)
I think a big part is the constant adversarial review by a different model with no prior context except the coding standards. Also important is context management, we do an onboarding and wrap-up for each session where we have a batch of continuity documents- Learnings, musings, and roadtrips where we let the most successful high-context sessions research and then write about something that they “ became curious about” during the session. That actually brings in a lot of insight to the team and occasionally is brilliant.
Also critical is crossing compaction barriers (standard re-onboarding protocols, writing transition documents prior to compaction, etc)
Also critical is being able to smell when a session is going off the rails. What we do there is sideline the session, wait for master to advance a bit, then have it do a “4c’s” review of the committed work, blast radius analysis, and remediation. If there’s still useful context left we put it into a project unrelated to its failure context. That salvages the valuable session context without staying in the failure trench.
It’s a lot like herding cats.
Re: The AI Productivity Gap
#108Earlier quoted context omitted.
Idk, ymmv. I’ve had very good outcomes writing embedded firmware for complex systems, drivers, etc. it’s really, really good at not bodging the kinds of things that always catch meaty developers unawares. OTOH, we run extensive harness optimization, where everything is specified in advance, then a plan is made, then a naive review of the plan vs the specification vs the blast radius, then implementation, tests, then…
How much does your whole setup cost? Just curious. An estimated range would be interesting to have
About $300 per dev per month. Importantly, we are not buying tokens, but the coding plans.
Re: The AI Productivity Gap
#109Then consider on top that the perceived time needed for implementation has dropped so additional work that would have never been taken up in the past gets added.
Re: The AI Productivity Gap
#110I don't know if I buy the "Pre AI (hours)" and "Post AI (hours)"