Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things that helped me get out of the AI 10x engineer imposter syndrome

colton.dev

Post reply on HN
Print view View original

361–370 of 675 posts

Re: Things that helped me get out of the AI 10x engineer imposter syndrome

#361

Post by rightbyte »

coffeebeqn wrote: post #269
doc_manhat_3 wrote: post #190

Earlier quoted context omitted.

I think even claims of 2-5x are highly suspect. It would imply that if your team is using AI then all else equal they accomplish 2-5 times as much in a quarter. I don't know about you but I'm certainly not seeing this and most people on my team use AI. [And to those saying we're using it wrong... well I can't argue with something that's not falsifiable]

My company is all in on LLMs and honestly the improvement seems to be like 0.9x to 1.2x depending on the project. None of them are moving at break neck speed and many projects are just as bogged down by complexity as ever. Pretty big (3000+) company with a large mature codebase in multiple languages. For god knows how much money spent on it

Do you mean -10 to +20% or 90 to 120%?

I am not allowed to use LLMs at work for work code so I can't tell what claims are real. Just my 80s game reimplementations of Snake and Asteroids.

Re: Things that helped me get out of the AI 10x engineer imposter syndrome

#362

Post by SkyPuncher »

simonw wrote: post #77

I found myself agreeing with quite a lot of this article. I'm a pretty huge proponent for AI-assisted development, but I've never found those 10x claims convincing. I've estimated that LLMs make me 2-5x more productive on the parts of my job which involve typing code into a computer, which is itself a small portion of that I do as a software engineer. That's not too far from this article's assumptions. From the artic…

I've basically come to the same 2x to 5x conclusion as you. Problem is that "5x productivity" is really only a small portion of my actual job.

The hardest part of my job is actually understanding the problem space and making sure we're applying the correct solution. Actual coding is probably about 30% of my job.

That means, I'm only looking at something like 30% productivity gain by being 5x as effective at coding.

Re: Things that helped me get out of the AI 10x engineer imposter syndrome

#363

Post by coltonv »

kasey_junk wrote: post #230
coltonv wrote: post #117

Earlier quoted context omitted.

My goal here was not to publicly call out any specific individual or article. I don't want to make enemies and I don't want to be cast as dunking on someone. I get that that opens me up to criticism that I'm fighting a strawman, I accept that. Your article does not specifically say 10x, but it does say this: > Kids today don’t just use agents; they use asynchronous agents. They wake up, free-associate 13 different th…

I believe that quote in Thomas’ blog can be attributed to me. I’ve at least said something near enough to him that I don’t mind claiming it. I _never_ made the claim that you could call that 10x productivity improvement. I’m hesitant to categorize productivity in software in numeric terms as it’s such a nuanced concept. But I’ll stand by my impression that a developer using ai tools will generate code at a perceptibl…

Appreciate the comment!

> I mentioned in another comment the major flaw in your productivity calculation, is that you aren’t accounting for the work that wouldn’t have gotten done otherwise. That’s where my improvements are almost universally coming from. I can improve the codebase in ways that weren’t justifiable before in places that do not suffer from the coordination costs you rightly point out.

I'm a bit confused by this. There is work that apparently is unlocking big productivity boosts but was somehow not justified before? Are you referring to places like my ESLint rule example, where eliminating the startup costs of learning how to write one allows you to do things you wouldn't have previously bothered with? If so, I feel like I covered this pretty well in the article and we probably largely agree on the value that productivity boost. My point is still stands that that doesn't scale. If this is not what you mean, feel free to correct me.

Appreciate your thoughts on hallucinations. My guess is the difference between what we're experiencing is that in your code hallucinations are still happening but getting corrected after tests are run, whereas my agents typically get stuck in these write-and-test loops and can't figure out how to solve the problem, or it "solves" it by deleting the tests or something like that. I've seen videos and viewed open source AI PRs which end up in similar loops as to what I've experienced, so I think what I see is common.

Perhaps that's an indication of that we're trying to solve different problems with agents, or using different languages/libraries, and that explains the divergence of experiences. Either way, I still contend that this kind of productivity boost is likely going to be hard to scale and will get tougher to realize as time goes on. If you keep seeing it, I'd really love to hear more about your methods to see what I'm missing. One thing that has been frustrating me is that people rarely share their workflows after makign big claims. This is unlike previous hype cycles where people would share descriptions of exactly what they did ("we rewrote in Rust, here's how we did it", etc.) Feel free to email me at the address in my about page[1] or send me a request on LinkedIn or whatever. I'm being 100% genuine that I'd love to learn from you!

[1] https://colton.dev/about/

Re: Things that helped me get out of the AI 10x engineer imposter syndrome

#364

Post by skinnymuch »

docmars wrote: post #330
mavamaarten wrote: post #101

Earlier quoted context omitted.

Yeah. I just need to babysit it too much. Take copilot, it gives good suggestions and blows me away sometimes with a block of code which is exactly what I'd type. But actively letting it code (at least with gpt4.1 or gpt4o) just doesn't work well enough for me. Half of the time it doesn't even compile, and after fixing that it's just not really correctly working either. I'd expect it to work like a very junior progra…

With something like Devin, where it integrates directly with your repo and generates documentation based on your project(s), it's much more productive to use as an agent. I can delegate like 4-5 small tasks that would normally take me a full day or two (or three) of context switching and mental preparation, and knock them out in less than a day because it did 50-80% of the work, leaving only a few fixes or small pivo…

Is work paying for Devin or you are? How pricey is it to delegate the task example you gave?

Re: Things that helped me get out of the AI 10x engineer imposter syndrome

#365

Post by tptacek »

timmytokyo wrote: post #344
tptacek wrote: post #125

Earlier quoted context omitted.

That's not what people mean when they bring up "hallucinations". What the author apparently meant was that they had an agent generating Terraform for them, and that Terraform was broken. That's not surprising to me! I'm sure LLMs are helpful for writing Terraform, but I wouldn't expect that agents are at the point of being able to reliably hand off Terraform that actually does anything, because I can't imagine an age…

As if a compiler or linter is the sole arbiter of correctness.

Nobody said anything about "correctness". Hallucinations aren't bugs. Everybody writes bugs. People writing code don't hallucinate.

It's a pretty obvious rhetorical tactic: everybody associates "hallucination" with something distinctively weird and bad that LLMs do. Fair enough! But then they smuggle more meaning into the word, so that any time an LLM produces anything imperfect, it has "hallucinated". No. "Hallucination" means that an LLM has produced code that calls into nonexistent APIs. Compilers can and do in fact foreclose on that problem.

Re: Things that helped me get out of the AI 10x engineer imposter syndrome

#366

Post by itissid »

Human goals are more important. I think conceptually the idea should always be strong goals set by humans and then sub goals each with a praticularly well defined *plan* for meeting them. This needs to be the conceptual basis, if you are having to plan for 50% or 75%(gasp) of the time for a feature and then AI just writes code, that is not intelligence much less a 10x engineer.

My use case is not for a 10x engineer but instead for *cognitive load sharing*. I use AI in a "non-linear" fashion. Do you? Here is what that means:

1. Brainstorm an idea and write down detailed enough plan. Like tell me how I might implement something or here is what I am thinking can you critique and compare it with other approaches. Then I quickly meet with 2 more devs and make a design decision for which one to use.

2. Start manual coding and let AI "fill the gaps": Write these test for my code or follow this already existing API and create the routes from this new spec. This is non-linear because I would complete 50-75% of the feature and let the rest be completed by AI.

3. I am tired and about to end my shift and there is this last bug, I go read the docs but I also ask AI to read my screen and come up with some hypothesis to come up with. I decide which hypothesis are most promising after some reading and then ask the AI to just test that(not fix it on auto mode).

4. Voice mode: I have a shortcut that triggers claude code and uses it like a quick "lookup/search" in my code base. This avoids context switching.

Re: Things that helped me get out of the AI 10x engineer imposter syndrome

#367

Post by tptacek »

bit1993 wrote: post #354
kasey_junk wrote: post #257

Earlier quoted context omitted.

Every. Single. Time. You say you get productivity gains from ai tools on the internet someone will tell you that you weren’t good at your job before the ai tooling. Perhaps, start from the assumption that I have in fact spent a fair bit of time doing this job at a high level. Where does that mental exercise take you with regard to your own position on ai tools. In fact, you don’t have to assume I’m qualified to speak…

I think what GP is saying is that in most cases generating allot of code is not a good thing. Every line of LLM generated code has to be audited because they are prone to hallucinations and auditing someone else's code is much more difficult and time consuming than auditing your own code. Allot of code also requires more maintenance.

The comment is premised on the idea that Kasey either doesn't know what a "masterful developer" is or needs to be corrected back to it.

Re: Things that helped me get out of the AI 10x engineer imposter syndrome

#368

Post by coltonv »

jonas21 wrote: post #335

> You can't compress the back and forth of 3 months of code review into 1.5 weeks. If your organization is routinely spending 3 months on a code review, it sounds like there's probably a 10 to 100x improvement you can extract from fixing your process before you even start using AI.

I think I may have worded this poorly. I mean the total amount of code review time that goes into 3 months of work (likely on hundreds of PRs) can't be compressed into 1.5 weeks at the same portion of time being allocated to code review. Each code review has a "floor" time, a minimum amount of time loss due to context switching, reading, writing, etc.

Re: Things that helped me get out of the AI 10x engineer imposter syndrome

#369

Post by jonas21 »

bumblehean wrote: post #275
lvl155 wrote: post #227

Earlier quoted context omitted.

Counter: you are looking at it wrong. You can get work done in 1/2 of the time it used to. Now you got 1/2 of the day to just mess around. Socialize or network. It’s not necessarily that you’re producing 2x.

> You can get work done in 1/2 of the time it used to. Now you got 1/2 of the day to just mess around. Socialize or network. This has never been the case in any company I've ever worked at. Even if you can finish your day's work in, say, 4 hours, you can't just dip out for the other 4 hours of the day. Managers and teammates expect you to be available at the drop of a hat for meetings, incidents, random questions, "e…

If you're remote, you can. This is the crux of why a lot of developers love remote work and management hates it.
Post reply on HN