i'm not sure why people expect agents to one shot everything to perfection with just a prompt. There's a reason why we talk about software development lifecycle, design, architecture, testing ... It's because it's been the most reliable way to build and ship software. We shouldn't expect discard this and expect agents to perform well outside of this. I'm treating LLM agents as junior devs who happen to have vast know…
> i'm not sure why people expect agents to one shot everything to perfection with just a prompt. because that's how agents are marketed.
The Vibe Tax
131–136 of 136 posts
Re: The Vibe Tax
#132Earlier quoted context omitted.
we should exercise critical thinking then heaps of people on this site expect them to be omnipotent then claim it’s fake when it doesn’t read minds
I think it's perfectly fair to evaluate the tools based on how well they live up to the hype that is being pumped out by the sellers of said tools. If they want us to compare their products to a more measured, reasonable take then they can advertise them as that.
I don't care what the company claims, I just use the tool the way I want to. I work very closely with the AI. I'll tell it to plan a change, review the plan, then execute. Then I'll test the changes and have it fix whatever I'm not happy with one thing at a time. I'll specify in detail both what to do and loosely describe how to do it or if I'm not sure I'll ask it to plan the change then review the plan and ask for changes if I want them etc. I also review my own PRs before I submit them to colleagues.
This way I maintain full control of everything, it just saves me hours of googling, planning and typing code - which I do miss a bit but I can't really justify writing code myself when I can achieve the same thing just by loosely describing my idea instead. It also saves a lot of time debugging, I think I'm generally a pretty good programmer but the AI makes fewer mistakes than me. It'll often catch some logic error I made during planning and suggest a good alternative.
A lot of developers seem to give up control entirely and then complain that they're no longer in control. Trying for that 10-100x speedup doing weeks of work in a day. I'm happy doing one week of work in a day. There's a limit to how much I can oversee without compromising quality.
Re: The Vibe Tax
#133i'm not sure why people expect agents to one shot everything to perfection with just a prompt. There's a reason why we talk about software development lifecycle, design, architecture, testing ... It's because it's been the most reliable way to build and ship software. We shouldn't expect discard this and expect agents to perform well outside of this. I'm treating LLM agents as junior devs who happen to have vast know…
Using AI is kayfabe. What I mean is, you create interaction patterns that resemble how humans work. This is because it is what the models are trained on but also because we've all been trained to interact in this way. So it manipulates you into providing more useful prompts. But I don't really want to play a part in a simulation, trying to cajole my scene partners into saying the lines I need them to say. I want to u…
That's why you tell it how to write the code, and then review the code to ensure it is what you wanted.
> We still haven't nailed the right amount of follow up and correction
I feel like I've got it under control. It's not really a problem at all to me. I just work closely with the AI. I do small tasks, I don't just have it generate thousands of lines at once. I tell it what to do and how, or give it some vague guidance and ask it to make a plan. Then review the plan, ask for some changes if necessary and execute. Then I go over all the changes, test them to ensure they work properly, have it fix any issues I find and so on.
I think the main problem with AI coding is people try to do too much. You can't keep a tight leash on it while also having it do a week's worth of work in an hour. I do one task at a time and I am heavily involved in it, deciding exactly how it's done. I micromanage the crap out of that thing. I write commits myself and I always review my own PR before submitting it to colleagues.
Works great. I get things done much faster than I used to, with better quality than before.
Re: The Vibe Tax
#134Earlier quoted context omitted.
- https://openai.com/index/introducing-the-codex-app/ no where does this document suggest that codex can "one shot everything to perfection with just a prompt". It describes using a prompt plus agent skills (which are essentially many other prompts) to develop a playable game.. nothing about it being perfect or anything more than being in a playable state.
Oh please, you sound so disingenuous, the claim wasn’t that the documents contained a specific phrase. You’re moving the goal posts. Their name is literally a play on anthropomorphising the models, such as… the ceo going on tv shows and repeatedly saying the models may be conscious and they may start nuclear wars, etc.
Re: The Vibe Tax
#135Earlier quoted context omitted.
I think this is the only sensible way to work with agents, if you care about code quality and reliability but still want the benefits of AI. There seem to be three camps that people more or less fall into: (a) AI is terrible/bad/evil and should never be used, (b) you should one-shot everything and be happy if it seems to "work" when you try it, (c) the middle ground, where the AI writes code which you carefully revie…
I think c is the only way it can sustainably work. The idea of b, that software is running and nobody there knows how it works, doesn’t seem like a good foundation for a business to run on.
The sunk cost fallacy bites and then bites again and again.
Re: The Vibe Tax
#136Earlier quoted context omitted.
I think it must depend at least partially on the task, too. At the extreme, there are things where you won't care beyond "it seems to work" because you only needed it to run once and it got useful results.
Yes, but all the "don't care" stuff needs to be inside a "don't care" module. For example, I don't actually care about GUI code. I just need "pressing this button emits this event or callback etc". If the GUI module interface is well defined then it doesn't matter how it's implemented as long as it works. The problems start when you don't have clean separations. If your GUI code is also implementing ad hoc business l…