Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vibe engineering

simonwillison.net

Post reply on HN
Print view View original

681–690 of 759 posts

Re: Vibe engineering

#681

Post by array_key_first »

herval wrote: post #591
pron wrote: post #580

Earlier quoted context omitted.

Because I reached that position 15 years ago, I can tell you that this is untrue (in the sense that the experience is completely different from an LLM). Training is one thing, but training doesn't increase the productivity of the trainer; it's meant to improve the capability of the trainee. At any level of capability, though - whether we're talking about an intern after one year of university or a senior developer wi…

> effective management requires that you're able to trust that the person tells you when they've hit a snag or anything else you may need to know Nope, effective management is on YOU, not them. If everyone you’re managing is completely transparent and immediately tells you stuff, you’re playing in easy mode

Yes, I want to play in easy mode. Why would I want to play in hard mode?

You're trying to sell AI here, right? And the argument is that AI is like hard mode... which developers are already in, but might not be.

It's just not a very good sales pitch.

Re: Vibe engineering

#682

Post by array_key_first »

latexr wrote: post #396
simonw wrote: post #168

Earlier quoted context omitted.

Absolutely this. LLM assistance means we can work faster, and that we can build things that previously weren't feasible given the available time and resources. Which makes the hardest problem in software even harder: what should we build ? It doesn't matter how fast you can move if you're consistently solving the wrong problems.

> Which makes the hardest problem in software even harder: what should we build? You should build what’s personally fun and challenging to you and/or what is useful and solves a problem. Building for any other reason, including and especially the unfettered pursuit of profit, is what turns everything to shit.

Well, there goes all our jobs.

I agree with you, but unfortunately, big boy gotta eat, and that means we have to sit around writing shitty commercial software. I already hate that, I would prefer if it didn't get worse.

Re: Vibe engineering

#683

Post by sarchertech »

simonw wrote: post #679
sarchertech wrote: post #621

Earlier quoted context omitted.

Nice to see that you recognize that! > One of the more "engineering" like skills in using this stuff is methodically figuring out what's a superstition and what actually works. The problem is there are so many variables and the system is so chaotic that this is a nearly impossible task for things that don’t have an absolutely enormous effect size. For most things you’re testing, you need to run the experiment many ma…

Here's an example from today of something I just figured out. I had Claude Code do some work which I pushed as a branch to GitHub. Then I opened a PR so I could more easily review it and added a bunch of notes and comments there. On a hunch, I pasted the URL to that PR into Claude Code and said "use the GitHub API to fetch the notes on this PR"... ... and it did exactly that. It guesses the API URL, fetched the JSON…

How is that an example of something you are doing that might be a superstition?

You asked it to do a single easily verifiable task and it did it. You don’t know whether that’s something it can do reliably until you test it sure.

An example of a possible superstitious action would be always adding commands as notes in a PR because you believe Claude gives PR notes more weight.

That’s something that sounds crazy, but it’s perfectly believable that some artifact of training could lead some model to actually behave this way. And you can imagine that someone picking up on this pattern could continue to favor writing commands as PR notes years after model changes have removed this behavior.

Re: Vibe engineering

#684

Post by simonw »

sarchertech wrote: post #683
simonw wrote: post #679

Earlier quoted context omitted.

Here's an example from today of something I just figured out. I had Claude Code do some work which I pushed as a branch to GitHub. Then I opened a PR so I could more easily review it and added a bunch of notes and comments there. On a hunch, I pasted the URL to that PR into Claude Code and said "use the GitHub API to fetch the notes on this PR"... ... and it did exactly that. It guesses the API URL, fetched the JSON…

How is that an example of something you are doing that might be a superstition? You asked it to do a single easily verifiable task and it did it. You don’t know whether that’s something it can do reliably until you test it sure. An example of a possible superstitious action would be always adding commands as notes in a PR because you believe Claude gives PR notes more weight. That’s something that sounds crazy, but i…

When I'm working with models I'm always looking for the simplest possible way to express a task. I've never been a fan of the whole "you're a world expert in X", "I'll tip you a million dollars if..." etc school of prompting.

I wrote up another real world example of how I use Claude Code this afternoon: https://simonwillison.net/2025/Oct/8/claude-datasette-plugin...

Re: Vibe engineering

#685

Post by sarchertech »

simonw wrote: post #684
sarchertech wrote: post #683

Earlier quoted context omitted.

How is that an example of something you are doing that might be a superstition? You asked it to do a single easily verifiable task and it did it. You don’t know whether that’s something it can do reliably until you test it sure. An example of a possible superstitious action would be always adding commands as notes in a PR because you believe Claude gives PR notes more weight. That’s something that sounds crazy, but i…

When I'm working with models I'm always looking for the simplest possible way to express a task. I've never been a fan of the whole "you're a world expert in X", "I'll tip you a million dollars if..." etc school of prompting. I wrote up another real world example of how I use Claude Code this afternoon: https://simonwillison.net/2025/Oct/8/claude-datasette-plugin...

Those are some obvious potential superstitious incantations. They might not be superstitions though. They might actually work. It’s entirely feasible that bribes produce higher quality code. Unfortunately it’s not as easy as avoiding things that sound ridiculous.

The black box, random, chaotic nature of LLMs virtually ensures that you will pick up superstitions even if they aren’t as obvious as the above. Numbered lists work better than bullets. Prompts work better if they are concise and you remove superfluous words. You should reset your context as soon as the agent starts doing x.

All of those things may be true. They may have been true for one model, but not others. They may have never been generally true for any model, but randomness led someone to believe they were.

Re: Vibe engineering

#686

Post by iFreilicht »

mejutoco wrote: post #384
tkgally wrote: post #244

Around the time GPT-4 was released in early 2023, a similar issue arose with another profession: translation. It was at that point that machine translation between languages like English and Japanese (the language pair I have worked with) started to approach human level for the first time. I took part in a lot of discussions then with other professional translators, and the reaction of many was similar to that of som…

Translation is great to discuss LLMs. Thanks for sharing your experience. On one side translation is not very valued by most people. It is rare that people know who translated a book, for example. It is a pity but people do not read much these days. Additionally, or maybe because of the above, translation is often paid in terms of number of lines or words, even before AI. A bit like software security, it is often sad…

> It is a pity but people do not read much these days.

This is a common belief, but it's just not true. The book industry is healthier than ever.

Re: Vibe engineering

#687

Post by tartieret »

joduplessis wrote: post #199

To people reading the article: replace the word "agent" with "intern". > Without tests? Your intern might claim something works without having actually tested it at all, plus any new change could break an unrelated feature without you realizing it. Test-first development is particularly effective with interns that can iterate in a loop. Vibe engineer? No, try technical manager.

As a CTO of a small company, spending a lot of time reviewing other developers' PRs, I completely agree. I recently enabled Github Copilot Agent and use it a lot for dealing with smaller and lower priority tasks in an async workflow (e.g. while I am doing other things). A lot of developers are not very good writers, and reviewing PRs from Copilot, with access to the full "thinking process" and being able to request changes in a few comments is sometimes more pleasant and effective

Re: Vibe engineering

#688

Post by tartieret »

ares623 wrote: post #646
sevenseacat wrote: post #321

Earlier quoted context omitted.

I realize this is likely being facetious, but just in case - code reviews are so much more than just 'check the syntax and style of the code'. They check the intention, check the actual functionality, find issues on the larger scale that LLMs literally can't. Yes, PRs start piling up because devs can vibe code them faster than they can be competently reviewed. This is a problem with the vibe code process, not the cod…

I was being half facetious, yes. But wouldn't the invoker of the LLM be already doing a review in that case? It just feels a bit redundant, to have engineer one do a code review of LLM's work, and then have engineer two do the same review.

The business cost with PRs isn't the first review but another developer, it's the number of iterations on a pull request due to defects and change requests. The way I am trying to promote the use of LLMs with more junior developers in my team (I am the CTO) is to use AI-assisted tools (we used Windsurf and recently switched to Github Copilot) for a first pass, e.g asking the agent for a review and catching potential defect before involving the human reviewer.

This doesn't mean the human reviewer will need to spend less time reviewing, but potentially this PR will be merged faster with on average a lower number of iterations and improved code quality.

I do have in my team some senior developers that are excellent, and it's very very rare I catch an issue in their PRs (maybe 1 out of 50). But I also have greener developers for who the ratio is way higher (like 8 or 9 out of 10) and this means repeated context switching for the reviewers.

Re: Vibe engineering

#689

Post by tartieret »

hollowonepl wrote: post #189

I’m glad to see that more and more articles and opinions about AI are focused on how it works and it works great just the process and mind thoughtfulness has to adapt and evolve to fully utilize the tool. That’s so much better than wasting time on the frustrated who just negate without a meaningful try. I share most of the experience and learning with the author, I just still don’t know how to name the whole process.…

I agree with your analogy with pair programming. Except that now your co-developer types way faster!

Re: Vibe engineering

#690

Post by simonw »

sarchertech wrote: post #685
simonw wrote: post #684

Earlier quoted context omitted.

When I'm working with models I'm always looking for the simplest possible way to express a task. I've never been a fan of the whole "you're a world expert in X", "I'll tip you a million dollars if..." etc school of prompting. I wrote up another real world example of how I use Claude Code this afternoon: https://simonwillison.net/2025/Oct/8/claude-datasette-plugin...

Those are some obvious potential superstitious incantations. They might not be superstitions though. They might actually work. It’s entirely feasible that bribes produce higher quality code. Unfortunately it’s not as easy as avoiding things that sound ridiculous. The black box, random, chaotic nature of LLMs virtually ensures that you will pick up superstitions even if they aren’t as obvious as the above. Numbered li…

I just realized I picked up a new superstition quite recently involving ChatGPT search.

I've been asking it for "credible" reports on topics, because when I use that word its thinking trace seems to consider the source of the information more carefully. I've noticed it saying things like "but that's just a random blog, I should find a story from a news organization".

But... I haven't done a measured comparison, so for all I know it has the same taste in sources even if I don't nudge it with "credible" in the mix!

Post reply on HN