Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vibe engineering

simonwillison.net

Post reply on HN
Print view View original

671–680 of 759 posts

Re: Vibe engineering

#671

Post by pron »

herval wrote: post #653
pron wrote: post #636

Earlier quoted context omitted.

So the role of a coding agent is to challenge me to play in hard mode? And suppose getting developers to not lie or hide important information is on me, what should I do to get an LLM to not do that?

no, the point is LLMs will behave the same way humans you have to manage do (there's obviously differences - eg LLMs tend to forget context more often than most humans, but also they tend to know a lot more than the average human). So some of the same skills that'll help you manage humans will also help you get more consistency out of LLMs.

I don't know of anyone who would like to work with someone who lies to them over and over, and will never stop. LLMs do certain things better than people, but my point is that there's nothing you can trust them to do. That's fine for research (we don't trust, and don't need to trust, any human or tool to do a fully exhaustive research, anyway), but not for most other work tasks. That's not to say that LLMs can't be utilised usefully, but something that can never be trusted behaves like neither person nor tool.

Re: Vibe engineering

#672

Post by hollowturtle »

simonw wrote: post #582
hollowturtle wrote: post #576

Earlier quoted context omitted.

Another void argument, we're speaking about tools, dogs are not tools

Yes they are. Guide dogs, hunting dogs, sheep dogs. The comparison to LLMs is genuinely useful here, because dogs are unreliable tools that you have to work with over a period of time to figure out. I've used this argument for real in the past with people who complain that it's unethical to set sightless people up with vision LLM tools because those tools are unreliable and make mistakes. My counter is that a) so are…

No they're not, they're animals as we are, otherwise one would claim that you simon are a tool too. I'm finally starting to understand your world beliefs from post and comments, it's aberrating and dystopian

Re: Vibe engineering

#673

Post by hollowturtle »

simonw wrote: post #614
azemetre wrote: post #606

Earlier quoted context omitted.

Comparing dogs, things that experience sentience, to software feels deeply dystopian and antihuman.

I'm fine with it. I love dogs, and I find suggestions that LLMs may achieve sentience or become conscious either laughable or abhorrent, depending on how serious the person is who's making them. It's still OK to use dogs as an analogy. In this case the analogy is to unreliable tools, and dogs are unreliable tools. I don't find "stochastic parrot" offensive as an analogy, even though it's got parrots in it.

btw if you have emotional bonds with LLMs like you do with your dog "tools" there isn't much to argue over anymore

Re: Vibe engineering

#674

Post by hollowturtle »

holbrad wrote: post #666
hollowturtle wrote: post #576

Earlier quoted context omitted.

Another void argument, we're speaking about tools, dogs are not tools

You can 100% use dogs as tools as we've done for thousands of years.

They're not, they re partners/coworkers with emotional bonds and a reciprocal relationship. Reducing them to tools is so Bay Area level of BS.

Re: Vibe engineering

#675

Post by schwartzworld »

mlsu wrote: post #624
schwartzworld wrote: post #494

Earlier quoted context omitted.

> incredibly impressive and completely unreliable. There have been methods of protecting against this since before AI, and they still apply. LLMs work great with test driven development, for example. I would say that high-level knowledge and good engineering practices more important than ever, but they were always important.

If you're writing your own tests, sure, AI is fast at writing code that passes the tests. But if you write a comprehensive test suite for a problem, you've effectively done the hard development work to solve the problem in the first place. How did the AI help? Oh have the AI write unit tests you say? Claude cheats constantly at the tests ime. It frequently tests the mock instead of the UUT and reports a pass. That's…

> But if you write a comprehensive test suite for a problem, you've effectively done the hard development work to solve the problem in the first place. How did the AI help?

By making you not write the implementation?

Also, the AI writing anything bad isn’t an excuse. You’re the one piloting that ship, and if not, you’re probably the one reviewing the code. It’s your job to review your own and others’ code with a critical eye, and that goes double in the LLM age.

Re: Vibe engineering

#676

Post by ewoodrich »

JamesSwift wrote: post #469
cadamsdotcom wrote: post #120

A better term is agentic coding, agentic software engineering, etc. rather than being vibe based. My process starts from a Claude Code plan, whose first step is to write a spec. I use TDD, and enforce my "unspoken rules of code quality" using a slew of generated tools. One tiny tool blocks code which violates our design system. Another tool blocks code which violates our separation of layering - this forces the HTTP…

I spent some time trying to think of a better term because I also think "vibe" detracts from the intent, and I think you nailed it with "agentic coding". Ill do my part by using that term now, hopefully it catches on : D

I was writing up some recommendations to my team for using Github Copilot in its different modes + Roo Code/Cline and tried so hard to avoid using “vibe coding” in a professional context. For sake of clarity I ended up including it once in a “you may have heard the term…” style reference but it just comes across as so deeply unserious and felt ridiculous being forced to use it at all.

I also landed on “agentic coding” FWIW. It’s a bit clunky but didn’t feel like an embarrassment to the entire software industry typing it so I’ll happily use it over the alternative.

Re: Vibe engineering

#677

Post by selcuka »

anthonypasq wrote: post #623
selcuka wrote: post #153

Earlier quoted context omitted.

Don't worry, it's probably only the impostor syndrome. Your development skills are still relevant. Think of agents as junior developers that assist you in coding tasks, whom you constantly need to mentor, review, and correct.

You think theyre still gonna be juniors 5 years from now? A couple years ago they could barely even write a function

No, I don't think they will always be junior developers. Obviously there will be a day that they will surpass humans.

However, the progress doesn't look linear with the current technology, and I don't expect to see the same big jump in the next 5 years as we've seen in the last 5 unless we discover a disruptive, new technology.

This can also be observed by comparing models with ~3B, ~30B, and ~300B parameters. You can see a huge performance boost when going from 3B to 30B, but we don't see the same when going to 300B. Simply adding 10x more RAM and GPU power brings diminishing returns.

Re: Vibe engineering

#678

Post by simonw »

hollowturtle wrote: post #673
simonw wrote: post #614

Earlier quoted context omitted.

I'm fine with it. I love dogs, and I find suggestions that LLMs may achieve sentience or become conscious either laughable or abhorrent, depending on how serious the person is who's making them. It's still OK to use dogs as an analogy. In this case the analogy is to unreliable tools, and dogs are unreliable tools. I don't find "stochastic parrot" offensive as an analogy, even though it's got parrots in it.

btw if you have emotional bonds with LLMs like you do with your dog "tools" there isn't much to argue over anymore

I don't think you're very good with analogies.

Re: Vibe engineering

#679

Post by simonw »

sarchertech wrote: post #621
simonw wrote: post #537

Earlier quoted context omitted.

I'm sure a lot of them are superstitions! I've written about that before: https://simonwillison.net/2023/Aug/27/wordcamp-llms/#superst... One of the more "engineering" like skills in using this stuff is methodically figuring out what's a superstition and what actually works.

Nice to see that you recognize that! > One of the more "engineering" like skills in using this stuff is methodically figuring out what's a superstition and what actually works. The problem is there are so many variables and the system is so chaotic that this is a nearly impossible task for things that don’t have an absolutely enormous effect size. For most things you’re testing, you need to run the experiment many ma…

Here's an example from today of something I just figured out.

I had Claude Code do some work which I pushed as a branch to GitHub. Then I opened a PR so I could more easily review it and added a bunch of notes and comments there.

On a hunch, I pasted the URL to that PR into Claude Code and said "use the GitHub API to fetch the notes on this PR"...

... and it did exactly that. It guesses the API URL, fetched the JSON and read my notes back to me.

I told it to address each note in turn and commit the result. It did.

If a future model changes such that it can no longer correctly guess the URL to fetch JSON notes for a GitHub PR I'll notice when this trick fails. For the moment it's something I get to tuck in my ever expanding list of things that Claude (and likely other good models) can do.

Re: Vibe engineering

#680

Post by qazxcvbnmlp »

baby wrote: post #622
tkgally wrote: post #244

Around the time GPT-4 was released in early 2023, a similar issue arose with another profession: translation. It was at that point that machine translation between languages like English and Japanese (the language pair I have worked with) started to approach human level for the first time. I took part in a lot of discussions then with other professional translators, and the reaction of many was similar to that of som…

I spoke with some professional translators early on and they were just in denial, getting even upset at the idea that an AI could replace them. I didn't push too much but felt bad for them, as they couldn't realize what was going to happen to their field. I really think that translation must be the most impacted field by AI.

Software engineers are the translators. We (as a metaphorical community are in denial). Read the 100s of comments on this post: either (AI code is wrong or this isn’t correct or it’s not effective or some other justification). At the end of the day it’s really hard to accept the change.
Post reply on HN