“Vibe Coding” vs. Reality

Remember that 80% of your time and resources is going to be spent finishing up the last 20% of the project. If the first 80% is borked by LLM code salad, you’re going to need to spend time fixing that code and making it actually work. That might take just as much time, if not more, than only using AI as an assistant (i.e. code completion) instead of the main source of code.

.. new stuff that quite obviously won't be able to live up to its hype. Indeed.

Or would you argue that NFTs actually did live up to the BS that was ascribed to them in some circles during their hype?

Obviously not.

But "some circles" ascribe BS to any new technology.

I think the bigger "AI hype vs. Reality" gap is about the productivity numbers people casually throw around, like "10x as productive" or even 100x.

For example, here are YC partners quoting a company in a batch claiming "100x speedup" in coding performance compared to the previous month:

https://www.youtube.com/watch?v=IACHfKmZMr8&t=1837s

You can tell this claim is false, because that level of productivity increase would be glaringly obvious to an outside observer; it wouldn't need to be self-reported.

A YC summer batch is 84 days culminating in Demo Day. So a 100x speed improvement would be like a team spending less than 1 day of coding and ending up with something that's on par with Demo Day in terms of functionality. Maybe the design would be wrong, but that wrong design would be just as fully-featured as a Demo Day app.

So if 100x were true, the partners in that video would be talking about how the new batch dynamic is "They get breakfast with a customer, learn something new, have an epiphany, and then later the same day they have their entire app rewritten based on what they learned, and that scratch-rewrite is already at a Demo Day level of functionality." The partners aren't talking about that dynamic because it's not happening. So clearly the self-reported 100x is inaccurate.

Even 10x would result in partners saying "Whoa, in this batch people have a Demo Day-quality app in production by the end of week 1 instead of week 12." The partners have a huge sample size on how much teams get done in what time period, so it would be glaringly obvious to them if this batch were shipping 10x as fast as previous batches.

That external observation would be the headline if it were what the partners were actually seeing. Since that's not the headline, it's clearly not what they're seeing, so 10x can't be the number either.

The only defense of vibe coding I'll make is that LLMs are very good at identifying decent implementations of business logic, such as workflows I may not have otherwise considered or found on StackOverflow. That then becomes a decent starting point for future iteration, but would never trust the "vibe" of the code itself even if that's what all the AI hypesters are doing.

Despite developing LLMs for years I haven't actually used them much in day-to-day work, but asking Claude 3.7 Sonnet my coding questions has been a superior experience to just Googling them (particularly if there are specific functional requirements/constraints)