This has been my experience as well.
What I use LLMs for
I don't use Claude Code, Codex or any other hyped-up, VC-approved, buzzword-compliant, productivity-theater SF tool for actual day-to-day coding. I just use ChatGPT outside of my editor in browser.
The only tasks that I feel LLMs right now can do somewhat-reliably without human supervision include:
1. Search: To search and compare different things; that is boring to do manually. As an example, I checked how M1 Pro compares to M4. ChatGPT crawls the web to compare both the processors on similar specs and provides a conclusion but I need to check and verify the claims it reached by briefly reading the sources.
2. Documentation: I have to do things on time so I offload reading documentations for syntax or how-to-do-it-with-this-library to ChatGPT.
3. Single task scripts: I let ChatGPT create me boring and one-time-use python scripts for trivial tasks.
I don't spend ChatGPT's limited but free GPT-5-(latest-model) tokens for such trivial tasks - I have a special keybinding on my mac that fires up ChatGPT in incognito for me.
But don't get me wrong, I find LLMs pretty useful for PoC-level stuff and I do like instructing my agents to make me a half-baked PoC that makes me feel like a revolutionary SF-founder for an afternoon before I delete the repo and go back to writing real code like a responsible adult. Because nothing says ‘disruptive genius’ like ctrl+z and moving on.
---
Why I find vibe coding in production not useful
Like sean goedecke, I feel that using LLMs in production is not useful and (in fact) anti-productive because it gives junior engineer's free pass to push code without actually learning/reading it (Humans often tend to chose the easiest path to accomplish given problem). And due to this, its a pain to review junior PR these days.
I have two reasons why LLMs are not good at production coding:
1. Context: There are enough discussions out on the internet that says LLMs are limited by context. Humans are better because they can handle and work with multiple contexts in mind - an LLMs knows how to solve the user problem, but the engineer knows what business problem this piece of code solves, in which context the code will run, with other process on same service and in which context of other services, what company code idioms to follow while writing code and what will make your pair-programmer think that you are smart.
2. Lack of deterministic output: I would not over-simply by saying its just a probabilistic system but you can feel it right? It would sometimes give correct, decent and sometimes wrong answers for the same prompt. Sometimes it would enlighten you with a very interesting insight/perspective and other times it would be as dumb as a toaster trying to file your taxes - consistently inconsistent (when you reached context rot), yet somehow still sold as “intelligent.”
---
A brief rant
My manager fear-mongers me nearly everyday saying LLMs are really good at coding, there are at-most 10 years left to earn as programmer and stuff. The fact he is hasn't touched coding in last 8 years.
He thinks his half-baked trivial ideas translated to code by an overpriced probabilistic system is a breakthrough for his company.
I am non-native english speaker and you would have probably sensed that by now. I often get tasks to write documentations, and every time my manager doesn't understand something, he takes it as a chance to break my confidence saying that you are bad at using LLM for language related tasks because my english is poor.
But (at-least right now) I strongly feel this is just a hype:
- CEOs want to get-rich-quick or they want to increase company profits by "ai-enabling" it and getting salary hikes.
- Companies are hyping this tech to recalibrate engineer salaries that has skyrocketed since covid.
- My manager wants to make me feel replaceable.