Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The current state of LLM-driven development

blog.tolki.dev

Post reply on HN
Print view View original

111–120 of 244 posts

Re: The current state of LLM-driven development

#111

Post by mark_l_watson »

simonw wrote: post #99
donperignon wrote: post #68

LLM’s are basically glorified slot machines. Some people try very hard to come up with techniques or theories about when the slot machine is hot, it’s only an illusion, let me tell you, it’s random and arbitrary, maybe today is your lucky day maybe not. Same with AI, learning the “skill” is as difficult as learning how to google or how to check stackoverflow, trivial. All the rest is luck and how many coins do you ha…

Learning how to Google is not trivial.

So true! About ten years ago Peter Norvig recommended the short Google online course on how to use Google Search: amazing how much one hour of structured learning permanently improved my search skills.

I have used neural networks since the 1980s, and modern LLM tech simply makes me happy, but there are strong limits to what I will use the current tech for.

Re: The current state of LLM-driven development

#112

Post by Terretta »

throwdbaaway wrote: post #43
Terretta wrote: post #3

Earlier quoted context omitted.

Which part of the opinion? I tend to strongly agree with the "unpopular opinion" about the IDEs mentioned versus CLI (specifically, aider.chat and Claude Code). Assuming (this is key) you have mastery of the language and framework you're using, working with the CLI tool in 25 year old XP practices is an incredible accelerant. Caveats: - You absolutely must bring taste and critical thinking, as the LLM has neither. -…

> use repomap tools on dependencies if creating new code that leverages those dependencies, and have that in context for that work. It seems to me that currently there are 2 schools of thought: 1. Use repomap and/or LSP to help the models navigate the code base 2. Let the models figure things out with grep Personally, I am 100% a grep guy, and my editor doesn't even have LSP enabled. So, it is very interesting to see…

I agree with the 2 schools, but different conclusion:

The strategy of one or the other brings differing big gaps and require context or prompt work to compensation.

They should be using 1 to keep overall lay of the land, and 2 before writing any code.

Re: The current state of LLM-driven development

#113

Post by mark_l_watson »

fnordsensei wrote: post #89

> By being particularly bad at anything outside of the most popular languages and frameworks, LLMs force you to pick a very mainstream stack if you want to be efficient. Do they? I’ve found Clojure-MCP[1] to be very useful. OTOH, I’m not attempting to replace myself, only augment myself. 1: https://github.com/bhauman/clojure-mcp

Thanks for the link! I used to use Clojure a lot professionally, but now just for fun projects, and to occasionally update my old Clojure book. I had bookmarked Clojure-MCP a while ago, but never got back to it but I will give it a try.

I like your phrasing of “OTOH, I’m not attempting to replace myself, only augment myself.” because that is my personal philosophy also.

Re: The current state of LLM-driven development

#114

Post by bootsmann »

simonw wrote: post #109
bootsmann wrote: post #102

Earlier quoted context omitted.

Simon, I have mad respect for your work but I think on this your view might be skewed because your day to day work involves a codebase where a single developer can still hold the whole context in their head. I would argue that the inadequacies of LLMs become more evident the more you have to make changes to systems that evolve at the speed of 15+ concurrent developers.

One of the things I'm using LLMs for a lot right now is quickly generating answers about larger codebases I'm completely unfamiliar with. Anything up to 250,000 tokens I pipe into GPT-5 (prior to that o3), and beyond that I'll send them to Gemini 2.5 Pro. For even larger code than that I'll fire up Codex CLI or Claude Code and let them grep their way to an answer. This stuff has gotten good enough now that I no longe…

Don't you see how this opens up a blindspot in your view of the code?

You don't have the luxury of having someone who is deeply familiar with the code sanity check your perceived understanding of the code, i.e. you don't see where the LLM is horribly off-track because you don't have sufficient understanding of that code to see the error. In enterprise contexts this is very common tho so its quite likely that a lot of the haters here have seen PRs submitted by vibecoders to their own work which have been inadequate enough that they started to blame the tool. For example I have seen someone reinvent the wheel of the session handling by a client library because they were unaware that the existing session came batteries included and the LLM didn't hesitate to write the code again for them. The code worked, everything checked out but because the developer didn't know what they didn't know about they submitted a janky mess.

Re: The current state of LLM-driven development

#115

Post by simonw »

bootsmann wrote: post #114
simonw wrote: post #109

Earlier quoted context omitted.

One of the things I'm using LLMs for a lot right now is quickly generating answers about larger codebases I'm completely unfamiliar with. Anything up to 250,000 tokens I pipe into GPT-5 (prior to that o3), and beyond that I'll send them to Gemini 2.5 Pro. For even larger code than that I'll fire up Codex CLI or Claude Code and let them grep their way to an answer. This stuff has gotten good enough now that I no longe…

Don't you see how this opens up a blindspot in your view of the code? You don't have the luxury of having someone who is deeply familiar with the code sanity check your perceived understanding of the code, i.e. you don't see where the LLM is horribly off-track because you don't have sufficient understanding of that code to see the error. In enterprise contexts this is very common tho so its quite likely that a lot of…

The LLMs go off track all the time. I spot that when I try putting what I've learned from them into action.

Re: The current state of LLM-driven development

#116

Post by 0points »

simonw wrote: post #103
hitarpetar wrote: post #93

Earlier quoted context omitted.

That's not a problem, that is the argument. People are bad at measuring their own productivity. Just because you feel more productive with an LLM does not mean you are. We need more studies and less anecdata

I'm afraid all you're going to get from me is anecdata, but I find a lot of it very compelling. I talk to extremely experienced programmers whose opinions I have valued for many years before the current LLM boom who are now flying with LLMs - I trust their aggregate judgement. Meanwhile my own https://tools.simonwillison.net/colophon collection has grown to over 120 in just a year and a half, most of which I wouldn't…

> my own https://tools.simonwillison.net/colophon collection has grown to over 120

What in the wooberjabbery is this even.

List of single-commit LLM generated stuff. Vibe coded shovelware like animated-rainbow-border [1] or unix-timestamp [2].

Calling these tools seems to be overstating it.

1: https://gist.github.com/simonw/2e56ee84e7321592f79ceaed2e81b...

2: https://gist.github.com/simonw/8c04788c5e4db11f6324ef5962127...

Re: The current state of LLM-driven development

#117

Post by Palmik »

mikeshi42 wrote: post #91
donperignon wrote: post #68

LLM’s are basically glorified slot machines. Some people try very hard to come up with techniques or theories about when the slot machine is hot, it’s only an illusion, let me tell you, it’s random and arbitrary, maybe today is your lucky day maybe not. Same with AI, learning the “skill” is as difficult as learning how to google or how to check stackoverflow, trivial. All the rest is luck and how many coins do you ha…

There's plenty of evidence that good prompts (prompt engineering, tuning) can result in better outputs. Improving LLM output through better inputs is neither an illusion, nor as easy as learning how to google (entire companies are being built around improving llm outputs and measuring that improvement)

Sure, but tricks & techniques that work with one model often don't translate or are actively harmful with others. Especially when you compare models from today and 6 or more months ago.

Keep in mind that the first reasoning model (o1) was released less than 8 months ago and Claude Code was released less than 6 months ago.

Re: The current state of LLM-driven development

#118

Post by Tainnor »

tptacek wrote: post #50
simonw wrote: post #48

Earlier quoted context omitted.

The problem with your argument here is that you're effectively saying that developers (like myself) who put effort into figuring out good workflows for coding with LLMs are deceiving themselves, and are effectively wasting their time. Either I've wasted significant chunks of the past ~3 years of my life or you're missing something here. Up to you to decide which you believe. I agree that it's hard to take solid measu…

Another problem with it is that you could have said the same thing about virtually any advancement in programming over the last 30 years.

There have been so many "advances" in software development in the last decades - powerful type systems, null safety, sane error handling, Erlang-style fault tolerance, property testing, model checking, etc. - and yet people continue to write garbage code in unsafe languages with underpowered IDEs.

I think many in the industry have absolutely no clue what they're doing and are bad at evaluating productivity, often prioritising short term delivery over longterm maintenance.

LLMs can absolutely be useful but I'm very concerned that some people just use them to churn out code instead of thinking more carefully about what and how to build things. I wish we had at least the same amount of discussions about those things I mentioned above as we have about whether Opus, Sonnet, GPT5 or Gemini is the best model.

Re: The current state of LLM-driven development

#119

Post by Palmik »

simonw wrote: post #100
Palmik wrote: post #81

People that comment on and get defensive about this bit: > Learning how to use LLMs in a coding workflow is trivial. There is no learning curve. You can safely ignore them if they don’t fit your workflows at the moment. How much of your workflow or intuition from 6 months ago is still relevant today? How long would it take to learn the relevant bits today? Keep in mind that Claude Code was released less than 6 months…

Pretty much all of the intuition I've picked up about getting good results from LLMs has stayed relevant. If I was starting from fresh today I expect it would take me months of experimentation to get back to where I am now. Working thoughtfully with LLMs has also helped me avoid a lot of the junk tips ("Always start with 'you are the greatest world expert in X', offer to tip it, ...") that are floating around out the…

All of the intuition? Definitely not my experience. I have found that optimal prompting differs significantly between models, especially when you look at models that are 6months old or older (the first reasoning model, o1, is less than 8 months old).

Speaking mostly from experience of building automated, dynamic data processing workflows that utilize LLMs:

Things that work with one model, might hurt performance or be useless with another.

Many tricks that used to be necessary in the past are no longer relevant, or only applicable for weaker models.

This isn't me dimissing anyone's experience. It's ok to do things that become obsolete fairly quickly, especially if you derive some value from it. If you try to stay on top of a fast moving field, it's almost inevitable. I would not consider it a waste of time.

Re: The current state of LLM-driven development

#120

Post by [deleted] »

donperignon wrote: post #68

LLM’s are basically glorified slot machines. Some people try very hard to come up with techniques or theories about when the slot machine is hot, it’s only an illusion, let me tell you, it’s random and arbitrary, maybe today is your lucky day maybe not. Same with AI, learning the “skill” is as difficult as learning how to google or how to check stackoverflow, trivial. All the rest is luck and how many coins do you ha…

[deleted]
Post reply on HN