Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The current state of LLM-driven development

blog.tolki.dev

Post reply on HN
Print view View original

121–130 of 244 posts

Re: The current state of LLM-driven development

#121

Post by JimDabell »

hiAndrewQuinn wrote: post #73

>I made a CLI logs viewers and querier for my job, which is very useful but would have taken me a few days to write (~3k LoC) I recall The Mythical Man-Month stating a rough calculation that the average software developer writes about 10 net lines of new, production-ready code per day. For a tool like this going up an order of magnitude to about 100 lines of pretty good internal tooling seems reasonable. OP sounds a…

A lot of the Mythical Man-Month is timeless, but for a stat like that, it really is worth bearing in mind the book was written half a century ago about developers working on 1970s mainframes.

Re: The current state of LLM-driven development

#122

Post by CuriouslyC »

simonw wrote: post #48
troupo wrote: post #23

Earlier quoted context omitted.

> I have never heard anybody successfully using LLMs say this before. Most of what I've learned from talking to people about their workflows is counterintuitive and subtle. Because for all our posturing about being skeptical and data driven we all believe in magic. Those "counterintuitive non-trivial workflows"? They work about as well as just prompting "implement X" with no rules, agents.md, careful lists etc. Becau…

The problem with your argument here is that you're effectively saying that developers (like myself) who put effort into figuring out good workflows for coding with LLMs are deceiving themselves, and are effectively wasting their time. Either I've wasted significant chunks of the past ~3 years of my life or you're missing something here. Up to you to decide which you believe. I agree that it's hard to take solid measu…

So far I've found that the people who are hating on AI are stuck maintaining highly coupled that they've invested a significant amount of mental energy internalizing. AI is bad on that type of code, and since they've invested so much energy on understanding the code, it ends up taking longer for them to load context and guide the AI than to just do the work. Their code base is hot coupled garbage, and rather than accept that the tools aren't working because of their own lack of architectural rigor, they just shit on the tools. This is part of the reason that that study of open source maintainers using Cursor didn't consistently produce improvement (also, Cursor is pretty mid).

https://www.youtube.com/watch?v=tbDDYKRFjhk&t=4s is one of the largest studies I've seen so far and it shows that when the codebase is small or engineered for AI use, >20% productivity improvements are normal.

Re: The current state of LLM-driven development

#124

Post by donperignon »

simonw wrote: post #14

Learning how to use LLMs in a coding workflow is trivial. There is no learning curve. [...] LLMs will always suck at writing code that has not be written millions of times before. As soon as you venture slightly offroad, they falter. That right there is your learning curve! Getting LLMs to write code that's not heavily represented in their training data takes experience and skill and isn't obvious to learn.

I’m still waiting that someone claiming how prompting is such an skill to learn, explain just once a single technique that is not obvious, like: storing checkpoint to go back to working version (already a good practice without using Llm see:git) or launch 10 tabs with slightly different prompts and choose the best, or ask the Llm to improve my prompt, or adding more context … is that an skill? I remember when I was a child that my mom thought that programming a vcr to record the night show to be such a feat…

Re: The current state of LLM-driven development

#125

Post by dwheeler »

> Learning how to use LLMs in a coding workflow is trivial. There is no learning curve. You can safely ignore them if they don’t fit your workflows at the moment.

I'm sprry, but I disagree with this claim. That is not my experience, nor many others. It's true that you can make them do something without learning anything. However, it takes time to learn what they are good amd bad at, what information they need, and what nonsense they'll do without express guidance. It also takes time to know what to look for when reviewing results.

I also find that they work fine for languages without static types. You need need tests, yes, but you need them anyway.

Re: The current state of LLM-driven development

#126

Post by pyb »

Palmik wrote: post #81

People that comment on and get defensive about this bit: > Learning how to use LLMs in a coding workflow is trivial. There is no learning curve. You can safely ignore them if they don’t fit your workflows at the moment. How much of your workflow or intuition from 6 months ago is still relevant today? How long would it take to learn the relevant bits today? Keep in mind that Claude Code was released less than 6 months…

A fraction of the LLM maximalists are being defensive, because they don't want to consider that they've maybe invested too much time in those tools ; considering what said tools are currently genuinely good at.

Re: The current state of LLM-driven development

#127

Post by simonw »

0points wrote: post #116
simonw wrote: post #103

Earlier quoted context omitted.

I'm afraid all you're going to get from me is anecdata, but I find a lot of it very compelling. I talk to extremely experienced programmers whose opinions I have valued for many years before the current LLM boom who are now flying with LLMs - I trust their aggregate judgement. Meanwhile my own https://tools.simonwillison.net/colophon collection has grown to over 120 in just a year and a half, most of which I wouldn't…

> my own https://tools.simonwillison.net/colophon collection has grown to over 120 What in the wooberjabbery is this even. List of single-commit LLM generated stuff. Vibe coded shovelware like animated-rainbow-border [1] or unix-timestamp [2]. Calling these tools seems to be overstating it. 1: https://gist.github.com/simonw/2e56ee84e7321592f79ceaed2e81b... 2: https://gist.github.com/simonw/8c04788c5e4db11f6324ef59621…

Cool right? It's my playground for vibe coded apps, except I started it nearly a year before the term "vibe coding" was introduced.

I wrote more about it here: https://simonwillison.net/2024/Oct/21/claude-artifacts/ - and a lot of them have explanations in posts under my tools tag: https://simonwillison.net/tags/tools/

It might also be the largest collection of published chat transcripts for this kind of usage from a single person - though that's not hard since most people don't publish their prompts.

Building little things like this is really effective way of gaining experience using prompts to get useful code results out of LLMs.

Re: The current state of LLM-driven development

#128

Post by ebiester »

deadbabe wrote: post #110
ebiester wrote: post #5

I disagree from almost the first sentence: > Learning how to use LLMs in a coding workflow is trivial. There is no learning curve. You can safely ignore them if they don’t fit your workflows at the moment. Learning how to use LLMs in a coding workflow is trivial to start, but you find you get a bad taste early if you don't learn how to adapt both your workflow and its workflow. It is easy to get a trivially good resu…

If it’s not trivial, it’s worthless, because writing things out manually yourself is usually trivial, but tedious. With LLMs, the point is to eliminate tedious work in a trivial way. If it’s tedious to get an LLM to do tedious work, you have not accomplished anything. If the work is not trivial enough for you to do yourself, then using an LLM will probably be a disaster, as you will not be able to judge the final out…

So, I'd like you to talk to a fair number of emacs and vim users. They have spent hours and hours learning their tools, tweaking their configurations, and learning efficiencies. They adapt their tool to them and themselves to the tool.

We are learning that this is not going to be magic. There are some cases where it shines. If I spend the time, I can put out prototypes that are magic and I can test with users in a fraction of the time. That doesn't mean I can use that for production.

I can try three or four things during a meeting where I am generally paying attention, and look afterwards to see if it's pursuing.

I can have it work through drudgery if I provide it an example. I can have it propose a solution to a problem that is escaping me, and I can use it as a conversational partner for the best rubber duck I've ever seen.

But I'm adapting myself to the tool and I'm adapting the tool to me through learning how to prompt and how to develop guardrails.

Outside of coding, I can write chicken scratch and provide an example of what I want, and have it write a proposal for a PRD. I can have it break down a task, generate a list of proposed tickets, and after I've went through them have it generate them in jira (or anything else with an API). But the more I invest into learning how to use the tool, the less I have to clean up after.

Maybe one day in the future it will be better. However, the time invested into the tool means that 40 bucks of investment (20 into cursor, 20 into gpt) can add 10-15% boost in productivity. Putting 200 into claude might get you another 10% and it can get you 75% in greenfield and prototyping work. I bet that agency work can be sped up as much as 40% for that 200 bucks investment into claude.

That's a pretty good ROI.

And maybe some workloads can do even better. I haven't seen it yet but some people are further ahead than me.

Re: The current state of LLM-driven development

#129

Post by simonw »

donperignon wrote: post #124
simonw wrote: post #14

Learning how to use LLMs in a coding workflow is trivial. There is no learning curve. [...] LLMs will always suck at writing code that has not be written millions of times before. As soon as you venture slightly offroad, they falter. That right there is your learning curve! Getting LLMs to write code that's not heavily represented in their training data takes experience and skill and isn't obvious to learn.

I’m still waiting that someone claiming how prompting is such an skill to learn, explain just once a single technique that is not obvious, like: storing checkpoint to go back to working version (already a good practice without using Llm see:git) or launch 10 tabs with slightly different prompts and choose the best, or ask the Llm to improve my prompt, or adding more context … is that an skill? I remember when I was a…

See my comment here about designing environments for coding agents to operate in: https://news.ycombinator.com/item?id=44854680

Effective LLM usage these days is about a lot more than just the prompts.

Re: The current state of LLM-driven development

#130

Post by streb-lo »

simonw wrote: post #97
antonvs wrote: post #63

Earlier quoted context omitted.

Define "not trivial". Obviously, experience helps, as with any tool. But it's hardly rocket science. It seems to me the biggest barrier is that the person driving the tool needs to be experienced enough to recognize and assist when it runs into issues. But that's little different from any sophisticated tool. It seems to me a lot of the criticism comes from placing completely unrealistic expectations on an LLM. "It's…

As of about three months ago, one of the most important skills in effective LLM coding is coding agent environment design. If you want to use a tool like Claude Code (or Gemini CLI or Cursor agent mode or Code CLI or Qwen Code) to solve complex problems you need to give them an environment they can operate in where they can solve that problem without causing too much damage if something goes wrong. You need to think…

The situation you’re outlining is trivial though.

Yea, there’s some grunt work involved but in terms of learned ability all of that is obvious to someone who knew only a little bit about LLMs.

Post reply on HN