Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The current state of LLM-driven development

blog.tolki.dev

Post reply on HN
Print view View original

141–150 of 244 posts

Re: The current state of LLM-driven development

#141

Post by simonw »

donperignon wrote: post #132
simonw wrote: post #97

Earlier quoted context omitted.

As of about three months ago, one of the most important skills in effective LLM coding is coding agent environment design. If you want to use a tool like Claude Code (or Gemini CLI or Cursor agent mode or Code CLI or Qwen Code) to solve complex problems you need to give them an environment they can operate in where they can solve that problem without causing too much damage if something goes wrong. You need to think…

Yeah if I want I to develop I need tooling around me. Still trivial to learn. Not a difficult skill. Not an specific skill to llm.

Why would you need to take all of these additional sandboxing measures if you weren't using an LLM?

Re: The current state of LLM-driven development

#142

Post by simonw »

bgwalter wrote: post #135
simonw wrote: post #127

Earlier quoted context omitted.

Cool right? It's my playground for vibe coded apps, except I started it nearly a year before the term "vibe coding" was introduced. I wrote more about it here: https://simonwillison.net/2024/Oct/21/claude-artifacts/ - and a lot of them have explanations in posts under my tools tag: https://simonwillison.net/tags/tools/ It might also be the largest collection of published chat transcripts for this kind of usage from a…

[flagged]

I've been using LLM-assistance for my larger open source projects - https://github.com/simonw/datasette https://github.com/simonw/llm and https://github.com/simonw/sqlite-utils - for a couple of years now.

Also literally hundreds of smaller plugins and libraries and CLI tools, see https://github.com/simonw?tab=repositories (now at 880 repos, though a few dozen of those are scrapers and shouldn't count) and https://pypi.org/user/simonw/ (340 published packages).

Unlike my tools.simonwillison.net stuff the vast majority of those products are covered by automated tests and usually have comprehensive documentation too.

What do you mean by my script?

Re: The current state of LLM-driven development

#143

Post by kketch »

This article makes me wanna try building a token field in Flutter using a LLM chat or agent. Chat should be enough. A few iterations to get the behaviour and the tests right. A bit of style to make it look Apple-nice. As if a regular dev would do much better/quicker for this use case, such a bad example imo I don't buy it

Re: The current state of LLM-driven development

#144

Post by ModernMech »

SkyPuncher wrote: post #60
majormajor wrote: post #45

Earlier quoted context omitted.

> That's a wild statement. I'm now extremely productive with LLMs in my core codebases, but it took a lot of practice to get it right and repeatable. There's a lot of little contextual details you need to learn how to control so the LLM makes the right choices. > Whenever I start working in a new code base, it takes a a non-trivial amount of time to ramp back up to full LLM productivity. Do you find that these detail…

Yes and no. The broader business context translates well, but each model has it's own blindspots and hyperfocuses that you need to massage out. * Business context - these are things like code quality/robustness, expected spec coverage, expected performance needs, domain specific knowledge. These generally translate well between models, but can vary between code bases. For example, a core monolith is going to have hig…

This is like UB in compilers but 100x worse, because there's no spec, it's not even documented, and it could change without a compiler update.

Re: The current state of LLM-driven development

#145

Post by mikeshi42 »

Palmik wrote: post #117
mikeshi42 wrote: post #91

Earlier quoted context omitted.

There's plenty of evidence that good prompts (prompt engineering, tuning) can result in better outputs. Improving LLM output through better inputs is neither an illusion, nor as easy as learning how to google (entire companies are being built around improving llm outputs and measuring that improvement)

Sure, but tricks & techniques that work with one model often don't translate or are actively harmful with others. Especially when you compare models from today and 6 or more months ago. Keep in mind that the first reasoning model (o1) was released less than 8 months ago and Claude Code was released less than 6 months ago.

Yes, though that just means the probability of success is a function of not only user input but also the model version.

Slot machines on the other hand are truly random and success is luck based with no priors (the legal ones in the US anyways)

Re: The current state of LLM-driven development

#146

Post by donperignon »

simonw wrote: post #141
donperignon wrote: post #132

Earlier quoted context omitted.

Yeah if I want I to develop I need tooling around me. Still trivial to learn. Not a difficult skill. Not an specific skill to llm.

Why would you need to take all of these additional sandboxing measures if you weren't using an LLM?

I usually work with containers for repeatability and portability. Also makes the local env closer to the final prod env.

Re: The current state of LLM-driven development

#147

Post by skydhash »

FeepingCreature wrote: post #137
bootsmann wrote: post #102

Earlier quoted context omitted.

Simon, I have mad respect for your work but I think on this your view might be skewed because your day to day work involves a codebase where a single developer can still hold the whole context in their head. I would argue that the inadequacies of LLMs become more evident the more you have to make changes to systems that evolve at the speed of 15+ concurrent developers.

This just sounds 1:1 equivalent to "there are things LLMs are good for and things LLMs are bad for."

I'll bite.

What are those things that they are good for? And consistently so?

Re: The current state of LLM-driven development

#148

Post by SkyPuncher »

deadbabe wrote: post #106
SkyPuncher wrote: post #44

Earlier quoted context omitted.

> Learning how to use LLMs in a coding workflow is trivial. There is no learning curve. You can safely ignore them if they don’t fit your workflows at the moment. That's a wild statement. I'm now extremely productive with LLMs in my core codebases, but it took a lot of practice to get it right and repeatable. There's a lot of little contextual details you need to learn how to control so the LLM makes the right choice…

He’s not wrong. Getting 80% of the benefit of LLMs is trivial. You can ask it for some functions or to write a suite of unit tests and you’re done. The last 20%, while possible to attain, is ultimately not worth it for the amount of time you spend in context hells. You can just do it yourself faster.

> The last 20%, while possible to attain, is ultimately not worth it for the amount of time you spend in context hells. You can just do it yourself faster.

I'm arguing that there's a skill that has to be learned in order to break through this. As you start in a new code base, you should be quick to jump in when you hit that 20%. But, as you spend more time in it, you learn how to avoid the same "context hell" issues and move that number down to 15%, 10%, 5% of the time.

You're still going to need to jump in, but when you can learn to get the LLM to write 95% of the code for you, that's incredibly powerful.

Re: The current state of LLM-driven development

#149

Post by candiddevmike »

throwawaybob420 wrote: post #140

Judging from all the comments here, it’s going to be amazing seeing the fallout of all the LLM generated code in a year or so. The amount of people who seemingly relish the ability to stop thinking and let the model generate giant chunks of their code base, is uh, something else lol.

Software "engineering" at it's finest

Re: The current state of LLM-driven development

#150

Post by d_silin »

Relying on LLM for any skill, especially programming, is like cutting your own healthy legs and buying crutches to walk. Plus you now have to pay $49/month for basic walking ability and $99/month for "Walk+" plan, where you can also (clumsily) jog.
Post reply on HN