Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The current state of LLM-driven development

blog.tolki.dev

Post reply on HN
Print view View original

221–230 of 244 posts

Re: The current state of LLM-driven development

#221

Post by credit_guy »

blks wrote: post #214
credit_guy wrote: post #193

Earlier quoted context omitted.

Sure. But it happens that I have 20 years of experience, and I know quite well how to code. Everything the LLM does for me I can do myself. But the LLM does that 100 times faster than me. Most of the days nowadays I push thousands of lines of code. And it's not garbage code, the LLMs write quite high quality code. Of course, I still have to go through the code and make sure it all makes sense. So I am still the bottl…

> Most of the days nowadays I push thousands of lines of code Insane stuff. It’s clear you can’t review so much changes in a day, so you’re just flooding your code base with code that you barely read. Or is your job just re-doing the same boilerplate over and over again?

You are a bit quick to jump to conclusions. With LLMs, test driven development becomes both a necessity and a pleasure. The actual functional code I push in a day is probably in the low hundreds LOC’s. But I push a lot of tests too. And sure, lots of that is boilerplate. But the tests run, pass, and if anything have better coverage than when I was writing all the code myself.

Re: The current state of LLM-driven development

#222

Post by antonvs »

skydhash wrote: post #84
antonvs wrote: post #63

Earlier quoted context omitted.

Define "not trivial". Obviously, experience helps, as with any tool. But it's hardly rocket science. It seems to me the biggest barrier is that the person driving the tool needs to be experienced enough to recognize and assist when it runs into issues. But that's little different from any sophisticated tool. It seems to me a lot of the criticism comes from placing completely unrealistic expectations on an LLM. "It's…

I don’t think anyone expects perfection. Programs crash, drives die, and computers can break anytime. But we expect our tools to be reliable and not fight with it everyday to get it to work. I don’t have to debug Emacs every day to write code. My CI workflow just runs every time a PR is created. When I type ‘make tests’, I get a report back. None of those things are perfect, but they are reliable.

If you work in a team, you work with other people, whose reliability is more akin to LLMs than to the deterministic processes you're describing.

What you're describing is a case of mismatched expectations.

Re: The current state of LLM-driven development

#223

Post by donperignon »

simonw wrote: post #99
donperignon wrote: post #68

LLM’s are basically glorified slot machines. Some people try very hard to come up with techniques or theories about when the slot machine is hot, it’s only an illusion, let me tell you, it’s random and arbitrary, maybe today is your lucky day maybe not. Same with AI, learning the “skill” is as difficult as learning how to google or how to check stackoverflow, trivial. All the rest is luck and how many coins do you ha…

Learning how to Google is not trivial.

Do you have an entry in your CV saying: proficiency in googling? It difficult not because it is complex, it difficult because Google want it to be opaque and as harder as possible to figure out.

Re: The current state of LLM-driven development

#224

Post by tptacek »

ModernMech wrote: post #204
tptacek wrote: post #188

Earlier quoted context omitted.

It is nothing at all like UB in a compiler. UB creates invisible bugs that tend to be discovered only after things have shipped. This is code generation. You can just read the code to see what it does, which is what most professionals using LLMs do.

With the volume of code people are generating, no you really can't just read it all. pg recently posted [1] that someone he knows is generating 10kloc/day now. There's no way people are using AI to generate that volume of code and reading it. How many invisible bugs are lurking in that code base, waiting to be found some time in the future after the code has shipped? [1] https://x.com/paulg/status/1953289830982664236

I read every line I generate and usually adjust things; I'm uncomfortable merging a PR I haven't put my fingerprints on somehow. From the conversations I have with other practitioners, I think this is pretty normal. So, no, I reject your premise.

Re: The current state of LLM-driven development

#225

Post by skajzbxbbj »

tptacek wrote: post #224
ModernMech wrote: post #204

Earlier quoted context omitted.

With the volume of code people are generating, no you really can't just read it all. pg recently posted [1] that someone he knows is generating 10kloc/day now. There's no way people are using AI to generate that volume of code and reading it. How many invisible bugs are lurking in that code base, waiting to be found some time in the future after the code has shipped? [1] https://x.com/paulg/status/1953289830982664236

I read every line I generate and usually adjust things; I'm uncomfortable merging a PR I haven't put my fingerprints on somehow. From the conversations I have with other practitioners, I think this is pretty normal. So, no, I reject your premise.

[dead]

Re: The current state of LLM-driven development

#226

Post by ModernMech »

tptacek wrote: post #224
ModernMech wrote: post #204

Earlier quoted context omitted.

With the volume of code people are generating, no you really can't just read it all. pg recently posted [1] that someone he knows is generating 10kloc/day now. There's no way people are using AI to generate that volume of code and reading it. How many invisible bugs are lurking in that code base, waiting to be found some time in the future after the code has shipped? [1] https://x.com/paulg/status/1953289830982664236

I read every line I generate and usually adjust things; I'm uncomfortable merging a PR I haven't put my fingerprints on somehow. From the conversations I have with other practitioners, I think this is pretty normal. So, no, I reject your premise.

My premise didn't have anything to do with you, so what you do isn't a basis for rejecting it. No matter what you or your small group of peers do, AI is generating code at a volume that all the developers in the world combined couldn't read if they dedicated 24hrs/day.

Re: The current state of LLM-driven development

#227

Post by simonw »

donperignon wrote: post #223
simonw wrote: post #99

Earlier quoted context omitted.

Learning how to Google is not trivial.

Do you have an entry in your CV saying: proficiency in googling? It difficult not because it is complex, it difficult because Google want it to be opaque and as harder as possible to figure out.

If anything getting good information out of Google has become harder for us expert users because Google have tried to make it easier for everyone else.

The power-user tricks like "double quote phrase searches" and exclusion though -term are treated more as gentle guidelines now, because regular users aren't expected to figure them out.

There's always "verbatim" mode, though amusingly that appears to be almost entirely undocumented! I tried using Google to find the official documentation for that feature just now and couldn't do better than their 2011 blog entry introducing it: https://search.googleblog.com/2011/11/search-using-your-term...

Maybe if I was more skilled at Google I'd be able to use it to find documentation on its own features?

Re: The current state of LLM-driven development

#228

Post by skydhash »

antonvs wrote: post #222
skydhash wrote: post #84

Earlier quoted context omitted.

I don’t think anyone expects perfection. Programs crash, drives die, and computers can break anytime. But we expect our tools to be reliable and not fight with it everyday to get it to work. I don’t have to debug Emacs every day to write code. My CI workflow just runs every time a PR is created. When I type ‘make tests’, I get a report back. None of those things are perfect, but they are reliable.

If you work in a team, you work with other people, whose reliability is more akin to LLMs than to the deterministic processes you're describing. What you're describing is a case of mismatched expectations.

Yep, but I don’t have to do their job for them. If they’re not reliable, at some point decisions will be taken to get them out of the project.

Re: The current state of LLM-driven development

#229

Post by KallDrexx »

dogcomplex wrote: post #218
KallDrexx wrote: post #177

Earlier quoted context omitted.

I would love to work at the places you have been where you are given enough time to throw out the prototype and do it properly. In my almost 20 years of professional experience this has never been the case and prototype and exploratory code has only been given minimal polishing time before reaching production and in use state.

We are all too well aware of the tragedy that is modern software engineering lol. Sadly I too have never seen that situation where I was given enough time to do the requisite multiple passes for proper design... I have been reprimanded and tediously spent collectively combing over said quick prototype code for far longer than the time originally provided to work on it though, as a proof of my incompetence! Does that…

Hah my bad, I misread your original comment as you saying you usually get the chance to do multiple passes on a prototype to productionalize it :)

Re: The current state of LLM-driven development

#230

Post by ruszki »

credit_guy wrote: post #101
tptacek wrote: post #8

Learning how to use LLMs in a coding workflow is trivial. There is no learning curve. You can safely ignore them if they don’t fit your workflows at the moment. I have never heard anybody successfully using LLMs say this before. Most of what I've learned from talking to people about their workflows is counterintuitive and subtle. It's a really weird way to open up an article concluding that LLMs make one a worse prog…

Fully agree. It takes months to learn how to use LLMs properly. There is an initial honeymoon where the LLMs blow your mind out. Then you get some disappointments. But then you start realizing that there are some things that LLMs are good at and some that they are bad at. You start creating a feel for what you can expect them to do. And more importantly, you get into the habit of splitting problems into smaller probl…

> There is an initial honeymoon where the LLMs blow your mind out.

What does this even mean?

In the first one and half years after ChatGPT released, when I used them there was a 100% rate, when they lied to me, I completely missed this honeymoon phase. The first time when it answered without problems was about 2 months ago. And that time was the first time when it answered one of them (ChatGPT) better than Google/Kagi/DDG could. Even yesterday, I tried to force Claude Opus to answer when is the next concert in Arena Wien, and it failed miserably. I tried other models too from Anthropic, and all failed. It successfully parsed the page of next events from the venue, then failed miserably. Sometimes it answered with events from the past, sometimes events in October. The closest was 21 August. When I asked what’s on 14 August, it said sorry, I’m right. When I asked about “events”, it simply ignored all of the movie nights. When I asked about them specifically, it was like I would have started a new conversation.

The only time when they made anything comparable to my code of quality was when they got a ton of examples of tests which looked almost the same. Even then, it made mistakes… when basically I had to change two lines, so copy pasting would have been faster.

There was an AI advocate here, who was so confident in his AI skill, that he showed something exact, which most of the people here try to avoid: recorded how he works with AIs. Here is the catch: he showed the same thing. There were already examples, he needed minimal modifications for the new code. And even then, copy pasting would have been quicker, and would have contained less mistakes… which he kept in the code, because it didn’t fail right away.

Post reply on HN