Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I program with LLMs

crawshaw.io

Post reply on HN
Print view View original

221–230 of 342 posts

Re: How I program with LLMs

#221

Post by davepeck »

dns_snek wrote: post #214
harrall wrote: post #207

Earlier quoted context omitted.

I believe it’s more that people hate trying new tools because they’ve already made their choice and made it their identity. However, there are also people who love everything new and jump onto the latest hype too. They try new things but then immediately advocate it without merit. Where are the sane people in the middle?

As an experienced software developer, I paid for ChatGPT for a couple of months, I trialed Gemini Pro for a couple of months, and I've used the current version of Claude. I'd be happy if LLMs could produce working code as often and as quickly as the evangelist claim, but whenever I try to use LLM to work on my day to day tasks, I almost always walk away frustrated and disappointed - and most of my work is boring on t…

I see less "painting as a luddite" in response to statements like this, and more... surprise. Mild skepticism, perhaps!

Your experience diverges from that of other experienced devs who have used the same tools, on probably similar projects, and reached different conclusions.

That includes me, for what it's worth. I'm a graybeard whose current work is primarily cloud data pipelines that end in fullstack web. Like most devs who have fully embraced LLMs, I don't think they are a magical panacea. But I've found many cases where they're unquestionably an accelerant -- more than enough to justify the cost.

I don't mean to say your conclusions are wrong. There seems to be a bimodal distribution amongst devs. I suspect there's something about _how_ these tools are used by each dev, and in the specific circumstances/codebases/social contexts, that leads to quite different outcomes. I would love to read a better investigation of this.

Re: How I program with LLMs

#222

Post by th4t1sW13rd »

brabel wrote: post #157
EagnaIonat wrote: post #104

Earlier quoted context omitted.

Have you tried using Ollama? You can download and run an LLM locally on your machine. You can also pick the right model for the right need and it's free.

Yes. If the AI is not integrated with the IDE, it's not as helpful. If there were an IDE plugin that let you use a local model, perhaps that would be an option, but I haven't seen that (Github Copilot allows selecting different models, but I didn't check more carefully whether that also includes a local one, anyone knows?).

This can use Ollama: https://www.continue.dev/

Re: How I program with LLMs

#223

Post by klibertp »

e12e wrote: post #45

Interesting. I wonder what the equivalent of sketch.dev would look like if it targeted Smalltalk and was embedded in a Smalltalk image (preferably with a local LLM running in smalltalk)? I'd love to be able to tell my (hypothetical smalltalk) tablet to create an app for me, and work interactively, interacting with the app as it gets built... Ed: I suppose I should just try and see where cloud ai can take smalltalk to…

Worth a look: https://github.com/feenkcom/gt4llm If you load this in GT, you'll get a Lepiter book with interactive tutorials.

Re: How I program with LLMs

#224

Post by travisporter »

gopalv wrote: post #88
dewitt wrote: post #52

One interesting bit of context is that the author of this post is a legit world-class software engineer already (though probably too modest to admit it). Former staff engineer at Google and co-founder / CTO of Tailscale. He doesn't need LLMs. That he says LLMs make him more productive at all as a hands-on developer, especially around first drafts on a new idea, means a lot to me personally. His post reminds me of an…

> That he says LLMs make him more productive at all as a hands-on developer, especially around first drafts on a new idea, means a lot to me personally. There is likely to be a great rift in how very talented people look at sharper tools. I've seen the same division pop up with CNC machines, 3d printers, IDEs and now LLMs. If you are good at doing something, you might find the new tool's output to be sub-par over wha…

> I got into this profession simply because I could Ctrl-Z to the previous step much more easily than my then favourite chemical engineering goals.

That is interesting. Asking as a complete ignoramus - is there not a way to do this now? Like start off with a 100 of reagent and at every step use a bit and discard if wrong

Re: How I program with LLMs

#225

Post by ssivark »

travisporter wrote: post #224
gopalv wrote: post #88

Earlier quoted context omitted.

> That he says LLMs make him more productive at all as a hands-on developer, especially around first drafts on a new idea, means a lot to me personally. There is likely to be a great rift in how very talented people look at sharper tools. I've seen the same division pop up with CNC machines, 3d printers, IDEs and now LLMs. If you are good at doing something, you might find the new tool's output to be sub-par over wha…

> I got into this profession simply because I could Ctrl-Z to the previous step much more easily than my then favourite chemical engineering goals. That is interesting. Asking as a complete ignoramus - is there not a way to do this now? Like start off with a 100 of reagent and at every step use a bit and discard if wrong

But for every step that turns out to be "correct" you now have to go back and redo that in your held-out sample anyways. So it's not like you get to save on repeating the work -- IIUC you just changed it from depth-first execution order to breadth-first execution order.

Re: How I program with LLMs

#226

Post by efnx »

davepeck wrote: post #221
dns_snek wrote: post #214

Earlier quoted context omitted.

As an experienced software developer, I paid for ChatGPT for a couple of months, I trialed Gemini Pro for a couple of months, and I've used the current version of Claude. I'd be happy if LLMs could produce working code as often and as quickly as the evangelist claim, but whenever I try to use LLM to work on my day to day tasks, I almost always walk away frustrated and disappointed - and most of my work is boring on t…

I see less "painting as a luddite" in response to statements like this, and more... surprise. Mild skepticism, perhaps! Your experience diverges from that of other experienced devs who have used the same tools, on probably similar projects, and reached different conclusions. That includes me, for what it's worth. I'm a graybeard whose current work is primarily cloud data pipelines that end in fullstack web. Like most…

I think it also depends on _what_ the domain is, and also to a certain degree the tools / stack you use. LLMs aren’t coherent or correct when working on novel problems, novel domains or using novel tools.

They’re great for doing something that has been done before, but their hallucinations are wildly incorrect when novelty is at play - and I’ll add they’re always very authoritative! I’m glad my languages of choice have a compiler!

Re: How I program with LLMs

#227

Post by davepeck »

efnx wrote: post #226
davepeck wrote: post #221

Earlier quoted context omitted.

I see less "painting as a luddite" in response to statements like this, and more... surprise. Mild skepticism, perhaps! Your experience diverges from that of other experienced devs who have used the same tools, on probably similar projects, and reached different conclusions. That includes me, for what it's worth. I'm a graybeard whose current work is primarily cloud data pipelines that end in fullstack web. Like most…

I think it also depends on _what_ the domain is, and also to a certain degree the tools / stack you use. LLMs aren’t coherent or correct when working on novel problems, novel domains or using novel tools. They’re great for doing something that has been done before, but their hallucinations are wildly incorrect when novelty is at play - and I’ll add they’re always very authoritative! I’m glad my languages of choice ha…

Yeah, absolutely.

LLMs work best for code when both (a) there's sufficient relevant training data aka we're not doing something particularly novel and (b) there's sufficient context from the current codebase to pick up expected patterns, the peculiarities of the domain models, etc.

Drop (a) and get comical hallucinations; drop (b) and quickly find that LLMs are deeply mediocre at top-level architectural and framework/library choices.

Perhaps there's also a (c) related to precision. You can write code to issue a SQL query and return JSON from an API endpoint in multiple just-fine ways. Misplace a pthread_mutex_lock, however, and you're in trouble. I certainly don't trust LLMs to get things like this right!

(It's worth mentioning that "novelty" is a tough concept in the context of LLM training data. For instance, maybe nobody has implemented a font rasterizer in Rust before, but plenty of people have written font rasterizers and plenty of others have written Rust; LLMs seem quite good at synthesizing the two.)

Re: How I program with LLMs

#228

Post by goatlover »

evilfred wrote: post #215
harrall wrote: post #207

Earlier quoted context omitted.

I believe it’s more that people hate trying new tools because they’ve already made their choice and made it their identity. However, there are also people who love everything new and jump onto the latest hype too. They try new things but then immediately advocate it without merit. Where are the sane people in the middle?

Middle Ground Fallacy

The middle ground between hyping the new tech and being completely skeptical about it is usually right. New tech is usually not everything it's hyped up to be, but also usually not completely useless or bad for society. It's likely we're not about to usher in the singularity or doom society, but LLMs are useful enough to stick around in various tools. Also it's probably the case that a percentage of they hype is driven by wanting funding.

Re: How I program with LLMs

#229

Post by dboreham »

gopalv wrote: post #88
dewitt wrote: post #52

One interesting bit of context is that the author of this post is a legit world-class software engineer already (though probably too modest to admit it). Former staff engineer at Google and co-founder / CTO of Tailscale. He doesn't need LLMs. That he says LLMs make him more productive at all as a hands-on developer, especially around first drafts on a new idea, means a lot to me personally. His post reminds me of an…

> That he says LLMs make him more productive at all as a hands-on developer, especially around first drafts on a new idea, means a lot to me personally. There is likely to be a great rift in how very talented people look at sharper tools. I've seen the same division pop up with CNC machines, 3d printers, IDEs and now LLMs. If you are good at doing something, you might find the new tool's output to be sub-par over wha…

Calculators vs slide rules.

Re: How I program with LLMs

#230

Post by godelski »

tikkun wrote: post #179
gopalv wrote: post #88

Earlier quoted context omitted.

> That he says LLMs make him more productive at all as a hands-on developer, especially around first drafts on a new idea, means a lot to me personally. There is likely to be a great rift in how very talented people look at sharper tools. I've seen the same division pop up with CNC machines, 3d printers, IDEs and now LLMs. If you are good at doing something, you might find the new tool's output to be sub-par over wha…

Intersting. So engineers that like to iterate and explore are more likely to like LLMs. Whereas engineers that like have a more rigid specific process are more likely to dislike LLMs.

I frequently iterate and explore when writing code. Code gets written multiple times before being merged. Yet, I still haven't found LLMs to be helpful in that way. The author gives "autocomplete", "search", and "chat-driven programming" as 3 paradigms. I get the most out of search (though a lot of this is due to the decreasing value of Google), autocomplete is pretty weak to me especially as I macro or just use contextual complete, and I've failed miserably at chat-driven programming on every attempt. I spend more time debugging the AI than it would to debug myself. Albeit it __feels__ faster because I'm doing more typing + waiting rather than continuous thinking (but the latter has extra benefits).
Post reply on HN