They’re pretty great for printf debugging. Yesterday I was confounded by a bug so I rapidly added a ton of logging that the LLM wrote instantly, then I had the LLM analyze the state difference between the repro and non repro logs. It found something instantly that it would have taken me a few hours to find, which led me to a fix.
How I program with LLMs
211–220 of 342 posts
Re: How I program with LLMs
#212I no longer work in tech, but I still write simple applications to make my work life easier. I frequently use what OP refers to as chat-driven programming, and I find it incredibly useful. My process starts by explaining a minimum viable product to the chat, which then generates the code for me. Sometimes, the code requires a bit of manual tweaking, but it’s usually a solid starting point. From there, I describe each…
>The biggest downside, however, is the rapid accumulation of technical debt.
Is that really related to the LLM?
Even in pre-LLM times, anytime I've scrapped together some code to solve some small immediate problem it grows tech debt at an amazing rate. Getting a feel for when a piece of code is going to be around long enough that it needs to be refactored, cleaned up, documented, etc. is a skill I developed over time. Even now it isn't a prefect guess, as there is an ongoing tug of war between wasting time today refactoring something I might not touch again with wasting time tomorrow having to pick up something I didn't clean up.
Re: How I program with LLMs
#213One interesting bit of context is that the author of this post is a legit world-class software engineer already (though probably too modest to admit it). Former staff engineer at Google and co-founder / CTO of Tailscale. He doesn't need LLMs. That he says LLMs make him more productive at all as a hands-on developer, especially around first drafts on a new idea, means a lot to me personally. His post reminds me of an…
I have been using LLM to generate functional code from *pseudo-code* with excellent results. I am starting to experiment with UML diagrams, both with LLM and computer vision to actually generate code from UML diagrams; for example a simple activity diagram could be the prompt on LLM 's, and might look like: Start -> Enter Credentials -> Validate -> [Valid] -> Welcome Message -> [Invalid] -> Error Message Correspondin…
This example illustrates one of the risks of using LLMs without subject expertise though. I just tested this with claude and got that exact same validation method back. Using string comparison is dangerous from a security perspective [1], so this is essentially unsafe validation, and there was no warning in the response about this.
1. https://sqreen.github.io/DevelopersSecurityBestPractices/tim...
Re: How I program with LLMs
#214Earlier quoted context omitted.
> That he says LLMs make him more productive at all as a hands-on developer, especially around first drafts on a new idea, means a lot to me personally. There is likely to be a great rift in how very talented people look at sharper tools. I've seen the same division pop up with CNC machines, 3d printers, IDEs and now LLMs. If you are good at doing something, you might find the new tool's output to be sub-par over wha…
I believe it’s more that people hate trying new tools because they’ve already made their choice and made it their identity. However, there are also people who love everything new and jump onto the latest hype too. They try new things but then immediately advocate it without merit. Where are the sane people in the middle?
As an experienced software developer, I paid for ChatGPT for a couple of months, I trialed Gemini Pro for a couple of months, and I've used the current version of Claude.
I'd be happy if LLMs could produce working code as often and as quickly as the evangelist claim, but whenever I try to use LLM to work on my day to day tasks, I almost always walk away frustrated and disappointed - and most of my work is boring on technical merits, I'm not writing novel comp-sci algorithms or cryptography libraries.
Every time I say this, I'm painted as some luddite who just hates change when the reality is that no, current LLMs are just not fit for many of the purposes they're being evangelized for. I'd love nothing more than to be a 2x developer on my side projects, but it just hasn't happened and it's not for the lack of trying or open mindedness.
edit: I've never actually seen any LLM-driven developers work in real time. Are there any live coding channels that could convince the skeptics what we're missing out on something revolutionary?
Re: How I program with LLMs
#215Earlier quoted context omitted.
> That he says LLMs make him more productive at all as a hands-on developer, especially around first drafts on a new idea, means a lot to me personally. There is likely to be a great rift in how very talented people look at sharper tools. I've seen the same division pop up with CNC machines, 3d printers, IDEs and now LLMs. If you are good at doing something, you might find the new tool's output to be sub-par over wha…
I believe it’s more that people hate trying new tools because they’ve already made their choice and made it their identity. However, there are also people who love everything new and jump onto the latest hype too. They try new things but then immediately advocate it without merit. Where are the sane people in the middle?
Middle Ground Fallacy
Re: How I program with LLMs
#216Earlier quoted context omitted.
Communication skills are the keys to using LLMs. Think about it: every type of information you want is in them, in fact it is there multiple times, with multiple levels of seriousness in the treatment of the idea. If one is casual in their request, using casual language, then the LLM will reply with a casual reply because that matched your request best. To get a hard, factual answer from those that are experts in a s…
>every type of information you want is in them Actually, I'm afraid that no. It won't give us the step by step scalable processes to make humanity as a whole enter in a loop of indefinitely long period of world peace, with each of us enjoying life in its own thriving manner. That would be great information to broadcast, though. Also it equally has ability to produce large pile of completely delusional answers, that m…
After a small prompt engineering: https://0bin.net/paste/zolMrjVz#dgZrZzKU-PlxdkJTdG0pZU9bsCM3...
Re: How I program with LLMs
#217> I could not go a week without getting frustrated by how much mundane typing I had to do before having a FIM model For those not in-the-know, I just learned today that code autocomplete is actually called "Fill-in-the-Middle" tasks
Says who? I've been in the industry for nearly 25 years and have heard auto complete throughout but not once have I heard fill in the middle. Stop taking these blogs as oracle's of truth, they are not. These AI articles are full of this nonsense, to the point where it would appear to me many responses might just be Nvidia bots or whatever.
Author here.
FIM is a term of art in LLM research for a style of tokens used to implement code completion. In particular, it refers to training an LLM with the extra non-printing tokens:
func add(a, b int) int {
return
}
func add(a, b int) int {
return
}
It is, in effect, an encoding scheme for getting the prefix and suffix into the LLM context while positioning the next token to be where the cursor is.
(There are several variants of this scheme.)
Re: How I program with LLMs
#218Earlier quoted context omitted.
> His post reminds me of an old idea I had of a language where all you wrote was function signatures and high-level control flow Regardless of language, that's basically how you approach the design of a new large project - top down architecture first, then split the implementation into modules, design the major data types, write function signatures. By the time you are done what is left is basically the grunt work of…
> the grunt work of implementing it all you mean the fun part. I can really empathize with digital artists. I spent twenty years honing my ability to write code and love every minute of it and you're telling me that in a few years all that's going to be left is PM syncs and OKRs and then telling the bot what to write if I'm lucky to have a job at all
I think it depends on the size of the project. To me, the real fun of being a developer is the magic of being able to conceive of something and then conjure it up out of thin air - to go from an idea to reality. For a larger more complex project the major effort in doing this is the solution conception, top-down design (architecture), and design of data structures and component interfaces... The actual implementation (coding), test cases and debugging, then does become more like drudgework, not the most creative or demanding part of the project, other than the occasional need for some algorithmic creativity.
Back in the day (I've been a developer for ~45 years!) it was a bit different as hardware constraints (slow 8-bit processors with limited memory) made algorithmic and code efficiency always a primary concern, and that aspect was certainly fun and satisfying, and much more a part of the overall effort than it is today.
Re: How I program with LLMs
#219Earlier quoted context omitted.
I believe it’s more that people hate trying new tools because they’ve already made their choice and made it their identity. However, there are also people who love everything new and jump onto the latest hype too. They try new things but then immediately advocate it without merit. Where are the sane people in the middle?
Middle Ground Fallacy
Fallacy fallacy
Re: How I program with LLMs
#220Earlier quoted context omitted.
I believe it’s more that people hate trying new tools because they’ve already made their choice and made it their identity. However, there are also people who love everything new and jump onto the latest hype too. They try new things but then immediately advocate it without merit. Where are the sane people in the middle?
As an experienced software developer, I paid for ChatGPT for a couple of months, I trialed Gemini Pro for a couple of months, and I've used the current version of Claude. I'd be happy if LLMs could produce working code as often and as quickly as the evangelist claim, but whenever I try to use LLM to work on my day to day tasks, I almost always walk away frustrated and disappointed - and most of my work is boring on t…
You're the middle ground I was talking about. You tried it. You know where it works and where it doesn't.
I've used LLM to generate code samples and my IDE (IntelliJ) uses an LLM for auto-suggestions. That's mostly about it for me.