[deleted]
I'm dialing back my LLM usage
71–80 of 252 posts
Re: I'm dialing back my LLM usage
#72I think that people are just too quick to assume this is amazing, before it is there. Which doesn't mean it won't get there.
Somehow if I take the best models and agents, most hard coding benchmarks are at below 50% and even swe bench verified is like at 75 maybe 80%. Not 95. Assuming agents just solve most problems is incorrect, despite it being really good at first prototypes.
Also in my experience agents are great to a point and then fall off a cliff. Not gradually. Just the type of errors you get past one point is so diverse, one cannot even explain it.
Re: I'm dialing back my LLM usage
#73Over the last couple months I've gone from highly skeptical to a regular user (Copilot in my case). Two big things changed: First, I figured out that only some models are good enough to do the tasks I want (Claude Sonnet 3.7 and 4 out of everything I've tested). Second, it takes some infrastructure. I've added around 1000 words of additional instructions telling Copilot how to operate, and that's on top of tests (which you should have anyway) and 3rd party documentation. I haven't tried the fleet-of-agents thing, one VS Code instance is enough and I want to understand the changes in detail.
Edit: In concrete terms the workflow is to allow Copilot to make changes, see what's broken, fix those, review the diff against the goal, simplify the changes, etc, and repeat, until the overall task is done. All hands off.
Re: I'm dialing back my LLM usage
#74Am I spending too much time on HN or is every post/comment section filled with this same narrative? Basically, LLMs are exciting but they produce messy code for which the dev feels no ownership. Managing a codebase written by an LLM is difficult because you have not cognitively loaded the entire thing into your head as you do with code written yourself. They're okay for one-off scripts or projects you do not intend t…
> workflow is without actually showing any code. an argument can be made that the code doesn't matter as long as the product works as it's supposed to (big asterisk here)
> the code doesn't matter
The only goal of a code generator is the code. I don't care whether it works or not (for specific scenarios and it could break 90% of the time). I want to see the generated code and, so far, I have never seen anything interesting besides todo lists made with ReactJS.
Re: I'm dialing back my LLM usage
#75Am I spending too much time on HN or is every post/comment section filled with this same narrative? Basically, LLMs are exciting but they produce messy code for which the dev feels no ownership. Managing a codebase written by an LLM is difficult because you have not cognitively loaded the entire thing into your head as you do with code written yourself. They're okay for one-off scripts or projects you do not intend t…
> The other side of it is people who seem to have 'gotten it' and can dispatch multiple agents to plan/execute/merge changes across a project and want to tell you how awesome their workflow is without actually showing any code. This is a great read on the situation. Do you think these people are just making it up/generating baseless hype?
I think people are rightly hesitant to share code that has their name on it but for which they know nothing about.
I have seen a few of these full blown llm coded projects and every one of them has has some giant red flashing warning at the top of the README about the project being llm generated.
So I think it’s probably a mix of avoiding embarrassment and self preservation.
Re: I'm dialing back my LLM usage
#76Earlier quoted context omitted.
> [LLMs] save me a ton of time doing either boilerplate work I hear this frequently from LLM aficionados. I have a couple of questions about it: 1) If there is so much boilerplate that it takes a significant amount of coding time, why haven't you invested in abstracting it away? 2) The time spent actually writing code is not typically the bottleneck in implementing a system. How much do you really save over the devel…
I don't know about the boilerplate part but when you are e.g. adding a new abstraction that will help simplify an existing pattern across the code base something like Copilot saves a ton of time. Write down what has to happen and why, then let the machine walk across the code base and make updates, update tests and docs, fix whatever ancillary breaks happen, etc. The real payoff is making it cheaper to do exploratory…
That's an interesting approach. You still have to review all the changes to make sure they're correct and that the code is maintainable, though. I could see this being a net savings on a legacy code base or a brand new system still in the "sketching" phase.
Re: I'm dialing back my LLM usage
#77I’m using ChatGPT (enterprise version paid by my employer) quite a lot lately, and I find it a useful tool. Here’s what I learned over time.
Don’t feed many pages of code to AI, it works best for isolated functions or small classes with little dependencies.
In 10% of cases when I ask to generate or complete code, the quality of the code is less than ideal but fixable with extra instructions. In 25% of cases, the quality of generated code is bad and remains so even after telling it what’s wrong and how to fix. When it happens, I simply ignore the AI output and do something else reasonable.
Apart from writing code, I find it useful at reviewing new code I wrote. Half of the comments are crap and should be ignored. Some others are questionable. However, I remember a few times when the AI identified actual bugs or other important issues in my code, and proposed fixes. Again, don’t copy-paste many pages at once, do it piecewise.
For some niche areas (examples are HLSL shaders, or C++ with SIMD intrinsics) the AI is pretty much useless, probably was not enough training data available.
Overall, I believe ChatGPT improved my code quality. Not only as a result of reviews, comments, or generated codes, but also my piecewise copy-pasting workflow improved overall architecture by splitting the codebase into classes/functions/modules/interfaces each doing their own thing.
Re: I'm dialing back my LLM usage
#78LLMs save me a lot of time as a software engineer because they save me a ton of time doing either boilerplate work or mundane tasks that are relatively conceptually easy but annoying to actually have to do/type/whatever in an IDE. But I still more-or-less have to think like a software engineer. That's not going to go away. I have to make sure the code remains clean and well-organized -- which, for example, LLMs can h…
I'm in a similar boat. I've only started using it more very recently, and it's really helping my "white-page syndrome" when I'm starting a new feature. I still have to fix a bunch of stuff, but I think it's easier for me to fix, tweak and refactor existing code than it is to write a new file from scratch. Often times there's a lot of repetition in the app I'm working on, and there's a lot of it that's already been ab…
Maybe it's my Lisp background, where it's arguably easier, but I find myself immediately thinking "Eliminate that repetition."
Re: I'm dialing back my LLM usage
#79I think LLMs have made a lot of developers forget the lessons in "Simple Made Easy": https://www.youtube.com/watch?v=SxdOUGdseq4 LLMs seem to be really good at reproducing the classic Ball of Mud, that can't really be refactored or understood. There's a lot of power in creating simple components that interact with other simple components to produce complex functionality. While each component is easy to understand and…
> I suppose once LLMs get really good at that skill, will be when we really won't need developers any more.
I don't really get this argument. So when LLMs become "perfect" software developers are we just going to have them running 24/7 shitting out every conceivable piece of software ever? What would anyone do with that?
Or do you expect every doctor, electrician, sales assistant, hairdresser, train driver etc. to start developing their own software on top of their existing job?
What's more likely is a few people will make it their jobs to find and break down problems people have that could use a piece of software and develop said piece of software using whatever means they have available to them. Today we call these people software developers.
Re: I'm dialing back my LLM usage
#80Am I spending too much time on HN or is every post/comment section filled with this same narrative? Basically, LLMs are exciting but they produce messy code for which the dev feels no ownership. Managing a codebase written by an LLM is difficult because you have not cognitively loaded the entire thing into your head as you do with code written yourself. They're okay for one-off scripts or projects you do not intend t…
It feels like a bell curve: - one big set of users who don't like it because it generates a lot of code and uses its own style of algorithms, and it's a whole lot of unfamiliar code that the user has to load up in their mind - as you said. Too much to comprehend, and quickly overwhelming. And then to either side - it unblocks users who simply couldn't have written the code on their own, who aren't even trying to load…
I'm in that last bracket. I don't really have LLMs do tasks that given enough time and scouring docs I couldn't have implemented myself. I set hard rules around architecture, components, general design patterns and then let the LLM go at it, after which review the result in multiple passes, like I would a junior's code. I could not care less about the minutiae of the actual implementation, as long as it conforms to my conventions and style guides and instructions.