Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I write software with LLMs

stavros.io

Post reply on HN
Print view View original

371–380 of 544 posts

Re: How I write software with LLMs

#371

Post by borski »

staticassertion wrote: post #238
wiseowise wrote: post #137

Earlier quoted context omitted.

> But writing the code was never the point. Is that why most prestigious jobs grilled you like a devil on algos/system design? > The point has always been delivering the product to the customer, in any industry. Code is rarely the deliverable. That’s just nonsense. It’s like saying “delivering product was always the most important thing, not drinking water”.

It's well understood that programming interviews are a pretty shitty tool. They're a proxy for understanding if you have basic skills required to understand a computer. Notably, most companies don't rely on these alone, they have behavioral questions, architecture questions, etc. Have you ever done an interview at these companies you're talking about? They're 8 hours lol maybe 1 is spent programming. But it's just ve…

Yeah, this is precisely what I meant.

Re: How I write software with LLMs

#372

Post by raw_anon_1111 »

AstroBen wrote: post #367
petcat wrote: post #237

Earlier quoted context omitted.

As human developers, I think we're struggling with "letting go" of the code. The code we write (or agents write) is really just an intermediate representation (IR) of the solution. For instance, GCC will inline functions, unroll loops, and myriad other optimizations that we don't care about (and actually want!). But when we review the ASM that GCC generates we are not concerned with the "spaghetti" and the "high coup…

This is fantasy completely disconnected from reality. Have you ever tried writing tests for spaghetti code? It's hell compared to testing good code. LLMs require a very strong test harness or they're going to break things. Have you tried reading and understanding spaghetti code? How do you verify it does what you want, and none of what you don't want? Many code design techniques were created to make things easy for h…

I’ve found much more utility even pre AI in a good suite of integration tests than unit tests. For instance if you are doing a test harness for an API, it doesn’t matter if you even have access to the code if you are writing tests against the API surface itself.

Re: How I write software with LLMs

#373

Post by 3form »

cousin_it wrote: post #271

I'm thinking more and more that there's an ethical problem with using LLMs for programming. You might be reusing someone's GPL code with the license washed off. It's especially worrisome if the results end up in a closed product, competing with the open source project and making more money than it. Of course neither you nor the AI companies will face any consequence, the government is all-in and won't let you be hurt…

I would also add: if you're paying, supporting their cause with your money.

Sometimes I would like to have magical make-my-project tool for my selfish reasons; sometimes I know it would be a bad choice to fall behind on what's to come. But I really, really don't want to support that future.

Re: How I write software with LLMs

#374

Post by rirze »

zackify wrote: post #239
miguelgrinberg wrote: post #151

> One thing I’ve noticed is that different people get wildly different results with LLMs, so I suspect there’s some element of how you’re talking to them that affects the results. It's always easier to blame the prompt and convince yourself that you have some sort of talent in how you talk to LLMs that other's don't. In my experience the differences are mostly in how the code produced by the LLM is reviewed. Develope…

This definitely is the case. I was talking to someone complaining about how llms don't work good. They said it couldn't fix an issue it made. I asked if they gave it any way to validate what it did. They did not, some people really are saying "fix this" instead of saying "x fn is doing y when someone makes a request to it. Please attempt to fix x and validate it by accessing the endpoint after and writing tests" Its…

Untested Hypothesis: LLM instruction is usually an intelligence+communication-based skill. I find in my non-authoritative experience that users who give short form instructions are generally ill prepared for technical motivation (whether they're motivating LLMs or humans).

Re: How I write software with LLMs

#375

Post by anthonyrstevens »

sweaterkokuro wrote: post #319
danbruc wrote: post #139

I randomly clicked and scrolled through the source code of Stavrobot - The largest thing I’ve built lately is an alternative to OpenClaw that focuses on security. [1] and that is not great code. I have not used any AI to write code yet but considered trying it out - is this the kind of code I should expect? Or maybe the other way around, has someone an example of some non-trivial code - in size and complexity - writt…

In my experience its in all Language Models' nature to maximize token generation. They have been natively incentivized to generate more where possible. So if you dont put down your parameters tightly it will let loose. I usually put hard requirements of efficient code (less is more) and it gets close to how I would implement it. But like the previous comments say, it all depends on how deeply you integrate yourself i…

>> They have been natively incentivized to generate more where possible

Do you have any evidence of this?

Re: How I write software with LLMs

#376

Post by devlinMckhay »

the failure mode section is the most honest thing i have read about this whole thing. I hit that exact wall building something evenings after work. you miss one bad architectural decision because you are tired or in a hurry, and three sessions later the llm is confidently making it worse and you are not even sure when it started going wrong. the only thing that helped was slowing down on the planning side even when i did not feel like i had time for it.

Re: How I write software with LLMs

#377

Post by sarchertech »

raw_anon_1111 wrote: post #287
sarchertech wrote: post #267

Earlier quoted context omitted.

1. Agents aren’t humans. A human can write a working 100k LOC application with zero tests (not saying they should but they could and have). An agent cannot do this. Agents require tests to keep them from spinning out and your tests do not cover all of the behaviors you care about. 2. If you doubt that your tests don’t cover all your requirements, 99.9% of every production bug you’ve ever had completely passed your te…

I have never known a human that could or did write 100K lines of bug free working code without running parts of it first and testing. So humans also don’t write bug free code or tests that cover all use cases - how is that an argument that humans are better?

Not that humans can't write 100k line programs bug free or without running parts of it.

An AI cannot write a 100k line program on its own without external guard rails otherwise it spins out. This has nothing to do with whether the agent is allowed to run the code itself. This is well documented. Look at what was required to allow Claude to write a "C compiler".

This has nothing to do with whether it's bug free. It literally can't produce a working 100k LOC program without external guardrails.

Re: How I write software with LLMs

#378

Post by neobrain »

benterris wrote: post #322

One thing I don't get with this workflow, and all the ones we see in similar articles: do the authors run their agents in YOLO mode (full unchecked permission on their machine)? It seems their agents have full edit rights (scoped to a directory, which seems reasonable), but can also run tests autonomously (which means they can run any code), which equates to full read/write access on the machine? I mean, there are wa…

Claude has a sandbox mode that uses bubblewrap to build a lightweight filesystem sandbox that only exposes the project directory: https://code.claude.com/docs/en/sandboxing

It's disabled by default though, and in general (especially with other agents) you very much still have to get out of your way to get any sort of reasonable access control indeed.

In principle though, just running the agent CLI in something like firejail would get you very far if you know what you're doing.

Re: How I write software with LLMs

#379

Post by tracker1 »

miguelgrinberg wrote: post #151

> One thing I’ve noticed is that different people get wildly different results with LLMs, so I suspect there’s some element of how you’re talking to them that affects the results. It's always easier to blame the prompt and convince yourself that you have some sort of talent in how you talk to LLMs that other's don't. In my experience the differences are mostly in how the code produced by the LLM is reviewed. Develope…

I'm relatively forgiving on bugs that I kind of expect to have happen... just from experience working with developers... a lot of the bugs I catch in LLMs are exactly the same as those I have seen from real people. The real difference is the turn around time. I can stay relatively busy just watching what the LLM is doing, while it's working... taking a moment to review more solidly when it's done on the task I gave it.

Sometimes, I'll give it recursive instructions... such as "these tests are correct, please re-run the test and correct the behavior until the tests work as expected." Usually more specific on the bugs, nature and how I think they should be fixed.

I do find that sometimes when dealing with UI effects, the agent will go down a bit of a rabbit hole... I wanted an image zoom control, and the agent kept trying to do it all with css scaling and the positioning was just broken.. eventually telling it to just use nested div's and scale an img element itself, using CSS positioning on the virtual dom for the positioning/overflow would be simpler, it actually did it.

I've seen similar issues where the agent will start changing a broken test, instead of understanding that the test is correct and the feature is broken... or tell my to change my API/instructions, when I WANT it to function a certain way, and it's the implementation that is wrong. It's kind of weird, like reasoning with a toddler sometimes.

Re: How I write software with LLMs

#380

Post by AstroBen »

raw_anon_1111 wrote: post #372
AstroBen wrote: post #367

Earlier quoted context omitted.

This is fantasy completely disconnected from reality. Have you ever tried writing tests for spaghetti code? It's hell compared to testing good code. LLMs require a very strong test harness or they're going to break things. Have you tried reading and understanding spaghetti code? How do you verify it does what you want, and none of what you don't want? Many code design techniques were created to make things easy for h…

I’ve found much more utility even pre AI in a good suite of integration tests than unit tests. For instance if you are doing a test harness for an API, it doesn’t matter if you even have access to the code if you are writing tests against the API surface itself.

I do too, but it comes from a bang-for-your-buck and not a test coverage standpoint. Test coverage goes up in importance as you lean more on AI to do the implementation IMO.
Post reply on HN