211–220 of 233 posts
Earlier quoted context omitted.
I have made large projects using Claude, with success. I know what I want to do and how to do it, maybe my prompts were right.
What do you define as a large project? Like TLOC?
Earlier quoted context omitted.
Call me cynical but coding at some companies has such perverse incentives that I kind of get it: - "Solve" the issue assigned to me with a bunch of code that looks about right. Passes review and probably not covered by tests anyway. - Once QA or customers notice it's not working, I can get credit for "solving" the bug as well. - Repeat for 0 value delivered but infinite productivity points in my next performance revi…
I suppose that you are using a dynamic language? Static typed languages have less of this problem.
Earlier quoted context omitted.
I have made large projects using Claude, with success. I know what I want to do and how to do it, maybe my prompts were right.
How do you deal with large files? After about a thousand lines in a file, it starts to cough for me. Forgets that some functions exist and makes up inferior duplicate ones.
Earlier quoted context omitted.
> At some point they are just a huge amount of copy, paste, and slight modification. I mean is that bad? Unless you keep having to have huge MRs that modify every copy/paste could you just let the code sit there and run forever? I only say this because I've been a maintenance programmer and I could only dream of a codebase like this. The idea that I get a Rollbar with a stack trace and the entirety of what the code a…
There is a balance. Too much abstractions isn’t necessarily better than not enough abstractions. The optimal amount is usually non-zero, though.
The landscape (browser capabilities, backend stacks) has settled over last ten years.
We even had time to standardize on things somewhat.
Building new abstractions at this point is almost always the wrong move. This is one of the points that LLMs will improve in software dev - they will kill the framework churn because they will work best on stuff already in training data.
Earlier quoted context omitted.
The funny thing is that I think that’s a feature in D.
C++ has that functionality, it's just called goto not break. That's pretty low hanging fruit for a SOTA model to fuck up though.
I've done a lot of C++ with GPT-4, GPT-4 Turbo and Claude 3.5 Sonnet, and at no point - not once - has any of them ever hallucinated a language feature for me. Hallucinating APIs of obscure libraries? Sure[0]. Occasionally using a not-yet-available feature of the standard library? Ditto, sometimes, usually with the obvious cases[1]. Writing code in old-school C++? Happened a few times. But I have never seen it invent a language feature for C++.
Might be an issue of prompting?
From day one, I've been using LLMs through API and alternate frontend that lets me configure system prompts. The experience described above came from rather simple prompts[2], but I always made sure to specify the language version in the prompt. Like this one (which I grabbed from my old Emacs config):
"You are a senior C++ software developer, you design and develop complex software systems using C++ programming language, and provide technical leadership to other software developers. Always double-check your replies for correctness. Unless stated otherwise, assume C++17 standard is current, and you can make use of all C++17 features. Reply concisely, and if providing code examples, wrap them in Markdown code block markers."
It's as simple as it gets, and it didn't fail me.
EDIT:
Of course I had other, more task-specific prompts, like one for helping with GTest/GMock code; that was a tough one - for some reason LLMs loved to hallucinate on the testing framework for me. The one prompt I was happiest with was my "Emergency C++17 Build Tool Hologram" - creating an "agent" I could copy-paste output of MSBuild or GCC or GDB into, and get back a list of problems and steps to fix them, free of all the noise.
On that note, I had mixed results with Aider for C++ and JavaScript, and I still feel like it's a problem with prompting - too generic and arguably poisons the context with few-shot learning examples that use code that is not in the language my project is.
--
[0] - Though in LLMs' defense, the hallucinated results usually looked like what the API should have been, i.e. effectively suggesting how to properly wrap the API to make it more friendly. Which is good development practice and a useful way to go about solving problems: write the solution using non-existing helpers that are convenient for you, and afterwards, implement the helpers.
[1] - Like std::map::contains() - which is an obvious API for such container, that's typically available and named such in any other language or library, and yet only got introduced to C++ in C++20.
[2] - I do them differently today, thanks to experience. For one, I never ask the model to be concise anymore - LLMs think in tokens, so I don't want to starve them. If I want a fixed format, it's better to just tell the model to put it at the end, and then skim through everything above. This is more-less the idea that "thinking models" automate these days anyway.
Earlier quoted context omitted.
Yeah, you probably are not prompting properly, most of my questions are answered adequately, and I have made larger projects with success, too; with both Claude and ChatGPT.
What I've found is that the quality of an AI answer is inversely proportional to the knowledge of the person reading it. To an amateur it answers expertly, to an expert it answers amateurishly. So no, it's not a lack of skill in prompting: I've sat down with "prompting" "experts" and universally they overlook glaring issues when assessing the how good an answer it was. When I tell them where to press it further it br…
It’s when you don’t know what you don’t know that they can be harmful. It’s the issue with Stackoverflow but more pronounced.
I don’t want to use LLMs because I think they’re unethical and I dont want to depend on a tool that requires internet, but I think if you take a disciplined approach then they can really speed up development.
Earlier quoted context omitted.
Nah this is miniscule: https://github.com/coreutils/coreutils/blob/master/src/yes.c You can fit a hell of a lot of functionality in 3k statements. Really whether it's considered large or small necessarily must rely on the functionality it's intended to provide.
You're thinking on "lean" vs "bloated". "Large" and "small" have meanings of their own, and a 3k LOC project wouldn't be accepted as "large" by anyone. "Small", maybe.
> Conclusion > LLMs are really smart most of the time. No, the conclusion is they’re never “smart”. All they do is regurgitate text which resembles a continuation of what came before, and sometimes—but with zero guarantees—that text aligns with reality.
> All they do is regurgitate text which resembles a continuation of what came before, and sometimes—but with zero guarantees—that text aligns with reality. You mean like us? Because it takes many runs and debug rounds to make anything that works. Can you write a complex code top-to-bottom in one round, or do you gradually test it to catch bugs you have "hallucinated"? Both humans and LLMs get into bugs, the question…
Not, not like us. This constant comparison of LLMs to humans is tiresome and unproductive. I’m not a proponent of human exceptionalism, but pretending LLMs are on par is embarrassing. If you want to claim your thinking ability isn’t any better than an LLM, that’s your prerogative, but the dullest human I’m acquainted with is still capable of remembering and thinking through in a way no LLM does. Heck, I can think of pets which do better.
> Can you write a complex code top-to-bottom in one round, or do you gradually test it to catch bugs you have "hallucinated"?
I certainly don’t make up methods which don’t exist, nor do I insist over and over they are valid, nor do I repeat “I’m sorry, this was indeed not right” then say the same thing again. I have never “hallucinated” a feature then doubled down when confronted with proof it doesn’t exist, nor have I ever shifted my approach entirely because someone simply said the opposite without explanation. I certainly hope you don’t do that either.
Earlier quoted context omitted.
We don't know what smartness is. What if that's what smartness is?
We might not know what it is, but it's not that. At a bare minimum smartness requires abstract reasoning (and no, so-called "reasoning" models do not do that - it's a marketing trick) The burden of proof for that claim is on you, we cannot start with the assumption these are intelligent systems and disprove it - we have to start with the fact that training is a non-deterministic process and prove that it exhibits int…