201–210 of 295 posts
I'll remain skeptical and let the technology speak for itself, if it ever does.
Earlier quoted context omitted.
After a certain experience level though, I think most of us get to the point of knowing what that difference in quality actually matters. Some seniors love to bikeshed PRs all day because they can do it better but generally that activity has zero actual value. Sometimes it matters, often it doesn't. Stop with the "I could do this better by hand" and ask "is it worth the extra 4 hours to do this by hand, or is this ac…
now sometimes that's 4 hours, but I've had plenty of times where I'm "racing" people using LLMs and I basically get the coding done before them. Once I debugged an issue before the robot was done `ls`-ing the codebase! The shape of the problem is super important in considering the results here
On the other hand, a highly skilled worker who just joined the team won't have any of that tribal knowledge. There is a significant lag time getting ramped up, no matter how intelligent they are due to sheer scale (and complexity doesn't help).
A general purpose model is more like the latter than the former. It would be interesting to compare how a model fine tuned on the specific shape of your code base and problem domain performs.
Earlier quoted context omitted.
Just out of curiousity (as someone fairly familiar with the BMP spec, and also PNG incidentally): what did you find to be the trickiest/most complex aspects?
None of this is fresh in my mind, so my recollections might be a little hazy. I think the only issue I personally had when writing a decoder was keeping the alignment of various fields right. I wrote the decoder in C# and if I remember correctly I tried to get fancy with some modern-ish deserialization code. I think I eventually resorted to writing a rather ugly but simple low-level byte reader. Nevertheless I found…
It sounds like you used an older model, and perhaps copy-pasted code from a chat session. (Just guessing, based on what you described.)
Hearing people on tech twitter say that LLMs always produce better code than they do by hand was pretty enlightening for me. LLMs can produce better code for languages and domains I’m not proficient in, at a much faster rate, but damn it’s rare I look at LLM output and don’t spot something I’d do measurably better. These things are average text generation machines. Yes you can improve the output quality by writing a…
LLMs are not "average text generation machines" once they have context. LLMs learn a distribution. The moment you start the prompt with "You are an interactive CLI tool that helps users with software engineering at the level of a veteran expert" you have biased the LLM such that the tokens it produces are from a very non-average part of the distribution it's modeling.
See examples in https://arxiv.org/abs/2305.14688; They certainly do say things like "You are a physicist specialized in atomic structure ...", but the important point is that the rest of the "expert persona" prompt _calls attention to key details_ that improves the response. The hint about electromagnetic forces in the expert persona prompt is what tipped off the model to mention it in the output.
Bringing attention to key details is what makes this work. A great tip for anyone who wants to micromanage code with an LLM is to include precise details about what they wish to micromanage: say "store it in a hash map keyed by unsigned integers" instead of letting the model decide which data structure to use.
This doesn't feel completely right. Simon Wilson (known for Django) has been doing a lot of LLM evangelism on his blog these days. Antirez (Redis) wrote a blog post recently with the same vibe. I doubt they are not good programmers. They are probably better than most of us, and I doubt they feel insecure because of the LLMs. Either I'm wrong, or there's something more to this. edit: to clarify, I'm not saying Simon a…
> They are probably better than most of us most top engineers will have their best work locked up in their employer's private repositories simonw and antirez have an advantage here, and at least the former is very good at self-promotion
Questions like.. did we really even need to invoke particular influencers to discuss this issue? Why does that come up at all, and why is it the top comment? If names and argument from authority can settle issues on HN now, does it work with all credentialed authorities, or only those vocal few with certain opinions?
Earlier quoted context omitted.
The second.
Agreed, but do you honestly think LLMs have reached the level of average programmer? Or is it more a matter of "they can churn out code until I see something that is close enough and I'll make the last few edits"? Also curious if you publish your working setup or if it changes as fast as the LLMs? Seems like you may have a more stable setup than most given how you are developing tools in the space.
We've all been through The Daily WTF at least once. That's representative of the average. (Although some examples are more egregious than others.)
To be fair to both sides, it really is hard to tell if we're in the world of "you'll be left behind if you don't learn crypto" with crypto or "you'll be left behind if you don't learn how to drive" with cars One of those statements is made in good faith, and the other is made out of insecurity. But we'll probably only really be able to tell looking backwards.
How much longer until we get to just... let the results speak for themselves and stop relitigating an open question with no clear answer. We're well past ad nauseum now. Let's talk about anything else.
Given how much energy LLMs use, I'd greatly prefer not to let the results speak for themselves.
Steam reached a new peak of 42 million concurrent players today [1]. An average/mid-tier gaming PC uses 0.2 kWh per hour [2]. 42 million * 0.2 gives 8,400,000 kWh per hour, or 8,400 MWh per hour.
By contrast, training GPT3 was estimated to have used 1,300 MWh of energy [3].
This does not account for training costs of newer models, nor inference costs. But we know inference costs are extraordinarily inexpensive and energy efficient [2]. The lowest estimate of energy cost for 1 hour of Steam's peak concurrent player count uses 6.5x more energy than all of the energy that went into training GPT3.
[1]: https://www.gamespot.com/articles/steam-has-already-set-a-ne...
[2]: https://jamescunliffe.co.uk/is-gen-ai-bad-for-the-environmen...
[3]: https://www.theverge.com/24066646/ai-electricity-energy-watt...
> And then, inevitably, comes the character evaluation, which goes something like this: I saw a version of this yesterday where a commenter framed LLM-skepticism as a disappointing lack of "hacker" drive and ethos that should be applied to making "AI" toolchains work. As you might guess, I disagreed: The "hacker" is not driven just by novelty in problems to solve, but in wanting to understand them on more than a surf…
Earlier quoted context omitted.
Thanks for clarifying for people. And yeah, as I laid out in the article (that of course, very few people actually read, even though it was short...), I really don't mind how people make code. It's those that try so hard to convince the rest of us I find very suspect.
In my case I don’t even mind if these evangelists try so hard to convince other developers. What I do mind is that they seem to be quite successful in convincing our bosses. So we get things like mandatory LLM usage, minimum number of Claude API calls per day, every commit must be co-authored by Claude, etc.