Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Honest Review of AI Programming

mropert.github.io

Post reply on HN
Print view View original

71–80 of 89 posts

Re: An Honest Review of AI Programming

#71

Post by K0balt »

lnenad wrote: post #20
K0balt wrote: post #7

Earlier quoted context omitted.

To be fair, hammers are bad screwdrivers. They are faster at driving screws and require less finesse to do so, but are really bad at taking them out, often break the screw, and often result in a fastening that superficially looks correct but fails to perform well under stress. Two new types of screw, the ring-shank nonhelical fastener and the rivet, however, are changing everything. Soon, using screwdrivers will be r…

I don't really see your point. You're still using the screwdriver and it's up to you to use it when it makes sense, see if it's doing it's job properly, fix issues it creates etc... If it's not doing a good job for your particular screw don't use it. But don't say it's a horrible tool since you choose to not use it properly. There is a middle ground between CEOisms "LLMs are literally Jesus" and people similar to thi…

[dead]

Re: An Honest Review of AI Programming

#72

Post by hahahaa »

kamranjon wrote: post #41
hahahaa wrote: post #25

Earlier quoted context omitted.

> Hallucinations are an inherent property of how LLMs work. While technically true the hallucination rates on modern models is low and other checks can ensure that by the time a human sees it it is most likely solid. For research there is more danger as there is less feedback loop other than other LLM scrutinising the first. For research I get it to come to a conclusion but provide me with links so I can judge. More…

“While technically true the hallucination rates on modern models is low…” Isn’t this entirely context dependent? Where did you get the information that modern models have low hallucination rates? I’d love to see the benchmark if there is one, it seems like it would be useful to track.

Anecdotal. Older models felt they were on drugs.

Re: An Honest Review of AI Programming

#73

Post by hahahaa »

righthand wrote: post #33
hahahaa wrote: post #25

Earlier quoted context omitted.

> Hallucinations are an inherent property of how LLMs work. While technically true the hallucination rates on modern models is low and other checks can ensure that by the time a human sees it it is most likely solid. For research there is more danger as there is less feedback loop other than other LLM scrutinising the first. For research I get it to come to a conclusion but provide me with links so I can judge. More…

> it it is most likely solid. Most likely? That’s not reassuring at all. So you’re saying the other checks can result in hallucinations?

Most likely solid is definitely better than a lot of human made PRs I have seen.

But making a decision the human reviewer disagrees with or misunderstanding a spec but maybe not asking a follow up isn't hallucination in my book.

Hallucination is just making stuff up without checking.

Re: An Honest Review of AI Programming

#74

Post by ryanackley »

cronin101 wrote: post #58
ryanackley wrote: post #56

Earlier quoted context omitted.

LLMs work perfectly fine for me but I’m building web app equivalents of binder keepers, like most people. Essentially store data, display data. The novelty is in what/how we’re displaying. LLMs are owning this market. Unless you yourself are doing commercial game development, calling the author disingenuous puts your own pro-AI bias on display. It’s based on speculation. I prefer to take his very specific examples an…

I’ve specifically used Opus to diagnose and fix performance bottlenecks in parallel Rust code on multiple occasions (e.g improving NPS for a chess engine) and it works well. I’ve done plenty of performance architecting in my day-job and rule #1 is generally “you can’t fix what you can’t see/measure”. I have a suspicion that many folks aren’t investing in letting AI actually introspect iterative execution via the appr…

I understand you're trying to draw a parallel (no pun intended) between what you did with Rust and the work the author performs as a professional game developer who optimizes game code for a living (it says this in his bio).

Since, I'm assuming you are not a professional game developer, the parallel is speculative and sort of reaching. Therefore, is it feasible to you the author of the blog knows better than you what tools work for his chosen field and that he came to the conclusions he did in good faith?

Re: An Honest Review of AI Programming

#75

Post by keeda »

ryanackley wrote: post #34
cronin101 wrote: post #32

At the risk of sharing an unpopular take, this reads a lot like someone who decided how they feel about LLM-driven engineering ~5 months ago and doesn’t seem to reflect the current reality. (Or they’re working in an organization with lower budgets and not cranking the frontier models of today) I fully agree about the cost/sustainability parts, but to suggest you can’t build a high quality coding/verifying/iterating l…

His experience is completely plausible. He’s in a niche that requires highly performant code and most complex, highly performant games do nit have source available for models to train on. It’s a very common observation that the farther you stray from mainstream, the less effective the LLM models become.

> It’s a very common observation that the farther you stray from mainstream, the less effective the LLM models become.

It is a common observation but I don't buy it. AI is clearly very good at Rust, but that is probably one of the least represented languages in its dataset. Anecdotally, I've also been having very good outcomes with a rather niche combination of technologies (opencv.js + JS in a browser extension) since early 2024. I would imagine there is way more C++ game code in the training set than that particular combination.

I think the more likely reason is that certain languages, projects or technologies tend to be organized in ways that are not ideal for LLMs. Specifically, I think Object Oriented approaches are not ideal for LLMs.

My theory is the key factor for effective LLM use is how effectively you can stuff the context with only the relevant data. OO tends to result in logic spread across inheritance hierarchies and templates (and even overloaded operators /shudder) which resides in a bunch of different files comingled with a whole lot of other logic. This just tends to confuse the LLM. On the other hand, I ended up using a lot more functional programming style which let me pinpoint the exact files or snippets of code relevant to a task, and the LLM pretty much never went wrong.

These days the models (and likely the harnesses) are much stronger and need much less curation of context, and hence can power through any kind of project organization. But I suspect they are still a bit sensitive to all the noise polluting their contexts and hence can produce very inconsistent results.

Re: An Honest Review of AI Programming

#76

Post by archagon »

spprashant wrote: post #66

> While I have found LLMs useful for researching and planning code changes, my attempts at actually making them write code have been quite lackluster. I found them to be slow and expensive to generate, for a mediocre result. I think this observation is generally true for the kind of problems the author is working on. But I would not make the leap to avoid using LLMs for any kind of code writing. LLMs do fantastically…

All this completely ignores the cataclysmic negative externalities, including environmental costs, knowledge atrophy and workforce erosion, accelerated burnout, slop-pollution of the open internet, shifting of power from engineers to management and SAAS corps, and empowerment/enrichment of some of the most morally vacuous people in tech history.

Re: An Honest Review of AI Programming

#77

Post by archagon »

daishi55 wrote: post #17
nesarkvechnep wrote: post #10

Earlier quoted context omitted.

So you basically agree but complain there are no new arguments against AI programming in the article?

No, I’m not sure where you are getting the idea that I agree with what I am referring to as cliches last relevant in 2024.

Why are you posting such cliche responses?

There is literally nothing to see in this subthread.

Re: An Honest Review of AI Programming

#78

Post by daishi55 »

gqrt100 wrote: post #38
daishi55 wrote: post #3

> “Agentic workflow” (or whatever they’re calling it at the time you’re reading this article) > Unlike the silver bullets of the past (like microservices or NoSQL) > Hallucinations are an inherent property of how LLMs work. > It’s all marketing and buzzwords Not a serious article or thinker. I can get this stuff on Reddit if I want to read thrice-regurgitated cliches about AI.

The author is a real C++ programmer. Not like the PyTorch guys who use std::shrared_ptr for cyclical graphs and then go on to overpromise on threaded Python only to find out that the slowdown is much larger and everything is brittle as expected years ago . The times that Meta people can browbeat honest engineers are over. Nice corporate take.

lol

Re: An Honest Review of AI Programming

#79

Post by sigseg1v »

This is likely a completely honest and plausible experience of the author, while at the same time still feeling like someone trying to use a pottery wheel and then concluding that pottery wheels are not suitable for producing very good art due to the clay often not centering. It says more about the author than about the topic.

Re: An Honest Review of AI Programming

#80

Post by psd1 »

munksbeer wrote: post #70
psd1 wrote: post #63

Earlier quoted context omitted.

TFA's author's experience is the opposite of your claims. Your claims may be right in _your_ circumstances, but not theirs. > High performance algorithms are quite well documented That may be true for bloom filters or what have you. But the author states the obvious: all recent games are closed-source. So any algorithms or techniques for real-time 3D that an LLM was trained on are going to be a long way behind the st…

> That may be true for bloom filters or what have you. But the author states the obvious: all recent games are closed-source. So any algorithms or techniques for real-time 3D that an LLM was trained on are going to be a long way behind the state of the art. Neither you, nor the author of the article has to use LLMs for coding. But if you want to, there are some practices you should follow to get best results, and tha…

It can do A, ergo it can do B.

By the same logic, you have experience in low-latency numerical decision-making, ergo you could optimise a 3D rendering pipeline.

Yeah, no.

Post reply on HN