Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Where's the shovelware? Why AI coding claims don't add up

mikelovesrobots.substack.com

Post reply on HN
Print view View original

151–160 of 498 posts

Re: Where's the shovelware? Why AI coding claims don't add up

#151

Post by goalieca »

jes5199 wrote: post #118
goalieca wrote: post #26

There's a relatively monotonous task in software engineering that pretty much everyone working no a legacy c/c++ code base has had to face: static analysis and compiler warnings. That seems about as boring and routine of an operation that exists. As simple as can be. I've seen this task farmed out to interns paid barely anything just to get it done. My question to HN is... can LLMs do this? Can they convert all the u…

if you have a static analysis tool that gives a list of problems to fix, and something like a unit test suite that makes sure nothing got badly broken due to a sloppy edit, then yes. If you don’t have these things, you’ll accumulate mistakes

We’re talking legacy code bases. Automated Test coverage is generally poor.

Re: Where's the shovelware? Why AI coding claims don't add up

#152

Post by atleastoptimal »

The_Fox wrote: post #146
atleastoptimal wrote: post #127

Earlier quoted context omitted.

The article is one person recording their own use of AI, finding no statistical significance but claiming since that the evaluated ratio of AI:human speed in performing various coding tasks resembled the METR study, that AI has no value. People have already talked about issues with the METR study, but importantly with that study and this blog post, it querying a small number of people using AI tools for the first tim…

> The rapid growth of revenue of AI model companies, OpenAI, Anthropic, etc. You can't use growth of AI companies as evidence to refute the article. The premise is that it's a bubble. The growth IS the bubble, according to the claim. > I don't know why the author of TFA claims that he can't make a bunch of one-off apps I agree... One-off apps seem like a place where AI can do OK. Not that I care about it. I want AI t…

Bubbles are born out of evaluations, not revenue. Web3 was a bubble because the money its made wasn't real productivity, but hype cycles, pyramid schemes, etc. AI companies are merely selling API calls, there is no financial scheming, it is very simply that the product is worth what it is being sold for.

> I want AI that can build and maintain my enterprise B2B app just as well as I can in a fraction of the time, and that's not what has been delivered.

AI isn't at that level yet but it is making fast strides in subsets of it. I can't imagine systems of models and the models themselves won't reach there in a couple years given how bad AI coding tools were just a couple years ago.

Re: Where's the shovelware? Why AI coding claims don't add up

#153

Post by noidesto »

searls wrote: post #73

The answer is that we're making it right now. AI didn't speed me up at all until agents got good enough, which was April/May of this year. Just today I built a shovelware CLI that exports iMessage archives into a standalone website export. Would have taken me weeks. I'll probably have it out as a homebrew formula in a day or two. I'm working on an iOS app as well that's MUCH further along than it would be if I hand-r…

Agreed. Agentic AI is a completely different tool than “traditional” AI.

Im curious what the author’s data and experiment would look like a year from now.

Re: Where's the shovelware? Why AI coding claims don't add up

#154

Post by atleastoptimal »

ozgrakkurt wrote: post #145
atleastoptimal wrote: post #130

Earlier quoted context omitted.

I agree that if there are productivity gains that everyone should benefit, but the only thing that would allow this to happen are systems and incentive structures that allow that to happen. A manager's job is to increase revenue and cut costs, that's how they get their job, how they keep their job, and how they are promoted. People very rarely get free benefits outside the range of what the incentive structures they…

And people don’t like this. Something being logical doesn’t mean people have to accept it. Also AI has been basically useless every time I tried it except converting some struct definitions across languages or similar tasks, it seems very unlikely that it would boost productivity by more than 10% let alone 400%.

What AI coding tools/models have you been using?

Re: Where's the shovelware? Why AI coding claims don't add up

#155

Post by noidesto »

larve wrote: post #16

In case the author is reading this, I have the receipts on how there's a real step function in how much software I build, especially lately. I am not going to put any number on it because that makes no sense, but I certainly push a lot of code that reasonably seems to work. The reason it doesn't show up online is that I mostly write software for myself and for work, with the primary goal of making things better, not…

Agree. In the hands of a seasoned dev not only does productivity improve but the quality of outputs.

If I’m working against a deadline I feel more comfortable spending time on research and design knowing I can spend less time on implementation. In the end, it took the same amount of time, though hopefully with an increase of reliability, observability, and extendibility. None of these things show up in the author’s faulty dataset and experiment.

Re: Where's the shovelware? Why AI coding claims don't add up

#156

Post by baxtr »

some-guy wrote: post #15

These claims wouldn't matter if the topic weren't so deadly serious. Tech leaders everywhere are buying into the FOMO, convinced their competitors are getting massive gains they're missing out on. This drives them to rebrand as AI-First companies, justify layoffs with newfound productivity narratives, and lowball developer salaries under the assumption that AI has fundamentally changed the value equation. This is my…

AI has become the ultimate excuse for weak managers to pressure tech folks.

Re: Where's the shovelware? Why AI coding claims don't add up

#158

Post by shepherdjerred »

sumeno wrote: post #120
searls wrote: post #73

The answer is that we're making it right now. AI didn't speed me up at all until agents got good enough, which was April/May of this year. Just today I built a shovelware CLI that exports iMessage archives into a standalone website export. Would have taken me weeks. I'll probably have it out as a homebrew formula in a day or two. I'm working on an iOS app as well that's MUCH further along than it would be if I hand-r…

It's amazing how whenever criticisms pop up the responses for the last 3 years have been "well you aren't using , it's finally good!"

isn't this likely to be the case when a field is developing quickly and there are a large number of people who have different opinions on the subject?

e.g. I liked GitHub Copilot but didn't find it to be a game changer. I tried Cursor this year and started to see how AI can be today.

Re: Where's the shovelware? Why AI coding claims don't add up

#159

Post by vkou »

mandeepj wrote: post #139
vkou wrote: post #78

Earlier quoted context omitted.

And when it hallucinates a non-existant library, what are the magic prompts that you give it that makes it stop trying to bullshit you?

> what are the magic prompts that you give it that makes it stop trying to bullshit you? Maybe keep your eyes open? :-)

As I thought.

And for the record - my eyes are open. I'm aware I'm being bullshitted. I don't trust, I verify.

But I also don't have a magical lever that I can pull to make it stop hallucinating.

... and every time I ask if one exists, I get either crickets, or a response that doesn't answer the question.

Re: Where's the shovelware? Why AI coding claims don't add up

#160

Post by neilv »

falcor84 wrote: post #131
neilv wrote: post #111

There's also the questionable copyright/IP angle. As an analogy, can you imagine being a startup that hired a developer, and months later finding out the bulk of the new Web app they "coded" for you was actually copy&pasted open source code, loosely obfuscated, which they were passing it off as something they developed, and to which the company had IP rights? You'd immediately convene the cofounders and a lawyer, abo…

That's a fantastic piece of short fiction, but it is fiction. In practice though, I've seen so many copy&pasted unsourced open source snippets in proprietary code that I've lost all ability to be surprised by it, and I can't think of any one time where the company was sued about that, let alone anyone facing any personal repercussions, not even those junior devs. And if anything, by being "lossy encyclopedias" rather…

Though one recent related scandal was:

https://techcrunch.com/2024/09/30/y-combinator-is-being-crit...

Post reply on HN