Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's talk about LLMs

b-list.org

Post reply on HN
Print view View original

111–120 of 201 posts

Re: Let's talk about LLMs

#111

Post by handoflixue »

wavemode wrote: post #85
atleastoptimal wrote: post #73

Earlier quoted context omitted.

Complaining about every one off issue with LLM's ignores the bigger picture: they are getting better every month and there is no fundamental reason why they wouldn't surpass humans in coding. Everything else is secondary. All I would need from an LLM doubter is evidence that at tractable software engineering task LLM's are not improving. The strongest argument against the increasing general capabilities of LLM's are…

Your logic is flawed because, a thing can improve for an infinite amount of time while never surpassing a certain limit. It's called an asymptote. That being said, I don't even think that arguing about this from a mathematical perspective is a worthwhile use of time. Calling something an asymptote in the first place requires defining a quantifiable "X" and "Y", which we don't even have. What we have are a bunch of sy…

> Your logic is flawed because, a thing can improve for an infinite amount of time while never surpassing a certain limit. It's called an asymptote.

Have you ever once looked at a METR chart? https://files.civai.org/assets/METR_Chart.jpg

That's not an asymptote.

> there's also the fundamental fact that performance on benchmarks is not the same thing as performance in the real world

Again, yes, you're correct in the general case but it has very little to do with the specific case.

Would you find it convincing if I simply said "some internet arguments are wrong"? It's certainly a true statement, and you've made an internet argument here, so clearly you should accept that you're wrong, right?

Re: Let's talk about LLMs

#112

Post by handoflixue »

paganel wrote: post #53
aspenmartin wrote: post #41

Earlier quoted context omitted.

You say this as though performance has not followed a very clear and extremely rapid improvement in a startlingly short amount of time. You’re definitely right that people adopt agentic workflows and are disappointed or worse, but the point is the disappointment has already reduced substantially and will continue to do so. We know this because we know the scaling laws, and also because learning theory has been around…

> very clear and extremely rapid improvement in a startlingly short amount of time. We're almost 6 months into all this AI-code madness and I've yet to see that "rapid improvement" you mention. As in software products that are genuinely better compared to 6 months ago, or new software products (and good software products at that) which would have not existed had this AI craze not happened.

Can you name literally any other technology that had hundreds of millions of users within the first six months of being invented?

Six months after the internet was invented, you could send email between a few universities.

Six months after the computer was invented, they still hadn't actually built one.

The first transcontinental railroad, took about six YEARS just to build.

Re: Let's talk about LLMs

#113

Post by aspenmartin »

paganel wrote: post #53
aspenmartin wrote: post #41

Earlier quoted context omitted.

You say this as though performance has not followed a very clear and extremely rapid improvement in a startlingly short amount of time. You’re definitely right that people adopt agentic workflows and are disappointed or worse, but the point is the disappointment has already reduced substantially and will continue to do so. We know this because we know the scaling laws, and also because learning theory has been around…

> very clear and extremely rapid improvement in a startlingly short amount of time. We're almost 6 months into all this AI-code madness and I've yet to see that "rapid improvement" you mention. As in software products that are genuinely better compared to 6 months ago, or new software products (and good software products at that) which would have not existed had this AI craze not happened.

Way more than six months. You may be talking about how the world looks from your vantage point, as well you should. But there’s a reason why the world doesn’t allocate trillions of dollars of capital based on that.

I really value skeptical people and skepticism generally. But what I think skeptical people would prefer to consider themselves is: rational and reasonable, with their beliefs well calibrated.

You’re not the only one to think that literally nothing major or significant has happened with AI but that’s simply wrong. Every major tech company - the ones poised to get the first best rewards, have already gotten good incremental revenue from AI via ads ranking/recommendations (Google, Meta, etc.), good productivity increases due to scale of workforce and advanced in house tooling. You won’t see these numbers and you don’t have to believe them. But I have seen them and I believe them, and I, like you, hate bullshit.

Re: Let's talk about LLMs

#114

Post by aspenmartin »

leptons wrote: post #86
aspenmartin wrote: post #41

Earlier quoted context omitted.

You say this as though performance has not followed a very clear and extremely rapid improvement in a startlingly short amount of time. You’re definitely right that people adopt agentic workflows and are disappointed or worse, but the point is the disappointment has already reduced substantially and will continue to do so. We know this because we know the scaling laws, and also because learning theory has been around…

You say this as though AI company debt has not followed a very clear and extremely rapid ballooning in a startlingly short amount of time. It's the "YOLO" of business strategies.

Always amazes me how we’re on a platform with “ycombinator” in the url and people don’t understand how private companies scale to capture market share. You’re right Uber was that company that ran at a loss for so long and collapsed, another YOLO business strategy. Or maybe it was Amazon or…hmm I forget

Re: Let's talk about LLMs

#115

Post by aspenmartin »

slopinthebag wrote: post #49
aspenmartin wrote: post #41

Earlier quoted context omitted.

You say this as though performance has not followed a very clear and extremely rapid improvement in a startlingly short amount of time. You’re definitely right that people adopt agentic workflows and are disappointed or worse, but the point is the disappointment has already reduced substantially and will continue to do so. We know this because we know the scaling laws, and also because learning theory has been around…

Yes but we don't know the shape of the curve and where we are on it.

See chinchilla scaling laws, we have the functional form of the curve and know the constants (though they change and are domain and model specific):

L(N,D) ~= 1.69 + 406 / N^0.339 + 411 / D^0.285

L is loss (pre training test loss) D is the scale of the data N is the number of model parameters

Re: Let's talk about LLMs

#116

Post by [deleted] »

zapataband1 wrote: post #93
jatora wrote: post #78

Earlier quoted context omitted.

Uninformed opinion of someone who clearly doesnt consistently use AI coding tools, clearly. And why are you limiting it to 6 months? Whats wrong with you?

How many years of real-life, in-production problem solving/coding have you done? That's what I base how informed you are not how much you use your favorite new $100/month token-prediction subscription

[deleted]

Re: Let's talk about LLMs

#117

Post by wavemode »

handoflixue wrote: post #111
wavemode wrote: post #85

Earlier quoted context omitted.

Your logic is flawed because, a thing can improve for an infinite amount of time while never surpassing a certain limit. It's called an asymptote. That being said, I don't even think that arguing about this from a mathematical perspective is a worthwhile use of time. Calling something an asymptote in the first place requires defining a quantifiable "X" and "Y", which we don't even have. What we have are a bunch of sy…

> Your logic is flawed because, a thing can improve for an infinite amount of time while never surpassing a certain limit. It's called an asymptote. Have you ever once looked at a METR chart? https://files.civai.org/assets/METR_Chart.jpg That's not an asymptote. > there's also the fundamental fact that performance on benchmarks is not the same thing as performance in the real world Again, yes, you're correct in the g…

You're scoring rhetorical points while talking past my entire comment. Hard to say if you even read it.

I'm not "convincing" anyone of anything. I'm stating the reasons that I, personally, am unconvinced of a specific claim being made to me.

Re: Let's talk about LLMs

#118

Post by aspenmartin »

gerdesj wrote: post #102
aspenmartin wrote: post #99

Earlier quoted context omitted.

I would argue LLMs are possibly the largest paradigm shift the world has ever seen, and we are only at the beginning. The entire scaffolding and structure of programming is in the process of changing — coding has moved to orchestration and testing and governance of how to manage and productionalize code that has surpassed the capacity of human review. If this sounds melodramatic it’s likely that it hasn’t fully taken…

[How did you bang out this: — on your keyboard? Why did you decide to use backticks and 66/99 for quotes - nice but its not you is it?] Engage as a person, please.

Oh also! Two dashes on my phone converts to an EN dash I think (not em dash!)

Re: Let's talk about LLMs

#119

Post by rglover »

zapataband1 wrote: post #95
rglover wrote: post #92

We are obsessed with fortune telling. Use the damn thing or don't. It's that simple.

it would be that simple if a mental-health altering token-predictor wasn't being consistently shoved down our collective throats.

Fair. But it's worth considering that anything "they" need to shove is a tell that they need you more than you need them. The sky-hath-fallen narrative is just what top-dollar marketing gets you these days. Clever, but mostly bark.

Re: Let's talk about LLMs

#120

Post by spopejoy »

marcus_holmes wrote: post #94

I was waiting for the "so I tried coding something with an LLM myself, and I found..." paragraph. But apparently the author never did try it, or at least if they did, they didn't write about it. This is a very academic approach to the subject - read what other people have written about it without ever doing it yourself. Study what someone said about LLM coding 50 years ago, before they were even invented, to see what…

> I was waiting for the "so I tried coding something with an LLM myself, and I found..."

Why? Most of the article was about the productivity of teams.

> This is a very academic approach to the subject - read what other people have written about it

Meta-studies have tremendous value. He's asking a simple question: if LLMs are changing the world, let's look at what studies are showing.

> My experience has been remarkable, and, like others, I'm finding real joy in being able to move past the code to actually design and play with whole systems and architectures

Great! What does that have to do with the age-old problem that software development doesn't scale to teams well? It is indeed a "50 year old problem", so please tell us how LLMs solve it.

Post reply on HN