Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

2025: The Year in LLMs

simonwillison.net

Post reply on HN
Print view View original

561–570 of 643 posts

Re: 2025: The Year in LLMs

#561

Post by jheez3 »

threethirtytwo wrote: post #546
jheez3 wrote: post #535

Earlier quoted context omitted.

"You don’t have a solid background. If you want to go around huffing and puffing your chest about a subject area, you kinda do fella. Credibility.

Not only is what he saying in direct contradiction to what people with credibility have said, but his claimed credentials can be utter bullshit. This is the internet bro. Credibility is irrelevant because identities can never be verified. So the only thing that matters is the strength and rationality of an argument. That’s the point of hacker news substantive content not some battle of comparison of credentials or us…

[flagged]

Re: 2025: The Year in LLMs

#562

Post by jheez3 »

tim333 wrote: post #553
jheez3 wrote: post #537

Earlier quoted context omitted.

Thats all irrelevant. Is/was there tremendous value to be had by being able to transport data? Of course. No doubt about it. Everything else got figured out and investments were made because of that. The same line of thinking does not hold with LLMs given their non-deterministic nature. Time will tell where things land.

There's value in intelligence too.

Intelligence? No. Get the wording right. It’s driven by probability.

Re: 2025: The Year in LLMs

#563

Post by threethirtytwo »

jheez3 wrote: post #561
threethirtytwo wrote: post #546

Earlier quoted context omitted.

Not only is what he saying in direct contradiction to what people with credibility have said, but his claimed credentials can be utter bullshit. This is the internet bro. Credibility is irrelevant because identities can never be verified. So the only thing that matters is the strength and rationality of an argument. That’s the point of hacker news substantive content not some battle of comparison of credentials or us…

[flagged]

Strong point. I especially liked the part where you addressed literally anything.

Re: 2025: The Year in LLMs

#564

Post by threethirtytwo »

jheez3 wrote: post #538
coffeefirst wrote: post #156

Earlier quoted context omitted.

Well, this is the internet. Arguing about everything is its favorite pastime. But generally yes, I think back to Mongo/Node/metaverse/blockchain/IDEs/tablets and pretty much everything has had its boosters and skeptics, this is just more... intense. Anyway I've decided to believe my own eyes. The crowds say a lot of things. You can try most of it yourself and see what it can and can't do. I make a point to compare no…

Only those with great taste are well-equipped to make assertions about what we have infront of us. The rest is all noise and personally I just block it out.

Then why are you still here?

Re: 2025: The Year in LLMs

#565

Post by habinero »

CamperBob2 wrote: post #484
habinero wrote: post #483

Earlier quoted context omitted.

No, frankly it's the difference between actual engineers and hobbyists/amateurs/non-SWEs. SWEs are trained to discard surface-level observations and be adversarial. You can't just look at the happy path, how does the system behave for edge cases? Where does it break down and how? What are the failure modes? The actual analogy to a machine shop would be to look at whether the machines were adequate for their use case,…

You sound pretty certain. There's often good money to be made in taking the contrarian view, where you have insights that the so-called "smart money" lacks. What are some good investments to make in the extreme-bear case, in which we're all just Clever Hans-ing ourselves as you put it? Do you have skin in the game?

My dude, I assure you "humans are really good at convincing themselves of things that are not true" is a very, very well known fact. I don't know what kind of arbitrage you think exists in this incredibly anodyne statement lol.

If you want a financial tip, don't short stock and chase market butterflies. Instead, make real professional friends, develop real skills and learn to be friendly and useful.

I made my money in tech already, partially by being lucky and in the right place at the right time, and partially because I made my own luck by having friends who passed the opportunity along.

Hope that helps!

Re: 2025: The Year in LLMs

#566

Post by threethirtytwo »

jheez3 wrote: post #562
tim333 wrote: post #553

Earlier quoted context omitted.

There's value in intelligence too.

Intelligence? No. Get the wording right. It’s driven by probability.

Calling intelligence “just probability” is like calling music “just vibrations” and thinking you said something deep.

Re: 2025: The Year in LLMs

#567

Post by threethirtytwo »

jheez3 wrote: post #536
didip wrote: post #67

Indeed. I don't understand why Hacker News is so dismissive about the coming of LLMs, maybe HN readers are going through 5 stages of grief? But LLM is certainly a game changer, I can see it delivering impact bigger than the internet itself. Both require a lot of investments.

"I can see it delivering impact bigger than the internet itself. Both require a lot of investments." lol.... Just make sure you screenshot your post so you have a good reminder in a few years re. your predictive ability.

Predicting your own victory instead of defending it is a bold strategy.

Re: 2025: The Year in LLMs

#568

Post by habinero »

threethirtytwo wrote: post #491
habinero wrote: post #483

Earlier quoted context omitted.

No, frankly it's the difference between actual engineers and hobbyists/amateurs/non-SWEs. SWEs are trained to discard surface-level observations and be adversarial. You can't just look at the happy path, how does the system behave for edge cases? Where does it break down and how? What are the failure modes? The actual analogy to a machine shop would be to look at whether the machines were adequate for their use case,…

What you’re describing is just competent engineering, and it’s already been applied to LLMs. People have been adversarial. That’s why we know so much about hallucinations, jailbreaks, distribution shift failures, and long-horizon breakdowns in the first place. If this were hobbyist awe, none of those benchmarks or red-teaming efforts would exist. The key point you’re missing is the type of failure. Search systems fai…

> If this were hobbyist awe, none of those benchmarks or red-teaming efforts would exist.

Absolutely not true. I cannot express how strongly this is not true, haha. The tech is neat, and plenty of real computer scientists work on it. That doesn't mean it's not wildly misunderstood by others.

> Concluding from those failure modes that this is just Clever Hans is not adversarial engineering.

I feel like you're maybe misunderstanding what I mean when I refer to Clever Hans. The Clever Hans story is not about the horse. It's about the people.

A lot of people -- including his owner-- were legitimately convinced that a horse could do math, because look, literally anyone can ask the horse questions and it answers them correctly. What more proof do you need? It's obvious he can do math.

Except of course it's not true lol. Horses are smart critters, but they absolutely cannot do arithmetic no matter how much you train them.

The relevant lesson here is it's very easy to convince yourself you saw something you 100% did not see. (It's why magic shows are fun.)

Re: 2025: The Year in LLMs

#569

Post by aspenmartin »

Denzel wrote: post #533
aspenmartin wrote: post #478

Earlier quoted context omitted.

Sure: at my MAANG company, where I watch the data closely on adoption of CC and other internal coding agent tools, most (significant) LOC are written by agents, and most employees have adopted coding agents as WAU, and the adoption rate is positively correlated with seniority. Like a lot of things LLM related (Simon Willison's pelican test, researchers + product leaders implementing AI features) I also heavily "vibe"…

We probably work at the same company, given you used MAANG instead of FAANG. As one of the WAU (really DAU) you’re talking about, I want to call out a couple things: 1) the LOC metrics are flawed, and anyone using the agents knows this - eg, ask CC to rewrite the 1 commit you wrote into 5 different commits, now you have 5 100% AI-written commits; 2) total speed up across the entire dev lifecycle is far below 10x, mos…

Oh yes all of this I agree with. I had tried to clarify this above but your examples are clearer: my point is: all measures and studies I have personally seen of AI impact on productivity have been deeply flawed for one reason or another.

Total speed up is WAY less than 10x by any measure. 2x seems too high too.

By data alone it’s a bit unclear of impact I agree. But I will say there seems to be a clear picture that to me, starting from a prior formed from personal experience, indicates some real productivity impact today, with a trajectory that suggests these claims of a lot of SWE work being offloaded to agents over the next few years seems not that far fetched.

- adoption and retention numbers internally and externally. You can argue this is driven by perverse incentives and/or the perception performance mismatch but I’m highly skeptical of this even though the effects of both are probably really, it would be truly extraordinary to me if there weren’t at least a ~10-20% bump in productivity today and a lot of headroom to go as integration gets better and user skill gets better and model capabilities grow

- benchmark performance, again benchmarks are really problematic but there are a lot of them and all of them together paint a pretty clear picture of capabilities truly growing and growing quickly

- there are clearly biases we can think of that would cause us to overestimate AI impact, but there are also biases that may cause us to underestimate impact: e.g. I’m now able to do work that I would have never attempted before. Multitasking is easier. Experiments are quicker and easier. That may not be captured well by e.g. task completion time or other metrics.

I even agree: quality of agentic code can be a real risk, but:

- I think this ignores the fact that humans have also always written shitty code and always will; there is lots of garbage in production believe me, and that predates agentic code

- as models improve, they can correct earlier mistakes

- it’s also a muscle to grow: how to review and use humans in the loop to improve quality and set a high bar

Re: 2025: The Year in LLMs

#570

Post by aspenmartin »

ben_w wrote: post #556
aspenmartin wrote: post #467

Earlier quoted context omitted.

The cope + disappointment will be knowing that a large population of HN users will paint a weird alternative reality. There are a multitude of messages about AI that are out there, some are highly detached from reality (on the optimistic and pessimistic side). And then there is the rational middle, professionals who see the obvious value of coding agents in their workflow and use them extensively (or figure out how t…

> You then have massive hardware buildouts and improvements to stack + a ton of R&D + competition to squeeze the juice out of the current paradigm (there are 4 orders of magnitude of scaling left before we hit real bottlenecks) This is a surprising claim. There's only 3 orders of magnitude between US data centre electricity consumption and worldwide primary energy (as in, not just electricity) production. Worldwide e…

This is not my own claim, it’s based on the following analysis from Epoch: https://epoch.ai/blog/can-ai-scaling-continue-through-2030

But I forgot how old that article is: it’s 4 orders of magnitude past GPT-4 in terms of total compute which is I think only 3.5 orders of magnitude from where we are today (based on 4.4x scaling/yr)

Post reply on HN