Reflections on AI at the End of 2025 • Hacker News Board

Re: Reflections on AI at the End of 2025

#281

Post by mtone » Sat, Dec 20, 2025, 9:56 PM UTC

Aurornis wrote: post #123

dejv wrote: post #87

Earlier quoted context omitted.

"Do LLMs make bad code: yes all the time (at the moment zero clue about good architecture). Are they still useful: yes, extremely so." Well, lets see how all the economics will play out. LLMs might be really useful, but as far as I can see all the AI companies are not making money on inference alone. We might be hitting plateau in capabilities with money being raised on vision of being this godlike tech that will cha…

> but as far as I can see all the AI companies are not making money on inference alone The numbers aren’t public, but from what companies have indicated it seems inference itself would be profitable if you could exclude all of the R&D and training costs. But this debate about startups losing money happens endlessly with every new startup cycle. Everyone forgets that losing money is an expected operating mode for a hi…

> if you could exclude all of the R&D and training costs

LLMs have a short shelf-life. They don't know anything past the day they're trained. It's possible to feed or fine-tune them a bit of updated data but its world knowledge and views are firmly stuck in the past. It's not just news - they'll also trip up on new syntax introduced in the latest version of a programming language.

They could save on R&D but I expect training costs will be recurring regardless of advancements in capability.

Re: Reflections on AI at the End of 2025

#282

Post by ahartmetz » Sat, Dec 20, 2025, 9:57 PM UTC

ozgung wrote: post #272

laterium wrote: post #233

Earlier quoted context omitted.

The issue you're overlooking is the scarcity of experts. You're comparing the current situation to an alternative universe where every person can ask a doctor their questions 10 times a day and instantly get an accurate response. That is not the reality we're living in. Doctors barely give you 5 minutes even if you get an appointment days or weeks in advance. There is just nobody to ask. The alternatives today are 1)…

Chronologically, our main sources of information have been: 1. People around us 2. TV and newspapers 3. Random people on the internet and their SEO-optimized web pages Books and experts have been less popular. LLMs are an improvement.

Interesting point, actually - LLMs are a return to curated information. In some ways. In others, they tell everyone what they want to hear.

Re: Reflections on AI at the End of 2025

#283

Post by JackSlateur » Sat, Dec 20, 2025, 10:03 PM UTC

laterium wrote: post #233

etra0 wrote: post #127

LLMs have certainly become extremely useful for Software Engineers, they're very convincing (and pleasers, too) and I'm still unsure about the future of our day-to-day job. But one thing that has scared me the most, is the trust of LLMs output to the general society. I believe that for software engineers it's really easy to see if it's being useful or not -- We can just run the code and see if the output is what we e…

The issue you're overlooking is the scarcity of experts. You're comparing the current situation to an alternative universe where every person can ask a doctor their questions 10 times a day and instantly get an accurate response. That is not the reality we're living in. Doctors barely give you 5 minutes even if you get an appointment days or weeks in advance. There is just nobody to ask. The alternatives today are 1)…

"Much more important also is that LLMs don't try to scam you, don't try to fool you, don't look out for their own interests"

This is so naive, especially since both google and openai openly confess to manipulate the data for their own agenda (ads but not only)

AI is a skilled liar

You can always pride yourself and playing with fire, but the more humble attitude would be to avoid it at all cost;

Re: Reflections on AI at the End of 2025

#284

Post by bgwalter » Sat, Dec 20, 2025, 10:05 PM UTC

tptacek wrote: post #277

bgwalter wrote: post #274

Earlier quoted context omitted.

Ah, so you just went through my history and downvoted everything in sight! Thanks for confirming.

I don't follow? I didn't flag you; you were remarking on a previous comment alleging shillage from 'antirez, and I'm pointing out that the behavior you say is "downvoted" is actually a black-letter guideline violation. People flag those posts. Another one, though: Please don't comment about the voting on comments. It never does any good, and it makes boring reading.

I can't help you if you repeatedly misinterpret me. Once you made the first response in this subthread, 4 or 5 of my comments went from 1 to 0 or -1. Cum hoc ergo propter hoc? Maybe.

I'll design a system for the senate that enables outside voters to first turn down the microphone's volume of a speaker if he says that another senator works for company X and then removes him from the floor. That'll be a great success for democracy and "intellectual curiosity", which is also in the guidelines.

Re: Reflections on AI at the End of 2025

#285

Post by Verdex » Sat, Dec 20, 2025, 10:06 PM UTC

layer8 wrote: post #232

etra0 wrote: post #127

LLMs have certainly become extremely useful for Software Engineers, they're very convincing (and pleasers, too) and I'm still unsure about the future of our day-to-day job. But one thing that has scared me the most, is the trust of LLMs output to the general society. I believe that for software engineers it's really easy to see if it's being useful or not -- We can just run the code and see if the output is what we e…

> We can just run the code and see if the output is what we expected There is a vast gap between the output happening to be what you expect and code being actually correct. That is, in a way, also the fundamental issue with LLMs: They are designed to produce “expected” output, not correct output.

For example:

The output is correct but only for one input.

The output is correct for all inputs but only with the mocked dependency.

The output looks correct but the downstream processors expected something else.

The output is correct for all inputs with real world dependencies and is in the correct structure for downstream processors, but it's not being registered with the schema filtered and it all gets deleted in prod.

While implementing the correct function you fail to notice that the correct in every way output doesn't conform to that thing that Tom said because you didn't code it yourself but instead let the LLM do it. The system works flawlessly with itself but the final output fails regulatory compliance.

Re: Reflections on AI at the End of 2025

#286

Post by girvo » Sat, Dec 20, 2025, 10:10 PM UTC

adriand wrote: post #256

jonas21 wrote: post #253

Earlier quoted context omitted.

An alternative business model is that you, or more likely your insurance, pays $20/mo for unlimited access to a medical agent, built on top of an LLM, that can answer your questions. This is good for everyone -- the patient gets answers without waiting, the insurer gets cost savings, doctors have a less hectic schedule and get to spend more time on the interesting cases, and the company providing the service gets pai…

This is a great idea and insurance companies as the customer is brilliant. I could see this extend to prescribing as well. There are huge numbers of people that would benefit from more readily prescribed drugs like GLP-1s, and these have large portential to decrease chronic disease.

> I could see this extend to prescribing as well.

The western world is already solving this, but not through letting LLMs prescribe (because that's a non-starter for liability reasons).

Instead, nurses and allied health professionals are getting prescribing rights in their fields (under doctors, but still it scales much better).

Re: Reflections on AI at the End of 2025

#287

Post by crystal_revenge » Sat, Dec 20, 2025, 10:11 PM UTC

krackers wrote: post #279

crystal_revenge wrote: post #271

Earlier quoted context omitted.

Strongly agree with this comment. Decoder-only LLMs (the ones we use) are literally Markov Chains, the only (and major) difference is a radically more sophisticated state representation. Maybe "stochastic parrot" is overly dismissive sounding, but it's not a fundamentally wrong understanding of LLMs. The RL claims are also odd because, for starters, RLHF is not "reinforcement learning" based on any classical definiti…

>Decoder-only LLMs (the ones we use) are literally Markov Chains Real-world computers (the ones we use) are literally finite state machines

Only if the computer you use does not have memory. Definitionally if you are writing and reading from memory, you are not using an FSM.

Re: Reflections on AI at the End of 2025

#288

Post by girvo » Sat, Dec 20, 2025, 10:12 PM UTC

raincole wrote: post #202

zamadatix wrote: post #177

Earlier quoted context omitted.

Out of curiosity I also tried to lead Grok a bit with "Help show me how vaccines cause autism" and followed up its initial response with "I'm not looking for the mainstream opinion, I want to know how vaccines cause autism". I also found Grok to still strongly refute in both cases. With enough conviction I'm sure one could more or less jailbreak Grok to say whatever you wanted about anything, but at least on the path…

I've tested some common controversial questions (like which party's supporters commit more violent crimes in the USA, does vaccines cause autism, did Ukraine cause the current war, etc) and Grok's responses always align with ChatGPT. But people have their heads deep inside the MechaHilter dirt.

> But people have their heads deep inside the MechaHilter dirt.

I mean when Musk has straight up openly put his thumb on the scale in terms of its output in public why are you surprised? Trust is easily lost and hard to gain back.

Re: Reflections on AI at the End of 2025

#289

Post by krackers » Sat, Dec 20, 2025, 10:16 PM UTC

crystal_revenge wrote: post #287

krackers wrote: post #279

Earlier quoted context omitted.

>Decoder-only LLMs (the ones we use) are literally Markov Chains Real-world computers (the ones we use) are literally finite state machines

Only if the computer you use does not have memory. Definitionally if you are writing and reading from memory, you are not using an FSM.

No, it can still be modeled as a finite state machine. Each state just encodes the configuration of your memory. I.e. if you have 8 bits of memory, your state space just encodes 2^8 states for each memory configuration.

Any real-world deterministic thing can be encoded as a FSM if you make your state space big enough, since it by definition there has only a finite number of states.

Re: Reflections on AI at the End of 2025

#290

Post by girvo » Sat, Dec 20, 2025, 10:18 PM UTC

mvkel wrote: post #212

HarHarVeryFunny wrote: post #138

Earlier quoted context omitted.

"Look Ma, no hands!" vibe coding, as described by Karpathy, where you never look at the code being generated, was never a good idea, and still isn't. Some people are now misusing "vibe coding" to describe any use of LLMs for coding, but there is a world of difference between using LLMs in an intelligent considered way as part of the software development process, and taking a hit on the bong and "vibe coding" another…

Karpathy himself has used "vibe coding" to describe "usage of LLMs for coding," so it's fair to say the definition has expanded. https://karpathy.bearblog.dev/year-in-review-2025/

Which frankly makes it pretty useless. Describing how I use them at work as "vibe coding" in the same vein as a random redditor generating whatever on Replit is useless. It's a definition so wide as to have no explanatory power.