Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I genuinely don't understand why some people are still bullish about LLMs

twitter.com

Post reply on HN
Print view View original

761–770 of 1001 posts

Re: I genuinely don't understand why some people are still bullish about LLMs

#761

Post by maxdoop »

mattmanser wrote: post #525
bjornsing wrote: post #520

I’ve been using Claude a lot lately, and I must say I very much disagree. For example, the other day I was chatting with it about the health risks associated with my high consumption of grown salmon. It then generated a small program to simulate the accumulation of PCB in my body. I could review the program, ask questions about the assumptions, etc. It all seemed very reasonable. A toxicokinetic analysis it called it…

Google it. There's a bunch of scientific papers talking about that. Bet that plenty of those researchers have written python programs they've uploaded to GitHub and you just got one of their programs regurgitated It's not intelligence mate, it's just copying an existing program.

And how would you define intelligence then?

Re: I genuinely don't understand why some people are still bullish about LLMs

#762

Post by spamizbad »

mwigdahl wrote: post #759
Thlom wrote: post #669

Earlier quoted context omitted.

A municipality in Norway used LLM to create a report about the school structure in the municipality (how many schools are there, how many should there be, where should they be, how big should they be, pros and cons of different size schools and classes etc etc). Turns out the LLM invented scientific papers to use as references and the whole report is complete and utter garbage based on hallucinations.

All of these anecdotal stories about "LLM" failures need to go into more detail about what model, prompt, and scaffolding was used. It makes a huge difference. Were they using Deep Research, which searches for relevant articles and brings facts from them into the report? Or did they type a few sentences into ChatGPT Free and blindly take it on faith? LLMs are _tools_, not oracles. They require thought and skill to us…

If any non-trivial ask of an LLM also requires the prompts/scaffolding to be listed, and independently verified, along with its output, their utility is severely diminished. They should be saving time not giving us extra homework.

Far better to just get these problems resolved.

Re: I genuinely don't understand why some people are still bullish about LLMs

#763

Post by crazygringo »

Terr_ wrote: post #559
crazygringo wrote: post #42

If there's one common thread across LLM criticisms, it's that they're not perfect. These critics don't seem to have learned the lesson that the perfect is the enemy of the good . I use ChatGPT all the time for academic research. Does it fabricate references? Absolutely, maybe about a third of the time. But has it pointed me to important research papers I might never have found otherwise? Absolutely . The rate of inac…

> Verifying the accuracy of its statements is easy. For single statements, sometimes, but not always. For all of the many statements, no. Having the human attention and discipline to mindfully verify every single one without fail? Impossible. Every software product/process that assumes the user has superhuman vigilance is doomed to fail badly. > Automation centaurs are great: they relieve humans of drudgework and let…

> Having the human attention and discipline to mindfully verify every single one without fail? Impossible.

I mean, how do you live life?

The people you talk to in your life say factually wrong things all the time.

How do you deal with it?

With common sense, a decent bullshit detector, and a healthy level of skepticism.

LLM's aren't calculators. You're not supposed to rely on them to give perfect answers. That would be crazy.

And I don't need to verify "every single statement". I just need to verify whichever part I need to use for something else. I can run the code it produces to see if it works. I can look up the reference to see if it exists. I can Google the particular fact to see if it's real. It's really very little effort. And the verification is orders of magnitude easier and faster than coming up with the information in the first place. Which is what makes LLM's so incredibly helpful.

Re: I genuinely don't understand why some people are still bullish about LLMs

#764

Post by matthewkayin »

gilbetron wrote: post #109

I get so confused on this. I play around, test, and mess with LLMs all the time and they are miraculous. Just amazing, doing things we dreamed about for decades. I mean, I can ask for obscure things with subtle nuance where I misspell words and mess up my question and it figures it out. It talks to me like a person. It generates really cool images. It helps me write code. And just tons of other stuff that astounds me…

My issue with higher ups pushing LLMs is that what slows me down at work is not having to write the code. I can write the code. If all I had to do was sit down and write code, then I would be incredibly productive because I'm a good programmer.

But instead, my productivity is hampered by issues with org communication, structure, siloed knowledge, lack of documentation, tech debt, and stale repos.

I have for years tried to provide feedback and get leadership to do something about these issues, but they do nothing and instead ask "How have you used AI to improve your productivity?"

Re: I genuinely don't understand why some people are still bullish about LLMs

#765

Post by tempfile »

mwigdahl wrote: post #759
Thlom wrote: post #669

Earlier quoted context omitted.

A municipality in Norway used LLM to create a report about the school structure in the municipality (how many schools are there, how many should there be, where should they be, how big should they be, pros and cons of different size schools and classes etc etc). Turns out the LLM invented scientific papers to use as references and the whole report is complete and utter garbage based on hallucinations.

All of these anecdotal stories about "LLM" failures need to go into more detail about what model, prompt, and scaffolding was used. It makes a huge difference. Were they using Deep Research, which searches for relevant articles and brings facts from them into the report? Or did they type a few sentences into ChatGPT Free and blindly take it on faith? LLMs are _tools_, not oracles. They require thought and skill to us…

Why do you think these details are important? The entire point of these tools is that I am supposed to be able to trust what they say. The hard work is precisely to be able to spot which things are true and false. If I could do that I wouldn't need an assistant.

Re: I genuinely don't understand why some people are still bullish about LLMs

#766

Post by casey2 »

WhyOhWhyQ wrote: post #646
gilbetron wrote: post #109

I get so confused on this. I play around, test, and mess with LLMs all the time and they are miraculous. Just amazing, doing things we dreamed about for decades. I mean, I can ask for obscure things with subtle nuance where I misspell words and mess up my question and it figures it out. It talks to me like a person. It generates really cool images. It helps me write code. And just tons of other stuff that astounds me…

The technology is not just less than superintelligence, for many applications it is less than prior forms of intelligence like traditional search and Stack Exchange, which were easily accessible 3 years ago and are in the process of being displaced by LLMs. I find that outcome unimpressive. And this Tweeter's complaints do not sound like a demand for superintelligence. They sound like a demand for something far more…

The 404 links are hilarious, like you can't even parse the output and retry until it returns a link that doesn't 404? Even ignoring the billions in valuation, this is so bad for a $20 sub.

Re: I genuinely don't understand why some people are still bullish about LLMs

#767

Post by Fanmade »

gilbetron wrote: post #109

I get so confused on this. I play around, test, and mess with LLMs all the time and they are miraculous. Just amazing, doing things we dreamed about for decades. I mean, I can ask for obscure things with subtle nuance where I misspell words and mess up my question and it figures it out. It talks to me like a person. It generates really cool images. It helps me write code. And just tons of other stuff that astounds me…

I guess it totally depends on the specific use cases. Just yesterday, I asked Gemini 2.5 for how to test a Laravel artisan command that internally calls another artisan command using the Pest testing framework. Gemini showed me a method that looked perfect. But when I tried to execute it, I got an error that the method doesn't exist. So I asked Gemini about this, and it told me: "Ah, apologies! That's a classic sign that you might be using a version of Laravel older than 5.7." So I told it that I am actually using the latest release (12.3), to which it responded with: "You're right to question things when you get an error! However, let's clarify the Laravel version first.

As of today, March 27, 2025, the latest stable version of Laravel is Laravel 11, which was released in March 2024. Laravel 12 has not been released yet (it's expected roughly in Q1 2026 based on the usual schedule).

Could you please double-check the exact Laravel version you are using?" So it did not believe me and I had to convince it first that I was using a real version. This went on for a while, with Gemini not only hallucinating stuff, but also being very persistent and difficult to convince of anything else.

Well, in the end it was still certain that this method should exist, even though it could not provide any evidence for it and my searching through the internet and the Git history of the related packages did also not provide any results.

So I gave up and tried it with Claude 3.7 which could also not provide any working solution.

In the end, I found an entirely different solution for my problem, but that wasn't based on anything the AIs told me, but just my own thinking and talking to other software developers.

I would not go that far to call these AIs useless. In software development they can help with simple stuff and boilerplate code, and I found them a lot more helpful in creative work. This is basically the opposite from what I would have expected 5 years ago ^^

But for any important tasks, these LLMs are still far too unreliable. They often feel like they have a lot of knowledge, but no wisdom. They don't know how to apply their knowledge ideally, and they often basically brute-force it with a mix of strange creativity and statistical models that are apparently based on a vast amount of internet content that has big parts of troll content and satire.

Re: I genuinely don't understand why some people are still bullish about LLMs

#768

Post by casey2 »

tempfile wrote: post #765
mwigdahl wrote: post #759

Earlier quoted context omitted.

All of these anecdotal stories about "LLM" failures need to go into more detail about what model, prompt, and scaffolding was used. It makes a huge difference. Were they using Deep Research, which searches for relevant articles and brings facts from them into the report? Or did they type a few sentences into ChatGPT Free and blindly take it on faith? LLMs are _tools_, not oracles. They require thought and skill to us…

Why do you think these details are important? The entire point of these tools is that I am supposed to be able to trust what they say. The hard work is precisely to be able to spot which things are true and false. If I could do that I wouldn't need an assistant.

Because it's the difference between a fleshy hallucination and something that might related to reality.

Re: I genuinely don't understand why some people are still bullish about LLMs

#769

Post by belter »

gilbetron wrote: post #109

I get so confused on this. I play around, test, and mess with LLMs all the time and they are miraculous. Just amazing, doing things we dreamed about for decades. I mean, I can ask for obscure things with subtle nuance where I misspell words and mess up my question and it figures it out. It talks to me like a person. It generates really cool images. It helps me write code. And just tons of other stuff that astounds me…

You did not mention any useful business case, with real economic impact, either than a very smart template generator for Developers.

Choose a very narrow domain, that you known well, and you quickly realize they are just repeating the training data.

Re: I genuinely don't understand why some people are still bullish about LLMs

#770

Post by dlevine »

I think of LLMs like smart but unreliable humans. You don't want to use them for anything that you need to have right. I would never have one write anything that I don't subsequently go over with a fine-toothed comb.

With that said, I find that they are very helpful for a lot of tasks, and improve my productivity in many ways. The types of things that I do are coding and a small amount of writing that is often opinion-based. I will admit that I am somewhat of a hacker, and more broad than deep. I find that LLMs tend to be good at extending my depth a little bit.

From what I can tell, Sabine Hossenfelder is an expert in physics, and I would guess that she already is pretty deep in the areas that she works in. LLMs are probably somewhat less useful at this type of deep, fact-based work, particularly because of the issue where LLMs don't have access to paywalled journal articles. They are also less likely to find something that she doesn't know (unlike with my use cases, where they are very likely to find things that I don't know).

What I have been hearing recently is that it will take a long time for LLMs will be better than humans at everything. However, they are already better than many many humans at a lot of things.

Post reply on HN