Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Unreliability of LLMs and What Lies Ahead

verissimo.substack.com

Post reply on HN
Print view View original

111–120 of 164 posts

Re: The Unreliability of LLMs and What Lies Ahead

#111

Post by esafak »

jvalencia wrote: post #81
esafak wrote: post #75

Earlier quoted context omitted.

This is the model conflating correlation with causation. Perhaps with more data spurious correlations would disappear, but the 'right' way is to make the models learn causal, world models.

Well, and I think the future of LLMs is not just in the pure LLM, but the agentic ones. LLMs with deterministic tools to ferret out specifics. We're only starting here but the results will be far better than what we do today.

Agentic LLM by itself provides value, to be sure, but they could also be part of learning a causal model. That's how humans do it; by interacting with the world.

Re: The Unreliability of LLMs and What Lies Ahead

#112

Post by boardwaalk »

ok123456 wrote: post #64

MongoDB was basically "vibe coding" for RBDMs. After the hype cycle, there will be a wasteland of unmaintainable vibe-coded products that companies will have to pump unlimited amounts of money into to maintain.

Or we’ll just leave them behind and that’s fine. And I work day maintaining old stuff of varying quality. Conceptually, software composting.

Re: The Unreliability of LLMs and What Lies Ahead

#113

Post by bagacrap »

tveita wrote: post #110
brentm wrote: post #87

This is a good articulation of what is a real concern around the AI bull thesis. If a calculator works great 99% of the time you could not use that calculator to build a bridge. Using AI for more than code generation is still very difficult and requires a human in the loop to verify the results. Sometimes using AI ends up being less productive because you're spending all your time debugging it's outputs. It's great b…

> If a calculator works great 99% of the time you could not use that calculator to build a bridge. We know for certain that certified lawyers have committed malpractice by using ChatGPT, in part because the made-up citations are relatively easy to spot. Malpractice by engineers might take a little more time to discover.

Engineers' work is also externally verifiable, e.g. by unit tests for software, but I'm assuming by other sorts of automated protocols for civil engineering. I would hope a bridge is not built without triple checking the various outcomes.

Re: The Unreliability of LLMs and What Lies Ahead

#114

Post by 65 »

asadotzler wrote: post #83
65 wrote: post #80

Earlier quoted context omitted.

Yes, which is why it's slightly confusing why programming is being pushed so hard to use with LLMs. For things that don't need completely accurate information, sure. But for programming, data, and factual information, it's surprising to see so many people using LLMs.

Code runs or it doesn't, that's a sort of verification feedback that other use cases don't offer, at least not so immediately. Formal code verification is a thing, not so much for verification of say legal citations. Code is language with some well documented rules all over the training corpora. Many other use cases are hardly so well represented in model training. These are just a few of many, many reasons that code…

Code runs or it doesn't... but that doesn't mean it does what you want it to do.

An LLM could generate code that takes raw user input and adds it to a raw SQL query. Does it work? Yeah. Is it a terrible security flaw? Also yeah.

Additionally, if you want a certain UX and the LLM cannot get there but the code works, that doesn't mean it's successful.

Re: The Unreliability of LLMs and What Lies Ahead

#115

Post by danans »

mjburgess wrote: post #108
asadotzler wrote: post #88

Earlier quoted context omitted.

Bullshit works on lots of people. Seeming to be true, or even just plausible, is enough for most people. This is why powerful bullshit machines are dangerous tools.

If people were easy enough to convince that they had been deceived, then I'd not mind so much. It's the extraordinary lengths people will go to in order to protect the bullshit they acquired with far less scepticism. Genuinely wild leaps of logic, shallowness of reasoning, on-the-face-of-it non-sequiturs, claims offered as great defeaters which require only a single moment of reflection to see through. This is the pr…

> It's the extraordinary lengths people will go to in order to protect the bullshit they acquired with far less scepticism

That's the narrative bias at play. We all are subject to it, and for good reason. People need stories to help maintain a stable mental equilibrium and a sense of identity. Knowledge that contradicts the stories that form the foundation of their understanding of the world can be destabilizing, which nobody wants.

Especially when they are facing struggle and stress, people will cling to their stories, even if a lie or deception in the story might be harming them. Religious cults and conspiracy theories are often built on this tendency, but so is culture in general.

Re: The Unreliability of LLMs and What Lies Ahead

#116

Post by fellowniusmonk »

yoyohello13 wrote: post #95
fellowniusmonk wrote: post #16

Earlier quoted context omitted.

I really don't understand people who are down on LLM. In terms of code output. I have gone from the productivity of being a Sr. Engineer to a team with .8 of a Sr. Engineer, 5 Jr. Engineers and One dude solely dedicated to reading/creating documentation. Unlike a lot of my fellow engineers who are also from traditional CS backgrounds and haven't worked in revenue restricted startup environments, I also have been VERY…

> Those who are on the MOST extreme end of the mathematic side and are linguistically bereft HATE LLM's and effectively cannot use them. This is an interesting observation. It at least aligns with my experience. I wouldn't say I'm "linguistically bereft" lol, but I do lean more toward the "functional programming is beautiful" side. I even have a degree in math. I'm not totally down on LLM coding, but I do fall more o…

Yeah, I certainly don't mean to imply that's the only reason. There are MANY reasons to hate LLMs and people all up and down the spectrum hate them for any number of reasons. I definitely think utility is still language specific as well (LLMs are just terrible with some languages), project specific, etc.

I think currently there are prompts and approaches that help ensure functions stay small and easy to reason about but it's very context dependent. Certainly any language or framework that has large amount of boilerplate will be less painful to work with if you hate boilerplate, I think that could arguably be increasing enshitification though in a sense. The people who say tons of code is being generated and it will all come crashing down in an unmaintainable mess... I do kinda agree.

I'm glad I am not writing code in medical/flight control systems or something like that, I think LLMs can be used in that context but idk if they would save or increase time?

Certain types of tasks require greater precision. Like in working with wood, framing a house is fine but building a dovetailed cabinet drawer is not on the table if that makes sense?

My impression is that at this point work in high precision environments is still in the human domain and LLMs are not. Multi-agent approaches maybe, treating humans like the final agent in multi-agent approaches, maybe, idk, I'm not working on any life or death libraries or projects ATM but I do feel good about test coverage so maybe that's good enough in a lot of cases.

People who say non-devs can dev with ai or cursor, I think at this point that's just a way of getting non-technical people to burn tokens and give them more money, but idk if that will be true in six months you know?

Re: The Unreliability of LLMs and What Lies Ahead

#117

Post by vinni2 »

brentm wrote: post #87

This is a good articulation of what is a real concern around the AI bull thesis. If a calculator works great 99% of the time you could not use that calculator to build a bridge. Using AI for more than code generation is still very difficult and requires a human in the loop to verify the results. Sometimes using AI ends up being less productive because you're spending all your time debugging it's outputs. It's great b…

> If a calculator works great 99% of the time you could not use that calculator to build a bridge.

But if the alternative is doing calculations by hand (writing code manually) there is a higher chance of making mistakes.

Just like calculations are double checked while building bridges unit tests and code reviews should catch bugs introduced by LLM written code.

Re: The Unreliability of LLMs and What Lies Ahead

#118

Post by mjburgess »

danans wrote: post #115
mjburgess wrote: post #108

Earlier quoted context omitted.

If people were easy enough to convince that they had been deceived, then I'd not mind so much. It's the extraordinary lengths people will go to in order to protect the bullshit they acquired with far less scepticism. Genuinely wild leaps of logic, shallowness of reasoning, on-the-face-of-it non-sequiturs, claims offered as great defeaters which require only a single moment of reflection to see through. This is the pr…

> It's the extraordinary lengths people will go to in order to protect the bullshit they acquired with far less scepticism That's the narrative bias at play. We all are subject to it, and for good reason. People need stories to help maintain a stable mental equilibrium and a sense of identity. Knowledge that contradicts the stories that form the foundation of their understanding of the world can be destabilizing, whi…

I think there is a certain sort of person who, if not "wants" this destabilization, doesn't really experience the alternative. People primarily relating to the world through irony, say. So, characteristically, socrates (, some stand up comedians, and the like) who trade in aporia -- this feeling of destablization.

Re: The Unreliability of LLMs and What Lies Ahead

#119

Post by bgnn »

smeeger wrote: post #107

hallucinations are essentially the only thing keeping all knowledge workers from being made permanently redundant. if that doesnt make you a little concerned then you are a fool. and the predictions of all the experts in 2010 is that what is currently happening right in front of us could never happen within a hundred years. why are the predictions of experts more reliable now? anyone who dismisses the risks is just a…

I'm a knowledge worker (electrical engineer) but not one bit worried about being replaced by AI in yhe foreseeable future. It does not only neet to be reliable, but also should be able to create, as in create physically working complex systems for me to be worried. I have not seen anything remotely close this yet.

I believe AI/ML will eventually get there but definitely not with LLMs or hoarding the whole internet. Most of the human know-how isn't on internet!

Oh, I guess I'm a fool.

Re: The Unreliability of LLMs and What Lies Ahead

#120

Post by eterm »

whyowhy3484939 wrote: post #77
eterm wrote: post #44

There are jobs out there that have always been unreliable. A classic example is the Travel Agent. This was already a job driven to near-extinction just by Google, but LLMs are a nail in the travel agent coffin. The job was always fuzzy. It was always unreliable. A travel agent recommendation was never a stamp of quality or guarentee of satisfaction. But now, I can ask an LLM to compare and contrast two weeks in the S…

Once they start making deals with the relevant organizations, book rooms, handle insurance, replacement hotels, etc, then they'll replace travel agents. These guys don't just Google a bunch of tickets you know.

We're getting into semantics now, but I'm talking about the kind of person who used to sit in a physical store, waiting for someone to walk by and go into the travel agency.

In the 80's and 90's, this is how most people booked their holidays. It was labour intensive, people would spend some time talking with a travel agent in a store, who would have a good idea of the packages available, and be able to make recommendations and match people with holidays.

The remnants of agencies still provide the same services, but (for the most of us) it's all online, it's all tick-box based, and much of the protection is via ATOL/ABTA.

These services still exist, but they're no longer all over the high-street. Names like Thomas Cook, Lunn Poly, have either been absorbed (mostly by TUI), or collapsed, and largely disappeared from the high-street with just a few left. (Mostly Tui).

And those that are left, have been reduced, much like retail banking, to entering your details into the same websites and services available to anyone, and talking you through the results that the computer spits out, that you could have browsed yourself at home. The underpaid travel agent in the store isn't any better connected than you are. In fact, they're possibly even more pushy about pushing you toward the hotels with the best commission than the website is.

Post reply on HN