51–59 of 59 posts
The problem I have with LLM-powered products is that they’re not marketed as LLMs, but as magic answer machines with phd-level pan-expertise. Lots of people in tech get frustrated and defensive when people criticize LLM-powered products and offer a defense as if people are criticizing LLMs as a technology. It’s perfectly reasonable for people to judge these products based on the way they’re presented as products. Kag…
Right, this is why I (author here) close the article mentioning that product design needs to keep the humans in the loop for these models to be useful. If the product is designed assuming humans will turn their brain off while using it, the fundamental unreliability of LLM behavior will create problems.
Earlier quoted context omitted.
Could you provide the steps you take to use LLMs as a tool? I'd like to compare them to the steps I would take to delegate a task to another human.
Keep feedback loops short and critical output to be verified by humans short. So this means that outputted answers in something like Kagi Assistant shouldn't be like those "Deep Research" report products where humans inevitably skim over the pages of outputted text. Similarly if you're using an LLM for coding or to write, keep diffs small and iteration cycles short. The point is to design the workflow to keep the hum…
Essentially they were asking if there's no meaningful difference between your "working with the tool" and "mindlessly 'delegating' work". I'm not seeing anything in your reply that would indicate such difference, so you could say that your "you shouldn't 'delegate' work" claim was bullshit.
Which makes total sense, because humans are also bullshitters. Yes, even I.
It successfully argues that LLMs are limited in usefulness without access to ground truth.
But that’s not the whole story!
Giving LLMs an ability to check their assertions, eg. by emitting and executing code to see if reality matches their word-vomit, or being able to research online - I wish the author had discussed how much of a game changer that is.
Yes I know I’m “only” talking about agents - “LLMs with tools and a goal, running in a loop”..
But adding ground truth takes you out of the loop. That’s super powerful. Make it so the LLM can ask something other than you to point out that that extra R in strawberry that they missed. In code we have code-writing agents but other industries can benefit from the same idea. Maybe a creative writer agent can be given a grammar checker for example.
It helps the thing do more on its own, and you’ll trust its output a lot more so you can use it for more things.
Yes - plain LLMs are stream-of-consciousness machines and basically emit bullshit, but that bullshit is often only minor corrections away from becoming highly useful autonomously emitted output.
They just need to validate against consensus reality to become insanely more useful than they are alone.
Earlier quoted context omitted.
LLMs can be useful as a tool, you shouldn't "delegate" work mindlessly to them. I don't "delegate" work to my nail gun or dishwasher, I work with the tool to achieve better productivity than without. When viewed in this framing, LLMs are undoubtedly a useful tool.
Could you provide the steps you take to use LLMs as a tool? I'd like to compare them to the steps I would take to delegate a task to another human.
Treat it exactly as the direct-able powerful autocomplete that it is, NOT an answering/reasoning engine.
> You should not go to an LLM for emotional conversations I'm more worried about who's keeping track of what's being shared with LLM's. Even if you could trust the model to respond with something meaningful, it's worth being very careful how much of your inner thoughts you share directly with a model that knows exactly who you are.
indeed:
```
Weizenbaum's own secretary reportedly asked Weizenbaum to leave the room so that she and ELIZA could have a real conversation. Weizenbaum was surprised by this, later writing: "I had not realized ... that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people."[23]
Every time people post these 'gotcha' LLM failures, they never work when I try them myself. E.g. ChatGPT has no problem with the surgeon being a dog: https://chatgpt.com/share/691e04cc-5b30-800c-8687-389756f36d... Neither does Gemini: https://gemini.google.com/share/6c2d08b2ca1a
Hi, author here! One issue with private LLM tests (including gotcha questions) is that they take time to design and once public, they become irrelevant. So I'm wary of sharing too many in a public blog. The surgeon dog was well known in May, the newest generation of models have all corrected against it. Those gotcha questions are generally called "misguided attention" traps, they're useful for blogs because they're s…
Here's my river crossing puzzle one, from 2023.
https://chatgpt.com/share/691f0bb2-6498-8009-b327-791c14ae81...
ChatGPT-3 got the wrong answer. It merely pattern matched against having seen the river crossing problem before, and simply regurgitated the solution to the unaltered version of the puzzle.
But later versions have been able to one-shot solve the "puzzle".
Here's GPT-5.1 getting the right answer in one shot:
https://chatgpt.com/share/691f0c27-e284-8009-96a9-a17bf37939...
>to talk nonsense to especially with the intention of deceiving or misleading https://www.merriam-webster.com/dictionary/bullshit
like say Musk saying there'd be a million robotaxis on the road by next year in 2020. Gemini 2.5 getting the riddle wrong seems an honest mistake - a confused guess rather than an intention to deceive.
Slightly related, Hinton was amusing accusing Gary Marcus of confabulating rather than the LLMs https://youtu.be/d7ltNiRrDHQ
This post is great! It successfully argues that LLMs are limited in usefulness without access to ground truth. But that’s not the whole story! Giving LLMs an ability to check their assertions, eg. by emitting and executing code to see if reality matches their word-vomit, or being able to research online - I wish the author had discussed how much of a game changer that is. Yes I know I’m “only” talking about agents -…