Adding a feature because ChatGPT incorrectly thinks it exists • Hacker News Board

Re: Adding a feature because ChatGPT incorrectly thinks it exists

#401

Post by pbhjpbhj » Tue, Jul 08, 2025, 3:59 PM UTC

eru wrote: post #317

suzzer99 wrote: post #53

Earlier quoted context omitted.

> Sometimes it comes up with a better approach than I had thought of. IMO this has always been the killer use case for AI—from Google Maps to Grammarly. I discovered Grammarly at the very last phase of writing my book. I accepted maybe 1/3 of its suggestions, which is pretty damn good considering my book had already been edited by me dozens of times AND professionally copy-edited. But if I'd have accepted all of Gram…

> The problem is executives want to completely remove humans from the loop, which almost universally leads to disastrous results. That's how you get economics of scale. Google couldn't have a human in the loop to review every page of search results before handing them out in response to queries.

Sure they could. We just want it to be otherwise.

What benefit might human review have? Maybe they could make sure the SERP list entries actually have the keywords you're looking for. Even better, they could make sure the prices in the shopping section are correct! Maybe even make sure they relate to the product you actually searched for... I might actually pay money for that.

Re: Adding a feature because ChatGPT incorrectly thinks it exists

#402

Post by tpmoney » Tue, Jul 08, 2025, 4:10 PM UTC

Sophira wrote: post #370

visarga wrote: post #364

Earlier quoted context omitted.

Oh the web was full of slop long before LLMs arrived. Nothing new. If anything, AI slop is higher quality than was SEO crap. And of course we can't uninvent AI just like we can't unborn a human.

It depends on the metric you use. Yes, AI text could be considered higher quality than traditional SEO, but at the same time, it's also very much not, because it always sounds like it might be authoritative, but you could be reading something hallucinated. In the end, the text was still only ever made to get visitors to websites, not to provide accurate information.

> it's also very much not, because it always sounds like it might be authoritative, but you could be reading something hallucinated.

People telling lies on the internet is an old enough and well known enough issue that it’s appeared in children’s TV shows. One need only dive down the rabbit hole of 9/11 “truthers” to see how much completely made up bullshit is published online as absolute fact with authoritative certainty. AI is the hot new thing and gets all the headlines, but Scottish Wikipedia was a problem long before AI and long after society largely settled its mind about how reliable Wikipedia is.

Re: Adding a feature because ChatGPT incorrectly thinks it exists

#403

Post by epidemian » Tue, Jul 08, 2025, 4:22 PM UTC

nomel wrote: post #323

epidemian wrote: post #299

Earlier quoted context omitted.

> Requiring perfection would either mean guardrails that would make it useless for most cases, or no LLM access until AGI exists What?? What does AGI have to do with this? (If this was some kind of hyperbolic joke, sorry, i didn't get it.) But, more importantly, the GP only said that in a sane world, the ChatGPT creators should be the ones trying to fix this mistake on ChatGPT. After all, it's obviously a mistake on…

> What does AGI have to do with this? Their requirement is no hallucinations [1], also stated as "be sure it didn't happen again" in the original comment. If you define a hallucination as something that wasn't in the training data, directly or indirectly (indirectly being something like an "obvious" abstract concept), then you've placed a profound constraint on the system, requiring determinism . That requirement fun…

> If you define a hallucination as something that wasn't in the training data, directly or indirectly (indirectly being something like an "obvious" abstract concept), then [...]

Ok, sure. But why would you choose to define hallucinations in a way that is contrary to common sense and the normal understanding of what an AI hallucination is?

The common definition of hallucinations is basically: when AI makes shit up and presents it as fact. (And the more technical definition also basically aligns with that.)

No one would say that if the AI takes the data you provide in the prompt and can deduce a correct answer for that specific data —something that is not directly or indirectly present in its training data— it would be hallucinating. In fact that would be an expected thing for an intelligent system to do.

It seems to me you're trying to discuss with something nobody said. You're making it seem that saying "it's bad that LLMs can invent wrong/misleading information like this and present it as fact, and that the companies that deploy them don't seem to care" is equivalent to "i want LLMs to be perfect and have no bugs whatsoever", and then discuss about how ridiculous is to state the latter.

Re: Adding a feature because ChatGPT incorrectly thinks it exists

#404

Post by sigbottle » Tue, Jul 08, 2025, 4:23 PM UTC

slowmovintarget wrote: post #263

kragen wrote: post #30

I've found this to be one of the most useful ways to use (at least) GPT-4 for programming. Instead of telling it how an API works, I make it guess, maybe starting with some example code to which a feature needs to be added. Sometimes it comes up with a better approach than I had thought of. Then I change the API so that its code works. Conversely, I sometimes present it with some existing code and ask it what it does…

That's not creativity. That's closer to simply observing the mean. For an analogy, it's like waiting to pave a path until people tread the grass in a specific pattern. (Some courtyard designers used to do just that. Wait to see where people were walking first.) Making things easy for Chat GPT means making things close to ordinary, average, or mainstream. Not creative, but can still be valuable.

Best way to put it. It's very hard to discuss even slightly unique concepts with GPT. It just keeps strawmanning ideas back to a common consensus without actually understanding the deep idea.

On the bright side, a lot of work is just finding the mean solution so.

Re: Adding a feature because ChatGPT incorrectly thinks it exists

#405

Post by kragen » Tue, Jul 08, 2025, 4:34 PM UTC

sjs382 wrote: post #269

jll29 wrote: post #169

Earlier quoted context omitted.

> The problem is executives want to completely remove humans from the loop, which almost universally leads to disastrous results Thanks for your words of wisdom, which touch on a very important other point I want to raise: often, we (i.e., developers, researchers) construct a technology that would be helpful and "net benign" if deployed as a tool for humans to use, instead of deploying it in order to replace humans.…

> Grammarly is great for sniffing out extra words and passive voice. But it doesn't get writing for humorous effect, context, deliberate repetition, etc. > But then along comes a greedy business manager who reckons recklessly Thanks for this. :)

I recklessly reckon I will go through the gateless gate to hear the sound of one hand clapping.

Re: Adding a feature because ChatGPT incorrectly thinks it exists

#406

Post by mikewarot » Tue, Jul 08, 2025, 4:51 PM UTC

So this model of ChatGPT obviously has been trained with the July 2028 dataset by mistake, including this discussion.

It'll all be fine in a few years. :-;

Re: Adding a feature because ChatGPT incorrectly thinks it exists

#407

Post by QuercusMax » Tue, Jul 08, 2025, 4:56 PM UTC

kragen wrote: post #212

adrianh wrote: post #122

Earlier quoted context omitted.

I don't fully understand your comment, but Soundslice has had first-class support for tablature for more than 10 years now. There's an excellent built-in tab editor, plus importers for various formats. It's just the ASCII tab support that's new.

I wonder if LLMs will stimulate ASCII formats for more things, and whether we should design software in general to be more textual in order to work better with LLMs.

I've had AI create ascii-art (Nethack-style) dungeon diagrams when I asked it to write me a D&D adventure. Last time I tried it these dungeon diagrams were completely nonsensical, but that was a few years ago.

Re: Adding a feature because ChatGPT incorrectly thinks it exists

#408

Post by kragen » Tue, Jul 08, 2025, 4:57 PM UTC

pvg wrote: post #303

kragen wrote: post #143

Earlier quoted context omitted.

These are good points, but I think they represent a somewhat narrow view of the issue. What's happening here is that we're discussing among ourselves what kinds of actions would be good or bad with respect to AI, just as we would with any other social issue, such as urban development, immigration, or marital infidelity. You could certainly argue that saying "please don't replace wetlands with shopping malls" or "plea…

Hectoring someone to 'stop doing this' is not 'starting a conversation', it's just hectoring.

A conversation on the topic certainly did ensue; see https://news.ycombinator.com/item?id=44492524 and https://news.ycombinator.com/item?id=44493015. Perhaps you mean to say that this wasn't the intended effect? But it was at least a highly predictable effect. Perhaps it would have gone better for the flamer if they had made the request without flaming not only the author in question and simonw.

To me the request in question seems to be in the same spirit as "Please don't play your music so loud at night", "Please don't look at my sister", or "Please don't throw your trash out your car window". In each of these cases, there's clearly a conflict between different people's desires, probably accompanied with underlying disagreements about relevant duties; perhaps one person believes the other has a duty to avert their gaze from the sister in question to show respect to her chastity, while their interlocutor does not subscribe to any such duty, believing he is entitled to look at whomever he pleases. Or perhaps one person believes the other has a duty to carry their trash to a trash can, while the other does not.

Given that such a conflict has arisen, how can we resolve it? We could merely refrain from trying to influence one another's behavior entirely, which is the lowest-effort approach, but this clearly leads to deeply suboptimal outcomes in many cases; perhaps the cost of turning down the stereo or carrying the garbage to a trash can would be almost trivial, so doing it to accommodate others' preferences results in a net improvement in welfare. Alternatively, we could try to exclude people whose normative beliefs differ from our own from the spaces that most affect us, but it should be obvious that this also often causes harms far out of proportion from the good that results, such as ethnic cleansing.

All the other approaches to resolving the conflict that I can think of—bargaining, mediation, arbitration, collective deliberation, etc.—begin unavoidably with stating the unfulfilled desire. Or, as you put it, hectoring someone to 'stop doing this'.

Re: Adding a feature because ChatGPT incorrectly thinks it exists

#409

Post by kunzhi » Tue, Jul 08, 2025, 4:58 PM UTC

garfij wrote: post #294

kunzhi wrote: post #270

Funny this article is trending today because I had a similar thought over the weekend - if I'm in Ruby and the LLM hallucinates a tool call...why not metaprogram it on the fly and then invoke it? If that's too scary, the failed tool call could trigger another AI to go draft up a PR with that proposed tool, since hey, it's cheap and might be useful.

We've done varying forms of this to differing degrees of success at work. Dynamic, on-the-fly generation & execution is definitely fascinating to watch in a sandbox, but is far to scary (from a compliance/security/sanity perspective) without spending a lot more time on guardrails. We do however take note of hallucinated tool calls and have had it suggest an implementation we start with and have several such tools in…

>Dynamic, on-the-fly generation & execution is definitely fascinating to watch in a sandbox, but is far to scary (from a compliance/security/sanity perspective) without spending a lot more time on guardrails.

Would love love love to hear more on what you are doing here? This seems super fascinating (and scary). :)

Re: Adding a feature because ChatGPT incorrectly thinks it exists

#410

Post by kragen » Tue, Jul 08, 2025, 5:07 PM UTC

QuercusMax wrote: post #407

kragen wrote: post #212

Earlier quoted context omitted.

I wonder if LLMs will stimulate ASCII formats for more things, and whether we should design software in general to be more textual in order to work better with LLMs.

I've had AI create ascii-art (Nethack-style) dungeon diagrams when I asked it to write me a D&D adventure. Last time I tried it these dungeon diagrams were completely nonsensical, but that was a few years ago.

I think ASCII art in particular is generally a weak point for LLMs; maybe they could do better if they used an image model. Other ASCII syntaxes using matched { } delimiters seem like they would be easier.