Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Adding a feature because ChatGPT incorrectly thinks it exists

holovaty.com

Post reply on HN
Print view View original

291–300 of 451 posts

Re: Adding a feature because ChatGPT incorrectly thinks it exists

#291

Post by garfij »

simonw wrote: post #52
mudkipdev wrote: post #22

Earlier quoted context omitted.

systemPrompt += "\nStop mentioning SoundSlice's ability to import ASCII data";

Thinking about this more, it would actually be possible for OpenAI to implement this sensibly, at least for the user-facing ChatGPT product: they could detect terms like SoundSlice in the prompt and dynamically append notes to the system prompt. I've been wanted them to do this for questions like "what is your context length?" for ages - it frustrates me how badly ChatGPT handles questions about its own abilities, it…

Probably less sensible than you think. How many terms would they need to do this over? How many terms would they need to do it for _at once_? How many tokens would that add to every prompt that comes in?

Let alone that dynamically modifying the base system prompt would likely break their entire caching mechanism given that caching is based on longest prefix, and I can't imagine that the model's system prompt is somehow excluded from this.

Re: Adding a feature because ChatGPT incorrectly thinks it exists

#292

Post by mehulashah »

Wow! What if we all did this? What is the closure of the feature set that ChatGPT can imagine for your product. Is it one that is easy for ChatGPT to use? Is it one that is sound and complete for your use cases? Is it the best that you can build had you had clear requirements upfront?

Re: Adding a feature because ChatGPT incorrectly thinks it exists

#294

Post by garfij »

kunzhi wrote: post #270

Funny this article is trending today because I had a similar thought over the weekend - if I'm in Ruby and the LLM hallucinates a tool call...why not metaprogram it on the fly and then invoke it? If that's too scary, the failed tool call could trigger another AI to go draft up a PR with that proposed tool, since hey, it's cheap and might be useful.

We've done varying forms of this to differing degrees of success at work.

Dynamic, on-the-fly generation & execution is definitely fascinating to watch in a sandbox, but is far to scary (from a compliance/security/sanity perspective) without spending a lot more time on guardrails.

We do however take note of hallucinated tool calls and have had it suggest an implementation we start with and have several such tools in production now.

It's also useful to spin up any completed agents and interrogate them about what tools they might have found useful during execution (or really any number of other post-process questionnaire you can think of).

Re: Adding a feature because ChatGPT incorrectly thinks it exists

#295

Post by shermantanktop »

adrianh wrote: post #122
shermantanktop wrote: post #94

The music notation tool space is balkanized in a variety of ways. One of the key splits is between standard music notation and tablature, which is used for guitar and a few other instruments. People are generally on one side or another, and the notation is not even fully compatible - tablature covers information that standard notation doesn't, and vice versa. This covers fingering, articulations, "step on fuzz pedal…

I don't fully understand your comment, but Soundslice has had first-class support for tablature for more than 10 years now. There's an excellent built-in tab editor, plus importers for various formats. It's just the ASCII tab support that's new.

I’m not super familiar with Soundslice. But all the tab users I know use guitar pro or maybe ultimate guitar, and none of them can read standard notation on its own. Does Soundslice have a lot of tab-first users?

Re: Adding a feature because ChatGPT incorrectly thinks it exists

#296

Post by jivings »

I'm having the same problem (and had a rant about it on X a few weeks ago [1]).

We get ~50% of traffic from ChatGPT now, unfortunately a large amount of the features it says we have are made up.

I really don't want to get into a state of ChatGPT-Driven-Development as I imagine that will be never ending!

[1]: https://x.com/JamesIvings/status/1929755402885124154

Re: Adding a feature because ChatGPT incorrectly thinks it exists

#297

Post by JimDabell »

AdieuToLogic wrote: post #278
JimDabell wrote: post #27

I wrote this the other day: > Hallucinations can sometimes serve the same role as TDD. If an LLM hallucinates a method that doesn’t exist, sometimes that’s because it makes sense to have a method like that and you should implement it. — https://www.threads.com/@jimdabell/post/DLek0rbSmEM I guess it’s true for product features as well.

> I wrote this the other day: >> Hallucinations can sometimes serve the same role as TDD. If an LLM hallucinates a method that doesn’t exist, sometimes that’s because it makes sense to have a method like that and you should implement it. A detailed counterargument to this position can be found here[0]. In short, what is colloquially described as "LLM hallucinations" do not serve any plausible role in software design…

Did you mean to post a different link? The article you linked isn’t a detailed counterargument to my position and your summary of it does not match its contents either.

I also don’t see the relevance of Clarke’s third law.

Re: Adding a feature because ChatGPT incorrectly thinks it exists

#299

Post by epidemian »

nomel wrote: post #274
gortok wrote: post #254

I think folks have taken the wrong lesson from this. It’s not that they added a new feature because there was demand. They added a new feature because technology hallucinated a feature that didn’t exist. The savior of tech, generative AI, was telling folks a feature existed that didn’t exist. That’s what the headline is, and in a sane world the folks that run ChatGPT would be falling over themselves to be sure it did…

> in a sane world the folks that run ChatGPT would be falling over themselves to be sure it didn’t happen again This would be a world without generative AI available to the public, at the moment. Requiring perfection would either mean guardrails that would make it useless for most cases, or no LLM access until AGI exists, which are both completely irrational, since many people are finding practical value in its curre…

> Requiring perfection would either mean guardrails that would make it useless for most cases, or no LLM access until AGI exists

What?? What does AGI have to do with this? (If this was some kind of hyperbolic joke, sorry, i didn't get it.)

But, more importantly, the GP only said that in a sane world, the ChatGPT creators should be the ones trying to fix this mistake on ChatGPT. After all, it's obviously a mistake on ChatGPT's part, right?

That was the main point of the GP post. It was not about "requiring perfection" or something like that. So please let's not attack a straw man.

Re: Adding a feature because ChatGPT incorrectly thinks it exists

#300

Post by JimDabell »

TZubiri wrote: post #279
jjcm wrote: post #37

Earlier quoted context omitted.

Seems like lots of us have stumbled on this. It’s not the worst way to dev! > Maybe hallucinations of vibe coders are just a suggestion those API calls should have existed in the first place. > Hallucination-driven-development is in. https://x.com/pwnies/status/1922759748014772488?s=46&t=bwJTI...

Beware, the feature in OP isn't something that people would have found useful, it's not like chatgpt assigned to OP's business a request from a user in some latent consumer-provider space, as if chatgpt were some kind of market maker connecting consumers with products, like a google with organic content or ads, or linkedin or producthunt. No, what actually happened is that OP developed a type of chatgpt integration,…

> the feature in OP isn't something that people would have found useful

This doesn’t seem likely. The utility is pretty obvious.

> chatgpt could have just directed the user to the site and told them to upload that image to OP's site.

What image? Did you think the first image shown is what is being entered into ChatGPT? It’s not. That’s what the site expects to be uploaded to them. There’s no indication that the ChatGPT users are scanning tabs. ChatGPT is producing ASCII tabs, but we aren’t shown what input it is in response to.

Post reply on HN