Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons after a Half-billion GPT Tokens

kenkantzer.com

Post reply on HN
Print view View original

41–50 of 179 posts

Re: Lessons after a Half-billion GPT Tokens

#41

Post by ein0p »

CharlesW wrote: post #35
ein0p wrote: post #33

Same here: I’m subscribed to all three top dogs in LLM space, and routinely issue the same prompts to all three. It’s very one sided in favor of GPT4 which is stunning since it’s now a year old, although of course it received a couple of updates in that time. Also at least with my usage patterns hallucinations are rare, too. In comparison Claude will quite readily hallucinate plausible looking APIs that don’t exist w…

This was with Claude Opus, vs. one of the lesser variants? I really like Opus for English copy generation.

Opus, yes, the $20/mo version. I usually don’t generate copy. My use cases are code (both “serious” and “the nice to have code I wouldn’t bother writing otherwise”), learning how to do stuff in unfamiliar domains, and just learning unfamiliar things in general. It works well as a very patient teacher, especially if you already have some degree of familiarity with the problem domain. I do have to check it against primary sources, which is how I know the percentage of hallucinations is very low. For code, however I don’t even have to do that, since as a professional software engineer I am the “primary source”.

Re: Lessons after a Half-billion GPT Tokens

#42

Post by 7thpower »

msp26 wrote: post #36

> But the problem is even worse – we often ask GPT to give us back a list of JSON objects. Nothing complicated mind you: think, an array list of json tasks, where each task has a name and a label. > GPT really cannot give back more than 10 items. Trying to have it give you back 15 items? Maybe it does it 15% of the time. This is just a prompt issue. I've had it reliably return up to 200 items in correct order. The tr…

Can you elaborate? I am currently beating my head against this.

If I give GPT4 a list of existing items with a defined structure, and it is just having to convert schema or something like that to JSON, it can do that all day long. But if it has to do any sort of reasoning and basically create its own list, it only gives me a very limited subset.

I have similar issues with other LLMs.

Very interested in how you are approaching this.

Re: Lessons after a Half-billion GPT Tokens

#43

Post by dougb5 »

The lessons I wanted from this article weren't in there: Did all of that expenditure actually help their product in a measurable way? Did customers use and appreciate the new features based on LLM summarization compared to whatever they were using before? I presume it's a net win or they wouldn't continue to use it, but more specifics around the application would be helpful.

Re: Lessons after a Half-billion GPT Tokens

#44

Post by pbhjpbhj »

_pdp_ wrote: post #17

The biggest realisation for me while making ChatBotKit has been that UX > Model alone. For me, the current state of AI is not about questions and answers. This is dumb. The presentation matters. This is why we are now investing in generative UI.

Generative UI being creation of a specific UI dependent on an obedience from your model? What model is it?

Google Gemini were showing something that I'd call 'adapted output UI' in their launch presentation. Is that close to what you're doing in any way?

Re: Lessons after a Half-billion GPT Tokens

#45

Post by msp26 »

7thpower wrote: post #42
msp26 wrote: post #36

> But the problem is even worse – we often ask GPT to give us back a list of JSON objects. Nothing complicated mind you: think, an array list of json tasks, where each task has a name and a label. > GPT really cannot give back more than 10 items. Trying to have it give you back 15 items? Maybe it does it 15% of the time. This is just a prompt issue. I've had it reliably return up to 200 items in correct order. The tr…

Can you elaborate? I am currently beating my head against this. If I give GPT4 a list of existing items with a defined structure, and it is just having to convert schema or something like that to JSON, it can do that all day long. But if it has to do any sort of reasoning and basically create its own list, it only gives me a very limited subset. I have similar issues with other LLMs. Very interested in how you are ap…

If you show your task/prompt with an example I'll see if I can fix it and explain my steps.

Are you using the function calling/tool use API?

Re: Lessons after a Half-billion GPT Tokens

#46

Post by thibaut_barrere »

7thpower wrote: post #42
msp26 wrote: post #36

> But the problem is even worse – we often ask GPT to give us back a list of JSON objects. Nothing complicated mind you: think, an array list of json tasks, where each task has a name and a label. > GPT really cannot give back more than 10 items. Trying to have it give you back 15 items? Maybe it does it 15% of the time. This is just a prompt issue. I've had it reliably return up to 200 items in correct order. The tr…

Can you elaborate? I am currently beating my head against this. If I give GPT4 a list of existing items with a defined structure, and it is just having to convert schema or something like that to JSON, it can do that all day long. But if it has to do any sort of reasoning and basically create its own list, it only gives me a very limited subset. I have similar issues with other LLMs. Very interested in how you are ap…

Not sure if that fits the bill, but here is an example with 200 sorted items based on a question (example with Elixir & InstructorEx):

https://gist.github.com/thbar/a53123cbe7765219c1eca77e03e675...

Re: Lessons after a Half-billion GPT Tokens

#47

Post by haolez »

That has been my experience too. The null hypothesis explains almost all of my hallucinations.

I just don't agree with the Claude assessment. In my experience, Claude 3 Opus is vastly superior to GPT-4. Maybe the author was comparing with Claude 2? (And I've never tested Gemini)

Re: Lessons after a Half-billion GPT Tokens

#48

Post by waldrews »

msp26 wrote: post #36

> But the problem is even worse – we often ask GPT to give us back a list of JSON objects. Nothing complicated mind you: think, an array list of json tasks, where each task has a name and a label. > GPT really cannot give back more than 10 items. Trying to have it give you back 15 items? Maybe it does it 15% of the time. This is just a prompt issue. I've had it reliably return up to 200 items in correct order. The tr…

I've been telling it the user is from a culture where answering questions with incomplete list is offensive and insulting.

Re: Lessons after a Half-billion GPT Tokens

#49

Post by satisfice »

I keep seeing this pattern in articles like this:

1. A recitation of terrible problems 2. A declaration of general satisfaction.

Clearly and obviously, ChatGPT is an unreliable toy. The author seems pleased with it. As an engineer, I find that unacceptable.

Re: Lessons after a Half-billion GPT Tokens

#50

Post by FranklinMaillot »

In my limited experience, I came to the same conclusion regarding simple prompt being more efficient than very detailed list of instructions. But if you look at OpenAI's system prompt for GPT4, it's an endless set of instructions with DOs and DONTs so I'm confused. Surely they must know something about prompting their model.
Post reply on HN