Lessons after a Half-billion GPT Tokens • Hacker News Board

Re: Lessons after a Half-billion GPT Tokens

#141

Post by xiphias2 » Sun, Apr 14, 2024, 9:38 AM UTC

checkyoursudo wrote: post #94

thisgoesnowhere wrote: post #11

The team I work on processes 5B+ tokens a month (and growing) and I'm the EM overseeing that. Here are my take aways 1. There are way too many premature abstractions. Langchain, as one of may examples, might be useful in the future but at the end of the day prompts are just a API call and it's easier to write standard code that treats LLM calls as a flaky API call rather than as a special thing. 2. Hallucinations are…

> get it to say when it's not sure This is a function of the language model itself. By the time you get to the output, the uncertainty that is inherent in the computation is lost to the prediction. It is like if you ask me to guess heads or tails, and I guess heads, I could have stated my uncertainty (e.g. Pr [H] = .5) before hand, but in my actual prediction of heads, and then the coin flip, that uncertainty is lost…

> so unless the prediction itself is uncertainty (which it should rarely be based on the training corpus, I think)

Why shouldn't you ask for uncertainaty?

I love asking for scores / probabilities (usually give a range, like 0.0 to 1.0) whenever I ask for a list, and it makes the output much more usable

Re: Lessons after a Half-billion GPT Tokens

#142

Post by az226 » Sun, Apr 14, 2024, 9:51 AM UTC

squigz wrote: post #122

egonschiele wrote: post #103

I have a personal writing app that uses the OpenAI models and this post is bang on. One of my learnings related to "Lesson 1: When it comes to prompts, less is more": I was trying to build an intelligent search feature for my notes and asking ChatGPT to return structured JSON data. For example, I wanted to ask "give me all my notes that mention Haskell in the last 2 years that are marked as draft", and let Chat GPT f…

This seems like something that would be better suited by a database and good search filters rather than an LLM...

Something something about everything looking like a nail when you’re holding a hammer

Re: Lessons after a Half-billion GPT Tokens

#143

Post by sebastiennight » Sun, Apr 14, 2024, 11:05 AM UTC

thibaut_barrere wrote: post #138

sebastiennight wrote: post #100

Earlier quoted context omitted.

There are a few improvements I'd suggest with that prompt if you want to maximise its performance. 1. You're really asking for hallucinations here. Asking for factual data is very unreliable, and not what these models are strong at. I'm curious how close/far the results are from ground truth. I would definitely bet that outside of the top 5, numbers would be wobbly and outside of top... 25?, even the ranking would be…

Thanks for the suggestions, appreciated! For some context, this snippet is just an educational demo to show what can be done with regard to structured output & data types validation. Re 1: for more advanced cases (using the exact same stack), I am using ensemble techniques & automated comparisons to double-check, and so far this has really well protected the app from hallucinations. I am definitely careful with this…

Ah, I understand your use case better! If you're teaching students this stuff, I'm in awe. I would expect it would take several years at many institutions before these tools became part of the curriculum.

Re: Lessons after a Half-billion GPT Tokens

#144

Post by thibaut_barrere » Sun, Apr 14, 2024, 11:09 AM UTC

sebastiennight wrote: post #143

thibaut_barrere wrote: post #138

Earlier quoted context omitted.

Thanks for the suggestions, appreciated! For some context, this snippet is just an educational demo to show what can be done with regard to structured output & data types validation. Re 1: for more advanced cases (using the exact same stack), I am using ensemble techniques & automated comparisons to double-check, and so far this has really well protected the app from hallucinations. I am definitely careful with this…

Ah, I understand your use case better! If you're teaching students this stuff, I'm in awe. I would expect it would take several years at many institutions before these tools became part of the curriculum.

I am not directly a professor (although I homeschool one of my sons for a number of tracks), but indeed this is one of my goals :-)

Re: Lessons after a Half-billion GPT Tokens

#145

Post by ShamelessC » Sun, Apr 14, 2024, 11:13 AM UTC

Kiro wrote: post #135

phillipcarter wrote: post #84

Earlier quoted context omitted.

> I wonder why? It seems to work pretty well for me. I read this as "what we do works just fine to not need to use JSON mode". We're in the same boat at my company. Been live for a year now, no need to switch. Our prompt is effective at getting GPT-3.5 to always produce JSON.

There's nothing to switch to. You just enable it. No need to change the prompt or anything else. All it requires is that you mention "JSON" in your prompt, which you obviously already do.

I think that’s only true when using ChatGPT via the web/app, not when used via API as they likely are. Happy to be corrected however.

Re: Lessons after a Half-billion GPT Tokens

#146

Post by littlestymaar » Sun, Apr 14, 2024, 11:15 AM UTC

> One part of our pipeline reads some block of text and asks GPT to classify it as relating to one of the 50 US states, or the Federal government.

Using a multi-billion tokens like GPT-4 for such a trivial classification task[1] is an insane overkill. And in an era where ChatGPT exists, and can in fact give you what you need to build a simpler classifier for the task, it shows how narrow minded most people are when AI is involved.

[1] to clarify, it's either trivial or impossible to do reliably depending on how fucked-up your input is

Re: Lessons after a Half-billion GPT Tokens

#147

Post by cheema33 » Sun, Apr 14, 2024, 11:22 AM UTC

ein0p wrote: post #33

Same here: I’m subscribed to all three top dogs in LLM space, and routinely issue the same prompts to all three. It’s very one sided in favor of GPT4 which is stunning since it’s now a year old, although of course it received a couple of updates in that time. Also at least with my usage patterns hallucinations are rare, too. In comparison Claude will quite readily hallucinate plausible looking APIs that don’t exist w…

> It’s very one sided in favor of GPT4

My experience has been the opposite. I subscribe to multiple services as well and copy/paste the same question to all. For my software dev related questions, Claude Opus is so far ahead that I am thinking that it no longer is necessary to use GPT4.

For code samples I request, GPT4 produced code fails to even compile many times. That almost never happens for Claude.

Re: Lessons after a Half-billion GPT Tokens

#148

Post by CuriouslyC » Sun, Apr 14, 2024, 11:54 AM UTC

azinman2 wrote: post #132

codewithcheese wrote: post #28

Earlier quoted context omitted.

Yeah also prompts should not be developed in abstract. Goal of a prompt is to activate the models internal respentations for it to best achieve the task. Without automated methods, this requires iteratively testing the models reaction to different input and trying to understand how it's interpreting the request and where it's falling down and then patching up those holes. Need to verify if it even knows what you mean…

Which automated method do you use?

The only public prompt optimizer that I'm aware of now is DSPy, but it doesn't optimize your main prompt request, just some of the problem solving strategies the LLM is instructed to use, and your few shot learning examples. I wouldn't be surprised if there's a public general prompt optimizing agent by this time next year though.

Re: Lessons after a Half-billion GPT Tokens

#149

Post by meowtimemania » Sun, Apr 14, 2024, 11:55 AM UTC

ein0p wrote: post #33

Same here: I’m subscribed to all three top dogs in LLM space, and routinely issue the same prompts to all three. It’s very one sided in favor of GPT4 which is stunning since it’s now a year old, although of course it received a couple of updates in that time. Also at least with my usage patterns hallucinations are rare, too. In comparison Claude will quite readily hallucinate plausible looking APIs that don’t exist w…

Have you tried Poe.com? You can access all the major llm’s with one subscription

Re: Lessons after a Half-billion GPT Tokens

#150

Post by chx » Sun, Apr 14, 2024, 11:58 AM UTC

satisfice wrote: post #49

I keep seeing this pattern in articles like this: 1. A recitation of terrible problems 2. A declaration of general satisfaction. Clearly and obviously, ChatGPT is an unreliable toy. The author seems pleased with it. As an engineer, I find that unacceptable.

https://hachyderm.io/@inthehands/112006855076082650

> You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to.

== End of toot.

The price you pay for this bullshit in energy when the sea temperature is literally off the charts and we do not know why makes it not worth it in my opinion.