Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons after a Half-billion GPT Tokens

kenkantzer.com

Post reply on HN
Print view View original

51–60 of 179 posts

Re: Lessons after a Half-billion GPT Tokens

#51

Post by jstummbillig »

larodi wrote: post #40

Agree largely with author, but this ‘wait for OpenAI to do it’ sentiment is not something valid. Opus for example is already much better (not only per my experience, but like… researchers evaluaiton). And even for the fun of it - try some local inference, boy. If u know how to prompt it you definitely would be able to run local for the same tasks. Like listening to my students all going to ‘call some API’ for their p…

It sounds like you are a tech educator, which potentially sound like a lot of fun with llms right now.

When you are integrating these things into your business, you are looking for different things. Most of our customers would for example not find it very cool to have a service outage because somebody wanted to not kill all the joy.

Re: Lessons after a Half-billion GPT Tokens

#52

Post by jstummbillig »

satisfice wrote: post #49

I keep seeing this pattern in articles like this: 1. A recitation of terrible problems 2. A declaration of general satisfaction. Clearly and obviously, ChatGPT is an unreliable toy. The author seems pleased with it. As an engineer, I find that unacceptable.

ChatGPT is probably in the top 5 value/money subscriptions I have ever had (and that includes utilities).

The relatively low price point certainly plays a role here, but it's certainly not a mainly recreational thing for me. These thing's are kinda hard to measure but roughly most + is engagement with hard stuff goes up, and rate of learning goes up, by a lot.

Re: Lessons after a Half-billion GPT Tokens

#53

Post by gnat »

jsemrau wrote: post #38
sroussey wrote: post #6

Earlier quoted context omitted.

If someone were to create something new, a blank slate approach, what would you find valuable and why?

Use a local model. For most tasks they are good enough. Let's say Mistral 0.2 instruct is quite solid by now.

Do different versions react to prompts in the same way? I imagined the prompt would be tailored to the quirks of a particular version rather than naturally being stably optimal across versions.

Re: Lessons after a Half-billion GPT Tokens

#54

Post by Civitello »

> Every use case we have is essentially “Here’s a block of text, extract something from it.” As a rule, if you ask GPT to give you the names of companies mentioned in a block of text, it will not give you a random company (unless there are no companies in the text – there’s that null hypothesis problem!). Make it two steps, first: > Does this block of text mention a company? If no, good you've got your null result. If yes: > Please list the names of companies in this block of text.

Re: Lessons after a Half-billion GPT Tokens

#55

Post by ctxc »

msp26 wrote: post #45
7thpower wrote: post #42

Earlier quoted context omitted.

Can you elaborate? I am currently beating my head against this. If I give GPT4 a list of existing items with a defined structure, and it is just having to convert schema or something like that to JSON, it can do that all day long. But if it has to do any sort of reasoning and basically create its own list, it only gives me a very limited subset. I have similar issues with other LLMs. Very interested in how you are ap…

If you show your task/prompt with an example I'll see if I can fix it and explain my steps. Are you using the function calling/tool use API?

Hi! My work is similar and I'd love to have someone to bounce ideas off of if you don't mind.

Your profile doesn't have contact info though. Mine does, please send me a message. :)

Re: Lessons after a Half-billion GPT Tokens

#56

Post by jsemrau »

gnat wrote: post #53
jsemrau wrote: post #38

Earlier quoted context omitted.

Use a local model. For most tasks they are good enough. Let's say Mistral 0.2 instruct is quite solid by now.

Do different versions react to prompts in the same way? I imagined the prompt would be tailored to the quirks of a particular version rather than naturally being stably optimal across versions.

I suppose that is one of the benefits of using a local model, that it reduces model risk. I.e., given a certain prompt, it should always reply in the same way. Using a hosted model, operationally you don't have that control over model risk.

Re: Lessons after a Half-billion GPT Tokens

#58

Post by lordofmoria »

dougb5 wrote: post #43

The lessons I wanted from this article weren't in there: Did all of that expenditure actually help their product in a measurable way? Did customers use and appreciate the new features based on LLM summarization compared to whatever they were using before? I presume it's a net win or they wouldn't continue to use it, but more specifics around the application would be helpful.

Hey, OP here!

The answer is a bit boring: the expenditure definitely has helped customers - in that, they're using AI generated responses in all their work flows all the time in the app, and barely notice it.

See what I did there? :) I'm mostly serious though - one weird thing about our app is that you might not even know we're using AI, unless we literally tell you in the app.

And I think that's where we're at with AI and LLMs these days, at least for our use case.

You might find this other post I just put up to have more details too, related to how/where I see the primary value: https://kenkantzer.com/gpt-is-the-heroku-of-ai/

Re: Lessons after a Half-billion GPT Tokens

#59

Post by andenacitelli »

waldrews wrote: post #48
msp26 wrote: post #36

> But the problem is even worse – we often ask GPT to give us back a list of JSON objects. Nothing complicated mind you: think, an array list of json tasks, where each task has a name and a label. > GPT really cannot give back more than 10 items. Trying to have it give you back 15 items? Maybe it does it 15% of the time. This is just a prompt issue. I've had it reliably return up to 200 items in correct order. The tr…

I've been telling it the user is from a culture where answering questions with incomplete list is offensive and insulting.

This is absolutely hilarious. Prompt engineering is such a mixed bag of crazy stuff that actually works. Reminds me of how they respond better if you put them under some kind of pressure (respond better, or else…).

I haven’t looked at the prompts we run in prod at $DAYJOB for a while but I think we have at least five or ten things that are REALLY weird out of context.

Re: Lessons after a Half-billion GPT Tokens

#60

Post by sungho_ »

I'm curious if the OP has tried any of the libraries that control the output of LLM (LMQL, Outliner, Guadiance, ...), and for those who have: do you find them as unnecessary as LangChain? In particular, the OP's post mentions the problem of not being able to generate JSON with more than 15 items, which seems like a problem that can be solved by controlling the output of LLM. Is that correct?
Post reply on HN