Lessons after a Half-billion GPT Tokens • Hacker News Board

Re: Lessons after a Half-billion GPT Tokens

#61

Post by LASR » Sat, Apr 13, 2024, 8:08 PM UTC

sungho_ wrote: post #60

I'm curious if the OP has tried any of the libraries that control the output of LLM (LMQL, Outliner, Guadiance, ...), and for those who have: do you find them as unnecessary as LangChain? In particular, the OP's post mentions the problem of not being able to generate JSON with more than 15 items, which seems like a problem that can be solved by controlling the output of LLM. Is that correct?

If you want x number of items every time, ask it to include a sequence number in each output, it will consistently return x number of items.

Numbered bullets work well for this, if you don’t need JSON. With JSON, you can ask it to include an ‘id’ in each item.

Re: Lessons after a Half-billion GPT Tokens

#62

Post by orbatos » Sat, Apr 13, 2024, 8:15 PM UTC

Statements like this tell me your analysis is poisoned by misunderstandings: "Why is this crazy? Well, it’s crazy that GPT’s quality and generalization can improve when you’re more vague – this is a quintessential marker of higher-order delegation / thinking." No, there is no "higher-order thought" happening, or any at all actually. That's not how these models work.

Re: Lessons after a Half-billion GPT Tokens

#63

Post by Xenoamorphous » Sat, Apr 13, 2024, 8:24 PM UTC

> We always extract json. We don’t need JSON mode

I wonder why? It seems to work pretty well for me.

> Lesson 4: GPT is really bad at producing the null hypothesis

Tell me about it! Just yesterday I was testing a prompt around text modification rules that ended with “If none of the rules apply to the text, return the original text without any changes”.

Do you know ChatGPT’s response to a text where none of the rules applied?

“The original text without any changes”. Yes, the literal string.

Re: Lessons after a Half-billion GPT Tokens

#64

Post by cpursley » Sat, Apr 13, 2024, 8:24 PM UTC

jsemrau wrote: post #38

sroussey wrote: post #6

Earlier quoted context omitted.

If someone were to create something new, a blank slate approach, what would you find valuable and why?

Use a local model. For most tasks they are good enough. Let's say Mistral 0.2 instruct is quite solid by now.

What are the best local/open models for accurate tool-calling?

Re: Lessons after a Half-billion GPT Tokens

#65

Post by cpursley » Sat, Apr 13, 2024, 8:28 PM UTC

Oras wrote: post #24

eigenvalue wrote: post #12

I agree with most of it, but definitely not the part about Claude3 being “meh.” Claude3 Opus is an amazing model and is extremely good at coding in Python. The ability to handle massive context has made it mostly replace GPT4 for me day to day. Sounds like everyone eventually concludes that Langchain is bloated and useless and creates way more problems than it solves. I don’t get the hype.

+1 for Claude Opus, it had been my go to for the last 3 weeks compared to GPT4. The generated texts are much better than GPT4 when it comes to follow the prompt. I also tried the API for some financial analysis of large tables, the response time was around 2 minutes, still did it really well and timeout errors were around 1 to 2% only.

How are you sending tabular data in a reliable way. And what is the source document type? I'm trying to solve this for complex financial-related tables in PDFs right now.

Re: Lessons after a Half-billion GPT Tokens

#66

Post by Oras » Sat, Apr 13, 2024, 8:39 PM UTC

cpursley wrote: post #65

Oras wrote: post #24

Earlier quoted context omitted.

+1 for Claude Opus, it had been my go to for the last 3 weeks compared to GPT4. The generated texts are much better than GPT4 when it comes to follow the prompt. I also tried the API for some financial analysis of large tables, the response time was around 2 minutes, still did it really well and timeout errors were around 1 to 2% only.

How are you sending tabular data in a reliable way. And what is the source document type? I'm trying to solve this for complex financial-related tables in PDFs right now.

Amazon Textract, to get tables, format them with Python as csv then send to your preferred AI model.

Re: Lessons after a Half-billion GPT Tokens

#67

Post by sroussey » Sat, Apr 13, 2024, 8:49 PM UTC

lordofmoria wrote: post #8

sroussey wrote: post #6

Earlier quoted context omitted.

If someone were to create something new, a blank slate approach, what would you find valuable and why?

This is a great question! I think we now know, collectively, a lot more about what’s annoying/hard about building LLM features than we did when LangChain was being furiously developed. And some things we thought would be important and not-easy, turned out to be very easy: like getting GPT to give back well-formed JSON. So I think there’s lots of room. One thing LangChain is doing now that solves something that IS ver…

How would testing work out ideally?

Re: Lessons after a Half-billion GPT Tokens

#68

Post by mind-blight » Sat, Apr 13, 2024, 8:50 PM UTC

gdiamos wrote: post #21

mind-blight wrote: post #4

Earlier quoted context omitted.

I suspect the biggest difference is the input data. Embeddings are great over datasets that look like FAQs and QA docs, or data that conceptually fits into very small chunks (tweets, some product reviews, etc). It does very badly over diverse business docs, especially with naive chunking. B2B use cases usually have old PDFs and word docs that need to be searched, and they're often looking for specific keywords (e.g.…

rare words are out of vocab errors in vectors Especially if they aren’t in the token vocab

Even worse, named entities vary from organization to organization.

We have a client who uses a product called "Time". It's software time management. For that customer's documentation, time should be close to "product" and a bunch of other things that have nothing to do with the normal concept of time.

I actually suspect that people would get a lot more bang for their buck fine tuning the embedding models on B2B datasets for their use case, rather than fine tuning an llm

Re: Lessons after a Half-billion GPT Tokens

#69

Post by bongodongobob » Sat, Apr 13, 2024, 8:57 PM UTC

FranklinMaillot wrote: post #50

In my limited experience, I came to the same conclusion regarding simple prompt being more efficient than very detailed list of instructions. But if you look at OpenAI's system prompt for GPT4, it's an endless set of instructions with DOs and DONTs so I'm confused. Surely they must know something about prompting their model.

That's for chatting and interfacing conversationally with a human. Using the API is a completely different ballgame because it's not meant to be a back and forth conversation with a human.

Re: Lessons after a Half-billion GPT Tokens

#70

Post by kromem » Sat, Apr 13, 2024, 9:00 PM UTC

Tip for your 'null' problem:

LLMs are set up to output tokens. Not to not output tokens.

So instead of "don't return anything" have the lack of results "return the default value of XYZ" and then just do a text search on the result for that default value (i.e. XYZ) the same way you do the text search for the state names.

Also, system prompts can be very useful. It's basically your opportunity to have the LLM roleplay as X. I wish they'd let the system prompt be passed directly, but it's still better than nothing.