71–80 of 179 posts
> We always extract json. We don’t need JSON mode I wonder why? It seems to work pretty well for me. > Lesson 4: GPT is really bad at producing the null hypothesis Tell me about it! Just yesterday I was testing a prompt around text modification rules that ended with “If none of the rules apply to the text, return the original text without any changes”. Do you know ChatGPT’s response to a text where none of the rules…
The team I work on processes 5B+ tokens a month (and growing) and I'm the EM overseeing that. Here are my take aways 1. There are way too many premature abstractions. Langchain, as one of may examples, might be useful in the future but at the end of the day prompts are just a API call and it's easier to write standard code that treats LLM calls as a flaky API call rather than as a special thing. 2. Hallucinations are…
Asking for analogies has been interesting and surprisingly useful.
Agree largely with author, but this ‘wait for OpenAI to do it’ sentiment is not something valid. Opus for example is already much better (not only per my experience, but like… researchers evaluaiton). And even for the fun of it - try some local inference, boy. If u know how to prompt it you definitely would be able to run local for the same tasks. Like listening to my students all going to ‘call some API’ for their p…
It's much better at critical thinking tasks and prose.
Don't mistake benchmarks for real world performance across actual usecases. There's a bit of Goodhart's Law going on with LLM evaluation and optimization.
Do I need langchain if I want to analyze a large document of many pages?
I keep seeing this pattern in articles like this: 1. A recitation of terrible problems 2. A declaration of general satisfaction. Clearly and obviously, ChatGPT is an unreliable toy. The author seems pleased with it. As an engineer, I find that unacceptable.
That doesn't mean they can't be incredibly useful - but it does mean you have to approach them in a bit of a different way, and design software around them that takes their unreliability into account.
I don't want a crap intro or waffley summary but it just can't help itself.
Same here: I’m subscribed to all three top dogs in LLM space, and routinely issue the same prompts to all three. It’s very one sided in favor of GPT4 which is stunning since it’s now a year old, although of course it received a couple of updates in that time. Also at least with my usage patterns hallucinations are rare, too. In comparison Claude will quite readily hallucinate plausible looking APIs that don’t exist w…
Earlier quoted context omitted.
How are you sending tabular data in a reliable way. And what is the source document type? I'm trying to solve this for complex financial-related tables in PDFs right now.
Amazon Textract, to get tables, format them with Python as csv then send to your preferred AI model.