Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons after a Half-billion GPT Tokens

kenkantzer.com

Post reply on HN
Print view View original

81–90 of 179 posts

Re: Lessons after a Half-billion GPT Tokens

#81

Post by Oras »

cpursley wrote: post #80
Oras wrote: post #66

Earlier quoted context omitted.

Amazon Textract, to get tables, format them with Python as csv then send to your preferred AI model.

Thanks. How does Textract compare to come of the common cli utilities like pdftotext, tesseract, etc (if you made a comparison)?

I did, none of the open source parser worked well with tables. I had the following issues:

- missing cells. - partial identification for number (ex: £43.54, the parser would pick it up as £43).

What I did to compare is drawing lines around identified text to visualize the accuracy. You can do that with tesseract.

Re: Lessons after a Half-billion GPT Tokens

#82

Post by robocat »

Me1000 wrote: post #78
ein0p wrote: post #33

Same here: I’m subscribed to all three top dogs in LLM space, and routinely issue the same prompts to all three. It’s very one sided in favor of GPT4 which is stunning since it’s now a year old, although of course it received a couple of updates in that time. Also at least with my usage patterns hallucinations are rare, too. In comparison Claude will quite readily hallucinate plausible looking APIs that don’t exist w…

Interesting, Claude 3 Opus has been better than GPT4 for me. Mostly in that I find it does a better (and more importantly, more thorough) job of explaining things to me. For coding tasks (I'm not asking it to write code, but instead to explain topics/code/etc to me) I've found it tends to give much more nuanced answers. When I give it long text to converse about, I find Claude Opus tends to have a much deeper underst…

How much of this is just that one model responds better to the way you write prompts?

Much like you working with Bob and opining that Bob is great, and me saying that I find Jack easier to work with.

Re: Lessons after a Half-billion GPT Tokens

#83

Post by motoxpro »

thisgoesnowhere wrote: post #11

The team I work on processes 5B+ tokens a month (and growing) and I'm the EM overseeing that. Here are my take aways 1. There are way too many premature abstractions. Langchain, as one of may examples, might be useful in the future but at the end of the day prompts are just a API call and it's easier to write standard code that treats LLM calls as a flaky API call rather than as a special thing. 2. Hallucinations are…

Devops is such an amazing analogy.

Re: Lessons after a Half-billion GPT Tokens

#84

Post by phillipcarter »

Xenoamorphous wrote: post #63

> We always extract json. We don’t need JSON mode I wonder why? It seems to work pretty well for me. > Lesson 4: GPT is really bad at producing the null hypothesis Tell me about it! Just yesterday I was testing a prompt around text modification rules that ended with “If none of the rules apply to the text, return the original text without any changes”. Do you know ChatGPT’s response to a text where none of the rules…

> I wonder why? It seems to work pretty well for me.

I read this as "what we do works just fine to not need to use JSON mode". We're in the same boat at my company. Been live for a year now, no need to switch. Our prompt is effective at getting GPT-3.5 to always produce JSON.

Re: Lessons after a Half-billion GPT Tokens

#85

Post by lordofmoria »

thisgoesnowhere wrote: post #11

The team I work on processes 5B+ tokens a month (and growing) and I'm the EM overseeing that. Here are my take aways 1. There are way too many premature abstractions. Langchain, as one of may examples, might be useful in the future but at the end of the day prompts are just a API call and it's easier to write standard code that treats LLM calls as a flaky API call rather than as a special thing. 2. Hallucinations are…

OP here - I had never thought of the analogy to DevOps before, that made something click for me, and I wrote a post just now riffing off this notion: https://kenkantzer.com/gpt-is-the-heroku-of-ai

Basically, I think we’re using GPT as the PaaS/heroku/render equivalent of AI ops.

Thank you for the insight!!

Re: Lessons after a Half-billion GPT Tokens

#86

Post by ryoshu »

thisgoesnowhere wrote: post #11

The team I work on processes 5B+ tokens a month (and growing) and I'm the EM overseeing that. Here are my take aways 1. There are way too many premature abstractions. Langchain, as one of may examples, might be useful in the future but at the end of the day prompts are just a API call and it's easier to write standard code that treats LLM calls as a flaky API call rather than as a special thing. 2. Hallucinations are…

> But all in all I think it's more of a force multiplier rather than a breakthrough like the internet.

Thank you. Seeing similar things. Clients are also seeing sticker shock on how much the big models cost vs. the output. That will all come down over time.

Re: Lessons after a Half-billion GPT Tokens

#87

Post by larodi »

jstummbillig wrote: post #51
larodi wrote: post #40

Agree largely with author, but this ‘wait for OpenAI to do it’ sentiment is not something valid. Opus for example is already much better (not only per my experience, but like… researchers evaluaiton). And even for the fun of it - try some local inference, boy. If u know how to prompt it you definitely would be able to run local for the same tasks. Like listening to my students all going to ‘call some API’ for their p…

It sounds like you are a tech educator, which potentially sound like a lot of fun with llms right now. When you are integrating these things into your business, you are looking for different things. Most of our customers would for example not find it very cool to have a service outage because somebody wanted to not kill all the joy.

Sure, when availability and SLA kicks in…, but reselling APIs will only get you that far. Perhaps the whole pro/cons cloud argument can also kick in here, not going into it. We may well be on the same page, or we both perhaps have valid arguments. Your comment is appreciated indeed.

But then is the author (and are we) talking experience in reselling APIs or experience in introducing NNs in the pipeline? Not the same thing IMHO.

Agreed that OpenAI provides very good service, Gemini is not quite there yet, Groq (the LPUs) delivered a nice tech demo, Mixtral is cool but lacks in certain areas, and Claude can be lengthy.

But precisely because I’m not sticking with OAI I can then restate my view that if someone is so good with prompts he can get the same results locally if he knows what he’s doing.

Prompting OpenAI the right way can be similarly difficult.

Perhaps the whole idea of local inference only matters for IoT scenarios or whenever data is super sensitive (or CTO super stubborn to let it embed and fly). But then if you start from day 1 with WordPress provisioned for you ready to go in Google Cloud, you’d never understand the underlying details of the technology.

There sure also must be a good reason why Phind tuned their own thing to offer alongside GPT4 APIs.

Disclaimer: tech education is a side thing I do, indeed, and been doing in person for very long time, more than dozen topics, to allow myself to have opinion. Of course business is different matter and strategic decisions arr not the same. Even though I’d not advise anyone to blindly use APIs unless they appreciate the need properly.

Re: Lessons after a Half-billion GPT Tokens

#88

Post by cpursley »

Oras wrote: post #81
cpursley wrote: post #80

Earlier quoted context omitted.

Thanks. How does Textract compare to come of the common cli utilities like pdftotext, tesseract, etc (if you made a comparison)?

I did, none of the open source parser worked well with tables. I had the following issues: - missing cells. - partial identification for number (ex: £43.54, the parser would pick it up as £43). What I did to compare is drawing lines around identified text to visualize the accuracy. You can do that with tesseract.

Interesting. Did you try MS's offering (Azure AI Document Intelligence). Their pricing seems better than Amazon.

Re: Lessons after a Half-billion GPT Tokens

#89

Post by richardw »

robocat wrote: post #82
Me1000 wrote: post #78

Earlier quoted context omitted.

Interesting, Claude 3 Opus has been better than GPT4 for me. Mostly in that I find it does a better (and more importantly, more thorough) job of explaining things to me. For coding tasks (I'm not asking it to write code, but instead to explain topics/code/etc to me) I've found it tends to give much more nuanced answers. When I give it long text to converse about, I find Claude Opus tends to have a much deeper underst…

How much of this is just that one model responds better to the way you write prompts? Much like you working with Bob and opining that Bob is great, and me saying that I find Jack easier to work with.

The first job of an AI company is finding model/user fit.

Re: Lessons after a Half-billion GPT Tokens

#90

Post by CuriouslyC »

Xenoamorphous wrote: post #63

> We always extract json. We don’t need JSON mode I wonder why? It seems to work pretty well for me. > Lesson 4: GPT is really bad at producing the null hypothesis Tell me about it! Just yesterday I was testing a prompt around text modification rules that ended with “If none of the rules apply to the text, return the original text without any changes”. Do you know ChatGPT’s response to a text where none of the rules…

You know all the stories about the capricious djinn that grants cursed wishes based on the literal wording? That's what we have. Those of us who've been prompting models in image space for years now have gotten a handle on this but for people who got in because of LLMs, it can be a bit of a surprise.

One fun anecdote, a while back I was making an image of three women drinking wine in a fancy garden for a tarot card, and at the end of the prompt I had "lush vegetation" but that was enough to tip the women from classy to red nosed frat girls, because of the double meaning of lush.

Post reply on HN