Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing ChatGPT and Whisper APIs

openai.com

Post reply on HN
Print view View original

471–480 of 696 posts

Re: Introducing ChatGPT and Whisper APIs

#471

Post by shostack »

dmw_ng wrote: post #116
rburhum wrote: post #85

Can somebody please clarify. Is the cost $0.002 per 1k tokens generated , read , or both?

Both, the API response includes a breakdown. In the best case 1 token = 1 word (for example "and", "the", etc). Depending on input, for English it seems reasonable to multiply the word count by about 1.3 to get a rough token count This pricing model seems fair since you can pass in huge prompts and request a single word reply, or a few words that expect a large reply

Is there anything akin to creating Stable Diffusion embeddings where it can train a very discrete concept that takes up a few kilobytes and use that with the base model?

Such an approach could in theory make it so you spend a little upfront to train more complex (read: concepts costing many tokens) and can subsequently reuse it cheaply because you're using an embedding of the vectors for that complex concept instead which may only take a single token.

Re: Introducing ChatGPT and Whisper APIs

#472

Post by vishal0123 »

kkielhofner wrote: post #430
ALittleLight wrote: post #377

Earlier quoted context omitted.

What do you mean one concurrent request? Can't you have a huge batch size to basically support a huge number of concurrent requests? e.g. Endpoint feeds a queue, queue fills a batch, batched results generate replies. You are simultaneously fulfilling many requests.

Hopefully they’re doing plenty of batching - you don’t even need to roll your own as you’re describing. Inference servers like Triton will dynamically batch requests with SLA params for max response time (for example). That said I don’t think anyone anyone outside of OpenAI knows what’s going on operationally. Same goes for VRAM usage, potential batch sizes, etc. This is all wild speculation. Same goes for whatever t…

> will set you back $100k/yr

This is 6 month of salary of one average developer's salary there. And BTW they are likely doing inference on 100s or 1000s of GPUs, not just 8.

Re: Introducing ChatGPT and Whisper APIs

#473

Post by panarky »

freeqaz wrote: post #236
vishal0123 wrote: post #82

Earlier quoted context omitted.

> I have no idea how OpenAI can make money on this. I did some quick calculation. We know the number of floating point operations per token for inference is approximately twice the number of parameters(175B). Assuming they use 16 bit floating point, and have 50% of peak efficiency, A100 could do 300 trillion flop/s(peak 624[0]). 1 hour of A100 gives openAI $0.002/ktok * (300,000/175/2/1000)ktok/sec * 3600=$6.1 back.…

It's also worth mentioning that, because Microsoft is an investor, they're likely getting these at cost or subsidized. OpenAI doesn't have to make money right away. They can lose a small bit of money per API request in exchange for market share (preventing others from disrupting them). As the cost of GPUs goes down, or they develop at ASIC or more efficient model, they can keep their pricing the same and then make mo…

Microsoft isn't using Nvidia A100s for inference are they? Seems like they'd use their Project Brainwave custom FPGAs.

Re: Introducing ChatGPT and Whisper APIs

#474

Post by dragonwriter »

jejeyyy77 wrote: post #467
shamino wrote: post #76

Earlier quoted context omitted.

exactly. this isn't a leetcode problem where all you have to do is re-run the function, or do it iteratively vs recursively.

Not sure what you mean, but for example, 2 separate competitors to DALL-E was released within months (SD and MJ). Arguable that both of these have since surpassed DALL-E's capabilities/ecosystem. Not sure why ChatGPT will be any different.

> Not sure why ChatGPT will be any different.

LLMs take vastly more resources to train and run than image generators. You can do quite a bit with SD on a few year old 4GB laptop GPU (that’s what I use mostly, though I’ve set up an instance with a better GPU on Compute Engine that I can fire up, too.)

GPT-NeoX-20B – an open (as in Open Source, not OpenAI) LLM intended as a start to move toward competing with GPT-3 (but still well behind, and smaller) requires a minimum 42GB of VRAM and 40GB system RAM to run for inference. The resources times time cost for training LLMs is…immense. The hardware cost alone of trying to catch up to ChatGPT is enormous, and unless a radical new approach that provides good results and insanely lower resource requirements is found, you aren’t going to have an SD-like community pushing things forward.

Will there be competition for ChatGPT? Yes, probably, but don’t expect it to look like the competition for Dall-E.

Re: Introducing ChatGPT and Whisper APIs

#475

Post by Hyzer »

juliennakache wrote: post #413

Does anyone have a sense of how the Shopify integration work under the hood? Specifically, how were they able to have Chat-GPT makes a recommendation from their product database? Given the model cannot be fine-tuned and there is a 4000 token context limit, I'm guessing the only thing really possible is to have chat gpt extract a few keyword and generate a query in their internal product search and return the first on…

Probably the embeddings API. This guide is what helped me understand the concept https://github.com/openai/openai-cookbook/blob/main/examples...

tl;dr is that you can pre-process each chunk of your database and use embeddings to quickly look up which chunk is most similar to the user's query, and then prepend that chunk to the user's query before giving it to GPT, so that GPT has the relevant context to give an answer.

Re: Introducing ChatGPT and Whisper APIs

#476

Post by astrange »

gremlinsinc wrote: post #213
Cantinflas wrote: post #53

Earlier quoted context omitted.

It's amazing that "being two months out of date" in AI means that you are already a dinosaur

That's what the singularity is all about, a moment in time when 2 seconds late turns you into a dinosaur, be greatful it's 2 months, not 2 weeks, 2 days, or 2 minutes.

The fun part of an S-curve looks exponential when you're on it.

Re: Introducing ChatGPT and Whisper APIs

#477

Post by astrange »

lordnacho wrote: post #351
eternalban wrote: post #324

Earlier quoted context omitted.

I just watched a video that convincingly showed that it is energy and energy alone that determines the production growth of humanity. Until the day AI can "generate" stuff (you know, something out of nothing) it can only at best streamline existing production, which is entirely capped by energy limits. We may drown in oceans of audio, video, novels, poems, films, porn, blue prints, chemical formulas, etc. dreamed up…

Couldn't the AI invent fusion?

It could imagine fusion really well, but inventing working fusion doesn't appear to rely on "intelligence". "Intelligence" ain't "omnipotence".

Re: Introducing ChatGPT and Whisper APIs

#478

Post by astrange »

Baeocystin wrote: post #254
anonylizard wrote: post #42

Earlier quoted context omitted.

You are like 2 months out of date. Stable diffusion now has a massive ecosystem around it (civitai/automatic1111), that when used well, completely crushes any competitors in terms of the images it produces. Midjourney is still competitive, but mostly because its easier to use. Dalle2 will get you laughed out of the room in any ai art discussion.

I still think that Midjourney is hamstringing themselves by being Discord-only. And their keyword nannying is pretty bad. It a testament too their overall quality that they're still as popular as they are are, but I really don't think they are doing themselves any favors, especially as the Stable Diffusion ecosystem continues to grow.

Only having a Discord interface is a cheap way of having someone else take care of abuse/CSAM issues for you.

Re: Introducing ChatGPT and Whisper APIs

#479

Post by astrange »

OkGoDoIt wrote: post #124
anonylizard wrote: post #42

Earlier quoted context omitted.

You are like 2 months out of date. Stable diffusion now has a massive ecosystem around it (civitai/automatic1111), that when used well, completely crushes any competitors in terms of the images it produces. Midjourney is still competitive, but mostly because its easier to use. Dalle2 will get you laughed out of the room in any ai art discussion.

Do you have any recently updated examples, blog posts, whatever showing that DALLE is worse than modern stable diffusion? I was still under the impression that DALLE was better (with better meaning the images are more likely to be what you asked for, more lifelike, more realistic, not necessarily artistically pleasing), with the downside of it being locked away and somewhat expensive. And my understanding is that sta…

DALLE2 is underpowered and has never improved since they released it. The actual quality of the images is very low (literally in the sense of they have lots of artifacts) because they saved CPU time by not running enough diffusion passes.

Re: Introducing ChatGPT and Whisper APIs

#480

Post by astrange »

soulofmischief wrote: post #285
jeron wrote: post #271

Earlier quoted context omitted.

>Dalle2 will get you laughed out of the room in any ai art discussion. and claiming AI art is art would get you laughed out of any art discussion. personally I think AI art is really cool, but to discount what Dalle 2 did for AI art is unfair.

My company has a team of AI-enpowered artists who would overwhelmingly disagree with you on the premise that AI art is not art. Maybe you're the only one doing the laughing.

A lot of online "artists" are mad about it. Generally not professionals who actually need productivity, but semipros who live off one-off commissions or else people who are just generally mad at tech bros.
Post reply on HN