Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI is too cheap to beat

generatingconversation.substack.com

Post reply on HN
Print view View original

331–340 of 426 posts

Re: OpenAI is too cheap to beat

#331

Post by mlsu »

Karrot_Kream wrote: post #326
mlsu wrote: post #221

Earlier quoted context omitted.

Gah, this is just not how it works. You are probably right that e.g. patient information, private conversations, proprietary code, etc would be safe with OpenAI. But it's not the on-prem team that needs to convince the rest of the organization to keep things on prem. Quite the opposite -- every single tech person would love to make our data someone else's problem (and get a big career boost from dealing with cloud te…

> because the people who actually give a fuck about the risk NEED to have granular detail of what data, readable by whom, is stored exactly where and for how long, and how can you make sure, and how do you know that access is scoped to the absolute minimum number of people, and is there a paper trail for that? You realize that Microsoft is a publicly traded company that has multiple privacy certifications? They have…

Absolutely. I am well aware of this, as are most tech people, that's what I'm saying. It's not us that are trying to convince our orgs to build a rack in the basement "to be more secure" just because we want to hear the fans running and see the lights blinking.

It's the lawyers that you need to convince. Good luck convincing any bigco lawyer that your company's data is safe on openAI because their legal agreement says "we don't train on API calls."

Re: OpenAI is too cheap to beat

#332

Post by wrsh07 »

cyode wrote: post #99
serjester wrote: post #55

I think this is under appreciated. I run a "talk-to-your-files" website with 5ish K MRR and a pretty generous free tier. My OpenAI costs have not exceeded $200 / mo. People talk about using smaller, cheaper models but unless you have strong data security requirements you're burdening yourself with serious maintenance work and using objectively worse models to save pennies. This doesn't even consider OpenAI continuous…

Are any OpenAI powered flows available to public, logged-out user traffic? I’ve worried (maybe irrationally) about doing this in a personal project and then dealing with malicious actors and getting stuck with a big bill.

You can set hard limits on the api when you configure billing, so you know your monthly bill will never be more than $X

Re: OpenAI is too cheap to beat

#333

Post by wrsh07 »

littlestymaar wrote: post #141
serjester wrote: post #55

I think this is under appreciated. I run a "talk-to-your-files" website with 5ish K MRR and a pretty generous free tier. My OpenAI costs have not exceeded $200 / mo. People talk about using smaller, cheaper models but unless you have strong data security requirements you're burdening yourself with serious maintenance work and using objectively worse models to save pennies. This doesn't even consider OpenAI continuous…

> you should probably just be using the best that's available (OpenAI). Sure, if you want to let a monopoly have all the added value while you get to keep the rest you can do that. Just make sure you're never successful enough to inspire them though, otherwise you're dead the next minute. Oops.

5k monthly revenue (or even 100k) isn't big enough for OpenAI to care. It's barely big enough for a competitor to care.

I suspect that OpenAI doesn't have the bandwidth to build most uses of ai and so is in the bill gates platform land: the ecosystem should be pocketing more money than the owner of the platform is.

Re: OpenAI is too cheap to beat

#334

Post by BoorishBears »

AJRF wrote: post #158

I read this and think "That won't last long". The pricing is too good to be true with you think about it rationally. If they raise prices they seem much, much less attractive than using AWS or Azure. Amazon seem to have a much better business built around their Bedrock offering. And all their other tools are available there like SageMaker, ec2, integration with MLFlow, etc, etc. I guess the same goes for Azure, if yo…

> The pricing is too good to be true with you think about it rationally

In what way shape or form?

> If they raise prices they seem much, much less attractive than using AWS or Azure.

They're already significantly more expensive than Azure. OpenAI charges something like $30k a month for dedicated capacity on a "call our sales team" basis: GPT 3.5/ GPT-4 on Azure comes with that for free.

And GPT-4 is already slow and expensive enough that no one just chooses it arbitrarily... they're using it for things no other model can do. They could charge double for GPT-4 and GPT-4 would still be the only model that can do those tasks: you wouldn't get to just switch off to some other GPT-4 equivalent provider.

> Amazon seem to have a much better business built around their Bedrock offering

Amazon is literally doing the same thing with Bedrock! They're offering Anthropic at competitive prices to OpenAI for a model that's no cheaper to run based on their own dedicated capacity numbers.

> OpenAI offering just models doesn't seem like it can last forever, and to compete with AWS or Azure at enterprise level they need to build all the things Amazon/MS have built.

OpenAI is not trying to become Azure: They actively go out of their way to hide the fact they even offer half the things they offer to enterprises, instead relying on Azure absorbing demand as much as possible.

OpenAI wants ChatGPT Plus to be the new Prime, as in no one should be able to afford to not pay OpenAI for their immensely valuable offering.

Except unlike Prime, the offering is software, not commerce: If Amazon could get AWS-like margins from their e-commerce business, AWS would be a footnote.

Re: OpenAI is too cheap to beat

#335

Post by Tokumei-no-hito »

Spivak wrote: post #234
latchkey wrote: post #186

Earlier quoted context omitted.

$20/mo... ~$1.3 day. I'm good with "vastly cheaper than a latte" pricing model and $20 to try something out for a month isn't bad at all.

Right but the API is so unbelievably cheap in comparison. I couldn't spend $20 if I tried using it constantly. My bill is a few bucks every month and you don't have to deal with "As a large language model…"

> and you don't have to deal with "As a large language model…"

what do you mean? is the API uncensored?

Re: OpenAI is too cheap to beat

#336

Post by pleoxy »

vikramkr wrote: post #277
pleoxy wrote: post #276

Earlier quoted context omitted.

It's not that hard. Particularly for simple deployments, which a startup should have or they are doing it wrong.

How would you do it for a simple webapp? Genuine question

Ingress that handles SSL. nginx, or caddy. Then stand up your app server behind that on the same VM. Database can be on the same or different VM.

I try to not use anything else if I can avoid it on a new project.

Ingress gives you the ability to load balance and is threaded and will scale with network transfer and cpu. Database should scale with a bump in VM specs as well, CPU and disk IOPS.

If you keep your app server stateless you can simply multiply it for the number of copies you need for your load.

Systemd can keep your app server running, it you docker it up and use that

Re: OpenAI is too cheap to beat

#337

Post by thegasman »

shon wrote: post #311
flir wrote: post #294

Earlier quoted context omitted.

The data is the moat. (If you can train your internally deployed LLM on data none of your competitors have, that's an advantage).

In that case, X.AI, powered by X/Twitter/Tesla data and possibly Facebook (both closed, and somewhat hard to crawl inside) have the largest moat.

China probably has the most comprehensive data on its users from a surveillance perspective

Re: OpenAI is too cheap to beat

#338

Post by tomcam »

appplication wrote: post #106

The premise of this is flawed. OpenAI is cheap because of has to be right now. They need to establish market dominance quickly, before competitors slide in. The winner of this horse race is not going to be the company with the best performing AI, it’s going to be the one who does the best job at creating an outstanding UX, ubiquitously presence, entrenching users, and building competitive moats that are not feature d…

How would a potential competitor obtain an equivalent body of training data?

Re: OpenAI is too cheap to beat

#339

Post by NikolaNovak »

appplication wrote: post #106

The premise of this is flawed. OpenAI is cheap because of has to be right now. They need to establish market dominance quickly, before competitors slide in. The winner of this horse race is not going to be the company with the best performing AI, it’s going to be the one who does the best job at creating an outstanding UX, ubiquitously presence, entrenching users, and building competitive moats that are not feature d…

Possibly unrealistic, but my fear is that they will end up like Netflix more than uber.

Some will scream in horror but I wanted Netflix to be a monopoly. A single place and app and account with all the content I need.

"competition" in streaming space has been nothing but disastrous for me as a consumer. It led to greedy heterogeneous islands of content, with proliferation of crappy apps and pointless restrictions and return to cable package mentality.

Again, Possibly irrationally and ignorantly, my fear is that 5 years from now I'll need a dozen subscriptions to less good services which will hoard their source data and models and be specialized based on which content they got licenses to. I. E. There'll be ai1 with new York times and Wikipedia, and ai2 with Washington post and encyclopedia Britannica, and ai3 with I don't know fox news and RT, and ai4 with mit and Harvard business libraries, and ai5 focused on math with extra subscription to wolfram, and ai6 with rights to stack overflow and JavaScript and so on.

There are many scenarios various writers have posited where we are actually in local maxima lf ll, with data being increasingly closed and or poisoned, and possibly segregated in the near future. :-/

Re: OpenAI is too cheap to beat

#340

Post by intended »

adam_arthur wrote: post #231
user3939382 wrote: post #223

Earlier quoted context omitted.

> The winner is going to be the consumers Cloud infra may be a comparable market, since computation is a big share of AI costs. Did consumers win big from competition between AWS, Azure, and GCP? Not sure. I see an uptick in write ups saying “We switched off cloud and reduced costs by 2/3rds.” Not a scientific sample but may leave the question open.

Much harder to switch cloud providers than to switch LLM models. How much time would it take most companies to move their product from AWS to GCP, for example? What if you use a cloud specific tool like DynamoDB? Margin is a function of stickiness/cost of switching (among other things). I suspect eventually we will enter a world where migrating cloud providers is mostly a click of a button, but we're a long ways off…

If you swap an LLM, at the very least, you have to run your entire Eval set again.

This will almost certainly lead to a prompt rebuild, to better accommodate new model idiosyncrasies.

If you are unlucky- your use case may be one where Evals require human review.

Unless you are YOLOing it without evals. In that case this is relevant.

Post reply on HN