OpenAI is too cheap to beat • Hacker News Board

Re: OpenAI is too cheap to beat

#411

Post by NikolaNovak » Fri, Oct 13, 2023, 5:59 PM UTC

0x008 wrote: post #352

NikolaNovak wrote: post #339

Earlier quoted context omitted.

Possibly unrealistic, but my fear is that they will end up like Netflix more than uber. Some will scream in horror but I wanted Netflix to be a monopoly. A single place and app and account with all the content I need. "competition" in streaming space has been nothing but disastrous for me as a consumer. It led to greedy heterogeneous islands of content, with proliferation of crappy apps and pointless restrictions and…

I don’t think you can really compare that. The streaming service market is not elastic because you cannot easily interchange one series for another. If other vendors LLMs become good enough it will actually be easily to interchange and then the race for the best UX and integration will be upon us (which the other commenter alluded to).

I think it's a circular argument.

"if other llms are good enough" assumes that in principle they have same opportunities, access to same data, or content. My fear is precisely that this assumption may be taken away - I. E. That news paper publishers or encyclopedia owners or big websites (stack overflow, web Md, etc) will enter into arrangement with specific llm companies - just like Netflix Disney prime etc aren't competing on their app or price or flexibility, but on exclusive underlying content. Nobody WANTS to subscribe to Paramount+ or cbs access... But if they hold enough material hostage some people will 'have to'. I can see a future, not far off, where different llm organizations selling feature is not how good their technology is - to your and overvodys point, THAT moat is likely to even out - but what underlying training data they have legal access to.

Re: OpenAI is too cheap to beat

#412

Post by latchkey » Fri, Oct 13, 2023, 7:15 PM UTC

Spivak wrote: post #314

vunderba wrote: post #287

Earlier quoted context omitted.

I'm astonished how often this comes up and also how wrong it is. The cost of the GPT-4 API is ballpark around $0.05 / 1000 tokens. If you want to include a rolling context window which you basically HAVE TO DO if you want to maintain a persistent conversation, you will easily meet or exceed 1000+ tokens. ChatGPT Pro gives you 50 GPT-4 queries every three hours. If you're using it all day you might average about 100 d…

I think you're being really wasteful with that kind of context window which is why it's a bit apples and oranges. I keep stats on this, my average message is around 50 tokens, the average individual response is around 200 tokens, and my average conversion length is 1.2 (only counting my messages). 50/convos/day * 30 days is $21 and I don't come close to that usage. Hell most of the time I don't even turn on GPT-4 bec…

Your usecase is that you don’t really use it very much, don’t want the advanced features and you’re arguing over $20 a month.

We are in a different category.

Re: OpenAI is too cheap to beat

#413

Post by kristopolous » Fri, Oct 13, 2023, 7:54 PM UTC

BoorishBears wrote: post #330

kristopolous wrote: post #313

That was said about uber ... it's just being subsidized by vc. Wait until they expect a profit huggingface will be more of a winner in the long run with an exit via an MSFT acquisition if I had to call it.

This is like Uber if self-driving cars had simply been a matter of making a faster GPU. OpenAI gets cheaper by twiddling their thumbs for the next few years, meanwhile they continue to amass more and more data for RLHF. It's weird that people are trying to drag non-software scaling into a software scaling problem: Lyft, Doordash, Instacart, etc. all relied on VC dollars to scale non-software growth like software. Ope…

ok so you'll have to help me here, I'm still learning this stuff.

RLHF I looked it up. Is this really useful? The average human has zero general expertise because people are specialized (I know nothing about say, 1960s avant garde french cinema and my responses in a conversation there would be garbage - given the breadth of human knowledge even the most accomplished scholars are useless for over 99% of it). Won't there be a quality decrease? How is this accommodated for?

If the chat systems simply gave the most popular answers it would cease to be useful real fast.

Re: OpenAI is too cheap to beat

#414

Post by wrsh07 » Fri, Oct 13, 2023, 9:24 PM UTC

Cwizard wrote: post #361

wrsh07 wrote: post #301

Earlier quoted context omitted.

They have a soc 2 so literally an (external) auditor has looked at their data retention policies and if your business is a customer you should request access to the report https://trust.openai.com/

but does that matter legally for health, finance and legal sectors? I am not familiar with the laws themselves but I worked in finance for a long time and the internal rules where that sensitive data cannot move off premises no matter what the external party promised/had certified.

Yes there are certifications for each of those sectors. Finance has pci compliance, health has HIPAA.

For legal issues it's a bit more nuanced (eg new york state has guidelines about best practices, but they're honestly fairly sensible and would probably allow SOC 2 or equivalent)

The best thing anybody can do with your data is not store it for very long. Beyond that, they should take sensible measures, like encrypt it at rest, have policies restricting access, etc

Re: OpenAI is too cheap to beat

#415

Post by wrsh07 » Fri, Oct 13, 2023, 9:27 PM UTC

littlestymaar wrote: post #358

wrsh07 wrote: post #333

Earlier quoted context omitted.

5k monthly revenue (or even 100k) isn't big enough for OpenAI to care. It's barely big enough for a competitor to care. I suspect that OpenAI doesn't have the bandwidth to build most uses of ai and so is in the bill gates platform land: the ecosystem should be pocketing more money than the owner of the platform is.

Now

Was this an accident? If it's a complete thought intending to say "for now" - eh, I would bet on it for quite a while. They don't care about your little use case.

It's possible gpt next or next++ ends up making whatever work you do trivial, but it's likely you'll still have customers

Re: OpenAI is too cheap to beat

#416

Post by YetAnotherNick » Fri, Oct 13, 2023, 9:31 PM UTC

zarzavat wrote: post #408

YetAnotherNick wrote: post #306

Earlier quoted context omitted.

Did you even read my comment? I specifically highlighted why openai might be cheaper in long run. One is they are already working on a chip that would be better just for running a single model.

They are not going to beat NVIDIA. Making a chip for one model is not really a good idea, there are more efficiency gains to be made by improving the model and using a general purpose AI chip, rather than keeping the model architecture static and building a special purpose chip for it. Regardless, whatever OpenAI can do, NVIDIA can do better, and on more recent process nodes because they have the volume.

No, because NVidia has to work for all the models. Nvidia has other constraints that they need to have for users like instructions, security etc. which openai doesn't have.

e.g. As they have a fixed model which they know they would get billions of request to, they could even work with analogue chip which is significantly cheaper and faster for inference. [1] could achieve 10-100x flops/watt for fixed models compared to nvidia for their first gen chip.

[1]: https://www.nature.com/articles/s41586-023-06337-5

Re: OpenAI is too cheap to beat

#417

Post by Ridj48dhsnsh » Fri, Oct 13, 2023, 10:15 PM UTC

taneq wrote: post #327

noah_buddy wrote: post #222

Earlier quoted context omitted.

It’s a funny point. After only taking Ubers and Lyfts all my adult life (young), I have recently switched to cabs because they are cheaper in my city (and I tell everyone I know to do the same). They have dominance now but if I had to guess whether cabs or Uber will still exist in 100 years… I know which I would bet on.

Uber's surge pricing is dumb in places where they're competing with traditional taxi ranks. I usually pay $35-$45 for an Uber to/from the airport, but if a couple of flights land at once, suddenly Uber is $85+ and the taxis are only $55.

Sounds to me like it's working perfectly.

The point of surge pricing is to rebalance supply and demand when demand for rides outstrips supply of drivers, whether by attracting more drivers or by discouraging price sensitive riders.

Some riders at the airport will strongly prefer Uber, for whatever reason, so they're less price sensitive than you. Because you're happy to substitute an Uber for a taxi, you decrease the demand as a response to the surge pricing, preserving the limited supply of drivers for the riders who really want them (or are at least are price insensitive enough to pay for that privilege).

Re: OpenAI is too cheap to beat

#418

Post by pleoxy » Sat, Oct 14, 2023, 2:13 PM UTC

vikramkr wrote: post #404

pleoxy wrote: post #336

Earlier quoted context omitted.

Ingress that handles SSL. nginx, or caddy. Then stand up your app server behind that on the same VM. Database can be on the same or different VM. I try to not use anything else if I can avoid it on a new project. Ingress gives you the ability to load balance and is threaded and will scale with network transfer and cpu. Database should scale with a bump in VM specs as well, CPU and disk IOPS. If you keep your app serv…

No that's all fine, I mean physically, where would I put a server and stuff if we didn't have cloud providers? I'd need to pay an isp for an IP address and maybe port forward and stuff like that right? I don't get why I wouldn't just do what you mentioned on a five dollar digital ocean droplet or an ec2 instance or whatever, the cloud still seems orders of magnitude easier to get off the ground.

If you rent a cloud vps as an ingress you can run an overlay network and your actual hosted services can literally be anywhere. See nebula, netbird, etc. You can also ssh forward, but that doesn't scale well past a handful of services and is a bit fragile.

For new small systems I suggest you start with a cloud VPS. If traffic is low, cost of downtime is low, and system requirements are high then a cheap mini PC ($150) at the home or office can keep your bill microscopic. If your app server and database are small then you can just throw them on the VPS too.

I run light traffic stuff at home in a closet so it doesn't occupy more costly cloud RAM. Production ready saas offerings I'm trying to sell right now are all in the cloud. Hosting all my stuff in the cloud would cost me hundreds per month. My home SLA is fine for the extras. I don't need colocation at this time, but I have spoken with data centers to understand my upgrade path.

You can run a live backup server at a second location and have pretty good redundancy should the primary lose power or connectivity.

When system requirements elevate (SLA, security, etc) you probably want to move into a data center for better physical security, reliable power and network. Bigger VPS is fine if it is big enough. Can also do a colocation if you don't want to rent, and you contract directly with a data center. I wouldn't look at colocation until your actual hosting needs exceed at least $100/mo and you're ready for a year long commitment.

Re: OpenAI is too cheap to beat

#419

Post by BoorishBears » Sat, Oct 14, 2023, 5:46 PM UTC

kristopolous wrote: post #413

BoorishBears wrote: post #330

Earlier quoted context omitted.

This is like Uber if self-driving cars had simply been a matter of making a faster GPU. OpenAI gets cheaper by twiddling their thumbs for the next few years, meanwhile they continue to amass more and more data for RLHF. It's weird that people are trying to drag non-software scaling into a software scaling problem: Lyft, Doordash, Instacart, etc. all relied on VC dollars to scale non-software growth like software. Ope…

ok so you'll have to help me here, I'm still learning this stuff. RLHF I looked it up. Is this really useful? The average human has zero general expertise because people are specialized (I know nothing about say, 1960s avant garde french cinema and my responses in a conversation there would be garbage - given the breadth of human knowledge even the most accomplished scholars are useless for over 99% of it). Won't the…

RLHF isn't used to teach the model what it knows, it's used to teach the model how to follow instructions

Before RLHF instruct tuning the models could only complete sentences

Technically they still complete sentences, but now they have a strong association for a format where a question is followed by an answer

Re: OpenAI is too cheap to beat

#420

Post by vikramkr » Sun, Oct 15, 2023, 2:41 AM UTC

pleoxy wrote: post #418

vikramkr wrote: post #404

Earlier quoted context omitted.

No that's all fine, I mean physically, where would I put a server and stuff if we didn't have cloud providers? I'd need to pay an isp for an IP address and maybe port forward and stuff like that right? I don't get why I wouldn't just do what you mentioned on a five dollar digital ocean droplet or an ec2 instance or whatever, the cloud still seems orders of magnitude easier to get off the ground.

If you rent a cloud vps as an ingress you can run an overlay network and your actual hosted services can literally be anywhere. See nebula, netbird, etc. You can also ssh forward, but that doesn't scale well past a handful of services and is a bit fragile. For new small systems I suggest you start with a cloud VPS. If traffic is low, cost of downtime is low, and system requirements are high then a cheap mini PC ($150…

But to the point of the original question that started this thread - the takeaway is still that cloud services made development massively easier right? The answer to the original question seems to still be "Yes cloud providers did lead to big wins for customers" since none of these other suggestions are able to get away from needing a cloud service provider without making starting something intensely difficult. And you wouldn't be able to get a vps for 5 bucks for ingress without all the other cloud competition in the market.

> Cloud infra may be a comparable market, since computation is a big share of AI costs. Did consumers win big from competition between AWS, Azure, and GCP? Not sure. I see an uptick in write ups saying “We switched off cloud and reduced costs by 2/3rds.” Not a scientific sample but may leave the question open.