Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI is too cheap to beat

generatingconversation.substack.com

Post reply on HN
Print view View original

371–380 of 426 posts

Re: OpenAI is too cheap to beat

#371

Post by selfhoster11 »

vlod wrote: post #196
Havoc wrote: post #45

Yep. Building a project that needs some LLMs. I'm very much of the self-hosting mindset so will try DIY, but it's very obviously the wrong choice by any reasonable metric. OpenAI will murder my solution by quality, by availability, by reliability and by scalability...all for the price of a coffee. It's a personal project though & partly intended for learning purposes so there is scope for accepting trainwreck level t…

Isn't it very expensive to do this? Don't you need a jacked up PC with multiple expensive NVidia cards?

For basic usage, you can get away with a small graphics card or no graphics card at all (albeit it will be very slow).

The general rule of thumb is, take a model size (7B, 13B, 34B, 70B) and multiply that by 0.5 or 0.625. If that number is smaller than the combined amount of system RAM and VRAM in your system, you can run the model at 4-bit and 5-bit quantization respectively.

Re: OpenAI is too cheap to beat

#372

Post by 7speter »

sangnoir wrote: post #201
outlace wrote: post #185

Earlier quoted context omitted.

But OpenAI appears to have some sort of data moat. I doubt their model is the best in the world, but more/better data generally beats better model, and GPT-4 definitely beats Claude, Bard, Bernie and the rest probably because they curated the best quality and largest data set. Maybe that moat doesn't last long but perhaps they have exclusive rights to some of that dataset through commercial agreements that could be a…

> But OpenAI appears to have some sort of data moat. I'm willing to bet dollars to doughnuts that Google and Facebook have at least one, possibly 2 or more orders of magnitude more latent training data to work with - not including Googles search index. My uninformed opinion is that Google and Meta's ML efforts are fragmented - with lots of serious effort going into increasing existing revenue streams with LLMs and th…

I think threads is a cover to get more natural language data from actual people for free, but maybe I’m wrong

Re: OpenAI is too cheap to beat

#373

Post by visarga »

PaulHoule wrote: post #160
ldjkfkdsjnv wrote: post #136

Earlier quoted context omitted.

Completely wrong, the best AI will win. There is insane demand for better models.

Depends how you define quality. This paper reflects my own experience https://arxiv.org/abs/2305.08377 and shows how LLM technology has a lot more to offer than "ChatGPT". The real takeaway is that by training LLMs with real training data (even with a "less powerful" model) you can get an error rate more than 10x less than you get with the "zero shot" model of asking ChatGPT to answer a question for you the same way…

Fresh approaches to AI hardware are emerging, like the Groq Chip which utilizes software-defined memory and networking without caches. To simplify reasoning about the chip, Groq makes it synchronous so the compiler can orchestrate data flows between memory and compute and design network flows between chips. Every run becomes deterministic, removing the need for benchmarking models since execution time can be precisely calculated during compilation. With these innovations, Groq achieved state-of-the-art speed of 240 tokens/s on 70B LLaMA.

Fascinating stuff - a synchronous distributed system allows treating 1000 chips as one, knowing exactly when data will arrive cycle-for-cycle and which network paths are open. The compiler can balance loads. No more indeterminism or complexity in optimizing performance (high compute utilization). A few basic operations suffice, with the compiler handling optimization, instead of 100 kernel variants of CONV for all shapes. Of course, it integrates with Pytorch and other frameworks.

https://youtu.be/hk4QGpQAvSY?t=57

Re: OpenAI is too cheap to beat

#374

Post by Narann »

appplication wrote: post #106

The premise of this is flawed. OpenAI is cheap because of has to be right now. They need to establish market dominance quickly, before competitors slide in. The winner of this horse race is not going to be the company with the best performing AI, it’s going to be the one who does the best job at creating an outstanding UX, ubiquitously presence, entrenching users, and building competitive moats that are not feature d…

OpenAI is not cheap, it’s selling at lost.

Re: OpenAI is too cheap to beat

#375

Post by chpatrick »

golergka wrote: post #369
chpatrick wrote: post #367

Earlier quoted context omitted.

Or a cryptocurrency where the proof of work is running a malicious AI...

You mean the basilisk?

Not exactly but it would be a good name for it!

If you run it you get some money and higher balances mean better treatment when it takes over.

Re: OpenAI is too cheap to beat

#376

Post by kevingadd »

singularity2001 wrote: post #349
appplication wrote: post #106

The premise of this is flawed. OpenAI is cheap because of has to be right now. They need to establish market dominance quickly, before competitors slide in. The winner of this horse race is not going to be the company with the best performing AI, it’s going to be the one who does the best job at creating an outstanding UX, ubiquitously presence, entrenching users, and building competitive moats that are not feature d…

One mayor difference between Uber and Open AI is that fundamentally openeyes technology will get cheaper for them to run, hardware wise and software wise. They just need to hold their position long enough for variants of Moores law to kick in.

the hardware these LLMs run on isn't going to get 10x faster/cheaper in the span of a couple years, it will get incrementally faster at the cost of having to buy new expensive datacenter GPU hardware. It's not going to magically save them from losing money on serving requests.

Re: OpenAI is too cheap to beat

#377

Post by RamblingCTO »

appplication wrote: post #106

The premise of this is flawed. OpenAI is cheap because of has to be right now. They need to establish market dominance quickly, before competitors slide in. The winner of this horse race is not going to be the company with the best performing AI, it’s going to be the one who does the best job at creating an outstanding UX, ubiquitously presence, entrenching users, and building competitive moats that are not feature d…

> The premise of this is flawed. OpenAI is cheap because of has to be right now. They need to establish market dominance quickly, before competitors slide in. The winner of this horse race is not going to be the company with the best performing AI, it’s going to be the one who does the best job at creating an outstanding UX, ubiquitously presence, entrenching users, and building competitive moats

I hate this. Not the best will win, but the one with the biggest pockets. Nothing that helps with technofeudalism. Proper competition would be good.

Re: OpenAI is too cheap to beat

#378

Post by j-a-a-p »

Common sense let me think you need to calculate the value of the data that OpenAI receives. But the article quickly goes into a comparison between OpenAI, which is highly optimised for the AI workload versus the pricing of an instance at AWS. AWS, renowned for their hefty cost and in the center of the cloud repatriation movement. That alone kills the argument that OpenAI is 'too cheap'.

Re: OpenAI is too cheap to beat

#379

Post by [deleted] »

littlestymaar wrote: post #358
wrsh07 wrote: post #333

Earlier quoted context omitted.

5k monthly revenue (or even 100k) isn't big enough for OpenAI to care. It's barely big enough for a competitor to care. I suspect that OpenAI doesn't have the bandwidth to build most uses of ai and so is in the bill gates platform land: the ecosystem should be pocketing more money than the owner of the platform is.

Now

[deleted]

Re: OpenAI is too cheap to beat

#380

Post by martypitt »

Bukhmanizer wrote: post #103
serjester wrote: post #55

I think this is under appreciated. I run a "talk-to-your-files" website with 5ish K MRR and a pretty generous free tier. My OpenAI costs have not exceeded $200 / mo. People talk about using smaller, cheaper models but unless you have strong data security requirements you're burdening yourself with serious maintenance work and using objectively worse models to save pennies. This doesn't even consider OpenAI continuous…

The bleeding obvious is that OpenAI is doing what most tech companies for the last 20 years have done. Offer the product for dirt cheap to kill off competition, then extract as much value from your users as possible by either mining data or hiking the price. I don’t understand how people are surprised by this anymore. So yeah, it’s the best option right now, when the company is burning through cash, but they’re plann…

I agree with this.

However, what will be interesting is if the price of delivering ChatGPT-style experiences drops as the industry matures/advances, and their pricing moat erodes.

Unlike Uber - where prices are dictated by factors unlikely to move significantly (Labour / Vehicles / Fuel / etc), The LLM space doesn't have these types of overheads.

Post reply on HN