p4d.24xlarge spot price is $8.2 / hour in US East 1 at the moment...
[deleted]
OpenAI is too cheap to beat
81–90 of 426 posts
Re: OpenAI is too cheap to beat
#82Earlier quoted context omitted.
Please Apple let me replace worthless Siri with ChatGPT on my iPhone. Would completely change how I use the device.
Just yesterday, while driving: "Read last message." Siri: "Sorry. Dictation service is unavailable at the moment." It's past time for excuses. High-level people at Apple need to be fired over this. Hello? Tim? Do your job. Hello? Anybody home...?
Nobody is switching away from Apple over this, so ultimately Tim is doing his job. Under his watch Apple has become the defacto choice for entire generations. Between vendor-lockin/walled gardens and societal/cultural pressures (don't want to be a green bubble!), they have one of the stickiest user bases there are.
Re: OpenAI is too cheap to beat
#83I think this is under appreciated. I run a "talk-to-your-files" website with 5ish K MRR and a pretty generous free tier. My OpenAI costs have not exceeded $200 / mo. People talk about using smaller, cheaper models but unless you have strong data security requirements you're burdening yourself with serious maintenance work and using objectively worse models to save pennies. This doesn't even consider OpenAI continuous…
I completely agree — open-source models and custom deployments just can't compete with the cost and efficiency here. The only exception here is if open-source models can get way smaller and faster than they are now while maintaining existing quality. That will make private deployments and custom fine-tuning way more likely.
Or FOSS models remain the same size and speed, but hardware for running them, especially locally, steadily improves till the AI is "good enough" for a large enough segment of the market.
Re: OpenAI is too cheap to beat
#84I think this is under appreciated. I run a "talk-to-your-files" website with 5ish K MRR and a pretty generous free tier. My OpenAI costs have not exceeded $200 / mo. People talk about using smaller, cheaper models but unless you have strong data security requirements you're burdening yourself with serious maintenance work and using objectively worse models to save pennies. This doesn't even consider OpenAI continuous…
how do ur customers feel about you uploading potentially confidential documents to a 3rd party?
Re: OpenAI is too cheap to beat
#85I’m confused don’t a100s cost 10,000 to buy? Why would you pay 166k per year to rent?
AWS is extremely overpriced for nearly every service. I don’t know why anyone else outside of startups with VC money to burn or bigcos that need the “no one ever got fired for buying IBM” guarantee would use them. You’re better off with Lambdalabs or others which charge only $1.1/h per A100. Also that is a 8xA100 system as others have noted, but it is the 40GB one which can be found on eBay for as low as $3k if you g…
There are only two services that are dirt cheap and way too reliable, useful.That's S3 and SQS. Rest can get very expensive very soon.
You can build a lot of stuff on top of these two.
Re: OpenAI is too cheap to beat
#86Nothing in that article convinces me the situation couldn't change entirely in any given month. Google Gemini could be more capable. Any number of new players (AWS, Microsoft, Apple) could enter the market in a serious way. The head-start OpenAI has in usage data is small and probably eclipsed by the clickstream and data stores that Google and Microsoft have access to. I see no durable advantage for OpenAI.
Gemini very well might be the biggest threat to OpenAI. ChatGPT has first-mover advantage so has a decent moat, but the amount of people willing to pay $20 per month for something worse[1] than they get for free with google.com is going to dwindle. I'd be very worried if I were them.
[1]: That knowledge cutoff and terrible UX of browse the web is brutal compared to the experience of Bard
Re: OpenAI is too cheap to beat
#87I signed up for OpenAI's ChatGPT tool, and entered a query, like 'What does the notation 1e100 mean?' (just to try it out). And then when displaying the output it would start outputting the reply in a slow way, like, it was dripfeeded to me, and I was like: 'what? surely this could be faster?' Maybe I'm missing something crucial here, but why does it dripfeed answers like this? Does it have to think really hard about…
The non-technical way to think about it is that ChatGPT “thinks out loud” and can only “think out loud”.
Future products would be able to hide some of that, but for now, that’s what the ChatGPT / Bing Assistant product does.
Re: OpenAI is too cheap to beat
#88I think this is under appreciated. I run a "talk-to-your-files" website with 5ish K MRR and a pretty generous free tier. My OpenAI costs have not exceeded $200 / mo. People talk about using smaller, cheaper models but unless you have strong data security requirements you're burdening yourself with serious maintenance work and using objectively worse models to save pennies. This doesn't even consider OpenAI continuous…
Just curious, could you briefly mention some of the custom use cases with negligible AI costs? Thanks
Re: OpenAI is too cheap to beat
#89Yep. Building a project that needs some LLMs. I'm very much of the self-hosting mindset so will try DIY, but it's very obviously the wrong choice by any reasonable metric. OpenAI will murder my solution by quality, by availability, by reliability and by scalability...all for the price of a coffee. It's a personal project though & partly intended for learning purposes so there is scope for accepting trainwreck level t…
A lot of tools for constraint, creativity, and related rely on manipulating the entire log probability distribution. OpenAI won’t expose this information and is therefor shockingly uncompetitive on things like poetry generation
Re: OpenAI is too cheap to beat
#90Earlier quoted context omitted.
Not really. A full rack with 16 amps usable power and some bandwidth is $400/month in Kansas City, MO. That is enough to power 5x A100s 24x7, so 10k plus $80 per month each, amortized, of course many more A100s would drop the price. Once installed in the rack ($250 1 time cost) you shouldn't need to touch it. So 10k plus $1250 per A100, per year including power. You can put 2 or 3 A100s per cheapo Celeron based CPU w…
And how many A100s do you need to do something meaningful with LLMs?
The funding has to come from somewhere, right? You either pay up front and save money over time, or pay as you go and pay more...