Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI is too cheap to beat

generatingconversation.substack.com

Post reply on HN
Print view View original

281–290 of 426 posts

Re: OpenAI is too cheap to beat

#281

Post by pleoxy »

throwaway50182 wrote: post #275
jokethrowaway wrote: post #256

Earlier quoted context omitted.

I disagree, at early startup scale you don't need much, you just buy a better VPS when you need to scale up Learning aws vs learning how to operate a vps are of comparable complexity It took me less then a day to setup infra for my startup (more than 10 years ago)

If you're just running VPS on the big clouds, you're not going to get much advantage out of it, indeed. But tell me what third tier cheap provider has managed scale-to-zero-or-infinity functions? Managed storage with S3-like API? Where can I get an API gateway cheaper than Amazon? What about managed databases? These tools allow me to develop insanely scalable software incredibly easily. Agreed - all the big clouds ar…

You get portability. Which the functions do not provide. Open source solutions have a longer career utility than proprietary offerings. I remember when NetWare certs were all the rage. Useless now. I remember msce. But if you learned open tools 35 years ago instead... You get the picture.

You can scale from 1 thread 512MB RAM, to 500 threads and 12TB of RAM (off the shelf). Which is good enough for almost everyone who isn't planet scale.

Auto scaling also comes with auto billing. Oops, your accidental infinite loop spawning functions has bankrupted your company. You don't have that risk starting with a VPS.

Re: OpenAI is too cheap to beat

#282

Post by DonHopkins »

CamperBob2 wrote: post #195
layer8 wrote: post #92

Earlier quoted context omitted.

Apple is ramping up spending in that area: https://www.macrumors.com/2023/09/06/apple-conversational-ai... It’ll probably take a while though.

Yeah, at this rate they'll be at parity with Infocom by 2030.

So their customers will have a difficult time figuring out that they should put a fish in their ear?

Re: OpenAI is too cheap to beat

#283

Post by BoorishBears »

mlsu wrote: post #221
BoorishBears wrote: post #151

Earlier quoted context omitted.

For years people have essentially made a living off FUD like "ignore the literal legal agreement and imagine all the worst case scenarios!!!" to justify absolutely farcical on-premise deployments of a lot of software, but AI is starting to ruin the grift. There are some cases where you really can't afford to send Microsoft data for their OpenAI offering... but there are a lot more where some figurehead solidified the…

Gah, this is just not how it works. You are probably right that e.g. patient information, private conversations, proprietary code, etc would be safe with OpenAI. But it's not the on-prem team that needs to convince the rest of the organization to keep things on prem. Quite the opposite -- every single tech person would love to make our data someone else's problem (and get a big career boost from dealing with cloud te…

You're not saying anything counter to what I said.

> You cannot trust the scrappy startup OpenAI

Not saying you do: Azure has a dedicated capacity driven GPT-4/3.5 offering that you can stick in your VPC with everything from PCI to HITRUST certs. These are the things that come out if you actually care about delivering solutions vs jumping to deliver the right sounding words for the figureheads like "We'd never trust those scrappy OpenAI guys!!!!"

> Quite the opposite -- every single tech person would love to make our data someone else's problem (and get a big career boost from dealing with cloud tech instead of the dead-end that is local sysadmin!).

You're attracting the least equipped people who tumbled into what you just admitted is a dead end trajectory, usually paying below market rates as a result, and then expecting them to outperform the people paying the most money for competent security outlays with much bigger fish (Azure is working with teams that need FedRAMP, DoD certs, HIPPA compliance, and much more)

The end result is that you end up with a poorly maintained leak sieve of an infrastructure in which Azure would likely be the most secure component you have to lean on in your entire organization.

You say:

> because the people who actually give a fuck about the risk NEED to have granular detail of what data, readable by whom, is stored exactly where and for how long, and how can you make sure, and how do you know that access is scoped to the absolute minimum number of people, and is there a paper trail for that

They don't care about risk, they care about flawed perceptions of risk that don't align with reality. These are the same companies that get pwned for years through some basic social engineering, and all that they ever have to show for it is audit logs that show who ac... ah wait no one ever actually checked the logs and it turns out they're useless because subsystem X Y and Z aren't even connected to it.

Re: OpenAI is too cheap to beat

#284

Post by herval »

agentcoops wrote: post #270
adam_arthur wrote: post #179

Earlier quoted context omitted.

The winner is going to be the consumers of AI. It's a race to the bottom on pricing on the provider/infra side. It seems very unlikely that any single LLM provider will achieve a sustained and durable advantage enough to achieve large margins on their product. Consumers can swap between providers with relative ease, and there is very little stickiness to these LLM APIs, since the interfaces are general and operate on…

Certainly the undisputed winners will be the very few firms with enough engineering resources and GPUs to train their own models (not just fine-tune) where the models in question increase the productivity of workers in their non-ai-related profit centers. After that we have the real question of what the future will be of open source LLMs, on the one hand, and the question most relevant to this article of what sort an…

This was definitely a theory that made people burn tons of money on the past couple of years, but I don’t think it holds water. These models are getting obsolete so fast, and there’s so many open ones, I doubt any one’s privately trained model can stay relevant for long

Re: OpenAI is too cheap to beat

#285

Post by moffkalast »

Havoc wrote: post #45

Yep. Building a project that needs some LLMs. I'm very much of the self-hosting mindset so will try DIY, but it's very obviously the wrong choice by any reasonable metric. OpenAI will murder my solution by quality, by availability, by reliability and by scalability...all for the price of a coffee. It's a personal project though & partly intended for learning purposes so there is scope for accepting trainwreck level t…

> all for the price of a coffee

Really depends what your API input is, using full GPT 4 context will drive you bankrupt with ~a dollar per prompt.

Re: OpenAI is too cheap to beat

#286

Post by omeze »

mbb70 wrote: post #228
jdminhbg wrote: post #166

Earlier quoted context omitted.

Google has a flywheel where its dominant position in search results in more users, whose data refines the search algorithm over time. The question is whether OpenAI has a similar thing going, or whether they just have done the best job of training a model against a static dataset so far. If they're able to incorporate customer usage to improve their models, that's a moat against competitors. If not, it's just a battl…

But that's exactly what they have: millions of high quality, rated chat interactions that no one else has. I don't know how they could _not_ incorporate customer usage to improve their models.

well, this assumes the chat (where the ratings are given) is what people are using and paying for. I think most businesses pay for some combination of API access and specific use cases like code generation (at least, thats what I pay for) that don't really impact RLHF data. General search for consumers is likely to schism since chatGPT isn't especially different from Bard or Edge's AI assistant or the myriad of other product surface areas that can add it.

Re: OpenAI is too cheap to beat

#287

Post by vunderba »

matteoraso wrote: post #184
latchkey wrote: post #16

I just paid the $20 for a month to try it out. In my super limited experience, GPT-4 is actually impressive and worth the money.

Unless you're an extremely heavy user, it's cheaper to just use the API. I've been tempted to do that, but OpenAI doesn't have a free trial for me to see the quality of GPT-4 first.

I'm astonished how often this comes up and also how wrong it is.

The cost of the GPT-4 API is ballpark around $0.05 / 1000 tokens. If you want to include a rolling context window which you basically HAVE TO DO if you want to maintain a persistent conversation, you will easily meet or exceed 1000+ tokens.

ChatGPT Pro gives you 50 GPT-4 queries every three hours. If you're using it all day you might average about 100 daily queries. Using a dedicated GPT4 API would run you approximately five dollars a day for the same thing - that's $150 a month as opposed to flat cost of $20.

Re: OpenAI is too cheap to beat

#288

Post by moffkalast »

redox99 wrote: post #236
oceanplexian wrote: post #144

Has anyone actually used GPT4? It's not "cheap". It was roughly $150 for me to build a small dataset with a few thousand quarter-page chunks of text for a data project using GPT4. GPT3 is substantially cheaper but it would hallucinate 30% of the time; honestly a nice fine-tune of LlaMA is on-par with GPT3 and after the sunk cost all it costs is a few $0.01 in electricity to generate the same sized dataset.

Even GPT3.5 can be much more expensive. In some specific tasks, a finetuned 7B llama can work as well as GPT3.5. You can rent a 3090 at $0.20/h on vast.ai, or $0.40/h on runpod. Using VLLM at 400t/s that's 1440000 generated tokens. Generating that amount of tokens with GPT3.5 would be $2.88.

Unless your use case isn't in English in which case LLama is as useful as a one-legged man in an ass-kicking contest.

LLama models only really shine for things that GPTs would refuse to even consider because of corporate RLHF, and if you need to keep your data local I suppose. For the rest they're second rate at best.

Re: OpenAI is too cheap to beat

#289

Post by stormfather »

PaulHoule wrote: post #160
ldjkfkdsjnv wrote: post #136

Earlier quoted context omitted.

Completely wrong, the best AI will win. There is insane demand for better models.

Depends how you define quality. This paper reflects my own experience https://arxiv.org/abs/2305.08377 and shows how LLM technology has a lot more to offer than "ChatGPT". The real takeaway is that by training LLMs with real training data (even with a "less powerful" model) you can get an error rate more than 10x less than you get with the "zero shot" model of asking ChatGPT to answer a question for you the same way…

Moore's law is irrelevant. Large language models are going to leave the digital paradigm behind altogether.

Neural nets don't need fully precise digital computing. Especially with quantization we're seeing that losing a bit of precision in the weights isn't impactful. Now that we're serving huge foundation models with static weights there's an enormous incentive to develop analog hardware to run them.

Mark my words, this will lead to a renaissance in analog computing, and in the future we will be shocked at the enormous waste of having run huge models on digital chips.

Just think, how many multiplications per second is the light refracting through your window right now clocking? More or less than is required to ChatGPT do you think? If only the crystals were configured correctly and the patterns of light coming through could be interpreted...

Re: OpenAI is too cheap to beat

#290

Post by jjeaff »

ra7 wrote: post #203
strangemonad wrote: post #133

Earlier quoted context omitted.

This argument has always felt to me like saying “google has no moat in search, they just happen to currently have the best page rank. Nothing is stopping yahoo from creating a better one”

Google's moat in search has always been systems and data center infrastructure. You can create your own search ranking algorithm, but you can't crawl the web and serve search results to billions of worldwide users in a few milliseconds.

I think it's also more than just systems and data centers. it is also difficult to scrape the web the way Google does without using Google IP addresses. a lot of the web now will block you or severely throttle you if you aren't one of the well know engines that they want indexing them.
Post reply on HN