Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Status: Multiple engines are down

status.openai.com

Post reply on HN
Print view View original

121–130 of 167 posts

Re: OpenAI Status: Multiple engines are down

#121

Post by tstrimple »

152334H wrote: post #4

With all of these outages, you have to wonder if it's a lack of skilled engineers on their part, or if they simply don't have enough GPUs to keep the lights on all the time.

In every company I've worked at that does ML work, the data scientists building the models don't have a strong software delivery background. They aren't building a production system with fail safes and redundancy. Their focus is the model itself. Consequently, when they stand up services in front of these models they are not even close to "production grade". I'd hope that a company focused on ML (versus an enterprise just dipping their toes into ML) would have solved these problems, but I wonder if it's related.

Re: OpenAI Status: Multiple engines are down

#122

Post by tstrimple »

Waterluvian wrote: post #95
localhost wrote: post #88

Earlier quoted context omitted.

GPT3 is ~175B parameters. At float16 precision, that's 350GB of weights. BLOOM-176B is about the same size. Here's one person's experience; a token is ~0.75 words. "The Python code in this tutorial generates one token every 3 minutes on a computer with an i5 11gen processor, 16GB of RAM, and a Samsung 980 PRO NVME..." [1] https://towardsdatascience.com/run-bloom-the-largest-open-ac...

Goodness... And to think my brain was free!

I wonder if your parents would agree with this statement.

Re: OpenAI Status: Multiple engines are down

#123

Post by drdaeman »

ronsor wrote: post #57
TOMDM wrote: post #24

Earlier quoted context omitted.

Don't get me wrong, I'd love to be able to run GPT3 and the subsequent finetuned versions thereof myself, but OpenAI has essentially no financial incentive to do so. A few years ago, we could maybe lean on their open aspirations to get that done, but with the "limited profitibility model" they've since instead adopted, I think that dream is mostly gone. At least we still get the occasional treat like Whisper out of t…

It's not as if anyone could afford to self-host the giant GPT-3 model anyway.

Why not? Please correct me if I'm wrong but from what I've heard, It's not millions or hundreds of thousands, it's "only" tens of thousands dollars' of hardware we're talking about. There are enthusiasts that spend comparably on their hobbies.

And a few dozen like minded people banding up together, shelling out a couple grands each? I'd say that's a totally realistic scenario.

Re: OpenAI Status: Multiple engines are down

#124

Post by naet »

danielscrubs wrote: post #47

To put this in perspective: the outage was 52 min. The total number of employees OpenAI have are 375. The launch date was just some months ago. At Google, the Cloud SQL dashboard was unavailable for around 12 hours a couple of weeks ago if I read this correctly: https://status.cloud.google.com/incidents/xg2qrL1UuSJiPDZALJ... The total number of employees of Google are 156 500. Google Cloud was launched 2008. So when…

Not sure what "perspective" you are trying to put this in by offering these comparison points between OpenAI and Google. Are you trying to say that OpenAI is more reliable than Google?

For additional perspective, OpenAI has a very small number of highly similar offerings (neural net with API access), while Google has a huge host of very different offerings, from web indexing and search, to email, to file hosting, to video streaming, to cloud compute... etc. Google also has a vastly larger user pool by at least a couple orders of magnitude. Google's core services are extremely solid even at ridiculous scale and have few outages, any of which would be considered major news.

Google also operates all this at a profit, while OpenAI works at a deficit. Google has had to scale larger than nearly any other service while maintaining profitability while OpenAI is more or less free to throw more compute power to solve problems, at any cost, to build valuation. Google has written entire programming languages to help them keep up at an unprecedented scale.

Comparing a single minor Google offering going down to an OpenAI outage isn't a fair comparison to either company. Yes, Google has, by your numbers, about 400 times the number of OpenAI employees. I'd be willing to bet that a single large Google service like YouTube handles more than 400 times the amount of compute, data, and traffic that OpenAI does. I wouldn't draw a comparison between the size and efficacy of the employees, but again, Google operates at a very different scale, and has operated at that scale very well.

Also anecdotally, half the time I've tried to use ChatGPT it's "at capacity" or throws an internal error, and I've also seen Dall-E unavailable even though I've barely tried to use it, so I wouldn't say that OpenAI service has been ironclad this whole time.

Re: OpenAI Status: Multiple engines are down

#126

Post by spaceman_2020 »

TOMDM wrote: post #14

LLM's are incredibly expensive to run. I imagine the huge demand that ChatGPT is seeing would make any cloud vendor sweat if you were to suddenly lump it on top of the usual demand. To me it's entirely unsurprising that OpenAI would have trouble keeping up. Good luck to them.

Is this something that can come down with better tech (both hardware and software) or is the high cost just baked into LLMs?

Re: OpenAI Status: Multiple engines are down

#127

Post by TOMDM »

somenameforme wrote: post #113
TOMDM wrote: post #80

Earlier quoted context omitted.

I'd agree if Apple were in the business of making datacentre infrastructure. Nobody is running these large scale models on their personal devices. Sure, some of the image generation tech is seeing personal use, so you'd have a point there, but these immense language models are something else entirely.

GLM-130B[1] (a 130 billion parameter model vs GPT-3's 175 billion parameter model) is able to run optimally on consumer level high-end hardware, 4xRTX 3090 in particular. That's It's a whole lot cheaper to run neural net style systems than to train them. "Somebody on Twitter"[2] got it setup, and broke down the costs, demonstrated some prompts, and what not. Cliff notes being a fraction of a penny per query, with eac…

Apples hardware, and 4 RTX 3090's being able to run a GPT scale model quantized to 4 byte ints are worlds apart.

Who knows though, maybe someone manages to get 4byte quantization producing good results and Apple makes a chip that can do the required ops for whatever that looks like with ~100GB of memory attached and then GP's comment might be relevant.

I started this reply rather skeptical, but with the boundaries Apple has been pushing, and the pace of AI research, honestly who knows.

Re: OpenAI Status: Multiple engines are down

#128

Post by ipython »

cellis wrote: post #38
la64710 wrote: post #23

Earlier quoted context omitted.

ChatGPT might but The LLLM model may not require so many nodes specially if someone is running for their own dataset.

GPT NeoX (open source) requires at least 45GB of GPU ram. So no, you're not running it on your laptop.

Wouldn’t that run on a new MacBook Pro m2 with 96gb of unified ram?

Re: OpenAI Status: Multiple engines are down

#129

Post by d2049 »

152334H wrote: post #4

With all of these outages, you have to wonder if it's a lack of skilled engineers on their part, or if they simply don't have enough GPUs to keep the lights on all the time.

Many companies have skilled engineers and enough hardware, and still have outages. It's an expected part of running any software service. Outages can be minimized, but the idea that they can be eliminated with enough people and computing resources is not how things work, from my experience.

Re: OpenAI Status: Multiple engines are down

#130

Post by verdverm »

danielscrubs wrote: post #111
verdverm wrote: post #104

Earlier quoted context omitted.

It was the dashboard, applications depending on the database service were not impacted. You're comparing apples and oranges and ignoring both a much larger context and the nuances

And most of those customers are paying customers? All I'm saying is that some outages are to be expected from a young company and they are doing excellent work.

Are they doing excellent work? We all have raved about various startups and how great they were, until the externalities became widely understood. Is there something that excludes OpenAI from this possibility? My near term concern is that these language models are going to cause a host of issues for society. Botnets, misinformation, students using for homework, influence operations...

I believe trust in information is a bigger concern than global warming. As an aside, what is OpenAI's relative contribution to CO2 emissions compared with other companies of similar size?

Post reply on HN