Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Salesforce Global Outage

status.salesforce.com

Post reply on HN
Print view View original

121–130 of 185 posts

Re: Salesforce Global Outage

#121

Post by shimman »

eastbound wrote: post #95
shimman wrote: post #74

Earlier quoted context omitted.

This is the case for every single B2B saas product. This is like the "bar is rolling on the floor" level of competence required. Please have higher standards for paid products.

What do you think of Atlassian?

Terrible, I'd argue the vast majority of modern big tech offerings are extremely poor quality where the need for surveillance in the form of constant monitoring/advertising metrics deliberately makes these types of services more costly to maintain and repair over time.

Sure there are like 3 or 5 decent services out there (like S3) but the vast majority are over engineered to be user hostile while extracting out whatever resources they can from their customers.

Re: Salesforce Global Outage

#123

Post by dd8601fn »

WinstonSmith84 wrote: post #99
reddalo wrote: post #91

I can't understand how such a huge company can have such a lousy UX.

Peak Tech Salesforce was 2010 +/- 2 years - i.e. after Visualforce and before Aura era. It used to be a developer oriented platform and it became shiny/flashy garbage eventually. But all these shiny things allowed them to get a large market cap with very brilliant sales people, it's hard to deny.

VF and Aura overlapped. Aura was just a bad start and janky. We sometimes just did React instead, for a while.

LWC is worlds better. And the local tooling with the cli and VSCode extensions is miles better than the old Eclipse/Sublime FMT days.

Re: Salesforce Global Outage

#124

Post by toomuchtodo »

ocdtrekkie wrote: post #119
Anon1096 wrote: post #103

Earlier quoted context omitted.

Hacker News is much easier to read when you realize that 95% of people have never worked on a "high" (maybe we could say >1B requests per day as a starting point) scale distributed service and think it's trivial to run one with more than 2 nines. You see comments all the time here mentioning that their own desktop at home is achieving more than that which belies deep misunderstanding of how systems are measured. Or t…

> which belies deep misunderstanding I think you are missing the point. When I state my Exchange server is more reliable than Exchange Online, I don't think I'm a better engineer. I recognize Microsoft has harder problems to solve than I do. I think building overengineered, oversized SaaS environments is introducing extreme risk. It's an inherent flaw of the current approach. Smaller is, in fact, better, because it's…

Indeed, the scale Anon1096 refers to wrt distributed systems is anti pattern. It is designed to vacuum up revenue and create enterprise value with scale, not to create resiliency for customers (although resiliency might be a byproduct of a well architected and operated distributed system at scale).

"Simplicity is the ultimate sophistication." -- Da Vinci

Re: Salesforce Global Outage

#125

Post by jedberg »

ocdtrekkie wrote: post #119
Anon1096 wrote: post #103

Earlier quoted context omitted.

Hacker News is much easier to read when you realize that 95% of people have never worked on a "high" (maybe we could say >1B requests per day as a starting point) scale distributed service and think it's trivial to run one with more than 2 nines. You see comments all the time here mentioning that their own desktop at home is achieving more than that which belies deep misunderstanding of how systems are measured. Or t…

> which belies deep misunderstanding I think you are missing the point. When I state my Exchange server is more reliable than Exchange Online, I don't think I'm a better engineer. I recognize Microsoft has harder problems to solve than I do. I think building overengineered, oversized SaaS environments is introducing extreme risk. It's an inherent flaw of the current approach. Smaller is, in fact, better, because it's…

Is it? When your internet is out for five days because your ISP takes a few days to get to you, do you acknowledge that you're now at 98.5% availability for the year, far worse than any SaaS email service?

I think people forget that those large environments are there for a reason. To make sure the service stays up in the face of problems outside your own control.

Re: Salesforce Global Outage

#126

Post by Jach »

gibsonf1 wrote: post #104
stmw wrote: post #66

Despite all of the snark here, in my experience Salesforce SRE team is quite competent. The engineering challenges of running a large PaaS - not just with own apps, but with millions of customer-written apps running on it - are quite interesting, and sadly things happen. The status page makes sense to actual customers, it's the particular "pods" where a given service runs.

Hmm, could the use of genAI have anything to do with this failure and the inability to quickly fix it?

It's not impossible, but Salesforce has had big outages before LLMs. For a disruption that began at 1am pacific, the response time isn't that bad. 3 hours total to give up on restarts, 4 hours total to validate a quick fix and begin rollout, and the rest of the time since has been waiting for the rollout + addressing subsets of instances that had some issues with restarting+the quick fix. It's nearly 9am pacific now, so Dreamforce is saved~ (It's Dreamforce week this week. Most devs are either focused on that or on soft-vacation / working on lower priority non-feature-work items, it's surprising anything would be updated to production this week that could do this.) The architecture and approval process of everything there has long been setup so that things can't be changed quickly.

Re: Salesforce Global Outage

#127

Post by toomuchtodo »

jedberg wrote: post #125
ocdtrekkie wrote: post #119

Earlier quoted context omitted.

> which belies deep misunderstanding I think you are missing the point. When I state my Exchange server is more reliable than Exchange Online, I don't think I'm a better engineer. I recognize Microsoft has harder problems to solve than I do. I think building overengineered, oversized SaaS environments is introducing extreme risk. It's an inherent flaw of the current approach. Smaller is, in fact, better, because it's…

Is it? When your internet is out for five days because your ISP takes a few days to get to you, do you acknowledge that you're now at 98.5% availability for the year, far worse than any SaaS email service? I think people forget that those large environments are there for a reason. To make sure the service stays up in the face of problems outside your own control.

In my entire adult lifetime (mid 40s), my ISP has never been out for five days. Compare to Github, Microsoft, Salesforce, and AWS outages that are always occurring in some fashion. Reddit is down constantly in various ways and still continues to operate as a business, public no less, so I disagree about the need to chase five nines and broadly speaking, large distributed systems that are potentially unnecessary for the use case and target outcome.

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

Re: Salesforce Global Outage

#128

Post by troyvit »

mulmen wrote: post #73
stmw wrote: post #66

Despite all of the snark here, in my experience Salesforce SRE team is quite competent. The engineering challenges of running a large PaaS - not just with own apps, but with millions of customer-written apps running on it - are quite interesting, and sadly things happen. The status page makes sense to actual customers, it's the particular "pods" where a given service runs.

I honestly don’t get the snark. The status page has: Seemingly meaningful IDs Search Region filter Email update signup Predictable URLs for instance status so they can be deep linked in runbooks What appears to be the actual live instance status . What appears to be the actual live service status in each instance. An update log with frequent detailed updates.

I despise Salesforce, but when I landed on this page I was like, huh. wow. honesty. Looks at GitHub

So yeah you're exactly right, the snark is not deserved if you ask me, and I'm 82% snark.

Re: Salesforce Global Outage

#129

Post by jedberg »

toomuchtodo wrote: post #127
jedberg wrote: post #125

Earlier quoted context omitted.

Is it? When your internet is out for five days because your ISP takes a few days to get to you, do you acknowledge that you're now at 98.5% availability for the year, far worse than any SaaS email service? I think people forget that those large environments are there for a reason. To make sure the service stays up in the face of problems outside your own control.

In my entire adult lifetime (mid 40s), my ISP has never been out for five days. Compare to Github, Microsoft, Salesforce, and AWS outages that are always occurring in some fashion. Reddit is down constantly in various ways and still continues to operate as a business, public no less, so I disagree about the need to chase five nines and broadly speaking, large distributed systems that are potentially unnecessary for t…

Consider yourself lucky that you’ve never been the victim of a fiber cut. But what about if the power to your house goes out? Or what if your server blows the power supply?

My entire point is that you have no redundancy in your system and you also aren’t big enough to have any pull with the vendors who can fix these types of outages so you’re basically at the mercy of your providers with no recourse.

That’s why these systems are built the way they are.

And generally four nines is considered the gold standard these days. I can tell you for sure that both Netflix and Ebay would lose money anytime they drop below four nines because I have at some point been responsible for both. You’re correct that Reddit has a lot more leeway and outage time before they start losing money but not that much leeway.

Re: Salesforce Global Outage

#130

Post by temp_praneshp »

toomuchtodo wrote: post #127
jedberg wrote: post #125

Earlier quoted context omitted.

Is it? When your internet is out for five days because your ISP takes a few days to get to you, do you acknowledge that you're now at 98.5% availability for the year, far worse than any SaaS email service? I think people forget that those large environments are there for a reason. To make sure the service stays up in the face of problems outside your own control.

In my entire adult lifetime (mid 40s), my ISP has never been out for five days. Compare to Github, Microsoft, Salesforce, and AWS outages that are always occurring in some fashion. Reddit is down constantly in various ways and still continues to operate as a business, public no less, so I disagree about the need to chase five nines and broadly speaking, large distributed systems that are potentially unnecessary for t…

Curious, what AWS outage has affected you for days?

(I hope you'll agree that the middle east outage is a true outlier)

Post reply on HN