Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Salesforce Global Outage

status.salesforce.com

Post reply on HN
Print view View original

131–140 of 185 posts

Re: Salesforce Global Outage

#131

Post by toomuchtodo »

jedberg wrote: post #129
toomuchtodo wrote: post #127

Earlier quoted context omitted.

In my entire adult lifetime (mid 40s), my ISP has never been out for five days. Compare to Github, Microsoft, Salesforce, and AWS outages that are always occurring in some fashion. Reddit is down constantly in various ways and still continues to operate as a business, public no less, so I disagree about the need to chase five nines and broadly speaking, large distributed systems that are potentially unnecessary for t…

Consider yourself lucky that you’ve never been the victim of a fiber cut. But what about if the power to your house goes out? Or what if your server blows the power supply? My entire point is that you have no redundancy in your system and you also aren’t big enough to have any pull with the vendors who can fix these types of outages so you’re basically at the mercy of your providers with no recourse. That’s why these…

I've contributed to building out data centers, as well as managed colos for others, primarily in downtown Chicago at Level3 and at 350 E Cermak. I am familiar with architecture required for reliability and diversity, from power and fiber in all the way up the stack to the Kubernetes cluster and software defined networking. If you participate in the capital markets, your data traverses systems I've participated in designing and implementing. There is a time and place for complexity (in this context, large/global distributed systems), but too often, complexity exists where it need not (imho).

"What are you optimizing for?" is always an important question, as is "The Five Whys."

Re: Salesforce Global Outage

#133

Post by leptons »

Anon1096 wrote: post #103
stmw wrote: post #66

Despite all of the snark here, in my experience Salesforce SRE team is quite competent. The engineering challenges of running a large PaaS - not just with own apps, but with millions of customer-written apps running on it - are quite interesting, and sadly things happen. The status page makes sense to actual customers, it's the particular "pods" where a given service runs.

Hacker News is much easier to read when you realize that 95% of people have never worked on a "high" (maybe we could say >1B requests per day as a starting point) scale distributed service and think it's trivial to run one with more than 2 nines. You see comments all the time here mentioning that their own desktop at home is achieving more than that which belies deep misunderstanding of how systems are measured. Or t…

While not a home-run server, the NTP system is a distributed service that receives 100 billion to trillions of requests per day, and it's running pretty smoothly - it's never gone down completely since it started in 1985. It's also very simple. The reason it has so many 9's uptime is because it is simple. Given a low amount of complexity, it's not unreasonable to think that an individual could run a >1B requests per day service.

Salesforce is not simple. It's wildly, overly complex. It's amazing it has any 9's at all and not 8's or 7's. Salesforce offers three 9's, which allows for 43 minutes downtime per month. The current outage is at 8 hours (and counting) so Salesforce is now at 98.9% uptime for the month - there's an "8" in there now. Not good, but considering the complexity of Salesforce, it's still kind of amazing.

Re: Salesforce Global Outage

#134

Post by ocdtrekkie »

jedberg wrote: post #129
toomuchtodo wrote: post #127

Earlier quoted context omitted.

In my entire adult lifetime (mid 40s), my ISP has never been out for five days. Compare to Github, Microsoft, Salesforce, and AWS outages that are always occurring in some fashion. Reddit is down constantly in various ways and still continues to operate as a business, public no less, so I disagree about the need to chase five nines and broadly speaking, large distributed systems that are potentially unnecessary for t…

Consider yourself lucky that you’ve never been the victim of a fiber cut. But what about if the power to your house goes out? Or what if your server blows the power supply? My entire point is that you have no redundancy in your system and you also aren’t big enough to have any pull with the vendors who can fix these types of outages so you’re basically at the mercy of your providers with no recourse. That’s why these…

I've dealt with a fiber cut, it wasn't nearly that bad. Fiber cuts impacting my SaaS providers were worse because there was nothing I could do about it.

Re: Salesforce Global Outage

#135

Post by stefanfisk »

dd8601fn wrote: post #116
cmiles8 wrote: post #105

Salesforce has turned into the monolithic messy bloatware it set out to replace. Please VCs stop with the AI FOMO and find a few good startups to just go destroy Salesforce and give folks a simple inexpensive replacement.

There are 10,000 “simple inexpensive replacements”. Always have been. If you need a glorified three object contact list… you shouldn’t buy Salesforce. Those just, obviously, can’t do almost any of the forty million serious things that Salesforce does, and real businesses do need.

This is an honest question: which serious things?

Re: Salesforce Global Outage

#136

Post by xboxnolifes »

jackdecker wrote: post #25

Ah yes. Exactly what a status page should look like: an endless list of random ID’s that don’t mean anything and no information whatsoever At least salesforce is consistent with their design language

This might actually be one of the most useful status pages ive seen. Its not just random green tick marks representing the entire service that only change yellow when someone gives and admits that 5 hours of bad service is an outage.

Re: Salesforce Global Outage

#137

Post by dehrmann »

ramesh31 wrote: post #68
mergy wrote: post #51

Unplanned outage timing is never good but this is really not good. https://www.salesforce.com/dreamforce/ Sept 15-17

Probably not a coincidence

Most places at this scale have code freezes in place well before conferences. The most likely issues are some launch couldn't handle the scale or periodic deployments have been saving them from some sort of long-standing leak bug, and pausing going into Dreamforce meant some service hasn't been restarted in a week. Historically, Salesforce sharded by customer, so that goes against both of these, unless it's in a routing layer.

Re: Salesforce Global Outage

#138

Post by IneffablePigeon »

sschueller wrote: post #107
cmiles8 wrote: post #105

Salesforce has turned into the monolithic messy bloatware it set out to replace. Please VCs stop with the AI FOMO and find a few good startups to just go destroy Salesforce and give folks a simple inexpensive replacement.

As someone who has never used Salesforce nor hubspot. What are the core features that has people using these services? Is it just the integration between all the different areas where customer data lives?

They're an incredible marketing machine.

But mostly it's a mix of the integration network effects you mention, cost of reimplementation if you want to leave, and the good old "nobody gets fired for buying IBM" dynamic.

Re: Salesforce Global Outage

#139

Post by a_conservative »

toomuchtodo wrote: post #124
ocdtrekkie wrote: post #119

Earlier quoted context omitted.

> which belies deep misunderstanding I think you are missing the point. When I state my Exchange server is more reliable than Exchange Online, I don't think I'm a better engineer. I recognize Microsoft has harder problems to solve than I do. I think building overengineered, oversized SaaS environments is introducing extreme risk. It's an inherent flaw of the current approach. Smaller is, in fact, better, because it's…

Indeed, the scale Anon1096 refers to wrt distributed systems is anti pattern. It is designed to vacuum up revenue and create enterprise value with scale, not to create resiliency for customers (although resiliency might be a byproduct of a well architected and operated distributed system at scale). "Simplicity is the ultimate sophistication." -- Da Vinci

Hidden in this discussion around self-hosting reliability are other options as well.

Depending on your time and appetite for tinkering with all of this, it's not hard to imagine a home setup that fails over to a cheap Hetzner or DO VM. A manual failover at the DNS level isn't overly complex, and could be scripted.

Keeping a database in sync between home and the instance might be simple or more complex depending on needs, but would it really be that hard to have Claude help you setup a replicating Postgres server? If your database (or data files) are 1 gigabyte and don't update that often... maybe just rsync it every night or something

There's a thread you and others are pulling on here, and we need to pull it. Hosting doesn't have to be the domain of the big vendors anymore.

Re: Salesforce Global Outage

#140

Post by SoftTalker »

cube00 wrote: post #54
raffraffraff wrote: post #47

Have you tried turning it off and then on again? > We're no longer pursuing restarts as a path to remediation. Oh you have

Kind of surprised they admit they're going to try restarting and see what happens. I'm sure it happens everywhere but nobody admits it. > We've attempted a rolling restart on one of the impacted instances to see if that resolves the issue. At least it didn't fix the problem so they can actually start finding the real cause. > We're no longer pursuing restarts as a path to remediation. Why isn't the AI they sell telli…

In my experience this is very common on linux hosts. Not that you would do a full system reboot as a generic first attempt (this was much more common when I worked with Windows) but restarting a wedged or misbehaving service/daemon is a pretty common thing.
Post reply on HN