Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Salesforce Global Outage

status.salesforce.com

Post reply on HN
Print view View original

161–170 of 185 posts

Re: Salesforce Global Outage

#161

Post by chias »

Semaphor wrote: post #153
jedberg wrote: post #129

Earlier quoted context omitted.

Consider yourself lucky that you’ve never been the victim of a fiber cut. But what about if the power to your house goes out? Or what if your server blows the power supply? My entire point is that you have no redundancy in your system and you also aren’t big enough to have any pull with the vendors who can fix these types of outages so you’re basically at the mercy of your providers with no recourse. That’s why these…

I mean, if you really need redundancy, isn’t a second instance on a VPS somewhere that you manually switch over to, enough? Over multiple ISPs, so far internet outages for more then a few minutes is very rare (though the few minutes would make me not want to host something requiring high availability; and a cut cable is really annoying because there simply is no quick fix), power outages even rares, I experienced 3 i…

Generally, no.

Ignoring for now how you are synchronizing the database and filesystem, and how doing so may well result in your duplicate experiencing the same failure as the original, you can maybe recover from a small class of availability issues that could knock you out of an SLA.

But that assumes you can get online and can fully orchestrate the transition within less than 53 minutes of it starting. Including the time you took to become aware of it. And including the time to diagnose and decide that a switchover would resolve the problem. Including the time it takes for DNS caches to expire and point to the new host. Including the DNS caches which may ignore your TTL. And including all these things again when you switch back.

And assuming, of course, that it doesn't happen again for a whole year.

Re: Salesforce Global Outage

#162

Post by Bluestein »

sllabres wrote: post #158
Bluestein wrote: post #24

Earlier quoted context omitted.

"Make a salesforce clone but with 100% uptime" ⎿ You've hit your session limit · resets 2:53am (48°52.6′S, 123°23.6′W Etc/GMT+8) /upgrade to increase your usage limit.

Point Nemo :-D https://en.wikipedia.org/wiki/Pole_of_inaccessibility

Finally! You win the Internet today good sir! (At least as far as I am concerned :)

Re: Salesforce Global Outage

#163

Post by r3trohack3r »

torginus wrote: post #155
Anon1096 wrote: post #103

Earlier quoted context omitted.

Hacker News is much easier to read when you realize that 95% of people have never worked on a "high" (maybe we could say >1B requests per day as a starting point) scale distributed service and think it's trivial to run one with more than 2 nines. You see comments all the time here mentioning that their own desktop at home is achieving more than that which belies deep misunderstanding of how systems are measured. Or t…

Well ackchually.. I get that large scale systems pose their own challenges on their own, but it also matters what's the smallest isolable unit. What I mean by this is a CDN consists of nodes that are horizontally replicable and don't really talk to each other, and thus are easy to run even at scale. In contrast, something like a bank or social media isn't really reducible - every user needs to be able to interact wit…

> What I mean by this is a CDN consists of nodes that are horizontally replicable and don't really talk to each other, and thus are easy to run even at scale.

This is only true if you exclude problems like “finding a CDN node from the device,” “managing congestion,” etc. as part of the problem statement

Re: Salesforce Global Outage

#164

Post by prettychill »

WinstonSmith84 wrote: post #144
dd8601fn wrote: post #123

Earlier quoted context omitted.

VF and Aura overlapped. Aura was just a bad start and janky. We sometimes just did React instead, for a while. LWC is worlds better. And the local tooling with the cli and VSCode extensions is miles better than the old Eclipse/Sublime FMT days.

Aura existed simply because Salesforce thought to be smarter than open source, well. Can't deny it though: Salesforce engineers were great on the backend, but frontend dev has never been their thing. Back then there was Angular 1 which was miles ahead. React was released shortly after Aura itself, so to say. LWC is what Aura shall have been 10+ years ago. And that ties back to what OP wrote: awful UX extremely slow b…

aura was open source for a long time. https://github.com/forcedotcom/aura

Re: Salesforce Global Outage

#165

Post by stmw »

trebligdivad wrote: post #111
stmw wrote: post #66

Despite all of the snark here, in my experience Salesforce SRE team is quite competent. The engineering challenges of running a large PaaS - not just with own apps, but with millions of customer-written apps running on it - are quite interesting, and sadly things happen. The status page makes sense to actual customers, it's the particular "pods" where a given service runs.

What I'm curious about is why it is a single-PaaS; I'd have expected Salesforce to have the customers quite isolated so the chance of bringing down multiple customers at once was much smaller.

The customers are quite isolated, but it doesn't mean that some services or errors do not propagate. In public cloud terms, think back on some AWS or Azure or even Gmail outages - you probably wouldn't even hear about them if it didn't affect millions of users at once, across security and availability boundaries.

Re: Salesforce Global Outage

#166

Post by WinstonSmith84 »

prettychill wrote: post #164
WinstonSmith84 wrote: post #144

Earlier quoted context omitted.

Aura existed simply because Salesforce thought to be smarter than open source, well. Can't deny it though: Salesforce engineers were great on the backend, but frontend dev has never been their thing. Back then there was Angular 1 which was miles ahead. React was released shortly after Aura itself, so to say. LWC is what Aura shall have been 10+ years ago. And that ties back to what OP wrote: awful UX extremely slow b…

aura was open source for a long time. https://github.com/forcedotcom/aura

That's not what I meant. Salesforce had a choice to use (and support) battle tested frameworks, and decided instead to build their own one.

Re: Salesforce Global Outage

#167

Post by prettychill »

WinstonSmith84 wrote: post #166
prettychill wrote: post #164

Earlier quoted context omitted.

aura was open source for a long time. https://github.com/forcedotcom/aura

That's not what I meant. Salesforce had a choice to use (and support) battle tested frameworks, and decided instead to build their own one.

I think around 2012 there was not a lot of options for an enterprise rally around. Aura was designed and built in the same timeframe as react/angular/ember/etc iirc.

Re: Salesforce Global Outage

#169

Post by DANmode »

lifeisstillgood wrote: post #152
pixl97 wrote: post #149

Earlier quoted context omitted.

>Salesforce is not simple. It's wildly, overly complex. It turns out business environments are wildly overly complex.

I think it’s like advertising - 50% of my code is wildly over complicated - I just don’t know which 50% But the GP is essentially correct - there is a 2% of salesforce that could be built run and keep 80% of salesforce users happy. Except that you could not charge enough to be able to advertise on F1 cars and take SVPs out to dinner. So you could not actually make 80% of them happy - they would ever buy it.

> I just don’t know which 50%

Yes, you largely do - they’re the commits that get rushed to, and through.

This take that showstopping technical debt is unavoidable is very new, and will age like milk.

Re: Salesforce Global Outage

#170

Post by carefree-bob »

pixl97 wrote: post #149
leptons wrote: post #133

Earlier quoted context omitted.

While not a home-run server, the NTP system is a distributed service that receives 100 billion to trillions of requests per day, and it's running pretty smoothly - it's never gone down completely since it started in 1985. It's also very simple. The reason it has so many 9's uptime is because it is simple. Given a low amount of complexity, it's not unreasonable to think that an individual could run a >1B requests per…

>Salesforce is not simple. It's wildly, overly complex. It turns out business environments are wildly overly complex.

It's the scaling nature of enterprise software. If you have a mature B2C app, you have millions of users. What each user wants isn't so important, so it's more of a take it or leave it experience. If you have an enterprise app, one big company can and does push you around to get their features in. And as you grow, you get a few hundred big companies that push you around. The result is this huge mess of features, and now you have to maintain this mess.

I remember Cisco before iOS used to have hundreds of branches for their router, one branch for each major customer that was demanding specific features. It was unmanageable, but that's what you needed to do to win those "enterprise customers".

It also turns out customers aren't very good at articulating their needs and putting them into a cohesive vision of the product. But they sure have specific demands to get stuff in. I'm not blaming the customer, this is just how this world works -- All of the "enterprise software" apps are extremely complex with hidden knobs and weird behavior that was pushed in by a customer twenty years ago all over the place.

Post reply on HN