Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Salesforce Global Outage

status.salesforce.com

Post reply on HN
Print view View original

171–180 of 184 posts

Re: Salesforce Global Outage

#171

Post by fragmede »

DANmode wrote: post #169
lifeisstillgood wrote: post #152

Earlier quoted context omitted.

I think it’s like advertising - 50% of my code is wildly over complicated - I just don’t know which 50% But the GP is essentially correct - there is a 2% of salesforce that could be built run and keep 80% of salesforce users happy. Except that you could not charge enough to be able to advertise on F1 cars and take SVPs out to dinner. So you could not actually make 80% of them happy - they would ever buy it.

> I just don’t know which 50% Yes, you largely do - they’re the commits that get rushed to, and through. This take that showstopping technical debt is unavoidable is very new, and will age like milk.

> showstopping technical debt is unavoidable is very new

No it's not. The push and pull between shipping and paying down technical debt is as old as there's been software to sell. Sales has been selling features that don't exist quite yet ever since they've been talking to customers, and engineering has been pushing back on implementing them yesterday since there's been features to implement. Showstopping technical debt is merely a side effect of who wins that argument in a given org.

Re: Salesforce Global Outage

#172

Post by gregw2 »

karmakaze wrote: post #102
ramesh31 wrote: post #68

Earlier quoted context omitted.

Probably not a coincidence

Yup. Could be that everyone was rushing to get all their products and demos ready leading up to it.

AWS has a history of this before re:Invents (its annual big conference) in my experience.

They've gotten a little better in recent years.

Re: Salesforce Global Outage

#173

Post by bobsmooth »

stefanfisk wrote: post #135
dd8601fn wrote: post #116

Earlier quoted context omitted.

There are 10,000 “simple inexpensive replacements”. Always have been. If you need a glorified three object contact list… you shouldn’t buy Salesforce. Those just, obviously, can’t do almost any of the forty million serious things that Salesforce does, and real businesses do need.

This is an honest question: which serious things?

My boss recently added a custom integration to allow us to log hours on jobs. It includes GPS coordinates so we have proof of our whereabouts.

Re: Salesforce Global Outage

#174

Post by abeyer »

WinstonSmith84 wrote: post #166
prettychill wrote: post #164

Earlier quoted context omitted.

aura was open source for a long time. https://github.com/forcedotcom/aura

That's not what I meant. Salesforce had a choice to use (and support) battle tested frameworks, and decided instead to build their own one.

To be fair, they came in with some requirements that I don't think anyone else had at the time, and even today aren't in any mainstream ones afaik.

I think the biggest difference was around providing security and stability barriers between front-end components on the same page, with the intent of allowing you to compose a page that contains your own components and those of other third party applications you've installed with guarantees about how they can (and can't) interact. Not sure they couldn't have tacked that onto another framework, but it comes with enough trade-offs and compromises that I'm not sure anyone else would have wanted to upstream it, so they would have been forking something anyway.

Aura wasn't much fun to work with, was never really feature complete, and not advocating for it... but it actually kind of made sense if you thought about front end with the context of how salesforce did security and multitenancy in mind.

Re: Salesforce Global Outage

#175

Post by ocdtrekkie »

b112 wrote: post #90
cube00 wrote: post #54

Earlier quoted context omitted.

Kind of surprised they admit they're going to try restarting and see what happens. I'm sure it happens everywhere but nobody admits it. > We've attempted a rolling restart on one of the impacted instances to see if that resolves the issue. At least it didn't fix the problem so they can actually start finding the real cause. > We're no longer pursuing restarts as a path to remediation. Why isn't the AI they sell telli…

Restart should be a very last emergency step, as if it works, a restart often might wipe out evidence of why. So hopefully it's not done often.

Rebooting is the first step: If it fixes it you don't have a problem. If it doesn't, you know more about the problem.

It's a joke, but like, after nearly two decades of engineering I have something break on me, due to updates. I figure the updates broke it, call the vendor, and they go... did you try rebooting it again?

Rebooting it a second time fixed it.

Re: Salesforce Global Outage

#176

Post by SaucyWrong »

dehrmann wrote: post #137
ramesh31 wrote: post #68

Earlier quoted context omitted.

Probably not a coincidence

Most places at this scale have code freezes in place well before conferences. The most likely issues are some launch couldn't handle the scale or periodic deployments have been saving them from some sort of long-standing leak bug, and pausing going into Dreamforce meant some service hasn't been restarted in a week. Historically, Salesforce sharded by customer, so that goes against both of these, unless it's in a rout…

I'm an ex-Salesforce, and yes, at the time I left, there was a huge change freeze surrounding Dreamforce. Unless a demo of an announced feature was coming in really at the buzzer, change velocity would have been low since a few weeks ago. I worked in a sub-cloud, so I can't even speculate as to the reason for the failure.

Something I wonder about is whether SRE responses were delayed due to having to be emergency-change-approved because Dreamforce was on. I don't recall a global outage ever occurring during a change freeze when I worked there, so /shrug.

Re: Salesforce Global Outage

#177

Post by SaucyWrong »

minimeow wrote: post #69
mergy wrote: post #51

Unplanned outage timing is never good but this is really not good. https://www.salesforce.com/dreamforce/ Sept 15-17

This is what happens when more than half the company is away attending the Salesforce cult-indoctrination stuff while spending all their bandwidth making customers/partners feel good.... The stuff that matters to keep the lights on gets overlooked.

I'm an ex-Salesforce engineer and I can attest that very few line-level engineers are ever invited to attend Dreamforce. This would have been an ordinary day at the office for the SRE team.

Re: Salesforce Global Outage

#178

Post by abeyer »

sparkling wrote: post #9

Remind me please, what are folks currently paying per seat for this glorified CRUD app?

There is a published sticker price, but I don't think anyone outside of tiny installs with just a few users pays that... it's very much the enterprise sales model of "let's schedule a call and talk about it." They also for many years would let previously negotiated prices stand during renewals even when sticker price went up, so older orgs often have the "same" license being charged at many different prices based on when it was first purchased and how the negotiations went at the time.

Re: Salesforce Global Outage

#179

Post by codeulike »

stmw wrote: post #66

Despite all of the snark here, in my experience Salesforce SRE team is quite competent. The engineering challenges of running a large PaaS - not just with own apps, but with millions of customer-written apps running on it - are quite interesting, and sadly things happen. The status page makes sense to actual customers, it's the particular "pods" where a given service runs.

Yes, the thing Salesforce are good at, and is little understood here, is that they've kept their platform online for 27 years so far. Its constantly evolving, three upgrades per year, but changes that require customers to change their customisations are rare, and when they happen they are communicated at least a year in advance. Approx 150,000 tenants, all with different configurations and some so heavily customized that they are effectively unique apps. Salesforce keeps them all online and evolving. In those 27 years there hasnt been a 'lets trash this and rewrite from scratch' and there hasn't been a 'you must migrate your data to our new platform, we're closing the old one'. They've just evolved it while running. They must have got some things very right in the original architecture to be able to do that.

One thing that I find interesting is that they launched their platform language Apex (a sortof subset of Java) in 2007 when TDD was the hot new thing, so TDD is baked into the platform - your Apex code must have at least 75% test coverage, and the tests must pass, before you are allowed to deploy to prod.

They leverage that test coverage when they are upgrading the platform - they have an internal process called The Hammer where they run all customer-created tests against customers own unique configs on the current platform and then again on the next version of the platform to see if any customer tests are being broken. Look it up, its really interesting.

Re: Salesforce Global Outage

#180

Post by supriyo-biswas »

torginus wrote: post #155
Anon1096 wrote: post #103

Earlier quoted context omitted.

Hacker News is much easier to read when you realize that 95% of people have never worked on a "high" (maybe we could say >1B requests per day as a starting point) scale distributed service and think it's trivial to run one with more than 2 nines. You see comments all the time here mentioning that their own desktop at home is achieving more than that which belies deep misunderstanding of how systems are measured. Or t…

Well ackchually.. I get that large scale systems pose their own challenges on their own, but it also matters what's the smallest isolable unit. What I mean by this is a CDN consists of nodes that are horizontally replicable and don't really talk to each other, and thus are easy to run even at scale. In contrast, something like a bank or social media isn't really reducible - every user needs to be able to interact wit…

> What I mean by this is a CDN consists of nodes that are horizontally replicable and don't really talk to each other, and thus are easy to run even at scale.

They do though! They mostly try to avoid it since hitting the network to serve any kind of latency would unacceptably increase latency, but you wildly underestimated the amount of complexity there is to running a CDN.

Post reply on HN