Live data from Hacker News

Salesforce Global Outage

status.salesforce.com

71–80 of 184 posts

Re: Salesforce Global Outage

#71
post #12

Could it be people doing Claude/GPT automations and they just can't handle it?

Could be. With a big conference going on and the updates mentioning resource exhaustion it could be a bunch of people doing demos, AI driven or not. Basically slashdotting themselves.

Re: Salesforce Global Outage

#72
post #51

Unplanned outage timing is never good but this is really not good. https://www.salesforce.com/dreamforce/ Sept 15-17

This is what happens when more than half the company is away attending the Salesforce cult-indoctrination stuff while spending all their bandwidth making customers/partners feel good.... The stuff that matters to keep the lights on gets overlooked.

Arguably, making customers and partners feel good is the more important part of the business

Re: Salesforce Global Outage

#73
post #66

Despite all of the snark here, in my experience Salesforce SRE team is quite competent. The engineering challenges of running a large PaaS - not just with own apps, but with millions of customer-written apps running on it - are quite interesting, and sadly things happen. The status page makes sense to actual customers, it's the particular "pods" where a given service runs.

I honestly don’t get the snark. The status page has:

Seemingly meaningful IDs

Search

Region filter

Email update signup

Predictable URLs for instance status so they can be deep linked in runbooks

What appears to be the actual live instance status.

What appears to be the actual live service status in each instance.

An update log with frequent detailed updates.

Re: Salesforce Global Outage

#74
post #66

Despite all of the snark here, in my experience Salesforce SRE team is quite competent. The engineering challenges of running a large PaaS - not just with own apps, but with millions of customer-written apps running on it - are quite interesting, and sadly things happen. The status page makes sense to actual customers, it's the particular "pods" where a given service runs.

This is the case for every single B2B saas product. This is like the "bar is rolling on the floor" level of competence required. Please have higher standards for paid products.

Re: Salesforce Global Outage

#76

Earlier quoted context omitted.

This is what happens when more than half the company is away attending the Salesforce cult-indoctrination stuff while spending all their bandwidth making customers/partners feel good.... The stuff that matters to keep the lights on gets overlooked.

Arguably, making customers and partners feel good is the more important part of the business

They wont feel good if the product they pay for doesnt work

Re: Salesforce Global Outage

#77

Earlier quoted context omitted.

This is what happens when more than half the company is away attending the Salesforce cult-indoctrination stuff while spending all their bandwidth making customers/partners feel good.... The stuff that matters to keep the lights on gets overlooked.

Arguably, making customers and partners feel good is the more important part of the business

arguably, this is what sales cares about and the half-measures taken to tackle what must be the Mt. Everest of tech debt at Salesforce is what leads to large, systemically degraded customer trust in products that keep shipping bugs

Re: Salesforce Global Outage

#78
post #54

Have you tried turning it off and then on again? > We're no longer pursuing restarts as a path to remediation. Oh you have

Kind of surprised they admit they're going to try restarting and see what happens. I'm sure it happens everywhere but nobody admits it. > We've attempted a rolling restart on one of the impacted instances to see if that resolves the issue. At least it didn't fix the problem so they can actually start finding the real cause. > We're no longer pursuing restarts as a path to remediation. Why isn't the AI they sell telli…

In my experience it’s a safe way to do something useful while everyone is getting their bearings. It immediately partitions the situation space between being persisted or systemic vs local or caused by long-running processes. Plus everyone’s going to ask if you’ve tried that already, so you might as well get it out of the way if it makes any amount of sense

Re: Salesforce Global Outage

#79
Kind of ironic. Salesforce is basically one of the major spiritual grandfathers of Slop. It is not uncommon in production systems to find that objects like Contact and Account have hundreds of custom fields. Sometimes, you find out that several of them have the same meaning and semantics, but were used at different times. Digging out you discover that some Marketing guy that used to work at the company did some task in a certain way that was lost when he was gone, and then a few months later his substitute had the same need and went ahead and created the same field with a slightly different name.

Doing data engineering work with Salesforce data is an exercise on archeology, psychology and organizational politics.

Slop is basically the ontological and teleological philosophy behind Salesforce very existence. Despite the official discourse that the "No Software" meant no infrastructure, no toil with updates and configuration, the subtext as intended for executives was very clear: "No need for you to be blocked by those pricks from engineering and their stupid, bureaucratic and gatekeeping rules".

"No software" was a call-to-arms to a certain subset of managers that were radicalized by Nicholas Carr's 2023 HBR article "IT Doesn't matter". It doesn't matter that Carr was a journalist and a writer with a masters in English that has never ever run even a small bodega, or has never managed an IT department. Anti-intellectualism and the abundance of capital brought in by the petrodollar that allowed the US government to run deficits year by year while exporting the ensuing inflationary effects to rest of world, would ensure that this message would ressonate and then even be amplified during the years of ZIRP and the Baillouts. Play fast and loose, first come, first served, a rising tide rises all boats and all that jazz. Wall Street favors bold, and the heck with the long term! This quarter will only live once!

Frankly, this is just poetic justice: Kill by slop, be killed by slop.

Re: Salesforce Global Outage

#80

Ah yes. Exactly what a status page should look like: an endless list of random ID’s that don’t mean anything and no information whatsoever At least salesforce is consistent with their design language

Random Ids? If you mean the “USA324” ones, those are pods. If you’re a customer you know which one(s) you care about.
Post reply on HN