Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Salesforce Global Outage

status.salesforce.com

Post reply on HN
Print view View original

141–150 of 185 posts

Re: Salesforce Global Outage

#141

Post by toomuchtodo »

a_conservative wrote: post #139
toomuchtodo wrote: post #124

Earlier quoted context omitted.

Indeed, the scale Anon1096 refers to wrt distributed systems is anti pattern. It is designed to vacuum up revenue and create enterprise value with scale, not to create resiliency for customers (although resiliency might be a byproduct of a well architected and operated distributed system at scale). "Simplicity is the ultimate sophistication." -- Da Vinci

Hidden in this discussion around self-hosting reliability are other options as well. Depending on your time and appetite for tinkering with all of this, it's not hard to imagine a home setup that fails over to a cheap Hetzner or DO VM. A manual failover at the DNS level isn't overly complex, and could be scripted. Keeping a database in sync between home and the instance might be simple or more complex depending on ne…

That was my intent, pull the thread.

Re: Salesforce Global Outage

#142

Post by karagenit »

gibsonf1 wrote: post #104
stmw wrote: post #66

Despite all of the snark here, in my experience Salesforce SRE team is quite competent. The engineering challenges of running a large PaaS - not just with own apps, but with millions of customer-written apps running on it - are quite interesting, and sadly things happen. The status page makes sense to actual customers, it's the particular "pods" where a given service runs.

Hmm, could the use of genAI have anything to do with this failure and the inability to quickly fix it?

The bug was from 2009, so probably not :)

Re: Salesforce Global Outage

#143

Post by CoffeeOnWrite »

b112 wrote: post #90
cube00 wrote: post #54

Earlier quoted context omitted.

Kind of surprised they admit they're going to try restarting and see what happens. I'm sure it happens everywhere but nobody admits it. > We've attempted a rolling restart on one of the impacted instances to see if that resolves the issue. At least it didn't fix the problem so they can actually start finding the real cause. > We're no longer pursuing restarts as a path to remediation. Why isn't the AI they sell telli…

Restart should be a very last emergency step, as if it works, a restart often might wipe out evidence of why. So hopefully it's not done often.

I wouldn't say so, rather you need to balance recovery time and evidence preservation. A good incident manager will give the service owning team a chance or two to debug, but not let them fall into the trap of needing to understand the problem fully before attempt a clumsy potential fix. And of course will take into account the total business impact of the ongoing disruption and the known and unknown risks of the proposed clumsy fix (it could make things worse).

Re: Salesforce Global Outage

#144

Post by WinstonSmith84 »

dd8601fn wrote: post #123
WinstonSmith84 wrote: post #99

Earlier quoted context omitted.

Peak Tech Salesforce was 2010 +/- 2 years - i.e. after Visualforce and before Aura era. It used to be a developer oriented platform and it became shiny/flashy garbage eventually. But all these shiny things allowed them to get a large market cap with very brilliant sales people, it's hard to deny.

VF and Aura overlapped. Aura was just a bad start and janky. We sometimes just did React instead, for a while. LWC is worlds better. And the local tooling with the cli and VSCode extensions is miles better than the old Eclipse/Sublime FMT days.

Aura existed simply because Salesforce thought to be smarter than open source, well. Can't deny it though: Salesforce engineers were great on the backend, but frontend dev has never been their thing. Back then there was Angular 1 which was miles ahead. React was released shortly after Aura itself, so to say. LWC is what Aura shall have been 10+ years ago. And that ties back to what OP wrote: awful UX extremely slow bloated with JS.

Now, don't talk me about VSCode Extensions. This is the perfect example of an awful dev experience. apex-jorje-lsp.jar with a JVM to parse Apex taking GB of memories, extensions taking dozens of seconds to load (when they load) ... In fact, the only decent LSP is aer, a simple decently working Go binary rather than the monster Salesforce shipped. The one good tooling Salesforce built in the last 15 years is, to some extent, the SF CLI - which came after the `force` CLI from the same guys who built `aer`, anyway. And nowadays, people can use that with their preferred editor from Zed to Vim with shortcuts from built upon the SF CLI.

So no, Salesforce didn't do great with tooling, they just did the bare minimum waiting on the (small) community to give them the right ideas.

Re: Salesforce Global Outage

#145

Post by elzbardico »

gregw2 wrote: post #98
elzbardico wrote: post #79

Kind of ironic. Salesforce is basically one of the major spiritual grandfathers of Slop. It is not uncommon in production systems to find that objects like Contact and Account have hundreds of custom fields. Sometimes, you find out that several of them have the same meaning and semantics, but were used at different times. Digging out you discover that some Marketing guy that used to work at the company did some task…

Typo: Carr wrote that in 2003, not 2023 for those of you who missed the foolishness and insane wreckage that article caused. The lost business value and competitiveness caused by outsourcing IT overseas to unmotivated parties under Carr's premise is hard to put your finger on but I have seen the aftermath and it's pretty massive.

Thanks for catching the typo!

Re: Salesforce Global Outage

#146

Post by elzbardico »

schnevets wrote: post #89
elzbardico wrote: post #79

Kind of ironic. Salesforce is basically one of the major spiritual grandfathers of Slop. It is not uncommon in production systems to find that objects like Contact and Account have hundreds of custom fields. Sometimes, you find out that several of them have the same meaning and semantics, but were used at different times. Digging out you discover that some Marketing guy that used to work at the company did some task…

Something tells me Troy the Salesforce Admin/BD Analyst did not cause the SAAS infrastructure to go down. And I think you're confusing crud with slop.

I didn't say anything even close to that.

Re: Salesforce Global Outage

#147

Post by reaperducer »

raffraffraff wrote: post #47

Have you tried turning it off and then on again? > We're no longer pursuing restarts as a path to remediation. Oh you have

Have you tried turning it off and then on again?

Flashbacks to "Thank you for calling Three-Ten-DELL. Have you tried turning it of and turning it back on again?"

Re: Salesforce Global Outage

#148

Post by b112 »

CoffeeOnWrite wrote: post #143
b112 wrote: post #90

Earlier quoted context omitted.

Restart should be a very last emergency step, as if it works, a restart often might wipe out evidence of why. So hopefully it's not done often.

I wouldn't say so, rather you need to balance recovery time and evidence preservation. A good incident manager will give the service owning team a chance or two to debug, but not let them fall into the trap of needing to understand the problem fully before attempt a clumsy potential fix. And of course will take into account the total business impact of the ongoing disruption and the known and unknown risks of the pro…

Yes, that's why it's the last emergency step. We're not disagreeing.

Re: Salesforce Global Outage

#149

Post by pixl97 »

leptons wrote: post #133
Anon1096 wrote: post #103

Earlier quoted context omitted.

Hacker News is much easier to read when you realize that 95% of people have never worked on a "high" (maybe we could say >1B requests per day as a starting point) scale distributed service and think it's trivial to run one with more than 2 nines. You see comments all the time here mentioning that their own desktop at home is achieving more than that which belies deep misunderstanding of how systems are measured. Or t…

While not a home-run server, the NTP system is a distributed service that receives 100 billion to trillions of requests per day, and it's running pretty smoothly - it's never gone down completely since it started in 1985. It's also very simple. The reason it has so many 9's uptime is because it is simple. Given a low amount of complexity, it's not unreasonable to think that an individual could run a >1B requests per…

>Salesforce is not simple. It's wildly, overly complex.

It turns out business environments are wildly overly complex.

Post reply on HN