Perhaps relevant that npm has been having issues although they only recently caught and fixed them. Scoped Private npm packages were getting cloudflare 503 errors
Facebook, Instagram go down around the world in an apparent outage
281–290 of 351 posts
Re: Facebook, Instagram go down around the world in an apparent outage
#282I’ve seen many systems go down over the last few days worldwide. Aside from the possibility of a mega-DDoS attack (which Facebook denies), all of these organizations have fairly diverse tech stacks to my knowledge. Google’s issue (supposedly) had to do with their Blobstore API, we don’t know what happened with Facebook, and many other, smaller services have had issues as well, including three intranet services at my…
Both companies are massive and have tons of developers. It becomes almost impossible to look at the system as a whole with the amount of changes coming through. And, you get scenarios where small failures cascade through the stack reaking havok. Often times its just one config change Its telling that one of the hottest areas of distributed systems research these days is the boring topic of configuration management. G…
>>Google, Microsoft, etc are paying researchers top dollar to figure out how to prevent massive outages through novel techniques
Curious what makes you think this. Are there specific job postings in either company that are focused on this?
Re: Facebook, Instagram go down around the world in an apparent outage
#283What manner of failure would cause such globally deployed and distributed systems to go down like this? I'm very interested to read up on this when they release details of the failure.
I have no inside knowledge of this one, but broadly speaking, these sorts of failures can be caused by a change thought innocent at the time to the core software that is then widely deployed using automated systems. If the core's tests didn't catch a real issue in production (and for whatever reason, the rollout happens faster than the regular small-release verification process can catch the error), things can go sour in a way that's expensive to un-sour.
Amazon once pushed a seemingly-innocuous change to their internal DNS that caused all the routers between and within datacenters to drop their IP tables on the floor. They had to re-establish the entire network by hand---datacenter heads calling each other up and reading IP address ranges over the phone to be hand-entered into lookup tables. Cost a fortune in lost sales for the time the whole site was inaccessible.
Re: Facebook, Instagram go down around the world in an apparent outage
#284Earlier quoted context omitted.
I don't believe Gmail was ever fully down. For me, I was just having problems with attachments. I also noticed app icons in the play store failing to load.
Didn't they just announce shutting down Flickr or something. Plus they could be decommissioning Google+ stuff. Maybe this is related. Just a guess.
Flickr is unrelated to Google and Facebook. Flicker is owned by SmugMug, previously by Yahoo, previously by Ludicorp.
Re: Facebook, Instagram go down around the world in an apparent outage
#285Earlier quoted context omitted.
So, a storm in Denver stops me from using Messengner in Estonia? I wonder where the butterfly flapped its wings.
Pretty sure it doesn't apply to Facebook, but Amazon's cheapest AWS tiers are around there. Same with Virginia.
Wrong, AWS has no regions near Denver.
Re: Facebook, Instagram go down around the world in an apparent outage
#286Can you imagine if Twitter and Google went down at the same time? People would be reactivating their Facebook accounts and having to sift through conspiracy theory posts about Hillary Clinton still just to figure out what was going on. Edit: The points on this post keep going up and down every time I check these comments. Yes, it was sarcasm, I was joking, but I was trying to point out that most people rely on a smal…
Whenever I hear when some service is down, I immediately go to that service to confirm. Then I repeatedly hit reload if it doesn't work to see if it can come up. I guess many people do the same and that may contribute to the problem...
Gmail's exponential back off, with it's visible countdown to next retry is a nice idea. Probably reduces that compounding wave of customer reloads.
Re: Facebook, Instagram go down around the world in an apparent outage
#287Earlier quoted context omitted.
FANGs all use white box hardware with “merchant silicon” meaning they buy the chipsets directly from Broadcom, Mellanox, etc. and build their own devices. However, they do all have Broadcom and Mellanox in common and Cisco, juniper, and arista do too.
Some FANGs definitely use Arista/Cisco. As far as I know white box hardware is mostly for top of the rack switches (as opposed to backbone infra).
Depends on which backbone infra :-) Take a look at the B4 and Jupiter papers from Google, for example.
Re: Facebook, Instagram go down around the world in an apparent outage
#288BTW, many apps are affected by this. I cannot log in to any app that uses Facebook for authentication.
This is one of the many many reasons I don't use fb for auth for any other websites.
Re: Facebook, Instagram go down around the world in an apparent outage
#289Earlier quoted context omitted.
That's the trust issue with current agreements we are solving. If an API is down the bound agreement is enforced instantly with our platform, no lies, no call, no pain. We are actually onboarding companies to try it out! https://stacktical.com
Why does this need a blockchain?
TLDR: Because Smart Contracts on the blockchain are the right tool for Secure Digital Agreements.
Paperweight contracts are irrelevant in a world of data
* A Smart Contract is cheaper to publish that the stack of paper handled by lawyers.
* Code is cheap to iterate from whereas traditional SLA are expensive/slow to renegociate. Over time, SLAs drive behaviors that are focused on delivering a minimum level of service at minimum cost to the provider.
* A Smart Contract is a code you can trust, understand and expect to behave instantly compared to the traditional SLM.
* A Smart Contract is immutable. More about the benefit of Smart Contract over paper/digital agreements: https://www.forbes.com/sites/cognitiveworld/2019/03/10/rise-...
Why not a python script? I don't trust the guy that handle that script. Blockchain is very good at bringing trust.
I hope I answered your question
[edit] formatting
Re: Facebook, Instagram go down around the world in an apparent outage
#290So yesterday Google had a major (and out of character) outage across its apps, and today Facebook has a major (and also out of character) outage across its apps. I can't wait to see the RCA for both of these and if they're related.
Remember when youtube was down for like six hours a few months ago and we still haven't heard why?
Yeh why was that? I kept wondering when we'd hear. I've got bets to collect on!