Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update about the October 4th outage

engineering.fb.com

Post reply on HN
Print view View original

31–40 of 239 posts

Re: Update about the October 4th outage

#31

Post by tshaddox »

spoonjim wrote: post #10
0xy wrote: post #5

Knowing almost nothing about networking, isn't the way Facebook handles networking somewhat of a monolithic anti-pattern? Why is a single update responsible for taking out multiple services and why wouldn't each product or even each region within each product have their own routes, for resiliency which can then be used to rollout changes slower? By having a large centralized and monolithic system, aren't they guarant…

I mean if you own 3 independent businesses, each of which would be worth over $100 billion, and you break all of them simultaneously for an entire day, including your internal email and your badge entry systems, yes, that is definitionally an anti-pattern.

Surely “anti-pattern” doesn’t just mean “anything with negative outcomes.” Couldn’t this just be a really big mistake that isn’t indicative of an anti-pattern?

Re: Update about the October 4th outage

#32

Post by DSingularity »

paxys wrote: post #21

It was quite ironic that while every Facebook property was offline there was an immense amount of misinformation about the incident perpetuated across the internet (including right here on HN) which everyone just believed as fact.

Dang I missed the misinformation.

Re: Update about the October 4th outage

#33

Post by runawaybottle »

tomrod wrote: post #30
paxys wrote: post #21

It was quite ironic that while every Facebook property was offline there was an immense amount of misinformation about the incident perpetuated across the internet (including right here on HN) which everyone just believed as fact.

Like what?

We almost went down the ‘this is a subterfuge to delete whistleblower evidence’ rabbit hole.

Re: Update about the October 4th outage

#34

Post by colechristensen »

paxys wrote: post #21

It was quite ironic that while every Facebook property was offline there was an immense amount of misinformation about the incident perpetuated across the internet (including right here on HN) which everyone just believed as fact.

I didn’t see any disinformation, just initial reports that it was DNS which were later explained to be caused by BGP.

Re: Update about the October 4th outage

#35

Post by johnduhart »

geerlingguy wrote: post #14

Gotta love how painfully vague this is. Sounds like a PR piece for investors, not an engineering blog piece.

I think you need to re-adjust your expectations, it's not reasonable to have a fully fleshed out RCA blog post available within hours of incident resolution. Most other cloud providers take a few days for theirs.

Re: Update about the October 4th outage

#36

Post by jensensbutton »

paxys wrote: post #21

It was quite ironic that while every Facebook property was offline there was an immense amount of misinformation about the incident perpetuated across the internet (including right here on HN) which everyone just believed as fact.

Seriously. I also saw lots of posts about how quiet it would be with Facebook down, but I don't think I've ever been exposed to so many stories and so much chatter about Facebook in a single day.

Re: Update about the October 4th outage

#40

Post by dodobirdlord »

shahsyed wrote: post #27

> configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues This could be anything, potentially. I'm not very knowledgeable in computer networking, but this could be as trivial as an incorrect update to a DNS record, right?

There seems to be nothing uncertain about the immediate cause of the issue - Facebook revoked all of their BGP routes, and all of their IP addresses couldn't receive packets until they were restored.
Post reply on HN