Live data from Hacker News

Facebook-owned sites were down

facebook.com

781–790 of 1001 posts

Re: Facebook-owned sites were down

#781
post #36

For Facebook and WhatsApp it looks like a DNS issue, name resolution fails with SERVFAIL: $ dig facebook.com ; > DiG 9.16.21 > facebook.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER

It's always DNS

>It's always DNS

How is this not the top comment? Underrated

Re: Facebook-owned sites were down

#782
post #445

Earlier quoted context omitted.

If only the news reporting was not as stupid as "internet is not working at UPC", instead of DNS resolvers at UPC crashed, here's what you can do... Anyway, I didn't even notice since I run knot-resolver at home. I wonder what it will be like connecting Facebook back to the internet, thundering herd and everything...

I suspect that the DNS aspect will be fine. The middle DNS servers only need one valid response to cache it for $TTL, but they can't cache SERVFAIL.

I mean connecting your company to the internet when you have billions of devices waiting in the line to fetch updates, or whatever.

Will not that be an issue? Re-enabling routing to such a massive internet service...

Re: Facebook-owned sites were down

#784
post #686
post #314

Earlier quoted context omitted.

> the people with physical access is separate from the people with knowledge of [...] Welcome to the brave new world of troubleshooting. This will seriously bite us one day.

folks with physical access are also denied. source - https://twitter.com/YourAnonOne/status/1445100431181598723

FWIW that's not the original source, just some twitter account reposting info shared by someone else. See this sub-thread: https://news.ycombinator.com/item?id=28750888

Re: Facebook-owned sites were down

#786
The media coverage and lots of the comments don't make sense to me. FB would not be so stupid and put all of their crucial DNS servers into a single autonomous system (which is now offline due to BGP issues). They operate literally dozens of datacenters around the world, and are surely not using a single AS for them - why not put secondary Nameservers there? Can someone make a sense of this?

Re: Facebook-owned sites were down

#789

Earlier quoted context omitted.

I can't fathom how they didn't plan for this. In any business of size, you have to change configuration remotely on a regular basis, and can easily lock yourself out on a regular basis. Every single system has a local user with a random password that we can hand out for just this kind of circumstance...

> I can't fathom how they didn't plan for this Maybe because they were planning for a million other possible things to go wrong, likely with higher probability than this. And busy with each day's pressing matters.

Anyone who has actually worked in the field can tell you that a deploy or config change going wrong, at some point, and wiping out your remote access / ability to deploy over it is incredibly, crazy likely.
Post reply on HN