Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Today's Outage Post Mortem

blog.cloudflare.com

Post reply on HN
Print view View original

141–150 of 159 posts

Re: Today's Outage Post Mortem

#141

Post by seanp2k2 »

larrys wrote: post #41
ryguytilidie wrote: post #22

This is pretty impressive. Keep in mind most of the team is on the west coast so this happened at 1am on a Sunday and they put up a post mortem within hours. Obviously you would prefer it not happen at all, but that is a great response imo.

But this is not impressive: "Someone from our operations team is monitoring our network 24/7." "Someone" seems to indicate "1 person". Not "people are monitoring" but "someone". That's it, one person monitors the network? Like the single night guard at the warehouse?

It's not usually necessary to have more than one person working overnight shifts with good automated monitoring in place. That person is tasked with "watching the watchman" as it were.

Paying 5 people to be up all night staring at dashboards isn't generally (required|a good use of people).

Re: Today's Outage Post Mortem

#143

Post by seanp2k2 »

dubcanada wrote: post #76
larrys wrote: post #59

Earlier quoted context omitted.

The question remains how many people actually are monitoring the network in order to call the first responders. Is it one person, two or five? And my use of "not impressive" was in reply to someone who said "impressive" but more importantly thought it was "impressive" that they put up a post mortem within hours. That's nice but it doesn't answer the question that I had. I stand behind my comment and re ask the questi…

You seem to be looking for "problems" where non exist. There could be 15000000 people monitoring it. It can still go down.

This.

It seems that you care more about how many people are actively waiting for something to break vs. how long their response takes. Also, (those people | that person) probably (is | are) the first responder. I really believe that one "first mate" watching the automated ship sail at night ready to triage a technical problem is better than 10 guards who will promptly fall all over themselves when the bits hit the fan.

Re: Today's Outage Post Mortem

#145

Post by seanp2k2 »

tedchs wrote: post #135

What I don't understand is why Cloudflare is making changes to their border routers in the process of protecting their customers. I am a network engineer and I love Juniper, but the reality is with any complex system, every change you make has a possibility of inducing an unexpected failure. I would think Cloudflare would have increased stability by using an architecture where the border routers have a mostly static…

So now you have two pieces of gear to test changes on and another interaction where stuff could break / go weird.

I don't see how that would solve anything here.

Re: Today's Outage Post Mortem

#146

Post by seanp2k2 »

Dylan16807 wrote: post #53
jonknee wrote: post #45

Earlier quoted context omitted.

You can only account for it by having more hardware and then it's possible more of your hardware will fail which puts you right back to where you started.

I don't think that's the only solution. I would be willing to bet that outside of heavy-DDoS conditions that even a tiny fraction of Cloudflare's network could handle the incoming tcp connections and deny all of them. At that point you don't have to worry about traffic collapsing anything. You can wait to bring up more equipment. You can send a tiny error page. You can let X% of requests get through and be fully serv…

You're vastly oversimplifying the problem here by only accounting for one class of problems.

>". I would be willing to bet that outside of heavy-DDoS conditions that even a tiny fraction of Cloudflare's network could handle the incoming tcp connections and deny all of them." depends on the attack.

>"You can send a tiny error page. You can let X% of requests get through and be fully served." Not usually that easy.

Re: Today's Outage Post Mortem

#149

Post by noselasd »

cbsmith wrote: post #101
mprovost wrote: post #65

Earlier quoted context omitted.

Yes but they were still seeing packets bigger than the MTU of Ethernet (or Sonet or whatever other layer 1/2 tech they're connected to the rest of the net with). It doesn't matter what higher level protocols can handle.

You are joking right? The packet size at the higher layer is what they were matching against. The size of the layer 2 packets is irrelevant.

Maybe, but nothing in the the rule they showed hinted it was not at layer 3 (For IPv4 )
Post reply on HN