Today's Outage Post Mortem • Hacker News Board

Re: Today's Outage Post Mortem

#151

Post by tedchs » Mon, Mar 04, 2013, 5:04 PM UTC

seanp2k2 wrote: post #145

tedchs wrote: post #135

What I don't understand is why Cloudflare is making changes to their border routers in the process of protecting their customers. I am a network engineer and I love Juniper, but the reality is with any complex system, every change you make has a possibility of inducing an unexpected failure. I would think Cloudflare would have increased stability by using an architecture where the border routers have a mostly static…

So now you have two pieces of gear to test changes on and another interaction where stuff could break / go weird. I don't see how that would solve anything here.

The thing it would solve is risking all their BGP peerings going down as a result of day-to-day service operations (i.e. every time they add a filter).

Re: Today's Outage Post Mortem

#152

Post by cbsmith » Mon, Mar 04, 2013, 5:14 PM UTC

noselasd wrote: post #149

cbsmith wrote: post #101

Earlier quoted context omitted.

You are joking right? The packet size at the higher layer is what they were matching against. The size of the layer 2 packets is irrelevant.

Maybe, but nothing in the the rule they showed hinted it was not at layer 3 (For IPv4 )

It is at layer 3. IPv6 is layer 3.

Re: Today's Outage Post Mortem

#153

Post by manys » Mon, Mar 04, 2013, 7:02 PM UTC

larrys wrote: post #41

ryguytilidie wrote: post #22

This is pretty impressive. Keep in mind most of the team is on the west coast so this happened at 1am on a Sunday and they put up a post mortem within hours. Obviously you would prefer it not happen at all, but that is a great response imo.

But this is not impressive: "Someone from our operations team is monitoring our network 24/7." "Someone" seems to indicate "1 person". Not "people are monitoring" but "someone". That's it, one person monitors the network? Like the single night guard at the warehouse?

You're being That Guy.

Re: Today's Outage Post Mortem

#154

Post by Dylan16807 » Mon, Mar 04, 2013, 7:12 PM UTC

seanp2k2 wrote: post #146

Dylan16807 wrote: post #53

Earlier quoted context omitted.

I don't think that's the only solution. I would be willing to bet that outside of heavy-DDoS conditions that even a tiny fraction of Cloudflare's network could handle the incoming tcp connections and deny all of them. At that point you don't have to worry about traffic collapsing anything. You can wait to bring up more equipment. You can send a tiny error page. You can let X% of requests get through and be fully serv…

You're vastly oversimplifying the problem here by only accounting for one class of problems. >". I would be willing to bet that outside of heavy-DDoS conditions that even a tiny fraction of Cloudflare's network could handle the incoming tcp connections and deny all of them." depends on the attack. >"You can send a tiny error page. You can let X% of requests get through and be fully served." Not usually that easy.

I said outside of attacks.

I call BS on saying it's not easy to limit the number of served connections and RST the rest. Isn't this something every web server can do by itself it's so easy?

Re: Today's Outage Post Mortem

#155

Post by noselasd » Mon, Mar 04, 2013, 9:08 PM UTC

cbsmith wrote: post #152

noselasd wrote: post #149

Earlier quoted context omitted.

Maybe, but nothing in the the rule they showed hinted it was not at layer 3 (For IPv4 )

It is at layer 3. IPv6 is layer 3.

If it was IPv6, I'd assume the routing rule on their blog contained IPv6 addreses, not IPv4 addresses, even if the blog faked the IP addresses.

Re: Today's Outage Post Mortem

#156

Post by meh02 » Mon, Mar 04, 2013, 10:25 PM UTC

23david wrote: post #112

tlrobinson wrote: post #79

Earlier quoted context omitted.

The distinction is a bit arbitrary. As a customer you should care that their service is geographically distributed, not whether they own the buildings where the servers are kept.

Gotta disagree. Owning a datacenter doesn't mean that they own the building. The difference between renting space in a datacenter versus running an entire datacenter is very big, and has ramifications for their uptime, security of their data and disaster recovery. Not sure why they aren't clearer about this.

Because they want you to think they're doing something more than running a bunch of leased servers from dedicated server providers.

Re: Today's Outage Post Mortem

#157

Post by meh02 » Mon, Mar 04, 2013, 10:32 PM UTC

random42 wrote: post #66

OT: I want to pitch cloudflare for our CDN needs. Can someone estimate the scale of cloudflare wrt. akamai (current provider), in terms of operations, consumers etc.?

Cloudflare is not a replacement for Akamai. They've just admitted to having a single network ops guy monitoring their network at night. Cloudflare is what happens when amateurs attempt to run a CDN.

What reason do you have to pitch Cloudflare?

Re: Today's Outage Post Mortem

#158

Post by meh02 » Mon, Mar 04, 2013, 10:34 PM UTC

larrys wrote: post #41

ryguytilidie wrote: post #22

This is pretty impressive. Keep in mind most of the team is on the west coast so this happened at 1am on a Sunday and they put up a post mortem within hours. Obviously you would prefer it not happen at all, but that is a great response imo.

But this is not impressive: "Someone from our operations team is monitoring our network 24/7." "Someone" seems to indicate "1 person". Not "people are monitoring" but "someone". That's it, one person monitors the network? Like the single night guard at the warehouse?

Hehe. Downvoted for pointing out Cloudflare are amateurs.

Welcome to the club.

Re: Today's Outage Post Mortem

#159

Post by cbsmith » Tue, Mar 05, 2013, 3:34 AM UTC

noselasd wrote: post #155

cbsmith wrote: post #152

Earlier quoted context omitted.

It is at layer 3. IPv6 is layer 3.

If it was IPv6, I'd assume the routing rule on their blog contained IPv6 addreses, not IPv4 addresses, even if the blog faked the IP addresses.

Perhaps then you aren't aware that IPv6 stacks can reach IPv4 addresses, nor that IPv6 packets are a popular way to compromise systems that support both IPv6 and IPv4, because the IPv6 stacks are not as well hardened.