Presumably Jupiter's Junos is closed-source, making investigation more difficult? Do they provide it to some of their bigger clients under an agreement?
Today's Outage Post Mortem
51–60 of 159 posts
Re: Today's Outage Post Mortem
#52> attack packets were between 99,971 and 99,985 bytes long.
This should raise a red flag, as it must be impossible. Ethernet NICs would just bail out on packets longer than what you've set the MTU to, and ethernet frames would just come from the next hop in most cases. And IP packets have a max length field of 16 bit.
Re: Today's Outage Post Mortem
#53> Even though some data centers came back online initially, they fell back over again because all the traffic across our entire network hit them and overloaded their resources. I know very little of networking, but this seems to be a recurring pattern that aggravates many major outages. What surprises me is that this so often seems to be a scenario not accounted for.
You can only account for it by having more hardware and then it's possible more of your hardware will fail which puts you right back to where you started.
I don't think that's the only solution. I would be willing to bet that outside of heavy-DDoS conditions that even a tiny fraction of Cloudflare's network could handle the incoming tcp connections and deny all of them. At that point you don't have to worry about traffic collapsing anything. You can wait to bring up more equipment. You can send a tiny error page. You can let X% of requests get through and be fully served.
I bet that most of the time the domino effect happens to internet services in general it's with nodes that are accepting most requests. They allow themselves to be overloaded. An active HTTP session uses orders of magnitude more resources than simply denying the initial packet and forgetting about it forever.
Re: Today's Outage Post Mortem
#54This is pretty impressive. Keep in mind most of the team is on the west coast so this happened at 1am on a Sunday and they put up a post mortem within hours. Obviously you would prefer it not happen at all, but that is a great response imo.
But this is not impressive: "Someone from our operations team is monitoring our network 24/7." "Someone" seems to indicate "1 person". Not "people are monitoring" but "someone". That's it, one person monitors the network? Like the single night guard at the warehouse?
Why would you assume this and judge the company on it, based entirely on an offhanded comment? You couldnt have given them the benefit of the doubt long enough to find another of several comments which clearly indicate there was a team of people working on the issue?
Re: Today's Outage Post Mortem
#55this is the type of reason why i stopped using cloudflare. there are just too many eggs in one basket. it's as if their entire service becomes a SPOF to your infrastructure.
You could say the same thing about almost any of your service providers. Your DNS provider goes out, everything goes out. The routers at the data center with your servers go out, all your servers go out. Your CDN goes out, all of the static assets on your site go out.
There will always be potential SPOF.
Re: Today's Outage Post Mortem
#56[deleted]
Re: Today's Outage Post Mortem
#57> attack packets were between 99,971 and 99,985 bytes long. This should raise a red flag, as it must be impossible. Ethernet NICs would just bail out on packets longer than what you've set the MTU to, and ethernet frames would just come from the next hop in most cases. And IP packets have a max length field of 16 bit.
http://en.wikipedia.org/wiki/Jumbogram
> An optional feature of IPv6, the jumbo payload option, allows the exchange of packets with payloads of up to one byte less than 4 GiB
Re: Today's Outage Post Mortem
#58Earlier quoted context omitted.
Re: "there aren't really any viable options..." Total misconception. BGP, OSPF, ISIS, LISP, etc. are all non proprietary. Sure, the root cause of this particular problem is that CF is using something specific to Juniper, however router interoperability is not predicated on components like that. This example was a tool CF operationalized, and likely had little to do with their routing with the exception of it being a…
"there aren't really any viable options" to juniper or cisco for core/edge routers. There are some routing protocols which interoperate (which is how different sites on the Internet can talk to each other), but most of the protocols used for HA or management of a given set of routers, or, more importantly, most tested/debugged implementations of HA and device management, are Cisco or Juniper specific. No big deal ann…
Thats why software defined networking, Openflow etc, are going to take off, as you can get back control of the protocols and what is going on, and avoid the vendor lockin.
Re: Today's Outage Post Mortem
#59Earlier quoted context omitted.
But this is not impressive: "Someone from our operations team is monitoring our network 24/7." "Someone" seems to indicate "1 person". Not "people are monitoring" but "someone". That's it, one person monitors the network? Like the single night guard at the warehouse?
Why would you assume this and judge the company on it, based entirely on an offhanded comment? You couldnt have given them the benefit of the doubt long enough to find another of several comments which clearly indicate there was a team of people working on the issue?
The question remains how many people actually are monitoring the network in order to call the first responders. Is it one person, two or five?
And my use of "not impressive" was in reply to someone who said "impressive" but more importantly thought it was "impressive" that they put up a post mortem within hours. That's nice but it doesn't answer the question that I had.
I stand behind my comment and re ask the question (since the info is ambiguous we have jgramhmc saying "small team who monitor things" and we have the blog post saying "Someone from our operations team is monitoring our network 24/7."
I don't think it's unreasonable (in the interest of transparency) to know exactly the structure and # bodies of who monitors the network at any given time. What is the human point of failure in the system?
I don't depend on cloudflare. But if I was running a mission critical operation and depended on them I might setup a site visit to actually get a feel of what is going on.
As an aside back when the .org registry got started one of the dns servers sat in an open unguarded office under a desk accessible by the cleaning person. I saw it when I did a site visit. And of course if you've been around long enough you know there was a time when the root dns servers sat unguarded in university offices.
Re: Today's Outage Post Mortem
#60That was a pretty interesting writeup and I always like it when companies are totally (and quickly) upfront about negative events. One thing that occurred to me though is that performing a hard reboot of the routers required calling people to physically access the devices and took some time to perform (as you would expect). Although I wouldn't expect it to be needed very often, I'm sort of surprised CloudFlare doesn'…
I've never seen remote power cyclers on big routers in major facilities which have on-site remote hands, even when servers all get both IPMI/LOM board cyclers and physical external cyclers. At most, the routers get a serial port connected to a serial port console server or directly to a modem, and/or an admin ethernet network. I've seen smaller routers, CSU/DSU, etc. type devices in branch offices on cyclers, though.…
You would need another network (not just a vlan) to run this as well, if you are going to try to reach it when nothing else is working.