Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Level 3 Global Outage

puck.nether.net

Post reply on HN
Print view View original

351–360 of 393 posts

Re: Level 3 Global Outage

#351

Post by mikelward »

hinkley wrote: post #349
mikelward wrote: post #341

Earlier quoted context omitted.

Some queue processing systems I've seen have infinite retries. At least they have exponential backoff I guess.

But your queue will grow and grow and the fraction of time you spend servicing old messages grows and grows. Not a terribly big fan of these queueing systems. People always seem to bung things up in ways they are not quite equipped to fix (in the “you are not smart enough to debug the code you wrote” sense). Last time I had to help someone with such a situation, we discovered that the duplicate processing problem had…

We also alert if any message is in the queue too long.

If anything, the alert is too sensitive.

Re: Level 3 Global Outage

#352

Post by walrus01 »

akritrime wrote: post #238
q3k wrote: post #208

Earlier quoted context omitted.

Unfortunately, this infrastructure is at an uncanny intersection of technology, business and politics. To learn the technical aspect of it, you can follow any network engineering certification materials or resources that delve into dynamic routing protocols, notably BGP. Inter-ISP networking is nothing but setting up BGP sessions and filters at the technical level. Why you set these up, and under what conditions is a…

What resources can I follow to start a non-profit ISP? I want to start one in my hometown for students who couldn't afford internet to join online classes.

If you intend to start a facilities-based last mile access ISP, what last-mile tech do you intend to use? There's a number of resources out there for people who want to be a small hyper local WISP. But I would not recommend it unless you have 10+ years of real world network engineering experience at other, larger ISPs.

Re: Level 3 Global Outage

#353

Post by walrus01 »

akritrime wrote: post #258
q3k wrote: post #240

Earlier quoted context omitted.

There's a bunch of guidelines for starting a (W)ISP depending on your region.

I actually tried, but all I got was some consultancy services that would help you get an ISP with estimated cost of 10k USD (a middle class household earns half of that in a year here).

http://wndw.net/book.html

Re: Level 3 Global Outage

#354

Post by walrus01 »

gkanai wrote: post #236
aosaigh wrote: post #187

Has anyone any good resources for learning more about the "internet-level" infrastructure affected today and how global networks are connected?

Don't forget Neal Stephenson's classic, "Mother Earth, Mother Board." 25 years old but still relevant. https://www.wired.com/1996/12/ffglass/

The business structures, ISP ownership and national telecoms have changed quite a lot in the past 25 years. But in terms of the physical OSI layer 1 challenges of laying cable across an ocean, that remains the most difficult and costly part of the process.

Re: Level 3 Global Outage

#355

Post by cortesoft »

jacquesm wrote: post #347
phkahler wrote: post #262

Earlier quoted context omitted.

Contrary to what that link says, the software was not thoroughly tested. Normal testing was bypassed - per management request after a small code change. This was covered in a book (perhaps Safeware, but maybe another one I dont recall) along with the Therac 25, the Ariane V, and several others. Unfortunately these lessons need to be relearned by each generation. See the 737-Max...

> Normal testing was bypassed - per management request after a small code change. That lesson will really never be learned. This happens on a daily basis all over the planet with people who have not been bitten - yet.

It isn't learned because 99% of the time, it works fine and nothing bad happens.

We are very bad at avoiding these sorts of rare, catastrophic events.

Re: Level 3 Global Outage

#356

Post by twic »

ficklepickle wrote: post #313
twic wrote: post #300

Earlier quoted context omitted.

This reminds me of an incident on the early internet (perhaps ARPANET at that point) where a routing table got corrupted so it had a negative-length route which routers then propagated to each other, even after the original corrupt router was rebooted. As with AT&T, they had to reboot all the routers at once to get rid of the corruption. I can't remember where i read about this, but i recall the problem was called "T…

I can't find anything by that name either, but the details do match the major ARPANET outage of Oct 27, 1980. The incident is detailed in RFC 789: http://www.faqs.org/rfcs/rfc789.html#b

Interesting, thanks! That is different to the story i remember, but it's possible that i remember incorrectly, or read an incorrect explanation.

I believe that i read about this episode in Hans Moravec's book 'Mind Children'. I can see in Google Books that chapter 5 is on 'Wildlife', and there is a section 'Spontaneous Generation', which promises to talk about a "software parasite" which emerged naturally in the ARPAnet - but of which the bulk is not available:

https://books.google.co.uk/books?id=56mb7XuSx3QC&lpg=PA133&d...

Re: Level 3 Global Outage

#357

Post by [deleted] »

tpmx wrote: post #332
srosmd wrote: post #331

Earlier quoted context omitted.

What was the intent of posting this? This is an article on a global network outage - some folks want the technical nitty-gritty and others don't. You seem adversarial or pretentious when you unexpectedly post things like this even if your intentions are well-meaning.

I agree it was needlessly adversarial (sorry about that!) - but it got the desired effect - an excellent explanation of the concept with lots of relevant background information (thanks, kitteh, upvoted). I think this helps the discussion a lot since a lot more people would be able to join. Less gatekeeping.

[deleted]

Re: Level 3 Global Outage

#358

Post by coryrc »

jacquesm wrote: post #347
phkahler wrote: post #262

Earlier quoted context omitted.

Contrary to what that link says, the software was not thoroughly tested. Normal testing was bypassed - per management request after a small code change. This was covered in a book (perhaps Safeware, but maybe another one I dont recall) along with the Therac 25, the Ariane V, and several others. Unfortunately these lessons need to be relearned by each generation. See the 737-Max...

> Normal testing was bypassed - per management request after a small code change. That lesson will really never be learned. This happens on a daily basis all over the planet with people who have not been bitten - yet.

Or most of what we do isn't really important so it doesn't matter if it breaks every once in a while.

Re: Level 3 Global Outage

#359

Post by throwaway344655 »

eastdakota wrote: post #339

Analysis of what we saw at Cloudflare, how our systems automatically mitigated the worst of the impact to our customers, and some speculation on what may have gone wrong: https://blog.cloudflare.com/analysis-of-todays-centurylink-l...

I remember working the support queue _before_ this automatic re-routing mitigation system went in and it was a lifesaver. Having to run over to SRE and yell "look! look at grafana showing this big jump in 522s across the board for everything originating in ORD-XX where the next hop is ASYYYY! WHY ARE WE STILL SENDING TRAFFIC OVER THAT ARRRGHH please re-route and make the 522 tickets stop"

it's cool to see something large enough that the auto-healing mechanisms weren't able to handle it on their own, though shoutout to whoever was on the weekend support/SRE shift; that stuff was never fun to deal with when you were one of a few reduced staff on the weekend shifts

Re: Level 3 Global Outage

#360

Post by a1369209993 »

doctorshady wrote: post #326
p_l wrote: post #324

Earlier quoted context omitted.

you can inject changes in routing, but if the other other carrier doesn't route around the affected network, you're back to square one. That's part of why Level3/CenturyLink was depeered and why several prefixes that are normally announced through it were quickly rerouted by owners.

That's my point; as a subscriber, you can prefix a long distance call with a routing code to avoid, for example, a shut down long distance network without any administrator changes. Routing to the long distance networks is done independently through the local network, so if AT&T's long distance network was having issues, it'd have no impact on your ability to access Verizon's long distance network.

There's actually no technical reason why you couldn't do that with IP (4 or 6); although you'd need a approriately located host to be running a relay daemon[0].

0: ie something that takes, say, a UDP packet on port NNNN containing a whole raw IPv4 packet, throws away the wrapping, and drops the IPv4 packet onto its own network interface. This is safe - the packet must shrink by a dozen or two bytes with each retransmission - but usually not actually set up anywhere.

Edit: It probably wouldn't work for TCP though - maybe try TOR?

Post reply on HN