Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

East Coast Internet Outage

news.ycombinator.com

Post reply on HN
Print view View original

161–170 of 265 posts

Re: East Coast Internet Outage

#161

Post by vlan0 »

jzer0cool wrote: post #135

When houses are created there are building codes to adhere to. For example, when there is a water pipe behind the walls a plumber (a good one) would also put a plate in front of it. In this way, once the walls are in place, drilling away into the wall, or nailing, would hit the metal plate, thus provides some safety mechanism. As another example, underground, in public areas are markings (or at least procedures) to i…

Lots of ways, unfortunately. But generally a quality network will be built to have physical redundancy. Yet, the ISP game at the top is all about cost cutting. So we'll never get what we paid for back in the 90s.

Re: East Coast Internet Outage

#162

Post by mikem170 »

im3w1l wrote: post #109
kube-system wrote: post #96

Earlier quoted context omitted.

I wouldn't doubt it. Single misconfigured routers have caused issues of similar scale.

I believe the internet is designed to handle broken links, but is not very robust against misconfigured (or malicious) routers.

Just to add to that, sometimes the redundancy doesn't work out as planned, maybe another weird set of circumstances, or some routers can't handle spikes in load as the network reconverges, causing other reconverges, confusing bgp routing peers, etc. Fun stuff!

Re: East Coast Internet Outage

#163

Post by richwater »

Jonnax wrote: post #105

The FCC said that 3mbps upload speed is all internet users need. They might not be in tune with modern internet usage https://www.theverge.com/2021/1/21/22242635/fcc-ajit-pai-int...

You vastly underestimate the cost to provide broadband to rural communities.

Re: East Coast Internet Outage

#164

Post by FlyMoreRockets »

kridsdale1 wrote: post #130
ASalazarMX wrote: post #119

Earlier quoted context omitted.

It might sound conspiratorial, but when the WTC was attacked in 2001, the Internet slowed down a few days after. We'll probably never know, but I assumed it was deeper surveillance being implemented. Every time an outage like this happens without a following postmortem, I wonder if it was a surveillance oopsie.

Why not take the simpler interpretation that millions of dollars of equipment was in the buildings that were destroyed, and it took a while for the network's self-healing algorithms to discover a new steady-state configuration that approximated the performance of what was suddenly lost?

Occam's razor: when presented with competing hypotheses about the same prediction, one should select the solution with the fewest assumptions.

https://en.wikipedia.org/wiki/Occam%27s_razor

Why is it that conspiracy theories seem to always require a more complex answer than the truth?

Re: East Coast Internet Outage

#165

Post by ronnier »

arthurcolle wrote: post #85

I was just about to call FiOS to switch from Xfinity because these data caps are absolutely ridiculous. I am sad now. Aside: Comcast, wtf? 1.2 TB limit per month? Seriously if anyone works at Comcast know that I hate your company with an overwhelming passion.

What's worse is the 35 Mbps upload.

Re: East Coast Internet Outage

#166

Post by mikem170 »

ArchOversight wrote: post #114
sleepysysadmin wrote: post #100

>There is a fiber cut in Brooklyn. We have no ETR, as of yet. A fiber cut auto heals automatically via bgp. Absolutely no truth in this claim from verizon. Meanwhile google, zoom, aws, and microsoft went down? There isn't any shared datacenters here. There's at least 3 distinct datacenters. So what's the truth? what actually went down?

If 100 Gbps of capacity suddenly went away, then BGP attempted to route around it, and sent that traffic down 10 Gbps routers instead, which overloaded, or were not ready for the deluge of traffic, then those BGP routes could start flapping, at which point those routes would get pulled, forcing it onto yet other circuits, which may be 1 Gbps and that's how the outage continues to grow. Too much capacity is centralize…

Yep, and it's possible that running low on ipv4 address space exacerbated the situation, so many more small routes than there used to be, maybe not all getting summarized during an outage they way they used to be, causing problems on peer bgp routers like you said.

Re: East Coast Internet Outage

#168

Post by Florin_Andrei »

dylan604 wrote: post #101
Animats wrote: post #95

Too much is failing from single point outages. The FCC may have to establish standards to prevent over-centralization. In the entire history of the Bell System, no electromechanical central office was ever down for more than 30 minutes for any reason than a natural disaster or, one occasion, a major file. That record has not been maintained in the Internet era.

>The FCC may have to establish standards to prevent over-centralization. Bwahahahahaha. Sorry, but that's like expecting the FCC to do something that would benefit US citizens vs US corporations. I think we've seen that's just not something they are interested in doing. Maybe a new administration will put as the head of the FCC someone that is "good of citizens" minded and uses a normal size coffee cup, but I'm not h…

> Maybe a new administration will put as the head of the FCC someone that is "good of citizens" minded and uses a normal size coffee cup

Someone who is interested in anything beyond personal profit, status, and influence, would be nice for a change. Also someone who is smart would be very nice.

Re: East Coast Internet Outage

#169

Post by jzer0cool »

valbaca wrote: post #142
jzer0cool wrote: post #135

When houses are created there are building codes to adhere to. For example, when there is a water pipe behind the walls a plumber (a good one) would also put a plate in front of it. In this way, once the walls are in place, drilling away into the wall, or nailing, would hit the metal plate, thus provides some safety mechanism. As another example, underground, in public areas are markings (or at least procedures) to i…

Scale: wrapping all fiber optic cables in a protective metal sheath would be cost prohibitive. Whereas adding a metal sheet to prevent a house from flooding or someone from getting electrocuted makes sense. One idea, borrowed from some chainsaw safety pants, would be to wrap the cable in a fiber web sheath that tangles in whatever cutting equipment https://www.youtube.com/watch?v=r5VSivQe760

Literally took off the pants testing that! I'm unsure how much is worth getting paid to run a self-test. This product looks pretty neat. Half joking, anyone ever, or witnessed getting eletrocuted trying to get the internet / computer working again?

This spun threads reminds me of nature. Spiders, silk works spinning web. Wonder if that's where the inspiration came from.

Re: East Coast Internet Outage

#170

Post by mikem170 »

jasonjayr wrote: post #113
sleepysysadmin wrote: post #100

>There is a fiber cut in Brooklyn. We have no ETR, as of yet. A fiber cut auto heals automatically via bgp. Absolutely no truth in this claim from verizon. Meanwhile google, zoom, aws, and microsoft went down? There isn't any shared datacenters here. There's at least 3 distinct datacenters. So what's the truth? what actually went down?

Traffic is passing, although slowly. I think this revealed that there is insufficient failover capacity.

Could be that. Also could be too much complexity, as opposed to capacity. Could have been timing didn't help, sometimes you take calculated risks or have to deal with additional complexity during migrations, etc.

I've even seen where playing it safe while planning to back out of a manual fix used during a weird outage (circuit went down in one direction or something, just needed to be manually shutdown to re-route everything) brought down the entire network when another outage happened before that evening, i.e. most redundancy plans protect against single failures, not all combination of two failures. They wanted to do all planned changes after business hours, like manually swinging traffic back to the primary hub site. They didn't check the weather on the other side of the country, and the power at their new super-duper data center went down that afternoon during a thunderstorm...

I wonder what percentage of fiber cuts don't cause noticeable outages. We'll never know, unless they let us root around in their ticket system.

As a customer it is fair to judge them on what you see, though. Some telcos screw up more than others. Best to use two.

(Typed over my phone, cuz my broadband is down!)

Post reply on HN