Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

East Coast Internet Outage

news.ycombinator.com

Post reply on HN
Print view View original

191–200 of 265 posts

Re: East Coast Internet Outage

#191

Post by [deleted] »

ASalazarMX wrote: post #119
sleepysysadmin wrote: post #100

>There is a fiber cut in Brooklyn. We have no ETR, as of yet. A fiber cut auto heals automatically via bgp. Absolutely no truth in this claim from verizon. Meanwhile google, zoom, aws, and microsoft went down? There isn't any shared datacenters here. There's at least 3 distinct datacenters. So what's the truth? what actually went down?

It might sound conspiratorial, but when the WTC was attacked in 2001, the Internet slowed down a few days after. We'll probably never know, but I assumed it was deeper surveillance being implemented. Every time an outage like this happens without a following postmortem, I wonder if it was a surveillance oopsie.

[deleted]

Re: East Coast Internet Outage

#192

Post by Scoundreller »

fcsp wrote: post #180
korethr wrote: post #139

Earlier quoted context omitted.

And when it's a large bundle cut, it's going to take a bit before service is restored. I have been told that a highly skilled tech can splice a single strand in a matter of minutes. Looking it up, I find figures of an average of 30 minutes per joint for 4-strand bundles, and slowing down from there. So if you have a 144-strand bundle to repair, or multiple 144-strand bundles, it's going to take several hours to effec…

Would it maybe be an idea to, going forward, color the cables individually before putting them into the ground?

If they’re all in use and it’s point to point, might be easier connecting whatever to whatever. Then simultaneously swap line cards around as they get lit up or letting IP do its thing.

Source: I have no idea.

Re: East Coast Internet Outage

#193

Post by dang »

xhrpost wrote: post #61
atonse wrote: post #32

Honestly, at this point I've found HN to be the best "Status Page" for this sort of thing. I came here first when I found issues and fiddling around with my custom DNS lookup hierarchy didn't help. FiOS customer in DC Metro.

Except these posts seem to regularly get taken down or flagged. This isn't even on the front page anymore it seems.

It set off the flamewar detector. We monitor that list and eventually cancel the penalty for threads that aren't actual flamewars, but it takes a while. If anyone wants to make it faster, emailing hn@ycombinator.com is usually a forcing function.

Edit: Also, posts without URLs get a downweight by default. I've turned that off too, and merged the other threads into this one, since it was the first one.

Re: East Coast Internet Outage

#194

Post by overkill28 »

neogodless wrote: post #103
2dollars27cents wrote: post #93

I have no technical understanding of fiber networks. The explanation that this was caused by one fiber line in Brooklyn being severed seems unlikely. Curious to see what the experts think.

While I'm also not qualified to diagnose the problem, I would imagine it's something like this... The major internet connections are like superhighways, with major central routers acting like exchanges. If a major superhighway had an accident blocking traffic, there would be personnel stationed at exchanges to help re-route the traffic, though that often occurs too late to prevent a bunch of backups. At any rate, peo…

The odd thing about this situation though is the geographic location of Brooklyn: on an island. Why would a significant amount of traffic from outside Long Island be going across the harbor to Brooklyn, especially given all the major switching facilities in Manhattan (closer to shore).

Re: East Coast Internet Outage

#195

Post by eMGm4D0zgUAVXc7 »

dang wrote: post #193
xhrpost wrote: post #61

Earlier quoted context omitted.

Except these posts seem to regularly get taken down or flagged. This isn't even on the front page anymore it seems.

It set off the flamewar detector. We monitor that list and eventually cancel the penalty for threads that aren't actual flamewars, but it takes a while. If anyone wants to make it faster, emailing hn@ycombinator.com is usually a forcing function. Edit: Also, posts without URLs get a downweight by default. I've turned that off too, and merged the other threads into this one, since it was the first one.

How does one programmatically detect a flamewar? :)

Re: East Coast Internet Outage

#196

Post by betterunix2 »

jzer0cool wrote: post #135

When houses are created there are building codes to adhere to. For example, when there is a water pipe behind the walls a plumber (a good one) would also put a plate in front of it. In this way, once the walls are in place, drilling away into the wall, or nailing, would hit the metal plate, thus provides some safety mechanism. As another example, underground, in public areas are markings (or at least procedures) to i…

Construction crews need to pay attention to what is below them before they start digging. A backhoe is going to defeat your cable every time.

Re: East Coast Internet Outage

#198

Post by dang »

eMGm4D0zgUAVXc7 wrote: post #195
dang wrote: post #193

Earlier quoted context omitted.

It set off the flamewar detector. We monitor that list and eventually cancel the penalty for threads that aren't actual flamewars, but it takes a while. If anyone wants to make it faster, emailing hn@ycombinator.com is usually a forcing function. Edit: Also, posts without URLs get a downweight by default. I've turned that off too, and merged the other threads into this one, since it was the first one.

How does one programmatically detect a flamewar? :)

By proxy, which is why there are false positives.

Re: East Coast Internet Outage

#199

Post by ASalazarMX »

FlyMoreRockets wrote: post #164
kridsdale1 wrote: post #130

Earlier quoted context omitted.

Why not take the simpler interpretation that millions of dollars of equipment was in the buildings that were destroyed, and it took a while for the network's self-healing algorithms to discover a new steady-state configuration that approximated the performance of what was suddenly lost?

Occam's razor: when presented with competing hypotheses about the same prediction, one should select the solution with the fewest assumptions. https://en.wikipedia.org/wiki/Occam%27s_razor Why is it that conspiracy theories seem to always require a more complex answer than the truth?

Occam's razor is a guide to start with the simpler explanation. If the simpler explanation is not sufficient, you add more elements until it is.

My point is, the Internet rerouting around damage doesn't explain the slowdown, mostly because it happened days after, not immediately, meaning the damaged infrastructure was not critical for the global internet.

The attacks unleashed a huge intelligence operation, and it's reasonable to expect a parallel intelligence operation online. If the latter caused a traffic bottleneck or not, we can't know because it was never addressed, it's speculation on my part.

Post reply on HN