Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Level 3 Global Outage

puck.nether.net

Post reply on HN
Print view View original

241–250 of 393 posts

Re: Level 3 Global Outage

#241

Post by kitteh »

q3k wrote: post #237
tpmx wrote: post #225

How the xxxx did it take CenturyLink/Level3 like 3-4 hours to fix this problem? Again ( https://news.ycombinator.com/item?id=24322988 ) not a network engineer, but it seemed like their routers actively stopped other networks from working around the problem since L3 would still keep pushing other networks' old routes, even after those networks tried to stop that. Also: BGP probably needs to redesigned from the ground…

> Also: BGP probably needs to redesigned from the ground up by software engineers with experience from designing systems that can remain working with hostile actors. This has been attempted a number of times, but this is a political problem, not a technical problem: there's no single agreed source of truth for routing policy. A lot of US Internet providers won't even sign up for ARIN IRR, or even move their legacy sp…

RPKI is a totally diff problem here, though.

If people refuse to sign ROAs, then they don't get protection. The ARIN TAL thing is real and people have to keep fighting that.

As it is right now you can xfer v4 out of ARIN but not v6. So even if you wanted to you can't.

Re: Level 3 Global Outage

#242

Post by DyslexicAtheist »

tpmx wrote: post #225

How the xxxx did it take CenturyLink/Level3 like 3-4 hours to fix this problem? Again ( https://news.ycombinator.com/item?id=24322988 ) not a network engineer, but it seemed like their routers actively stopped other networks from working around the problem since L3 would still keep pushing other networks' old routes, even after those networks tried to stop that. Also: BGP probably needs to redesigned from the ground…

> Also: BGP probably needs to redesigned from the ground up

SCION from ETH Zurich:

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

Re: Level 3 Global Outage

#243

Post by tpmx »

q3k wrote: post #237
tpmx wrote: post #225

How the xxxx did it take CenturyLink/Level3 like 3-4 hours to fix this problem? Again ( https://news.ycombinator.com/item?id=24322988 ) not a network engineer, but it seemed like their routers actively stopped other networks from working around the problem since L3 would still keep pushing other networks' old routes, even after those networks tried to stop that. Also: BGP probably needs to redesigned from the ground…

> Also: BGP probably needs to redesigned from the ground up by software engineers with experience from designing systems that can remain working with hostile actors. This has been attempted a number of times, but this is a political problem, not a technical problem: there's no single agreed source of truth for routing policy. A lot of US Internet providers won't even sign up for ARIN IRR, or even move their legacy sp…

Build an industry coalition. Put pressure on those who don't join. Randomly throw away 1 out of 10000 packets from the providers that fail to get with the times. Increase that frequency according to some published time function.

Re: Level 3 Global Outage

#244

Post by toss1 »

suby wrote: post #16

I was doing development work which uses a server I've got hosted on digital ocean. I started getting intermittent responses which I thought weird as I hadn't changed anything on the server. I spent a good ten minutes trying to debug the issue before searching for something on duckduckgo, which also didn't respond. Cloudfare shouldn't be involved at all with my little site, so I don't think it's limited to just them.

Yup, definitely noticed earlier outages to both EU sites and also to HN. Looked far upstream because many sites/lots of things worked fine. Good to see it's at least largely fixed

Re: Level 3 Global Outage

#246

Post by MayeulC »

q3k wrote: post #230
jsjohnst wrote: post #228

Earlier quoted context omitted.

> or even just getting an ASN and some IPv6 PA space and trying to announce it somewhere That’s fairly expensive to do just for a hobby interest, but at least the price has came down since I last looked.

A RIPE ASN (as a end-user through a LIR) and PA v6 will cost you around $100 per year and some mild paperwork, there's plenty of companies/organizations that will help you with that (shameless plug: bgp.wtf, that's us). Afterwards, announcing this space is probably the cheapest with vultr (but their BGP connectivity to VMs is uh, erratic at times) or with ix-vm.cloud, or with packet.net (more expensive). You can also…

> bgp.wtf, that's us

I feel like you ought to be part of the ffdn database: https://db.ffdn.org/

Though the "French Data Networks Federation" is a French organization, their db tries to cover every independent, nonprofit ISP in the world :)

Re: Level 3 Global Outage

#247

Post by sneak »

tpmx wrote: post #243
q3k wrote: post #237

Earlier quoted context omitted.

> Also: BGP probably needs to redesigned from the ground up by software engineers with experience from designing systems that can remain working with hostile actors. This has been attempted a number of times, but this is a political problem, not a technical problem: there's no single agreed source of truth for routing policy. A lot of US Internet providers won't even sign up for ARIN IRR, or even move their legacy sp…

Build an industry coalition. Put pressure on those who don't join. Randomly throw away 1 out of 10000 packets from the providers that fail to get with the times. Increase that frequency according to some published time function.

Having a single, cryptographically assured source of truth for routing data is a turnkey censorship nightmare waiting to happen.

All it takes is a national military to care enough to put pressure on the database operator, legal or otherwise, and suddenly your legitimate routes are no longer accepted.

If you think this wouldn't be used to shut down things like future Snowden-style leaks or Wikileaks or The Shadow Brokers, you may not have been paying attention to the news.

Re: Level 3 Global Outage

#248

Post by tpmx »

sneak wrote: post #247
tpmx wrote: post #243

Earlier quoted context omitted.

Build an industry coalition. Put pressure on those who don't join. Randomly throw away 1 out of 10000 packets from the providers that fail to get with the times. Increase that frequency according to some published time function.

Having a single, cryptographically assured source of truth for routing data is a turnkey censorship nightmare waiting to happen. All it takes is a national military to care enough to put pressure on the database operator, legal or otherwise, and suddenly your legitimate routes are no longer accepted. If you think this wouldn't be used to shut down things like future Snowden-style leaks or Wikileaks or The Shadow Brok…

Yes, obviously. How is that related to the discussion above?

Re: Level 3 Global Outage

#249

Post by q3k »

tpmx wrote: post #243
q3k wrote: post #237

Earlier quoted context omitted.

> Also: BGP probably needs to redesigned from the ground up by software engineers with experience from designing systems that can remain working with hostile actors. This has been attempted a number of times, but this is a political problem, not a technical problem: there's no single agreed source of truth for routing policy. A lot of US Internet providers won't even sign up for ARIN IRR, or even move their legacy sp…

Build an industry coalition. Put pressure on those who don't join. Randomly throw away 1 out of 10000 packets from the providers that fail to get with the times. Increase that frequency according to some published time function.

> Build an industry coalition. Put pressure on those who don't join. Randomly throw away 1 out of 10000 packets from the providers that fail to get with the times. Increase that frequency according to some published time function.

What sort of incentive would anyone have to join such a coalition? Why would anyone work with providers from such a coalition, when they can work with an alternative ISP outside it and not have to deal with packet drops?

I think you're underestimating how many people have been attempting to solve this. The Internet community has some quite clever people in it, but it's also very, very large, and sweeping changes are difficult to pull off (see: IPv6 adoption).

Re: Level 3 Global Outage

#250

Post by jsjohnst »

q3k wrote: post #230
jsjohnst wrote: post #228

Earlier quoted context omitted.

> or even just getting an ASN and some IPv6 PA space and trying to announce it somewhere That’s fairly expensive to do just for a hobby interest, but at least the price has came down since I last looked.

A RIPE ASN (as a end-user through a LIR) and PA v6 will cost you around $100 per year and some mild paperwork, there's plenty of companies/organizations that will help you with that (shameless plug: bgp.wtf, that's us). Afterwards, announcing this space is probably the cheapest with vultr (but their BGP connectivity to VMs is uh, erratic at times) or with ix-vm.cloud, or with packet.net (more expensive). You can also…

> A RIPE ASN (as a end-user through a LIR) and PA v6 will cost you around $100 per year

ARIN starting price is 2.5x that much (used to be 5x) for just the ASN. Glad the pricing is better elsewhere in the world at least!

Post reply on HN