Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Verizon fixed a recent backbone provider issue

blog.edgecast.com

Post reply on HN
Print view View original

21–27 of 27 posts

Re: How Verizon fixed a recent backbone provider issue

#21

Post by X-Istence »

One of the issues with newer switching/routing gear (if you want to call it issues...) is that they have to work at such incredible speeds that they are unable to actually keep up with verifying packet integrity before shooting it down into the next hop.

If you want 10 Gbit/sec line rate, one of the ways to do that is to simply forego any and all verification that packets are not corrupted and shove it out of an ethernet port as fast as possible.

I recently ran into this at work, we had some issues with corrupted packets, and we ended up tracing it down to a Twinax cable that was failing. But every single switch in the path to the server happily forwarded the corrupt packets.

Luckily Cisco has some internal counters that show issues like that, and after tracing it down multiple switches/routers we found the culprit and fixed the issue!

Re: How Verizon fixed a recent backbone provider issue

#22

Post by kbenson »

minaguib wrote: post #4

I've blogged about an almost exact replica of this in 2012: http://mina.naguib.ca/blog/2012/10/22/the-little-ssh-that-so... Funny enough, I'm a direct client of EdgeCast and I believe I witnessed a case of file corruption on their layer last week.

I actually had to stop for a second, and wonder if they actually found (or were notified) of the exact same issue and did their own write up. I wasn't aware your post was from 2012, I first saw it in a recent HN repost.

Re: How Verizon fixed a recent backbone provider issue

#23

Post by jlgaddis »

X-Istence wrote: post #21

One of the issues with newer switching/routing gear (if you want to call it issues...) is that they have to work at such incredible speeds that they are unable to actually keep up with verifying packet integrity before shooting it down into the next hop. If you want 10 Gbit/sec line rate, one of the ways to do that is to simply forego any and all verification that packets are not corrupted and shove it out of an ethe…

> ... they have to work at such incredible speeds that they are unable to actually keep up with verifying packet integrity before shooting it down into the next hop.

Yep: http://en.wikipedia.org/wiki/Cut-through_switching

Re: How Verizon fixed a recent backbone provider issue

#24

Post by dredmorbius »

aroch wrote: post #2

The cynic in me wonders if this is blog post was written solely to serve as positive PR for Verizon in the face of their horrible transit/peering infrastructure management (ie. letting links run hot so they can charge transit providers more money for each increase in commit or letting peering ports run very hot because they don't want to spend the few thousand to add another 10G because they'd rather be paid for tran…

Given the "being good stewards of the Internet" headline: yeah, total submarine.

http://www.paulgraham.com/submarine.html

Re: How Verizon fixed a recent backbone provider issue

#25

Post by thehelix112 »

[Full disclosure: I am an employee at Verizon/EdgeCast. Wasn't sure how to pop that in the title to be honest]. We have updated the blog post to clarify some of the technical questions that have come up in this discussion, particularly around the confusion that Verizon/EdgeCast was the (Tier 1) transit provider in question.

Re: How Verizon fixed a recent backbone provider issue

#26

Post by PhantomGremlin »

aroch wrote: post #2

The cynic in me wonders if this is blog post was written solely to serve as positive PR for Verizon in the face of their horrible transit/peering infrastructure management (ie. letting links run hot so they can charge transit providers more money for each increase in commit or letting peering ports run very hot because they don't want to spend the few thousand to add another 10G because they'd rather be paid for tran…

You're not the only cynic on HN. When I read this I immediately thought of the somewhat different "infrastructure management" that occurred when Verizon extorted Netflix a while ago.

That blog post would have been written something like this:

Verizon: Those sure are nice packets you're asking us to deliver to our mutual customers. It'd be a shame if something happened to them at the interface between our networks.

Netflix: Ouch. Ouch. Ouch. Thank you for bending us over and screwing us. Here, take some of our money. Please make the pain stop.

Re: How Verizon fixed a recent backbone provider issue

#27

Post by PhantomGremlin »

shdon wrote: post #3
aroch wrote: post #2

The cynic in me wonders if this is blog post was written solely to serve as positive PR for Verizon in the face of their horrible transit/peering infrastructure management (ie. letting links run hot so they can charge transit providers more money for each increase in commit or letting peering ports run very hot because they don't want to spend the few thousand to add another 10G because they'd rather be paid for tran…

No company is all bad. I'm sure there are plenty of techs at Verizon and its subsidiaries that are just trying to do the best job they possibly can and are excited and rightfully proud to diagnose and fix such a problem. Looks like they did some good detective work and deserve the bragging rights. We see similar posts pop up once in a while with other such situations (CloudFlare springs to mind). It just happens to b…

> No company is all bad.

I think there are a few that are. I see very little good in a company like Philip Morris. They manufacture a delivery system for an addictive drug that, as a side effect, results in countless cancer deaths.

I couldn't possibly see myself working for a company like that.

OTOH under the right circumstances I'd work for Verizon. Their evil is venial compared to merchants of death. But a man's gotta eat, and that's why we prostitute ourselves out to these evil behemoths. And why we rationalize that we're "just trying to do the best job we possibly can".

Post reply on HN