Live data from Hacker News

How Verizon fixed a recent backbone provider issue

blog.edgecast.com

11–20 of 27 posts

Re: How Verizon fixed a recent backbone provider issue

#11
post #9
post #6

Full disclosure: thehelix112's profile indicates that it belongs to "Directory of Security Development ( http://www.edgecast.com)" .

Self submission / promotion is not an issue here if it's not illegitimated seeded with votes.

Nonetheless, I think it would be better if the title had been worded such that this relationship was made clear.

Re: How Verizon fixed a recent backbone provider issue

#12
post #11
post #9

Earlier quoted context omitted.

Self submission / promotion is not an issue here if it's not illegitimated seeded with votes.

Nonetheless, I think it would be better if the title had been worded such that this relationship was made clear.

No one does that except for Show HN and that's only for the debut of a new project.

Re: How Verizon fixed a recent backbone provider issue

#13
post #6

Full disclosure: thehelix112's profile indicates that it belongs to "Directory of Security Development ( http://www.edgecast.com)" .

As a guy that has done some hardware for router interfaces, this was an interesting post regardless of source.

There are lots of built-in monitoring functions for the hardware interfaces. For example, the laser driver for the transmitter often drops in power before failure, so there is a hardware function built-in on the receiver side to report the avg laser power. If the power drops below a certain threshold, an alarm goes off on the monitoring software.

The author is pointing out an interesting failure mode not automatically caught. As a shot in the dark: In the receiver, the hardware is spec'd to receive 31 zeros in a row (iirc, which btw goes way back to early SONET specs). If the detector (eg) is degrading in a way that causes a particular string of zeros (or some other fixed pattern) to fail, you would see it show up much like the test that was run.

What's interesting is that the errors did not light up alarms on the monitoring. Perhaps crc errors have a threshold, and the bit pattern causing the failure was showing up statistically below the threshold.

Anyway, I'm sure there's some hardware failure-analysis guy out there who would love to look harder at that link.

Re: How Verizon fixed a recent backbone provider issue

#14

I realize Verizon bought Edgecast last year but why is the post being promoted by Edgecast and not Verizon themselves? Edgecast doesn't own or operate a backbone, they are an edge provider so this is a completely separate concern. This is typical dirty Verizon/big telco PR in my opinion. Edgecast is a great CDN and company, I'm curious to see how this new ownership plays out.

Oh come on. It's an interesting article.

Re: How Verizon fixed a recent backbone provider issue

#15
post #2

The cynic in me wonders if this is blog post was written solely to serve as positive PR for Verizon in the face of their horrible transit/peering infrastructure management (ie. letting links run hot so they can charge transit providers more money for each increase in commit or letting peering ports run very hot because they don't want to spend the few thousand to add another 10G because they'd rather be paid for tran…

... dude, it's a company blog post. they're writing about what they do in their noc. it's basic SEO and marketing, not some telco conspiracy.

Re: How Verizon fixed a recent backbone provider issue

#16

As a (smaller) CDN, we routinely solve these sorts of problems. Never occurred to be to blog about it in such manner. But then again, we suck at PR.

Please blog about it! It's usually an interesting read. People eat up posts like this and from CloudFlare.

Re: How Verizon fixed a recent backbone provider issue

#18
As someone who lives in the UK, and purchases bandwidth all over the world I say this:

That's your fucking job, what do you want, a medal?

Seriously, the level of crap that verizon, Level 3 and AT&T put out is immense. I was trying to get a 100meg line in downtown redwood city. (this was q4 2013) at first I was told that the exchange was full. After that they said I could have bonded T3. After much screaming and shouting they decided that they would do me a favour and provision a fibre line.(bear in mind this was part of a large global account, with MPLS and other such niceties)

In the end it took 3 months of epic hassle (this was without way leave) just to get to the point of connection. after another 6 weeks of pointless meandering, I had fibre.

However because of the level of skill at the NOX it was another 3 fucking weeks to get it lit properly. (15% packet loss is not acceptable by the way)

The worst part of this is the cost, $4500 a month for a steaming pile of shite, backed by people to thick to open doors effectively.

In london this is how it went down: Phone up $provider, I want a 1 gig line please.

$provider: sure, that might be up to 90 days, pending legals and survey

Me: ok

$provider: (week later) survey is done, line should be lit in a month

$provider: oh and thats £1500 a month

In conclusion, just fuck right off, get off your arse and fucking do what we all pay you for, provide some fucking bandwidth.

Re: How Verizon fixed a recent backbone provider issue

#19

As a (smaller) CDN, we routinely solve these sorts of problems. Never occurred to be to blog about it in such manner. But then again, we suck at PR.

Yes, as Kalleboo said, please do. I can't speak for others, but personally I find such postmortems to be among the most interesting kind of articles on here.

Re: How Verizon fixed a recent backbone provider issue

#20

As someone who lives in the UK, and purchases bandwidth all over the world I say this: That's your fucking job, what do you want, a medal? Seriously, the level of crap that verizon, Level 3 and AT&T put out is immense. I was trying to get a 100meg line in downtown redwood city. (this was q4 2013) at first I was told that the exchange was full. After that they said I could have bonded T3. After much screaming and shou…

If they're provisioning fiber lines and not documenting loss, doing OTDR, etc they're doing it wrong. It should not have gone live and handed to the customer without passing a fucking smell test.

edit: my past experience with CenturyLink was always great -- they provided us with extremely detailed documents of the provisioned line with loss, bandwidth tests, etc.

Post reply on HN