Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Observations of an Internet Middleman

blog.level3.com

Post reply on HN
Print view View original

21–30 of 180 posts

Re: Observations of an Internet Middleman

#21

Post by jeffasinger »

wdewind wrote: post #16
preinheimer wrote: post #13

Earlier quoted context omitted.

Part of the issue is the monopoly most broadband providers have in the US and Canada. If you had two truly distinct cable options, say Comcast and TascCom. Your friends with Comcast say Netflix sux, low quality, lots of problems. Your friends with TascCom say netflix is awesome, great HD content, too bad there's only like 8 3D movies! You'll sign up for TascCom. Comcast's board will get angry they're losing customers…

Yeah, this does seem to be a core issue. How can we make it easier for people to start last-mile ISPs? It seems like a pretty capital intensive business to get involved in :)

The core issue isn't at the last mile, but in connecting a last mile network to the internet?

Maybe we make Cable companies share their last mile networks like we did with DSL?

Re: Observations of an Internet Middleman

#22

Post by xhrpost »

wdewind wrote: post #12
preinheimer wrote: post #5

Earlier quoted context omitted.

I think your issue is here: "Doesn't Level 3 buy a contract from Comcast that says "we get to send this much data per month"? If Netflix (or Level 3) tries to push more data through that, which causes congestion, it seems like a contractual fact that Comcast is either holding up their end of the bargain or not." Level 3 doesn't buy the connection from Comcast. Comcast buys a connection from Level 3. Comcast needs to…

Thanks a lot for your response (and to the others who responded as well, appreciate it!). What you're saying makes sense, but just to play devils advocate... Disclaimer: The blog I'm about to post was linked from Comcast's blog. They claim he is independent, I know nothing about him. "Netflix could use multiple providers to connect to ISPs and could also use third party CDNs like Akamai, EdgeCast and Limelight, who a…

If the ISP is letting a connection become saturated in order to place pressure on a third party provider to provide settlement, they are doing so at the cost of any other providers running over those trunks. Since switching to AT&T, I've noticed decreased Netflix performance and sometimes Imgur loads like I'm on dial-up again. I haven't checked the routes but it has me wondering if it's shared congestion. To your point, I'm not sure this is a moral hazard yet but it does seem like one of those things that could be very bad for the customer and the ISP can simply get away with it due to lack of competition.

Re: Observations of an Internet Middleman

#23

Post by notdonspaulding »

exelius wrote: post #3
wdewind wrote: post #2

Here's the part that doesn't make sense to me, and hopefully someone can explain it: Netflix pays Level 3 and Cogent to connect them to Comcast's network. Comcast claims that only the Level 3 connection is saturated, and that Netflix is sending all their bandwidth over Level 3 because it's cheaper for them. Doesn't Level 3 buy a contract from Comcast that says "we get to send this much data per month"? If Netflix (or…

I think Level3's issue is that Comcast is unwilling to increase the capacity of their links without payment from Level3. If Level3 engages in settlement-free peering with Comcast and their network traffic graph looks anything like the 100gbit link in the article, I don't blame them: it's a 20:1 disparity in traffic in vs traffic out. What's happening here is that Level3 and other transit providers are starting to see…

> it's a 20:1 disparity in traffic in vs traffic out.

Are you talking about on this graph?

http://blog.level3.com/wp-content/uploads/2014/05/route_info...

Because all the numbers I see are showing about a 5:1 or 6:1 imbalance.

> ...transit providers are starting to see their industry be squeezed by the big ISPs.

No.

> ...why do transit providers even exist?

Because it's untenable (and inefficient) for Comcast to build a separate fiber network to every service provider (e.g. to start providing a service on your version of the internet, you'd have to build your own network connection to Comcast, and then again to Verizon, and again to AT&T, and again to every other ISP on the planet).

Take another look at the L3 network map from the blog post:

http://blog.level3.com/wp-content/uploads/2014/05/network_ma...

If you think transit providers are purely middlemen for the sake of middlemen then you're basically saying that every service provider should build out a network of that same size in order to reach every ISP.

> Settlement-free peering assumes that bandwidth usage is roughly symmetric

No. It assumes that the benefit to each network is roughly symmetric. Put another way, if Netflix videos only degrade on Comcast's network, Comcast's goodwill with its own customers will increase as a result of increasing transit/peering capacity. To a normal company, that's a tangible benefit to the agreement enough to make it worthwhile.

Comcast empirically does not care about the goodwill of its customers, which is not surprising given its monopoly of local markets. If customers had alternatives to Comcast and it became known that Netflix worked on one ISP and not the other, customers would switch ISPs en masse.

Re: Observations of an Internet Middleman

#24

Post by exelius »

wmf wrote: post #11
exelius wrote: post #8

Earlier quoted context omitted.

Because the cost of a packet is borne by the receiver while the sender is the one monetizing it. The assumption is that if you're sending traffic to a customer, you are being paid for that activity through subscription fees, ads, donations, etc. That packet costs the receiver some minuscule amount to process. Even though it was requested by the ultimate receiver, the ultimate sender is only sending the packet because…

Because the cost of a packet is borne by the receiver... That's true for hot-potato routing, but CDNs generally use cold-potato routing. As for monetization, both Netflix and Comcast are getting paid by their respective customers.

It's no less true for cold-potato routing; the source is just closer to the endpoint. And CDNs doing cold-potato routing happily pay for their transit because the service THEY make money off of is providing CDN services to their customers, who theoretically make money off the content they pay the CDNs to distribute.

What it comes down to is Comcast has no incentive to ensure its routes to various Internet transit providers are GOOD if they're not monetizing them. Why should Comcast be treated differently than any other transit provider and not allowed to monetize its transit services? Netflix wasn't paying Comcast, so Comcast had no responsibility to help Netflix make its own service better when there were other ways Netflix could have alleviated the bandwidth situation (e.g. buy transit through another company).

Re: Observations of an Internet Middleman

#25

Post by cortesoft »

wdewind wrote: post #2

Here's the part that doesn't make sense to me, and hopefully someone can explain it: Netflix pays Level 3 and Cogent to connect them to Comcast's network. Comcast claims that only the Level 3 connection is saturated, and that Netflix is sending all their bandwidth over Level 3 because it's cheaper for them. Doesn't Level 3 buy a contract from Comcast that says "we get to send this much data per month"? If Netflix (or…

I think a big problem with your reasoning centers on word choice; you phrased it as Level 3 having a contract that says "we get to send this much data per month", and then Level 3 attempting to send more data than that agreement.

However, you must remember that every single packet that Level 3 sends to the Comcast network is a packet that was requested BY A COMCAST CUSTOMER. Level 3 isn't just deciding to send a bunch of traffic over Comcasts network, they are sending the data requested by Comcast customers to that network.

This is an important distinction, and is usually part of all of these 'settlement-free' peering arrangements. In agreeing to peer with Level 3, I am sure Comcast has an agreement that they will not send any traffic to Comcast's network that is not actually destined for a Comcast customer. Now, if Level 3 were sending traffic to Comcast and expecting Comcast to route, for free, that traffic to another 3rd part network, that would NOT be kosher, and Comcast could fairly ask for either money or for Level 3 to stop sending that traffic to them.

Level 3 is ONLY sending the traffic to Comcast that Comcast is requesting; it is THEIR customers who are creating this demand for Level 3's customer's content. Comcast told their customers that for $X per month, they would get Y amount of internet bandwidth, which will in almost all cases be traffic originating from a non-Comcast network; that is just the nature of the internet. Even though Level 3 is willing to send the data that Comcast's customers want to the Comcast network for free (settlement free), Comcast is instead refusing to accept all the traffic that their own customers are requesting and demanding additional payment to provide the service they already sold and are paid for.

Re: Observations of an Internet Middleman

#26

Post by cortesoft »

exelius wrote: post #10
preinheimer wrote: post #5

Earlier quoted context omitted.

I think your issue is here: "Doesn't Level 3 buy a contract from Comcast that says "we get to send this much data per month"? If Netflix (or Level 3) tries to push more data through that, which causes congestion, it seems like a contractual fact that Comcast is either holding up their end of the bargain or not." Level 3 doesn't buy the connection from Comcast. Comcast buys a connection from Level 3. Comcast needs to…

This would all be fine and good if Level3 didn't make money from selling bandwidth. Comcast doesn't have to buy a connection from Level3; Level3 will give Comcast a link for free because they know they can turn around and charge their own customers for access to it. Comcast is really just cutting out the middleman here; and that scares Level3.

They could only cut out the middle man if they built their own global network; they don't have any fiber running to Europe or Asia or elsewhere. If there were no middle man, Comcast customers could only reach other Comcast customers.

Re: Observations of an Internet Middleman

#27

Post by exelius »

mixologic wrote: post #17
exelius wrote: post #10

Earlier quoted context omitted.

This would all be fine and good if Level3 didn't make money from selling bandwidth. Comcast doesn't have to buy a connection from Level3; Level3 will give Comcast a link for free because they know they can turn around and charge their own customers for access to it. Comcast is really just cutting out the middleman here; and that scares Level3.

But they cant cut out the middleman. Comcast doesnt have a direct connection to netflix or anywhere else on the internet. I think Level 3 should straight up shut off their peering agreement with comcast and watch how all the last mile customers blame comcast for the issue, until such time as comcast stops playing games.

But the Internet doesn't work that way: if Level3 shut off their peering with Comcast, Level3's customers would just find another transit provider and route around L3 because those companies are still in business to make money. They don't care about squabbles between Comcast and L3; they just want their monetized traffic to get delivered. The last-mile customers probably wouldn't even notice; but Level3's customers would likely be forced to pay more for transit from another company. But they would still pay.

Re: Observations of an Internet Middleman

#28

Post by rayiner »

jeffasinger wrote: post #21
wdewind wrote: post #16

Earlier quoted context omitted.

Yeah, this does seem to be a core issue. How can we make it easier for people to start last-mile ISPs? It seems like a pretty capital intensive business to get involved in :)

The core issue isn't at the last mile, but in connecting a last mile network to the internet? Maybe we make Cable companies share their last mile networks like we did with DSL?

> Maybe we make Cable companies share their last mile networks like we did with DSL?

Which killed investment in DSL. Who wants to spend billions on infrastructure they are forced to lease out at wholesale rates to competitors?

Re: Observations of an Internet Middleman

#29

Post by danielweber »

xhrpost wrote: post #22
wdewind wrote: post #12

Earlier quoted context omitted.

Thanks a lot for your response (and to the others who responded as well, appreciate it!). What you're saying makes sense, but just to play devils advocate... Disclaimer: The blog I'm about to post was linked from Comcast's blog. They claim he is independent, I know nothing about him. "Netflix could use multiple providers to connect to ISPs and could also use third party CDNs like Akamai, EdgeCast and Limelight, who a…

If the ISP is letting a connection become saturated in order to place pressure on a third party provider to provide settlement, they are doing so at the cost of any other providers running over those trunks. Since switching to AT&T, I've noticed decreased Netflix performance and sometimes Imgur loads like I'm on dial-up again. I haven't checked the routes but it has me wondering if it's shared congestion. To your poi…

I have two net connections, and both are generally reliable, but I've noticed that the DSL connection (over Windstream) is very slow for certain sites like imgur and tumblr and AWS, so I've set those to use the cable modem (Time Warner) connection, where they work great.

Since I have 2 connections I definitely have competition in my area. (Wireless is another option.) So I don't get why if a connection has issues, it's always the DSL one, and always with certain other sites.

Re: Observations of an Internet Middleman

#30

Post by notdonspaulding »

wdewind wrote: post #12
preinheimer wrote: post #5

Earlier quoted context omitted.

I think your issue is here: "Doesn't Level 3 buy a contract from Comcast that says "we get to send this much data per month"? If Netflix (or Level 3) tries to push more data through that, which causes congestion, it seems like a contractual fact that Comcast is either holding up their end of the bargain or not." Level 3 doesn't buy the connection from Comcast. Comcast buys a connection from Level 3. Comcast needs to…

Thanks a lot for your response (and to the others who responded as well, appreciate it!). What you're saying makes sense, but just to play devils advocate... Disclaimer: The blog I'm about to post was linked from Comcast's blog. They claim he is independent, I know nothing about him. "Netflix could use multiple providers to connect to ISPs and could also use third party CDNs like Akamai, EdgeCast and Limelight, who a…

> "It seems like it would be great if they could figure out a mutually beneficial situation, but I guess I fail to see the moral hazard."

This article in particular is making the argument that at the level that L3, Comcast, AT&T, Netflix and others are operating, it's almost always mutually beneficial to peer. The article also makes the point that the capital expenditure is negligible compared to the benefit to both sides.

The moral hazard here is that Comcast is (shrewdly) taking the bet that their customers won't blame them for what appears to be Netflix's reduced performance. If there was actual competition in the local ISP space, customers would catch on that the performance is only worse on Comcast's network, but that's not the world we live in.

> Barring, of course, the monopoly argument, I see no moral hazard here.

You can't just ignore the monopoly argument. There's nothing inherently unfair about a monopoly, but if you use it to extort money from people then its no surprise that people call you out on it. In the Good Old Days, you'd see monopolies try to squash their competitors. This is more nuanced than that. Monopolistic-extortion-action-at-a-distance, if you will.

Post reply on HN