Live data from Hacker News

Linode Nextgen: The Network

blog.linode.com

181–190 of 193 posts

Re: Linode Nextgen: The Network

#181
post #147
post #146

Earlier quoted context omitted.

Is linode announcing their own netblocks under their own ASN, though? My impression has been that you have more diversity of immediate upstreams than linode...

No idea. I'm on he.net and cogent right now; I'm working on getting more routes, but it's pretty irritating. I have been having some trouble reaching folks on comcast in the midwest, so I contacted comcast and asked about paid peering (they think they are hot shit, so there is no way they would settlement-free peer with me) They want three grand for a 1g peering link. And they aren't on any of the silicon valley peer…

Perhaps it is because peering is not an entitled right. If it was, who would pay for all the costs of network growth?

AFA HE.net pricing, the reality is HE.net does not sell "their" network... they sell other ISPs network via their peering. Their costs are far lower as they don't carry the traffic more than 1-2 router hops.

HE.net does this so much that their peers are feeling that the relationship is no longer "fair" and are not interested in upgrading capacity to enable this business relationship. If HE.net had to pay as much as their peers for the network costs, the prices would be similar.

You get what you pay for....

Re: Linode Nextgen: The Network

#182
post #172

Earlier quoted context omitted.

Sounds like you would be better picking up a decent transit locally to you with a good blend of upstreams.

that's what they say... but the first rule of business is that you will get fucked if it's hard for you to switch providers. I'm already in that position with datacentere space. I've been in that position with bandwidth, too, and it's super painful. Also, nobody is going to give a shit about your three customers way out in the middle of "nobody cares" that can't reach your stuff 'cause cogent and at&t refuse to upgra…

If the problem turns out to be that AT&T and Cogent's peering links are overloaded, and the links between AT&T and Hurricane Electric don't have similar problems, then your border routers should be setting localpref to something low for any route with AT&T's ASN learned on the Cogent BGP session, which should send your outbound traffic to AT&T via Hurricane Electric except when your he.net link is down.

The opposite direction looks more problematic. Some ISPs will let you tag your routes with BGP communities that say ``when advertising to AT&T, prepend some extra copies of some ASN so that AT&T will be unlikely to send me traffic via this path.'' http://www.onesc.net/communities/as174/ does not list any communities which would affect how Cogent advertises routes to AT&T without affecting how Cogent advertises to all of their other peers. On the other hand, while 174:3001 is an extremely coarse tool, your traffic ratios may be sufficiently outbound only that sending most of your inbound traffic via Hurricane Electric might turn out OK, even with Hurricane Electric being only 1G and Cogent 10G (although this change might make your network less robust in the face of a DDOS attack, and you might want to drop 174:3001 when dealing with such an attack).

If the overloaded links are only overloaded in one direction, then you may only need to do one of these things, not both.

Re: Linode Nextgen: The Network

#183
post #172

Earlier quoted context omitted.

Sounds like you would be better picking up a decent transit locally to you with a good blend of upstreams.

that's what they say... but the first rule of business is that you will get fucked if it's hard for you to switch providers. I'm already in that position with datacentere space. I've been in that position with bandwidth, too, and it's super painful. Also, nobody is going to give a shit about your three customers way out in the middle of "nobody cares" that can't reach your stuff 'cause cogent and at&t refuse to upgra…

Do you have traceroutes that demonstrate that the path goes through Cogent and AT&T in both directions? Asymmetric routing is extremely common for things like this.

Re: Linode Nextgen: The Network

#184
post #182
post #172

Earlier quoted context omitted.

that's what they say... but the first rule of business is that you will get fucked if it's hard for you to switch providers. I'm already in that position with datacentere space. I've been in that position with bandwidth, too, and it's super painful. Also, nobody is going to give a shit about your three customers way out in the middle of "nobody cares" that can't reach your stuff 'cause cogent and at&t refuse to upgra…

If the problem turns out to be that AT&T and Cogent's peering links are overloaded, and the links between AT&T and Hurricane Electric don't have similar problems, then your border routers should be setting localpref to something low for any route with AT&T's ASN learned on the Cogent BGP session, which should send your outbound traffic to AT&T via Hurricane Electric except when your he.net link is down. The opposite…

I tried switching my outbound traffic to the customers in question over to he.net (with a static route, which I know is the dumb way to do it.) with no improvement.

Hm. I need to learn more about bgp, specifically tags and communities, can you recommend me a book?

Re: Linode Nextgen: The Network

#185
post #181
post #147

Earlier quoted context omitted.

No idea. I'm on he.net and cogent right now; I'm working on getting more routes, but it's pretty irritating. I have been having some trouble reaching folks on comcast in the midwest, so I contacted comcast and asked about paid peering (they think they are hot shit, so there is no way they would settlement-free peer with me) They want three grand for a 1g peering link. And they aren't on any of the silicon valley peer…

Perhaps it is because peering is not an entitled right. If it was, who would pay for all the costs of network growth? AFA HE.net pricing, the reality is HE.net does not sell "their" network... they sell other ISPs network via their peering. Their costs are far lower as they don't carry the traffic more than 1-2 router hops. HE.net does this so much that their peers are feeling that the relationship is no longer "fair…

>Perhaps it is because peering is not an entitled right. If it was, who would pay for all the costs of network growth?

My complaint here is that comcast charges as much for peering as for transport (with full routes) - I mean, if that's the case, why advertise paid peering services? Every time I've bought full routes I've had the option of asking the provider to only send me customer routes, or to filter on my end for customer routes. Hell, most people will give me a feed with customer routes, and another with full routes, so I could use them as a de-prioritized transport provider of last resort as well as gaining all the benefits of being a peer if I bought transport from them, so yeah, advertising paid peering at transport prices is just silly.

My second complaint is that they aren't on any peering exchanges. Peering exchanges are important because individual interconnects in a datacenter are expensive; the data center charges you a hefty monthly fee for every connection you have to other people at the same datacenter (like half what you'd pay for a shitty but local room in a shared house) /and/ then if you use direct interconnects both parties need to maintain a switch port.

With a peering exchange? you maintain one switchport for all your traffic that goes over the exchange. There are even route servers you can subscribe to if you want to automatically peer with everyone else on the route server at that exchange, making peering a nearly zero work, zero cost thing.

(Of course, if you are sending a /lot/ of traffic, then a individual link probably makes more sense, sure, but I'm not sending a lot of traffic.)

>If HE.net had to pay as much as their peers for the network costs, the prices would be similar.

But they are similar. after negotiation, he.net is more expensive than Cogent. to the tune that commits on he cost the same as overages on Cogent.

HE.net, for me, is actually pretty competitive with Cogent, who I think is their closest competitor, quality-wise. After negotiation, Cogent was slightly cheaper. (The interesting thing about he.net was that they started with a number that was fairly reasonable and stuck with it. It was clear that I didn't have to spend three months dicking around to get the real price, which I appreciate.) Cogent started off asking about as much as I'd expect to pay for L3 after negotiating well, but came down hard.

>You get what you pay for....

If you believe that... well, I've got some Premium VPS here for you. just $100/month per gigabyte ram. They are "premium" - way better than the regular stuff. I'll even wear a suit while I provision you.

People are always willing to charge you more for a shitty product. Always. Professionals spend huge amounts of effort getting you to pay more for the shitty product. (More for the good product, too.)

Hell, in the bandwidth world? It's completely normal to end up paying 1/5th the asking price after negotiation. Yeah, 20% of what the salesperson initially quoted is about what I normally end up paying for transit.

Do you think service is different for people who pay the asking price vs. people who negotiate down to the real price?

I mean, sure, buying transport from L3 is going to get you better connectivity than buying you transport from Cogent, and L3 is generally going to cost you more than Cogent. How much more? with my negotiation skills, about 10x more, but I only got L3 down half from their asking price, and I got Cogent down to 1/5th their asking price; I spent considerably more time on Cogent because I thought that was, well, more realistic, and at the time, Linode, the people I see as my competitors to beat, were mostly he.net, which, quality-wise, is on par with Cogent (I think he.net may be a bit better, and there is some argument if cogent or he.net is slightly better, but I think most people would agree that they are both near the bottom of the pile.)

So yeah. uh, I guess there is some relationship between "real price" and quality... but it is not nearly as close a relationship as people say.

Re: Linode Nextgen: The Network

#186
post #183
post #172

Earlier quoted context omitted.

that's what they say... but the first rule of business is that you will get fucked if it's hard for you to switch providers. I'm already in that position with datacentere space. I've been in that position with bandwidth, too, and it's super painful. Also, nobody is going to give a shit about your three customers way out in the middle of "nobody cares" that can't reach your stuff 'cause cogent and at&t refuse to upgra…

Do you have traceroutes that demonstrate that the path goes through Cogent and AT&T in both directions? Asymmetric routing is extremely common for things like this.

not now. Next time I get a complaint like this, though, I will attempt to document the debug process in more detail, in public.

Re: Linode Nextgen: The Network

#187
Someone in the world are always unsatisfied for everything, yes, everything, no matter what you changed, no matter what you improved, they always always can point something out there to complain about. Example!? Right here!

Re: Linode Nextgen: The Network

#188
post #184
post #182

Earlier quoted context omitted.

If the problem turns out to be that AT&T and Cogent's peering links are overloaded, and the links between AT&T and Hurricane Electric don't have similar problems, then your border routers should be setting localpref to something low for any route with AT&T's ASN learned on the Cogent BGP session, which should send your outbound traffic to AT&T via Hurricane Electric except when your he.net link is down. The opposite…

I tried switching my outbound traffic to the customers in question over to he.net (with a static route, which I know is the dumb way to do it.) with no improvement. Hm. I need to learn more about bgp, specifically tags and communities, can you recommend me a book?

I ended up learning most of what I know about BGP not via a book, which makes that hard.

But basically, back in the days of 16 bit ASNs, communities were just 32 bit values that were attached to routes being advertised by BGP, usually expressed as the two 16 bit halves separated by a colon, generally with the ASN that defines the meaning of the community being on the left of the colon, and that ASN defining arbitrary values on the right side of the colon for flags that they want to recognize.

Then, there are route maps or whatever the router vendor wants to call them that use the data from the communities.

Typically when the router is deciding what route to use, it looks for the route with the highest localpref, and if there's a tie, then it looks at how many ASNs are listed in the path, and if there's still a tie, then the router may use MED. http://www.cisco.com/en/US/tech/tk365/technologies_tech_note... explains how this works in the Cisco BGP implementation with all of the additional factors that are considered; Linux and Juniper BGP implementations may consider a slightly different set of factors.

IIRC, Level 3 generally sets localpref to 86 on routes heard on non-customer peer sessions, and 100 on routes heard on customer BGP sessions, but if customers want localpref to be 90 or 80 or 70 instead, they can send community 3356:90 or 3356:80 or 3356:70. If you had two links to Level 3 and wanted to use one as backup only, you could send community 3356:90 on your routes on your backup link, and potentially no communities on your primary link. If you had a really slow link to Level 3 that was backup only and some other ASN was your primary, then you'd want to send Level 3 3356:80 or 3356:70. I doubt you have any desire to set your upstream's localpref values in your configuration, but this may shed some light on why they offer it to their other customers. Also, if you ever have customers speaking BGP who want to use you as backup only, you need to offer some way for routes they advertise to you to have localpref less than what you set on routes you learn from your upstreams; typically this seems to be done by letting your customers send you a community so the customer controls this, but you could just have a special route-map for customers who are backup only that would just set all of their routes to low localpref regardless of the BGP communities, and just apply the right route-map to those BGP sessions.

http://www.onesc.net/communities/as3356/ mentions that Level 3 supports 65004:[ASN] for four prepends when advertising to ASN. If you were a Level 3 customer, and traffic coming to you via one of their peering links was overloaded, sending a community of that form would be a good way to discourage traffic from taking that link by making the path have a large number of ASNs and thus look undesirable. I say this mostly to illustrate the sort of thing that it looks like you might find useful for Cogent to support, though it appears that Cogent doesn't support that.

It's likely that Hurricane Electric and Cogent will recognize some communities that you could send them that will influence their routing, but as far as I know, their peers won't necessarily end up accepting any communities from you, so the key is to either find documentation on the web of what communties Hurricane Electric and Cogent recognize, or ask them for it.

You can also sometimes get geographic data about which major city is closest to a network block by recieving community data from an upstrem, but it's not obvious to me why you'd want that. It may sometimes be used when hot potato routing is not desired, or when a CDN is trying to identify which geographically close server to use.

Re: Linode Nextgen: The Network

#189
post #184
post #182

Earlier quoted context omitted.

If the problem turns out to be that AT&T and Cogent's peering links are overloaded, and the links between AT&T and Hurricane Electric don't have similar problems, then your border routers should be setting localpref to something low for any route with AT&T's ASN learned on the Cogent BGP session, which should send your outbound traffic to AT&T via Hurricane Electric except when your he.net link is down. The opposite…

I tried switching my outbound traffic to the customers in question over to he.net (with a static route, which I know is the dumb way to do it.) with no improvement. Hm. I need to learn more about bgp, specifically tags and communities, can you recommend me a book?

Also, playing with looking glasses can be helpful in understanding what's going on with BGP from the perspective of other networks. http://www.bgp4.net/wiki/doku.php?id=tools:ipv4_looking_glas... has a list, which doesn't seem to include AT&T. Going to http://www.nordu.net/connectivity/looking-glass/lg.cgi and selecting bgp and entering prgmr.com's IPv4 address isn't very exciting, in that it only lists one ASN in the path (it seems they have direct peering with Hurricane Electric, who is announcing the block that includes that IP address under their ASN). Putting in an IP address advertised under your ASN at the looking glass of a network that peers with both Hurricane Electric and Cogent might be more interesting if you manage to see routes via both.

Re: Linode Nextgen: The Network

#190
post #184
post #182

Earlier quoted context omitted.

If the problem turns out to be that AT&T and Cogent's peering links are overloaded, and the links between AT&T and Hurricane Electric don't have similar problems, then your border routers should be setting localpref to something low for any route with AT&T's ASN learned on the Cogent BGP session, which should send your outbound traffic to AT&T via Hurricane Electric except when your he.net link is down. The opposite…

I tried switching my outbound traffic to the customers in question over to he.net (with a static route, which I know is the dumb way to do it.) with no improvement. Hm. I need to learn more about bgp, specifically tags and communities, can you recommend me a book?

Also, do a Google search for

bgp localpref

and at least skim some of the articles that come up there and some of the things linked to from those articles.

Post reply on HN