51–60 of 69 posts
Earlier quoted context omitted.
Putting aside legal issues, you don't have any moral problems doing volumes of scraping content that is not yours?
Over time, I've learned to wget every web page and content archive I want to keep. The Internet forgets.
It's anti-climactic that all it was going to take was a compelling business case. Netflix made it easy with their peering initiative [1]. Now Comcast gets to count these bytes against their customers' quotas, and it costs them nearly nothing to deliver the traffic. This reminds me of NNTP, but Netflix is still running their own hardware. [1] Netflix's "Open Connect" https://signup.netflix.com/openconnect/guidelines
I wouldn't be so sure there's no money changing hands. One of the sides caved; and it was probably Netflix. The amount of money the ISPs were asking for from Netflix was not outrageous; the ISPs were more worried about the precedent that providing free interconnects might set. Give one to Netflix for free, and Apple, Google, Amazon and Microsoft all think they're entitled to one too. Strategically, Netflix was holdin…
Yes. Right now the average person only hates Comcast because its a monthly bill, it increases frequently, and there is not much competition, they don't yet hate Comcast for the technical reasons "internet" people do.
But if Comcast starts making Netflix, YouTube, etc not work correctly and starts affecting the average American's daily life it will become a political issue. Congressmen will start seriously discussing turning the internet into a utility.
Earlier quoted context omitted.
I've done this before for large scraping projects. I find the datacenter the target website is hosted in, then get a dedicated server right next to it. I've never gotten better performance.
dumb question, but how were you finding their datacenter?
Earlier quoted context omitted.
Yeah, it's a value proposition. The argument ISPs keep using is that Netflix traffic is costing them too much. I don't think Netflix should have to engineer a solution to that problem AND pay the ISP for the privilege of saving them tons of money. If the costs (4U of rack space, networking equipment, and network engineering around privacy/security) are more than the ISP would save, it's an easy decision. Note that in…
No; no ISP would ever run externally managed equipment inside their network. They would run it in a separate cage with a direct fiber connection to the gateway routers. Rule #1 of network security is that you never run someone else's equipment inside your network; you ALWAYS make them go through a gateway. Also it's largely semantics where a device like OpenConnect is hosted. Netflix could host it at a datacenter acr…
The second paragraph is wrong too, there are much closer relationships possible. They're much deeper than that today, and I would expect in the future to see CDNs much deeper in provider networks.
Earlier quoted context omitted.
I wouldn't be so sure there's no money changing hands. One of the sides caved; and it was probably Netflix. The amount of money the ISPs were asking for from Netflix was not outrageous; the ISPs were more worried about the precedent that providing free interconnects might set. Give one to Netflix for free, and Apple, Google, Amazon and Microsoft all think they're entitled to one too. Strategically, Netflix was holdin…
> is it possible to hate Comcast more than most people already do? Yes. Right now the average person only hates Comcast because its a monthly bill, it increases frequently, and there is not much competition, they don't yet hate Comcast for the technical reasons "internet" people do. But if Comcast starts making Netflix, YouTube, etc not work correctly and starts affecting the average American's daily life it will bec…
I loled. Congress has a lot more important problems on its plate than this (social security, immigration, health care, etc.) On this issue they're just going to do what ever their friendly telecom lobbyist tells them to do.
Earlier quoted context omitted.
Putting aside legal issues, you don't have any moral problems doing volumes of scraping content that is not yours?
Certainly not! If I was re-selling the data, maybe. But I'm generally using it for statistics and data viz. I include the source of the data and I always obey robots.txt. Sometimes I'm even able to talk with the owners beforehand to get their ok. (Don't downvote him, it's a valid question)
300TB is quite a lot, even today.
Earlier quoted context omitted.
Yeah, it's a value proposition. The argument ISPs keep using is that Netflix traffic is costing them too much. I don't think Netflix should have to engineer a solution to that problem AND pay the ISP for the privilege of saving them tons of money. If the costs (4U of rack space, networking equipment, and network engineering around privacy/security) are more than the ISP would save, it's an easy decision. Note that in…
No; no ISP would ever run externally managed equipment inside their network. They would run it in a separate cage with a direct fiber connection to the gateway routers. Rule #1 of network security is that you never run someone else's equipment inside your network; you ALWAYS make them go through a gateway. Also it's largely semantics where a device like OpenConnect is hosted. Netflix could host it at a datacenter acr…
You have no idea how big CDNs do it. I've personally installed CDN gear into ISP racks. Sometimes you get an uplink into a router, other times you sit on a switch with other gear.
That "direct fiber interconnect" is called private peering and mostly used to fill cache boxes on the providers network.
Earlier quoted context omitted.
Putting aside legal issues, you don't have any moral problems doing volumes of scraping content that is not yours?
Over time, I've learned to wget every web page and content archive I want to keep. The Internet forgets.
I spent way too much time pruning stupid crap such as slashdot and started to learn this 'Bayesian classifier' thing.
Your idea is much better.