Is "quarter billion" used to make it sound like a bigger number? Even "half" is aggressive, imo.
How to crawl a quarter billion webpages in 40 hours (2012)
11–20 of 25 posts
Re: How to crawl a quarter billion webpages in 40 hours (2012)
#12Earlier quoted context omitted.
Common Crawl contains the HTML? I wonder how this is legal and considered acceptable. I wish I knew how even Google and others get away with scraping content, saving it, and utilizing it for profit without sharing any revenue with the original webmasters. I know people can opt out of crawling, for those that actually respect that. But still, am I the only one who feels like this is wrong? I guess I have this view tha…
> I wonder how this is legal and considered acceptable. I wish I knew how Google and others gets away with scraping content [...] Well, here's the answer: "transformative" reuse of content is explicitly permitted under copyright law. Simply reproducing the content and charging for it would not fall under this provision, but building an archive of publicly available information is - quite appropriately, permissible. T…
I believe the Internet Archive is much less clear in this regard than public search engines. IA doesn't even have a clear takedown policy and no webmaster tools in place to give owners control on the archived content.
Their crowler does obey to robots.txt rules but if you want content to be removed permanently you have to ask politely by email and in my experience they simply block the site urls from being searched but they don't make clear at all if content was actually removed from their servers.
Re: How to crawl a quarter billion webpages in 40 hours (2012)
#13old HN comments ( 3 years ago ) https://news.ycombinator.com/item?id=4367933
Re: How to crawl a quarter billion webpages in 40 hours (2012)
#14Earlier quoted context omitted.
Common Crawl contains the HTML? I wonder how this is legal and considered acceptable. I wish I knew how even Google and others get away with scraping content, saving it, and utilizing it for profit without sharing any revenue with the original webmasters. I know people can opt out of crawling, for those that actually respect that. But still, am I the only one who feels like this is wrong? I guess I have this view tha…
> I wonder how this is legal and considered acceptable. I wish I knew how Google and others gets away with scraping content [...] Well, here's the answer: "transformative" reuse of content is explicitly permitted under copyright law. Simply reproducing the content and charging for it would not fall under this provision, but building an archive of publicly available information is - quite appropriately, permissible. T…
I don't think "Sharing is good" is true in the real world. If you apply that as a blanket statement, you'll end up in trouble.
What is legal is not always ethical. I think there's an interesting story there about how Google is legal, if someone doesn't automatically assume it should be just because it is.
The text online isn't always similar to a published text of the past. There is a personal overlap today that changes the rules. Such as this text I'm publishing right now. Forgetting about all the legalities and technicalities, I still feel like it is different than a page published in a book. I still feel like I should have the power to edit or delete it whenever I want in the future, yet Hacker News disagrees and removes my right to modify it, forever capturing it as if it owns it, not me. I still feel like this text is more transitory, where its relevance is mostly right now, and if it were deleted in a month it would be fine, because it's mostly just chit chat.
Certainly we could live in a world where everyone has microphones transcribing everything they ever say, which is transmitted to Google, and provided to researchers, where all kinds of uses could emerge. But that's a different world than the one where we've developed rules for today. Right now, I feel like most things I say are in passing, and should not only disappear, but won't spread where someone is capturing and propagating it beyond my control.
What control do I have over my text that is in this Common Crawler database? What if it captured information that was considered to be ephemeral in the website's context, and ripped it out of its home where it's now part of this collective publication, where anyone can use it for anything?
Sharing could be good in a world where people are not selfish and malicious. But in this one, many people will use whatever data they can get their hands on for selfish and malicious purposes, that do not benefit you, the author, in any way. I bet a large percentage of use for that Common Crawler database was harmful to society, such as for helping spammers generate fake content.
Re: How to crawl a quarter billion webpages in 40 hours (2012)
#15I feel like more companies are building their businesses around web crawling and parsing data. There are lots of players in the eCommerce space that monitor pricing, search relevance, and product integrity. Each one of these companies has to build some sort of a templating system for defining crawl jobs, a definition of parsing rules to extract the data, and monitoring system to alert when the underlining HTML of a site has changed from their predefined rules. I'm interested in these aspects. Building a distributed crawler is easier than ever.
Re: How to crawl a quarter billion webpages in 40 hours (2012)
#16Is "quarter billion" used to make it sound like a bigger number? Even "half" is aggressive, imo.
I was going to ask this very question. Seems like the appropriate way would be to say 250 million. I mean you wouldn't say I got a quarter hundred dollars or quarter thousand...
Re: How to crawl a quarter billion webpages in 40 hours (2012)
#17No one ever talks about a particular topic though when it comes to web crawling etc. How do you avoid all the "bad" sites as in, really bad shit? The stuff that your ISP could use as evidence against you when in fact it was just your code running and it happened to come across one of these sorts of sites. How do you deal with all that? That is the only thing stopping me from experimenting around with web crawling.
Re: How to crawl a quarter billion webpages in 40 hours (2012)
#18No one ever talks about a particular topic though when it comes to web crawling etc. How do you avoid all the "bad" sites as in, really bad shit? The stuff that your ISP could use as evidence against you when in fact it was just your code running and it happened to come across one of these sorts of sites. How do you deal with all that? That is the only thing stopping me from experimenting around with web crawling.
I used to do some crawling on comcast using what is now IBM Watson Explorer. I got a ton of phone calls from them. It sounded like those calls would go away if I just paid a little bit more for business class service.
Re: How to crawl a quarter billion webpages in 40 hours (2012)
#19This isn't particularly difficult anymore. The most interesting challenges in web crawling around turning a diaspora of web content into usable data. E.g., how to get prices from 10 million product listings from 1,000 different e-retailers?
Re: How to crawl a quarter billion webpages in 40 hours (2012)
#20I don't understand his hesitancy in releasing his crawler code. I imagine there are plenty for people to access and alter for malicious use if they desired, so why does releasing his such a big deal?