81–90 of 132 posts
The internet is not vital to democracy, and perhaps as we are learning more now, antithetical to it. So then it logically follows that web scraping is not vital to democracy.
This article focuses a lot on _good_ of web-scraping but I don't think we even need to go that far. Fight against web-scrapers just seems like a complete logical oxymoron: they want data to be public but also select who gets to see it. Our whole web infrastructure are based around clearly distinct public/private exchanges - there's no middleground this and yet people create these absurd hacks like captchas and finger…
Web-scraping can do harm to a business if the scraper does not implement things like timeouts or sleeps between processes. Websites and businesses do not factor bandwidth for the rogue scraper.
I think of web scraping as nothing less than automation of human work. There really is no hacking or unauthorized access involved. It's either me, using a browser, to read something publicly accessible on the internet, or it's my computer, which I've programmed to read those things for me, so I can focus on creative work or spend time with friends and family. This applies not only to personal life, but businesses as…
I think of placing a zillion cameras in public places as nothing less than automation of human work. Instead of hiring a million agents to record what the population is doing, you hire one programmer to automate that data collection process. Totally different? Yes, but it shows that scale can affect whether something is OK to do or not, at least for some (I accept that people watching the streets can be useful, but a…
> If, for example, I start scraping all the articles on a newspaper’s web site, publish them on a web site, adding my own ads
Using a Q-Tip to murder someone is illegal. We don't have special laws against it—it's just that murder is illegal, we have criminal punishment for murder, and the ordinary machinery of the courts and the law is sufficient to handle any instances of murder by Q-Tip. Because it being done with a Q-Tip is the least important part.
If you're scraping and republishing someone else's content, it's not the scraping part that's the problem.
Web-scraping can do harm to a business if the scraper does not implement things like timeouts or sleeps between processes. Websites and businesses do not factor bandwidth for the rogue scraper.
You can easily block a harmful scraper like that from the website owner's side.
Bonus, if your site is scrapable, it is easier to preserve.
Now all we have is SPA 30MB main.min.js mess.
https://news.ycombinator.com/item?id=25254499
Can't really fault The Markup for looking out for their own techniques/strategies