Web Scraping Is Vital to Democracy
themarkup.org
Web Scraping Is Vital to Democracy
1–10 of 132 posts
Re: Web Scraping Is Vital to Democracy
#2Not only is web scraping in danger, the trend towards HTTPS makes it harder to log traffic, and manage the flow of data to/from browsers on your own computer.
Re: Web Scraping Is Vital to Democracy
#3Not only is web scraping in danger, the trend towards HTTPS makes it harder to log traffic, and manage the flow of data to/from browsers on your own computer.
Can you please explain how you got to that conclusion?
Re: Web Scraping Is Vital to Democracy
#4Re: Web Scraping Is Vital to Democracy
#5It's pretty typical and sad how the US is prosecuting a dirty cop who looks up confidential data in exchange for money: on behalf of the system itself rather than on behalf of the victims' privacy, any limitations on the actions of law enforcement officers in general, or public corruption. In libertarianism/liberalism's final form, the only two kinds of laws left are lèse majesté and violations of terms of service.
Re: Web Scraping Is Vital to Democracy
#6"In both of those instances, the pages and data scraped are publicly available on the internet-no hacking necessary-but sites involved could easily change the fine print on their terms of service to label the aggregation of that information "unauthorized.""
They could. But if they filed a claim under the CFAA without ever sending a cease and desist letter to the alleged intruder, I think the claim would be dismissed.
Is simply changing TOS enough for a CFAA claim to have a reasonable chance of success. I could be wrong, but I believe for every CFAA claim we have seen so far based on "scraping", there was some notice to stop directed specifically at the respondent. If someone thinks just changing TOS (public notice) is enough for a CFAA claim to survive a Motion to Dismiss and there is no need to also contact the alleged offender asking them to stop, then let's hear about the precedent that supports that idea. I do not think there is any such precedent, but I could be wrong.
Re: Web Scraping Is Vital to Democracy
#7"In both of those instances, the pages and data scraped are publicly available on the internet-no hacking necessary-but sites involved could easily change the fine print on their terms of service to label the aggregation of that information "unauthorized."" They could. But if they filed a claim under the CFAA without ever sending a cease and desist letter to the alleged intruder, I think the claim would be dismissed.…
So robots.txt has no meaning?
Re: Web Scraping Is Vital to Democracy
#8This article focuses a lot on _good_ of web-scraping but I don't think we even need to go that far.
Fight against web-scrapers just seems like a complete logical oxymoron: they want data to be public but also select who gets to see it. Our whole web infrastructure are based around clearly distinct public/private exchanges - there's no middleground this and yet people create these absurd hacks like captchas and fingerprints to fight the nature of the internet.
Finally everyone wants benefits of public data (search engine indexing etc) but don't really want to give anything back to the ecosystem. It's just pure greed and law, our society and government shouldn't aid it in any way, shape or form.
Re: Web Scraping Is Vital to Democracy
#9Ever considered Democracy doesnt work
Look at the last 5-25-55-255 years
Re: Web Scraping Is Vital to Democracy
#10Not only is web scraping in danger, the trend towards HTTPS makes it harder to log traffic, and manage the flow of data to/from browsers on your own computer.
Can you please explain how you got to that conclusion?
Not OP, but it's very hard to intercept and monitor the traffic of an app that uses code signing, obfuscation and certificate pinning properly.
What used to be a five minute "put tcpdump on the access point / router to work" job pre-HTTPS everywhere now is a many days worth job of messing around - out of reach for all but the really dedicated, and as a result it is very hard for an user to have actual visibility over where their data flows.