Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping Is Vital to Democracy

themarkup.org

Post reply on HN
Print view View original

61–70 of 132 posts

Re: Web Scraping Is Vital to Democracy

#61

Post by narag »

Someone wrote: post #58
mnmkng wrote: post #20

I think of web scraping as nothing less than automation of human work. There really is no hacking or unauthorized access involved. It's either me, using a browser, to read something publicly accessible on the internet, or it's my computer, which I've programmed to read those things for me, so I can focus on creative work or spend time with friends and family. This applies not only to personal life, but businesses as…

I think of placing a zillion cameras in public places as nothing less than automation of human work. Instead of hiring a million agents to record what the population is doing, you hire one programmer to automate that data collection process. Totally different? Yes, but it shows that scale can affect whether something is OK to do or not, at least for some (I accept that people watching the streets can be useful, but a…

If, for example, I start scraping all the articles on a newspaper’s web site, publish them on a web site, adding my own ads, most people would think that shouldn’t be legal.

Publishing? Ads?

Re: Web Scraping Is Vital to Democracy

#62

Post by Wowfunhappy »

mnmkng wrote: post #20

I think of web scraping as nothing less than automation of human work. There really is no hacking or unauthorized access involved. It's either me, using a browser, to read something publicly accessible on the internet, or it's my computer, which I've programmed to read those things for me, so I can focus on creative work or spend time with friends and family. This applies not only to personal life, but businesses as…

> I think of web scraping as nothing less than automation of human work.

I agree that scraping should be allowed. We wouldn't have Google otherwise.

However, there's something to be said for the fact that some activities are okay at a small scale but become problematic at a large scale—particularly the scale which becomes possible when a task is automated.

For instance, I recently bought an expensive camera and have been having fun walking around the city and taking interesting photos. Many of the photos have people in them. I don't think there's any harm in this.

I store my photos in Aperture, which automatically performs facial recognition on everything in my library. Perhaps some day, if I take enough pictures, Aperture will notice that the same stranger is present in two completely different images. That might be kind of cool—I can see myself fancifully trying to imagine this person's life story. I don't think there would be any harm in that, either.

However, if I aggregated millions of photos from different sources, and used them to track people's movements across the city, that would clearly be a huge problem! Sure, it would be merely automating the work a human could do, but the scale just changes everything!

Re: Web Scraping Is Vital to Democracy

#63

Post by mikem170 »

sokoloff wrote: post #26
mnmkng wrote: post #20

I think of web scraping as nothing less than automation of human work. There really is no hacking or unauthorized access involved. It's either me, using a browser, to read something publicly accessible on the internet, or it's my computer, which I've programmed to read those things for me, so I can focus on creative work or spend time with friends and family. This applies not only to personal life, but businesses as…

While I have substantial agreement with your point of view, when web content is substantially ad-funded, automated scraping effectively bypasses the “payment”. I say this as someone who runs an adblocker, installs them for family, and doesn’t derive income from hosting ads, so I’m not pro-ad; I just realize that while it’s not a crime for me to dump the whole bowl of waiting room candies into my backpack, that’s goin…

I wrote a scraper for myself only to keep an eye on several used truck websites. I was able to do this for craigslist, so I could run it every couple days and check the nearest 75 cities, the list of truck ads, filtering on/out certain keywords, and the pages for the interesting trucks, making a report for myself to help find the vehicle I want.

I could not do this for any of other several large sites advertising used trucks, like commercialtrucker.com, truckpaper.com, and machinio.com. I kept bumping into artificial javascript and captcha limitations that were not worth my time to try to work around.

The thing is that I will probably find what I want on a craigslist site that I can scrape, I'm getting a lot of great info from them, and I'm not going to bother with any of the ad-based sites, it takes to olong to run all the manual searches. They have effectively done a disservice to their (presumably paying) customers who want to sell their vehicles.

Re: Web Scraping Is Vital to Democracy

#64

Post by mschuster91 »

captn3m0 wrote: post #51
mschuster91 wrote: post #17

Earlier quoted context omitted.

> Setting a HTTPS-connect proxy (I use Zap) is a 2 minute process on a desktop that lets you strip SSL easily. Not if the app uses certificate pinning, ships its own version of a SSL library and uses code-signing and obfuscation to prevent you messing around with it. As for the walled garden part: I agree with the general sentiment, but on the other hand I also see the lengths malware authors go to gather data from p…

> Not if the app uses certificate pinning, ships its own version of a SSL library and uses code-signing and obfuscation to prevent you messing around with it. Are there desktop apps that behave this way? Atleast in my experience - I haven't come across anything like this on Linux.

Dropbox does: https://knowledge.broadcom.com/external/article/169397/dropb...

And I seriously hope cert pinning gets adopted by more applications.

Re: Web Scraping Is Vital to Democracy

#65

Post by drdeadringer »

mnmkng wrote: post #20

I think of web scraping as nothing less than automation of human work. There really is no hacking or unauthorized access involved. It's either me, using a browser, to read something publicly accessible on the internet, or it's my computer, which I've programmed to read those things for me, so I can focus on creative work or spend time with friends and family. This applies not only to personal life, but businesses as…

> I think of web scraping as nothing less than automation of human work.

I did this with Python regarding market prices for ETFs, Stocks, and Mutual Funds.

I wrote Python scripts, one of which was to web-scrape current market values, investment distribution, &c for ETFs, stocks, and Mutual Funds for whic I was invested in. I then would have to manually port that into a spreadsheet for "my own special graphs" and such.

I'm sure there are places online which would do this for me if I logged in, entered in all of my information and more, and provide that all for me ... but this story should sound familiar. And due to personal reasons, I've been lagging behind for far too long.

My Point: This information is publicly available and and it serves my purposes. There's no reason why I should not be able to do this. Yes, it's my fault for not using the information [and yes, the information dies with me], but the point is that I automated the gathering of that information for my own self -- and "everyone" else on the planet has that same information.

I never felt like I was illegal doing this. I'm happy to know if there is a "save" way to do this type of aggregate situation otherwise.

Re: Web Scraping Is Vital to Democracy

#66

Post by sokoloff »

mikem170 wrote: post #63
sokoloff wrote: post #26

Earlier quoted context omitted.

While I have substantial agreement with your point of view, when web content is substantially ad-funded, automated scraping effectively bypasses the “payment”. I say this as someone who runs an adblocker, installs them for family, and doesn’t derive income from hosting ads, so I’m not pro-ad; I just realize that while it’s not a crime for me to dump the whole bowl of waiting room candies into my backpack, that’s goin…

I wrote a scraper for myself only to keep an eye on several used truck websites. I was able to do this for craigslist, so I could run it every couple days and check the nearest 75 cities, the list of truck ads, filtering on/out certain keywords, and the pages for the interesting trucks, making a report for myself to help find the vehicle I want. I could not do this for any of other several large sites advertising use…

If AutoTempest covers your search criteria, I’ve found them to be excellent to do repetitive searches. Somewhat fortunately for you, they suck for craigslist searches, but have great coverage for all the other major sites.

Re: Web Scraping Is Vital to Democracy

#67

Post by perpetualpatzer »

jkingsbery wrote: post #57

The connection between web scraping and the case cited seems pretty tenuous. I can't say I know much about the definition of Computer Fraud and Abuse Act, but employment agreements commonly stipulate that the employee can only access information to the extent needed for his or her job. So defining this as unauthorized use wouldn't at all be surprising given how that term is used in practice. The article then goes on…

>The connection between web scraping and the case cited seems pretty tenuous.

The link is another case, Linkedin v hiQ, that is being held pending this case and presents the same question of what counts as "accessing without authorization or exceeding authorization" under the CFAA. The dispute there is whether hiQ could scrape public LinkedIn pages.

The crux of the issue is that if instructions on how to use data that someone has access to without "breaking and entering" don't count as revoking "authorization", this law doesn't cover this officer's actions (though other laws / job requirements may). If just breaking verbal or written terms of use is enough to criminalize it, that covers a whole bunch of things we'd think of as not federal crimes, like lying about your age to set up a facebook account.

Re: Web Scraping Is Vital to Democracy

#70

Post by diegoperini »

mnmkng wrote: post #20

I think of web scraping as nothing less than automation of human work. There really is no hacking or unauthorized access involved. It's either me, using a browser, to read something publicly accessible on the internet, or it's my computer, which I've programmed to read those things for me, so I can focus on creative work or spend time with friends and family. This applies not only to personal life, but businesses as…

> There really is no hacking or unauthorized access involved.

I'm not sure about that. If I had a phone book full of phone numbers (those heavy ones from 80s), would calling every number in that book to find the one I'm looking for be legal/ethical?

P.S: I agree that "Web Scraping Is Vital to Democracy".

Post reply on HN