Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping Is Vital to Democracy

themarkup.org

Post reply on HN
Print view View original

51–60 of 132 posts

Re: Web Scraping Is Vital to Democracy

#51

Post by captn3m0 »

mschuster91 wrote: post #17
captn3m0 wrote: post #16

Earlier quoted context omitted.

I'd say this is a consequence of making our phones opaque rather than HTTPS-everywhere. Setting a HTTPS-connect proxy (I use Zap) is a 2 minute process on a desktop that lets you strip SSL easily. Doing the same on our mobile devices is much more tougher, because we've let feature phones become walled gardens.

> Setting a HTTPS-connect proxy (I use Zap) is a 2 minute process on a desktop that lets you strip SSL easily. Not if the app uses certificate pinning, ships its own version of a SSL library and uses code-signing and obfuscation to prevent you messing around with it. As for the walled garden part: I agree with the general sentiment, but on the other hand I also see the lengths malware authors go to gather data from p…

> Not if the app uses certificate pinning, ships its own version of a SSL library and uses code-signing and obfuscation to prevent you messing around with it.

Are there desktop apps that behave this way? Atleast in my experience - I haven't come across anything like this on Linux.

Re: Web Scraping Is Vital to Democracy

#52

Post by smarx007 »

Okay, now is a good time to put my long-time idea on paper: we need a "cloud" power of attorney (not using the term digital so that people don't confuse it for a regular one just signed digitally).

Key idea is this: access to internet and its services is essential (and at the web scale, so is automation), but bot abuse is real. It should be possible for any company to ban bots but then for a person with a account on the website to say: I am giving this tool my cloud power of attorney (best if signed through a digital govt id system, e.g. https://en.wikipedia.org/wiki/BankID) and I take responsibility for what it does; you may not block it or erect CAPTCHAs for it, it scrapes/takes actions on your website on my behalf for my personal needs. This would make running a bunch of scripts on your own NUC or Pi an inalienable right while still allowing companies to fight unfair competition and plain simple DoS attacks.

Re: Web Scraping Is Vital to Democracy

#53

Post by mstipetic »

jagged-chisel wrote: post #38
artembugara wrote: post #35

Earlier quoted context omitted.

I totally agree with the first part where you say it's fine to scrape. It is just an automation. But you don't mention what you do with this data after. 1. You store it somewhere (not in your brain) 2. You extract value out of it (directly or indirectly) That's why I understand why it is a problem for those who publish this data. One thing I always try to say to everyone who argues about web scraping: web scraping is…

> That's why I understand why it is a problem for those who publish this data. I still don't understand why it's a problem for those who publish the data. If the "data" is facts (e.g. lists of values associated with objects, like the colors available on automobiles), this data is not protected by copyright, there is no remedy if it's republished, and if a business relies on limiting access to this data, that business…

So if somebody compiles the data, charges an N amount for access, it's fine that some entity buys a one time access and then releases it for free?

Re: Web Scraping Is Vital to Democracy

#55

Post by bangtoyib »

agustif wrote: post #34

Last weekend I made a little playwright script to checkout stock item status of decathlon products. The bike -Rodillo- Has been out of stock for several weeks. While coding it it was briefly on stock but I was too slow with my card input hahaha and so I neeed to now automate the buy part as soon as it's available, or make that run every 5min and mail/notify me if available... Btw this was super easy, playwright as a…

nice

Re: Web Scraping Is Vital to Democracy

#56

Post by naringas »

and so is free (FSF-style) and open source software. but apparently a healthy economy (with a 'free' market) is more important... o̶h̶,̶ ̶a̶l̶s̶o̶ ̶n̶a̶t̶i̶o̶n̶a̶l̶ ̶s̶e̶c̶u̶r̶i̶t̶y̶ ̶a̶n̶d̶ ̶s̶o̶m̶e̶t̶h̶i̶n̶g̶ ̶a̶b̶o̶u̶t̶ ̶t̶h̶e̶ ̶c̶h̶i̶l̶d̶r̶e̶n̶.̶

Re: Web Scraping Is Vital to Democracy

#57

Post by jkingsbery »

The connection between web scraping and the case cited seems pretty tenuous. I can't say I know much about the definition of Computer Fraud and Abuse Act, but employment agreements commonly stipulate that the employee can only access information to the extent needed for his or her job. So defining this as unauthorized use wouldn't at all be surprising given how that term is used in practice.

The article then goes on to defend web scraping which is very different because (1) the person in question accessed data manually, (2) the person in question had access to confidential information, (3) the person in question used the data in a way he must have agreed not to as part of his employment. It's hard to see how someone could connect whatever precedent this case sets to web scraping.

Re: Web Scraping Is Vital to Democracy

#58

Post by Someone »

mnmkng wrote: post #20

I think of web scraping as nothing less than automation of human work. There really is no hacking or unauthorized access involved. It's either me, using a browser, to read something publicly accessible on the internet, or it's my computer, which I've programmed to read those things for me, so I can focus on creative work or spend time with friends and family. This applies not only to personal life, but businesses as…

I think of placing a zillion cameras in public places as nothing less than automation of human work. Instead of hiring a million agents to record what the population is doing, you hire one programmer to automate that data collection process.

Totally different? Yes, but it shows that scale can affect whether something is OK to do or not, at least for some (I accept that people watching the streets can be useful, but also have my doubts about doing that at scale, whether by cameras or by hiring a million agents)

With both cameras and copy-pasting web content, there’s the issue what you do with the data. If, for example, I start scraping all the articles on a newspaper’s web site, publish them on a web site, adding my own ads, most people would think that shouldn’t be legal.

If you agree, we’re now haggling over the price (https://quoteinvestigator.com/2012/03/07/haggling/). That’s where things get difficult, but I think the entire spectrum from white (scraping for this goal is fine) via grey to black exists.

Re: Web Scraping Is Vital to Democracy

#60

Post by emiliobumachar »

SergeAx wrote: post #47

Okay, great point, but who will pay for that? Serving content costs money: traffic, extra load on CPU and storages, and now on serverless functions too. One spider may be equivalent of hundreds of real users. And it spoils caches too.

Crackpot regulation proposal: scraping is legal if you favor mirror sites and afterwards create a mirror yourself and maintain it to some arbitrary standard.
Post reply on HN