Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping Is Vital to Democracy

themarkup.org

Post reply on HN
Print view View original

71–80 of 132 posts

Re: Web Scraping Is Vital to Democracy

#71

Post by arthurcolle »

diegoperini wrote: post #70
mnmkng wrote: post #20

I think of web scraping as nothing less than automation of human work. There really is no hacking or unauthorized access involved. It's either me, using a browser, to read something publicly accessible on the internet, or it's my computer, which I've programmed to read those things for me, so I can focus on creative work or spend time with friends and family. This applies not only to personal life, but businesses as…

> There really is no hacking or unauthorized access involved. I'm not sure about that. If I had a phone book full of phone numbers (those heavy ones from 80s), would calling every number in that book to find the one I'm looking for be legal/ethical? P.S: I agree that "Web Scraping Is Vital to Democracy".

> If I had a phone book full of phone numbers (those heavy ones from 80s), would calling every number in that book to find the one I'm looking for be legal/ethical?

Sure, why not?

Re: Web Scraping Is Vital to Democracy

#72

Post by LunaSea »

krageon wrote: post #50
LunaSea wrote: post #36

Earlier quoted context omitted.

It's not founded on "but advertisements" it's founded on "but paying for content". The fact that websites are developed by people (wages), hosted on infrastructure (hardware renting) and require countless other jobs is somehow magically forgotten in these discussions.

It's not up to me or anyone consuming content to help with figuring out the right way to pay for it. What is up to me is the choice to not be potentially exposed to malware any time I go to read the news. While you may have a point, that point just doesn't matter within the context that we're talking about. Making companies money is not my responsibility.

It absolutely is your responsibility if you are consuming the content.

Consuming the content means agreeing to the premise that the content is paid for using ads.

If you disagree with said premise you may happily browse another website.

Malware is already illegal to install and you are free to sue the website for damages.

I don't like ads either but trying to justify that their should be a choice of browsing a website without the ads it hosts is ludicrous.

Re: Web Scraping Is Vital to Democracy

#73

Post by diegoperini »

arthurcolle wrote: post #71
diegoperini wrote: post #70

Earlier quoted context omitted.

> There really is no hacking or unauthorized access involved. I'm not sure about that. If I had a phone book full of phone numbers (those heavy ones from 80s), would calling every number in that book to find the one I'm looking for be legal/ethical? P.S: I agree that "Web Scraping Is Vital to Democracy".

> If I had a phone book full of phone numbers (those heavy ones from 80s), would calling every number in that book to find the one I'm looking for be legal/ethical? Sure, why not?

I have no idea, I've never done it.

Re: Web Scraping Is Vital to Democracy

#74

Post by PestoDiRucola »

notahaterbut wrote: post #69

Web-scraping can do harm to a business if the scraper does not implement things like timeouts or sleeps between processes. Websites and businesses do not factor bandwidth for the rogue scraper.

You can easily block a harmful scraper like that from the website owner's side.

Re: Web Scraping Is Vital to Democracy

#76

Post by bobobob420 »

Wowfunhappy wrote: post #62
mnmkng wrote: post #20

I think of web scraping as nothing less than automation of human work. There really is no hacking or unauthorized access involved. It's either me, using a browser, to read something publicly accessible on the internet, or it's my computer, which I've programmed to read those things for me, so I can focus on creative work or spend time with friends and family. This applies not only to personal life, but businesses as…

> I think of web scraping as nothing less than automation of human work. I agree that scraping should be allowed. We wouldn't have Google otherwise. However, there's something to be said for the fact that some activities are okay at a small scale but become problematic at a large scale—particularly the scale which becomes possible when a task is automated. For instance, I recently bought an expensive camera and have…

Can the case be made that recording in public is a right (as if always should be) but trying to track where everyone is at every point of time is stalking at a mass scale which should be illegal as stalking on a one to one scale is illegal?

For reference this is what one site (findlaw) has given as what constitutes the crime of stalking:

The crime of stalking can be simply described as the unwanted pursuit of another person. Examples of this type of behavior includes following a person, appearing at a person's home or place of business, making harassing phone calls, leaving written messages or objects, or vandalizing a person's property.

Re: Web Scraping Is Vital to Democracy

#77

Post by krageon »

LunaSea wrote: post #72
krageon wrote: post #50

Earlier quoted context omitted.

It's not up to me or anyone consuming content to help with figuring out the right way to pay for it. What is up to me is the choice to not be potentially exposed to malware any time I go to read the news. While you may have a point, that point just doesn't matter within the context that we're talking about. Making companies money is not my responsibility.

It absolutely is your responsibility if you are consuming the content. Consuming the content means agreeing to the premise that the content is paid for using ads. If you disagree with said premise you may happily browse another website. Malware is already illegal to install and you are free to sue the website for damages. I don't like ads either but trying to justify that their should be a choice of browsing a websit…

> Consuming the content means agreeing to the premise that the content is paid for using ads.

It is in fact not this way, because the content arrives with or without the ads. In the EU EULAs (the dystopian construct that you'd expect to enforce that bit of lunacy) that purport to apply to content you have already accessed are not legally valid. Leaving the legal interpretation aside, me doing one thing doesn't mean consent for something else. Believing otherwise is both unethical and amoral, stances I don't hugely feel like interacting with.

Re: Web Scraping Is Vital to Democracy

#79

Post by dylz »

captn3m0 wrote: post #51
mschuster91 wrote: post #17

Earlier quoted context omitted.

> Setting a HTTPS-connect proxy (I use Zap) is a 2 minute process on a desktop that lets you strip SSL easily. Not if the app uses certificate pinning, ships its own version of a SSL library and uses code-signing and obfuscation to prevent you messing around with it. As for the walled garden part: I agree with the general sentiment, but on the other hand I also see the lengths malware authors go to gather data from p…

> Not if the app uses certificate pinning, ships its own version of a SSL library and uses code-signing and obfuscation to prevent you messing around with it. Are there desktop apps that behave this way? Atleast in my experience - I haven't come across anything like this on Linux.

Many desktop apps behave this way, yes. Anything from a major tech co should at least

Re: Web Scraping Is Vital to Democracy

#80

Post by kordlessagain »

smarx007 wrote: post #52

Okay, now is a good time to put my long-time idea on paper: we need a "cloud" power of attorney (not using the term digital so that people don't confuse it for a regular one just signed digitally). Key idea is this: access to internet and its services is essential (and at the web scale, so is automation), but bot abuse is real. It should be possible for any company to ban bots but then for a person with a account on…

Building a distributed crowd sourced screen crawler.
Post reply on HN