Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping Is Vital to Democracy

themarkup.org

Post reply on HN
Print view View original

101–110 of 132 posts

Re: Web Scraping Is Vital to Democracy

#101

Post by smarx007 »

sundarurfriend wrote: post #96
smarx007 wrote: post #52

Okay, now is a good time to put my long-time idea on paper: we need a "cloud" power of attorney (not using the term digital so that people don't confuse it for a regular one just signed digitally). Key idea is this: access to internet and its services is essential (and at the web scale, so is automation), but bot abuse is real. It should be possible for any company to ban bots but then for a person with a account on…

> It should be possible for any company to ban bots but then for a person with a account on the website to say: I am giving this tool my cloud power of attorney That's basically what API keys are, aren't they?

No, many providers require to apply for API access, to pay for it, to be bound by T&C (some APIs require to you to cache data for max 24h) and also provide a crippled subset of actions through it. What I was saying is that none of that shall be allowed once a power of attorney has been given to a tool (obviously unless the website access is paid, then it's perfectly fine to charge for API access but then it should be offered on something like FRAND terms). Think of Chrome extensions detected and blocked by some websites, this would not be allowed under my proposal.

Re: Web Scraping Is Vital to Democracy

#102

Post by smarx007 »

kordlessagain wrote: post #80
smarx007 wrote: post #52

Okay, now is a good time to put my long-time idea on paper: we need a "cloud" power of attorney (not using the term digital so that people don't confuse it for a regular one just signed digitally). Key idea is this: access to internet and its services is essential (and at the web scale, so is automation), but bot abuse is real. It should be possible for any company to ban bots but then for a person with a account on…

Building a distributed crowd sourced screen crawler.

I am proposing to have laws that respect our rights, not to secure rights by circumventing and disrespecting the law.

Re: Web Scraping Is Vital to Democracy

#103

Post by dehrmann »

Wowfunhappy wrote: post #62
mnmkng wrote: post #20

I think of web scraping as nothing less than automation of human work. There really is no hacking or unauthorized access involved. It's either me, using a browser, to read something publicly accessible on the internet, or it's my computer, which I've programmed to read those things for me, so I can focus on creative work or spend time with friends and family. This applies not only to personal life, but businesses as…

> I think of web scraping as nothing less than automation of human work. I agree that scraping should be allowed. We wouldn't have Google otherwise. However, there's something to be said for the fact that some activities are okay at a small scale but become problematic at a large scale—particularly the scale which becomes possible when a task is automated. For instance, I recently bought an expensive camera and have…

The case I thought of is how public records are great, but the way Spokeo and friends scrape them, post them, and sell them isn't.

Re: Web Scraping Is Vital to Democracy

#104

Post by rdbell »

kordlessagain wrote: post #80
smarx007 wrote: post #52

Okay, now is a good time to put my long-time idea on paper: we need a "cloud" power of attorney (not using the term digital so that people don't confuse it for a regular one just signed digitally). Key idea is this: access to internet and its services is essential (and at the web scale, so is automation), but bot abuse is real. It should be possible for any company to ban bots but then for a person with a account on…

Building a distributed crowd sourced screen crawler.

https://packetstream.io is pretty close

Re: Web Scraping Is Vital to Democracy

#105

Post by IfOnlyYouKnew »

hhjj wrote: post #39
IfOnlyYouKnew wrote: post #11

Earlier quoted context omitted.

You just state that "there is no middle ground", then go on describing instances of what arguably is "middle ground" as "absurd hacks". What's it gonna be? There's a POV that, on the internet, "anything goes", i. e. whatever you can do, you're allowed to do. Then, there's a perspective that works a lot like the offline world, where any clear communication that a reasonable person would understand as denying them acce…

The offline world is the world you are by default, you can't choose to put your house elsewhere. On contrary, when you put information on internet you choose to do it there (at least for website owners...). If you want to put restrictions on the usage of your data, make people sign a contract before accessing it. Also a contract should not have "the public" as a party or a subset of, you should be able to identify th…

A contract doesn't need to be signed. All it takes is a "meeting of the minds". Ever bought a coffee from a street vendor just by pointing at something / handing them some cash or similar? That's a valid contract.

As long as people keep noticing how stupid Ayn Rand is before they come of voting age, we do have some protection against surprises: you can't just sign over your house or your first-born by clicking on a cookie banner. But I'm pretty sure Facebook could make you type "I won't scrape Facebook" into a box and it'd be (civil-law) binding.

Re: Web Scraping Is Vital to Democracy

#106

Post by Wowfunhappy »

dehrmann wrote: post #103
Wowfunhappy wrote: post #62

Earlier quoted context omitted.

> I think of web scraping as nothing less than automation of human work. I agree that scraping should be allowed. We wouldn't have Google otherwise. However, there's something to be said for the fact that some activities are okay at a small scale but become problematic at a large scale—particularly the scale which becomes possible when a task is automated. For instance, I recently bought an expensive camera and have…

The case I thought of is how public records are great, but the way Spokeo and friends scrape them, post them, and sell them isn't.

Right! Ensuring a process has the right amount of friction can actually be super important, to ensure it's available but isn't used wantonly.

Re: Web Scraping Is Vital to Democracy

#107

Post by LunaSea »

krageon wrote: post #77
LunaSea wrote: post #72

Earlier quoted context omitted.

It absolutely is your responsibility if you are consuming the content. Consuming the content means agreeing to the premise that the content is paid for using ads. If you disagree with said premise you may happily browse another website. Malware is already illegal to install and you are free to sue the website for damages. I don't like ads either but trying to justify that their should be a choice of browsing a websit…

> Consuming the content means agreeing to the premise that the content is paid for using ads. It is in fact not this way, because the content arrives with or without the ads. In the EU EULAs (the dystopian construct that you'd expect to enforce that bit of lunacy) that purport to apply to content you have already accessed are not legally valid. Leaving the legal interpretation aside, me doing one thing doesn't mean c…

> It is in fact not this way, because the content arrives with or without the ads

But that's simply a technical implementation detail.

If the articles you read had first party ads or videos with embedded ads in them, that choice wouldn't exist like it already doesn't when you watch TV.

> Leaving the legal interpretation aside, me doing one thing doesn't mean consent for something else. Believing otherwise is both unethical and amoral, stances I don't hugely feel like interacting with.

Yes and you stealing content (consuming without paying for it) is somehow moral?

It's easy to find all kinds of free content, should it be music, movies or series but lets not kid ourselves in thinking that it is some great human right to have access to things we didn't pay for.

Re: Web Scraping Is Vital to Democracy

#109

Post by amelius »

bobobob420 wrote: post #76
Wowfunhappy wrote: post #62

Earlier quoted context omitted.

> I think of web scraping as nothing less than automation of human work. I agree that scraping should be allowed. We wouldn't have Google otherwise. However, there's something to be said for the fact that some activities are okay at a small scale but become problematic at a large scale—particularly the scale which becomes possible when a task is automated. For instance, I recently bought an expensive camera and have…

Can the case be made that recording in public is a right (as if always should be) but trying to track where everyone is at every point of time is stalking at a mass scale which should be illegal as stalking on a one to one scale is illegal? For reference this is what one site (findlaw) has given as what constitutes the crime of stalking: The crime of stalking can be simply described as the unwanted pursuit of another…

> The crime of stalking can be simply described as the unwanted pursuit of another person. Examples of this type of behavior includes following a person, appearing at a person's home or place of business, making harassing phone calls, leaving written messages or objects, or vandalizing a person's property.

Except for the last point, that sounds like ads :)

Post reply on HN