Live data from Hacker News

Congrats! Web scraping is legal! (US precedent)

parsers.me

391–400 of 409 posts

Re: Congrats! Web scraping is legal! (US precedent)

#391
post #44

Earlier quoted context omitted.

> You don't just get to drop someone else's work product into your algorithm without their permission. Why not?

Because copyright law exists.

I don't think using data as input to an algorithm necessarily breaks copyright law.

I can read a book to post my impression on it somewhere right? I can read it and say "it was beautiful" on twitter.

I can then automate my "taste meter" through machine learning, it reads a given book character by character, and spits out what I'd think of it if I actually read it. Then posts it on twitter, says "it was beautiful".

Did I break copyright law? I don't think so.

Re: Congrats! Web scraping is legal! (US precedent)

#392
post #385

Earlier quoted context omitted.

Scrapers can just create a user account then. They still seem to be covered by this injunction

Legal: It will be hard to argue in court that you have right to create fake accounts. That is definitely against TOS, and at that point Linkedin can sue the company instead, so I believe it would scraper would not be successful at court. Technical: This is actually what you want, as limiting scraping per user is easier than per IP, since IP can be shared by users, so your rate limits have to be higher for IP address…

a real account could scrape tons of stuff given time

Re: Congrats! Web scraping is legal! (US precedent)

#393

Earlier quoted context omitted.

So, apply DOS/DDOS mitigations? You need to have those anyway, so you might as well use them for this too.

Yeah, but where do you draw the line between 'oh sorry we dropped your request because of rate limiting' (or whatever mitigation strategy) and 'oh, we dropped your request because u scraping us bro' legally? IANAL, but this lawsuit seems to indicated that putting barriers in front of scraping attempts is a no-no.

"You issued more than $X requests per $Y seconds. We don't care whether you're scraping or not; try again later." for any values of X and Y.

Re: Congrats! Web scraping is legal! (US precedent)

#394
IANAL, but IMO LinkedIn goofed by using the CFAA. eBay and BiddersEdge had a similar case, and eBay won, preventing scraping of its website using trespass:

https://casetext.com/case/ebay-v-bidders-edge

I can see both sides of this, and am not sure which is better. If a site is not allowed to block traffic, does that mean blocking a DDOS attack is illegal? What if the company doing the scraping is so aggressive about it that they are impacting performance?

One fair way (to me) would be to figure out how many pages a typical user looked at per month, and if a company is scraping the site, they are limited to accessing it like one customer; they can't launch 100 threads to scarf down data as fast as possible. This probably wouldn't work for most scrapers' businesses, but it seems reasonable from the origin web site's POV.

Another possibility would be to require web sites that accept and then publicize user submissions to give users a choice about whether they want to make their information public. If they do, the hosting site has to include the information in a publicly-accessible feed, like a daily compressed download. This seems reasonable to me, because the hosting site is getting all kinds of information for free from users. It's not like it would take a lot of effort to do a streaming JSON dump of the stuff that changed every day, and if necessary, they could throttle the download rate, but not so low that the data for a day couldn't be downloaded in a day. Ie, a competitor might be a day behind, but no more. Of course, the competitor would have to publish a feed too, allowing other people to lunch off of them just like they are lunching off other sites.

With LinkedIn, all of the information they obtain and publish is submitted by users, so for LI to say they "own" this information makes no sense. At least with eBay, the current auction price is not not something the user submitted, assuming it has bids.

Re: Congrats! Web scraping is legal! (US precedent)

#395
post #174

Earlier quoted context omitted.

I'm interested in hearing your take on "organizational transparency". Like please push the concept / idea to its 'full' realization and tell me that picture, even if it implies a little bit of "sci-fi"¹. Digging this because I think that domain / paradigm will see unparalleled evolution in the next few decades. [1]: I mean, don't stop at current law / values / behaviors; like people from the 1940s wouldn't have dared…

I don't think that looking too far ahead is useful: this is just a matter of pragmatics. Revolutionary change in a peaceful society happens via a long sequence of small, incremental changes, and that's a good thing, because you get to see how each of the changes plays out. I think the best sci-fi persuades you that it's looking at the distant future when in fact it's only using the future as a foil to provide deep in…

Can't agree more with your first paragraph.

1, 2, 6, 11 should be no-brainers if people were educated IMHO — but this is 1920 relative to electricity or cars; still a long way to go before the mainsteam masses get it (which very much includes political figures). I would think 2030-2040 for the emergence of ethical consensus and concern (the kind that pervades political parties and social classes).

That is assuming the needle doesn't move too much farther in the authoritarian direction until then (the 20-year trend is really not looking that way currently).

3, 4, 5, 9, 10 are/would be met by strong opposition from interest groups, I'm sure you see that too. Everything I know about 3 tells me it's never going to happen with current parties / politicians. It's at least 1 generation away and I'm not sure the concept itself isn't utopia. 9 and 10 as well, I think it largely depends on the cultural paradigm (and this world's in 2020 is really not aligned with that, nor does it trend or even look that way). 4, 5 likewise, complex topics, lots and lots and lots of gatekeepers and lobbyists.

My take on these is they're very costly in terms of political capital; and they are largely debatable (politically, legally, philosophically, etc., you'll find passionate captains on both sides); thus there are 'better' (more consensual, with direct net positive effect) lower hanging fruits imho.

7 and 8 are hard problems, notably because of scale and the need for automation — it's part of a much bigger domain, automation of compliance and building "trustable" systems etc.; the kind that bridge or plane engineers must build, and probably software engineers too, but you know we're far from that if you read this forum.

I'd say 1 2 6 11 and 7 8 on the way to scale/automation already paint a whole different regime and degree of maturity for a 21st century State. I'd like to think we're now ~1 generation away from enactment of such norms.

Re: Congrats! Web scraping is legal! (US precedent)

#396
post #341

Earlier quoted context omitted.

> Your analogy doesn't hold. It does, in the US. You're likely making an inconsistent comparison. Property ownership has nothing to do with visual access. You cannot legally be barred from casually (involuntarily) perceiving something. It's reasonable to put up physical barriers to reduce what is casually perceived. It's a very good analogy.

However it doesn't hold - as your neighbor I can't bar you from putting up a fence because it'll intrude on my view of your property... granted people try to do that _all the time_ but I think it's commonly understood that putting up a fence for privacy is allowed. It's also not a great analogy for this case because another party is given continued easy access to view my backyard while the first party is denied - and…

> it's commonly understood that putting up a fence for privacy is allowed.

Try building that fence into the stratosphere. A regulatory body will prevent that.

> I have no inherent right to view your private life at least as much as any of your other neighbors.

That's a different analogy, not a violation of the first.

It's not necessary for every part of the analogy to hold, being an analogy.

Re: Congrats! Web scraping is legal! (US precedent)

#397
post #387

Earlier quoted context omitted.

So, should it have become public knowledge that Brendan Eich donated $1,000 to support Proposition 8?

I'm gonna need some context there. To begin with you can't just donate to support a bill. What you probably mean is that he donated to an organization or politician who supported Prop 8, so say that. The important detail is whether the funds were his personal funds, or whether they belonged to his company: i.e. was he acting on his own behalf, or on behalf of an organization? It's an inane question, though, because t…

So a person spends money to support an odious cause, and that person gets money by being associated with an organization. You hate the odious cause, so you want to avoid giving your money to support people who will then give money to promoting that odious cause. How does that work? Do you no longer have the right to not indirectly support things in your ideal system?

Re: Congrats! Web scraping is legal! (US precedent)

#398
post #397

Earlier quoted context omitted.

I'm gonna need some context there. To begin with you can't just donate to support a bill. What you probably mean is that he donated to an organization or politician who supported Prop 8, so say that. The important detail is whether the funds were his personal funds, or whether they belonged to his company: i.e. was he acting on his own behalf, or on behalf of an organization? It's an inane question, though, because t…

So a person spends money to support an odious cause, and that person gets money by being associated with an organization. You hate the odious cause, so you want to avoid giving your money to support people who will then give money to promoting that odious cause. How does that work? Do you no longer have the right to not indirectly support things in your ideal system?

Since you're just openly ignoring the post you're "responding to", I'll just copy-paste my response to what you have just said, with some minor changes:

> So a person spends money to support an odious cause, and that person gets money by being associated with an organization. You hate the odious cause, so you want to avoid giving your money to support people who will then give money to promoting that odious cause. How does that work? Do you no longer have the right to not indirectly support things in your ideal system?

Yes, but it would also protect people who donated to a virtuous cause. The fact that privacy allows people to secretly donate to political campaigns is the point. Part of the reason virtuous causes have had any success at all is that people donated to support them, at a time when donating to those virtuous causes was risking your job and social standing.

Human rights still apply to humans who support odious causes. If you are willing to give up human rights to fight bad people trying to support odious causes, then those rights won't be there to protect good people trying to support virtuous causes, either.

Re: Congrats! Web scraping is legal! (US precedent)

#400
post #243

Earlier quoted context omitted.

> Why can't I have terms on my website that say how you can use my information? Neither Creative Commons nor copyleft (nor copyright in general!) can assert anything about private use . IP rights are commercial rights; they affect sellers of your IP. They don’t affect end-consumers of your IP. Note that even the GPL can’t force someone to publish the source of their GPLed-library-containing program, if they never pub…

Right, sure, I hadn't really thought about how copyright isn't really enforceable against individuals. That's very interesting. However, I'm really not sure how this is relevant to your moral stance on commercial use of "public" personal information. Why do you believe it's reasonable to prevent unauthorized commercial exploitation of creative works, but not to prevent unauthorized commercial exploitation of personal…

I think you got the wrong idea if you were thinking I was saying copyright isn’t enforceable “against individuals.”

My example of copyleft was specifically about the thing the Affero GPL tries to avoid (to unknown success): the possibility of someone using GPLed libraries to set up a commercial web service. Because they never release the binary, but only have people interact with it over the Internet, there’s no derivative work being made available in the commercial domain. So copyright doesn’t apply. Even though you’re a company making money off GPLed libraries!

Post reply on HN