This makes me think of Aaron Swartz. If he had the strength to keep from harming himself, his whole sentence would have likely been vacated. This is a reminder to those of us with suicidal thoughts: things can always get better.
Congrats! Web scraping is legal! (US precedent)
141–150 of 409 posts
Re: Congrats! Web scraping is legal! (US precedent)
#142The EFF has a writeup on this decision from September that I found more helpful than the linked article: https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l...
Re: Congrats! Web scraping is legal! (US precedent)
#143The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.
> People want their data to be public People don't want their data to be public. People want other people's data to be public. One's own data everyone thinks should be private and tightly controlled. This applies to people and businesses equally.
In this case, LinkedIn users kind of do want their “public profiles” to be public. They’re online CVs; by definition, if you make one, your goal is to get it into the hands of anyone who asks for it!
LinkedIn, likewise, has built its business model on an implicit contract with its users that it’s going to show their CV to anyone who asks for it.
I think LinkedIn users would be surprised that LinkedIn doesn’t let bots read (scrape) their public CV. A CV (individually, rather than in aggregate) is ultimately useful for only one thing: marketing the CV’s author’s skills. Why wouldn’t I want my marketable skills scraped into some private “talent matchmaking” agency’s databases, such that someone could find me—and hire me—when I show up as a result of some fancy OLAP query they paid that agency to run on their scraped data? It’s more roundabout than them just finding my CV on LinkedIn, but I’m still glad they found it!
Re: Congrats! Web scraping is legal! (US precedent)
#144German copyright has the concept of a "Datenbankwerk" (since the 90s). E.g. the telephone book contains lots of boring facts that are each in themselves not copyrightable. However the collection in itself is copyrightable, as it required substantial effort to create. It seems odd that US copyright law wouldn't have a similar provision, or that it doesn't apply here?
There is a concept in US copyright known as "thin" copyright. Collections of uncopyrightable information can in fact be copyrighted, but the arrangement of the information must have some spark or minimal creative energy. Mere "sweat of the brow" is insufficient to confer copyrightability on the work.
Such "thin" copyright tends to mean that there is a strong presumption against infringement. You generally need to demonstrate that the work has been copied virtually in its entirety to find infringement; partial borrowing is insufficient.
Re: Congrats! Web scraping is legal! (US precedent)
#145So, Google blocks me from automating my rank checking. Will this apply to google now?
Re: Congrats! Web scraping is legal! (US precedent)
#146"HiQ only takes information from public LinkedIn profiles. By definition, any member of the public has the right to access this information. Most importantly, the appeals court also upheld a lower court ruling that prohibits LinkedIn from interfering with hiQ’s web scraping of its site." Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawli…
[deleted]
Re: Congrats! Web scraping is legal! (US precedent)
#147Earlier quoted context omitted.
Wouldn't the solution be to offer a streamlined download (maybe even as a torrent if you're worried about bandwidth) of all the data then?
I work on a fully open data repository. The website has the API linked in 3 places, so when I find inappropriate scraping I block it with "HTTP 420 ... see or contact ". Some people probably switch to using the API, but no-one has ever contacted us. They either give up, or run their scraper on a different computer -- I've seen the same scraper move between university computers, departments, then (in the evening) to a…
I really don't understand why anyone would bother writing and using a web scraper when an API exists. Does the API not provide all the same data/functions as the website? Scrapers are a big PITA compared to just using an API: they're much harder to write to be reliable, and they can break at any time, whenever the site makes even the smallest change. APIs avoid all that mess, and make performance far better too (on both sides), since you're only downloading the data you want, not a ton of Javascript and HTML that you don't.
Re: Congrats! Web scraping is legal! (US precedent)
#148Anyone know the laws around scraping news article meta-data(image, headline, publisher, url) directly from a publishers website? As long as I'm not scraping the full article, could I have my users share the scraped meta-data between each other?
Re: Congrats! Web scraping is legal! (US precedent)
#149"HiQ only takes information from public LinkedIn profiles. By definition, any member of the public has the right to access this information. Most importantly, the appeals court also upheld a lower court ruling that prohibits LinkedIn from interfering with hiQ’s web scraping of its site." Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawli…
https://en.wikipedia.org/wiki/Tortious_interference This would mostly mean that you cannot start interfering with webscraping you previously allowed merely because you learned that they're making money with the scraped data.
It seems absurd if the 'interference' only directly affects their own property. Like, if my neighbors start monetizing livestreaming my backyard, suddenly I can't put up a fence? Except worse because in actuality, this third-party contract is costing them money through server load and bandwidth.
Re: Congrats! Web scraping is legal! (US precedent)
#150"HiQ only takes information from public LinkedIn profiles. By definition, any member of the public has the right to access this information. Most importantly, the appeals court also upheld a lower court ruling that prohibits LinkedIn from interfering with hiQ’s web scraping of its site." Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawli…
I wonder if it has anything to do with the fact that the data is actually owned by LinkedIn users, and they expressed that they want their data to be publicly available?
Unlikely. The license to LinkedIn retains ownership, but the user's retention of information ownership doesn't compel LinkedIn to affirmatively do things with that data (i.e. LinkedIn isn't forced to vend the data to a given consumer if the user says so).
The license further goes on to clarify that LinkedIn will vend public data to search engines, but the definition of "search engine" is almost certainly assumed (by LinkedIn, at least) to be up to them.