71–80 of 303 posts
Earlier quoted context omitted.
That's a load of controversial hogwash. If you have permission to view the page, then you have permission to scrape the page. Anyone can make a LinkedIn account, ergo anyone can scrape any public profile.
People have gone to jail for it. https://www.wired.com/2010/11/wiseguys-plead-guilty/
Earlier quoted context omitted.
At the risk of bringing out the "you wouldn't download a handbag" brigade - you could make the argument that just because I leave my bike unlocked on the street doesn't give you a right to ride it off. Even if you go for "forgiveness rather than permission", if you ride it a second time after I've told you I don't want you riding it, then you're in the wrong. Obviously, there are philosophical arguments to be made ab…
The bike example is a very poor analogy - the data isn't removed from LinkedIn, it's merely copied. It doesn't matter how many times the site is scraped, the data is unchanged and still available. A better analogy would be me taking a photo of the bike while walking past. It shouldn't matter how many times you tell me to stop, if it's on public property you can't really stop me from taking the photo. If you have an i…
Taking a photo of the bike is also a poor analogy. Scrapers don't take one photo, they take photos of all the bikes. And scrapers don't keep the photos for themselves, they sell them for a profit. Also, the original bike isn't parked, it's placed in a gallery (probably with an admission fee? I don't know the business model of linkedin).
I'm following this story with a lot of interest. I've done (and still do!) a lot of data crawling/scraping. In the past I've worked on so-called "alternative data" collection and analysis for financial forecasting. Without going into too much detail, a lot of hedge funds have teams constantly searching for kernels of data that can contribute some kind of signal for market movements. This data can come in the form of…
> Outside of finance I am familiar with many tech companies which (openly or otherwise), kickstarted what are now widely known enterprises through cleverly crawling or scraping massive amounts of data. Google comes to my mind. I'm only partly joking, as it seems to me that the line between search engine web crawling and other forms of web scraping is very thin.
A few weeks ago I was looking up public profiles on LinkedIn and I noticed what I interpreted to be some network-side fingerprinting of some kind. The first couple profiles came up, but from that point I was only served a sign up page. It didn't matter if I changed browsers or spawned new incognito sessions.
possibly just filtering based on your IP?
Not sure how to feel about this. While in theory, scraping data is a shady practice - companies like LinkedIn leave the door open for it.
How is scraping data shady? Is it also shady to walk into a store and look at things without buying anything?
If you post it on a public network, it is defacto, public. If you don't want it scraped, take it down, or put it behind a login. If the user provides the login to a scraper, then the scraper has permission.
Earlier quoted context omitted.
How is scraping data shady? Is it also shady to walk into a store and look at things without buying anything?
>Is it also shady to walk into a store and look at things without buying anything? Not making a claim about scraping, but it's maybe more apt to compare to walking into a store and writing a list of everything for sale their costs.
I'm following this story with a lot of interest. I've done (and still do!) a lot of data crawling/scraping. In the past I've worked on so-called "alternative data" collection and analysis for financial forecasting. Without going into too much detail, a lot of hedge funds have teams constantly searching for kernels of data that can contribute some kind of signal for market movements. This data can come in the form of…
I find scraping fascinating. Are there "societies of scrapers"? Inside, are certain sites more worthwhile - and which ones (eg reddit, eBay, trade union websites, whatever) How about scraper brokers? Do they exist? Are there scammy scrapers? Make up BS and sell as scraped data? How big is this?
Not that I'm aware of, no.
> Inside, are certain sites more worthwhile - and which ones (eg reddit, eBay, trade union websites, whatever)
Yes, absolutely. For many purposes websites that sell their own data are less useful (less signal exclusivity). Specific sources of data will be much more valuable depending on what the data is about.
> How about scraper brokers? Do they exist?
Yes. You're not getting access without an NDA in addition to paying quite a lot.
> Are there scammy scrapers? Make up BS and sell as scraped data?
That depends on how easy it is to verify the data. For most of what you'd term "alternative data" you'll know if it's real in 2 - 12 weeks, and it's not sustainable to sell crap.
But a lot of parties scrape dodgy financial timeseries data (ticks and quotes on equities or options) and sell it, priced as though it were tick data when it's barely accurately OHLC. They mostly sell this sort of data to amateurs who don't realize tick data is expensive for a reason.
> How big is this?
Very big. Most hedge funds ingest a lot of data whether they curate it internally or source it from elsewhere.
Earlier quoted context omitted.
>Is it also shady to walk into a store and look at things without buying anything? Not making a claim about scraping, but it's maybe more apt to compare to walking into a store and writing a list of everything for sale their costs.
I don't see a problem with that. Market transparency is a good thing, as it allows the market to be efficient.