Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A raw dump of companies from all over the world by LinkedIn handle

blog.bigpicture.io

Post reply on HN
Print view View original

51–60 of 108 posts

Re: A raw dump of companies from all over the world by LinkedIn handle

#51

Post by mcenedella »

babblingfish wrote: post #27

It's funny how OP does not address where this data comes from even though it's obviously from LinkedIn. I see many people in the comments asking questions so I will add my two cents as someone who is currently employed by LinkedIn and has an interest in web scraping. This dataset was taken from scraping the company pages from LinkedIn. A company has to pay to have this page, so this certainly does not include all com…

LinkedIn lost the HiQ case. Your GC claiming it’s a win doesn’t change that you lost, repeatedly, in multiple courts of law: https://en.m.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn

Re: A raw dump of companies from all over the world by LinkedIn handle

#52

Post by SnowHill9902 »

JohnFen wrote: post #48
prepend wrote: post #41

Earlier quoted context omitted.

> In any case, web scraping is a sort of gray area of the law. I don’t think it’s grey. It seems to be legal as the data are made freely available and the only grey part is that companies don’t want this to happen and would rather charge and not have people scrape.

It is a gray area (in the US) in the sense that there is no clear consensus about it in the courts. There have been court rulings in both directions.

So it’s not gray, it’s melange.

Re: A raw dump of companies from all over the world by LinkedIn handle

#54

Post by mfrye0 »

simonw wrote: post #37
mfrye0 wrote: post #30

Earlier quoted context omitted.

How did you compute this? I just did another check to verify (wc -l) and it's coming to 15,980,531.

I used wc -l at first, but I've just imported into SQLite and the count(*) is 15,263,246 - updated my previous comment (which had said 15,263,251). I downloaded the CSV and ran: sqlite-utils insert companies.db company companies-dataset-2023-02-ckgENv.csv --csv sqlite-utils enable-fts companies.db company name specialties sqlite-utils analyze-tables --save companies.db This lets me run searches against the name and s…

Ok. I'm not sure how this happened, but I think the dataset was somehow mislabeled. It appears that this dataset is the Q1 version, not the latest Q2. Can you please try re-downloading it?

We're probably going to have to make an public announcement about this...

Re: A raw dump of companies from all over the world by LinkedIn handle

#55

Post by foolswisdom »

mcenedella wrote: post #51
babblingfish wrote: post #27

It's funny how OP does not address where this data comes from even though it's obviously from LinkedIn. I see many people in the comments asking questions so I will add my two cents as someone who is currently employed by LinkedIn and has an interest in web scraping. This dataset was taken from scraping the company pages from LinkedIn. A company has to pay to have this page, so this certainly does not include all com…

LinkedIn lost the HiQ case. Your GC claiming it’s a win doesn’t change that you lost, repeatedly, in multiple courts of law: https://en.m.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn

According to that wiki page, the courts eventually did rule that HiQ violated LinkedIn's terms of service, and a settlement agreement was reached.

Re: A raw dump of companies from all over the world by LinkedIn handle

#56

Post by dheera »

babblingfish wrote: post #27

It's funny how OP does not address where this data comes from even though it's obviously from LinkedIn. I see many people in the comments asking questions so I will add my two cents as someone who is currently employed by LinkedIn and has an interest in web scraping. This dataset was taken from scraping the company pages from LinkedIn. A company has to pay to have this page, so this certainly does not include all com…

> In any case, web scraping is a sort of gray area of the law.

I feel like it's also only a matter of time before we have decentralized web scrapers that can consume data and assemble datasets and store the results in a decentralized fashion in a way that is completely non-enforceable. LinkedIn data would be aggregated in all sorts of interesting ways, and there would be literally no name you could sue, no address you could serve papers to, it would be a hundred thousand bots in 100 countries that collectively assembled the dataset and put it on some IPFS or torrent.

If data is visible with human eyes, it's visible by decentralized bots, the only missing piece is the technical and financial complexity of the current state of decentralized compute.

Re: A raw dump of companies from all over the world by LinkedIn handle

#57

Post by hallqv »

SnowHill9902 wrote: post #52
JohnFen wrote: post #48

Earlier quoted context omitted.

It is a gray area (in the US) in the sense that there is no clear consensus about it in the courts. There have been court rulings in both directions.

So it’s not gray, it’s melange.

Multiple court reaffirms in favor of scraping, including by the SC, seems more like verdant to me..

Re: A raw dump of companies from all over the world by LinkedIn handle

#59

Post by henryfjordan »

mcenedella wrote: post #51
babblingfish wrote: post #27

It's funny how OP does not address where this data comes from even though it's obviously from LinkedIn. I see many people in the comments asking questions so I will add my two cents as someone who is currently employed by LinkedIn and has an interest in web scraping. This dataset was taken from scraping the company pages from LinkedIn. A company has to pay to have this page, so this certainly does not include all com…

LinkedIn lost the HiQ case. Your GC claiming it’s a win doesn’t change that you lost, repeatedly, in multiple courts of law: https://en.m.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn

Linkedin lost on the CFAA claims, there were others that they did not lose.

Re: A raw dump of companies from all over the world by LinkedIn handle

#60

Post by nxqs »

paxys wrote: post #47

You use "open source" multiple times in the post, HN title, HN comments, but: 1. The source code for the project isn't shared anywhere. 2. The data isn't shared under any standard open source license. 3. The terms of your site explicitly prohibit commercial use of this data. So what exactly makes this "open source in the broadest sense"?

"It's 'open source'"... https://youtu.be/dTRKCXC0JFg?t=6
Post reply on HN