Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A raw dump of companies from all over the world by LinkedIn handle

blog.bigpicture.io

Post reply on HN
Print view View original

41–50 of 108 posts

Re: A raw dump of companies from all over the world by LinkedIn handle

#41

Post by prepend »

babblingfish wrote: post #27

It's funny how OP does not address where this data comes from even though it's obviously from LinkedIn. I see many people in the comments asking questions so I will add my two cents as someone who is currently employed by LinkedIn and has an interest in web scraping. This dataset was taken from scraping the company pages from LinkedIn. A company has to pay to have this page, so this certainly does not include all com…

> In any case, web scraping is a sort of gray area of the law.

I don’t think it’s grey. It seems to be legal as the data are made freely available and the only grey part is that companies don’t want this to happen and would rather charge and not have people scrape.

Re: A raw dump of companies from all over the world by LinkedIn handle

#43

Post by r3trohack3r »

This is awesome and in the ballpark of something I'm working on right now.

I currently have a list of developer handles, their associated aliases, and their associated email addresses - trying to map that set to employment history.

Do folks know of any good data sets for this?

Re: A raw dump of companies from all over the world by LinkedIn handle

#44

Post by GirishSharma643 »

simonw wrote: post #24

It's a 2.64GB CSV file with the following columns: handle type name website founded industry specialties size city state country_code 15,263,246 rows. I think the main listing for Google is this one (as an example): 10361050:company/google,Public Company,Google,goo.gle,,Software Development,"search, ads, mobile, android, online video, apps, machine learning, virtual reality, cloud, hardware, artificial intelligence,…

I don't have any work email. Can anyone please share some other source to download?

Re: A raw dump of companies from all over the world by LinkedIn handle

#45

Post by mpeg »

Murrawhip wrote: post #32

On your home page you list Microsoft as being one of your clients. I'm pretty impressed that you managed to sell them what appears to be (mostly) their own data.

We went through this in the ad world before GDPR existed. Companies were selling datasets for advertising and, I remember one of my clients (one of the largest ad agencies in the world) telling me they had over 100 sources of data they augmented their publisher data with but they tested and saw there was only about 20% uniqueness on average between sources.

They were buying the same data again and again, recycled and repackaged.

Re: A raw dump of companies from all over the world by LinkedIn handle

#47

Post by paxys »

You use "open source" multiple times in the post, HN title, HN comments, but:

1. The source code for the project isn't shared anywhere.

2. The data isn't shared under any standard open source license.

3. The terms of your site explicitly prohibit commercial use of this data.

So what exactly makes this "open source in the broadest sense"?

Re: A raw dump of companies from all over the world by LinkedIn handle

#48

Post by JohnFen »

prepend wrote: post #41
babblingfish wrote: post #27

It's funny how OP does not address where this data comes from even though it's obviously from LinkedIn. I see many people in the comments asking questions so I will add my two cents as someone who is currently employed by LinkedIn and has an interest in web scraping. This dataset was taken from scraping the company pages from LinkedIn. A company has to pay to have this page, so this certainly does not include all com…

> In any case, web scraping is a sort of gray area of the law. I don’t think it’s grey. It seems to be legal as the data are made freely available and the only grey part is that companies don’t want this to happen and would rather charge and not have people scrape.

It is a gray area (in the US) in the sense that there is no clear consensus about it in the courts. There have been court rulings in both directions.

Re: A raw dump of companies from all over the world by LinkedIn handle

#49

Post by tacker2000 »

simonw wrote: post #35
simonw wrote: post #24

It's a 2.64GB CSV file with the following columns: handle type name website founded industry specialties size city state country_code 15,263,246 rows. I think the main listing for Google is this one (as an example): 10361050:company/google,Public Company,Google,goo.gle,,Software Development,"search, ads, mobile, android, online video, apps, machine learning, virtual reality, cloud, hardware, artificial intelligence,…

Ran https://sqlite-utils.datasette.io/en/stable/cli.html#cli-ana... to figure out the most common values in each column: company.type: (2/22) Total rows: 15263246 Null rows: 0 Blank rows: 5032965 Distinct values: 189 Most common: 5128934: Privately Held 5032965: 1202598: Self-Owned 1007806: Partnership 952111: Public Company 763992: Nonprofit 749117: Self-Employed 319885: Educational 84829: Government Agency 2423: De…

Very interesting, thanks!

Re: A raw dump of companies from all over the world by LinkedIn handle

#50

Post by hallqv »

JohnFen wrote: post #48
prepend wrote: post #41

Earlier quoted context omitted.

> In any case, web scraping is a sort of gray area of the law. I don’t think it’s grey. It seems to be legal as the data are made freely available and the only grey part is that companies don’t want this to happen and would rather charge and not have people scrape.

It is a gray area (in the US) in the sense that there is no clear consensus about it in the courts. There have been court rulings in both directions.

What rulings are you referring to? Been reaffirmed that scraping LinkedIn is legal multiple times, even by Supreme Court. https://techcrunch.com/2022/04/18/web-scraping-legal-court/
Post reply on HN