Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A raw dump of companies from all over the world by LinkedIn handle

blog.bigpicture.io

Post reply on HN
Print view View original

21–30 of 108 posts

Re: A raw dump of companies from all over the world by LinkedIn handle

#21

Post by 1024core »

tomalaci wrote: post #13
1024core wrote: post #7

What would be really interesting is to turn this into a graph based on, say, past experience of CEOs/big dealings with each other/etc.

The moment those graph connections will show negative view of some of those CEOs they will start screaming about privacy issues and/or sue the graph creator. With scraping LinkedIn alone it should already be possible to create such graphs. Not sure why I haven't seen any. Either scraping LinkedIn isn't as straightforward or such grap-creating attempts have been shut down due to potential for defaming someone.

I would say that scraping LinkedIn is not easy.

Re: A raw dump of companies from all over the world by LinkedIn handle

#22

Post by cdkmoose »

data_maan wrote: post #4

I looked at the attributes they say the dataset has. Not too many (e.g. number of people, location). The really interesting ones, like who is doing business with whom, are missing.

From the deeper dataset documentation, it looks like this free dataset is a field subset of their paid product.

Re: A raw dump of companies from all over the world by LinkedIn handle

#23

Post by conzept »

jsty wrote: post #6
mfrye0 wrote: post #2

Hey HN, we're thrilled to announce our latest project - the World's Largest Open Source Company Dataset. Our team has been working hard on this product for the past few months, and we're excited to finally share it with you all. We started off years ago trying to build a B2B app, but getting basic company data at scale was a huge barrier for us. This 15M+ record dataset attempts to solve that and has all the key comp…

What's the license the dataset is released under? I poked around the documentation a bit but couldn't find it - apologies if it's in there!

The data is not licensed as opensource (only non-commercial use): "The Service and its entire contents, features and functionality (including but not limited to all information, software, text, displays, images, video and audio, and the design, selection and arrangement thereof), are owned by the Company, its licensors or other providers of such material and are protected by United States and international copyright, trademark, patent, trade secret and other intellectual property or proprietary rights laws. You are permitted to use the Service for Your personal, non-commercial use, or legitimate business purposes related to Your role as a customer of BigPicture.io." - https://bigpicture.io/terms

Re: A raw dump of companies from all over the world by LinkedIn handle

#24

Post by simonw »

It's a 2.64GB CSV file with the following columns:

    handle
    type
    name
    website
    founded
    industry
    specialties
    size
    city
    state
    country_code
15,263,246 rows.

I think the main listing for Google is this one (as an example):

10361050:company/google,Public Company,Google,goo.gle,,Software Development,"search, ads, mobile, android, online video, apps, machine learning, virtual reality, cloud, hardware, artificial intelligence, youtube, and software","10,001+",Mountain View,California,US

Re: A raw dump of companies from all over the world by LinkedIn handle

#25

Post by yolo3000 »

wg0 wrote: post #11

What I am curious to know about is - what company buys from whom and the whole dependency graph to visualise how complex our modern economy is. But not sure that kind of information is in there.

That sort of data is normally not public

Re: A raw dump of companies from all over the world by LinkedIn handle

#26

Post by mfrye0 »

mfrye0 wrote: post #2

Hey HN, we're thrilled to announce our latest project - the World's Largest Open Source Company Dataset. Our team has been working hard on this product for the past few months, and we're excited to finally share it with you all. We started off years ago trying to build a B2B app, but getting basic company data at scale was a huge barrier for us. This 15M+ record dataset attempts to solve that and has all the key comp…

So this blew up today. Reviewing all the comments now.

1. Yes, this is scraped from public sources. 2. Yes, this is free to use / is open source in the broadest sense. Apologies for the confusion on the lack of a license and no mention about this in our TOS. We probably should update our TOS to be clearer here. 3. This is a raw dump of companies from all over the world by LinkedIn handle. The handles are deduped, but the website is not.

Re: A raw dump of companies from all over the world by LinkedIn handle

#27

Post by babblingfish »

It's funny how OP does not address where this data comes from even though it's obviously from LinkedIn. I see many people in the comments asking questions so I will add my two cents as someone who is currently employed by LinkedIn and has an interest in web scraping.

This dataset was taken from scraping the company pages from LinkedIn. A company has to pay to have this page, so this certainly does not include all companies. If you have a premium account your search is not rate limited so you can iteratively scrape anything you want even though it's technically a violation of the terms of service.

There are many companies that sell data scraped from LinkedIn as a product. LinkedIn won a court case against hiQ Labs for scraping member data and other things[1]. I am not trying to compare this court case to the OP's website, just something worth mentioning.

In any case, web scraping is a sort of gray area of the law. In my opinion, this data set does not contain member data and is not being monetized so it feels kosher to me.

(Opinions expressed are solely my own and do not express the views or opinions of my employer.)

[1] https://www2.staffingindustry.com/Editorial/IT-Staffing-Repo...

Re: A raw dump of companies from all over the world by LinkedIn handle

#28

Post by yolo3000 »

rickette wrote: post #3
mfrye0 wrote: post #2

Hey HN, we're thrilled to announce our latest project - the World's Largest Open Source Company Dataset. Our team has been working hard on this product for the past few months, and we're excited to finally share it with you all. We started off years ago trying to build a B2B app, but getting basic company data at scale was a huge barrier for us. This 15M+ record dataset attempts to solve that and has all the key comp…

Out of curiosity: Would you be willing to share how you acquired this data? Website scraping or other means?

Most likely from scraping (crunchbase, yahoo,etc), unless they bought it from somewhere. In most countries you can get it from the chamber of commerce. Dun and Bradstreet and other similar companies. Some of these data aggregators will have partnership with other companies, and you can also (illegally) scrape it from there.

Re: A raw dump of companies from all over the world by LinkedIn handle

#29

Post by simonw »

babblingfish wrote: post #27

It's funny how OP does not address where this data comes from even though it's obviously from LinkedIn. I see many people in the comments asking questions so I will add my two cents as someone who is currently employed by LinkedIn and has an interest in web scraping. This dataset was taken from scraping the company pages from LinkedIn. A company has to pay to have this page, so this certainly does not include all com…

> A company has to pay to have this page, so this certainly does not include all companies.

I found a listing in there for my old startup derived from the LinkedIn page, and we never paid for that LinkedIn listing.

Re: A raw dump of companies from all over the world by LinkedIn handle

#30

Post by mfrye0 »

simonw wrote: post #24

It's a 2.64GB CSV file with the following columns: handle type name website founded industry specialties size city state country_code 15,263,246 rows. I think the main listing for Google is this one (as an example): 10361050:company/google,Public Company,Google,goo.gle,,Software Development,"search, ads, mobile, android, online video, apps, machine learning, virtual reality, cloud, hardware, artificial intelligence,…

How did you compute this? I just did another check to verify (wc -l) and it's coming to 15,980,531.
Post reply on HN