Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A raw dump of companies from all over the world by LinkedIn handle

blog.bigpicture.io

Post reply on HN
Print view View original

31–40 of 108 posts

Re: A raw dump of companies from all over the world by LinkedIn handle

#31

Post by sam_lowry_ »

yolo3000 wrote: post #25
wg0 wrote: post #11

What I am curious to know about is - what company buys from whom and the whole dependency graph to visualise how complex our modern economy is. But not sure that kind of information is in there.

That sort of data is normally not public

Individual countries know this, and even share rather successfully across unions.

EU exchanges VAT information which is exactly about who buys from whom... down to transactions of 120€ and more if I am not mistaken.

Commercial databases are more limited, but even those can trace ownership of companies to ultimate beneficial owners. Orbis from Bureau Van Dijk is probably the biggest of such databases.

Re: A raw dump of companies from all over the world by LinkedIn handle

#34

Post by rahimnathwani »

mfrye0 wrote: post #30
simonw wrote: post #24

It's a 2.64GB CSV file with the following columns: handle type name website founded industry specialties size city state country_code 15,263,246 rows. I think the main listing for Google is this one (as an example): 10361050:company/google,Public Company,Google,goo.gle,,Software Development,"search, ads, mobile, android, online video, apps, machine learning, virtual reality, cloud, hardware, artificial intelligence,…

How did you compute this? I just did another check to verify (wc -l) and it's coming to 15,980,531.

wc -l will count all newlines, even those that are escaped. Perhaps some company descriptions have newlines?

Re: A raw dump of companies from all over the world by LinkedIn handle

#35

Post by simonw »

simonw wrote: post #24

It's a 2.64GB CSV file with the following columns: handle type name website founded industry specialties size city state country_code 15,263,246 rows. I think the main listing for Google is this one (as an example): 10361050:company/google,Public Company,Google,goo.gle,,Software Development,"search, ads, mobile, android, online video, apps, machine learning, virtual reality, cloud, hardware, artificial intelligence,…

Ran https://sqlite-utils.datasette.io/en/stable/cli.html#cli-ana... to figure out the most common values in each column:

    company.type: (2/22)

      Total rows: 15263246
      Null rows: 0
      Blank rows: 5032965

      Distinct values: 189

      Most common:
        5128934: Privately Held
        5032965: 
        1202598: Self-Owned
        1007806: Partnership
        952111: Public Company
        763992: Nonprofit
        749117: Self-Employed
        319885: Educational
        84829: Government Agency
        2423: De financiación privada

    company.website: (4/22)

      Total rows: 15263246
      Null rows: 0
      Blank rows: 3043640

      Distinct values: 10675220

      Most common:
        3043640: 
        81214: facebook.com
        50174: instagram.com
        41050: business.site
        27134: linktr.ee
        24024: indiamart.com
        19683: wixsite.com
        16546: negocio.site
        15864: linkedin.com
        13201: yelp.com

    company.founded: (5/22)

      Total rows: 15263246
      Null rows: 0
      Blank rows: 7626823

      Distinct values: 1551

      Most common:
        7626823: 
        488964: 2020
        438343: 2017
        417847: 2018
        404720: 2016
        398489: 2019
        386049: 2015
        383686: 2021
        338120: 2014
        298768: 2013

    company.industry: (6/22)

      Total rows: 15263246
      Null rows: 0
      Blank rows: 1274455

      Distinct values: 2591

      Most common:
        1274455: 
        793016: IT Services and IT Consulting
        626267: Advertising Services
        623184: Construction
        424481: Real Estate
        417648: Business Consulting and Services
        416398: Software Development
        401791: Retail
        337914: Financial Services
        305029: Wellness and Fitness Services

    company.specialties: (7/22)

      Total rows: 15263246
      Null rows: 0
      Blank rows: 9681220

      Distinct values: 5226043

      Most common:
        9681220: 
        1678: Real Estate
        1353: Education
        557: Software Development
        537: real estate
        516: Recruitment
        456: Marketing
        420: Property Management
        396: Digital Marketing
        393: Hospitality

    company.size: (8/22)

      Total rows: 15263246
      Null rows: 0
      Blank rows: 2531526

      Distinct values: 226

      Most common:
        6317189: 2-10
        3459571: 11-50
        2531526: 
        1170922: 51-200
        730818: 1 employee
        417504: 201-500
        262702: 1
        149520: 501-1,000
        130678: 1,001-5,000
        43399: 10,001+

    company.city: (9/22)

      Total rows: 15263246
      Null rows: 0
      Blank rows: 3039470

      Distinct values: 393107

      Most common:
        3039470: 
        262475: London
        116915: Paris
        110220: New York
        96914: São Paulo
        72455: Los Angeles
        66837: Madrid
        65075: Toronto
        59321: New Delhi
        58151: Dubai

    company.state: (10/22)

      Total rows: 15263246
      Null rows: 0
      Blank rows: 4326168

      Distinct values: 55773

      Most common:
        4326168: 
        670232: England
        567647: California
        318576: Texas
        283576: New York
        276783: Florida
        215523: São Paulo
        172516: Maharashtra
        164761: Ontario
        161408: Île-de-France

    company.country_code: (11/22)

      Total rows: 15263246
      Null rows: 0
      Blank rows: 2858360

      Distinct values: 272

      Most common:
        3943846: US
        2858360: 
        1204699: GB
        816378: IN
        691296: FR
        638895: BR
        443326: DE
        401268: NL
        386207: ES
        373460: CA

    company_fts.specialties: (13/22)

      Total rows: 15263246
      Null rows: 0
      Blank rows: 9681220

      Distinct values: 5226043

      Most common:
        9681220: 
        1678: Real Estate
        1353: Education
        557: Software Development
        537: real estate
        516: Recruitment
        456: Marketing
        420: Property Management
        396: Digital Marketing
        393: Hospitality

Re: A raw dump of companies from all over the world by LinkedIn handle

#36

Post by justinclift »

mfrye0 wrote: post #26
mfrye0 wrote: post #2

Hey HN, we're thrilled to announce our latest project - the World's Largest Open Source Company Dataset. Our team has been working hard on this product for the past few months, and we're excited to finally share it with you all. We started off years ago trying to build a B2B app, but getting basic company data at scale was a huge barrier for us. This 15M+ record dataset attempts to solve that and has all the key comp…

So this blew up today. Reviewing all the comments now. 1. Yes, this is scraped from public sources. 2. Yes, this is free to use / is open source in the broadest sense. Apologies for the confusion on the lack of a license and no mention about this in our TOS. We probably should update our TOS to be clearer here. 3. This is a raw dump of companies from all over the world by LinkedIn handle. The handles are deduped, but…

> Yes, this is free to use ...

Including for commercial purposes?

Re: A raw dump of companies from all over the world by LinkedIn handle

#37

Post by simonw »

mfrye0 wrote: post #30
simonw wrote: post #24

It's a 2.64GB CSV file with the following columns: handle type name website founded industry specialties size city state country_code 15,263,246 rows. I think the main listing for Google is this one (as an example): 10361050:company/google,Public Company,Google,goo.gle,,Software Development,"search, ads, mobile, android, online video, apps, machine learning, virtual reality, cloud, hardware, artificial intelligence,…

How did you compute this? I just did another check to verify (wc -l) and it's coming to 15,980,531.

I used wc -l at first, but I've just imported into SQLite and the count(*) is 15,263,246 - updated my previous comment (which had said 15,263,251).

I downloaded the CSV and ran:

    sqlite-utils insert companies.db company companies-dataset-2023-02-ckgENv.csv --csv
    sqlite-utils enable-fts companies.db company name specialties
    sqlite-utils analyze-tables --save companies.db
This lets me run searches against the name and specialties columns, and gives me those aggregate stats too.

Re: A raw dump of companies from all over the world by LinkedIn handle

#38

Post by mfrye0 »

justinclift wrote: post #36
mfrye0 wrote: post #26

Earlier quoted context omitted.

So this blew up today. Reviewing all the comments now. 1. Yes, this is scraped from public sources. 2. Yes, this is free to use / is open source in the broadest sense. Apologies for the confusion on the lack of a license and no mention about this in our TOS. We probably should update our TOS to be clearer here. 3. This is a raw dump of companies from all over the world by LinkedIn handle. The handles are deduped, but…

> Yes, this is free to use ... Including for commercial purposes?

Yes.

Re: A raw dump of companies from all over the world by LinkedIn handle

#39

Post by byyll »

wg0 wrote: post #11

What I am curious to know about is - what company buys from whom and the whole dependency graph to visualise how complex our modern economy is. But not sure that kind of information is in there.

Closest I can think of: https://www.importyeti.com
Post reply on HN