Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A raw dump of companies from all over the world by LinkedIn handle

blog.bigpicture.io

Post reply on HN
Print view View original

71–80 of 108 posts

Re: A raw dump of companies from all over the world by LinkedIn handle

#71

Post by jehb »

GirishSharma643 wrote: post #44
simonw wrote: post #24

It's a 2.64GB CSV file with the following columns: handle type name website founded industry specialties size city state country_code 15,263,246 rows. I think the main listing for Google is this one (as an example): 10361050:company/google,Public Company,Google,goo.gle,,Software Development,"search, ads, mobile, android, online video, apps, machine learning, virtual reality, cloud, hardware, artificial intelligence,…

I don't have any work email. Can anyone please share some other source to download?

Not an answer to your question, but I did find their justification for why they require a work email address intriguing.

https://blog.bigpicture.io/how-we-stopped-spam-signups/

I've been working through this issue a lot lately. There tend to be two camps, the "Make the experience good for the person trying to access the data, because good relationships are more important than the ability to contact someone" versus "Why should I give someone something for free when there is zero chance I can ever make a sale to that person?"

Re: A raw dump of companies from all over the world by LinkedIn handle

#72

Post by simonw »

mfrye0 wrote: post #54
simonw wrote: post #37

Earlier quoted context omitted.

I used wc -l at first, but I've just imported into SQLite and the count(*) is 15,263,246 - updated my previous comment (which had said 15,263,251). I downloaded the CSV and ran: sqlite-utils insert companies.db company companies-dataset-2023-02-ckgENv.csv --csv sqlite-utils enable-fts companies.db company name specialties sqlite-utils analyze-tables --save companies.db This lets me run searches against the name and s…

Ok. I'm not sure how this happened, but I think the dataset was somehow mislabeled. It appears that this dataset is the Q1 version, not the latest Q2. Can you please try re-downloading it? We're probably going to have to make an public announcement about this...

OK, that one has 15,948,996 rows.

Here's what I got from running the same "sqlite-utils analyze-tables companies2.db company" command against it:

    company.handle: (1/10)

      Total rows: 15948996
      Null rows: 0
      Blank rows: 0

      Distinct values: 15948996

    company.type: (2/10)

      Total rows: 15948996
      Null rows: 0
      Blank rows: 5253878

      Distinct values: 92

      Most common:
        5311279: Privately Held
        5253878: 
        1290064: Self-Owned
        1055857: Partnership
        987045: Public Company
        828643: Self-Employed
        799655: Nonprofit
        334552: Educational
        87681: Government Agency
        35: De financiación privada

    company.name: (3/10)

      Total rows: 15948996
      Null rows: 0
      Blank rows: 1591

      Distinct values: 15549439

      Most common:
        1591: 
        1098: .
        277: A
        246: -
        164: None
        155: X
        142: N/A
        132: ...
        128: x
        122: 1

    company.website: (4/10)

      Total rows: 15948996
      Null rows: 0
      Blank rows: 3249552

      Distinct values: 11272926

      Most common:
        3249552: 
        86957: facebook.com
        57769: instagram.com
        46404: business.site
        31397: linktr.ee
        27882: indiamart.com
        21852: wixsite.com
        19008: negocio.site
        17366: linkedin.com
        13224: yelp.com

    company.founded: (5/10)

      Total rows: 15948996
      Null rows: 0
      Blank rows: 8040264

      Distinct values: 1561

      Most common:
        8040264: 
        524236: 2020
        451742: 2017
        441748: 2018
        426575: 2019
        418318: 2021
        411391: 2016
        389487: 2015
        339212: 2014
        299038: 2013

    company.industry: (6/10)

      Total rows: 15948996
      Null rows: 0
      Blank rows: 1334901

      Distinct values: 421

      Most common:
        1334901: 
        820156: IT Services and IT Consulting
        651746: Construction
        651557: Advertising Services
        465857: Software Development
        455111: Business Consulting and Services
        447922: Real Estate
        435151: Retail
        355049: Financial Services
        312937: Wellness and Fitness Services

    company.size: (7/10)

      Total rows: 15948996
      Null rows: 0
      Blank rows: 2655086

      Distinct values: 123

      Most common:
        6646929: 2-10
        3584483: 11-50
        2655086: 
        1197530: 51-200
        1091094: 1 employee
        421595: 201-500
        150053: 501-1,000
        129373: 1,001-5,000
        44755: 10,001+
        27742: 5,001-10,000

    company.city: (8/10)

      Total rows: 15948996
      Null rows: 0
      Blank rows: 3155391

      Distinct values: 410985

      Most common:
        3155391: 
        269708: London
        124059: Paris
        113135: New York
        99314: São Paulo
        75428: Los Angeles
        69691: Madrid
        67328: Toronto
        63738: Dubai
        63456: New Delhi

    company.state: (9/10)

      Total rows: 15948996
      Null rows: 0
      Blank rows: 4524015

      Distinct values: 58563

      Most common:
        4524015: 
        691167: England
        584141: California
        329584: Texas
        291639: New York
        286723: Florida
        222552: São Paulo
        185925: Maharashtra
        171885: Ontario
        171657: Île-de-France

    company.country_code: (10/10)

      Total rows: 15948996
      Null rows: 0
      Blank rows: 2961064

      Distinct values: 272

      Most common:
        4059985: US
        2961064: 
        1232403: GB
        885302: IN
        756411: FR
        664235: BR
        467501: DE
        414433: NL
        410372: ES
        389535: CA

Re: A raw dump of companies from all over the world by LinkedIn handle

#73

Post by simonw »

simonw wrote: post #72
mfrye0 wrote: post #54

Earlier quoted context omitted.

Ok. I'm not sure how this happened, but I think the dataset was somehow mislabeled. It appears that this dataset is the Q1 version, not the latest Q2. Can you please try re-downloading it? We're probably going to have to make an public announcement about this...

OK, that one has 15,948,996 rows. Here's what I got from running the same "sqlite-utils analyze-tables companies2.db company" command against it: company.handle: (1/10) Total rows: 15948996 Null rows: 0 Blank rows: 0 Distinct values: 15948996 company.type: (2/10) Total rows: 15948996 Null rows: 0 Blank rows: 5253878 Distinct values: 92 Most common: 5311279: Privately Held 5253878: 1290064: Self-Owned 1055857: Partner…

The thing I find most interesting is this:

        269708: London
        124059: Paris
        113135: New York
        99314: São Paulo
        75428: Los Angeles
        69691: Madrid
        67328: Toronto
        63738: Dubai
        63456: New Delhi
I would not have expected São Paulo to come fourth in this list, after New York but in front of Los Angeles. I just learned it's the 4th largest city in the world https://en.wikipedia.org/wiki/List_of_largest_cities - after Tokyo, Delhi, Shanghai - but I guess it has much more of a representation on LinkedIn than those other cities.

Re: A raw dump of companies from all over the world by LinkedIn handle

#75

Post by JohnFen »

hallqv wrote: post #50
JohnFen wrote: post #48

Earlier quoted context omitted.

It is a gray area (in the US) in the sense that there is no clear consensus about it in the courts. There have been court rulings in both directions.

What rulings are you referring to? Been reaffirmed that scraping LinkedIn is legal multiple times, even by Supreme Court. https://techcrunch.com/2022/04/18/web-scraping-legal-court/

I believe that decision was that web scraping doesn't violate the CFAA unless there are access controls (such as the need to log in) to get to it, but they specifically said that the ruling doesn't comment about other possible claims against it.

That's why I think it's still a gray area. But I could be wrong -- I haven't been following this stuff all that closely.

Re: A raw dump of companies from all over the world by LinkedIn handle

#76

Post by mfrye0 »

dang wrote: post #67

The submitted title was "World's largest open source company dataset", but (1) "world's largest" is linkbait and the article walks it back, (2) "open source" could be worded better per https://news.ycombinator.com/item?id=35979581 , and (3) the only thing left in the title after taking those out would be "company dataset", which is too generic to be a good title. I've therefore replaced the title above with what appe…

Hey dang. That's fair.

To be frank, we went back and forth on this, but in the end, thought the original title was ok. The only other large, "open source" dataset we could find was 9M. So after researching, we came to the conclusion that it sounded clickbaity, but was likely accurate.

And yes on "open source". We fully intend for this to be "open source" in the full meaning of the word, but it seems we were moving too fast and missed adding the formal license.

Re: A raw dump of companies from all over the world by LinkedIn handle

#77

Post by [deleted] »

mfrye0 wrote: post #76
dang wrote: post #67

The submitted title was "World's largest open source company dataset", but (1) "world's largest" is linkbait and the article walks it back, (2) "open source" could be worded better per https://news.ycombinator.com/item?id=35979581 , and (3) the only thing left in the title after taking those out would be "company dataset", which is too generic to be a good title. I've therefore replaced the title above with what appe…

Hey dang. That's fair. To be frank, we went back and forth on this, but in the end, thought the original title was ok. The only other large, "open source" dataset we could find was 9M. So after researching, we came to the conclusion that it sounded clickbaity, but was likely accurate. And yes on "open source". We fully intend for this to be "open source" in the full meaning of the word, but it seems we were moving to…

[deleted]

Re: A raw dump of companies from all over the world by LinkedIn handle

#78

Post by prepend »

JohnFen wrote: post #48
prepend wrote: post #41

Earlier quoted context omitted.

> In any case, web scraping is a sort of gray area of the law. I don’t think it’s grey. It seems to be legal as the data are made freely available and the only grey part is that companies don’t want this to happen and would rather charge and not have people scrape.

It is a gray area (in the US) in the sense that there is no clear consensus about it in the courts. There have been court rulings in both directions.

What rulings have been against it?

You don’t need consensus to allow something, you need consensus to be against something, otherwise it’s allowed.

This “scraping is Grey” sounds like FUD to me. Legally, you can scrape anything publicly available as long as you damage the server. You understand that google is just a giant web scraper right?

And all the other search engines spidering the web. Do you think they are risking their business being in the “grey?”

Post reply on HN