Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A raw dump of companies from all over the world by LinkedIn handle

blog.bigpicture.io

Post reply on HN
Print view View original

91–100 of 108 posts

Re: A raw dump of companies from all over the world by LinkedIn handle

#91

Post by SXX »

babblingfish wrote: post #27

It's funny how OP does not address where this data comes from even though it's obviously from LinkedIn. I see many people in the comments asking questions so I will add my two cents as someone who is currently employed by LinkedIn and has an interest in web scraping. This dataset was taken from scraping the company pages from LinkedIn. A company has to pay to have this page, so this certainly does not include all com…

> If you have a premium account your search is not rate limited so you can iteratively scrape anything you want even though it's technically a violation of the terms of service.

Well. This is certainly not true.

Even with Premium subscription you can hit a hard limit of personal profile views while just actively going through profiles without any automation. And Linkedin will temporary restrict and threaten to ban your account.

Re: A raw dump of companies from all over the world by LinkedIn handle

#92

Post by mfrye0 »

Wronnay wrote: post #66

I always get "Oops! We ran into an error. Contact us at support@bigpicture.io" when I try to sign-up

Apologies. We've had a huge problem with bots, so we have a number of security measures in place. The Google Captcha component is probably flagging you as a bot.

Try disabling your VPN if you're on one, or use a different IP.

Re: A raw dump of companies from all over the world by LinkedIn handle

#93

Post by joshspankit »

B1FF_PSUVM wrote: post #88
babblingfish wrote: post #27

It's funny how OP does not address where this data comes from even though it's obviously from LinkedIn. I see many people in the comments asking questions so I will add my two cents as someone who is currently employed by LinkedIn and has an interest in web scraping. This dataset was taken from scraping the company pages from LinkedIn. A company has to pay to have this page, so this certainly does not include all com…

> and is not being monetized It seems to have a paid version with more data: https://docs.bigpicture.io/docs/free-datasets/companies/#fie... (just the fact, I don't have an opinion)

I’m on the same page. It also requires users to create an account in order to get the free version, and I would argue that in most cases that’s part of a monetization funnel.

Re: A raw dump of companies from all over the world by LinkedIn handle

#94

Post by maxlin »

ttul wrote: post #80
decide1000 wrote: post #79

Not sure why you need an account. Download it here: https://bigpicture-datasets-public.s3.us-west-2.amazonaws.co...

Oh boy, DMCA request incoming...

That is literally their own site. I don't think they would DMCA themselves

Thanks for the link. The work email requirement for just the download is just bullshit (yes I did read their piece, with that kind of logic they could also require credit card info etc while still calling it free like many scammy sites do)

Re: A raw dump of companies from all over the world by LinkedIn handle

#96

Post by babblingfish »

B1FF_PSUVM wrote: post #88
babblingfish wrote: post #27

It's funny how OP does not address where this data comes from even though it's obviously from LinkedIn. I see many people in the comments asking questions so I will add my two cents as someone who is currently employed by LinkedIn and has an interest in web scraping. This dataset was taken from scraping the company pages from LinkedIn. A company has to pay to have this page, so this certainly does not include all com…

> and is not being monetized It seems to have a paid version with more data: https://docs.bigpicture.io/docs/free-datasets/companies/#fie... (just the fact, I don't have an opinion)

:O

Re: A raw dump of companies from all over the world by LinkedIn handle

#97

Post by babblingfish »

simonw wrote: post #29
babblingfish wrote: post #27

It's funny how OP does not address where this data comes from even though it's obviously from LinkedIn. I see many people in the comments asking questions so I will add my two cents as someone who is currently employed by LinkedIn and has an interest in web scraping. This dataset was taken from scraping the company pages from LinkedIn. A company has to pay to have this page, so this certainly does not include all com…

> A company has to pay to have this page, so this certainly does not include all companies. I found a listing in there for my old startup derived from the LinkedIn page, and we never paid for that LinkedIn listing.

Yeah apparently it's free, my bad, I remembered it incorrectly

Re: A raw dump of companies from all over the world by LinkedIn handle

#99

Post by prepend »

tyingq wrote: post #85
prepend wrote: post #78

Earlier quoted context omitted.

What rulings have been against it? You don’t need consensus to allow something, you need consensus to be against something, otherwise it’s allowed. This “scraping is Grey” sounds like FUD to me. Legally, you can scrape anything publicly available as long as you damage the server. You understand that google is just a giant web scraper right? And all the other search engines spidering the web. Do you think they are ris…

There does seem to be some distinction if you log into and scrape: "Only three claims remained for the final order - the violation of the CAN-SPAM Act, violation of the CFAA and California Penal Code. The district court then granted summary judgment to Facebook on all three of the remaining Facebook claims. The district court awarded statutory damages of $3,031,350" https://en.wikipedia.org/wiki/Facebook,_Inc._v._Pow…

That’s completely different though as they are scraping private info that’s proprietary to Facebook.

I think there’s no grey area for whether scraping publicly available data is legal as it is legal.

Re: A raw dump of companies from all over the world by LinkedIn handle

#100

Post by opportune »

medvezhenok wrote: post #81
decide1000 wrote: post #79

Not sure why you need an account. Download it here: https://bigpicture-datasets-public.s3.us-west-2.amazonaws.co...

Presumably as lead generation for their main upsell, which is the enriched version of the same data set :)

This whole thing is amateur to the core, it’s an improperly licensed (are you open source or not? What license? You don’t get to say “probably eventually MIT” lol) freemium scrape of linkedin masquerading as a product
Post reply on HN