Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Personal and social information of 1.2B people discovered in data leak

dataviper.io

Post reply on HN
Print view View original

201–210 of 440 posts

Re: Personal and social information of 1.2B people discovered in data leak

#201

Post by nfoz »

kaivi wrote: post #85
Havoc wrote: post #17

Out of curiosity how do you guys think they managed to scrape LinkedIn on such a large scale? I've been wanting to do some social graph experimentation on it (small scale - say 1000 people near me) but concluded I probably couldn't scrape enough via raw scraping without freaking out their anti-scraping. (And API is a non-starter since that basically says everything is verboten).

I've crawled a popular social network on a large scale, currently doing the same for dating services as a hobby. God, wish I'd still got paid for webscraping. Here are some tricks which may or may not work today: - Have an app where user logs in through said website, then scrape their friends using this user's token. That way you get exponential leverage on the number of API calls you can make, with just a handful of…

Don't you consider this unethical -- if not against the site itself, than against the other users of the site whose data you're scraping?

Re: Personal and social information of 1.2B people discovered in data leak

#202

Post by arpa »

hobofan wrote: post #160
sillysaurusx wrote: post #58

Earlier quoted context omitted.

So far, the answers have contained non-technical answers like "Distributed Scraping." Well, yes, obviously. A more useful answer is: I did this once, many years ago. Back then it was a matter of hooking up PhantomJS and making sure your user string was set correctly. Since PhantomJS was – I think – essentially the same as what headless chrome is today, the server can't determine that you're running a headless browser…

LinkedIn doesn't protection doesn't seem to be that sophisticated at the moment. Someone I know maintains ~weekly up-to-date profiles of a few million users via a headless scraper that uses ~10 different premium accounts and a very low number of different IPs.

That is a violation of ToS (using registerd accounts for scrape) and could carry potential legal implications.

Re: Personal and social information of 1.2B people discovered in data leak

#203

Post by godelski »

shadowgovt wrote: post #138

It's a tragedy that all of this data was available to anyone in a public database instead of.... checks notes... available to anyone who was willing to sign up for a free account that allowed them 1,000 queries. It seems like PDL's core business model is irresponsible regarding their stewardship of the data they've harvested.

Would it be better if this was a paid service? If the issue access to the data, then maybe we should ask if this data should be collected in the first place.

Re: Personal and social information of 1.2B people discovered in data leak

#204

Post by adatavizguy »

kaivi wrote: post #85
Havoc wrote: post #17

Out of curiosity how do you guys think they managed to scrape LinkedIn on such a large scale? I've been wanting to do some social graph experimentation on it (small scale - say 1000 people near me) but concluded I probably couldn't scrape enough via raw scraping without freaking out their anti-scraping. (And API is a non-starter since that basically says everything is verboten).

I've crawled a popular social network on a large scale, currently doing the same for dating services as a hobby. God, wish I'd still got paid for webscraping. Here are some tricks which may or may not work today: - Have an app where user logs in through said website, then scrape their friends using this user's token. That way you get exponential leverage on the number of API calls you can make, with just a handful of…

- Call their API through ipv6, because they may not yet have a proper, ipv6 subnet-based rate limiter.

How would someone do that using node.js? Asking for a friend.

Re: Personal and social information of 1.2B people discovered in data leak

#205

Post by dang »

modzu wrote: post #88
sparkywolf wrote: post #48

I found a vulnerability in linkedIn a few years back that allowed anyone to access a private profile (because client side validation was enough for them I guess..?) They didn't take my report seriously (still not completely patched) and I feel like that told me all I needed to know about their security practices.

linkedin is a computer virus

Could you please stop posting unsubstantive comments to Hacker News? We're trying for a bit better than internet default here.

Re: Personal and social information of 1.2B people discovered in data leak

#206

Post by lmilcin »

ibirman wrote: post #184
jerrac wrote: post #165

Earlier quoted context omitted.

I've been using ES off and on since before 1.0 came out. It has always baffled me that ES doesn't require a username and password by default. ES is a database that has to exist on a network to be usable. Heck, it expects that you have multiple nodes, and will complain if you don't. So one of the first things you do is expose it to the network so you can use it. Yes, it takes some serious incompetence to not realize y…

They offer security as a paid feature.

That's incorrect.

The usual way of using this service is to have backend network configured that connects your services that is not available from outside (ie you have to traverse through services to reach it).

The so called "security" is just a paid feature for companies that want to use ElasticSearch but want to use it in "legacy" way because, presumably, they don't have people to design it correctly.

Re: Personal and social information of 1.2B people discovered in data leak

#207

Post by skissane »

john-radio wrote: post #64
sparkywolf wrote: post #48

I found a vulnerability in linkedIn a few years back that allowed anyone to access a private profile (because client side validation was enough for them I guess..?) They didn't take my report seriously (still not completely patched) and I feel like that told me all I needed to know about their security practices.

I reported an issue to the LinkedIn competitor https://about.me two years ago where signing in with my Google credentials gives me access to some the account of some random other person with a similar name to me. I think that during registration, I attempted to register about.me/johnradio (except it's not "johnradio"), but he was already using it, and then the bug occurred that gave me this access. I randomly check e…

My gmail is my first initial followed by my last name. There are other people on this planet with same first initial and last name, some of whom seem to think that must be their email too, because I keep on getting emails where they used it to sign up for things.

Re: Personal and social information of 1.2B people discovered in data leak

#208

Post by arpa »

d33 wrote: post #62
kitotik wrote: post #4

Yet another Elasticsearch server wide open. This is going to make the flurry of open mongodb servers look trivial.

I wouldn't be surprised if the starting point for this vulnerability wasn't ES, but Docker. Docker by default modifies iptables and if you hack together a system that uses both software running directly on the host and in containers, it's going to expose the forwarded containers to the Internet - which you might not be expecting, since a bind to localhost would be enough to expose a service. It's always a good idea t…

No. It's not dockers' fault you did not read the manual and expose the ports wrong: you can bind the port to specific ips for export and tjat address should be 127.0.0.1

Re: Personal and social information of 1.2B people discovered in data leak

#209

Post by [deleted] »

troebr wrote: post #117
ThrustVectoring wrote: post #97

Earlier quoted context omitted.

It looks like it's a US-based company without enough of a European presence to fall under their jurisdiction.

https://gdpr.eu/companies-outside-of-europe/ it looks like it would? I'm no expert though.

[deleted]

Re: Personal and social information of 1.2B people discovered in data leak

#210

Post by opless »

spitfire wrote: post #192
opless wrote: post #174

Earlier quoted context omitted.

There’s a torrent going around

Do you know where I can find a torrent for this leak?

I don't think it's wise to add a magnet link here.

But as I recall looking for Breach Compilation may help finding the requisite gist on GitHub.

Post reply on HN