Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Personal and social information of 1.2B people discovered in data leak

dataviper.io

Post reply on HN
Print view View original

51–60 of 440 posts

Re: Personal and social information of 1.2B people discovered in data leak

#51

Post by fredley »

Havoc wrote: post #17

Out of curiosity how do you guys think they managed to scrape LinkedIn on such a large scale? I've been wanting to do some social graph experimentation on it (small scale - say 1000 people near me) but concluded I probably couldn't scrape enough via raw scraping without freaking out their anti-scraping. (And API is a non-starter since that basically says everything is verboten).

'Mobile Proxies' like https://oxylabs.io/mobile-proxies (no affiliation) allow you to use large pools of mobile or domestic IPs to scrape. It's expensive, but not prohibitively so. Once you've got a mobile IP you become incredible hard to throttle, since you're behind a mobile NAT gateway.

Re: Personal and social information of 1.2B people discovered in data leak

#52

Post by BoorishBears »

anilshanbhag wrote: post #31

People data labs's data is pretty accurate. Here is mine: https://api.peopledatalabs.com/v4/person?api_key=9c6a1382204... You can try it for yourself by changing the email. All of the information is public, so I don't mind. They are basically doing data integration.

Haha, when I was a kid and scared to use my real name for things, for some reason I used my email... which had my real name in it, to open a Github account with a fake name

So the api knows me as the famous architect, Art Vandelay

Re: Personal and social information of 1.2B people discovered in data leak

#53

Post by roywiggins »

badrequest wrote: post #46

Genuinely hope somebody goes to prison for this, but not gonna hold my breath.

This data is accessible at small scales just by registering for a free api key at People Data Labs and making a GET request, and if you want more robust access you could just pay PDL for it.

Re: Personal and social information of 1.2B people discovered in data leak

#54

Post by quotemstr »

kerng wrote: post #22

Seems like the ball is with Google at the moment, the exposed data is on their GCP servers. So, they can figure out next steps.

Get a court order. No infrastructure company on its own should be making value judgements about what it hosts.

Re: Personal and social information of 1.2B people discovered in data leak

#55

Post by bonestamp2 »

shantly wrote: post #45
ordinaryradical wrote: post #28

Earlier quoted context omitted.

Highly recommended. You can put in multiple email addresses, so you can help monitor your non-technical family members’ info as well.

At this point practically everything about me's available either for free or a few dollars. The only interesting thing left is whether a given password has been compromised. The answer to everything else is "yes, it's been leaked". Been that way for most of a decade at this point, guessing it's the same for most other folks with any modern digital or banking presence whatsoever.

I'm sure there are search engines for it too but I noticed that credit karma can tell you which of your passwords have been associated with your email addresses in data breaches.

Credit Karma is free but the CEO appears to be transparent in how they make money (recommending financial products to you based on what they see in your credit profile).

Re: Personal and social information of 1.2B people discovered in data leak

#56

Post by quotemstr »

manigandham wrote: post #3

This is all scrapped public social media data. No credentials or govt information. It's very easy to download or buy this data legally.

Yeah. Email addresses? Phone numbers? All of that is practically public at this point anyway. This article is crying wolf. Someday there is going to be a massive credential compromise.

Re: Personal and social information of 1.2B people discovered in data leak

#57

Post by StuffedParrot »

tyri_kai_psomi wrote: post #26
Havoc wrote: post #17

Out of curiosity how do you guys think they managed to scrape LinkedIn on such a large scale? I've been wanting to do some social graph experimentation on it (small scale - say 1000 people near me) but concluded I probably couldn't scrape enough via raw scraping without freaking out their anti-scraping. (And API is a non-starter since that basically says everything is verboten).

Distributed bot and scraper networks. Thousands of IPs geographically dispersed throughout the world. There is only so much you can do with rate limiting.

Proxies can also work well for cheaper than buying distributed compute.

Re: Personal and social information of 1.2B people discovered in data leak

#58

Post by sillysaurusx »

Havoc wrote: post #17

Out of curiosity how do you guys think they managed to scrape LinkedIn on such a large scale? I've been wanting to do some social graph experimentation on it (small scale - say 1000 people near me) but concluded I probably couldn't scrape enough via raw scraping without freaking out their anti-scraping. (And API is a non-starter since that basically says everything is verboten).

So far, the answers have contained non-technical answers like "Distributed Scraping." Well, yes, obviously.

A more useful answer is: I did this once, many years ago. Back then it was a matter of hooking up PhantomJS and making sure your user string was set correctly. Since PhantomJS was – I think – essentially the same as what headless chrome is today, the server can't determine that you're running a headless browser.

Now, it's not so easy nowadays to do that. There are mechanisms to detect whether the client is in headless mode. But most websites don't implement advanced detection and countermeasures. And in the ideal case, you can't really detect that someone is doing automated scraping. Imagine a VM that's literally running chrome, and the script is set up to interact with the VM using nothing but mouse movements and keyboard presses. You could even throw in some AI to the mix: record some real mouse movements and keyboard presses over time, then hook up some AI to your script such that it generates movements and keyboard presses that are impossible to distinguish from real human inputs. Such a system would be almost impossible to differentiate vs your real users.

The other piece of the puzzle is user accounts. You often have to have "aged" user accounts. For example, if you tried to scrape LinkedIn using your own account, it wouldn't matter if you were using 500 IPs. They would probably notice.

It's hard to counter a determined scraper.

Re: Personal and social information of 1.2B people discovered in data leak

#59

Post by EGreg »

BoorishBears wrote: post #52
anilshanbhag wrote: post #31

People data labs's data is pretty accurate. Here is mine: https://api.peopledatalabs.com/v4/person?api_key=9c6a1382204... You can try it for yourself by changing the email. All of the information is public, so I don't mind. They are basically doing data integration.

Haha, when I was a kid and scared to use my real name for things, for some reason I used my email... which had my real name in it, to open a Github account with a fake name So the api knows me as the famous architect, Art Vandelay

There is a way to get every developer’s email on github thanks to git commits adding it :))
Post reply on HN