Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Personal and social information of 1.2B people discovered in data leak

dataviper.io

Post reply on HN
Print view View original

181–190 of 440 posts

Re: Personal and social information of 1.2B people discovered in data leak

#181

Post by kaivi »

davidhyde wrote: post #134
kaivi wrote: post #85

Earlier quoted context omitted.

I've crawled a popular social network on a large scale, currently doing the same for dating services as a hobby. God, wish I'd still got paid for webscraping. Here are some tricks which may or may not work today: - Have an app where user logs in through said website, then scrape their friends using this user's token. That way you get exponential leverage on the number of API calls you can make, with just a handful of…

You forgot the part about exposing your finished database to unprotected elasticsearch http endpoint ;) In all seriousness does anyone know why you can even host an elasticsearch database as http and without credentials? Seems to be the default. What is the use case for this?

Tbh I'm still selling that data.

For a while I've had reoccurring nightmares that my DB had been stolen and published together with an article on how stupid and incompetent I am.

Re: Personal and social information of 1.2B people discovered in data leak

#182

Post by robbya »

sparkywolf wrote: post #48

I found a vulnerability in linkedIn a few years back that allowed anyone to access a private profile (because client side validation was enough for them I guess..?) They didn't take my report seriously (still not completely patched) and I feel like that told me all I needed to know about their security practices.

A few years of heads up is sufficient to disclose publicly. Full disclosure helps keep companies honest about security.

Re: Personal and social information of 1.2B people discovered in data leak

#183

Post by scraping_legal »

Havoc wrote: post #17

Out of curiosity how do you guys think they managed to scrape LinkedIn on such a large scale? I've been wanting to do some social graph experimentation on it (small scale - say 1000 people near me) but concluded I probably couldn't scrape enough via raw scraping without freaking out their anti-scraping. (And API is a non-starter since that basically says everything is verboten).

The US courts decided that scraping is legal, even if against EULA:

> In a long-awaited decision in hiQ Labs, Inc. v. LinkedIn Corp., the Ninth Circuit Court of Appeals ruled that automated scraping of publicly accessible data likely does not violate the Computer Fraud and Abuse Act (CFAA). This is an important clarification of the CFAA’s scope, which should provide some relief to the wide variety of researchers, journalists, and companies who have had reason to fear cease and desist letters threatening liability simply for accessing publicly available information in a way that publishers object to. It’s a major win for research and innovation, which will hopefully pave the way for courts and Congress to further curb abuse of the CFAA.

https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l...

Re: Personal and social information of 1.2B people discovered in data leak

#184

Post by ibirman »

jerrac wrote: post #165
jillesvangurp wrote: post #155

I was at an Elasticsearch meetup yesterday where we had a good laugh about several similar scandals in Germany recently involving completely unprotected Elasticsearch running on a public IP address without a firewall (e.g. https://www.golem.de/news/elasticsearch-datenleak-bei-conrad... , in German). This beats any of that. Out of the box it does not even bind to a public internet address. Somebody configured this to…

I've been using ES off and on since before 1.0 came out. It has always baffled me that ES doesn't require a username and password by default. ES is a database that has to exist on a network to be usable. Heck, it expects that you have multiple nodes, and will complain if you don't. So one of the first things you do is expose it to the network so you can use it. Yes, it takes some serious incompetence to not realize y…

They offer security as a paid feature.

Re: Personal and social information of 1.2B people discovered in data leak

#185

Post by cookiecaper »

jillesvangurp wrote: post #155

I was at an Elasticsearch meetup yesterday where we had a good laugh about several similar scandals in Germany recently involving completely unprotected Elasticsearch running on a public IP address without a firewall (e.g. https://www.golem.de/news/elasticsearch-datenleak-bei-conrad... , in German). This beats any of that. Out of the box it does not even bind to a public internet address. Somebody configured this to…

I've come across several such ES instances that are 100% exposed to the world without even trying, and ES is by no means the first tool to have this problem. People are never going to stop doing this. Making it annoyingly difficult within ES just weakens them such that some other "wow it's so easy" search product will be better positioned to eat their lunch.

Re: Personal and social information of 1.2B people discovered in data leak

#186

Post by wccrawford »

pc86 wrote: post #162
gbear605 wrote: post #147

Earlier quoted context omitted.

I’m also not an expert, but my understanding is that it applies but would be hard for the EU to take action against them

A law isn't a law if you can't enforce it, so "applies" has kind of a strange meaning in this context then, doesn't it?

A law always has a jurisdiction. EU laws generally don't apply to the US, even if the EU wants them to. There are exceptions, of course.

Re: Personal and social information of 1.2B people discovered in data leak

#187

Post by jillesvangurp »

jerrac wrote: post #165
jillesvangurp wrote: post #155

I was at an Elasticsearch meetup yesterday where we had a good laugh about several similar scandals in Germany recently involving completely unprotected Elasticsearch running on a public IP address without a firewall (e.g. https://www.golem.de/news/elasticsearch-datenleak-bei-conrad... , in German). This beats any of that. Out of the box it does not even bind to a public internet address. Somebody configured this to…

I've been using ES off and on since before 1.0 came out. It has always baffled me that ES doesn't require a username and password by default. ES is a database that has to exist on a network to be usable. Heck, it expects that you have multiple nodes, and will complain if you don't. So one of the first things you do is expose it to the network so you can use it. Yes, it takes some serious incompetence to not realize y…

It has to exist on a private network behind a firewall with ports open to application servers and other es nodes only. Running things on a public ip address is a choice that should not be taken lightly. Clustering over the public internet is not a thing with Elasticsearch (or similar products).

If you are running mysql or postgres on a public ip address it would be equally stupid and irresponsible regardless of the useless default password that many people never change unless you also set up TLS properly (which would require knowing what you are doing with e.g. certificates). The security in those products is simply not designed for being exposed on a public ip address over a non TLS connection. Pretending otherwise would be a mistake. Having basic authentication in Elasticsearch would be the pointless equivalent. Base64 (i.e. basic authentication over http) encoded plaintext passwords is not a form of security worth bothering with. Which is why they never did this. It would be a false sense of security.

At some point you just have to call out people for being utter morons. The blame is on them, 100%. The only deficiency here is with their poor decision making. Going "meh http, public IP, no password, what could possibly go wrong?! lets just upload the entirety of linkedin to that." That level of incompetence, negligence, and indifference is inexcusable. I bet, MS/Linkedin is considering legal action against individuals and companies involved. IMHO they'd be well within their rights to sue these people into bankruptcy.

Re: Personal and social information of 1.2B people discovered in data leak

#189

Post by gbear605 »

pc86 wrote: post #162
gbear605 wrote: post #147

Earlier quoted context omitted.

I’m also not an expert, but my understanding is that it applies but would be hard for the EU to take action against them

A law isn't a law if you can't enforce it, so "applies" has kind of a strange meaning in this context then, doesn't it?

Theoretically, if it were egregious enough, the EU could say to the owners or management of the company that if they went to the EU they would be arrested. That’s enough of a threat that it might convince them.

Re: Personal and social information of 1.2B people discovered in data leak

#190

Post by zmmmmm »

From what I could see the data returned on me was all derived from publicly available sources (eg: my "public" LinkedIn page, my public github page etc). Perhaps others have more but this looks more like an aggregator of public information than a breach of non-public information.

Having said that, I find these companies unspeakably evil - their intent is to make money by harming people (eroding their privacy by making otherwise private personal information easier to get, obviously a gold mine for identity thieves etc).

Post reply on HN