Personal and social information of 1.2B people discovered in data leak • Hacker News Board

Re: Personal and social information of 1.2B people discovered in data leak

#281

Post by bound008 » Fri, Nov 22, 2019, 9:38 PM UTC

jillesvangurp wrote: post #187

jerrac wrote: post #165

Earlier quoted context omitted.

I've been using ES off and on since before 1.0 came out. It has always baffled me that ES doesn't require a username and password by default. ES is a database that has to exist on a network to be usable. Heck, it expects that you have multiple nodes, and will complain if you don't. So one of the first things you do is expose it to the network so you can use it. Yes, it takes some serious incompetence to not realize y…

It has to exist on a private network behind a firewall with ports open to application servers and other es nodes only. Running things on a public ip address is a choice that should not be taken lightly. Clustering over the public internet is not a thing with Elasticsearch (or similar products). If you are running mysql or postgres on a public ip address it would be equally stupid and irresponsible regardless of the u…

This is a fallacy of distributed systems. Never trust the network. Best case you get packets destined for somewhere else, worst case you your network segmented wasn't actually segmented.

Re: Personal and social information of 1.2B people discovered in data leak

#282

Post by outworlder » Fri, Nov 22, 2019, 9:39 PM UTC

thegeomaster wrote: post #221

lacker wrote: post #216

Earlier quoted context omitted.

If you set up elasticsearch on a cloud service like AWS, by default your firewall will prevent the outside world from interacting with it, and no authentication is really necessary. If you do use authentication, you probably wouldn't want username+password, you would probably want it to hook into your AWS role manager thing. So to me, username+password seems useful, but it isn't going to be one of the top two most co…

I don't see why, though. It's much safer to start with a secure setup and then have the user disable the security explicitly (hopefully knowing what they're doing). Yes, username/password auth is not that common, but isn't it better than having no auth at all?

Ok, let's say username/password is mandatory and enabled by default. I see to options.

Option one, they generate an unique password for every installation – non trivial to do, because at which point do you do it? It can't be before a cluster is formed, as you'll have a split brain generating a bunch of credentials. If you do it afterwards, then there is a period of time when you cluster is not yet protected. Worse yet, unprotected and handshaking authentication. So you don't do that.

You could make the user input the credentials. What is to prevent them from creating weak credentials? And worse, they have to do that for every node (or at least the masters). Not a good experience and lost credentials will probably be the subject of a good many support calls.

So most products don't do that. What they do is default passwords. Which is arguably no security at all and doesn't protect anything. It may make it just a tiny bit easier to do the right thing afterwards (by changing to better credentials). Still, there's a period of time while the cluster is unprotected (default credentials are as good as no credentials).

Authentication does little to protect against the sort of people who are exposing databases to the public. If it is easily disabled, then they will be doing just that. Because they are already doing that by forcing databases to bind to publicly accessible interfaces.

Re: Personal and social information of 1.2B people discovered in data leak

#283

Post by Havoc » Fri, Nov 22, 2019, 9:40 PM UTC

kaivi wrote: post #85

Havoc wrote: post #17

Out of curiosity how do you guys think they managed to scrape LinkedIn on such a large scale? I've been wanting to do some social graph experimentation on it (small scale - say 1000 people near me) but concluded I probably couldn't scrape enough via raw scraping without freaking out their anti-scraping. (And API is a non-starter since that basically says everything is verboten).

I've crawled a popular social network on a large scale, currently doing the same for dating services as a hobby. God, wish I'd still got paid for webscraping. Here are some tricks which may or may not work today: - Have an app where user logs in through said website, then scrape their friends using this user's token. That way you get exponential leverage on the number of API calls you can make, with just a handful of…

>- Call their API through ipv6, because they may not yet have a proper, ipv6 subnet-based rate limiter.

Clever. VMs with IPV6 are cheap as a bonus :)

Same for non-js mobile. Thanks for the tips

Re: Personal and social information of 1.2B people discovered in data leak

#284

Post by dredmorbius » Fri, Nov 22, 2019, 9:41 PM UTC

ars wrote: post #79

I don't know about other people, but I have zero personal info with LinkedIn and Facebook. They only info they have about me is info I don't mind being public. If I want something to be private I don't tell it to them. It's as simple as that. Google on the other hand, knows lots of private things.

Through shadow profiles, third-party submissions, cross-site cookie tracking, and integration of offline data records, this almost certainly is absolutely false.

Unless you've directly pursued all legal (or otherwise) mechanisms to ascertain this directly, the best you can say is that you're unaware of any information that's been acquired, and that you didn't knowingly or intentionally contribute any yourself.

The article here describes precisely this practice, in its fourth paragraph and following, in the section titled "Data Enrichment":

For a very low price, data enrichment companies allow you to take a single piece of information on a person (such as a name or email address), and expand (or enrich) that user profile to include hundreds of additional new data points of information. As seen with the Exactis data breach, collected information on a single person can include information such as household sizes, finances and income, political and religious preferences, and even a person’s preferred social activities.

Please let's put this canard to rest.

Re: Personal and social information of 1.2B people discovered in data leak

#285

Post by jillesvangurp » Fri, Nov 22, 2019, 9:41 PM UTC

LaGrange wrote: post #200

jillesvangurp wrote: post #187

Earlier quoted context omitted.

It has to exist on a private network behind a firewall with ports open to application servers and other es nodes only. Running things on a public ip address is a choice that should not be taken lightly. Clustering over the public internet is not a thing with Elasticsearch (or similar products). If you are running mysql or postgres on a public ip address it would be equally stupid and irresponsible regardless of the u…

> It has to exist on a private network behind a firewall with ports open to application servers and other es nodes only. Running things on a public ip address is a choice that should not be taken lightly. Clustering over the public internet is not a thing with Elasticsearch (or similar products). I've met at least one cloud provider in the past (small Dutch thing) that provides _only_ public IP addresses. They do hav…

Well, lets agree it's a sad thing. Very sad.

Re: Personal and social information of 1.2B people discovered in data leak

#286

Post by Havoc » Fri, Nov 22, 2019, 9:42 PM UTC

davidhyde wrote: post #134

kaivi wrote: post #85

Earlier quoted context omitted.

I've crawled a popular social network on a large scale, currently doing the same for dating services as a hobby. God, wish I'd still got paid for webscraping. Here are some tricks which may or may not work today: - Have an app where user logs in through said website, then scrape their friends using this user's token. That way you get exponential leverage on the number of API calls you can make, with just a handful of…

You forgot the part about exposing your finished database to unprotected elasticsearch http endpoint ;) In all seriousness does anyone know why you can even host an elasticsearch database as http and without credentials? Seems to be the default. What is the use case for this?

>You forgot the part about exposing your finished database to unprotected elasticsearch http endpoint ;)

I'll cut straight to the chase and post it on hn. This intermediate step of waiting for someone to discover it takes too long

Re: Personal and social information of 1.2B people discovered in data leak

#287

Post by outworlder » Fri, Nov 22, 2019, 9:43 PM UTC

Why are there people running anything publicly accessible?

If you are running on the cloud, there is no need for any VMs to have any public IPs at all. Exception for your Bastion host, and even that should be restricted to known networks.

All incoming traffic needs a layer of indirection. On cloud providers that's usually their load balancers.

Re: Personal and social information of 1.2B people discovered in data leak

#288

Post by WA » Fri, Nov 22, 2019, 9:44 PM UTC

I wonder how high the GDPR fine will be.

Re: Personal and social information of 1.2B people discovered in data leak

#289

Post by golemiprague » Fri, Nov 22, 2019, 9:46 PM UTC

jillesvangurp wrote: post #187

jerrac wrote: post #165

Earlier quoted context omitted.

I've been using ES off and on since before 1.0 came out. It has always baffled me that ES doesn't require a username and password by default. ES is a database that has to exist on a network to be usable. Heck, it expects that you have multiple nodes, and will complain if you don't. So one of the first things you do is expose it to the network so you can use it. Yes, it takes some serious incompetence to not realize y…

It has to exist on a private network behind a firewall with ports open to application servers and other es nodes only. Running things on a public ip address is a choice that should not be taken lightly. Clustering over the public internet is not a thing with Elasticsearch (or similar products). If you are running mysql or postgres on a public ip address it would be equally stupid and irresponsible regardless of the u…

The responsibility is always on the user but it doesn't make the decision to make it open by default less stupid. I remember being puzzled about this when using elastic search for the first time. I was trying to set up a log server with kibana and the first thing I was thinking about was how do I block this thing, why it is open for everyone? It should have a user/pass like any other database, it is not rocket science, that should be the default.

Re: Personal and social information of 1.2B people discovered in data leak

#290

Post by oblio » Fri, Nov 22, 2019, 9:47 PM UTC

sebsito wrote: post #278

jillesvangurp wrote: post #187

Earlier quoted context omitted.

It has to exist on a private network behind a firewall with ports open to application servers and other es nodes only. Running things on a public ip address is a choice that should not be taken lightly. Clustering over the public internet is not a thing with Elasticsearch (or similar products). If you are running mysql or postgres on a public ip address it would be equally stupid and irresponsible regardless of the u…

I agree. Every anti-moronic default adds friction. I love that I can play with ES quickly via simple URL without any auth.

That's how we got PHP, Javascript, Visual Basic, MySQL (before version 5), Mongo.

You'd think that at some point we'd understand that there's way more morons out there than sensible people.