Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Personal and social information of 1.2B people discovered in data leak

dataviper.io

Post reply on HN
Print view View original

271–280 of 440 posts

Re: Personal and social information of 1.2B people discovered in data leak

#271

Post by isoos »

kaivi wrote: post #85
Havoc wrote: post #17

Out of curiosity how do you guys think they managed to scrape LinkedIn on such a large scale? I've been wanting to do some social graph experimentation on it (small scale - say 1000 people near me) but concluded I probably couldn't scrape enough via raw scraping without freaking out their anti-scraping. (And API is a non-starter since that basically says everything is verboten).

I've crawled a popular social network on a large scale, currently doing the same for dating services as a hobby. God, wish I'd still got paid for webscraping. Here are some tricks which may or may not work today: - Have an app where user logs in through said website, then scrape their friends using this user's token. That way you get exponential leverage on the number of API calls you can make, with just a handful of…

> Don't be too kind on the big websites.

I usually recommend latency-based dynamic load control for that. Once the website starts to reply 500-1000ms longer than the average one-thread latency, it is time to take a bit of it back. It is also a co-operative strategy between fellow scrapers, even if they don't know about the other ones pushing larger load on the servers.

Re: Personal and social information of 1.2B people discovered in data leak

#273

Post by GhettoMaestro »

jillesvangurp wrote: post #187
jerrac wrote: post #165

Earlier quoted context omitted.

I've been using ES off and on since before 1.0 came out. It has always baffled me that ES doesn't require a username and password by default. ES is a database that has to exist on a network to be usable. Heck, it expects that you have multiple nodes, and will complain if you don't. So one of the first things you do is expose it to the network so you can use it. Yes, it takes some serious incompetence to not realize y…

It has to exist on a private network behind a firewall with ports open to application servers and other es nodes only. Running things on a public ip address is a choice that should not be taken lightly. Clustering over the public internet is not a thing with Elasticsearch (or similar products). If you are running mysql or postgres on a public ip address it would be equally stupid and irresponsible regardless of the u…

> At some point you just have to call out people for being utter morons. The blame is on them, 100%. [...]

Your attitude is a symptom of a broader issue that plagues this industry: Indifference to risk*probability. If you don't ship software with "secure defaults" (depending on the threat/attack model), you essentially are handing out loaded shotguns, then blaming the "dumb" user when they inevitably point it at their foot and click the trigger. Easy solution: Don't hand out the gun loaded -- make the user do specific actions that enable the usage. Yeah, it creates some friction to first time deployment, but that's a secondary concern to having your freaking DB leaking all over the place.

Re: Personal and social information of 1.2B people discovered in data leak

#274

Post by rvrabec »

Scoundreller wrote: post #19
rvrabec wrote: post #12

Earlier quoted context omitted.

It's weird because for oxydata you have to contact their sales team... but peopledatalabs has an opt out form. https://www.peopledatalabs.com/opt-out-form People Data Labs privacy policy: 3. ACCESS TO AND CONTROL OVER INFORMATION A person may do any of the following at any time by contacting People Data Labs at support@peopledatalabs.com. People Data Labs will reply to a person’s request within five business days. A.…

I want someone to start an opt-out service, where I send them $20, and they send a book of names by registered mail for opt-outs every month. An online opt-out system is too easy for them. I want each one to get a phone-book sized list of opt-outs every month. And the same for data requests. Someone that curates the data collectors, and sends them requests every month. Do you know which country’s “do not call” list I…

Actually working on that project right now - www.thekanary.com. Super early stage but have a big list of brokers and opt out links that I'm automating. Would love early feedback.

Re: Personal and social information of 1.2B people discovered in data leak

#275

Post by z3t4 »

jillesvangurp wrote: post #187
jerrac wrote: post #165

Earlier quoted context omitted.

I've been using ES off and on since before 1.0 came out. It has always baffled me that ES doesn't require a username and password by default. ES is a database that has to exist on a network to be usable. Heck, it expects that you have multiple nodes, and will complain if you don't. So one of the first things you do is expose it to the network so you can use it. Yes, it takes some serious incompetence to not realize y…

It has to exist on a private network behind a firewall with ports open to application servers and other es nodes only. Running things on a public ip address is a choice that should not be taken lightly. Clustering over the public internet is not a thing with Elasticsearch (or similar products). If you are running mysql or postgres on a public ip address it would be equally stupid and irresponsible regardless of the u…

Software should be secure by default. Don't blame the user.

mySQL in comparison wont even let you install without setting a root password. And it only listen on localhost/unix-socket by default. Then you need to explicitly add another user if you want to allow it to login from a non local ip. I don't think it's even possible - to both set a blank root password and allow it to login from a public IP.

So you really think the solution is to blame some low level worker, and sue him/her? The blame should always be on the people in charge, usually the CEO, who set the bar for engineering practices, proper training, etc, or the lack of.

Re: Personal and social information of 1.2B people discovered in data leak

#277

Post by outworlder »

GhettoMaestro wrote: post #273
jillesvangurp wrote: post #187

Earlier quoted context omitted.

It has to exist on a private network behind a firewall with ports open to application servers and other es nodes only. Running things on a public ip address is a choice that should not be taken lightly. Clustering over the public internet is not a thing with Elasticsearch (or similar products). If you are running mysql or postgres on a public ip address it would be equally stupid and irresponsible regardless of the u…

> At some point you just have to call out people for being utter morons. The blame is on them, 100%. [...] Your attitude is a symptom of a broader issue that plagues this industry: Indifference to risk*probability. If you don't ship software with "secure defaults" (depending on the threat/attack model), you essentially are handing out loaded shotguns, then blaming the "dumb" user when they inevitably point it at thei…

But ES doesn't hand over a loaded gun . Someone went out of their way to load the gun up.

Re: Personal and social information of 1.2B people discovered in data leak

#278

Post by sebsito »

jillesvangurp wrote: post #187
jerrac wrote: post #165

Earlier quoted context omitted.

I've been using ES off and on since before 1.0 came out. It has always baffled me that ES doesn't require a username and password by default. ES is a database that has to exist on a network to be usable. Heck, it expects that you have multiple nodes, and will complain if you don't. So one of the first things you do is expose it to the network so you can use it. Yes, it takes some serious incompetence to not realize y…

It has to exist on a private network behind a firewall with ports open to application servers and other es nodes only. Running things on a public ip address is a choice that should not be taken lightly. Clustering over the public internet is not a thing with Elasticsearch (or similar products). If you are running mysql or postgres on a public ip address it would be equally stupid and irresponsible regardless of the u…

I agree. Every anti-moronic default adds friction. I love that I can play with ES quickly via simple URL without any auth.

Re: Personal and social information of 1.2B people discovered in data leak

#279

Post by [deleted] »

jillesvangurp wrote: post #187
jerrac wrote: post #165

Earlier quoted context omitted.

I've been using ES off and on since before 1.0 came out. It has always baffled me that ES doesn't require a username and password by default. ES is a database that has to exist on a network to be usable. Heck, it expects that you have multiple nodes, and will complain if you don't. So one of the first things you do is expose it to the network so you can use it. Yes, it takes some serious incompetence to not realize y…

It has to exist on a private network behind a firewall with ports open to application servers and other es nodes only. Running things on a public ip address is a choice that should not be taken lightly. Clustering over the public internet is not a thing with Elasticsearch (or similar products). If you are running mysql or postgres on a public ip address it would be equally stupid and irresponsible regardless of the u…

[deleted]

Re: Personal and social information of 1.2B people discovered in data leak

#280

Post by GhettoMaestro »

outworlder wrote: post #277
GhettoMaestro wrote: post #273

Earlier quoted context omitted.

> At some point you just have to call out people for being utter morons. The blame is on them, 100%. [...] Your attitude is a symptom of a broader issue that plagues this industry: Indifference to risk*probability. If you don't ship software with "secure defaults" (depending on the threat/attack model), you essentially are handing out loaded shotguns, then blaming the "dumb" user when they inevitably point it at thei…

But ES doesn't hand over a loaded gun . Someone went out of their way to load the gun up.

Bullshit.

If firing up a piece of software creates an unauthenticated, unprotected (non-TLS) endpoint to read-write data, that's a loaded gun. That is PRECISELY the default behavior of ES.

ES has jacked around for years by making TLS and other standard security features premium. To that, I say this: Screw ES and their bullshit business model. Their business model is a leading cause to dumbasses dumping extremely sensitive PII data into a DB that is unprotected - those same folks aren't going to go the extra mile to secure the DB, either by licensing or 3rd party bolt-ons.

Thus, why it must be shipped secure by default. Anything less is a professional felony, in my eyes. Also, screw ES again, in-case I wasn't clear.

Post reply on HN