Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hacker deleted all of NewsBlur’s Mongo data and is now holding the data hostage

newsblur.com

Post reply on HN
Print view View original

131–140 of 336 posts

Re: Hacker deleted all of NewsBlur’s Mongo data and is now holding the data hostage

#131

Post by mtippett »

wjossey wrote: post #99
vngzs wrote: post #75

Earlier quoted context omitted.

I think there are some good lessons here: 1. Even if you have one way to protect your database (e.g., firewall rules), you should have another. In this case, use a database password or (better) client TLS certificate to authenticate traffic. We're all human and we mess up. You should be designing systems that are graceful in response to your inevitable mistakes. 2. If you can afford another server/a hosting provider…

Just want to yes and you. In general, put everything in private subnets, and make the only way any traffic can get to a server is through a load balancer. There are very few reasons to have a server itself have its own public IP address, and using your load balancer as a chokepoint, means you can set up layers and layers of redundancy to prevent traffic from ever being able to reach a database under your control. Thi…

+1 on the private subnet.

You want to design your system so that if a network a critical misconfiguration occurs you don't open yourself up - you simply stop working.

(Too many years chasing EMR clusters getting dropped onto the internet by users with AWS console access).

Re: Hacker deleted all of NewsBlur’s Mongo data and is now holding the data hostage

#132

Post by wolverine876 »

dfadsadsf wrote: post #11
wolverine876 wrote: post #4

These are crimes. Where is law enforcement? I am not sure why the public still gives them a pass and treats computer crimes differently. With the resources of the federal government, it shouldn't be hard to find and take down the criminals. Think of how easily the criminals exploit their victims - it is just as hard for the criminals to play defense as it is for everyone else.

Actually federal government resources are pretty limited and finding hackers is hard. What's more after you find hackers you often can't do anything about them as they are in China, Iran, Russia or some other non-US friendly country. Even if country if friendly, extradition is expensive and time consuming. After all that you may end up putting random 20 year old guy in jail for a few years without putting any dent in…

> federal government resources are pretty limited

The resources are whatever Congress and the President makes them. The budget is typically over $2 trillion. Dedicating overwhelming resources - billions of dollars, far beyond any hacker group - to the purpose would be a rounding error. The potential access to expertise and equipment is far beyond any hacker group. The legal power is enormous - the government can access, through cooperation and compulsion, so many systems that the hackers would have to try to access at great expense. What major company is going to resist cooperating against these people? They may enthusiastically participate. Just think of the potential of backbone operators, cloud providers, etc. etc. cooperating.

> zero evidence presented that it was even Russians

I believe that's false.

> (jurisdiction issues)

It's a problem, but not impossible. The US has enormous power in the world if they want to use it.

Re: Hacker deleted all of NewsBlur’s Mongo data and is now holding the data hostage

#133

Post by rank0 »

jacquesm wrote: post #90
rank0 wrote: post #79

Earlier quoted context omitted.

Good luck! I appreciate your transparent account of the situation but what does it say about your company if your database got popped by a “script kiddie”?

It says more about Docker than anything else. This is an insane default setting, it's something that should have been fixed when it was first brought to their attention. Computer security is hard enough without loaded footguns like these lying around.

I understand your point but how do you explain the complete absence of database security controls? That part is on Newsblur. Defense in depth is important!

Re: Hacker deleted all of NewsBlur’s Mongo data and is now holding the data hostage

#134

Post by gnyman »

pm90 wrote: post #108
conesus wrote: post #20

NewsBlur's founder here. I'll attempt to explain what's happening. This situation is more of a script kiddie than a hacker. I'm in the process of moving everything on NewsBlur over to Docker containers in prep for the big redesign launching next week. It's been a great year of maintenance and I've enjoyed the fruits of Ansible + Docker for NewsBlur's 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and…

Is 3 hours a large time for an open server to be discovered? Do the attackers just have a giant list of ips that they constantly scan and can instantly know if it’s suddenly open to traffic?

Yes if you publish a known insecure service like mongodb, on the standard port, on a well known VPS provider you can expect it to be automatically compromised within hours if not minutes.

As others commented, scanning the whole Internet is even not a problem so scanning a "limited" part where you are likely to see these services pop up is even less of a problem.

I think the takeaway is that you cannot hide in the masses on the Internet anymore, 10-20 years ago you could throw up a insecure server and it could be fine for a long time.

Nowadays you must assume someone will find and try to login to your service, even if you put it on a non-standard port.

Also, if it's a HTTPS service take note they when you get a certificate you will be announcing that domain to the whole world and publish it to a searchable database (for example https://crt.sh/ ).

Re: Hacker deleted all of NewsBlur’s Mongo data and is now holding the data hostage

#135

Post by risyachka »

jacquesm wrote: post #90
rank0 wrote: post #79

Earlier quoted context omitted.

Good luck! I appreciate your transparent account of the situation but what does it say about your company if your database got popped by a “script kiddie”?

It says more about Docker than anything else. This is an insane default setting, it's something that should have been fixed when it was first brought to their attention. Computer security is hard enough without loaded footguns like these lying around.

Docker is a minor part of the issue.

The main problem is lack of basic security.

Re: Hacker deleted all of NewsBlur’s Mongo data and is now holding the data hostage

#136

Post by lmm »

vngzs wrote: post #95
lmm wrote: post #81

Earlier quoted context omitted.

I agree with only the third of those. The other lessons I'd take would be: 1. Be cautious about trendy technologies that promise to make life easy - often they cut corners to do so, and often security is one of those corners 2. Use real authentication rather than network firewalling. Make your datastore TLS-only and require a valid client certificate to connect; that way it doesn't matter if it's exposed to the inter…

Yeah, there's the whole "zero trust" possibility which I didn't mention. If you do authentication/authorization really well, you can stop doing (2). For things like databases, I think it's better to treat them as if they _could_ be exposed to the open internet, without actually doing so. It's generally not the case that anyone on the internet needs to query your DBs. As described, you're _only_ relying on client TLS…

My view is that for most cases the cost/benefit of multiple layers doesn't stack up. Given a fixed amount of available time and effort, you'll generally get better results by focusing that effort on making one really good layer - e.g. putting active monitoring in place so that you detect when your single layer breaks (whether that's an attacker from outside the network being able to connect to inside the network or an attacker with an expired certificate being able to connect to a live system) - than by spending half as much time each on two layers, IME.

Re: Hacker deleted all of NewsBlur’s Mongo data and is now holding the data hostage

#137

Post by radicalbyte »

vngzs wrote: post #75
conesus wrote: post #20

NewsBlur's founder here. I'll attempt to explain what's happening. This situation is more of a script kiddie than a hacker. I'm in the process of moving everything on NewsBlur over to Docker containers in prep for the big redesign launching next week. It's been a great year of maintenance and I've enjoyed the fruits of Ansible + Docker for NewsBlur's 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and…

I think there are some good lessons here: 1. Even if you have one way to protect your database (e.g., firewall rules), you should have another. In this case, use a database password or (better) client TLS certificate to authenticate traffic. We're all human and we mess up. You should be designing systems that are graceful in response to your inevitable mistakes. 2. If you can afford another server/a hosting provider…

What strikes me about your reply is that 1, 2, 3 are all absolutely basic table-stakes things. SQL injection mitigation level things.

Have we suddenly stopped teaching the basics?

Re: Hacker deleted all of NewsBlur’s Mongo data and is now holding the data hostage

#138

Post by herpderperator »

conesus wrote: post #20

NewsBlur's founder here. I'll attempt to explain what's happening. This situation is more of a script kiddie than a hacker. I'm in the process of moving everything on NewsBlur over to Docker containers in prep for the big redesign launching next week. It's been a great year of maintenance and I've enjoyed the fruits of Ansible + Docker for NewsBlur's 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and…

I'm sorry that you have to go through this, it seems inevitable these days. However, while it's nice that you're sharing your analysis of the situation, you start off by downplaying the attack and calling them a script kiddie. If for example someone finds out they can brute-force Facebook's 6-digit password reset token because they didn't put any rate-limiting in[0], are they a hacker? Is there major skill involved in doing so or just a million-iteration loop to go through all the combinations? They received a $15,000 payout indicating that Facebook values their mistake seriously (although I'd say it's worth much more given it's a guaranteed full account takeover.) So regardless of what you think of the attack, easy or not, you still made a security mistake and you should admit that first and foremost rather than brushing it off as a "script kiddie situation".

Other than that, it's commendable that you have working backups and are responding calmly and with a plan. I hope you get everything back in working order smoothly :)

[0] https://www.theverge.com/2016/3/8/11179926/facebook-account-...

Re: Hacker deleted all of NewsBlur’s Mongo data and is now holding the data hostage

#139

Post by jacquesm »

rank0 wrote: post #133
jacquesm wrote: post #90

Earlier quoted context omitted.

It says more about Docker than anything else. This is an insane default setting, it's something that should have been fixed when it was first brought to their attention. Computer security is hard enough without loaded footguns like these lying around.

I understand your point but how do you explain the complete absence of database security controls? That part is on Newsblur. Defense in depth is important!

Yes, they absolutely have some culpability, but making a change like this without alerting the administrator of the system is the rough equivalent of any process with 'root' privileges on any one of your servers suddenly executing an iptables command to allow all access. You'd only know about it because you got hacked.

Such drastic changes to the security model should only be one after explicit instruction.

The number of companies that operate without access controls between servers on the same segment is unfortunately quite large, database security controls are - again - more often than not left at their default setting and those too are quite often insecure.

Defense in depth always has limited depth, though I totally agree that running a database without access controls is not the way to go.

Re: Hacker deleted all of NewsBlur’s Mongo data and is now holding the data hostage

#140

Post by traskjd »

I saw this mentioned on twitter yesterday, Microsoft Attack Surface Analyzer, an open source toolset for seeing what config changed when performing software installation.

https://github.com/microsoft/AttackSurfaceAnalyzer

Interesting to see such a strong example of where tools like this could help the very next day.

Note: I haven't used this yet, just saw it and made a note.

Post reply on HN