Live data from Hacker News

A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

blog.newsblur.com

191–200 of 275 posts

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#191
post #110

Earlier quoted context omitted.

And now Google is picking up the definition of footgun(1) from Hacker news! (1) https://i.imgur.com/pHlLFJA.png

I wonder if someone who's not known to Google to like HN would get the same result?

I get a quote from Wiktionary, not HN.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#192
post #187

> When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world Can Docker really be blamed here? It sounds like when they ran MongoDB they explicitly published Mongo's port to the internet by either adding the ports property to docker-compose.yml or the -p flag with a Docker command to open Mongo's port to the outside world. Not to pour salt on an open wound but…

Or there just should've been a password.

This whole incident would've been prevented and hopefully eventually rectified if they'd put auth on the localhost interface as well.

Let your tools save passwords, but it's insanity to have anything just blindly accept requests - you're one mis-directed script away from people wiping out your prod DB, let alone dedicated hackers.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#193
post #192
post #187

> When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world Can Docker really be blamed here? It sounds like when they ran MongoDB they explicitly published Mongo's port to the internet by either adding the ports property to docker-compose.yml or the -p flag with a Docker command to open Mongo's port to the outside world. Not to pour salt on an open wound but…

Or there just should've been a password. This whole incident would've been prevented and hopefully eventually rectified if they'd put auth on the localhost interface as well. Let your tools save passwords, but it's insanity to have anything just blindly accept requests - you're one mis-directed script away from people wiping out your prod DB, let alone dedicated hackers.

Yeah, passwords are good.

I remember back in 2016 I had a similar issue with Redis. I ran it with -p 6379:6379 with no password and someone got into my Redis container back when "crackit"[0] was a thing.

But I never blamed Docker or Redis. I blamed myself because I was careless for not fully understanding the implications of what -p did at the time.

[0]: http://antirez.com/news/96

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#194

Network services really should have auth by default even if they only bind to localhost . There’s so many ways that a localhost service can become accessible to attackers. Any process with network access, no matter what user it’s running as, can access a localhost service - UNIX sockets on the other hand can be restricted by the usual user/group permissions. A localhost service can be exposed by e.g. an SSRF bug from…

100% agree.

'easy to start developing against' falls flat on its face once you deploy to production and realize you have to bolt on authn and authz to your code and data connections instead of properly designing it in from the get go. (it might work if you never deploy your products though. god knows how many products don't see the light of day.)

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#195
post #186
post #83

Earlier quoted context omitted.

MySQL/MariaDB have a completely open root account too... although default firewall rules should prevent public access too, unless Docker likes to punch that hole open too. Yes, root account password and access permissions should be changed upon a fresh install, but the real issue here is Docker's "helpfulness" by opening ports without explicit permission. That's absurd, and has no reasonable excuse.

That's how both Mongo and MySQL, back in the day, became so popular. They had super lax security, making them easy to use for newbie devs, who are frequently scared/easily distracted by security settings. I'm quite convinced the lack of security was by design. Growth hacking and all that. Get everyone onboard and once you have big business going on, you can focus on the minutiae of security, scaling, not losing data.…

does docker get money, though? anyone with just a bit of common sense has their own docker hub mirror and I don't know how else they earn money...

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#196
post #9

It's not all that great blaming the victim. They clearly made the right moves with at least some of their configuration decisions and leaned on the underlying platform not being bonkers (but alas, it was: https://github.com/moby/moby/issues/4737 and https://github.com/moby/moby/issues/22054 ). Should they have hardened in all the other ways for defense in depth, e.g requiring authentication from localhost? Sure. Shou…

> It's not all that great blaming the victim. Everyone wants to be treated as a (software) engineer here, but the engineer's perspective in this situation would be the opposite: The victims are the customers, and the perpetrator, acting in negligence, was Newsblur. Risk management is a core part of the engineer's job. https://www.sebokwiki.org/wiki/Risk_Management

This is completely wrong. By this logic me killing a parent turns the parent into a perpetrator because the parent didn't learn proper self defense.

No, the parent is a victim and the children are dependents of the victim. Thus hurting the parent also hurts the children. It's the same thing with Newsblur. Hurting Newsblur hurts their customers.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#197

Network services really should have auth by default even if they only bind to localhost . There’s so many ways that a localhost service can become accessible to attackers. Any process with network access, no matter what user it’s running as, can access a localhost service - UNIX sockets on the other hand can be restricted by the usual user/group permissions. A localhost service can be exposed by e.g. an SSRF bug from…

It’s an unpopular opinion but it’s 100% a good idea. If at all possible, bind local services to UNIX sockets instead of localhost ports. At least that way you’ll get some measure of access control effectively for free. If you’re writing software that deals with network connections (as a client or server), write the code for UNIX sockets first. It’s usually trivial to bolt network connection code on top, and being abl…

A similar but more flexible solution I like is to create a wireguard network between all nodes and developer workstations, which creates a separate interface called wg0. Bind every private service to this and publish what you want through a reverse proxy.

It has an added benefit of not needing vendor-specific TLS configuration and firewall rules easy to configure since everything goes through the single VPN port.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#198

Holy shit, I've been running open ports for a year with docker, completely ignoring my firewall! This has to be fixed ASAP.

One of the best things one can do with Docker, for security, but at the expense of convenience is to configure it to disable the docker-proxy and IPTables manipulation and manage IPTables yourself. It'll be slow at first, and you'll need new rules each time you bring a container up but you'll have far more control. We switched to this approach when we had an IP dual-stack issue with the proxy and not only do we know…

note that if the containers are bridged the iptables FORWARD table wont affect packet forwarding between containers either. you would need to use ebtables for layer 2 firewalling i am almost certain.

Edit: apparently there is a sysctl setting that makes the bridge netfilter code call the iptables filter routines (bridge-nf-call-iptables) so i am kinda wrong...

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#199
post #171
post #109

Earlier quoted context omitted.

Alternatively, running all your services as VMs also helps. Having root in a VM doesn't typically give you any rights on the hypervisor (at least not on eg Xen).

Well, if they get root on your mongo vm they can still drop all your tables (or ransomware you) right? So would it make a difference in this particular case? Outside the VM tooling probably not being so insane as to bypass the firewall?

Well, in this case docker was trying to be helpful.

On a hypervisor, it's much harder for VMs to influence each other.

Linux containers (and docker amongst them) started out as convenient and reasonably performant, and added security later. One patch at a time.

Historically, hypervisors typically started secure and added performance and convenience over time.

(Very simplified. But I used to work for XenSource back in the day.)

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#200
post #132
post #9

It's not all that great blaming the victim. They clearly made the right moves with at least some of their configuration decisions and leaned on the underlying platform not being bonkers (but alas, it was: https://github.com/moby/moby/issues/4737 and https://github.com/moby/moby/issues/22054 ). Should they have hardened in all the other ways for defense in depth, e.g requiring authentication from localhost? Sure. Shou…

> They clearly made the right moves The machine with DB had a public interface. No matter firewalls, this is just bad. DB machine should be in private subnets, preferably with no inet access at all, even via NAT. Proof that it had public interface. TFA: "Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world"

>The machine with DB had a public interface. No matter firewalls, this is just bad. DB machine should be in private subnets, preferably with no inet access at all, even via NAT.

This is the type of thinking that lead to the attack in the first place. No, you should not pretend that your database is secure because it is hidden deep behind dozens of layers of firewalls.

You should assume that your database is always accessible from the interne and make sure that internet access to your database is not relevant.

With mongodb you are supposed to use TLS encryption and authentication even for communication within the same host because a low privilege hack or mistake could forward a database port or make it public.

In this scenario you not only need to bypass all the network security theater, you also need the password and a valid client certificate signed by a private CA.

Post reply on HN