Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

blog.newsblur.com

Post reply on HN
Print view View original

71–80 of 275 posts

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#71

Post by grayhatter »

This isn't a footgun. A footgun is when something happens that should be expected, but isn't for reasons of negligence or ignorance to the thing that should be expected. The example that everyone seems to love is pointer arithmetic. If you make a basic error in your math, invalid memory access may occur and then likely more bad stuff.

Docker altering firewall rules without explicit instructions to do so is either a feature, if you're looking to defend docker, Or a security defect if you're feeling rational. There's an argument to be made for the root cause of the issue, The docker container manifest, or the application itself. But to pretend like this is what docker should be doing, users should know better just seems wrong.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#72

Post by jessikat »

jbverschoor wrote: post #53
thatwasunusual wrote: post #41

Earlier quoted context omitted.

That might be true, but regardless of docker (or other, similar solutions), shouldn't MongoDB have had auth protection?

From 3.6 the default binding is localhost. That would be ook, except that if you mail the power, you listen to all interfaces. This will happen with Redis as well and is a problem with both docker and its dockerfile

Even on localhost, you should still use auth. One shouldn't naively assume someone can't break into your server.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#73

Post by plasma »

I feel there are two issues to fix:

1. Docker doing this iptables change out of the box, and

2. MongoDB not having a password set out of the box

The life of a developer (and solo dev) means you often have limited time you need to navigate a project and try and do your best to understand, deploy and use it -- this is just one of many tasks on your TODO today to get you closer to operating your product.

I really wish these kinds of things were more secure in a few ways:

1. Defensive defaults (passwords, not opening holes in firewalls), and

2. Not making the security hard to use

If its painful to work with the security feature and "get it working", someone with limited time may just undo the defensive defaults to get things working again (doh).

But I get it, sometimes a security piece on by default is so cryptically painful to understand that you get so frustrated with it you just turn it off.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#74

Post by drdaeman »

jbverschoor wrote: post #57
foobarbazetc wrote: post #12

Wait... people run their databases on public IPs?

No, they run databases that listen to localhost, and then use docker to forward that private binding to the public ip

No, they run database that listens to 0.0.0.0/0 and/or ::0/0 on a private isolated network interface (eth0 inside Docker namespace), then let Docker create a NAT to forward packets from a public network interface on the host to this database.

The latter is a mistake. It doesn't ever make sense to expose hosted container ports to host, with the exception of public-facing stuff (like a webserver). The whole backoffice should be on a separate network.

And this is how Docker works by default, since forever, unless one very explicitly requests that it exposes ("publishes") the ports.

Just don't expose stuff. If the host needs to talk to a database or something, it can always talk to it via that `docker0` (or however it's called, I don't really remember the details) interface. (I believe it doesn't always add a route, because on one of my machines I have to manually add a route so Traefik on a host is able to talk to the containers on a isolated virtual network.)

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#75

Post by arp242 »

mdellavo wrote: post #8
sneak wrote: post #4

The mistake here is a MongoDB that didn't require authentication, not that docker's clunky iptables setup exposed it to the internet. Relying solely on a host-based firewall for access control is, for reasons which must now be obvious, admin incompetence. They are responsible for securing the containers. They didn't. Your services should be using authentication even if they are only bound to localhost.

I don't know why this is being downvoted. Multiple overlapping layers of security would have given newsblur a backup in case of accidental "footguns". Unauthenticated mongodb instances are a pretty common problem - it's why a "script kiddie" was so successful.

> I don't know why this is being downvoted.

The core message ("using auth on MongoDB would have prevented this, it's always a good idea to add password auth just in case") is perfectly reasonable; we can all learn from this, and it's perfectly fine to point out such things.

But the way it was phrased was absolutely not okay. People make mistakes all the time and they are not "incompetent". This is the classic "I am very smart, I never make mistakes, if you made a mistake then you're an idiot. You probably eat poop. I am smart btw"-attitude that's just ... ugh...

People rely on firewalls to prevent mistakes from becoming disastrous. Defeating that silently is super surprising. People don't know everything about every piece of tech they use either; very few people do: it's just too much information.

And it's not like auth alone is perfect. Remember when a bug in MySQL allowed people to bypass the auth? Good thing I put a firewall in front of my Doc... oh, no, wait...

So this is why I downvoted it.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#76

Post by judge2020 »

ramraj07 wrote: post #66

The biggest issue with docker is the false sense of security it seems to give a lot of engineers into thinking they know infra when they really don't. I stay away from this because I don't understand it fundamentally, and now this proves it's better to not think these new technologies are your friend unless you actually know what you're doing (which apparently most don't).

Staying away from it is still not the best strategy - at least learn and play with it to understand its strengths and weaknesses.

> it seems to give a lot of engineers into thinking they know infra when they really don't

Maybe, but technology changes over time - I don’t see many new projects choosing VMware over docker/OCI for new infrastructure deployment since you usually don’t need a full VM for applocations that just need isolation and easy static deployments.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#77

Post by spockz »

caymanjim wrote: post #62

Docker has worked like this for a long time. It's a really bad default, and they damn well ought to stop operating this way by default, but it's also like Docker Devops 101. When you install Docker on a Linux system, you should configure the DOCKER-USER chain to drop everything originating on your public network interface. You should also stop running services bound to localhost and instead run them on a private netw…

When I moved my docket setup to a new fedora server a few months ago I was surprised I had to add firewall rules to allow traffic to the ports exposed in the containers. I’m not sure whether this is for any port or specially for things running under the docker user but RHEL should be safe by default.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#78

Post by qbasic_forever »

onebike wrote: post #45

This (Docker opening a hole in my firewall) is why I moved my dev server from Linode to Digital Ocean. DO provides a “cloud firewall” that provides something akin to AWS security groups and therefore can’t be messed by Docker. Linode doesn’t have anything like that (last time I checked at least).

Make sure to flip on the feature though, it's not on by default last I saw spinning up some droplets a few months ago.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#80

Post by cbo100 »

Firerouge wrote: post #19
eganist wrote: post #9

It's not all that great blaming the victim. They clearly made the right moves with at least some of their configuration decisions and leaned on the underlying platform not being bonkers (but alas, it was: https://github.com/moby/moby/issues/4737 and https://github.com/moby/moby/issues/22054 ). Should they have hardened in all the other ways for defense in depth, e.g requiring authentication from localhost? Sure. Shou…

This same one got me after years of using Docker, I only discovered it after using the combination of Ubuntu Server (and it's ufw) on a DMZed device. I was running what I thought was an internal FTP instance for almost a week. Luckily it was about as hardened as regular ftp can be, but I noticed the problem when my service wasn't able to log in as the (very low) connection limit was filled by someone attempting passw…

Yep. This exact interaction between UFW and Docker had me stumped for a while too.

Totally unexpected outcome.

Luckily I discovered it in testing, so didn't make it to production. But annoying that those issues still remain for so long.

Post reply on HN