A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021) • Hacker News Board

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#111

Post by yjftsjthsd-h » Mon, Feb 20, 2023, 1:58 AM UTC

remram wrote: post #109

yjftsjthsd-h wrote: post #108

Earlier quoted context omitted.

I don't think so; at worst , each of those is passive and might fail to make you more secure. Docker goes out of its way to add holes to existing security. It's like... if iptables decided to ship a feature that detected when it was running in AWS and "helpfully" automatically reconfigured your security groups to allow any traffic that was allowed in iptables, that would be the same.

I posted that a bunch of times already but that's not what happens. Docker does not override the firewall, it just uses forwarding which kick in before filtering (iptables has separate "nat" and "filter" tables). The host's firewall just doesn't apply to containers and VMs, because they are not listening on a port on the host. Docker could not extend the host's firewall to containers without changing how iptables wor…

> I posted that a bunch of times already but that's not what happens. Docker does not override the firewall, it just uses forwarding which kick in before filtering (iptables has separate "nat" and "filter" tables).

That's exactly what happens. Docker sticking its rules before the normal filters is exactly what I mean when I say it bypasses the firewall rules. Like... you're literally describing the implementation details of what I said it does.

> The host's firewall just doesn't apply to containers and VMs, because they are not listening on a port on the host.

They clearly are? If a docker container was listening but not on a port on the host's internet-facing interface (indirected though it may be), none of this would be a problem. The problem is precisely that if you have a host firewall rule that says "this port is blocked", docker will "helpfully" preempt that and connect that port to a container, which is a massive and unreasonable footgun.

> Docker could not extend the host's firewall to containers without changing how iptables work.

podman seems to manage fine, so I have trouble believing that.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#112

Post by remram » Mon, Feb 20, 2023, 2:34 AM UTC

yjftsjthsd-h wrote: post #111

remram wrote: post #109

Earlier quoted context omitted.

I posted that a bunch of times already but that's not what happens. Docker does not override the firewall, it just uses forwarding which kick in before filtering (iptables has separate "nat" and "filter" tables). The host's firewall just doesn't apply to containers and VMs, because they are not listening on a port on the host. Docker could not extend the host's firewall to containers without changing how iptables wor…

> I posted that a bunch of times already but that's not what happens. Docker does not override the firewall, it just uses forwarding which kick in before filtering (iptables has separate "nat" and "filter" tables). That's exactly what happens. Docker sticking its rules before the normal filters is exactly what I mean when I say it bypasses the firewall rules. Like... you're literally describing the implementation det…

I am not saying that it isn't a footgun, but it is not a conscious decision of Docker to "override" or "insert an allow rule", who actively bypass restrictions if they exist. Rather the situation is a result of implementing this the "naive" way, doing forwarding in the "nat" table that is meant for other hosts (e.g. treating containers as distinct hosts, which in a sense they are, since they have their own (virtual) network interfaces and port namespace).

VirtualBox works the same way, if you use the bridge networking. So does kvm. And so does Podman, I am not sure why you thought otherwise (maybe there is a mode where it uses a userspace proxy rather than iptables? When running rootless for example? Does not happen on my machine, ufw rules are ignored).

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#113

Post by ThePhysicist » Mon, Feb 20, 2023, 7:20 AM UTC

return_to_monke wrote: post #81

ThePhysicist wrote: post #18

Using "ufw" on a production server is an anti-pattern, in my opinion. You should use something like ferm instead.

can you please explain why?

First, UFW only exposes a small subset of iptables functionality through its UI, if you want anything else you'll have to add it manually, which is error-prone. And UFW can't (or couldn't, when I last looked at it) read config from multiple files, which makes it really hard to build up configuration declaratively & separate concerns. When deploying using Ansible you want to be able to add rules for different services independently, which is easy using Ferm as that reads config files from a directory, but very difficult using UFW.

Also, disabling UFW by accident can happen quite easily, if you e.g. remove your systems' Python version (to install a new one) it will uninstall UFW as well, opening up all ports & chains in the process. On a desktop system that might be OK, but it's just not acceptable for a server system IMHO. Hence the first thing I do in any Ansible base role is to remove UFW and install Ferm.

Ferm isn't perfect either but it has much better support for most iptables functionality and the way it builds up and applies rules makes it way more suited to a production environment.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#114

Post by nickjj » Mon, Feb 20, 2023, 3:40 PM UTC

nickstinemates wrote: post #106

nickjj wrote: post #60

Am I missing something? The article wrote: > When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world But the blog post doesn't mention how Docker "helpfully inserted an allow rule". Is this because NewsBlur ran the container using the -p 27017:27017 flag without reading the docs around what publishing a port does? You don't need to publish a port for (2) co…

> Is this because NewsBlur ran the container using the -p 27017:27017 flag without reading the docs around what publishing a port does? Yep, and ended up shifting blame to Docker by calling it a "footgun."

I think if this happened in 2014 it would be more accepted as a footgun since Docker was much less known then.

But in 2021 they:

    - Ran their main DB in Docker without knowing the fundamentals of Docker
    - Shipped it to production without anyone catching it in a review process

I mean, the incident isn't fun and I wish no one harm on their production data but this is like saying you never used Linux before, stumbled upon a prompt onto your production server, ran `rm -rf /var/lib/mongodb` and now you wrote a blog about how you're upset at Ken Thompson and Dennis Ritchie[0] because an `rm` footgun allowed you to delete your production database.

Now, to be constructive about this. Can or should Docker do anything to help prevent this? Would adding a warning to the terminal output that you're opening a port to the outside world be worth it? That sort of feels like overkill. Maybe a Docker plugin could be created to add more text or extra protection through y/n prompts when running anything dangerous, such as `docker compose down -v`. But this feels like a scenario where someone would do the bad thing without protection, get bitten by it and then discover the plugin afterwards where it's too late.

[0]: The creators of the `rm` tool.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#115

Post by nickstinemates » Mon, Feb 20, 2023, 9:29 PM UTC

yjftsjthsd-h wrote: post #72

nickstinemates wrote: post #67

Earlier quoted context omitted.

How is it that you can spawn many containers and have them all bind to `0.0.0.0:80` in your example? Try doing the same with Apache2. Multiple instances listening on Port 80. This is the difference, and the source of ignorance.

You... can't? I mean, you can bind it inside the container, of course, but you can't bind it on the host ("publish"), which seems like the analogous thing to running on the host. Also, AIUI podman manages to not ignore the firewall, so it's not like it's some inherent issue.

You can. Because each container has its own ip address. Publish IS EXACTLY the creation of the iptables entry to route traffic over the docker0 bridge.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#116

Post by Badumtss8 » Tue, Feb 21, 2023, 5:44 AM UTC

nickstinemates wrote: post #63

Badumtss8 wrote: post #47

Earlier quoted context omitted.

nginx listens on port 80. Accessible from machine. Inaccessible from outside unless allowed by ufw. docker listens on port X. Accessible on machine. Also accessible from outside regardless of ufw. No amount of time and experience will make you think that configuring a software to listen on a port will automagically poke a hole in the firewall.

The scope of Docker and nginx are incomparable, so the comparison is wrong. It starts with the simple truth: `docker` doesn't `listen` on any port. Or maybe a simple question: How can I run `docker run -p 8080 nginx` over and over without port conflict? Or - lets expand scope even more. How is docker supposed to know about your choice of firewall? What about upstream firewalls? What about multiple versions of firewal…

The same way nginx doesn't care about the choice of firewall because it doesn't automagically poke holes in the firewall.

May be a simple question: How are you able run nginx and open http:// localhost without it making any changes to the firewall. Can go on and on.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#117

Post by cpuguy83 » Thu, Feb 23, 2023, 6:52 PM UTC

remram wrote: post #103

cpuguy83 wrote: post #85

Earlier quoted context omitted.

To be clear, Docker inserted an allow rule because the user asked it to. Docker is relatively easy to set up without needing to use port forwarding to allow communication. Docker's networking model is explicitly designed to prevent the scenario in the post. Not to negate the fact that many are surprised by how docker bypasses ufw rules. This is painful and I would love to see a way for people to safely use port forwa…

Docker does not "insert an allow rule", it uses the "nat" table to forward connections to the container, and not the "filter" table where the firewall rules live. So packets never go through those filters. This is not a consequence of Docker's action but a fact of how Linux networking works. Docker could have bigger warnings, or default to only accepting connections from localhost (I think it should), but it could no…

Thanks, I was just using the parent's vocabulary to make my point.