Live data from Hacker News

A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

blog.newsblur.com

71–80 of 117 posts

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#71
When I first read the original post, I felt inclined to shift the "blame" on the author, much like they shifted it on Docker, though they are aware of how this footgun can also be a benefit to some:

> It would make for a much more dramatic read if I was hit through a vulnerability in Docker instead of a footgun. By having Docker silently override the firewall, Docker has made it easier for developers who want to open up ports on their containers at the expense of security.

When I want to run my own web server on an arbitrary Linux distro inside of a container, I want to bind to 0.0.0.0:80 and 0.0.0.0:443. That's it.

When I want to run my own database on an arbitrary Linux distro inside of a container, without making it accessible externally, I want to just omit exposing ports. That's it.

I don't want to think about the firewall on the system (even what kind of a firewall it is running, if any) or configure it separately, otherwise it'd be as bind mount target directories on host system not being automatically created, making me use "mkdir -p PATH", which already adds unnecessary friction (if I want the files to be available in the host OS, as opposed to just in abstracted away Docker volumes). After all, if there were more steps, then an awkward dance would inevitably start: "Hey, we launched the container on SOME_DISTRO but we can't access the UI/API/whatever." and you'd have to write platform-specific instructions, as opposed to being able to just give a Docker Compose file or an equivalent and be done with it.

Thus, my original thoughts were along the lines of: "It's not at the expense of security, as much as it is to the benefit of your convenience without any changes to security, if you are aware of what you are doing." though that isn't charitable enough. Something like this would be more helpful instead:

  # Suppose you want to let people know that your container wants to listen on the port 2000
  # You can specify it in the Dockerfile, though this will be more like documentation for the container, as opposed to publishing anything
  # Docs: https://docs.docker.com/engine/reference/builder/#expose
  # File: Dockerfile
  EXPOSE 2000/tcp
  ...
  
  # If you want your port to be available locally, then Docker gives you that ability as well
  # You can even change the port, for example, to 2005 on the host
  # Docs: https://docs.docker.com/compose/compose-file/compose-file-v3/#ports
  # File: docker-compose.yml
  ports:
    - "127.0.0.1:2005:2000/tcp"
  ...
  
  # If you want your port to be available remotely, then you can either omit the IP address or use 0.0.0.0
  # Let's take the above example and use 2005 as the exposed port again
  # File docker-compose.yml
  ports:
    - "2005:2000/tcp"
  ...
  
  # For examples of Docker CLI without Docker Compose/Swarm, have a look at the docs: https://docs.docker.com/config/containers/container-networking/
  # There's also good information there about additional networking options, such as creating custom networks for limiting what can talk to what.
Actually, here's an example that anyone can run, with Docker CLI (using the httpd web server as example, same idea applies to DBs):

  # Launch everything (detached, so we can use the same terminal session, remove containers once stopped)
  docker run -d --rm --name "net_test_not_exposed" httpd:alpine3.17
  docker run -d --rm --name "net_test_local_access" -p "127.0.0.1:2005:80/tcp" httpd:alpine3.17
  docker run -d --rm --name "net_test_public_access" -p "0.0.0.0:2006:80/tcp" httpd:alpine3.17
  docker run -d --rm --name "net_test_public_access_shorthand" -p "2007:80" httpd:alpine3.17
  
  # Check that everything is running
  docker ps
  
  # Or different formatting
  docker ps --format "{{.Names}} is {{.Status}} with ports {{.Ports}}"
  
  # Then you can use a browser to check them, or do it manually, either locally or from another node (with actual IP address)
  # 2005 will be available locally, whereas 2006 and 2007 will be available remotely
  curl http://127.0.0.1:2005
  curl http://127.0.0.1:2006
  curl http://127.0.0.1:2007
  
  # Finally, the cleanup (the --rm will get rid of the containers)
  docker kill net_test_not_exposed net_test_local_access net_test_public_access net_test_public_access_shorthand
I will say that this is a good suggestion:

> Better would be for Docker to issue a warning when it detects that the most popular firewall on Linux is active and filtering traffic to a port that Docker is about to open.

However, the thing with warnings is that they're easy to miss. Either way, I've been there - when I was learning Docker a number of years back, I left its socket open and by the morning the tiny VPS was mining crypto. Good docs help, maybe exposing things publicly by default with a port binding or even using the shorthand in the first place isn't such a good idea either.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#72

Earlier quoted context omitted.

What's ignorant about it? Or wrong, for that matter, since it can't be "slander" if it's true. Docker does bypass default firewall rules and expose container ports to the public. If I run anything else on a server (apache2, say), and tell it to bind to 0.0.0.0:80, it doesn't matter, because the firewall will block it. If I tell a docker container to bind to 0.0.0.0:80, it magically skips over any other protections an…

How is it that you can spawn many containers and have them all bind to `0.0.0.0:80` in your example? Try doing the same with Apache2. Multiple instances listening on Port 80. This is the difference, and the source of ignorance.

You... can't? I mean, you can bind it inside the container, of course, but you can't bind it on the host ("publish"), which seems like the analogous thing to running on the host.

Also, AIUI podman manages to not ignore the firewall, so it's not like it's some inherent issue.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#73
Personally I blame the irritation and cost of setting up private subnets that can access the Internet via NAT in AWS VPCs in addition to Docker's interactions with UFW. Private VPC subnets with NAT Gateways are both unacceptably costly and complicated to setup (source: recently dealt with the combination of NAT Gateways + the Network Firewall service in AWS). Throw in the need for VPC endpoints for many workloads and AWS networking starts to look like an extortion racket for those who want the majority of their infrastructure to be sequestered from the Internet.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#74

Earlier quoted context omitted.

what. docker isn’t the problem here. dbs on public subnets, and the lack of monitoring for accidental db exposure are the actual issues here.

> docker isn’t the problem here I mean... if they weren't using docker it would have been fine, but because they used docker it wasn't fine. That reads like docker is the problem. That further layers could have mitigated it doesn't make docker not the problem.

Where does the buck stop? This was simply a few layers of bad configuration.

They didn't secure their database with auth/access control and they misconfigured docker.

They have a few things at their disposal:

- Using the docker-user chain to set firewall rules

- Running docker such that the default bind address for the port directive is 127.0.0.1 instead of 0.0.0.0. This puts a safety on the footgun.

- Explicitly setting the bind address of the port directive when bringing up the container.

Docker didn't come along, install itself, and open up the port. The sysadmin did. It's unfortunate they didn't know how it interacted with the firewall or how to properly configure it, but given that, should they really have been rolling it out to production systems?

They tested on prod and learned in trial by fire.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#75

This is why you shouldn't be using Docker in production. It's a great tool, but it's simply not designed for that kind of environment. Edit: note I said Docker specifically, nothing about containerization.

I respectfully disagree. It solves a huge problem - reproducible app environment. It comes with its share of footguns too (hint: always explicitly specify pool of IP addresses it can use!), yes, but what doesn't?

I didn’t say he shouldn’t use containerization. He just shouldn’t have used Docker. Docker has always been very dev environment focused.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#76

Earlier quoted context omitted.

I respectfully disagree. It solves a huge problem - reproducible app environment. It comes with its share of footguns too (hint: always explicitly specify pool of IP addresses it can use!), yes, but what doesn't?

its probably not impossible to do this without the footguns

True, it just wasn't done in this particular universe we live in. :shrugs: Still worth using it in production, but being careful.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#77

Earlier quoted context omitted.

> docker isn’t the problem here I mean... if they weren't using docker it would have been fine, but because they used docker it wasn't fine. That reads like docker is the problem. That further layers could have mitigated it doesn't make docker not the problem.

Where does the buck stop? This was simply a few layers of bad configuration. They didn't secure their database with auth/access control and they misconfigured docker. They have a few things at their disposal: - Using the docker-user chain to set firewall rules - Running docker such that the default bind address for the port directive is 127.0.0.1 instead of 0.0.0.0. This puts a safety on the footgun. - Explicitly set…

> They didn't secure their database with auth/access control

True, although they had reason to believe that that was safe.

> they misconfigured docker.

Ah, no, that's where we disagree. They didn't configure it, docker shipped an insane default that bypasses existing security measures. There is absolutely no reason to expect that running a program in docker magically makes it ignore the system firewall.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#78

Earlier quoted context omitted.

I respectfully disagree. It solves a huge problem - reproducible app environment. It comes with its share of footguns too (hint: always explicitly specify pool of IP addresses it can use!), yes, but what doesn't?

I didn’t say he shouldn’t use containerization. He just shouldn’t have used Docker. Docker has always been very dev environment focused.

Are you aware of another containerization technology that would be more suitable?

(genuinly curious - I only have experience with Docker and never felt the need to look elsewhere, even with footguns, but I'm still curious)

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#79

Earlier quoted context omitted.

> docker isn’t the problem here I mean... if they weren't using docker it would have been fine, but because they used docker it wasn't fine. That reads like docker is the problem. That further layers could have mitigated it doesn't make docker not the problem.

If they had understood how docker works, it would've been fine, too. But because they didn't and used a software firewall as their only line of defense and didn't bother with authentication for the DB server, it wasn't fine.

> If they had understood how docker works, it would've been fine, too.

They shouldn't have needed to, for this.

> But because they didn't and used a software firewall as their only line of defense

Okay? That should have been safe. Like, sure, there are ways to add more layers, but that layer shouldn't have failed them.

> didn't bother with authentication for the DB server, it wasn't fine.

They shouldn't have needed to. Like yes, running a DB without authentication isn't a great idea, but they had good reason to believe that it wouldn't be exposed.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#80
post #73

Personally I blame the irritation and cost of setting up private subnets that can access the Internet via NAT in AWS VPCs in addition to Docker's interactions with UFW. Private VPC subnets with NAT Gateways are both unacceptably costly and complicated to setup (source: recently dealt with the combination of NAT Gateways + the Network Firewall service in AWS). Throw in the need for VPC endpoints for many workloads and…

I’m not sure I’m understanding this entirely. I thought the way you should set it up is have e.g. your nginx / istio gateways exposed to the public internet on EIPs. You shouldn’t have your databases exposed to the world, or the hardware running your backend services directly exposed either. Also, security groups. Yes, if you have to use NAT gateways for serving traffic, those are very expensive.
Post reply on HN