Live data from Hacker News

A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

blog.newsblur.com

121–130 of 275 posts

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#121

Network services really should have auth by default even if they only bind to localhost . There’s so many ways that a localhost service can become accessible to attackers. Any process with network access, no matter what user it’s running as, can access a localhost service - UNIX sockets on the other hand can be restricted by the usual user/group permissions. A localhost service can be exposed by e.g. an SSRF bug from…

It’s an unpopular opinion but it’s 100% a good idea. If at all possible, bind local services to UNIX sockets instead of localhost ports. At least that way you’ll get some measure of access control effectively for free.

If you’re writing software that deals with network connections (as a client or server), write the code for UNIX sockets first. It’s usually trivial to bolt network connection code on top, and being able to deal with sockets directly can make automated testing significantly easier.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#122
The real issue here is that the “backend” database engine was exposed on the public internet. It should have been a layer behind that with suitable protection, as mentioned a VPC with security group rules or equivalent firewall.

Regardless of the docker foot gun this could have been a human fudge as well.

Having done exactly this myself before with SQL server and Slammer I didn’t blame the default configuration, I blamed myself.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#123
Props to the developer for being this open and letting others learn and discuss the implications of this situation! I rather enjoyed the original discussion as well, even though the situation itself was unfortunate: https://news.ycombinator.com/item?id=27613217

I agree with the other posters, that Docker can cause problems with firewalls, as pointed out both in the article, in the GitHub issue, and in the other thread as well: https://github.com/moby/moby/issues/4737

Furthermore, it also seems to me, that not only Docker should respect the firewall rules (even though it'd confuse another group of people about why their services aren't accessible externally and would necessitate manual firewall rule management, short of some explicit way to do that, such as docker run ... -p 80:3000 --expose-in-firewall 80 ...) and that MongoDB also should have secure defaults, as any other piece of software!

That said, at least to me it appears that Docker Compose, Docker Swarm and other technologies have attempted to introduce a networking abstraction to allow running services (more) securely and privately, by not exposing their ports on the host directly.

For example, see the following example of a Compose file, which would only expose a web server to the world:

  version: '3.4'
  services:
    # you would probably want to run MongoDB not exposed to the outside world at all
    # notice that there are no ports exposed here, given that we're using Docker networking
    mongodb_container:
      image: mongo:bionic
      restart: "unless-stopped"
      environment:               
        - MONGO_INITDB_ROOT_USERNAME=lazy_example_of_a_single_file_config
        - MONGO_INITDB_ROOT_PASSWORD=but_you_should_be_able_to_store_this_in_secrets
        # see docs for MONGO_INITDB_ROOT_PASSWORD_FILE at https://hub.docker.com/_/mongo
      deploy:
        replicas: 1
        placement:
          constraints:
            - node.hostname == your-mongo-server.com # just an example, probably use labels instead
        resources:
          limits:
            cpus: "4"
            memory: "8G"
    # your web application wouldn't need to be exposed to the outside world either
    # it could just receive the requests from a web server that ensures SSL/TLS
    web_application:
      image: hello-world # which you would replace with your actual app
      restart: "unless-stopped"
      environment:               
        - APP_CONFIG=variables_go_here_or_in_a_config_map
        - MONGO_HOST=mongodb_container
        - MONGO_PORT=27017
        - MONGO_USERNAME=either_the_username_above_or_a_non_root_one
        - MONGO_PASSWORD=though_the_non_root_needs_separate_creation
        # maybe with a helper container, like some people use dbmate for RDBMS
      deploy:
        replicas: 4
        placement:
          constraints:
            - node.hostname == your-application-server.com # just an example, probably use labels instead
        resources:
          limits:
            cpus: "2"
            memory: "2G"
    # finally, the actual web server is the ingress point, which you can self host
    # in the case of Kubernetes and other fancy tech you'd be able to use external solutions, probably
    web_server:
      image: caddy:alpine # or another web server, that would actually be exposed
      restart: "unless-stopped"
      ports:
        - 80:80
        - 443:443
        # though sometimes you need the long format for the above so client IPs resolve correctly
      volumes:
        - some_config_directory:/etc/caddy/Caddyfile
        - some_other_config_directory:/data
        - yet_another_config_directory:/config
      deploy:
        replicas: 4
        placement:
            constraints:
            - node.hostname == your-ingress-server.com # just an example, probably use labels instead
        resources:
          limits:
            cpus: "2"
            memory: "2G"
That's not to say that everyone should use Swarm or other orchestrators, but personally i find it a good approach, which allows defining some of the network topology in the same file as the application deployment, thus making the risk of human error slightly smaller.

Docker even has a page on networking, that provides information about the additional functionality that's available as well: https://docs.docker.com/network/

Of course, that's not to say that defense in depth and external solutions wouldn't perhaps be a better option.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#124
post #122

The real issue here is that the “backend” database engine was exposed on the public internet. It should have been a layer behind that with suitable protection, as mentioned a VPC with security group rules or equivalent firewall. Regardless of the docker foot gun this could have been a human fudge as well. Having done exactly this myself before with SQL server and Slammer I didn’t blame the default configuration, I bl…

The issue is that docker circumvents a platform security control without warning.

Docker on Ubuntu has a critical security vulnerability, with a track record of being exploited. That, clearly, is something that should be fixed ASAP.

If you have just a single layer, sure, blame yourself for poor security practices. Services should listen to unix domain sockets or internal networks only. They should have authentication in place. Machines should not have routable IP addresses. Machines should have externally applied firewalls. And so on. That doesn't mean the security layer that did fail shouldn't be fixed.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#126
post #122

The real issue here is that the “backend” database engine was exposed on the public internet. It should have been a layer behind that with suitable protection, as mentioned a VPC with security group rules or equivalent firewall. Regardless of the docker foot gun this could have been a human fudge as well. Having done exactly this myself before with SQL server and Slammer I didn’t blame the default configuration, I bl…

The issue is that docker circumvents a platform security control without warning. Docker on Ubuntu has a critical security vulnerability, with a track record of being exploited. That, clearly, is something that should be fixed ASAP. If you have just a single layer, sure, blame yourself for poor security practices. Services should listen to unix domain sockets or internal networks only. They should have authentication…

Correct. my point is that security is no good full stop if there is only one layer.

Even temporary rules and default states aren’t protected against until your automation is complete if your box is on the public internet. Or if you screw your automation up, the same happens.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#127
post #122

The real issue here is that the “backend” database engine was exposed on the public internet. It should have been a layer behind that with suitable protection, as mentioned a VPC with security group rules or equivalent firewall. Regardless of the docker foot gun this could have been a human fudge as well. Having done exactly this myself before with SQL server and Slammer I didn’t blame the default configuration, I bl…

The issue is that docker circumvents a platform security control without warning. Docker on Ubuntu has a critical security vulnerability, with a track record of being exploited. That, clearly, is something that should be fixed ASAP. If you have just a single layer, sure, blame yourself for poor security practices. Services should listen to unix domain sockets or internal networks only. They should have authentication…

Sorry if it sounds like a blame the victim (I'm probably doing it, yeah) but if you know all that, and even if you don't, having a DB server meant to be accessed only internally with a public IP is basically wrong under all circumstances. I mean, the docker "footgun" can be valid if you are running a single host with multiple services, some of them public and some of them private, and dockerizing the private ones gets them exposed to the Internet, even if you had a firewall rule to manage that. That's fine, let's blame Docker. But in this case I'm sorry but the Docker behavior just exposed a broken design.

I really hope they learn from all this the right lesson, which is not to just blame docker, but to carefully think whether you really need to use the Internet as a mean of internal communication for your services.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#128

Earlier quoted context omitted.

I suppose my premise is that engineers often delude themselves into thinking that they know it well enough to implement docker when they dont. Id much rather prefer that the regular developers focus on the coding and leave those decisions to actual devops folks who can focus on getting this stuff right. But the simplicity of dockerfile lulls many into thinking if they run something with it it's production ready.

Doesn't that run counter to what devops actually should be? I know that in practice, this split is what happens at companies, but we should work to close that rift, not widen it.

Whatever devops is supposed to be, it shouldn't involve relying on a tool whose security characteristics the team does not fully understand. If requiring such knowledge results in widening a pre-existing rift, so be it.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#129
post #75
post #8

Earlier quoted context omitted.

I don't know why this is being downvoted. Multiple overlapping layers of security would have given newsblur a backup in case of accidental "footguns". Unauthenticated mongodb instances are a pretty common problem - it's why a "script kiddie" was so successful.

> I don't know why this is being downvoted. The core message ("using auth on MongoDB would have prevented this, it's always a good idea to add password auth just in case") is perfectly reasonable; we can all learn from this, and it's perfectly fine to point out such things. But the way it was phrased was absolutely not okay. People make mistakes all the time and they are not "incompetent". This is the classic "I am v…

Having unauthenticated private services on directly-internet-connected hosts, regardless of the state of the host based firewall, is a mistake that a competent sysadmin does not make. (Then again, so is running MongoDB.)

It's not an insult to call someone incompetent.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database

#130

Holy shit, I've been running open ports for a year with docker, completely ignoring my firewall! This has to be fixed ASAP.

One of the best things one can do with Docker, for security, but at the expense of convenience is to configure it to disable the docker-proxy and IPTables manipulation and manage IPTables yourself.

It'll be slow at first, and you'll need new rules each time you bring a container up but you'll have far more control.

We switched to this approach when we had an IP dual-stack issue with the proxy and not only do we know it's secure because we wrote the rules, we get better performance without the userland proxy and more control over routing.

It's important to understand that because your containers sit on a bridged network, INPUT rules do nothing to stop incoming connections, it's all about the FORWARD table.

Post reply on HN