Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

blog.newsblur.com

Post reply on HN
Print view View original

91–100 of 117 posts

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#91

Post by cpuguy83 »

Tijdreiziger wrote: post #44
Temporary_31337 wrote: post #28

Earlier quoted context omitted.

If we accept that this is the default state of the Internet (insecure) then I think it is a correct assumption and forces everyone to think twice before exposing anything to the Internet by default. Here the default Docker behaviour was very much at fault but also the assumption that a Linux level firewall is good for Docker. We have docker deployments that are completely hidden behind NAT in AWS and such things are…

I have run into this firewall problem too (not in any sort of production environment, thankfully). The problem is that your mental model of a firewall tends to be that it sits 'in front of' all applications on your machine. Docker breaks this assumption by implicitly inserting rules into iptables. Worse still, if you then inspect the state of the firewall (using UFW), it does not show you any of the rules that Docker…

Docker is not implicitly doing anything. The user requested for the port to get published and so docker published it.

As nice as it would be for Docker to be more configurable in terms of where rules get inserted, that would not have helped here because the user already misconfigured things (should be using docker networks for db access). Technically docker even has a chain that you can throw rules into that will get hit before docker's rules... so it's there just very manual (instead of, say, having a ufw mode).

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#92

Post by [deleted] »

jacquesm wrote: post #9

It's a pity that the chances of nailing the perps is so low. Obviously docker and the person that put this together share some of the blame but: the original internet would have never gotten off the ground if it wasn't for people cooperating with each other rather than to try to tear things down all the time. And with the chances of your average script kiddie/hacker/idiot getting caught being lower than a typical bik…

[deleted]

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#93

Post by ilyt »

> Turns out the ufw firewall I enabled and diligently kept on a strict allowlist with only my internal servers didn’t work on a new server because of Docker.

OOF. Those can be nasty. Many of the tools "helpfully" inject jumps to their chains at the very front of the table, libvirt example:

    Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination         
    9105K 4151M LIBVIRT_INP  all  --  *      *       0.0.0.0/0            0.0.0.0/0        

    Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination         
        0     0 LIBVIRT_FWX  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
        0     0 LIBVIRT_FWI  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
        0     0 LIBVIRT_FWO  all  --  *      *       0.0.0.0/0            0.0.0.0/0

    Chain OUTPUT (policy ACCEPT 229K packets, 39M bytes)
     pkts bytes target     prot opt in     out     source               destination         
    6893K 1628M LIBVIRT_OUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0
So if you relied on your firewall blocking access and didn't test it it's easy to be bamboozled.

The common approach is to allow other tools only manage their own chains and call those chains from your but not that many tools allow for that and want to take over your firewall.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#94

Post by Tijdreiziger »

cpuguy83 wrote: post #91
Tijdreiziger wrote: post #44

Earlier quoted context omitted.

I have run into this firewall problem too (not in any sort of production environment, thankfully). The problem is that your mental model of a firewall tends to be that it sits 'in front of' all applications on your machine. Docker breaks this assumption by implicitly inserting rules into iptables. Worse still, if you then inspect the state of the firewall (using UFW), it does not show you any of the rules that Docker…

Docker is not implicitly doing anything. The user requested for the port to get published and so docker published it. As nice as it would be for Docker to be more configurable in terms of where rules get inserted, that would not have helped here because the user already misconfigured things (should be using docker networks for db access). Technically docker even has a chain that you can throw rules into that will get…

> The user requested for the port to get published and so docker published it.

Yes, which runs counter to how every other application works. If you ask, say, sshd to listen on port 22, your firewall will still sit in front of it, unless you explicitly poke a hole in it.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#95

Post by ary »

lambdasquirrel wrote: post #80
ary wrote: post #73

Personally I blame the irritation and cost of setting up private subnets that can access the Internet via NAT in AWS VPCs in addition to Docker's interactions with UFW. Private VPC subnets with NAT Gateways are both unacceptably costly and complicated to setup (source: recently dealt with the combination of NAT Gateways + the Network Firewall service in AWS). Throw in the need for VPC endpoints for many workloads and…

I’m not sure I’m understanding this entirely. I thought the way you should set it up is have e.g. your nginx / istio gateways exposed to the public internet on EIPs. You shouldn’t have your databases exposed to the world, or the hardware running your backend services directly exposed either. Also, security groups. Yes, if you have to use NAT gateways for serving traffic, those are very expensive.

What I'm noting is in line with your thinking.

Whatever ways AWS marketing might encourage people to do things via their "Well Architected" series of publications, the reality is that the path of least resistance with VPC networking is what the linked article describes: everything on public subnets (with maybe security groups or network ACLs to restrict it somewhat).

Splitting infrastructure into public/private subnets is very tedious, and can at times be difficult to debug. In some of my most recent work it took some time to understand why and how the components needed to be linked to make a combination of an Internet Gateway, Network Firewall, and NAT Gateway work together. Combined with the use of multiple private subnets residing in different AZs meant testing permutations and fussing with the "reachability" analyzer. It is this kind of thing that makes people go with the first thing that works so that they can move on to getting productive work done.

Typically one wouldn't serve traffic via a NAT Gateway, and that's true for our case as well. The problem we ran into was not properly configuring VPC endpoints for a workload that heavily interacts with S3 and running up a tiny, but still not ideal, bill for network utilization over the NAT Gateway instead. It's not obvious to may people that interacting with AWS APIs happens over the public Internet even from within AWS infrastructure. The cost of ongoing cost NAT Gateways versus the free per-VPC Internet Gateway feels like it disincentivizes starting with a more secure setup.

Ultimately what I'm trying to say is that it's somewhat negligent of AWS to not make an almost universal requirement of modern web and SaaS cloud networking be simple and cheap to implement.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#96

Post by c7DJTLrn »

mannykannot wrote: post #56

"When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world." Yet another reminder that the most important ability in systems engineering is good judgement.

The poor judgement here is not using an appliance firewall in addition to a system firewall. Exposing an instance directly to the public Internet is just not a good move.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#97

Post by ilyt »

cpuguy83 wrote: post #91
Tijdreiziger wrote: post #44

Earlier quoted context omitted.

I have run into this firewall problem too (not in any sort of production environment, thankfully). The problem is that your mental model of a firewall tends to be that it sits 'in front of' all applications on your machine. Docker breaks this assumption by implicitly inserting rules into iptables. Worse still, if you then inspect the state of the firewall (using UFW), it does not show you any of the rules that Docker…

Docker is not implicitly doing anything. The user requested for the port to get published and so docker published it. As nice as it would be for Docker to be more configurable in terms of where rules get inserted, that would not have helped here because the user already misconfigured things (should be using docker networks for db access). Technically docker even has a chain that you can throw rules into that will get…

Docker is implicitly overriding existing rules by putting their own ones in front. The least it should do is to put a warning "hey, there are other rules here,I'm adding my own, please verify.

It should also have option (I only found "complete on/complete off" one) to "just" create its DOCKER* chains and leave the jumping to them to the user

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#98

Post by ilyt »

c7DJTLrn wrote: post #96
mannykannot wrote: post #56

"When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world." Yet another reminder that the most important ability in systems engineering is good judgement.

The poor judgement here is not using an appliance firewall in addition to a system firewall. Exposing an instance directly to the public Internet is just not a good move.

And not having password. And not even trying to nmap the service after configuration to verify if firewall is doing what is intended.

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#99

Post by ilyt »

djm_ wrote: post #20

Fantastic write-up. This seems to me like a combination of multiple foot-guns, first being the Docker one - followed by the fact Mongo was not configured to authenticate the connection. Heroku by default run PostgreSQL open to the world (which is problematic for other reasons) but they get away with it by relying on PG's decent authentication. My default is to prefer to build systems with multiple layers of security,…

If it is on same machine you can even get away with using PostgreSQL via socket, eliminating all of the auth problems as now app user = db login

Re: A Docker footgun led to a vandal deleting NewsBlur's MongoDB database (2021)

#100

Post by cpuguy83 »

Tijdreiziger wrote: post #94
cpuguy83 wrote: post #91

Earlier quoted context omitted.

Docker is not implicitly doing anything. The user requested for the port to get published and so docker published it. As nice as it would be for Docker to be more configurable in terms of where rules get inserted, that would not have helped here because the user already misconfigured things (should be using docker networks for db access). Technically docker even has a chain that you can throw rules into that will get…

> The user requested for the port to get published and so docker published it. Yes, which runs counter to how every other application works. If you ask, say, sshd to listen on port 22, your firewall will still sit in front of it, unless you explicitly poke a hole in it.

Listening on a port and opening a port are different things. `-p` is for opening ports. Docker is not listening on behalf of the application. It is routing traffic to the application.
Post reply on HN