Live data from Hacker News

I got hacked: My Hetzner server started mining Monero

blog.jakesaunders.dev

281–290 of 422 posts

Re: I got hacked: My Hetzner server started mining Monero

#281
post #279
post #231

Earlier quoted context omitted.

Yup the regular "8080:8080" bind resulted in a ransom note in my database on day 1. Bound it to localhost only now.

I had the same experience (postgres/postgres on default port). It took me a few days to find out, because the affected database was periodically re-built from another source. I just noticed that for some periods the queries failed until the next rebuild.

Yea plenty of bots scouting the internet for these kinda vulnerabilities, good learning experience, wont happen again :D

Re: I got hacked: My Hetzner server started mining Monero

#282
post #20

Earlier quoted context omitted.

There would be, but a lot of docker containers are misconfigured or unnecessarily privileged, allowing for escape. Also, if you've been compromised, you may have a rootkit that hides itself from the filesystem, so you can't be sure of a file's existence through a simple `ls` or `stat`.

> but a lot of docker containers are misconfigured or unnecessarily privileged, allowing for escape Honestly, citation needed. Very rare unless you're literally giving the container access to write to /usr/bin or other binaries the host is running, to reconfigure your entire /etc, access to sockets like docker's, or some other insane level of over reach I doubt even the least educated docker user would do. While of c…

Half the vendor software I come across asks you to mount devices from the host, add capabilities or run the container in privileged mode because their outsourced lowest bidder developers barely even know what a container is. I doubt even the smallest minority of their customers protest against this because apparently the place I work at is always the first one to have a problem with it.

Re: I got hacked: My Hetzner server started mining Monero

#283
After reading some comments: this probably goes without saying, but one should be very careful what to expose to the internet. Sounds like the analytics-service maybe could have been available only over VPN (or similar, like mTLS etc.)

And for basic web sites, it's much better if it requires no back-end.

Every service exposed increases risk and requires additional vigilance to maintain. Which means more effort.

Re: I got hacked: My Hetzner server started mining Monero

#284
post #273

Earlier quoted context omitted.

Hetzner has a free firewall service outside of your machine. You can use that as the first line of defence.

The problem with Hetzner's firewall service is it nukes network performance especially on ipv6.

It also killed my docker networking, so portainer stopped working.

Re: I got hacked: My Hetzner server started mining Monero

#285

$ sudo ufw default deny incoming $ sudo ufw default allow outgoing $ sudo ufw allow ssh $ sudo ufw allow 80/tcp $ sudo ufw allow 443/tcp $ sudo ufw enable As a user of iptables this order makes me anxious. I used to cut myself out from the server many times because first blocking then adding exceptions. I can see that this is different here as the last command commits the rules...

I had this one too: I first denied all incoming requests and was about to allow SSH, but my SSH connection dropped :) Fortunately, I was able to restore the VM with the provider's VM console.

Re: I got hacked: My Hetzner server started mining Monero

#287

Was dad notified of the security breach? If not he may want to consider switching hosting providers. Dad deserves a proper LLM-free post mortem.

Hahaha, I did tell him this afternoon. This is the bloke who has the same password for all his banking apps despite me buying him 1password though. The imminent threat from RCE's just didn't land.

Careful, HN isn't your average IRC channel.

Re: I got hacked: My Hetzner server started mining Monero

#288
post #155
post #106

> I also enabled UFW (which I should have done ages ago) I disrecommend UFW. firewalld is a much better pick in current year and will not grow unmaintainable the way UFW rules can. firewall-cmd --persistent --set-default-zone=block firewall-cmd --persistent --zone=block --add-service=ssh firewall-cmd --persistent --zone=block --add-service=https firewall-cmd --persistent --zone=block --add-port=80/tcp firewall-cmd --…

it doesn't matter what netfilter frontend you use if you allow outbound connections from any binary. In order to stop these attacks, restrict outbound connections from unknown / not allowed binaries. This kind of malware in particular requires outbound connections to the mining pools. Others downloads scripts or binaries from remote servers, or try to communicate with their c2c servers. On the other hand, removing ex…

Wasn’t there that npm malware thing a while ago that trashed your home folder if it couldn’t phone home?

Re: I got hacked: My Hetzner server started mining Monero

#289
post #278
post #216

Earlier quoted context omitted.

If you can, do not expose ports like this 8080:8080, but do this "192.168.0.1:8080:8080" so its bound to a private IP. Then use any old method expose only what you want to the world. In my own use I have 10.0.10.11 on the vm that I host docker stuff. It doesn't even have its own public IP meaning I could actually expose to 0.0.0.0 if I wanted to but things might change in the future so it's a precaution. That IP is o…

It's really a trap. I'm surprised they never changed the default to 127.0.0.1 instead of 0.0.0.0. So you would need to explicitly specify it, if you want to bind to all interfaces.

The reason is convenience. There would be a lot more friction if they didn't do it like this for everything other than local development.

Docker also has more traps and not quite as obvious as this. For example, it can change the private IP block its using without telling you. I got hit by this once due to a clash with a private block I was using for some other purpose. There's a way to fix it in the config but it won't affect already created containers.

By the way. While we're here. A public service announcement. You probably do NOT need the userland-proxy and can disable it.

/etc/docker/daemon.json

{ "userland-proxy": false }

Re: I got hacked: My Hetzner server started mining Monero

#290
post #63

Just a note - you can very much limit cpu usage on the docker containers by setting --cpus="0.5" (or cpus:0.5 in docker compose) if you expect it to be a very lightweight container, this isolation can help prevent one roudy container from hitting the rest of the system regardless of whether it's crypto-mining malware, a ddos attempt or a misbehaving service/software.

While this is a good idea I wonder if doing this could allow the intrusion to go undetected for longer - how many people/monitoring systems would notice a small increase in CPU usage compared to all CPUs being maxed out.
Post reply on HN