Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I got hacked: My Hetzner server started mining Monero

blog.jakesaunders.dev

Post reply on HN
Print view View original

181–190 of 422 posts

Re: I got hacked: My Hetzner server started mining Monero

#181

Post by 3abiton »

gus_ wrote: post #155
3np wrote: post #106

> I also enabled UFW (which I should have done ages ago) I disrecommend UFW. firewalld is a much better pick in current year and will not grow unmaintainable the way UFW rules can. firewall-cmd --persistent --set-default-zone=block firewall-cmd --persistent --zone=block --add-service=ssh firewall-cmd --persistent --zone=block --add-service=https firewall-cmd --persistent --zone=block --add-port=80/tcp firewall-cmd --…

it doesn't matter what netfilter frontend you use if you allow outbound connections from any binary. In order to stop these attacks, restrict outbound connections from unknown / not allowed binaries. This kind of malware in particular requires outbound connections to the mining pools. Others downloads scripts or binaries from remote servers, or try to communicate with their c2c servers. On the other hand, removing ex…

Is there an automated way of doing this?

Re: I got hacked: My Hetzner server started mining Monero

#182

Post by 3abiton »

danparsonson wrote: post #92

No firewall! Wow that's brave. Hetzner will let you configure one that runs outside of the box so you might want to add that too, as part of your defense in depth - that will cover you if you make a mistake with ufw. Personally I keep SSH firewalled only to my home address in this way; if I'm out and about and need access, I can just log into Hetzner's website and change it temporarily.

Honestly fail2ban is amazing. I might doa write up on the countless of attempts on my servers.

Re: I got hacked: My Hetzner server started mining Monero

#183

Post by CGamesPlay »

I took issue with this paragraph of the article, on account of several pieces of misinformation, presumably courtesy of Claude hallucinations:

> Here’s the test. If /tmp/.XIN-unix/javae exists on my host, I’m fucked. If it doesn’t exist, then what I’m seeing is just Docker’s default behavior of showing container processes in the host’s ps output, but they’re actually isolated.

1. Files of running programs can be deleted while the program is running. If the program were trying to hide itself, it would have deleted /tmp/.XIN-unix/javae after it started. The nonexistence of the file is not a reliable source of information for confirming that the container was not escaped.

2. ps shows program-controlled command lines. Any program can change what gets displayed here, including the program name and arguments. If the program were trying to hide itself, it would change this to display `login -fp ubuntu` instead. This is not a reliable source of information for diagnosing problems.

It is good to verify the systemd units and crontab, and since this malware is so obvious, it probably isn't doing these two hiding methods, but information-stealing malware might not be detected by these methods alone.

Later, the article says "Write your own Dockerfiles" and gives one piece of useless advice (using USER root does not affect your container's security posture) and two pieces of good advice that don't have anything to do with writing your own Dockerfiles. "Write your own Dockerfiles" is not useful security advice.

Re: I got hacked: My Hetzner server started mining Monero

#184

Post by egberts1 »

denkmoon wrote: post #124
3np wrote: post #106

> I also enabled UFW (which I should have done ages ago) I disrecommend UFW. firewalld is a much better pick in current year and will not grow unmaintainable the way UFW rules can. firewall-cmd --persistent --set-default-zone=block firewall-cmd --persistent --zone=block --add-service=ssh firewall-cmd --persistent --zone=block --add-service=https firewall-cmd --persistent --zone=block --add-port=80/tcp firewall-cmd --…

I’ll just mention Foomuuri here. Its bit of a spiritual successor to shorewall and has firewalld emulation to work with tools compatible with firewalld

Foomuuri is ALMOST there.

I mean there are some payload over payload like GRE VPE/VXLAN/VLAN or IPSec that needs to be written in raw nft if using Foomuuni but it works!.

But I love the Shorewall approach and your configuration gracefully encapsulated Shorewall mechanic.

Disclaimer: I maintain vim-syntax-nftables syntax highlighter repo at Github.

Re: I got hacked: My Hetzner server started mining Monero

#185

Post by 3eb7988a1663 »

tracker1 wrote: post #100
tgtweak wrote: post #63

Just a note - you can very much limit cpu usage on the docker containers by setting --cpus="0.5" (or cpus:0.5 in docker compose) if you expect it to be a very lightweight container, this isolation can help prevent one roudy container from hitting the rest of the system regardless of whether it's crypto-mining malware, a ddos attempt or a misbehaving service/software.

Another is running containers in read-only mode, assuming they support this configuration... will minimize a lot of potential attack surface.

Never looked into this. I would expect the majority of images would fail in this configuration. Or am I unduly pessimistic?

Re: I got hacked: My Hetzner server started mining Monero

#186

Post by [deleted] »

d4mi3n wrote: post #37
V__ wrote: post #5

> The Reddit post I’d seen earlier? That guy got completely owned because his container was running as root. The malware could: [...] Is that the case, though? My understanding was, that even if I run a docker container as root and the container is 100% compromised, there still would need to be a vulnerability in docker for it to “attack” the host, or am I missing something?

While this is true, the general security stance on this is: Docker is not a security boundary. You should not treat it like one. It will only give you _process level_ isolation. If you want something with better security guarantees, you can use a full VM (KVM/QEMU), something like gVisor[1] to limit the attack surface of a containerized process, or something like Firecracker[2] which is designed for multi-tenancy. Th…

[deleted]

Re: I got hacked: My Hetzner server started mining Monero

#187

Post by drnick1 »

Nextgrid wrote: post #109
danparsonson wrote: post #105

Earlier quoted context omitted.

No you're right, I didn't mean the firewall would have saved them, but just as a general point of advice. And yes a second VPS running opnSense or similar makes a nice cheap proxy and then you can firewall off the main server completely. Although that wouldn't have saved them either - they'd still need to forward HTTP/S to the main box.

A firewall blocking outgoing connections (except those whitelisted through the proxy) would’ve likely prevented the download of the malware (as it’s usually done by using the RCE to call a curl/wget command rather than uploading the binary through the RCE) and/or its connection to the mining server.

In practice, this is basically impossible to implement. As a user behind a firewall you normally expect to be able to open connections with any remote host.

Re: I got hacked: My Hetzner server started mining Monero

#188

Post by figassis »

exceptione wrote: post #122
3np wrote: post #106

> I also enabled UFW (which I should have done ages ago) I disrecommend UFW. firewalld is a much better pick in current year and will not grow unmaintainable the way UFW rules can. firewall-cmd --persistent --set-default-zone=block firewall-cmd --persistent --zone=block --add-service=ssh firewall-cmd --persistent --zone=block --add-service=https firewall-cmd --persistent --zone=block --add-port=80/tcp firewall-cmd --…

> Specifically for docker it is a very common gotcha that the container runtime can and will bypass firewall rules and open ports anyway. Like I said in another comment, drop Docker, install podman.

In docker, simply clearly define the interface (ip) and port. It can be 0.0.0.0:80 for example. No bypass happens.

Re: I got hacked: My Hetzner server started mining Monero

#189

Post by scottyeager »

kopirgan wrote: post #171

Only lesson seems to be use ufw! (or equivalent)

As others have mentioned, a firewall might have been useful in restricting outbound connections to limit the usefulness of the machine to the hacker after the breach.

An inbound firewall can only help protect services that aren't meant to be reachable on the public internet. This service was exposed to the internet intentionally so a firewall wouldn't have helped avoid the breach.

The lesson to me is that keeping up with security updates helps prevent publicly exposed services from getting hacked.

Re: I got hacked: My Hetzner server started mining Monero

#190

Post by cyphar »

V__ wrote: post #5

> The Reddit post I’d seen earlier? That guy got completely owned because his container was running as root. The malware could: [...] Is that the case, though? My understanding was, that even if I run a docker container as root and the container is 100% compromised, there still would need to be a vulnerability in docker for it to “attack” the host, or am I missing something?

You really need to use user namespaces to get this kind of security protection -- running as root inside a container without user namespaces is not secure. Yes, breakouts often require some other bug or misconfiguration but the margin for error is non-existent (for instance, if you add CAP_SYS_PTRACE to your containers it is trivial to break out of them and container runtimes have no way of protecting against that). Almost all container breakouts in the past decade were blocked by user namespaces.

Unfortunately, user namespaces are still not the default configuration with Docker (even though the core issues that made using them painful have long since been resolved).

Post reply on HN