Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I got hacked: My Hetzner server started mining Monero

blog.jakesaunders.dev

Post reply on HN
Print view View original

191–200 of 422 posts

Re: I got hacked: My Hetzner server started mining Monero

#191

Post by danparsonson »

tete wrote: post #139
danparsonson wrote: post #92

No firewall! Wow that's brave. Hetzner will let you configure one that runs outside of the box so you might want to add that too, as part of your defense in depth - that will cover you if you make a mistake with ufw. Personally I keep SSH firewalled only to my home address in this way; if I'm out and about and need access, I can just log into Hetzner's website and change it temporarily.

Firewalls in the majority of cases don't get you much. Yes it's a last line of defense if you do something really stupid and don't even know where or what you configure your services to listen on, but if you don't the difference between running firewalls and not is minuscule. There are way more important things like actually knowing that you are running software with widely known RCE that don't even use established m…

The author mentioned they had other services exposed to the internet (Postgres, RabbitMQ) which increases their attack surface area. There may be vulnerabilities or misconfigurations in those services for example.

Good security is layered.

Re: I got hacked: My Hetzner server started mining Monero

#192

Post by cyphar »

miladyincontrol wrote: post #128
ronsor wrote: post #20

Earlier quoted context omitted.

There would be, but a lot of docker containers are misconfigured or unnecessarily privileged, allowing for escape. Also, if you've been compromised, you may have a rootkit that hides itself from the filesystem, so you can't be sure of a file's existence through a simple `ls` or `stat`.

> but a lot of docker containers are misconfigured or unnecessarily privileged, allowing for escape Honestly, citation needed. Very rare unless you're literally giving the container access to write to /usr/bin or other binaries the host is running, to reconfigure your entire /etc, access to sockets like docker's, or some other insane level of over reach I doubt even the least educated docker user would do. While of c…

As a maintainer of runc (the runtime Docker uses), if you aren't using user namespaces (which is the case for the vast majority of users) I would consider your setup insecure.

And a shocking number of tutorials recommend bind-mounting docker.sock into the container without any warning (some even tell you to mount it "ro" -- which is even funnier since that does nothing). I have a HN comment from ~8 years ago complaining about this.

Re: I got hacked: My Hetzner server started mining Monero

#193

Post by figassis »

danparsonson wrote: post #92

No firewall! Wow that's brave. Hetzner will let you configure one that runs outside of the box so you might want to add that too, as part of your defense in depth - that will cover you if you make a mistake with ufw. Personally I keep SSH firewalled only to my home address in this way; if I'm out and about and need access, I can just log into Hetzner's website and change it temporarily.

Yup. All my servers are behind Tailscale. The only thing I expose is a load balancer that routes tcp (email) and http. That balancer is running docker, fully firewalled (incl docker bypasses). Every server is behind herzner’s firewall in addition to the internal firewall.

App servers run docker, with images that run a single executable (no os, no shell), strict cpu and memory limits. Most of my apps only require very limited temporary storage so usually no need to mount anything. So good luck executing anything in there.

I used, way back in the day, to run Wordpress sites. Would get hacked monthly every possible way. Learned so much, including the fact that often your app is your threat. With Wordpress, every plugin is a vector. Also the ability to easily hop into an instance and rewrite running code (looking at you scripting languages incl JS) is terrible. This motivated my move to Go. The code I compiled is what will run. Period.

Re: I got hacked: My Hetzner server started mining Monero

#194

Post by kopirgan »

scottyeager wrote: post #189
kopirgan wrote: post #171

Only lesson seems to be use ufw! (or equivalent)

As others have mentioned, a firewall might have been useful in restricting outbound connections to limit the usefulness of the machine to the hacker after the breach. An inbound firewall can only help protect services that aren't meant to be reachable on the public internet. This service was exposed to the internet intentionally so a firewall wouldn't have helped avoid the breach. The lesson to me is that keeping up…

Yes thanks for the clarification.

Re: I got hacked: My Hetzner server started mining Monero

#195

Post by 3np »

3abiton wrote: post #181
gus_ wrote: post #155

Earlier quoted context omitted.

it doesn't matter what netfilter frontend you use if you allow outbound connections from any binary. In order to stop these attacks, restrict outbound connections from unknown / not allowed binaries. This kind of malware in particular requires outbound connections to the mining pools. Others downloads scripts or binaries from remote servers, or try to communicate with their c2c servers. On the other hand, removing ex…

Is there an automated way of doing this?

Two paths:

- Configuration management (ansible, salt, chef, puppet)

- Preconfigured images (NixOS, packer, Guix, atomic stuffs)

For a one-off: pssh

Re: I got hacked: My Hetzner server started mining Monero

#196

Post by windexh8er »

TacticalCoder wrote: post #157
socalgal2 wrote: post #66

Earlier quoted context omitted.

that's a really good point .. but, I think 99% of docker users believe it is a a sandbox and treat it as such.

Not 99%. Many people run an hypervisor and then a VM just for Docker. Attacker now needs a Docker exploit and then a VM exploit before getting to the hypervisor (and, no, pwning the VM ain't the same as pwning the hypervisor).

Agreed - this is actually pretty common in the Proxmox realm of hosters. I segment container nodes using LXC, and in some specific cases I'll use a VM.

Not only does it allow me to partition the host for workloads but I also get security boundaries as well. While it may be a slight performance hit the segmentation also makes more logical sense in the way I view the workloads. Finally, it's trivial to template and script, so it's very low maintenance and allows for me to kill an LXC and just reprovision it if I need to make any significant changes. And I never need to migrate any data in this model (or very rarely).

Re: I got hacked: My Hetzner server started mining Monero

#197

Post by tgtweak »

fragmede wrote: post #89
tgtweak wrote: post #63

Just a note - you can very much limit cpu usage on the docker containers by setting --cpus="0.5" (or cpus:0.5 in docker compose) if you expect it to be a very lightweight container, this isolation can help prevent one roudy container from hitting the rest of the system regardless of whether it's crypto-mining malware, a ddos attempt or a misbehaving service/software.

The other thing to note is that docker is for the most part, stateless. So if you're running something that has to deal with questionable user input (images and video or more importantly PDFs), is to stick it on its own VM and then cycle the docker container every hour and the VM every 12, and then still be worried about it getting hacked and leaking secrets.

Most of this is mitigated by running docker in an LXC containers (like proxmox does) which grants a lot more isolation than docker on it's own - closer in nature to running separate VMs.

Re: I got hacked: My Hetzner server started mining Monero

#198

Post by 3np »

CGamesPlay wrote: post #183

I took issue with this paragraph of the article, on account of several pieces of misinformation, presumably courtesy of Claude hallucinations: > Here’s the test. If /tmp/.XIN-unix/javae exists on my host, I’m fucked. If it doesn’t exist, then what I’m seeing is just Docker’s default behavior of showing container processes in the host’s ps output, but they’re actually isolated. 1. Files of running programs can be dele…

> "Write your own Dockerfiles" is not useful security advice.

I actually think it is. It makes you more intimate with the application and how it runs, and can mitigate one particular supply-chain security vector.

Agreeing that the reasoning is confused but that particular advice is still good I think.

Re: I got hacked: My Hetzner server started mining Monero

#199

Post by seszett »

danparsonson wrote: post #191
tete wrote: post #139

Earlier quoted context omitted.

Firewalls in the majority of cases don't get you much. Yes it's a last line of defense if you do something really stupid and don't even know where or what you configure your services to listen on, but if you don't the difference between running firewalls and not is minuscule. There are way more important things like actually knowing that you are running software with widely known RCE that don't even use established m…

The author mentioned they had other services exposed to the internet (Postgres, RabbitMQ) which increases their attack surface area. There may be vulnerabilities or misconfigurations in those services for example. Good security is layered.

But if they have to be exposed then a firewall won't help, and if they don't have to be exposed to the internet then a firewall isn't needed either, just configure them not to listen on non-local interfaces.

Re: I got hacked: My Hetzner server started mining Monero

#200

Post by beeflet »

asdff wrote: post #156
tgtweak wrote: post #69

Earlier quoted context omitted.

Monero's proof of work (RandomX) is very asic-resistant and although it generates a very small amount of earnings, if you exploit a vulnerability like this with thousands or tens of thousands of nodes, it can add up (8 modern cores 24/7 on Monero would be in the 10-20c/day per node range). OPs Vps probably generated about $1 for those script kiddies.

So $40 a year? Does that imply all monero is mined like this because it's clearly not cost effective at all to mine legitimately?

I think so, but it is hard to say. Could be a lot of people with extra power (or stolen power), but their own equipment. I mine myself with waste solar power
Post reply on HN