Live data from Hacker News

Post Mortem on Salt Incident

blog.algolia.com

11–20 of 70 posts

Re: Post Mortem on Salt Incident

#11
post #2

The intruders had root access to every server in a salt deployment for who knows how long and yet everyone is claiming there's no evidence that any data or secrets (customer's or otherwise) were exfiltrated from the network. This is a very dangerous assumption. Nobody has any idea what was run on the servers since it seems that once the initial attack script was deployed it downloaded and executed new scripts every 6…

Nuke the entire site from orbit. It's the only way to be sure

Re: Post Mortem on Salt Incident

#12
post #9
post #7

Earlier quoted context omitted.

+1, however, from what I read, the vulnerability can only be exploited if the attacker has network access to the salt masters port, which should never occur. The people that got compromised had Salt exposed to the Internet, which is obviously ridiculous. Not trying to downplay the critical nature of the vulnerability but the ones that were compromised by this issue have deeper security issues to deal with.

If you look at their current `hardening` document it still has pretty unclear language about what is acceptable and what isn't. > Use a hardened bastion server or a VPN to restrict direct access to the Salt master from the internet Is this SSH access or is this access to the salt master from minions? Or just access in general?

Apparently it includes minion-master interaction. If that’s to be “hardened” over SSH, what’s the point of all the salt keys?

Re: Post Mortem on Salt Incident

#13
post #12
post #9

Earlier quoted context omitted.

If you look at their current `hardening` document it still has pretty unclear language about what is acceptable and what isn't. > Use a hardened bastion server or a VPN to restrict direct access to the Salt master from the internet Is this SSH access or is this access to the salt master from minions? Or just access in general?

Apparently it includes minion-master interaction. If that’s to be “hardened” over SSH, what’s the point of all the salt keys?

[deleted]

Re: Post Mortem on Salt Incident

#14
post #5
post #3

Earlier quoted context omitted.

I agree. I would like to seem more details of how they determined it was only crypto mining. Finding only mining scripts in your logs doesn't mean they were not running other code once they had root.

It seems bizarre to me that a crypto miner got in. It wouldn't make much money on regular CPUs, and the high processor usage would immediately draw attention. So it looks like a low-effort botnet, which is embarrassing to get pwned by. (The coin mining could be a cover like you mention, but it seems unlikely since it naturally draws attention.)

I once worked at a place where a minor piece of cloud infra got exploited. All the attacker did was run a monero miner on it.

Re: Post Mortem on Salt Incident

#16
post #10
post #4

“We’ve secured the impacted SaltStack service by updating it and adding additional IP filtering, allowing only our servers to connect to it.” So this means they had Salt master ports publicly accessible? Why would anyone have salt ports open/exposed to public/internet?

Yeah, that jumped out for me too. I'm guessing that they didn't want to deploy some sort of private network layer.

That's easier said then done. There are no simple cross cloud provider solutions for a private networking other then ZeroTier, which has it's own issues.

Re: Post Mortem on Salt Incident

#17
Trusting a central control server is the fundamental mistake here.

It creates a very high value target that is difficult to secure.

I prefer a model where the management commands are signed at a management workstation and those commands are pushed by the server and authenticated at the managed node against a security policy.

Re: Post Mortem on Salt Incident

#20
post #16
post #10

Earlier quoted context omitted.

Yeah, that jumped out for me too. I'm guessing that they didn't want to deploy some sort of private network layer.

That's easier said then done. There are no simple cross cloud provider solutions for a private networking other then ZeroTier, which has it's own issues.

As a hobbyist, I might use tinc or PeerVPN. Or Tor plus OnionCat, with restrictive ip6tables rules. I've used that for a private Docker repository.

But those are probably not secure enough. Or too much hassle to setup.

Post reply on HN