Post Mortem on Salt Incident
blog.algolia.com
Post Mortem on Salt Incident
1–10 of 70 posts
Re: Post Mortem on Salt Incident
#2The intruders had root access to every server in a salt deployment for who knows how long and yet everyone is claiming there's no evidence that any data or secrets (customer's or otherwise) were exfiltrated from the network. This is a very dangerous assumption. Nobody has any idea what was run on the servers since it seems that once the initial attack script was deployed it downloaded and executed new scripts every 60s and then removed themselves. Pretty standard C&C ops. It may have started as a mining operation, but that doesn't mean it was the only thing it was doing.
Re: Post Mortem on Salt Incident
#3The intruders had root access to every server in a salt deployment for who knows how long and yet everyone is claiming there's no evidence that any data or secrets (customer's or otherwise) were exfiltrated from the network. This is a very dangerous assumption. Nobody has any idea what was run on the servers since it seems that once the initial attack script was deployed it downloaded and executed new scripts every 6…
I agree. I would like to seem more details of how they determined it was only crypto mining. Finding only mining scripts in your logs doesn't mean they were not running other code once they had root.
Re: Post Mortem on Salt Incident
#4“We’ve secured the impacted SaltStack service by updating it and adding additional IP filtering, allowing only our servers to connect to it.”
So this means they had Salt master ports publicly accessible? Why would anyone have salt ports open/exposed to public/internet?
Re: Post Mortem on Salt Incident
#5The intruders had root access to every server in a salt deployment for who knows how long and yet everyone is claiming there's no evidence that any data or secrets (customer's or otherwise) were exfiltrated from the network. This is a very dangerous assumption. Nobody has any idea what was run on the servers since it seems that once the initial attack script was deployed it downloaded and executed new scripts every 6…
I agree. I would like to seem more details of how they determined it was only crypto mining. Finding only mining scripts in your logs doesn't mean they were not running other code once they had root.
It seems bizarre to me that a crypto miner got in. It wouldn't make much money on regular CPUs, and the high processor usage would immediately draw attention. So it looks like a low-effort botnet, which is embarrassing to get pwned by.
(The coin mining could be a cover like you mention, but it seems unlikely since it naturally draws attention.)
Re: Post Mortem on Salt Incident
#6This whole salt-stack incident could've been handled a lot better by salt themselves:
- the notification was a week ago to a small mailing list, which is tucked away on their site
- no notification to the registry to when you go to download salt (at least I never received an email, but still get plenty of marketing spam)
- no posts on social media as far as I can tell, I couldn't find a tweet, anything on reddit, or anything on hn.
- they only blogged about it on their official site yesterday, way after damage had been done
- one week's notice between the initial announcement and the patch coming out. The patch being released is basically a disclosure of the vulnerability
- the patch was released late Thursday early Friday depending on your timezone, giving attackers the weekend head start
- the official salt docker images were only patched yesterday
- You can't get a patch for older versions without filling out a form and supplying details
- Ubuntu and other repositories are still vulnerable
Re: Post Mortem on Salt Incident
#7This whole salt-stack incident could've been handled a lot better by salt themselves: - the notification was a week ago to a small mailing list, which is tucked away on their site - no notification to the registry to when you go to download salt (at least I never received an email, but still get plenty of marketing spam) - no posts on social media as far as I can tell, I couldn't find a tweet, anything on reddit, or…
+1, however, from what I read, the vulnerability can only be exploited if the attacker has network access to the salt masters port, which should never occur. The people that got compromised had Salt exposed to the Internet, which is obviously ridiculous.
Not trying to downplay the critical nature of the vulnerability but the ones that were compromised by this issue have deeper security issues to deal with.
Re: Post Mortem on Salt Incident
#8Earlier quoted context omitted.
I agree. I would like to seem more details of how they determined it was only crypto mining. Finding only mining scripts in your logs doesn't mean they were not running other code once they had root.
It seems bizarre to me that a crypto miner got in. It wouldn't make much money on regular CPUs, and the high processor usage would immediately draw attention. So it looks like a low-effort botnet, which is embarrassing to get pwned by. (The coin mining could be a cover like you mention, but it seems unlikely since it naturally draws attention.)
> It wouldn't make much money on regular CPUs
Not true; some PoWs such as Random-X are designed to be most efficient CPUs.
Re: Post Mortem on Salt Incident
#9This whole salt-stack incident could've been handled a lot better by salt themselves: - the notification was a week ago to a small mailing list, which is tucked away on their site - no notification to the registry to when you go to download salt (at least I never received an email, but still get plenty of marketing spam) - no posts on social media as far as I can tell, I couldn't find a tweet, anything on reddit, or…
+1, however, from what I read, the vulnerability can only be exploited if the attacker has network access to the salt masters port, which should never occur. The people that got compromised had Salt exposed to the Internet, which is obviously ridiculous. Not trying to downplay the critical nature of the vulnerability but the ones that were compromised by this issue have deeper security issues to deal with.
If you look at their current `hardening` document it still has pretty unclear language about what is acceptable and what isn't.
> Use a hardened bastion server or a VPN to restrict direct access to the Salt master from the internet
Is this SSH access or is this access to the salt master from minions? Or just access in general?
Re: Post Mortem on Salt Incident
#10“We’ve secured the impacted SaltStack service by updating it and adding additional IP filtering, allowing only our servers to connect to it.” So this means they had Salt master ports publicly accessible? Why would anyone have salt ports open/exposed to public/internet?
Yeah, that jumped out for me too. I'm guessing that they didn't want to deploy some sort of private network layer.