Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post Mortem on Salt Incident

blog.algolia.com

Post reply on HN
Print view View original

61–70 of 70 posts

Re: Post Mortem on Salt Incident

#61

Post by [deleted] »

trabant00 wrote: post #28
kureikain wrote: post #25

It's weird that these salt master are reach-able from internet and they can sleep well with it. Even with zero-trust network or beyondcorp idea, I still found one extra layer of protection a VPC give are so great. Few years ago, it has an issue with K8S API Server, and updating k8s isn't a walk in the park. I felt relax back then because we have everything inside VPC. You can use SSH or VPN to access service inside V…

I never understood the current trent to say VPN is a thing of the past. Redundancy in security layers is how you dont't get affected by every CVE out there. Imo this is THE lesson to learn from this story. Seondary: salt and ansible are not very mature yet.

[deleted]

Re: Post Mortem on Salt Incident

#62

Post by VWWHFSfQ »

Jedd wrote: post #57
VWWHFSfQ wrote: post #2

The intruders had root access to every server in a salt deployment for who knows how long and yet everyone is claiming there's no evidence that any data or secrets (customer's or otherwise) were exfiltrated from the network. This is a very dangerous assumption. Nobody has any idea what was run on the servers since it seems that once the initial attack script was deployed it downloaded and executed new scripts every 6…

> ... and yet everyone is claiming there's no evidence that any data or secrets (customer's or otherwise) were exfiltrated from the network. A number of people have carefully reviewed the payload that was deployed to servers, especially during what we're calling v1-v4 of the attack. (v5 onwards got more complex, but that wasn't until Monday (with variability for timezone). > Nobody has any idea what was run on the se…

I hope for everyone's sake that it was just a naive crypto mining operation. But given the length of time this vulnerability was available, and the extent of access it allowed, I just find it very hard to say with any certainty that we know everything that it was doing. Exploits like this get passed around in nefarious circles pretty regularly. One of the scripts I saw went to great lengths to eliminate competing crypto miners from the systems so they could run their own. That tells me there were multiple people (or groups) exploiting this in competition with each other.

You said the v5 of the attack got more sophisticated. How do we know there wasn't a "v0" that was even more sophisticated and innocuous? You can't trust the server logs. Firewall tables were flushed, SELinux was disabled. It's just really hard to say the full extent of damages.

Re: Post Mortem on Salt Incident

#63

Post by brianjlogan »

lrpublic wrote: post #17

Trusting a central control server is the fundamental mistake here. It creates a very high value target that is difficult to secure. I prefer a model where the management commands are signed at a management workstation and those commands are pushed by the server and authenticated at the managed node against a security policy.

What configuration management tools use this methodology?

Re: Post Mortem on Salt Incident

#64

Post by Shish2k »

mtam wrote: post #7
cetra3 wrote: post #6

This whole salt-stack incident could've been handled a lot better by salt themselves: - the notification was a week ago to a small mailing list, which is tucked away on their site - no notification to the registry to when you go to download salt (at least I never received an email, but still get plenty of marketing spam) - no posts on social media as far as I can tell, I couldn't find a tweet, anything on reddit, or…

+1, however, from what I read, the vulnerability can only be exploited if the attacker has network access to the salt masters port, which should never occur. The people that got compromised had Salt exposed to the Internet, which is obviously ridiculous. Not trying to downplay the critical nature of the vulnerability but the ones that were compromised by this issue have deeper security issues to deal with.

I was in this situation; I went with “salt master exposed to the internet” because it’s the only service on that box - if I’d wrapped it in a VPN, then I’m replacing one exposed service with a different exposed service, and VPNs aren’t immune to exploits either (plus an extra layer of configuration means an extra layer of things that can go wrong)

Re: Post Mortem on Salt Incident

#65

Post by mixedCase »

mike_d wrote: post #21
mtam wrote: post #7

Earlier quoted context omitted.

+1, however, from what I read, the vulnerability can only be exploited if the attacker has network access to the salt masters port, which should never occur. The people that got compromised had Salt exposed to the Internet, which is obviously ridiculous. Not trying to downplay the critical nature of the vulnerability but the ones that were compromised by this issue have deeper security issues to deal with.

> has network access to the salt masters port, which should never occur You seem to prescribe to the "hard shell soft gooey center" network security philosophy. Should people expose an Oracle server to the internet? Absolutely not. Does moving it behind a firewall change the fact that every mildly skilled exploit developer is sitting on an Oracle 0day? Absolutely not. People have legitimate reasons for exposing Salt…

> It's how I bootstrap random VMs and bare metal from the internet.

What's stopping you from bootstrapping a bastion server through a simpler script and then bootstrapping with Salt from there?

Re: Post Mortem on Salt Incident

#66

Post by bawolff »

aneutron wrote: post #46
bawolff wrote: post #30

Earlier quoted context omitted.

> one week's notice between the initial announcement and the patch coming out. The patch being released is basically a disclosure of the vulnerability While your other points may be valid, one week should be plenty of time between announcement and patch. Any longer and i would call the time table problematic.

You have clearly never worked at a large enough OLD corporation. One week is nothing compared to what it would take to upgrade your configuration management system.

That sounds like an old corporation problem. The world should not pay the price of old corporations inflexibility.

If someone hacks your system you certainly wont have a week to respond. The longer a vendor sits on a vuln, the more likely it is to leak or to be rediscovered by a malicious party.

Re: Post Mortem on Salt Incident

#67

Post by lrpublic »

brianjlogan wrote: post #63
lrpublic wrote: post #17

Trusting a central control server is the fundamental mistake here. It creates a very high value target that is difficult to secure. I prefer a model where the management commands are signed at a management workstation and those commands are pushed by the server and authenticated at the managed node against a security policy.

What configuration management tools use this methodology?

A couple that I’ve built - they are not commercially available.

I’d consider open sourcing something based on them if there’s sufficient interest.

Perhaps as an integration for one of the major players.

Re: Post Mortem on Salt Incident

#68

Post by Jedd »

VWWHFSfQ wrote: post #62
Jedd wrote: post #57

Earlier quoted context omitted.

> ... and yet everyone is claiming there's no evidence that any data or secrets (customer's or otherwise) were exfiltrated from the network. A number of people have carefully reviewed the payload that was deployed to servers, especially during what we're calling v1-v4 of the attack. (v5 onwards got more complex, but that wasn't until Monday (with variability for timezone). > Nobody has any idea what was run on the se…

I hope for everyone's sake that it was just a naive crypto mining operation. But given the length of time this vulnerability was available, and the extent of access it allowed, I just find it very hard to say with any certainty that we know everything that it was doing. Exploits like this get passed around in nefarious circles pretty regularly. One of the scripts I saw went to great lengths to eliminate competing cry…

You're absolutely right that we can't be 100% confident, and best practice dictates a full rebuild from known sources, as usual after IOCs especially of this magnitude.

However, the number of public and non-patched salt servers might be considered a sufficiently small volume for bad actors to have investigated, who can say why it took so long to see genuinely malign attacks.

> One of the scripts I saw went to great lengths to eliminate competing crypto miners from the systems so they could run their own. That tells me there were multiple people (or groups) exploiting this in competition with each other.

It wasn't very sophisticated - just a series of kill statements. This tells me that the author of that script picked up an existing script that's probably been around for years and adjusted it to their needs.

The script also tried to kill confluence, amongst a handful of other large, relatively rare applications, which further suggests this was old fashioned copy-pasting by some non-sophisticated script kiddies ... or someone just wanting to do a PSA and draw attention to this exploit, and making a few BTC for their troubles. Who can say.

We don't know there wasn't a 'v0' - but we're fairly confident. Unless it was disabled as soon as 'v1' popped up, you'd expect honeypot systems to identify non-benign variants - and honeypot systems were identifying modest, reversible changes and nothing in the way of data exfiltration.

By Tuesday or Wednesday of this week I expect there were more (and worse) exploits than could be tracked, though, and some people are really going to suffer as a result.

Re: Post Mortem on Salt Incident

#69

Post by mike_d »

empath75 wrote: post #47
mike_d wrote: post #21

Earlier quoted context omitted.

> has network access to the salt masters port, which should never occur You seem to prescribe to the "hard shell soft gooey center" network security philosophy. Should people expose an Oracle server to the internet? Absolutely not. Does moving it behind a firewall change the fact that every mildly skilled exploit developer is sitting on an Oracle 0day? Absolutely not. People have legitimate reasons for exposing Salt…

> People have legitimate reasons for exposing Salt to the internet. I do. It's how I bootstrap random VMs and bare metal from the internet. I question that that is a legitimate reason to expose it to the internet. Defense in depth is a thing and putting the keys to the kingdom at layer 0 doesn’t seem wise even if a vpn or bastion doesn’t offer perfect protection.

Read the sentence after the ones you quoted. The internet connected salt master is used to provision accepted hosts in to the tunneled (VPN) network where the real master lives.

Re: Post Mortem on Salt Incident

#70

Post by mike_d »

mixedCase wrote: post #65
mike_d wrote: post #21

Earlier quoted context omitted.

> has network access to the salt masters port, which should never occur You seem to prescribe to the "hard shell soft gooey center" network security philosophy. Should people expose an Oracle server to the internet? Absolutely not. Does moving it behind a firewall change the fact that every mildly skilled exploit developer is sitting on an Oracle 0day? Absolutely not. People have legitimate reasons for exposing Salt…

> It's how I bootstrap random VMs and bare metal from the internet. What's stopping you from bootstrapping a bastion server through a simpler script and then bootstrapping with Salt from there?

That is exactly what the internet connected Salt master does. It bootstraps enough control that I can get the tunnels and keys properly configured, and the other 95% takes place once it is switched to a protected Salt master.
Post reply on HN