Post Mortem on Salt Incident • Hacker News Board

Re: Post Mortem on Salt Incident

#21

Post by mike_d » Wed, May 06, 2020, 6:08 AM UTC

mtam wrote: post #7

cetra3 wrote: post #6

This whole salt-stack incident could've been handled a lot better by salt themselves: - the notification was a week ago to a small mailing list, which is tucked away on their site - no notification to the registry to when you go to download salt (at least I never received an email, but still get plenty of marketing spam) - no posts on social media as far as I can tell, I couldn't find a tweet, anything on reddit, or…

+1, however, from what I read, the vulnerability can only be exploited if the attacker has network access to the salt masters port, which should never occur. The people that got compromised had Salt exposed to the Internet, which is obviously ridiculous. Not trying to downplay the critical nature of the vulnerability but the ones that were compromised by this issue have deeper security issues to deal with.

> has network access to the salt masters port, which should never occur

You seem to prescribe to the "hard shell soft gooey center" network security philosophy. Should people expose an Oracle server to the internet? Absolutely not. Does moving it behind a firewall change the fact that every mildly skilled exploit developer is sitting on an Oracle 0day? Absolutely not.

People have legitimate reasons for exposing Salt to the internet. I do. It's how I bootstrap random VMs and bare metal from the internet. But in my case the attack was mitigated by the fact that Salt cascades changes in a bunch of other systems and re-masters minions to a host only reachable over a tunnel. I blew away the internet master, restored from a backup, and patched.

> the ones that were compromised by this issue have deeper security issues to deal with

Or it was just another Monday. When you become sufficiently large you deal with incidents on a daily basis. Kudos to the people who publicly postmortem and talk about what went well and what didn't.

(For the record, I've already been working for a few months on a move to Ansible for non-security reasons)

Re: Post Mortem on Salt Incident

#22

Post by tasssko » Wed, May 06, 2020, 6:09 AM UTC

mtam wrote: post #7

cetra3 wrote: post #6

This whole salt-stack incident could've been handled a lot better by salt themselves: - the notification was a week ago to a small mailing list, which is tucked away on their site - no notification to the registry to when you go to download salt (at least I never received an email, but still get plenty of marketing spam) - no posts on social media as far as I can tell, I couldn't find a tweet, anything on reddit, or…

+1, however, from what I read, the vulnerability can only be exploited if the attacker has network access to the salt masters port, which should never occur. The people that got compromised had Salt exposed to the Internet, which is obviously ridiculous. Not trying to downplay the critical nature of the vulnerability but the ones that were compromised by this issue have deeper security issues to deal with.

+1 agree but exposing salt to the internet is not the problem. A simple ip whitelist ingress firewall rule on the salt master port would have helped, blocking access is also possible on this port. With cloud services it has become trivial to group server resources so that when they belong to the same group they can communicate with each other. I don’t use salt however i am not a proponent of network isolation as a form of security.

Re: Post Mortem on Salt Incident

#23

Post by alexbrower » Wed, May 06, 2020, 6:30 AM UTC

Can anyone describe the business benefits of an algolia implementation (vs Elasticsearch?) for a company that doesn't heavily rely on content searches? It seems expensive and something that I'd build on my own.

(Disclaimer: long-time operator and fledgling programmer)

Re: Post Mortem on Salt Incident

#24

Post by 0x0 » Wed, May 06, 2020, 6:32 AM UTC

Both this and the ghost cms updates seem to hint that the only reason this was discovered was the fact that loud crypto miners were exhausting resources. What are the chances a more quiet attacker hasn't thoroughly ploughed through the entire infrastructure days ahead?

Also think about how many years this vuln has been present and exposed. Who's to know blackhats haven't sat on this 0day for years, quietly compromosing private keys and other data? Spooky.

Re: Post Mortem on Salt Incident

#25

Post by kureikain » Wed, May 06, 2020, 6:35 AM UTC

It's weird that these salt master are reach-able from internet and they can sleep well with it.

Even with zero-trust network or beyondcorp idea, I still found one extra layer of protection a VPC give are so great. Few years ago, it has an issue with K8S API Server, and updating k8s isn't a walk in the park. I felt relax back then because we have everything inside VPC.

You can use SSH or VPN to access service inside VPC. But any of tools that had permission to manage your infrastructure should never expose to the internet.

Same thing with Jenkins, if you are using Jenkins to manage Terraform or trigger Ansible/Salt/Chef run, make sure Jenkins is not reachable from internet. Using different method to route webhook into it.

Re: Post Mortem on Salt Incident

#26

Post by sterlind » Wed, May 06, 2020, 6:35 AM UTC

itsajoke wrote: post #14

sterlind wrote: post #5

Earlier quoted context omitted.

It seems bizarre to me that a crypto miner got in. It wouldn't make much money on regular CPUs, and the high processor usage would immediately draw attention. So it looks like a low-effort botnet, which is embarrassing to get pwned by. (The coin mining could be a cover like you mention, but it seems unlikely since it naturally draws attention.)

I once worked at a place where a minor piece of cloud infra got exploited. All the attacker did was run a monero miner on it.

Heh, in a way it makes a good bug bounty. Like if popping calc got you a trickle of income.

Re: Post Mortem on Salt Incident

#27

Post by isodude » Wed, May 06, 2020, 6:53 AM UTC

cetra3 wrote: post #6

This whole salt-stack incident could've been handled a lot better by salt themselves: - the notification was a week ago to a small mailing list, which is tucked away on their site - no notification to the registry to when you go to download salt (at least I never received an email, but still get plenty of marketing spam) - no posts on social media as far as I can tell, I couldn't find a tweet, anything on reddit, or…

Weird that something with this title got buried here on HN: https://news.ycombinator.com/item?id=23041528

Re: Post Mortem on Salt Incident

#28

Post by trabant00 » Wed, May 06, 2020, 6:58 AM UTC

kureikain wrote: post #25

It's weird that these salt master are reach-able from internet and they can sleep well with it. Even with zero-trust network or beyondcorp idea, I still found one extra layer of protection a VPC give are so great. Few years ago, it has an issue with K8S API Server, and updating k8s isn't a walk in the park. I felt relax back then because we have everything inside VPC. You can use SSH or VPN to access service inside V…

I never understood the current trent to say VPN is a thing of the past. Redundancy in security layers is how you dont't get affected by every CVE out there.

Imo this is THE lesson to learn from this story.

Seondary: salt and ansible are not very mature yet.

Re: Post Mortem on Salt Incident

#29

Post by hajhatten » Wed, May 06, 2020, 6:58 AM UTC

mike_d wrote: post #21

mtam wrote: post #7

Earlier quoted context omitted.

+1, however, from what I read, the vulnerability can only be exploited if the attacker has network access to the salt masters port, which should never occur. The people that got compromised had Salt exposed to the Internet, which is obviously ridiculous. Not trying to downplay the critical nature of the vulnerability but the ones that were compromised by this issue have deeper security issues to deal with.

> has network access to the salt masters port, which should never occur You seem to prescribe to the "hard shell soft gooey center" network security philosophy. Should people expose an Oracle server to the internet? Absolutely not. Does moving it behind a firewall change the fact that every mildly skilled exploit developer is sitting on an Oracle 0day? Absolutely not. People have legitimate reasons for exposing Salt…

You piqued my interest, what made you move? I personally haven't touched Saltstack in years, but was super happy with it when I did.

Re: Post Mortem on Salt Incident

#30

Post by bawolff » Wed, May 06, 2020, 7:08 AM UTC

cetra3 wrote: post #6

This whole salt-stack incident could've been handled a lot better by salt themselves: - the notification was a week ago to a small mailing list, which is tucked away on their site - no notification to the registry to when you go to download salt (at least I never received an email, but still get plenty of marketing spam) - no posts on social media as far as I can tell, I couldn't find a tweet, anything on reddit, or…

> one week's notice between the initial announcement and the patch coming out. The patch being released is basically a disclosure of the vulnerability

While your other points may be valid, one week should be plenty of time between announcement and patch. Any longer and i would call the time table problematic.