Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post Mortem on Salt Incident

blog.algolia.com

Post reply on HN
Print view View original

41–50 of 70 posts

Re: Post Mortem on Salt Incident

#41

Post by ciprian_craciun »

I've seen mentioned in the comments various "deployment" tools (or call them "configuration management" if you will) being called "insecure" or "immature", or one being claimed better than another; however I think this is a good opportunity to talk about a deeper problem, namely the architectural choices each tool has taken.

These choices all impact the reliability and security of the resulting system, especially the following:

* do they rely on SSH, or they have implemented their own authentication / authorization techniques? (personally I would be very reluctant to trust anything that just listens on a network port for deployment commands, and it's not SSH;)

* do the agents run with full `root` privileges, or is there a builtin mechanism that allows the agent to act only in a limited capacity, within the confines of a set of whitelisted actions? (perhaps even requiring a secondary authentication mechanism for certain "sensitive" actions, for example something integrated with `sudo`, that provides a sort of 2-factor-authentication with a human in the loop;)

* do the operators have enough "visibility" into what is happening during the deployments? (more specifically, are the deployment scripts easily auditable or are they a spaghetti of dependencies? are the concrete actions to be taken clearly described, or are they hidden in the source code of the tool?)

* are there builtin mechanisms to "verify" the results of the deployments?

* and building upon the previous item, are there mechanisms to continuously "verify" if the deployment hasn't changed behind the scenes?

I understand that some of these features wouldn't have helped directly to prevent this particular case, however it would have helped in alerting and diagnosis.

Re: Post Mortem on Salt Incident

#42

Post by darkwater »

kureikain wrote: post #25

It's weird that these salt master are reach-able from internet and they can sleep well with it. Even with zero-trust network or beyondcorp idea, I still found one extra layer of protection a VPC give are so great. Few years ago, it has an issue with K8S API Server, and updating k8s isn't a walk in the park. I felt relax back then because we have everything inside VPC. You can use SSH or VPN to access service inside V…

Yeah, I completely agree and really don't see the point of having a Configuration Management server facing Internet and basically having all your servers connect to it through the Internet! One thing is BeyondCorp idea to eliminate the roadwarrior concept and another is having your infra management exposed to CVEs in the wild!

For Jenkins it's a bit more complicated because GitHub webhooks although they do publish their IPs in a programmatic form so you can whitelist them.

Re: Post Mortem on Salt Incident

#43

Post by darkwater »

dijit wrote: post #35
mtam wrote: post #4

“We’ve secured the impacted SaltStack service by updating it and adding additional IP filtering, allowing only our servers to connect to it.” So this means they had Salt master ports publicly accessible? Why would anyone have salt ports open/exposed to public/internet?

> Why would anyone have salt ports open/exposed to public/internet? If you're bootstrapping random servers, this is a fine approach. The whole Salt connection methodology is 'trust on first connect' (a bit like the default SSH) with a manual stage in accepting an incoming request and the connection stream is encrypted. If you're using salt to bootstrap your VPN servers or network appliances then it's understandable t…

> If you're bootstrapping random servers, this is a fine approach.

Define "random". I think there is an alternative method not involving exposing you CM server on the Internet for almost any definition of random. In the Algolia case it's pretty sure because they now filter the access by IP (so they KNOW the IPs)

Re: Post Mortem on Salt Incident

#44

Post by kureikain »

darkwater wrote: post #42
kureikain wrote: post #25

It's weird that these salt master are reach-able from internet and they can sleep well with it. Even with zero-trust network or beyondcorp idea, I still found one extra layer of protection a VPC give are so great. Few years ago, it has an issue with K8S API Server, and updating k8s isn't a walk in the park. I felt relax back then because we have everything inside VPC. You can use SSH or VPN to access service inside V…

Yeah, I completely agree and really don't see the point of having a Configuration Management server facing Internet and basically having all your servers connect to it through the Internet! One thing is BeyondCorp idea to eliminate the roadwarrior concept and another is having your infra management exposed to CVEs in the wild! For Jenkins it's a bit more complicated because GitHub webhooks although they do publish th…

For Jenkins, what I do is:

1. Configured webhook override in Jenkins. So Jenkins will register sth like https://ci-webhook.domain.com to github webhook.

2. This ci-webhook is a simple webapp that validate webhook and if it's valid(sign by correct key), write the payload to SQS queue

3. A small daemon, run on same Jenkins master, that pulls SQS queue, and replay it to local jenkins

I used to rely on Github IP whitelist but one day i realized anyone can hit my Jenkins use Github.

Re: Post Mortem on Salt Incident

#45

Post by kureikain »

lykr0n wrote: post #16
mirimir wrote: post #10

Earlier quoted context omitted.

Yeah, that jumped out for me too. I'm guessing that they didn't want to deploy some sort of private network layer.

That's easier said then done. There are no simple cross cloud provider solutions for a private networking other then ZeroTier, which has it's own issues.

Last time I was able to build Azure AWS and GCP AWS use their VPN tunneling and a strongswan server on AWS.

It's only AZure AWS GCP, Azure GCP I didn't try bcuz we just want to connect to central AWS node.

I think IPSec with the "right" config is good enough. But the pain is managing the route tables :(.

Re: Post Mortem on Salt Incident

#46

Post by aneutron »

bawolff wrote: post #30
cetra3 wrote: post #6

This whole salt-stack incident could've been handled a lot better by salt themselves: - the notification was a week ago to a small mailing list, which is tucked away on their site - no notification to the registry to when you go to download salt (at least I never received an email, but still get plenty of marketing spam) - no posts on social media as far as I can tell, I couldn't find a tweet, anything on reddit, or…

> one week's notice between the initial announcement and the patch coming out. The patch being released is basically a disclosure of the vulnerability While your other points may be valid, one week should be plenty of time between announcement and patch. Any longer and i would call the time table problematic.

You have clearly never worked at a large enough OLD corporation.

One week is nothing compared to what it would take to upgrade your configuration management system.

Re: Post Mortem on Salt Incident

#47

Post by empath75 »

mike_d wrote: post #21
mtam wrote: post #7

Earlier quoted context omitted.

+1, however, from what I read, the vulnerability can only be exploited if the attacker has network access to the salt masters port, which should never occur. The people that got compromised had Salt exposed to the Internet, which is obviously ridiculous. Not trying to downplay the critical nature of the vulnerability but the ones that were compromised by this issue have deeper security issues to deal with.

> has network access to the salt masters port, which should never occur You seem to prescribe to the "hard shell soft gooey center" network security philosophy. Should people expose an Oracle server to the internet? Absolutely not. Does moving it behind a firewall change the fact that every mildly skilled exploit developer is sitting on an Oracle 0day? Absolutely not. People have legitimate reasons for exposing Salt…

> People have legitimate reasons for exposing Salt to the internet. I do. It's how I bootstrap random VMs and bare metal from the internet.

I question that that is a legitimate reason to expose it to the internet.

Defense in depth is a thing and putting the keys to the kingdom at layer 0 doesn’t seem wise even if a vpn or bastion doesn’t offer perfect protection.

Re: Post Mortem on Salt Incident

#48

Post by CJefferson »

mtam wrote: post #7
cetra3 wrote: post #6

This whole salt-stack incident could've been handled a lot better by salt themselves: - the notification was a week ago to a small mailing list, which is tucked away on their site - no notification to the registry to when you go to download salt (at least I never received an email, but still get plenty of marketing spam) - no posts on social media as far as I can tell, I couldn't find a tweet, anything on reddit, or…

+1, however, from what I read, the vulnerability can only be exploited if the attacker has network access to the salt masters port, which should never occur. The people that got compromised had Salt exposed to the Internet, which is obviously ridiculous. Not trying to downplay the critical nature of the vulnerability but the ones that were compromised by this issue have deeper security issues to deal with.

If they wrote software which should never be visible to the internet, they should have made that clearer.

It's far too easy to make something internet-visible. They could have set up a simple check to see if the service is internet, and refused to work if it was.

Re: Post Mortem on Salt Incident

#49

Post by user5994461 »

cetra3 wrote: post #9
mtam wrote: post #7

Earlier quoted context omitted.

+1, however, from what I read, the vulnerability can only be exploited if the attacker has network access to the salt masters port, which should never occur. The people that got compromised had Salt exposed to the Internet, which is obviously ridiculous. Not trying to downplay the critical nature of the vulnerability but the ones that were compromised by this issue have deeper security issues to deal with.

If you look at their current `hardening` document it still has pretty unclear language about what is acceptable and what isn't. > Use a hardened bastion server or a VPN to restrict direct access to the Salt master from the internet Is this SSH access or is this access to the salt master from minions? Or just access in general?

SSH bastions and VPN are two standard ways to allow external clients access into an internal network, meaning salt is never exposed publicly.

I read this as a guideline that the salt master must not be exposed to the internet. Albeit could be better worded for a developer audience who doesn't understand bastions or VPN well.

Re: Post Mortem on Salt Incident

#50

Post by dijit »

darkwater wrote: post #43
dijit wrote: post #35

Earlier quoted context omitted.

> Why would anyone have salt ports open/exposed to public/internet? If you're bootstrapping random servers, this is a fine approach. The whole Salt connection methodology is 'trust on first connect' (a bit like the default SSH) with a manual stage in accepting an incoming request and the connection stream is encrypted. If you're using salt to bootstrap your VPN servers or network appliances then it's understandable t…

> If you're bootstrapping random servers, this is a fine approach. Define "random". I think there is an alternative method not involving exposing you CM server on the Internet for almost any definition of random. In the Algolia case it's pretty sure because they now filter the access by IP (so they KNOW the IPs)

"Random" can mean "I don't know before I start my instance".

If you're multi-cloud (vultr, DO, AWS and GCP) you almost certainly will not know your instances IP before it's provisioned and you can't make use of nice features like network tags or security labels.

If you're producing test environments then bootstrapping those is going to be significantly more painful than just opening up your salt-master and running an authenticated API request to allow those new machines.

As other people have mentioned, this was always supposed to be /possible/ it's akin to SSH. Sure, you can avoid some log spam and potential issues by firewalling it off- but it's meant to be possible to run it publicly, it has always been marketed this way so it's not "insane" that people did it.

Post reply on HN