Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post Mortem on Salt Incident

blog.algolia.com

Post reply on HN
Print view View original

51–60 of 70 posts

Re: Post Mortem on Salt Incident

#51

Post by user5994461 »

dijit wrote: post #50
darkwater wrote: post #43

Earlier quoted context omitted.

> If you're bootstrapping random servers, this is a fine approach. Define "random". I think there is an alternative method not involving exposing you CM server on the Internet for almost any definition of random. In the Algolia case it's pretty sure because they now filter the access by IP (so they KNOW the IPs)

"Random" can mean "I don't know before I start my instance". If you're multi-cloud (vultr, DO, AWS and GCP) you almost certainly will not know your instances IP before it's provisioned and you can't make use of nice features like network tags or security labels. If you're producing test environments then bootstrapping those is going to be significantly more painful than just opening up your salt-master and running an…

In a multi clouds setup, all the clouds are joined together with site to site VPNs. One doesn't just do a setup where they're public and connect to one another database over the public internet.

Re: Post Mortem on Salt Incident

#52

Post by darkwater »

kureikain wrote: post #44
darkwater wrote: post #42

Earlier quoted context omitted.

Yeah, I completely agree and really don't see the point of having a Configuration Management server facing Internet and basically having all your servers connect to it through the Internet! One thing is BeyondCorp idea to eliminate the roadwarrior concept and another is having your infra management exposed to CVEs in the wild! For Jenkins it's a bit more complicated because GitHub webhooks although they do publish th…

For Jenkins, what I do is: 1. Configured webhook override in Jenkins. So Jenkins will register sth like https://ci-webhook.domain.com to github webhook. 2. This ci-webhook is a simple webapp that validate webhook and if it's valid(sign by correct key), write the payload to SQS queue 3. A small daemon, run on same Jenkins master, that pulls SQS queue, and replay it to local jenkins I used to rely on Github IP whitelis…

> I used to rely on Github IP whitelist but one day i realized anyone can hit my Jenkins use Github.

That's a really good point but I guess you are talking about Actions egress right? Webhook in theory have dedicated IP ranges [1] and I think they are not shared with Actions egress, although TBH I haven't tested it.

[1] https://api.github.com/meta

Re: Post Mortem on Salt Incident

#53

Post by darkwater »

dijit wrote: post #50
darkwater wrote: post #43

Earlier quoted context omitted.

> If you're bootstrapping random servers, this is a fine approach. Define "random". I think there is an alternative method not involving exposing you CM server on the Internet for almost any definition of random. In the Algolia case it's pretty sure because they now filter the access by IP (so they KNOW the IPs)

"Random" can mean "I don't know before I start my instance". If you're multi-cloud (vultr, DO, AWS and GCP) you almost certainly will not know your instances IP before it's provisioned and you can't make use of nice features like network tags or security labels. If you're producing test environments then bootstrapping those is going to be significantly more painful than just opening up your salt-master and running an…

> As other people have mentioned, this was always supposed to be /possible/ it's akin to SSH. Sure, you can avoid some log spam and potential issues by firewalling it off- but it's meant to be possible to run it publicly, it has always been marketed this way so it's not "insane" that people did it.

I'm not blaming anyone, I'm just saying that if you put well-known software facing the Internet you are exposing yourself to more risks than not putting them on the Internet. And for a core infra software as SaltStack I don't really see a good reason to justify it. I don't justify either putting SSH publicly accessible unless you are a really, really small company or an individual.

Re: Post Mortem on Salt Incident

#54

Post by Conan_Kudo »

mike_d wrote: post #33
hajhatten wrote: post #29

Earlier quoted context omitted.

You piqued my interest, what made you move? I personally haven't touched Saltstack in years, but was super happy with it when I did.

Twice I encountered breaking changes between versions that required manually upgrading minions. I also got the overall feeling Salt was built by developers, Ansible by sysadmins - and I fit into the latter bucket.

Ansible (originally known as "Fedora Unified Network Controller" or "func") was made for solving the problems automating Fedora Infrastructure.

Puppet did not make Fedora Infrastructure administrators happy. So func was designed around solving their problems, and expanded its scope as people found it useful. Then it was renamed to Ansible, the developers left Red Hat to create AnsibleWorks, and the rest is history!

So yes, it was made for sysadmins. :)

Re: Post Mortem on Salt Incident

#55

Post by johann-algolia »

VWWHFSfQ wrote: post #2

The intruders had root access to every server in a salt deployment for who knows how long and yet everyone is claiming there's no evidence that any data or secrets (customer's or otherwise) were exfiltrated from the network. This is a very dangerous assumption. Nobody has any idea what was run on the servers since it seems that once the initial attack script was deployed it downloaded and executed new scripts every 6…

Hello,

I'll try to give you some insight as I'm a security engineer at Algolia.

Your concern is valid, and it's true, we cannot know for sure. That's the reason why, as explained in the blog post, we are reinstalling all impacted servers and rotating our secrets. If our assumption is false, this should contain the issue.

That being said, we have good reasons to make that assumption.

- Our analysis of the incident and how the malware behaved on our systems didn't find any evidence towards access and transfer of data.

- There are other public analysis of the malware. Other companies hit have the same analysis than us, and you can have a look at https://saltexploit.com/ which is maintaining an interesting list of what is known on the attack, how it behaved, and how it's evolving fast to adapt.

I hope this answers your concern.

Re: Post Mortem on Salt Incident

#56

Post by cat199 »

cetra3 wrote: post #6

This whole salt-stack incident could've been handled a lot better by salt themselves: - the notification was a week ago to a small mailing list, which is tucked away on their site - no notification to the registry to when you go to download salt (at least I never received an email, but still get plenty of marketing spam) - no posts on social media as far as I can tell, I couldn't find a tweet, anything on reddit, or…

> - Ubuntu and other repositories are still vulnerable

isn't really salt's problem though.. same could be said for relying on any distro-provided package

Re: Post Mortem on Salt Incident

#57

Post by Jedd »

VWWHFSfQ wrote: post #2

The intruders had root access to every server in a salt deployment for who knows how long and yet everyone is claiming there's no evidence that any data or secrets (customer's or otherwise) were exfiltrated from the network. This is a very dangerous assumption. Nobody has any idea what was run on the servers since it seems that once the initial attack script was deployed it downloaded and executed new scripts every 6…

> ... and yet everyone is claiming there's no evidence that any data or secrets (customer's or otherwise) were exfiltrated from the network.

A number of people have carefully reviewed the payload that was deployed to servers, especially during what we're calling v1-v4 of the attack. (v5 onwards got more complex, but that wasn't until Monday (with variability for timezone).

> Nobody has any idea what was run on the servers ...

Well that's not true - there's a number of victims that have useful IDS tools, including auditd, plus the review of binaries and shell scripts deployed, etc.

Some of us also have netflow collection at the edge, and can review connections initiated from within our networks.

> ... once the initial attack script was deployed it downloaded and executed new scripts every 60s and then removed themselves.

I don't think any of us have found scripts that removed themselves. While that may sound naive, there's a few researchers that have been analysing these tools, including via large honeypot networks, and this just hasn't (at least for the first 2-3 days) been a profile of the attack.

Thankfully - and I appreciate it's very weird to say this - the initial attacks were very much vanilla crypto currency mining opportunities. It could have been a lot worse, and algolia's assessment matches a lot of other independent assessments on this front.

Re: Post Mortem on Salt Incident

#58

Post by alexandercrohde »

dijit wrote: post #35
mtam wrote: post #4

“We’ve secured the impacted SaltStack service by updating it and adding additional IP filtering, allowing only our servers to connect to it.” So this means they had Salt master ports publicly accessible? Why would anyone have salt ports open/exposed to public/internet?

> Why would anyone have salt ports open/exposed to public/internet? If you're bootstrapping random servers, this is a fine approach. The whole Salt connection methodology is 'trust on first connect' (a bit like the default SSH) with a manual stage in accepting an incoming request and the connection stream is encrypted. If you're using salt to bootstrap your VPN servers or network appliances then it's understandable t…

Kind of a tough situation. I personally wouldn't be ready to accept this is the last such vulnerability that will be found.

In light of this attack, maybe going forward have a setup script that creates an SSH tunnel back to a machine that can talk to the salt-master for you. You could then have VPN, but if it's flakey at all, it could cost the ability to update machines.

Or perhaps (and I say this as a saltstack user) ansible really is the more secure model for those scenarios.

Re: Post Mortem on Salt Incident

#59

Post by tilolebo »

kureikain wrote: post #44
darkwater wrote: post #42

Earlier quoted context omitted.

Yeah, I completely agree and really don't see the point of having a Configuration Management server facing Internet and basically having all your servers connect to it through the Internet! One thing is BeyondCorp idea to eliminate the roadwarrior concept and another is having your infra management exposed to CVEs in the wild! For Jenkins it's a bit more complicated because GitHub webhooks although they do publish th…

For Jenkins, what I do is: 1. Configured webhook override in Jenkins. So Jenkins will register sth like https://ci-webhook.domain.com to github webhook. 2. This ci-webhook is a simple webapp that validate webhook and if it's valid(sign by correct key), write the payload to SQS queue 3. A small daemon, run on same Jenkins master, that pulls SQS queue, and replay it to local jenkins I used to rely on Github IP whitelis…

Hooks have a dedicated IP range.

With Terraform and AWS it's pretty simple to create a security group on an Application Load Balancer and whitelist these IPs using https://www.terraform.io/docs/providers/github/d/ip_ranges.h...

Re: Post Mortem on Salt Incident

#60

Post by techslave »

tasssko wrote: post #22
mtam wrote: post #7

Earlier quoted context omitted.

+1, however, from what I read, the vulnerability can only be exploited if the attacker has network access to the salt masters port, which should never occur. The people that got compromised had Salt exposed to the Internet, which is obviously ridiculous. Not trying to downplay the critical nature of the vulnerability but the ones that were compromised by this issue have deeper security issues to deal with.

+1 agree but exposing salt to the internet is not the problem. A simple ip whitelist ingress firewall rule on the salt master port would have helped, blocking access is also possible on this port. With cloud services it has become trivial to group server resources so that when they belong to the same group they can communicate with each other. I don’t use salt however i am not a proponent of network isolation as a fo…

that’s just plain negligent. of course you use network isolation as one of your defenses in depth.
Post reply on HN