Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post Mortem on Salt Incident

blog.algolia.com

Post reply on HN
Print view View original

31–40 of 70 posts

Re: Post Mortem on Salt Incident

#31

Post by section_me »

cetra3 wrote: post #6

This whole salt-stack incident could've been handled a lot better by salt themselves: - the notification was a week ago to a small mailing list, which is tucked away on their site - no notification to the registry to when you go to download salt (at least I never received an email, but still get plenty of marketing spam) - no posts on social media as far as I can tell, I couldn't find a tweet, anything on reddit, or…

It was posted, just no traction on it (eg. https://news.ycombinator.com/item?id=22972100 posted 11 days ago). But yeah supprising the lack of posts about it.

Re: Post Mortem on Salt Incident

#32

Post by vegannet »

alexbrower wrote: post #23

Can anyone describe the business benefits of an algolia implementation (vs Elasticsearch?) for a company that doesn't heavily rely on content searches? It seems expensive and something that I'd build on my own. (Disclaimer: long-time operator and fledgling programmer)

Search is hard to get right and the cost of Algolia is negligible vs. doing it yourself. As a programmer, every line of code you write is a line of code you own: the less code you own in production, the better off you are. Algolia has saved us hundreds of hours which translates to tens of thousands of dollars.

Re: Post Mortem on Salt Incident

#33

Post by mike_d »

hajhatten wrote: post #29
mike_d wrote: post #21

Earlier quoted context omitted.

> has network access to the salt masters port, which should never occur You seem to prescribe to the "hard shell soft gooey center" network security philosophy. Should people expose an Oracle server to the internet? Absolutely not. Does moving it behind a firewall change the fact that every mildly skilled exploit developer is sitting on an Oracle 0day? Absolutely not. People have legitimate reasons for exposing Salt…

You piqued my interest, what made you move? I personally haven't touched Saltstack in years, but was super happy with it when I did.

Twice I encountered breaking changes between versions that required manually upgrading minions. I also got the overall feeling Salt was built by developers, Ansible by sysadmins - and I fit into the latter bucket.

Re: Post Mortem on Salt Incident

#34

Post by nemo136 »

lasdfas wrote: post #3
VWWHFSfQ wrote: post #2

The intruders had root access to every server in a salt deployment for who knows how long and yet everyone is claiming there's no evidence that any data or secrets (customer's or otherwise) were exfiltrated from the network. This is a very dangerous assumption. Nobody has any idea what was run on the servers since it seems that once the initial attack script was deployed it downloaded and executed new scripts every 6…

I agree. I would like to seem more details of how they determined it was only crypto mining. Finding only mining scripts in your logs doesn't mean they were not running other code once they had root.

running the virus code in a container / vm and checking what gets modified

Re: Post Mortem on Salt Incident

#35

Post by dijit »

mtam wrote: post #4

“We’ve secured the impacted SaltStack service by updating it and adding additional IP filtering, allowing only our servers to connect to it.” So this means they had Salt master ports publicly accessible? Why would anyone have salt ports open/exposed to public/internet?

> Why would anyone have salt ports open/exposed to public/internet?

If you're bootstrapping random servers, this is a fine approach.

The whole Salt connection methodology is 'trust on first connect' (a bit like the default SSH) with a manual stage in accepting an incoming request and the connection stream is encrypted.

If you're using salt to bootstrap your VPN servers or network appliances then it's understandable that you'd have it exposed to a more public network, and the documentation was clear that this was fine.

Not everything is a virtual machine on a cloud provider.

Re: Post Mortem on Salt Incident

#36

Post by StreamBright »

mtam wrote: post #7
cetra3 wrote: post #6

This whole salt-stack incident could've been handled a lot better by salt themselves: - the notification was a week ago to a small mailing list, which is tucked away on their site - no notification to the registry to when you go to download salt (at least I never received an email, but still get plenty of marketing spam) - no posts on social media as far as I can tell, I couldn't find a tweet, anything on reddit, or…

+1, however, from what I read, the vulnerability can only be exploited if the attacker has network access to the salt masters port, which should never occur. The people that got compromised had Salt exposed to the Internet, which is obviously ridiculous. Not trying to downplay the critical nature of the vulnerability but the ones that were compromised by this issue have deeper security issues to deal with.

Coconut security is not great either. Hard shell, soft internals. Not exposing the ports to the internet is just one layer.

Re: Post Mortem on Salt Incident

#37

Post by dijit »

trabant00 wrote: post #28
kureikain wrote: post #25

It's weird that these salt master are reach-able from internet and they can sleep well with it. Even with zero-trust network or beyondcorp idea, I still found one extra layer of protection a VPC give are so great. Few years ago, it has an issue with K8S API Server, and updating k8s isn't a walk in the park. I felt relax back then because we have everything inside VPC. You can use SSH or VPN to access service inside V…

I never understood the current trent to say VPN is a thing of the past. Redundancy in security layers is how you dont't get affected by every CVE out there. Imo this is THE lesson to learn from this story. Seondary: salt and ansible are not very mature yet.

Salt is definitely immature (been using it for 5 years and the situation has actually gotten worse in that time) but Ansible is a weird thing to group.

What issues do you have with Ansible?

Re: Post Mortem on Salt Incident

#38

Post by vertex-four »

sterlind wrote: post #5
lasdfas wrote: post #3

Earlier quoted context omitted.

I agree. I would like to seem more details of how they determined it was only crypto mining. Finding only mining scripts in your logs doesn't mean they were not running other code once they had root.

It seems bizarre to me that a crypto miner got in. It wouldn't make much money on regular CPUs, and the high processor usage would immediately draw attention. So it looks like a low-effort botnet, which is embarrassing to get pwned by. (The coin mining could be a cover like you mention, but it seems unlikely since it naturally draws attention.)

It’s easier to sell Monero for cash than... some random data from some random company.

Re: Post Mortem on Salt Incident

#39

Post by aseure »

alexbrower wrote: post #23

Can anyone describe the business benefits of an algolia implementation (vs Elasticsearch?) for a company that doesn't heavily rely on content searches? It seems expensive and something that I'd build on my own. (Disclaimer: long-time operator and fledgling programmer)

Disclaimer: I'm a developer at Algolia.

IMHO the two main advantages in favor of Algolia, are the sane defaults for relevancy and speed and the fact that the service is hosted and can grow with your business without having dedicated engineers to manage both the configuration and the infrastructure.

Also, on top of the Algolia services per se (search, analytics, recommendation, etc.), we're providing a lot of backend and frontend libraries which one would otherwise need to reimplement when using an elastic- or Solr-based implementations.

Re: Post Mortem on Salt Incident

#40

Post by vbernat »

As a point of comparison, you can also expose Puppet masters to the public Internet but Puppet is using HTTP/HTTPS as a transport, so it is trivial to put a reverse proxy in front of it, requiring a valid certificate (managed and signed by Puppet) to contact the service. This way, no need to maintain a whitelist of legitimate clients.
Post reply on HN