ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough • Hacker News Board

Re: ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough

#21

Post by xbar » Sun, Nov 21, 2021, 3:19 PM UTC

junon wrote: post #7

technion wrote: post #6

Earlier quoted context omitted.

The part that's contraversial about the MS bounties is that they stopped covering the majority of on-premise products. For example, person that reported the two major Microsoft Exchange vulnerability chains received no payout at all. Ref: https://i.blackhat.com/USA21/Wednesday-Handouts/us-21-ProxyL...

MS is pretty bad about vulnerability bounties. I reported one about privilege escalation on WSL1 and received no response plus it was patched within the following months. Was a bit aggravated.

Truly horrible, despite tooting their horn about it a very awful lot.

Re: ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough

#22

Post by xbar » Sun, Nov 21, 2021, 3:21 PM UTC

beardedwizard wrote: post #16

native_samples wrote: post #13

That is completely insane. Getting root on one container = complete access to the entire system with administrator level access? What kind of security operation are they running there exactly? Local root exploits aren't exactly unheard of, so you'd think the infrastructure would be designed to tolerate that sort of thing, not simply hand out private keys to management APIs to all and sundry.

What kind of development operation is the question I would ask. Security mostly involves convincing developers to do the right thing with a lot of resistance. Not sure I would assume the security team is behind this, rather than some "risk acceptance" forced on them to launch the feature on time.

A close reading of the DevSecOps infinity lifecycle has lots of security+development touchpoints that yield better software security than this.

The trade-off you mention is a management decision, not really a development or security decision.

Re: ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough

#23

Post by Cpoll » Sun, Nov 21, 2021, 3:43 PM UTC

maltalex wrote: post #14

Ducki wrote: post #12

Earlier quoted context omitted.

Could you elaborate on the design?

Sure. Here are a few examples. They're all based on my experience with the MongoDB API for CosmosDb. Your mileage with other APIs may vary. 1. CosmosDB has a hardcoded 60-second timeout for queries. That means that queries that take longer than that are literally impossible to run without breaking the query into smaller chunks. This is worse than it sounds because CosmosDB doesn't have some of the basic optimizations…

> RUs

I've also run into a situation with obtuse/unexpected/excessive pricing. We were spiking a Mongo -> Cosmos migration and with our Cosmos configuration we were getting charged separately for each (empty) collection our app created.

Each collection was billed separately for provisioned throughput even though they were unused, and since the app created 50~ collections the cost added up pretty quickly before I noticed. Note: There is a way to turn this off afair, but the description of the setting made me decide to select it for a prod-like database.

In reality it's partly my fault. I should have kept a close eye on the costs until I was sure my mental model was correct. I also knew CosmosDB was just offering a Mongo API and wasn't actually hosted Mongo, so I should have been more vigilant about the differences in implementation. And of course I should have RTFM — although, even when I noticed the problem, it took me a long time and a support ticket to find the explanation in the docs.

Re: ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough

#24

Post by rawgabbit » Sun, Nov 21, 2021, 3:54 PM UTC

maltalex wrote: post #14

Ducki wrote: post #12

Earlier quoted context omitted.

Could you elaborate on the design?

Sure. Here are a few examples. They're all based on my experience with the MongoDB API for CosmosDb. Your mileage with other APIs may vary. 1. CosmosDB has a hardcoded 60-second timeout for queries. That means that queries that take longer than that are literally impossible to run without breaking the query into smaller chunks. This is worse than it sounds because CosmosDB doesn't have some of the basic optimizations…

Thank you for the detailed write up. I did a cursory investigation of CosmosDB but didn’t know it was this bad.

Re: ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough

#25

Post by maltalex » Sun, Nov 21, 2021, 3:55 PM UTC

Cpoll wrote: post #23

maltalex wrote: post #14

Earlier quoted context omitted.

Sure. Here are a few examples. They're all based on my experience with the MongoDB API for CosmosDb. Your mileage with other APIs may vary. 1. CosmosDB has a hardcoded 60-second timeout for queries. That means that queries that take longer than that are literally impossible to run without breaking the query into smaller chunks. This is worse than it sounds because CosmosDB doesn't have some of the basic optimizations…

> RUs I've also run into a situation with obtuse/unexpected/excessive pricing. We were spiking a Mongo -> Cosmos migration and with our Cosmos configuration we were getting charged separately for each (empty) collection our app created. Each collection was billed separately for provisioned throughput even though they were unused, and since the app created 50~ collections the cost added up pretty quickly before I noti…

> I should have RTFM — although, even when I noticed the problem, it took me a long time and a support ticket to find the explanation in the docs

I haven't raised it as an issue since it's not a design issue. But the CosmosDB documentation has been a constant source of pain for us too. Important details were often not mentioned, mentioned in unexpected places, or were downright wrong.

Re: ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough

#26

Post by bowmessage » Sun, Nov 21, 2021, 3:57 PM UTC

maltalex wrote: post #14

Ducki wrote: post #12

Earlier quoted context omitted.

Could you elaborate on the design?

Sure. Here are a few examples. They're all based on my experience with the MongoDB API for CosmosDb. Your mileage with other APIs may vary. 1. CosmosDB has a hardcoded 60-second timeout for queries. That means that queries that take longer than that are literally impossible to run without breaking the query into smaller chunks. This is worse than it sounds because CosmosDB doesn't have some of the basic optimizations…

From what I remember of my time in AWS, DynamoDB also suffers from 2. Not sure if that's changed, since.

Re: ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough

#27

Post by someuname » Sun, Nov 21, 2021, 4:10 PM UTC

skj wrote: post #17

When we designed the security model for Google Cloud Build (I do not work there anymore), we decided that containers were not valid security barriers. So, all partitioning was done on the VM and network (configured outside the VM) level. It wasn't hard to convince anyone that this was the right way to handle things.

Why are they not?

Re: ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough

#28

Post by notimetorelax » Sun, Nov 21, 2021, 5:13 PM UTC

shlosky wrote: post #11

The funny thing is the founder of Wiz is formerly the head of Microsoft Israel, and many many ex-Microsoft are in Wiz. I wonder if the knowledge about Microsoft internals helped them finding this vulnerability.

It might have, but ex-microsofties is a large set. Googling tells me Microsoft employs 181k people. Assuming 3% leave each year gives us 5k. Of course not all of them are software engineers.

Re: ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough

#29

Post by dagss » Sun, Nov 21, 2021, 5:20 PM UTC

rawgabbit wrote: post #24

maltalex wrote: post #14

Earlier quoted context omitted.

Sure. Here are a few examples. They're all based on my experience with the MongoDB API for CosmosDb. Your mileage with other APIs may vary. 1. CosmosDB has a hardcoded 60-second timeout for queries. That means that queries that take longer than that are literally impossible to run without breaking the query into smaller chunks. This is worse than it sounds because CosmosDB doesn't have some of the basic optimizations…

Thank you for the detailed write up. I did a cursory investigation of CosmosDB but didn’t know it was this bad.

It is much worse than this this was just the start of a very long list of awful problems (my team tried to live with Cosmos for a couple of years too,luckily got approved a port to another DB and now we are not looking back).

The worst thing is Microsoft reps would always recommend the database, Microsoft support (who came onsite to us to talk with us about Cosmos) had a hard time acknowledging what should be obvious problems and so on... so it took a while for us to catch on that it actually is as beyond awful as it is.

Re: ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough

#30

Post by maltalex » Sun, Nov 21, 2021, 6:08 PM UTC

dagss wrote: post #29

rawgabbit wrote: post #24

Earlier quoted context omitted.

Thank you for the detailed write up. I did a cursory investigation of CosmosDB but didn’t know it was this bad.

It is much worse than this this was just the start of a very long list of awful problems (my team tried to live with Cosmos for a couple of years too,luckily got approved a port to another DB and now we are not looking back). The worst thing is Microsoft reps would always recommend the database, Microsoft support (who came onsite to us to talk with us about Cosmos) had a hard time acknowledging what should be obvious…

Glad to hear we're not the only ones.

There were times when I felt like I was part of some elaborate prank or social psychology experiment to see how developers would react to an obviously broken product. Trivial use cases were, and probably still are utterly broken. As if no one before us ever tried to use them.

Still, I'd love to hear about your experience with cosmos and the issues you guys discovered.