The funny thing is the founder of Wiz is formerly the head of Microsoft Israel, and many many ex-Microsoft are in Wiz. I wonder if the knowledge about Microsoft internals helped them finding this vulnerability.
ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough
11–20 of 35 posts
Re: ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough
#12I had several absolutely awful experiences with CosmosDB even before this breach. Its design and engineering are the worst I've encountered on Azure or anywhere else that I remember. This vulnerability, and especially its handling by Microsoft, were the final nail in the coffin for us and we've put in the effort to migrate away.
Could you elaborate on the design?
Re: ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough
#13That is completely insane. Getting root on one container = complete access to the entire system with administrator level access? What kind of security operation are they running there exactly? Local root exploits aren't exactly unheard of, so you'd think the infrastructure would be designed to tolerate that sort of thing, not simply hand out private keys to management APIs to all and sundry.
Re: ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough
#14I had several absolutely awful experiences with CosmosDB even before this breach. Its design and engineering are the worst I've encountered on Azure or anywhere else that I remember. This vulnerability, and especially its handling by Microsoft, were the final nail in the coffin for us and we've put in the effort to migrate away.
Could you elaborate on the design?
Sure. Here are a few examples. They're all based on my experience with the MongoDB API for CosmosDb. Your mileage with other APIs may vary.
1. CosmosDB has a hardcoded 60-second timeout for queries. That means that queries that take longer than that are literally impossible to run without breaking the query into smaller chunks. This is worse than it sounds because CosmosDB doesn't have some of the basic optimizations that exist in other databases. For example, finding all distinct values of an indexed field required a full scan which wasn't doable in 60 seconds. Another example is deleting all documents with a specific value in an indexed field - again, not doable in 60 seconds. When deleting or updating multiple documents, we'd write short snippets of code that queried for the ids of all documents that need to be updated, and then updated or deleted them by id one by one.
2. Scaling up and down again can cause irrevocable performance changes since there's a direct link between the number of provisioned RUs and the number of "Physical Partitions" created by the database. A new "physical partition" is created for every 10K RUs or 50GB of data. CosmosDB knows how to create new physical partitions when scaling up, but doesn't know how to merge them when scaling down.
Say you have 10 logical partitions on 5 physical partitions, and you are paying for 50K RUs. Each physical partition holds exactly 2 logical partitions and is allocated 10K RUs. Now you had to temporarily scale up the database for some reason to 100K RUs, so you have 10 physical partitions with one logical partition on each one. When you scale back to 50K RUs, you'll still have 10 logical partitions, each with 5K. So now each of your logical partitions has exactly 5K RUs, while before it had 10K RUs shared with a different logical partition.
3. The allocation of logical partitions to physical partitions is static, hash-based and there's no control over it. This means that having hot logical partitions is a performance problem. Hot logical partitions might end up on the same physical partition and be starved for resources while other physical partitions are over-provisioned. Of course, you can allocate data to partitions completely randomly and hope for the best, but there's a performance penalty for querying multiple logical partitions. Plus, updates/deletes are limited to a single logical partition, so you'll be losing the ability to batch update/delete related documents.
4. Index construction is asynchronous and very slow because it uses some pool of "free" RUs that scale off the RUs allocated to your collection. It used to take us over 12 hours to build simple indexes on a ~30GB collection. Also, if you issue multiple index modification commands they will be queued even if they cancel each other out. So issuing a "create index" command, realizing you've made a mistake, then issuing a "drop index" followed by another "create index" is a 24-hour adventure. Over the next 12 hours the original index will be created, then immediately dropped, and created again. To top it off, there's no visibility into which indexes are being used, and the commands for checking the progress of index construction were broken and never worked for us.
Re: ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough
#15> August 17 2021 - MSRC awarded $40,000 bounty for the report. I don't know much about the bug bounty industry, is this the typical payout from what it seems to be a pretty severe vulnerability?
Yes. A competent researcher deciding to investigate their products can reasonably be expected to find a new, unique vulnerability of this severity given a few weeks to maybe a month of effort. At a standard consulting rate that amounts to a few thousand to low tens of thousands of dollars, so a $40,000 payout is a fair amount for the amount of effort and difficulty to find a bug that totally invalidates the security…
Not a great comparison. A researcher does not work for a company, a researcher is basically a mercenary; they'll go where the money is. If a company pays too little relative to its peers, or fails to pay out too often due to fine-print in the T&Cs, researchers will go elsewhere. That's a serious loss for Microsoft given the kinds of vulnerabilities and their impact that a researcher can find if they're focusing on your platform.
Re: ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough
#16That is completely insane. Getting root on one container = complete access to the entire system with administrator level access? What kind of security operation are they running there exactly? Local root exploits aren't exactly unheard of, so you'd think the infrastructure would be designed to tolerate that sort of thing, not simply hand out private keys to management APIs to all and sundry.
What kind of development operation is the question I would ask. Security mostly involves convincing developers to do the right thing with a lot of resistance. Not sure I would assume the security team is behind this, rather than some "risk acceptance" forced on them to launch the feature on time.
Re: ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough
#17When we designed the security model for Google Cloud Build (I do not work there anymore), we decided that containers were not valid security barriers. So, all partitioning was done on the VM and network (configured outside the VM) level.
It wasn't hard to convince anyone that this was the right way to handle things.
Re: ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough
#18Earlier quoted context omitted.
Yes. A competent researcher deciding to investigate their products can reasonably be expected to find a new, unique vulnerability of this severity given a few weeks to maybe a month of effort. At a standard consulting rate that amounts to a few thousand to low tens of thousands of dollars, so a $40,000 payout is a fair amount for the amount of effort and difficulty to find a bug that totally invalidates the security…
Not a great comparison. A researcher does not work for a company, a researcher is basically a mercenary; they'll go where the money is. If a company pays too little relative to its peers, or fails to pay out too often due to fine-print in the T&Cs, researchers will go elsewhere. That's a serious loss for Microsoft given the kinds of vulnerabilities and their impact that a researcher can find if they're focusing on yo…
I believe the last sentence in the parent comment was (top notch) sarcasm.
Re: ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough
#19You know some ms product manager thought they were "winning" when they included Jupyter notebook.
Re: ChaosDB Explained: Azure's Cosmos DB Vulnerability Walkthrough
#20That is completely insane. Getting root on one container = complete access to the entire system with administrator level access? What kind of security operation are they running there exactly? Local root exploits aren't exactly unheard of, so you'd think the infrastructure would be designed to tolerate that sort of thing, not simply hand out private keys to management APIs to all and sundry.
What kind of development operation is the question I would ask. Security mostly involves convincing developers to do the right thing with a lot of resistance. Not sure I would assume the security team is behind this, rather than some "risk acceptance" forced on them to launch the feature on time.
Tom This feature is due friday at 4pm but we want to review it so we need it done by lunch. Dont forget to make it secure.
Sure its as secure as i was givem time to test security none and none. Thanks Tom great work. See ya at 12.
This is more the reality than resistance.