Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Azure Cosmos DB, a globally distributed database

docs.microsoft.com

Post reply on HN
Print view View original

111–120 of 120 posts

Re: Azure Cosmos DB, a globally distributed database

#111

Post by GovindMS »

sargun wrote: post #92

Does CosmosDB have any relationship to Microsoft Cosmos ( http://web.stanford.edu/class/ee380/Abstracts/111026a-Hellan... )? Or is this another case of Dynamo / DynamoDB?

Hi ! Sargun, If you see Dharma's detailed reply which consists of papers and history from 2010. This is a different effort with a different focus.

Azure Cosmos DB has been many years in the making. Azure Cosmos DB started as “Project Florence” in late 2010 to address developer the pain-points faced by large scale applications inside Microsoft. Observing that the challenges of building globally distributed apps are not a problem unique to Microsoft, in 2015 we made the first generation of this technology available to Azure developers in the form of DocumentDB. Since that time, we’ve been steadily adding new capabilities both in the database engine as well as, larger distributed system components. Azure Cosmos DB is the result. It is the next big leap in globally distributed, at scale, cloud databases. As a part of this release of Azure Cosmos DB, DocumentDB customers, with their data, are automatically Azure Cosmos DB customers. They now have access to the new system and capabilities offered by Azure Cosmos DB today as well as, as we keep evolving the service.

The database engine design is inspired on LLAMA http://db.disi.unitn.eu/pages/VLDBProgram/pdf/research/p853-..., Bwtree - > https://pdfs.semanticscholar.org/7655/9c6cc259c6ab5baf7bd19d.... and schema-agnostic indexing techniques -> http://www.vldb.org/pvldb/vol8/p1668-shukla.pdf. Please note that these papers are significantly behind the current state of the implementation. The most crucial aspect that these papers dont cover is the integration of the database engine with the larger distributed system components of Cosmos DB including the resource governance, partition management, and the implementation of replication protocol /consistency models etc. Our goal is to publish all of the design specifications including TLA+ specs over time.

Re: Azure Cosmos DB, a globally distributed database

#112

Post by xapata »

GovindMS wrote: post #107
xapata wrote: post #104

Earlier quoted context omitted.

That didn't answer the question: will all regions be always consistent?

We would invite you to explore the consistency levels available in Cosmos DB. As a developer you can choose what makes most sense in distributed scenario. https://docs.microsoft.com/en-us/azure/documentdb/documentdb... Eventual is one end of the consistency. But then consistent prefix, session (if you can control) and bounded staleness all can play a role.

Tunable consistency. Very cool.

Re: Azure Cosmos DB, a globally distributed database

#113

Post by tracker1 »

smithkl42 wrote: post #66
tracker1 wrote: post #54

From the intro page[1]... Many of the descriptions comparing to NoSQL are wrong. There are plenty of NoSQL options that have similar features, though it isn't universal, it can and often is there. Cassandra, for example, probably does just as well in multi-zone/dc concurrency. Consistency options are also similarly tunable. Cockroach 1.0 was announced earlier as well. It's not that I don't appreciate the option. This…

Azure Table Storage? Uggh. Nasty. I've tried half a dozen times to use them, and every time I've given up. Great for write-only data that you never want to see again. Horrible for real-world querying.

Most of my usage has been mostly predictable single-key lookups, and some limited querying... The amount of data I've needed to put in it has always been fairly limited as well (generally under a million records per collection/table). It's a decent and really inexpensive option depending on your needs.

Re: Azure Cosmos DB, a globally distributed database

#114

Post by bshanks »

throwanem wrote: post #103
brianwawok wrote: post #101

Earlier quoted context omitted.

And white papers are full of it. Has random things like s Cassandra cluster needs 1 full time engineer per 100 nodes. Where do you come up with this stuff?

In my experience of Cassandra (working on an application client, not managing it), one FTE per hundred nodes is extremely generous.

Do you mean that you typically need more engineers, or less?

Re: Azure Cosmos DB, a globally distributed database

#115

Post by throwanem »

bshanks wrote: post #114
throwanem wrote: post #103

Earlier quoted context omitted.

In my experience of Cassandra (working on an application client, not managing it), one FTE per hundred nodes is extremely generous.

Do you mean that you typically need more engineers, or less?

The former. One FTE per dozen nodes is more in line with what I've seen in practice.

Re: Azure Cosmos DB, a globally distributed database

#116

Post by dharmashuklaMS »

lobster_johnson wrote: post #68

Is Cosmos related to the work on Corfu/CorfuDB [2] [1] in any way? [1] https://www.microsoft.com/en-us/research/publication/corfu-a... [2] https://github.com/CorfuDB/CorfuDB

Azure Cosmos DB is not related to the work on Corfu/CorfuDB.

Re: Azure Cosmos DB, a globally distributed database

#117

Post by dharmashuklaMS »

martinknafve wrote: post #21
oedenfield wrote: post #19

Earlier quoted context omitted.

Backup is automatic and restore is available within a time frame. https://docs.microsoft.com/en-us/azure/documentdb/documentdb...

> If the data is accidently dropped or corrupted, please contact Azure support within 8 hours Thats sounds less then optimal. :)

For supporting the "oops I accidentally deleted my table/collection/graph - how do I get it back?" scenario, Cosmos DB automatically takes periodic, full backups of all your data. Currently, we restore your backups on demand and its free - you just send a mail to us. In a few months, we will expose an API using which you can restore the data you accidentally deleted yourself.

Re: Azure Cosmos DB, a globally distributed database

#118

Post by dharmashuklaMS »

daxfohl wrote: post #90
dharmashuklaMS wrote: post #37

Hi, This is Dharma from Azure Cosmos DB team. We are super excited to make the service available today.We published the first of the series of technical blog posts here -> https://azure.microsoft.com/en-us/blog/a-technical-overview-... . Would love to answer any Cosmos DB questions.

Are graph ops/queries atomic? i.e. if you run a tree query on a tree-graph, at the same time you're re-parenting a treenode, is there a chance that the node could end up in the result tree twice or zero times? Also, if they're atomic, are they optimistic or pessimistic transactions? Also if they're atomic, does that mean queries done on the write server? (My understanding is that readonly transactions on read-replica…

Read only transactions existed in DocumentDB too. DocumentDB was a strict subset of the capabilities that Cosmos DB provides/existed underneath. Hence, read only transactions exist in Cosmos DB too. The first technical overview blog post attempted to provide you a high level overview. We are hoping to cover specific areas in either future blog posts, conference publications.

Re: Azure Cosmos DB, a globally distributed database

#119

Post by dharmashuklaMS »

ahelwer wrote: post #52

Designed with TLA+! :D Small interview with Leslie Lamport: https://techcrunch.com/2017/05/10/with-cosmos-db-microsoft-w... Hope Cosmos team releases a whitepaper on their experiences with the language. I'd heard snatches of gossip here and there that TLA+ was used inside Cosmos, but no concrete details. edit: apparently there's also a video of Lamport talking about this https://www.youtube.com/watch?v=L_PPKyAsR3w

Cosmos DB codebase consists of multi-million lines of C++ code. It is fully asynchronous and like all large scale (stateful) distributed systems, it is also extremely complex. The database engine of Cosmos DB (including the b-tree and the log structured storage engine) is fully latch free; the engine, resource governance subsystem, global distribution infrastructure, partition management etc -- are all deeply integrated. Each of these subsystems have extremely complex state machines which are hard to describe with the required degree of precision using the English language. Any correctness bug that gets introduced because of lack of precision, can potentially lead to data loss, corruption, partial or complete failures of the entire service. This is where TLA+ comes in.

Background: Dr. Leslie Lamport's work has been a constant source of inspiration for the Cosmos DB team. A few of the engineers on the team had learnt TLA+ initially on their own and started seeing its benefits. Subsequently, other members of the team started applying it as well. Leslie had also taught a fantastic class on TLA+ (across Microsoft), which engineers on the Cosmos DB team attended. It was a wonderful, once in a lifetime opportunity for the team to learn TLA+ from Leslie.

To be clear, Leslie personally didn't write any of the TLA+ specs for Cosmos DB. It was Cosmos DB engineers who wrote the TLA+ specs to specify & verify the design (incl. consistency models). The net result is that TLA+ made Cosmos DB a more robust system, which is crucial to offer strict and comprehensive SLAs encompassing availability, consistency, throughput and latency at the 99th percentile. Further, writing TLA+ specs for the five consistency models which we have exposed (as well as many that we have experimented, internally) enabled us to precisely define the semantics for each of the consistency models. This in-turn enables developers building apps on top of Cosmos DB, to rely on the well-defined semantics.

Hope this is helpful.

Re: Azure Cosmos DB, a globally distributed database

#120

Post by ahelwer »

dharmashuklaMS wrote: post #119
ahelwer wrote: post #52

Designed with TLA+! :D Small interview with Leslie Lamport: https://techcrunch.com/2017/05/10/with-cosmos-db-microsoft-w... Hope Cosmos team releases a whitepaper on their experiences with the language. I'd heard snatches of gossip here and there that TLA+ was used inside Cosmos, but no concrete details. edit: apparently there's also a video of Lamport talking about this https://www.youtube.com/watch?v=L_PPKyAsR3w

Cosmos DB codebase consists of multi-million lines of C++ code. It is fully asynchronous and like all large scale (stateful) distributed systems, it is also extremely complex. The database engine of Cosmos DB (including the b-tree and the log structured storage engine) is fully latch free; the engine, resource governance subsystem, global distribution infrastructure, partition management etc -- are all deeply integra…

Thanks, Dharma! I probably actually TA'd the TLA+ class which your engineers attended ;)

Still hope you write a whitepaper. The AWS paper is super valuable when making the case for TLA+ use in industry.

Post reply on HN