I was very impressed with the quality and depth of the book. I also appreciated how unbiased it felt. I feel that so many books tout approach/technology X as the best approach, but Martin really did a great job at explaining trade offs of various approaches and possible solutions to the problems they introduce.
Highly recommended--especially considering getting the Amazon price point (~ $25).
Designing Data-Intensive Applications
21–30 of 61 posts
Re: Designing Data-Intensive Applications
#22In "The Future of Data Systems", the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work like de-duping, committing the data to a RDBMS, invalidating caches, updating search indexes, etc. The application might read state directly from log updates, or have a system to sync w/ some sort of atomic state (e.g. RethingDB changefeeds). The architecture…
We're building an open source database like this. It's document-oriented and relies on a transaction log core (currently using Postgres, but it's pluggable), that feeds into a query subsystem (currently layered on top of Elasticsearch, but also pluggable). The transaction log encourages small logical "patches" (set a field, increment a number, replace a substring, move an array element, etc.) that are applied in sequ…
Curious how this compares to Couchbase + N1QL, since that gets you a document db that also supports full SQL.
Re: Designing Data-Intensive Applications
#23Hands down the best book in this area. Took me some time to read the book cover to cover but it is definitely worth it
Re: Designing Data-Intensive Applications
#24In "The Future of Data Systems", the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work like de-duping, committing the data to a RDBMS, invalidating caches, updating search indexes, etc. The application might read state directly from log updates, or have a system to sync w/ some sort of atomic state (e.g. RethingDB changefeeds). The architecture…
We're building an open source database like this. It's document-oriented and relies on a transaction log core (currently using Postgres, but it's pluggable), that feeds into a query subsystem (currently layered on top of Elasticsearch, but also pluggable). The transaction log encourages small logical "patches" (set a field, increment a number, replace a substring, move an array element, etc.) that are applied in sequ…
this looks very similar what I am doing, the difference appears to be that in delta a number of operations is just a subset that can be done in your system. for example it only allows the set operation on scalars/string fields and push operation on vectors, also the delete operations are placed in a way in which automatically let the merge algorithm to know if it needs to check past version in order to compute the current version of the resource, it also allows to compact all the history of a resource in distributed way.
Re: Designing Data-Intensive Applications
#25Really loving the book. Definitely a must read for the data engineers. I would add that redbook.io is another great read.
Re: Designing Data-Intensive Applications
#26Good book! It's only at the introductory level, but I liked that it lacks bias and does a survey of the field.
Re: Designing Data-Intensive Applications
#27In "The Future of Data Systems", the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work like de-duping, committing the data to a RDBMS, invalidating caches, updating search indexes, etc. The application might read state directly from log updates, or have a system to sync w/ some sort of atomic state (e.g. RethingDB changefeeds). The architecture…
We have built something like this on top of a distributed in-memory database. The changelog of the distributed in-memory database is the 'source of truth' that downstream clients would like to consume. The research problem is that the changelog is batches of transactions that complete within a given epoch (time). Transactions may execute in parallel on different data nodes within an epoch and there is no global ordering over them (no global time, only logical time). Downstream clients, however, may require an ordering over the transactions, so we worked on a solution to this problem when the database stores metadata for a filesystem. Our solution ensures we don't have to serialize the transactions to ensure a global ordering, and provides strong eventual consistency to clients.
Re: Designing Data-Intensive Applications
#28I am a backend engineer who has been working in the mobile gaming space for many years now. Most of the focus today for mobile gaming backends is to scale to millions of players while offering low latency and real-time interactions. Have been following the development of this book through its beta and having read it now, I think it is fair to say that this book is worth its weight in gold. I can relate directly to all the challenges faced over the years in implementing a real time distributed database _like_ functionality with a high read _and_ write throughput, both custom and off the shelf. Extremely well written.
Re: Designing Data-Intensive Applications
#29Earlier quoted context omitted.
We're building an open source database like this. It's document-oriented and relies on a transaction log core (currently using Postgres, but it's pluggable), that feeds into a query subsystem (currently layered on top of Elasticsearch, but also pluggable). The transaction log encourages small logical "patches" (set a field, increment a number, replace a substring, move an array element, etc.) that are applied in sequ…
this looks very similar what I am doing, the difference appears to be that in delta a number of operations is just a subset that can be done in your system. for example it only allows the set operation on scalars/string fields and push operation on vectors, also the delete operations are placed in a way in which automatically let the merge algorithm to know if it needs to check past version in order to compute the cu…
Not sure what you mean by subset. In our case, a single transaction contains one or more document operations (create, delete, etc.), one of which is a "patch" operation that applies a fine-grained transformation. The transformations use an extended version of JSONPath in order to be able to target deep tree nodes as well as apply transformations to multiple fields (e.g. authors[0].publications[*].title). Operations such as set, increment etc. are specific to the data type of the value being transformed and will fail if the data type does not support the transformation function.
Re: Designing Data-Intensive Applications
#30I highly recommend this book. Its a must read book IMHO. Especially useful when trying to select a database.