I've read it and highly recommend it.
Does anyone know books that are similar in style? (conceptual, showcasing different solutions to problems and their tradeoffs, high signal-to-noise)
Designing Data-Intensive Applications
11–20 of 61 posts
Re: Designing Data-Intensive Applications
#12In "The Future of Data Systems", the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work like de-duping, committing the data to a RDBMS, invalidating caches, updating search indexes, etc. The application might read state directly from log updates, or have a system to sync w/ some sort of atomic state (e.g. RethingDB changefeeds).
The architecture seems to solve two big problems
* Scaling RDBMS (there are solutions like Cloud Spanner but they rely heavily on Google's proprietary network for low latency and are expensive as balls)
* Keeping downstream systems in sync. A lot of companies have gone with a "Tail the RDBMS" of some kind, e.g. writing the MySQL binlog to a Kafka queue and having downstream consumers read from that, but this seems like a more elegant solution.
Are there any examples or experiences of people working with systems like this? What are some downsides, challenges, and actual benefits?
Re: Designing Data-Intensive Applications
#13I'm actually midway through this book and I definitely recommend it. The content manages to be both approachable and enlightening. I'm a backend software engineer with the latitude to architect systems at my company and the content so far has given me a stronger foundation for choosing how and where to manage our data. I really enjoy the mini-dives into the structures and decisions supporting the common databases you…
"..better equipped to evaluate the tools at our disposal for a given job"
That's a great review - I learned about the book recently, and it sounds like exactly what I need right now, to make a more informed decision about database choices.
Re: Designing Data-Intensive Applications
#14I'm halfway through this book and I highly recommend it. I'd say its required reading for anyone who wants to be an architect.
Re: Designing Data-Intensive Applications
#15It seems most of my comments these days are singing the praises of this book. It should be practically mandatory reading for anyone in the field. It ties concepts together and builds understanding in a way that doesn't rely on specific technologies. I wish it had been written and that I could have done a whole course on this in grad school.
Re: Designing Data-Intensive Applications
#16This book is a modern survey on practical distributed systems. I knew bits of pieces of the material going in, but the way it was brought together was just masterful.
It will not appeal to the absolute novice to be sure. But for anyone else who has worked on systems for moving data (ETL, streams) and storing data (databases and other data stores), this book will show you how things (probably stuff you've done bits of pieces of) fit together and expound on the few foundational big ideas that makes everything cohere. Once you've understood that, you are on your way to designing data systems that are much cleaner and more scalable.
My experience reading this book is a bit like that of a tradesperson going back to school to learn theory, and after being enlightened, coming away with a new understanding of how to put together theory and practice to better his craft.
I chanced upon this book through an excellent interview Martin Kleppmann did on Software Engineering Daily podcast. If you want the talk-show-host cliff notes version of what the book is about, you should listen to this particular episode:
https://softwareengineeringdaily.com/2017/05/02/data-intensi...
Re: Designing Data-Intensive Applications
#17In "The Future of Data Systems", the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work like de-duping, committing the data to a RDBMS, invalidating caches, updating search indexes, etc. The application might read state directly from log updates, or have a system to sync w/ some sort of atomic state (e.g. RethingDB changefeeds). The architecture…
HI I am working in one is called omniql, I am lone developer, so probably it will take time, but there are much of the what they said in the book planned for omniql, is based in observable pattern so it will have live queries by default, it will use system like kafka, nats, Redis or any message system, for communicate the partitions of the distributed databases and do computation that can be merged hierarchically, the computation are done by `serverless function` in any language, is a model that can be extended to ui interface to and create a framework very similar to mobx , but more lightwave, and ui components that can automatically query their data requirement to the server just when they are observed, I am creating a binary protocol of communication for this called delta https://github.com/nebtex/delta, that is perfect for this system an incremental computing, is a really a big project, but I hope to have something working for the next 6 months.
Re: Designing Data-Intensive Applications
#18I know a lot of people here already praise the book... but I have to do that as well. It's a great overview and it explains all the relevant concepts really nicely. Thanks, Martin!
Re: Designing Data-Intensive Applications
#19In "The Future of Data Systems", the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work like de-duping, committing the data to a RDBMS, invalidating caches, updating search indexes, etc. The application might read state directly from log updates, or have a system to sync w/ some sort of atomic state (e.g. RethingDB changefeeds). The architecture…
We're building an open source database like this. It's document-oriented and relies on a transaction log core (currently using Postgres, but it's pluggable), that feeds into a query subsystem (currently layered on top of Elasticsearch, but also pluggable).
The transaction log encourages small logical "patches" (set a field, increment a number, replace a substring, move an array element, etc.) that are applied in sequence but can be disentangled by clients to generate a consistent UI, and also used to resolve conflicts between multiple distributed writers. You can also follow the log through the gRPC and HTTP APIs, and you can register "watches" on queries that cause matching changes to trigger an event.
While the transaction log is the underlying data model, we also maintain a consistent view of the current version so that you can use it as a document database with classical CRUD operations. So on the surface it's a lot like Firebase or CouchDB, except you get things like joins and schemas.
Drop me an email (see profile) and I can send you some links.
Re: Designing Data-Intensive Applications
#20I've been massively recommending this book. I think it's very unlikely a backend engineer can avoid having their applications on multiple servers. This book does a wonderfully clear and practical description of the issues with distributed systems, and tools and techniques to address those issues. It's written in a way that is very accessible to someone without a traditional comp sci background.
I was excited about this book because there is a gap in distributed systems books. I feel like there's are a large amount of blogs but most of the books available on amazon are text books and/or include heavy math.