Earlier quoted context omitted.
I have been involved in using a system built like this. All I can say is... It feels like you're building a database out of an event stream. A shitty one at that... Basically the write log part, only without a way to apply that state reliably like a real database. So you have to keep the log around basically forever. It's like you're in the middle of a DB recovery all the time. After insane amounts of research and de…
Great insightful comment. I came to the same conclusion a number of years ago. We did something about it - we built a new Hadoop platform around a not very well known distributed, in-memory, open-source database - MySQL Cluster (NDB). It is not the MySQL Server you think you know. It is an in-memory OLTP engine used by most network operators as a call subscriber DB. It can handles millions reads or writes/sec on comm…
Designing Data-Intensive Applications
41–50 of 61 posts
Re: Designing Data-Intensive Applications
#42Good book! It's only at the introductory level, but I liked that it lacks bias and does a survey of the field.
do you have a reference with more advanced level? will be nice if you share it. thanks
A book that tries talk broadly necessarily has less room to go into detail. For advanced level references, one will have to drill down to sources that are more specific.
Thankfully, this book cites its sources and extensively documents references, and the author even maintains the reference links[0].
Re: Designing Data-Intensive Applications
#43I'm about 40% through this book, and it's been a stellar read so far. Using clear, well thought-out language it gets straight to business – chapter after chapter. Highly recommended!
Re: Designing Data-Intensive Applications
#44In "The Future of Data Systems", the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work like de-duping, committing the data to a RDBMS, invalidating caches, updating search indexes, etc. The application might read state directly from log updates, or have a system to sync w/ some sort of atomic state (e.g. RethingDB changefeeds). The architecture…
We built a kind of hybrid infrastructure around this idea.
Any component sends its data into kafka-esque (we're using combination of NATS and PubSub) pipeline where series of workers read, process and write data into our RDBMS which is the ultimate source of the truth.
This allowed us to run a double scale system, where all components are running on their own pace and the RDBMS is running on its own. There is some inherited delay in the data propagation, but it works for service like our (search engine) that doesn't require real-time exposure of newly acquired data.
This design also allows for a frugality as the RDBMS cluster is only scaled based on long term trends and now short term bursts. We also are able to buy committed usage for the cluster as we've great predictability in its growth.
Re: Designing Data-Intensive Applications
#45I have this book as well. As a more seasoned systems engineer, I think it covers a lot of the groundwork needed for people newer to the distributed disciplines.
Re: Designing Data-Intensive Applications
#46I was a bit dismayed that this was about the technical design of data-intensive applications, not about their UX design. There still seems to be a huge gap in the latter.
Take a look at Tufte's book The Visual Display of Quantitative Information
Re: Designing Data-Intensive Applications
#47I was a bit dismayed that this was about the technical design of data-intensive applications, not about their UX design. There still seems to be a huge gap in the latter.
Take a look at Tufte's book The Visual Display of Quantitative Information
Interaction works both ways (input and output). Tufte's work is generally in just one direction (visualization as output). It isn't very applicable to application as much as it is to presentation (think: how did we figure out what to present in that PowerPoint inthe first place?).
Re: Designing Data-Intensive Applications
#48I've read it and highly recommend it. Does anyone know books that are similar in style? (conceptual, showcasing different solutions to problems and their tradeoffs, high signal-to-noise)
Same question here. I am still reading this book but the way the author combines the concepts with the practices and the contents are structured really inspire me to keep reading. (Usually I gave up easily)
Re: Designing Data-Intensive Applications
#49Earlier quoted context omitted.
I have been involved in using a system built like this. All I can say is... It feels like you're building a database out of an event stream. A shitty one at that... Basically the write log part, only without a way to apply that state reliably like a real database. So you have to keep the log around basically forever. It's like you're in the middle of a DB recovery all the time. After insane amounts of research and de…
Great insightful comment. I came to the same conclusion a number of years ago. We did something about it - we built a new Hadoop platform around a not very well known distributed, in-memory, open-source database - MySQL Cluster (NDB). It is not the MySQL Server you think you know. It is an in-memory OLTP engine used by most network operators as a call subscriber DB. It can handles millions reads or writes/sec on comm…
Hopsworks looks like it might be exactly what I need, I do typical data science work for small to small-medium data and wanted to start properly playing with spark on a HDFS store.
Currently most work is just done in R/Python in VM's on a small proxmox cluster (where only 1 node is always on) but I'd like start gently moving to spark, run the stack on a single node and scale on demand.
Is Hopsworks for me, does this approach even make sense for such small data or am I crazy? Thanks for your response!
Re: Designing Data-Intensive Applications
#50I was a bit dismayed that this was about the technical design of data-intensive applications, not about their UX design. There still seems to be a huge gap in the latter.
Probably you interpreted the word "design" with its narrower meaning of visual(ly-oriented) design, rather than, say architecting data-intensive applications.
As with another poster, Edward Tufte's books came to mind - though it's about visual presentation of information, not user interface/experience design.
I've also felt that there's an unmet demand for books that provide a thorough overview of UI/UX design patterns, especially the way this book (Designing Data-Intensive Applications) does for its domain.