51–60 of 61 posts
I wrote a more detailed review at http://horia141.com/designing-data-intensive-applications-re... for the interested.
I was a bit dismayed that this was about the technical design of data-intensive applications, not about their UX design. There still seems to be a huge gap in the latter.
Probably you interpreted the word "design" with its narrower meaning of visual(ly-oriented) design, rather than, say architecting data-intensive applications. As with another poster, Edward Tufte's books came to mind - though it's about visual presentation of information, not user interface/experience design. I've also felt that there's an unmet demand for books that provide a thorough overview of UI/UX design patter…
It is an overloaded term for sure, but the title of the book caught my attention, it is only when I read the article that I realized it was talking about the design of application implementations, not the UX.
I've read it and highly recommend it. Does anyone know books that are similar in style? (conceptual, showcasing different solutions to problems and their tradeoffs, high signal-to-noise)
A few shorter books have come out that try to touch different approaches that I've liked: "Seven Languages in Seven weeks" -- and the series has also gained 7 databases, concurrency models and web frameworks.
Finally, there's this anthology where OSS authors described what they did in their applications, so there's a ton of practical information http://aosabook.org/en/index.html
In "The Future of Data Systems", the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work like de-duping, committing the data to a RDBMS, invalidating caches, updating search indexes, etc. The application might read state directly from log updates, or have a system to sync w/ some sort of atomic state (e.g. RethingDB changefeeds). The architecture…
I have been involved in using a system built like this. All I can say is... It feels like you're building a database out of an event stream. A shitty one at that... Basically the write log part, only without a way to apply that state reliably like a real database. So you have to keep the log around basically forever. It's like you're in the middle of a DB recovery all the time. After insane amounts of research and de…
I dislike to lose ACID. Mainly because my apps are all about financial/business stuff.
My ideal DB right know will be alike:
Incoming data ->
- Write WAL (disk)
- Convert to Logical JSON-like structure (memory). For Consisten API
- Pre-Triggers (BLOCK)
- Build caches, (secondary?) indexes, etc (NON-BLOCK)In "The Future of Data Systems", the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work like de-duping, committing the data to a RDBMS, invalidating caches, updating search indexes, etc. The application might read state directly from log updates, or have a system to sync w/ some sort of atomic state (e.g. RethingDB changefeeds). The architecture…
I have been involved in using a system built like this. All I can say is... It feels like you're building a database out of an event stream. A shitty one at that... Basically the write log part, only without a way to apply that state reliably like a real database. So you have to keep the log around basically forever. It's like you're in the middle of a DB recovery all the time. After insane amounts of research and de…
Earlier quoted context omitted.
I have been involved in using a system built like this. All I can say is... It feels like you're building a database out of an event stream. A shitty one at that... Basically the write log part, only without a way to apply that state reliably like a real database. So you have to keep the log around basically forever. It's like you're in the middle of a DB recovery all the time. After insane amounts of research and de…
Very interesting. What do you think about CockroachDB ( https://github.com/cockroachdb/cockroach )? Is that serious enough to the challenge, from outside Google?
The second major issue is technical. Building something like Spanner requires a very accurate time source that absolutely will not skew. Ever. This is how Google avoids partitions and essentially breaks CAP theory. Perfect time gives you globally accurate timestamps without exchanging data. Distributed transactions without locks or shared state, just usually benign contention.
They're not that hard to build, just no demand. Possibly some issues with ITAR preventing such accurate clocks from becoming commodity hardware? It could easily lead to extremely accurate IMU's which are definitely limited. Not sure, but that's what I ran into researching fibre optic gyros. Atomic clocks could probably be built on SMT scale for a few cents IMO.
As usual, it seems Google is already doing this and has been for years. We either need to wait for the trickle down that Google thankfully does after about 5 years... Or get some deep pocketed tech behemoth to foot the bill for everyone else
Earlier quoted context omitted.
Very interesting. What do you think about CockroachDB ( https://github.com/cockroachdb/cockroach )? Is that serious enough to the challenge, from outside Google?
Two problems. The name is a nightmare for any large company. Countless people have brought this up and the dev team is apparently deaf to the issue. Idiots. The second major issue is technical. Building something like Spanner requires a very accurate time source that absolutely will not skew. Ever. This is how Google avoids partitions and essentially breaks CAP theory. Perfect time gives you globally accurate timesta…
From their 2017 paper on Spanner and CAP:
> To the extent there is anything special, it is really Google’s wide-area network, plus many years of operational improvements, that greatly limit partitions in practice, and thus enable high availability.
[0] https://static.googleusercontent.com/media/research.google.c...
In "The Future of Data Systems", the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work like de-duping, committing the data to a RDBMS, invalidating caches, updating search indexes, etc. The application might read state directly from log updates, or have a system to sync w/ some sort of atomic state (e.g. RethingDB changefeeds). The architecture…
I have been involved in using a system built like this. All I can say is... It feels like you're building a database out of an event stream. A shitty one at that... Basically the write log part, only without a way to apply that state reliably like a real database. So you have to keep the log around basically forever. It's like you're in the middle of a DB recovery all the time. After insane amounts of research and de…
Many software applications, especially web application servers, are effectively a set of data structures updated incrementally from an incoming data stream, and then served to users. The analogy of many applications to a database (or an interpreter) is an accurate one and, in my personal experience, useful as well.
> So you have to keep the log around basically forever.
Different logs can have different retention periods, and you linearize across different logs by using a single writer per timeline. Many domains allow for enough splitting of timelines to enable effective parallel processing without compromising consistency (consider the case of distinct customer organizations using a time-tracking product - there's no reason they need to be able to write to the same database, and volume/contention within any individual organization is likely to be low, allowing for maintained performance).
Earlier quoted context omitted.
Great insightful comment. I came to the same conclusion a number of years ago. We did something about it - we built a new Hadoop platform around a not very well known distributed, in-memory, open-source database - MySQL Cluster (NDB). It is not the MySQL Server you think you know. It is an in-memory OLTP engine used by most network operators as a call subscriber DB. It can handles millions reads or writes/sec on comm…
Hopsworks looks like it might be exactly what I need, I do typical data science work for small to small-medium data and wanted to start properly playing with spark on a HDFS store. Currently most work is just done in R/Python in VM's on a small proxmox cluster (where only 1 node is always on) but I'd like start gently moving to spark, run the stack on a single node and scale on demand. Is Hopsworks for me, does this…
Earlier quoted context omitted.
Hopsworks looks like it might be exactly what I need, I do typical data science work for small to small-medium data and wanted to start properly playing with spark on a HDFS store. Currently most work is just done in R/Python in VM's on a small proxmox cluster (where only 1 node is always on) but I'd like start gently moving to spark, run the stack on a single node and scale on demand. Is Hopsworks for me, does this…
Yes, Hopsworks can run on anything from 1 server to 1000s. We are finalizing the first proper release now - Jupyter support, tensorflow, pyspark, sparkr, python-kernel for jupyter too,