Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Designing Data-Intensive Applications

dataintensive.net

Post reply on HN
Print view View original

31–40 of 61 posts

Re: Designing Data-Intensive Applications

#31

Post by lobster_johnson »

HodGreeley wrote: post #22
lobster_johnson wrote: post #19

Earlier quoted context omitted.

We're building an open source database like this. It's document-oriented and relies on a transaction log core (currently using Postgres, but it's pluggable), that feeds into a query subsystem (currently layered on top of Elasticsearch, but also pluggable). The transaction log encourages small logical "patches" (set a field, increment a number, replace a substring, move an array element, etc.) that are applied in sequ…

Curious how this compares to Couchbase + N1QL, since that gets you a document db that also supports full SQL.

In our case, we wanted something more compact, so a query looks more like GraphQL (it's technically a superset of JSON, but it usually doesn't look like JSON). Joins, for example, are just nested declarations that list which attributes on the joined collections to fetch. Here's a query that shows many of the features:

    *[_type == "blogpost" && published == true] {
      _id, title,

      "bodyExcerpt": regexReplace(body, "(.+?)(\. |$)", "\\1"),
      
      author -> { name, category },

      "sectionNames": sections -> name
    } | order(createdAt desc)[0..20]
I don't know much about N1QL, I will have to read more about it.

Re: Designing Data-Intensive Applications

#32

Post by erwan »

jamesblonde wrote: post #27
mshenfield wrote: post #12

In "The Future of Data Systems", the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work like de-duping, committing the data to a RDBMS, invalidating caches, updating search indexes, etc. The application might read state directly from log updates, or have a system to sync w/ some sort of atomic state (e.g. RethingDB changefeeds). The architecture…

We have built something like this on top of a distributed in-memory database. The changelog of the distributed in-memory database is the 'source of truth' that downstream clients would like to consume. The research problem is that the changelog is batches of transactions that complete within a given epoch (time). Transactions may execute in parallel on different data nodes within an epoch and there is no global order…

Can you elaborate on the solution? Do you use an aggregator node, or have you massaged real-time reqs to allow for an aggregation period on the client side? I'm super curious to hear how you proceeded!

Re: Designing Data-Intensive Applications

#33

Post by criloz2 »

lobster_johnson wrote: post #29
criloz2 wrote: post #24

Earlier quoted context omitted.

this looks very similar what I am doing, the difference appears to be that in delta a number of operations is just a subset that can be done in your system. for example it only allows the set operation on scalars/string fields and push operation on vectors, also the delete operations are placed in a way in which automatically let the merge algorithm to know if it needs to check past version in order to compute the cu…

Not sure what you mean by subset. In our case, a single transaction contains one or more document operations (create, delete, etc.), one of which is a "patch" operation that applies a fine-grained transformation. The transformations use an extended version of JSONPath in order to be able to target deep tree nodes as well as apply transformations to multiple fields (e.g. authors[0].publications[*].title). Operations s…

lobster nice, sorry for my bad english, is not native language :), in delta each message can be just appended to their past versions (binary format, not parsing required and based on flatbuffers) , in the same memory or disk region, which I have called a superposition (a superstition can represent any resource) you can see an image here https://github.com/nebtex/delta/blob/master/docs/version-lin..., each time that you append a message the tables in the new message are linked to their immediate past version if they exist (you can see tables like nested messages on protobuff), but if a deletion message of that table is found it does not create the link, when the program tries to find a field in a table it lookups in the latest message firsts and then go to past version till it found something, I believe that this should be fast due to how cache works in the modern computer architectures (still not tested), the superposition can be compacted to a single message, in order to free space, also the compaction can run in parallel if you have a lot of messages, for example is possible to maintain all the mutation of the db in a distribuited log, and if people need to recreate the latest state or any past state, should be a fast operation due that is possible to use all the nodes availables. I need to work in a better doc for sure, if someone has some recommendation to give me it will really nice, hehe.

Re: Designing Data-Intensive Applications

#34

Post by slackingoff2017 »

mshenfield wrote: post #12

In "The Future of Data Systems", the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work like de-duping, committing the data to a RDBMS, invalidating caches, updating search indexes, etc. The application might read state directly from log updates, or have a system to sync w/ some sort of atomic state (e.g. RethingDB changefeeds). The architecture…

I have been involved in using a system built like this. All I can say is... It feels like you're building a database out of an event stream.

A shitty one at that... Basically the write log part, only without a way to apply that state reliably like a real database. So you have to keep the log around basically forever. It's like you're in the middle of a DB recovery all the time.

After insane amounts of research and deep thought my personal opinion is that this is the wrong way to do scalable systems. Event sourcing and eventual consistency are taking industry for a ride in the wrong direction.

In my quest to find a better way I found some research/leaks/opinions of Googlers, and I think they're right. Even Netflix admits that using eventual consistency means they have to build systems that go around and "fixup" data that ends up in bad states. Ew. Service RPC loops in any such systems are Pandora's box. Are these calls getting the most recently updated version of the data? Nobody knows. Even replaying the event log can't save you, the log may be strongly ordered but the data state between services that call each other is party determined by timing. Undefined behavior.

You'll notice that LinkedIn/Netflix/Uber etc all seem to be building their systems using this pattern. Who is conspicuously absent? Google. The father of containers, VM's, and highly distributed systems is mum.

Researching Google's systems gives some fascinating answers to the problem of distributed consistency, a solution I'm stunned hasn't seen more attention. Google decided as early as 2005 that eventually consistent systems were too hard to use and manage. All of their databases, BigTable, MegaStore, Spanner, F1... They're all strongly consistent in certain ways.

How does Google do it? They make the database the source of truth. Service RPC calls either fail or succeed immediately. Service call loops, while bad for performance, produce consistent results. Failures are easy to find because data updates either succeed or fail immediately, not in some unbouded future time.

The rest of the industry is missing the point of microservices IMO. Google's massively distributed systems are enabled largely by their innovative database designs. The rest of the industry is trying to replicate the topography of Google's internal systems without understanding what makes them work well.

For microservices to be realistically usable for most use cases we need someone to come up with decent competition to Google's database systems. When you have a transactional distributed database all the problems with data spread across multiple services goes away.

HBase was a good attempt but doesn't get enough love. A point missed in the creation of HBase, that becomes clear when reading the papers about MegaStore and Spanner, is that it wasn't designed to be used as a data store by itself. Instead, it has the minimal features needed to build a MegaStore on top of it. The weirder features of HBase/BigTable (like keeping around 3 copies of changed data, and row level atomicity without transactions) are clearly designed to make it possible to build a database on top of it.

Unfortunately nobody thus far has taken up that challenge, and outside Google were all stuck with shitty databases that Google tossed away a decade ago.

Re: Designing Data-Intensive Applications

#35

Post by jamesblonde »

erwan wrote: post #32
jamesblonde wrote: post #27

Earlier quoted context omitted.

We have built something like this on top of a distributed in-memory database. The changelog of the distributed in-memory database is the 'source of truth' that downstream clients would like to consume. The research problem is that the changelog is batches of transactions that complete within a given epoch (time). Transactions may execute in parallel on different data nodes within an epoch and there is no global order…

Can you elaborate on the solution? Do you use an aggregator node, or have you massaged real-time reqs to allow for an aggregation period on the client side? I'm super curious to hear how you proceeded!

No, we're using a table as a 'queue' in the db. Client are decoupled and if our middleware is offline, it can restart and catch up by draining the table. Transactions over the table ensure the consistency and integrity of the 'queue'. We provide at-least-once semantics to downstream apps, by exploiting transactions in the DB. Actually, we have one sink in the DB itself and we get exactly once semantics for that. The work in under submission with anonymous reviewing, so can't elaborate massively on everything else. Performance numbers are good, though, a sigle server can forward more than 10k ops/sec from the database changelog to a downstream db used for freetext search.

Re: Designing Data-Intensive Applications

#36

Post by ChemicalWarfare »

mshenfield wrote: post #12

In "The Future of Data Systems", the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work like de-duping, committing the data to a RDBMS, invalidating caches, updating search indexes, etc. The application might read state directly from log updates, or have a system to sync w/ some sort of atomic state (e.g. RethingDB changefeeds). The architecture…

From my personal experience (which probably has to do with working in industries where incoming data has a wide array of clients who need it ASAP) - the biggest challenge with any distributed DB system is the problem of "reading your own writes" and how the system approaches it. Not to be confused with tx isolation levels.

It's a balancing act between two extremes - locking everything down and ensuring the tx has been committed on all nodes/propagated to all consumers on one hand and sending an "ack" back to the client with a loose promise of eventual consistency on the other.

Re: Designing Data-Intensive Applications

#37

Post by jamesblonde »

slackingoff2017 wrote: post #34
mshenfield wrote: post #12

In "The Future of Data Systems", the author imagines a system where the application writes events to a Kafka-like distributed log. Consumers of the log do work like de-duping, committing the data to a RDBMS, invalidating caches, updating search indexes, etc. The application might read state directly from log updates, or have a system to sync w/ some sort of atomic state (e.g. RethingDB changefeeds). The architecture…

I have been involved in using a system built like this. All I can say is... It feels like you're building a database out of an event stream. A shitty one at that... Basically the write log part, only without a way to apply that state reliably like a real database. So you have to keep the log around basically forever. It's like you're in the middle of a DB recovery all the time. After insane amounts of research and de…

Great insightful comment. I came to the same conclusion a number of years ago. We did something about it - we built a new Hadoop platform around a not very well known distributed, in-memory, open-source database - MySQL Cluster (NDB). It is not the MySQL Server you think you know. It is an in-memory OLTP engine used by most network operators as a call subscriber DB. It can handles millions reads or writes/sec on commodity hardware (it has been benched at 200m reads/sec, about 80m writes/sec). It has transactions (read committed isolation level) and row-level locks. It supports efficient cross-partition transactions using one transaction coordinator per database node (up to 48 of them). You can build scalable apps with strong consistency if you can write apps with primary key ops and partition-pruned index scans. We managed to scale out HDFS by 16X with this technique. Since then, we have been doing like you suggested - we built a microservices architecture for Hadoop called Hopsworks around the transactional distributed database. All the evils of eventually consistency go away - systems like Apache Ranger/Sentry become just tables in the DB. More reading is available here: http://www.hops.io/?q=content/news-events

Re: Designing Data-Intensive Applications

#40

Post by teacpde »

g___ wrote: post #11

I've read it and highly recommend it. Does anyone know books that are similar in style? (conceptual, showcasing different solutions to problems and their tradeoffs, high signal-to-noise)

I have the same question, I thoroughly enjoy the book and would love to see similar recommendations. Maybe it could be a good ask HN post.
Post reply on HN