Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using logs to build a solid data infrastructure

blog.confluent.io

Post reply on HN
Print view View original

21–30 of 34 posts

Re: Using logs to build a solid data infrastructure

#21

Post by skybrian »

moatra wrote: post #6

if you want to build a new derived datastore, you can just start a new consumer at the beginning of the log, and churn through the history of the log, applying all the writes to your datastore. For high-throughput environments with lots of appends to the log, how do you get around the ever-increasing size of your log file? I know the traditional answer is to take a periodic snapshot and compact the previous data, but…

One idea is to shard the logs. By analogy with git: any given repo has a log of its commits, but you can have as many repos as you like.

It does limit throughput for any given shard, though, and then you're left with a distributed transaction problem to solve when you need to commit changes to objects in different repos.

Re: Using logs to build a solid data infrastructure

#22

Post by tristanz »

boredandroid wrote: post #4
tristanz wrote: post #2

Martin's talks and blog posts are aways awesome. I'm really excited to see how this plays out for real applications. The one thing I'm always somewhat confused by though is how a "totally ordered log" intersects with the reality of a partitioned log. The simplicity of a log seems to break down a bit when you partition. For instance, imagine I want to implement multi-key transactions on top of a distributed datastore.…

One thing to realize is that a partitioned log is a generalization of an unpartitioned log (i.e. if you set # partitions = 1 in a partitioned log you have an unpartitioned log). In Kafka the purpose of partitions is to provide computational parallelism not model entities in the world. So if you have 100m users you would map that into a number of partitions based on your computational parallelism (maybe 10-100 machine…

Yeah, this is a good way to look at it. But it's also my point. Partitions are about parallelism and don't always fit the data model or domain. While you can reduce the partitions to 1, this is limits parallelism. It's not always easy to design a partitioning scheme that preserves the replication semantics you want. And these semantics vary from having something totally ordered, to having something that you can replay on a very fine-grained level (to replicate to the client for instance.) Most discussions of the advantages of logs really emphasize how amazing a totally ordered log is for replication, but that's not actually what production deployments look like so you still need to think carefully about what happens when writes are being applied to your datastores without a clear order.

Re: Using logs to build a solid data infrastructure

#23

Post by Terr_ »

gbrits wrote: post #20

The proposed architecture really works well for me. I've used it for a couple of projects now. To throw around some terms for those interested in reading up/background: - The separation of writes (through log) from reads (through any of the consumers) is sometimes called: CQRS (command query responsibility separation) - having a centralized log as the defining store for updates/ change events is sometimes called: eve…

> having a centralized log as the defining store for updates/ change events is sometimes called: eventsourcing

To clarify slightly: Event Sourcing isn't just emitting events for others, it means that your system is re-reading its own log to derive current state from past-events.

For performance reasons you try to avoid this with stuff like periodic "snapshots", but the capability of has to be there.

Re: Using logs to build a solid data infrastructure

#24

Post by robbles »

I've really learned a lot from these Confluent posts about building log-based architectures, but I feel like they're rehashing the same high-level architectural ideas again and again.

I'm already sold on this idea, and would really love to see more posts that get into the nitty-gritty details of how to integrate Kafka, how to migrate an existing infrastructure, case studies, sample code, etc. It all seems very handwavy otherwise.

Re: Using logs to build a solid data infrastructure

#25

Post by tristanz »

Sphax wrote: post #7
tristanz wrote: post #2

Martin's talks and blog posts are aways awesome. I'm really excited to see how this plays out for real applications. The one thing I'm always somewhat confused by though is how a "totally ordered log" intersects with the reality of a partitioned log. The simplicity of a log seems to break down a bit when you partition. For instance, imagine I want to implement multi-key transactions on top of a distributed datastore.…

You could always map the document id or channel id to be always the same partition, then that particular document or channel log would be ordered. Seems like it would work.

The problem with this is there is no way to replay the log when clients connect without replaying the whole topic, which can be huge.

Re: Using logs to build a solid data infrastructure

#26

Post by saganus »

I have actually found myself thinking a lot about logs lately and how I can end up using them for a lot of problems, and are sometimes very simple to implement. But I always wondered if I was actually using the right tool for the job...

I had no idea they had such far-reaching implications and so many jobs where this is the right tool.

Neat!

Re: Using logs to build a solid data infrastructure

#27

Post by divtxt »

samwgoldman wrote: post #12

I've been really interested in this architecture since Jay Kreps' blog post on it. One part that I'm less clear on is how this fits in with request-response style communication between, say, a Web browser and a Web server. In a simple Web-app-writes-to-DB scenario, it's easy to read my writes, but with a async log processing system, how am I supposed to organize my code so I can read my writes and respond with useful…

You're quite right. Switching from a db to a log as the "master" loses the simplicity of the app-to-db model.

As the article says: "For now, a better option is to extract the log from a database" - i.e you use some tooling to generate a log from the db.

Indeed, you can now see tools that go in this direction usually by using the replication stream. (eg https://github.com/shyiko/mysql-binlog-connector-java and https://github.com/xstevens/decoderbufs)

Re: Using logs to build a solid data infrastructure

#28

Post by dwenzek »

samwgoldman wrote: post #12

I've been really interested in this architecture since Jay Kreps' blog post on it. One part that I'm less clear on is how this fits in with request-response style communication between, say, a Web browser and a Web server. In a simple Web-app-writes-to-DB scenario, it's easy to read my writes, but with a async log processing system, how am I supposed to organize my code so I can read my writes and respond with useful…

To connect a request/response system to a log-oriented system such as presented here, an option is to use the request/reply server as a proxy for the log-oriented one :

  - producing the requests to the log-oriented system,

  - consuming the responses at some endpoint/topic of the dataflow,

  - linking responses to requests using some id propagated along the whole dataflow,

  - dealing with a cache of pending requests, asynchronous responses and timeout
In practice, to keep things manageable - latency and count of pending connections, we can't always wait for the very end of the request processing dataflow; but we can at least choose an intermediate log/topic where progress is sufficient to forge a response or to reply with 202 accepted request.

Re: Using logs to build a solid data infrastructure

#29

Post by compleat-ang »

itistoday2 wrote: post #3

Could someone compare Apache Kafka to Eris Industry's centralized blockchain model?

Disclaimer I am the CEO of Eris Industries.

Firstly, we are not doing centralized blockchains, we're doing permissioned blockchains which work most efficiently when actors within the system have been identified. These blockchains are simply one element of the Eris Blockchain Application platform we are developing.

Comparing a smart contract enabled blockchain to a distributed messaging broker is a difficult proposition as they are very much apples and oranges in their capabilities, usage, and goals.

That said let me give it a go.

## Similarities

* Kafka and eris:db both work on a distributed model * Kafka and eris:db both are inherently scalable and persist their data * Kafka and eris:db both include a distributed messaging layer optimized for their system which is not http

## Differences

* eris:db persists using merkle-ized encoding schemes with all data having been digitally signed via a node using an ECC-PKI and verified by a logic gateway (known commonly as a smart contract) prior to being persisted -- eris:db further builds logs into blocks and chains those together using traditional blockchain techniques; kafka uses a relatively sophisticated, but not inherently as verifiable "traditional" logging mechanism * eris:db, as with any blockchain, includes a [fork choice rule](https://eng.erisindustries.com/blockchain/2015/04/30/on-bloc...) for resolving differences within the nodes on the network as to prior history as well as adding a layer of byzantine fault tolerance; from my (admittedly cursory) overview of kafka while it is distributed it is not byzantine fault tolerant * kafka is fast and meant as a near real time message broker; eris:db is not built for speed and sacrifices some availability for consistency

## Overall

Message brokers are great complements to blockchains actually as they can provide an additional speed layer for messages which blockchains are not particularly good at.

Re: Using logs to build a solid data infrastructure

#30

Post by Sphax »

tristanz wrote: post #25
Sphax wrote: post #7

Earlier quoted context omitted.

You could always map the document id or channel id to be always the same partition, then that particular document or channel log would be ordered. Seems like it would work.

The problem with this is there is no way to replay the log when clients connect without replaying the whole topic, which can be huge.

True, hadn't thought about that.
Post reply on HN