Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using logs to build a solid data infrastructure

blog.confluent.io

Post reply on HN
Print view View original

11–20 of 34 posts

Re: Using logs to build a solid data infrastructure

#11

Post by Sphax »

moatra wrote: post #6

if you want to build a new derived datastore, you can just start a new consumer at the beginning of the log, and churn through the history of the log, applying all the writes to your datastore. For high-throughput environments with lots of appends to the log, how do you get around the ever-increasing size of your log file? I know the traditional answer is to take a periodic snapshot and compact the previous data, but…

There's a log compaction cleanup policy yes. Never used it myself but if I'm not mistaken it works like this: for each message you send to Kafka, you set a key with it. When Kafka does log compaction, it keeps only the last value for each key.

The other cleanup policy is to just have a retention time. After X minutes/days/weeks segments of the log are simply deleted.

Re: Using logs to build a solid data infrastructure

#12

Post by samwgoldman »

I've been really interested in this architecture since Jay Kreps' blog post on it. One part that I'm less clear on is how this fits in with request-response style communication between, say, a Web browser and a Web server.

In a simple Web-app-writes-to-DB scenario, it's easy to read my writes, but with a async log processing system, how am I supposed to organize my code so I can read my writes and respond with useful information?

Maybe the solution is to eschew request-response entirely and have all requests return 200, then poll or use two-way communication?

Alternatively, I could have my log-appending operation return a value indicating the position in the totally-ordered log, which I could pass to the query interfaces as a way of indicating "don't return until you've processed at least to here." Does anyone do that?

Am I totally off base here? I'd love to hear from anyone who is using these kinds of systems today.

Re: Using logs to build a solid data infrastructure

#13

Post by moatra »

Sphax wrote: post #11
moatra wrote: post #6

if you want to build a new derived datastore, you can just start a new consumer at the beginning of the log, and churn through the history of the log, applying all the writes to your datastore. For high-throughput environments with lots of appends to the log, how do you get around the ever-increasing size of your log file? I know the traditional answer is to take a periodic snapshot and compact the previous data, but…

There's a log compaction cleanup policy yes. Never used it myself but if I'm not mistaken it works like this: for each message you send to Kafka, you set a key with it. When Kafka does log compaction, it keeps only the last value for each key. The other cleanup policy is to just have a retention time. After X minutes/days/weeks segments of the log are simply deleted.

That sounds great if your messages in the logs are the complete state for that key, but I'm not seeing how to use that compaction system if the messages are change events.

Is there a system designed for snapshotting the aggregate and logging the delta?

Re: Using logs to build a solid data infrastructure

#14

Post by fizx »

samwgoldman wrote: post #12

I've been really interested in this architecture since Jay Kreps' blog post on it. One part that I'm less clear on is how this fits in with request-response style communication between, say, a Web browser and a Web server. In a simple Web-app-writes-to-DB scenario, it's easy to read my writes, but with a async log processing system, how am I supposed to organize my code so I can read my writes and respond with useful…

Typically, you cache the last write somewhere close so that you can manually splice it into the corresponding page render. Then you engineer the rest of the system such that the log processing system is rarely/never falling behind.

Re: Using logs to build a solid data infrastructure

#15

Post by sridca »

Has anyone thought of how the distributed commit log can be extended to client-side FRP?

Elm has this notion of signal – http://elm-lang.org/learn/What-is-FRP.elm – which is really a stream of changing values, that are used to construct varying model and its rendered view (virtual DOM and all that).

I am wondering if we can merge these two notions – signals and commit logs. Consequently, this would replace the traditional "request-response" model in REST API with nothing but signals, thus leveraging the simplicity of FRP for the whole application. Client side Elm code does a "send" on the signal that is connected to the server-side commit log, and also reads from another signal connected to another log that receives new data (added from various places).

Re: Using logs to build a solid data infrastructure

#16

Post by gbrits »

samwgoldman wrote: post #12

I've been really interested in this architecture since Jay Kreps' blog post on it. One part that I'm less clear on is how this fits in with request-response style communication between, say, a Web browser and a Web server. In a simple Web-app-writes-to-DB scenario, it's easy to read my writes, but with a async log processing system, how am I supposed to organize my code so I can read my writes and respond with useful…

You do req/ack. I.e: reply with 202.

Afterwards, either poll or communicate over sockets back to client side. If there's workers involved on the server: add socketid in the message envelope so it can be matched on return.

Re: Using logs to build a solid data infrastructure

#17

Post by boredandroid »

samwgoldman wrote: post #12

I've been really interested in this architecture since Jay Kreps' blog post on it. One part that I'm less clear on is how this fits in with request-response style communication between, say, a Web browser and a Web server. In a simple Web-app-writes-to-DB scenario, it's easy to read my writes, but with a async log processing system, how am I supposed to organize my code so I can read my writes and respond with useful…

Here is how I think about this, there are three high level paradigms you see for processing: 1. Request/response (e.g. most UI actions, REST, etc) 2. Stream (e.g. subscribing to a Kafka topic) 3. Batch/periodic (e.g. Hadoop, DWH)

You actually need all of these at least somewhere in a company, and they each have their place.

The dividing line for request/response is that someone is waiting on the other end and that if the request fails you can just show them an error and continue on your way. So when a web service gets overwhelmed it usually just times out requests.

Consider an e-commerce site as an example: 1. Request/response Displaying a product Making a sale

2. Stream or batch Restocking Logistics and shipping Price adjustments Analytics Product catalog import Search index updates Etc

The later category is asynchronous so it can be done in a batch fashion (once an hour or day) if latency is not a concern, or in a streaming fashion if it needs to be faster.

Re: Using logs to build a solid data infrastructure

#18

Post by kasey_junk »

moatra wrote: post #13
Sphax wrote: post #11

Earlier quoted context omitted.

There's a log compaction cleanup policy yes. Never used it myself but if I'm not mistaken it works like this: for each message you send to Kafka, you set a key with it. When Kafka does log compaction, it keeps only the last value for each key. The other cleanup policy is to just have a retention time. After X minutes/days/weeks segments of the log are simply deleted.

That sounds great if your messages in the logs are the complete state for that key, but I'm not seeing how to use that compaction system if the messages are change events. Is there a system designed for snapshotting the aggregate and logging the delta?

A common pattern is to publish a "checkpoint" message. Not sure if the concept is built into Kafka or not.

Re: Using logs to build a solid data infrastructure

#19

Post by erichmond »

moatra wrote: post #13
Sphax wrote: post #11

Earlier quoted context omitted.

There's a log compaction cleanup policy yes. Never used it myself but if I'm not mistaken it works like this: for each message you send to Kafka, you set a key with it. When Kafka does log compaction, it keeps only the last value for each key. The other cleanup policy is to just have a retention time. After X minutes/days/weeks segments of the log are simply deleted.

That sounds great if your messages in the logs are the complete state for that key, but I'm not seeing how to use that compaction system if the messages are change events. Is there a system designed for snapshotting the aggregate and logging the delta?

It's easy to store messages in HDFS or S3 for long-term storage. It's also easy to replay messages from those mediums, if you need to re-ingest data later on.

Re: Using logs to build a solid data infrastructure

#20

Post by gbrits »

The proposed architecture really works well for me. I've used it for a couple of projects now.

To throw around some terms for those interested in reading up/background: - The separation of writes (through log) from reads (through any of the consumers) is sometimes called: CQRS (command query responsibility separation) - having a centralized log as the defining store for updates/ change events is sometimes called: eventsourcing - as mentioned in article: elastic search as a consumer of the log, which only gets updates through the log, is an example of an Eager Read Derivation.

All defined on site of Fowler.

Glad to see this getting more attention. Asked about usage for Kafka as an eventsource here some time ago. Includes insightful answer of Kafka author. http://stackoverflow.com/questions/17708489/using-kafka-as-a...

Post reply on HN