Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

KSQL: Open Source Streaming SQL for Apache Kafka

confluent.io

Post reply on HN
Print view View original

11–20 of 87 posts

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#11

Post by stingraycharles »

daddykotex wrote: post #6

I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…

As always in these types of the scenarios, the answer is: it depends. It depends on the amount of data you have. It depends upon how big the diversion from the original schema is. Etcetera.

My personal philosophy is to always leave event data at rest alone: data is immutable, you don't convert it, and you treat it like a historical artifact. You version each event, but never convert it into a new version in the actual event store. Any version upgrades that should be applied are done when the event is read; this requires automated procedures to convery any event version N to another version N + 1, but having these kind of procedures in place is good practice anyway. Some might argue that doing this every time an event is read is a waste of CPU cycles, but in my experience this far outweights possible downsides of losing the actual event stored at that time in the past, and this type of data is accessed far less frequently than new event data.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#12

Post by daddykotex »

stingraycharles wrote: post #11
daddykotex wrote: post #6

I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…

As always in these types of the scenarios, the answer is: it depends. It depends on the amount of data you have. It depends upon how big the diversion from the original schema is. Etcetera. My personal philosophy is to always leave event data at rest alone: data is immutable, you don't convert it, and you treat it like a historical artifact. You version each event, but never convert it into a new version in the actua…

Ok this makes sense.

It matches what some others have been saying as well. Thanks

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#13

Post by assface »

daddykotex wrote: post #6

I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…

What you are referring to is called a "temporal database". Your specific example is called a "bitemporal database".

https://en.wikipedia.org/wiki/Temporal_database

https://en.wikipedia.org/wiki/Bitemporal_Modeling

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#14

Post by sixdimensional »

filereaper wrote: post #5

I really wish frameworks offering SQL would upfront say what level of SQL compatibility they have, is it SQL 2011, is it Postgres etc... Anyways, if anyone's wondering, here's the Github page. [1] Also from FAQ [2]: Is KSQL fully compliant to ANSI SQL? KSQL is a dialect inspired by ANSI SQL. It has some differences because it is geared at processing streaming data. For example, ANSI SQL has no notion of “windowing” f…

ANSI SQL should be extended to standardize streaming syntax. There are certainly enough vendors that have it now... for example, sqlstream.com. It's too bad that SQL:2011 didn't get these features since it had a lot of temporal-related extensions.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#16

Post by sixdimensional »

daddykotex wrote: post #6

I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…

I've often wondered the same.

In data warehousing, particularly the Kimball methodology, if descriptive attributes are missing from dimensions, for example, it is common to represent them using some standard value, like, "Not Applicable" or "Unknown" for string values. For integers, one might use -1 or similar. For dates it might be a specially chosen token date which means "Unknown Date" or "Missing Date".

It doesn't solve the problem of truly unknown missing information, but it at least gives a standard practice for labeling it consistently. Think of trying to do analytics on something where the value is unknown?? Not too easy, but at least it is all in one bucket.

Certainly, if past values can be derived, even if they were not created at the time the data was originally created, that is one way of "deriving" the past when it was missing. But, otherwise, I don't think there is any other way to make up for the lack of data/information.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#17

Post by stingraycharles »

daddykotex wrote: post #12
stingraycharles wrote: post #11

Earlier quoted context omitted.

As always in these types of the scenarios, the answer is: it depends. It depends on the amount of data you have. It depends upon how big the diversion from the original schema is. Etcetera. My personal philosophy is to always leave event data at rest alone: data is immutable, you don't convert it, and you treat it like a historical artifact. You version each event, but never convert it into a new version in the actua…

Ok this makes sense. It matches what some others have been saying as well. Thanks

There's a whole world out there about this kind of stuff. Take a look at CQRS and some of the posts by Greg Young; they're highly informative and one of the first people to really capture this way of dealing with data properly.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#18

Post by dm3 »

daddykotex wrote: post #6

I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…

Here's a pretty good, even if a bit too verbose, explanation of various issues and solutions related to event versioning: https://leanpub.com/esversioning/read.

The text is written by Greg Young - the lead on the EventStore [1] project.

[1] https://geteventstore.com/

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#20

Post by zjaffee »

daddykotex wrote: post #6

I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…

The best way I've seen it done, is to version all of your schemas and have a database that signals all the transformations needed to be done for any given schema version. That way, when your reading a particular event, you can query for all the operations needed to be done on such event and perform them.
Post reply on HN