KSQL: Open Source Streaming SQL for Apache Kafka

#1

Post by uptown » Mon, Aug 28, 2017, 4:34 PM UTC

KSQL: Open Source Streaming SQL for Apache Kafka
confluent.io

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#2

Post by lima » Mon, Aug 28, 2017, 5:55 PM UTC

Combine this with Debezium[1] and you get real-time SQL queries on your MySQL/PostgreSQL/MongoDB database!

RethinkDB, as far as I understand, does the same thing - their changefeed mechanism would consume the DB log and run the queries against the log.

[1]: http://debezium.io

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#3

Post by cbsmith » Mon, Aug 28, 2017, 5:56 PM UTC

This is the Kafka answer to Pipelinedb. I love it.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#4

Post by CSDude » Mon, Aug 28, 2017, 6:07 PM UTC

Apache Flink is also a good alternative, and works very well. We have used it in production for a while for generating live reports. I made simple example [1] and have a look at the docs if you are more interested [2]. Gonna definetely try Kafka's version, its version of stream processing [3] also interesting as well.

[1] https://medium.com/@mustafaakin/flink-streaming-sql-example-...

[2] https://ci.apache.org/projects/flink/flink-docs-release-1.3/...

[3] http://docs.confluent.io/current/streams/index.html

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#5

Post by filereaper » Mon, Aug 28, 2017, 6:24 PM UTC

I really wish frameworks offering SQL would upfront say what level of SQL compatibility they have, is it SQL 2011, is it Postgres etc...

Anyways, if anyone's wondering, here's the Github page. [1]

Also from FAQ [2]:

Is KSQL fully compliant to ANSI SQL?

KSQL is a dialect inspired by ANSI SQL. It has some differences because it is geared at processing streaming data. For example, ANSI SQL has no notion of “windowing” for use cases such as performing aggregations on data grouped into 5-minute windows, which is a commonly required functionality in the streaming world.

[1] https://github.com/confluentinc/ksql/blob/0.1.x/docs/syntax-...

[2] https://github.com/confluentinc/ksql/blob/0.1.x/docs/faq.md#...

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#6

Post by daddykotex » Mon, Aug 28, 2017, 6:45 PM UTC

I've had the question for a while so I'll ask it here, maybe someone can help me.

Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1.

Over time, things have changed and you have evolved your events to include some fields and deprecate others. You could do this without any downtime whatsoever by changing your events in a way that is backward compatible way.

What is the good approach to what I'd call a `replay`?

When you want to replay all events, the version of your apps that will consume the events may not know about the fields that were in the event for day one.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#7

Post by coolio222 » Mon, Aug 28, 2017, 6:46 PM UTC

Does anyone knows details about the KSQL-engine that computes the queries? According to their git, there can be multiple KSQL-engines in a Client-Server configuration. Is the workload for one query distributed? Is the SQL translated into a program using the Kafka streaming API?

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#8

Post by daddykotex » Mon, Aug 28, 2017, 6:47 PM UTC

daddykotex wrote: post #6

I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…

I'm also wondering: how one deal with changes to events when using KSQL?

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#9

Post by coolio222 » Mon, Aug 28, 2017, 6:52 PM UTC

daddykotex wrote: post #6

I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…

I encountered that problem. The ad hoc fix, was to have a version field in each event and functions that translate the old event into new event(s). The code that processes the events only processes events of the current version. If your old events had been denormalized this might result into repetition of events when splitted.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#10

Post by daddykotex » Mon, Aug 28, 2017, 6:54 PM UTC

coolio222 wrote: post #9

daddykotex wrote: post #6

I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…

I encountered that problem. The ad hoc fix, was to have a version field in each event and functions that translate the old event into new event(s). The code that processes the events only processes events of the current version. If your old events had been denormalized this might result into repetition of events when splitted.

Ok, thanks for the pointer