Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

KSQL: Open Source Streaming SQL for Apache Kafka

confluent.io

Post reply on HN
Print view View original

31–40 of 87 posts

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#31

Post by daddykotex »

dm3 wrote: post #18
daddykotex wrote: post #6

I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…

Here's a pretty good, even if a bit too verbose, explanation of various issues and solutions related to event versioning: https://leanpub.com/esversioning/read . The text is written by Greg Young - the lead on the EventStore [1] project. [1] https://geteventstore.com/

thank you

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#32

Post by elvinyung »

chicagobuss wrote: post #30
jasonrhaas wrote: post #22

Seems like a somewhat similar idea to AWS Aurora.

Hmm. I don't see that at all. Aurora is a way to query data in S3. This is a way to query Kafka streams in real time.

I think you mean Athena :P

This seems to be basically similar to Spark, which lets you perform full SQL queries on Kafka streams.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#34

Post by hojjat »

coolio222 wrote: post #7

Does anyone knows details about the KSQL-engine that computes the queries? According to their git, there can be multiple KSQL-engines in a Client-Server configuration. Is the workload for one query distributed? Is the SQL translated into a program using the Kafka streaming API?

Yes, queries are translated into Kafka Streams API. In the client-server(cluster) mode each query will run on every instance of engine the same way kafka streams apps run on multiple instances.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#35

Post by linkmotif »

daddykotex wrote: post #6

I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…

Recently asked on the Kafka users mailing list https://lists.apache.org/thread.html/82692004eb2292e1240c339...

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#36

Post by nehanarkhede »

CSDude wrote: post #4

Apache Flink is also a good alternative, and works very well. We have used it in production for a while for generating live reports. I made simple example [1] and have a look at the docs if you are more interested [2]. Gonna definetely try Kafka's version, its version of stream processing [3] also interesting as well. [1] https://medium.com/@mustafaakin/flink-streaming-sql-example-... [2] https://ci.apache.org/projec…

I'm one of the authors of Kafka. I've outlined some differences between Flink's support for streaming SQL and KSQL in this Twitter thread - https://twitter.com/juliusvolz/status/902283513382051840

Here's a summary: - KSQL has a completely Interactive SQL interface, so you don't have to switch between DSL code and SQL.

- KSQL upports local, distributed and embedded modes. Is tightly integrated with Kafka's Streams API and Kafka itself; doesn't reinvent the wheel. So is simple to use and deploy.

- KSQL doesn't have external dependencies, for orchestration, deployment etc.

- KSQL has native support for Kafka's exactly once processing semantics, supports and stream-table joins.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#37

Post by nehanarkhede »

rollulus wrote: post #26
andmarios wrote: post #15

At Landoop we submitted our proposal to present our KSQL at the Kafka summit but we were rejected. Ah, if we only knew... :)

Hmm, that's "interesting". Can you elaborate a bit on how yours and theirs differ, if any?

There's a pretty big difference to a point that Landoop's KCQL (Kafka Connect Query Language) and Confluent's KSQL (Streaming SQL for Apache Kafka) are two different products.

- KSQL is a full-fledged Streaming SQL engine for all kinds of stream processing operations from windowed aggregations, stream-table joins, sessionization and much more. So it does more powerful stream processing on Kafka than what Landoop's product supports which is simple projections and filters.

- KSQL can do that because it supports streams and tables as first-class constructs and tightly integrates with Kafka's Streams API and the Kafka log itself. We are not aware of any other products that do that today, including Landoop's tool.

- We will add support for Kafka connectors so you can stream data from different systems into Kafka through KSQL. This will cover what Landoop intended with KCQL (Kafka Connect Query Language.

- Confluent works with several very large enterprises and many of the companies that have adopted Kafka. We worked with those customers to learn what would solve real business problems and used that feedback to build KSQL. So it . models on real-world customer feedback.

- We'd love to hear feedback. Here's the repository https://github.com/confluentinc/ksql and here's the Slack Channel slackpass.io/confluentcommunity - #ksql

Hope that helps!

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#38

Post by nehanarkhede »

daddykotex wrote: post #8
daddykotex wrote: post #6

I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…

I'm also wondering: how one deal with changes to events when using KSQL?

Mind clarifying what you mean by changes to events?

If you create a STREAM or TABLE using KSQL, it makes sure that they are kept updated with every single event that arrives on the source Kafka topics.

That's what you'd expect in a true event-at-a-time Streaming SQL engine, which is what KSQL is.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#39

Post by nehanarkhede »

BMarkmann wrote: post #24

I understand the underlying architecture is fundamentally different, but the end result seems kind of akin to some of the "complex event processing" (CEP) tools out there like Esper (ie. feeding incoming data into established queries instead of executing queries against at-rest data). Would this fit similar sorts of use cases / fit into the CEP market?

Yes KSQL is ideal for use cases similar to those that CEP was initially targeted for and more - from real-time anomaly detection, monitoring, analytics to application development and Streaming ETL. As you alluded to, the big difference is that KSQL is designed as a distributed Streaming SQL engine that can run at Kafka scale.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#40

Post by nehanarkhede »

lima wrote: post #2

Combine this with Debezium[1] and you get real-time SQL queries on your MySQL/PostgreSQL/MongoDB database! RethinkDB, as far as I understand, does the same thing - their changefeed mechanism would consume the DB log and run the queries against the log. [1]: http://debezium.io

Yes, exactly. We plan to add support for various Kafka connectors to KSQL so you can ingest/export streams from external systems, like database, and then do stream processing all using KSQL queries.

The purpose is to bring Streaming ETL and Stream Processing together to make the user's life easy.

Post reply on HN