Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

KSQL: Open Source Streaming SQL for Apache Kafka

confluent.io

Post reply on HN
Print view View original

81–87 of 87 posts

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#81

Post by daddykotex »

linkmotif wrote: post #35
daddykotex wrote: post #6

I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…

Recently asked on the Kafka users mailing list https://lists.apache.org/thread.html/82692004eb2292e1240c339...

Thanks for sharing, it makes me realize that our messages are not independent.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#82

Post by raulk »

tedmiston wrote: post #72
dkersten wrote: post #71

Earlier quoted context omitted.

I suppose one needs to take care that migrations are never lossy so that the full information for upgrading or downgrading a version is available.

Yeah, that's the challenge. For instance, how do you handle when a column was one data type but then down the road was changed to another type when the two aren't cross compatible or could potentially break?

You could retain this info in a meta field of flexible type. For a DB, it could a JSON type. For messages, it could be an extra _meta field on the message that the systems themselves ignore.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#83

Post by edan0 »

nehanarkhede wrote: post #37
rollulus wrote: post #26

Earlier quoted context omitted.

Hmm, that's "interesting". Can you elaborate a bit on how yours and theirs differ, if any?

There's a pretty big difference to a point that Landoop's KCQL (Kafka Connect Query Language) and Confluent's KSQL (Streaming SQL for Apache Kafka) are two different products. - KSQL is a full-fledged Streaming SQL engine for all kinds of stream processing operations from windowed aggregations, stream-table joins, sessionization and much more. So it does more powerful stream processing on Kafka than what Landoop's pr…

KSQL is not a full-fledged streaming SQL engine.

If you look at its syntax reference page (https://github.com/confluentinc/ksql/blob/0.1.x/docs/syntax-...), you can see that it only has a small number of scalar functions, a handful of aggregate functions, and only four major SELECT operators including an invented and nonstandard WINDOW clause (even though ANSI standard SQL has had a WINDOW clause for 18 years, since SQL:99).

Contrast that with SQLstream Blaze, which is a full-fledged streaming SQL engine with over a hundred scalar functions, operators, CEP operators/temporal predicates, aggregate functions, analytic functions, UDXes/UDFs, and major statement operators. (See http://sqlstream.com/docs/index.html?conc_transforming_and_a... for details.)

This all comes into play when an experienced SQL developer wants to write a real-world query or port an existing business application from an RBDMS to a stream processor. Outside of toy applications, this just can't be done with without a full-fledged streaming SQL engine.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#84

Post by edan0 »

lima wrote: post #2

Combine this with Debezium[1] and you get real-time SQL queries on your MySQL/PostgreSQL/MongoDB database! RethinkDB, as far as I understand, does the same thing - their changefeed mechanism would consume the DB log and run the queries against the log. [1]: http://debezium.io

You’re absolutely right. Integration with the whole ecosystem can provide real-time streaming SQL queries against not only open source databases MySQL, PostgreSQL, and MongoDB, but also (1) enterprise relational databases Oracle, Teradata, Microsoft SQL Server, and IBM DB2; (3) message buses including Kafka, AWS Kinesis, ActiveMQ/AMQP, IBM WebSphere MQ, and Teradata Listener; (3) natural streaming data including HTTP, web sockets, network sockets, and files; and (4) cloud data warehouses including Amazon Redshift and Snowflake.

Imagine, for example, writing a query that would join a Kafka stream against a Kinesis stream and enrich it with a database lookup. You'll find all of that today with SQLstream Blaze.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#85

Post by miguno »

buremba wrote: post #46

I love the continuous queries over append-only datasets and it's great that Kafka added support for it. It looks like there is one main contributor ( https://github.com/confluentinc/ksql/graphs/contributors ) though, it seems that the other contributors either wrote the documentation or helped for the packaging. Not a great sign considering how big this project is (there are competitors which only do this such Pipeli…

To add to what nehanarkhede said: I'd rather see it as a testament to the powerful building blocks of Kafka and its Streams API that you can actually implement a project such as KSQL with very few engineers.

Also, the commits in KSQL reflect only parts of the work -- it doesn't include design discussions, code reviews, etc.

Lastly, keep in mind that the git repository was cleaned (think: squashed commits) prior to publication, see the very first commit in the repository's timeline. So you don't see the prior work/commits of other Confluent engineers that went into KSQL.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#86

Post by sewen »

filereaper wrote: post #5

I really wish frameworks offering SQL would upfront say what level of SQL compatibility they have, is it SQL 2011, is it Postgres etc... Anyways, if anyone's wondering, here's the Github page. [1] Also from FAQ [2]: Is KSQL fully compliant to ANSI SQL? KSQL is a dialect inspired by ANSI SQL. It has some differences because it is geared at processing streaming data. For example, ANSI SQL has no notion of “windowing” f…

Flink's SQL implementation actually follows ANSI SQL - which I think is very important.

There is a way of interpreting streams and tables to make ANSI SQL meaningful in the presence of streams, which we follow [1].

The big advantage is (besides not having to learn another syntax and retaining compatibility with SQL tools and dashboards) that this seamlessly handles the batch (bounded/static input) and streaming (unbounded/continuous) use cases with literally the same SQL statement.

[1] http://flink.apache.org/news/2017/04/04/dynamic-tables.html

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#87

Post by sewen »

nehanarkhede wrote: post #36
CSDude wrote: post #4

Apache Flink is also a good alternative, and works very well. We have used it in production for a while for generating live reports. I made simple example [1] and have a look at the docs if you are more interested [2]. Gonna definetely try Kafka's version, its version of stream processing [3] also interesting as well. [1] https://medium.com/@mustafaakin/flink-streaming-sql-example-... [2] https://ci.apache.org/projec…

I'm one of the authors of Kafka. I've outlined some differences between Flink's support for streaming SQL and KSQL in this Twitter thread - https://twitter.com/juliusvolz/status/902283513382051840 Here's a summary: - KSQL has a completely Interactive SQL interface, so you don't have to switch between DSL code and SQL. - KSQL upports local, distributed and embedded modes. Is tightly integrated with Kafka's Streams API…

Disclaimer: I am one of the Flink committers.

While Flink has in fact no direct SQL entry point right now (and many users simply wrap the API entry points themselves to form a SQL entry point), the other statements are actually not quite right.

  - Flink as a whole (and SQL sits just on the DataStream API) works local, distributed and embedded as well.

  - Flink does not have any external dependencies, not even Kafka/ZooKeeper; it is self-contained. One can even just receive a data stream via a socket if that works for the use case.

  - Flink itself has always had exactly-once semantics, and works also exactly-once with Kafka.
Post reply on HN