Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

KSQL: Open Source Streaming SQL for Apache Kafka

confluent.io

Post reply on HN
Print view View original

71–80 of 87 posts

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#71

Post by dkersten »

tedmiston wrote: post #49
raulk wrote: post #27

Earlier quoted context omitted.

Bingo. If you equate this technique to a DB migration, you could have "up" and "down" directions for the translations from version N N+1. Then if you have 90% confidence you'll only ever need to replay the upgraded stream, you can upgrade it and destroy the previous version. If at some point (the remaining 10%) you need to rescue the old stream, you can run the "down" direction and rehydrate the old version of the st…

It sounds good in theory. In practice, I haven't heard much around running backwards migrations on a data warehouse / massive collection of events but I'm sure some out there already do it.

I suppose one needs to take care that migrations are never lossy so that the full information for upgrading or downgrading a version is available.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#72

Post by tedmiston »

dkersten wrote: post #71
tedmiston wrote: post #49

Earlier quoted context omitted.

It sounds good in theory. In practice, I haven't heard much around running backwards migrations on a data warehouse / massive collection of events but I'm sure some out there already do it.

I suppose one needs to take care that migrations are never lossy so that the full information for upgrading or downgrading a version is available.

Yeah, that's the challenge. For instance, how do you handle when a column was one data type but then down the road was changed to another type when the two aren't cross compatible or could potentially break?

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#74

Post by edan0 »

filereaper wrote: post #5

I really wish frameworks offering SQL would upfront say what level of SQL compatibility they have, is it SQL 2011, is it Postgres etc... Anyways, if anyone's wondering, here's the Github page. [1] Also from FAQ [2]: Is KSQL fully compliant to ANSI SQL? KSQL is a dialect inspired by ANSI SQL. It has some differences because it is geared at processing streaming data. For example, ANSI SQL has no notion of “windowing” f…

Confluent doesn't understand the ANSI SQL standard. Support for aggregating over windows with standard syntax goes back to SQL:99, so it's been around for 18 years. Any company playing in the SQL space should make a point of understanding the standard.

There is a good explanation of ANSI SQL standard windowing at http://sqlstream.com/docs/conc_applicationdesign.html?zoom_h.... The document describes both tumbling and rolling windows as well as how they are achieved with standard syntax.

SQLstream could not agree with you more that an up-front explanation of SQL compatibility level is necessary to evaluate a product. SQLstream Blaze is SQL:2011 compliant.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#75

Post by edan0 »

nehanarkhede wrote: post #40
lima wrote: post #2

Combine this with Debezium[1] and you get real-time SQL queries on your MySQL/PostgreSQL/MongoDB database! RethinkDB, as far as I understand, does the same thing - their changefeed mechanism would consume the DB log and run the queries against the log. [1]: http://debezium.io

Yes, exactly. We plan to add support for various Kafka connectors to KSQL so you can ingest/export streams from external systems, like database, and then do stream processing all using KSQL queries. The purpose is to bring Streaming ETL and Stream Processing together to make the user's life easy.

If you want streaming ANSI standard SQL support for both ingest and egress to and from a vast number of systems including (1) open source databases MySQL, PostgreSQL, and MongoDB; (2) enterprise relational databases Oracle, Teradata, Microsoft SQL Server, IBM DB2, Actian Vectorwise, Ingres, and ParAccel; (3) message buses including Kafka, AWS Kinesis, ActiveMQ/AMQP, IBM WebSphere MQ, and Teradata Listener; (4) natural streaming data including HTTP, web sockets, network sockets, and files; and (5) cloud data warehouses including Amazon Redshift and Snowflake, you can download and use SQLstream Blaze (http://www.sqlstream.com/download) today. You can even JOIN streams from AWS Kinesis against streams in Kafka to combine cloud and on-premise data via a single query.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#76

Post by edan0 »

nehanarkhede wrote: post #39
BMarkmann wrote: post #24

I understand the underlying architecture is fundamentally different, but the end result seems kind of akin to some of the "complex event processing" (CEP) tools out there like Esper (ie. feeding incoming data into established queries instead of executing queries against at-rest data). Would this fit similar sorts of use cases / fit into the CEP market?

Yes KSQL is ideal for use cases similar to those that CEP was initially targeted for and more - from real-time anomaly detection, monitoring, analytics to application development and Streaming ETL. As you alluded to, the big difference is that KSQL is designed as a distributed Streaming SQL engine that can run at Kafka scale.

If you are interested in performing ultra low latency (http://www.sqlstream.com) and it's full implementation of Allen's Interval Algebra (https://en.wikipedia.org/wiki/Allen%27s_interval_algebra) via SQL temporal predicates: http://sqlstream.com/docs/sqlrf_planned_feature_temporal_pre.... This stuff runs at 1,000,000 events per second per core and can scale out in conjunction with Kafka across any number of servers.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#78

Post by vdm »

daddykotex wrote: post #6

I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…

https://martin.kleppmann.com/2012/12/05/schema-evolution-in-...

https://www.safaribooksonline.com/library/view/designing-dat...

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#79

Post by daddykotex »

1_2__4 wrote: post #53
daddykotex wrote: post #6

I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…

Don't persist the stream. The problem gets a lot easier if you stop thinking of a message bus as a data store.

How do you do a replay if you don't keep the event somewhere? I did not mean that messages were to be stored in the bus?

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#80

Post by daddykotex »

nehanarkhede wrote: post #38
daddykotex wrote: post #8

Earlier quoted context omitted.

I'm also wondering: how one deal with changes to events when using KSQL?

Mind clarifying what you mean by changes to events? If you create a STREAM or TABLE using KSQL, it makes sure that they are kept updated with every single event that arrives on the source Kafka topics. That's what you'd expect in a true event-at-a-time Streaming SQL engine, which is what KSQL is.

Suppose you build a STREAM or TABLE from a topic and assume a field in the event is `id`. Later on, you introduce an update to this event where where your replace `id` by `user_id`, how is KSQL reacting?
Post reply on HN