Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

KSQL: Open Source Streaming SQL for Apache Kafka

confluent.io

Post reply on HN
Print view View original

41–50 of 87 posts

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#41

Post by nehanarkhede »

elvinyung wrote: post #32
chicagobuss wrote: post #30

Earlier quoted context omitted.

Hmm. I don't see that at all. Aurora is a way to query data in S3. This is a way to query Kafka streams in real time.

I think you mean Athena :P This seems to be basically similar to Spark, which lets you perform full SQL queries on Kafka streams.

Spark SQL is different from KSQL in the following ways:

- Spark SQL is not an interactive Streaming SQL interface. To do stream processing, you have to switch between writing code using Java/Scala/Python and SQL statements. KSQL, on the other hand, is a completely interactive Streaming SQL engine. You can do sophisticated stream processing operations interactively using SQL statements alone.

- KSQL is a true event-at-a-time Streaming SQL engine. Spark SQL is micro-batch.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#44

Post by Fergi »

cmollis wrote: post #23
cbsmith wrote: post #3

This is the Kafka answer to Pipelinedb. I love it.

PipelineDB is great..and the guys are super helpful.. influxdb has something similar, and now so does AWS Kinesis (with their Kinesis Analytics product).

PipelineDB will become a standard, open-source PostgreSQL extension this year, enabling anybody using PostgreSQL to leverage continuous SQL queries in combination with all other PostgreSQL functionality.

https://www.pipelinedb.com/blog/pipelinedb-0-9-8-postgresql-...

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#45

Post by iaabtpbtpnn »

largote wrote: post #43

Fun fact: KSQL is the code of the airport closest to Oracle HQ (San Carlos airport).

Came here to post this... do you know if the code is meant to pay homage to Oracle, or were they just trying to make it sound like San QarLos?

Edit: Oh, the wiki page explains that it predates Oracle. So probably the latter.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#46

Post by buremba »

I love the continuous queries over append-only datasets and it's great that Kafka added support for it.

It looks like there is one main contributor (https://github.com/confluentinc/ksql/graphs/contributors) though, it seems that the other contributors either wrote the documentation or helped for the packaging. Not a great sign considering how big this project is (there are competitors which only do this such Pipelinedb), hopefully you can create a team just for KSQL.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#47

Post by cbsmith »

cmollis wrote: post #23
cbsmith wrote: post #3

This is the Kafka answer to Pipelinedb. I love it.

PipelineDB is great..and the guys are super helpful.. influxdb has something similar, and now so does AWS Kinesis (with their Kinesis Analytics product).

Are InfluxDB's continuous queries really like this? It's so much less of a streaming solution I hadn't realized it did this.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#48

Post by manigandham »

daddykotex wrote: post #6

I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…

If the fields are changing then you effectively have DDL and migrations in your code already... so decouple them and version the schema officially. Then record these schema changes as events in the same event stream.

Build a view on against these schema change events as a table of schema version by timestamp to allow for parsing any arbitrary event.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#49

Post by tedmiston »

raulk wrote: post #27
dkersten wrote: post #21

Earlier quoted context omitted.

I suppose you can always trade those CPU cycles off against storage and cache the N+1 version (in a separate Kafka topic or elsewhere), so now reading the latest-version data is fast, yet you still retain the original data intact, at the expense of more storage. This does complicate the storage though, as you now have multiple days a sources, but nothing that can't be solved.

Bingo. If you equate this technique to a DB migration, you could have "up" and "down" directions for the translations from version N N+1. Then if you have 90% confidence you'll only ever need to replay the upgraded stream, you can upgrade it and destroy the previous version. If at some point (the remaining 10%) you need to rescue the old stream, you can run the "down" direction and rehydrate the old version of the st…

It sounds good in theory. In practice, I haven't heard much around running backwards migrations on a data warehouse / massive collection of events but I'm sure some out there already do it.
Post reply on HN