Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

KSQL: Open Source Streaming SQL for Apache Kafka

confluent.io

Post reply on HN
Print view View original

61–70 of 87 posts

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#61

Post by joaodlf »

filereaper wrote: post #5

I really wish frameworks offering SQL would upfront say what level of SQL compatibility they have, is it SQL 2011, is it Postgres etc... Anyways, if anyone's wondering, here's the Github page. [1] Also from FAQ [2]: Is KSQL fully compliant to ANSI SQL? KSQL is a dialect inspired by ANSI SQL. It has some differences because it is geared at processing streaming data. For example, ANSI SQL has no notion of “windowing” f…

I'm assuming that most "SQL" implementations in distributed systems don't actually follow the SQL standard. They just offer a SQL like syntax to solve common problems. I could be wrong, but that's the impression I get working with tools like Cassandra/Spark.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#62

Post by rxin »

nehanarkhede wrote: post #41
elvinyung wrote: post #32

Earlier quoted context omitted.

I think you mean Athena :P This seems to be basically similar to Spark, which lets you perform full SQL queries on Kafka streams.

Spark SQL is different from KSQL in the following ways: - Spark SQL is not an interactive Streaming SQL interface. To do stream processing, you have to switch between writing code using Java/Scala/Python and SQL statements. KSQL, on the other hand, is a completely interactive Streaming SQL engine. You can do sophisticated stream processing operations interactively using SQL statements alone. - KSQL is a true event-at…

What you said about Spark was not true. One can totally interactively query a Kafka stream using Spark, in SQL.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#63

Post by vog »

stingraycharles wrote: post #11
daddykotex wrote: post #6

I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…

As always in these types of the scenarios, the answer is: it depends. It depends on the amount of data you have. It depends upon how big the diversion from the original schema is. Etcetera. My personal philosophy is to always leave event data at rest alone: data is immutable, you don't convert it, and you treat it like a historical artifact. You version each event, but never convert it into a new version in the actua…

> this requires automated procedures to convery any event version N to another version N + 1, but having these kind of procedures in place is good practice anyway

Note that this "good practice" already has a name, it is usually called "migrations".

Migrations may be as simple as SQL Update / ALTER TABLE statements. But they may also be transformation of JSON/XML/... structures, or any other complex calculation.

It may not be the best term, as "migrations" usually imply that the result is written back to the data store. But apart from that, I don't see any problem using this term here.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#64

Post by _pmf_ »

stingraycharles wrote: post #17
daddykotex wrote: post #12

Earlier quoted context omitted.

Ok this makes sense. It matches what some others have been saying as well. Thanks

There's a whole world out there about this kind of stuff. Take a look at CQRS and some of the posts by Greg Young; they're highly informative and one of the first people to really capture this way of dealing with data properly.

I wonder what happened to Greg Young. I think he had a book in the pipeline (which I was looking forward to), but as far as I can glean from social media, some burnout related stuff happened.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#65

Post by sandGorgon »

Fergi wrote: post #52
stuartaxelowen wrote: post #51

Earlier quoted context omitted.

Doesn't PipelinDB limit you to postgres' scaling capabilities?

Not for streaming analytic workloads, because PipelineDB fundamentally adds continuous queries to PostgreSQL, so data is continuously distilled and aggregated as it arrives, before it is stored, which drastically reduces the amount of data stored in PipelineDB (or soon, in PostgreSQL via the extension refactor). PipelineDB also offers a clustering extension for large workloads (see: http://enterprise.pipelinedb.com/d…

this is awesome! will it work on Amazon RDS for postgresql ?

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#66

Post by _pmf_ »

bamazizi wrote: post #29
daddykotex wrote: post #6

I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…

I highly recommend you look into gRPC. Building apps using event sourcing, CQRS and microservices can easily become hell if the data models are not thought through.

I thought RPC and CQRS are diametrically opposite patterns. (Although you can use RPC in a CQRS context, but only as a transport/encapsulation layer (so the response says "request queued" or "error", but does not divulge domain specific information ("item created", "item not created")).

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#68

Post by _pmf_ »

Fergi wrote: post #44
cmollis wrote: post #23

Earlier quoted context omitted.

PipelineDB is great..and the guys are super helpful.. influxdb has something similar, and now so does AWS Kinesis (with their Kinesis Analytics product).

PipelineDB will become a standard, open-source PostgreSQL extension this year, enabling anybody using PostgreSQL to leverage continuous SQL queries in combination with all other PostgreSQL functionality. https://www.pipelinedb.com/blog/pipelinedb-0-9-8-postgresql-...

> PipelineDB will become a standard, open-source PostgreSQL extension

What does this imply?

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#69

Post by blain_the_train »

sandGorgon wrote: post #60
nehanarkhede wrote: post #36

Earlier quoted context omitted.

I'm one of the authors of Kafka. I've outlined some differences between Flink's support for streaming SQL and KSQL in this Twitter thread - https://twitter.com/juliusvolz/status/902283513382051840 Here's a summary: - KSQL has a completely Interactive SQL interface, so you don't have to switch between DSL code and SQL. - KSQL upports local, distributed and embedded modes. Is tightly integrated with Kafka's Streams API…

@neha - where do you think kafka is going to evolve in the world of data processing. I'm very bullish on kafka. Today we have Spark for batch data computation and have already switched some of our streaming stuff to Kafka. Do you see yourselves entering into the batch processing space anytime ? Google has officially said that Flink is "compelling" because of its compatibility with the Beam model. If I can step on thi…

What do you mean by "batch processing" Personally i find that term to be confusing.

I believe if Kafka can do streaming then it effectively can do batch as batch is a subset of streaming.

Re: KSQL: Open Source Streaming SQL for Apache Kafka

#70

Post by [deleted] »

nehanarkhede wrote: post #55
andmarios wrote: post #15

At Landoop we submitted our proposal to present our KSQL at the Kafka summit but we were rejected. Ah, if we only knew... :)

An independent program committee makes these calls and had to pick from 166 submissions. The program committee picks talks that are insightful and technically challenging.

[deleted]
Post reply on HN