51–60 of 87 posts
Earlier quoted context omitted.
PipelineDB is great..and the guys are super helpful.. influxdb has something similar, and now so does AWS Kinesis (with their Kinesis Analytics product).
Doesn't PipelinDB limit you to postgres' scaling capabilities?
PipelineDB also offers a clustering extension for large workloads (see: http://enterprise.pipelinedb.com/docs/)
But in terms of ad hoc, exploratory analytics workloads, yes - the scaling limitations would be the same, since for ad hoc, exploratory analytics PipelineDB and PostgreSQL are the same. But with that said, the processed, aggregated data that gets stored is generally much smaller than large volumes of granular data, so there is much less data to comb through with PipelineDB.
I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…
I love the continuous queries over append-only datasets and it's great that Kafka added support for it. It looks like there is one main contributor ( https://github.com/confluentinc/ksql/graphs/contributors ) though, it seems that the other contributors either wrote the documentation or helped for the packaging. Not a great sign considering how big this project is (there are competitors which only do this such Pipeli…
At Landoop we submitted our proposal to present our KSQL at the Kafka summit but we were rejected. Ah, if we only knew... :)
Earlier quoted context omitted.
Doesn't PipelinDB limit you to postgres' scaling capabilities?
Not for streaming analytic workloads, because PipelineDB fundamentally adds continuous queries to PostgreSQL, so data is continuously distilled and aggregated as it arrives, before it is stored, which drastically reduces the amount of data stored in PipelineDB (or soon, in PostgreSQL via the extension refactor). PipelineDB also offers a clustering extension for large workloads (see: http://enterprise.pipelinedb.com/d…
At Landoop we submitted our proposal to present our KSQL at the Kafka summit but we were rejected. Ah, if we only knew... :)
An independent program committee makes these calls and had to pick from 166 submissions. The program committee picks talks that are insightful and technically challenging.
Regarding Landoops proposal. This was not about KCQL but SQL on Streams, also named KSQL which is integrated with their new product Kafka Lenses. We'll look at Confluents SQL and see which one to go forward with, maybe both but we have happy customers using our version.
But congratulations on your KSQL very nice! We (Landoop and DM staff) have proved many ex colleagues in the Investment bank world wrong about Kafka and this cements our decisions to use it. Thanks.
I've had the question for a while so I'll ask it here, maybe someone can help me. Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1. Over time,…
I encountered that problem. The ad hoc fix, was to have a version field in each event and functions that translate the old event into new event(s). The code that processes the events only processes events of the current version. If your old events had been denormalized this might result into repetition of events when splitted.
Apache Flink is also a good alternative, and works very well. We have used it in production for a while for generating live reports. I made simple example [1] and have a look at the docs if you are more interested [2]. Gonna definetely try Kafka's version, its version of stream processing [3] also interesting as well. [1] https://medium.com/@mustafaakin/flink-streaming-sql-example-... [2] https://ci.apache.org/projec…
I'm one of the authors of Kafka. I've outlined some differences between Flink's support for streaming SQL and KSQL in this Twitter thread - https://twitter.com/juliusvolz/status/902283513382051840 Here's a summary: - KSQL has a completely Interactive SQL interface, so you don't have to switch between DSL code and SQL. - KSQL upports local, distributed and embedded modes. Is tightly integrated with Kafka's Streams API…
I'm very bullish on kafka. Today we have Spark for batch data computation and have already switched some of our streaming stuff to Kafka.
Do you see yourselves entering into the batch processing space anytime ? Google has officially said that Flink is "compelling" because of its compatibility with the Beam model.
If I can step on thin ice... is it easier for Flink to commandeer Kafka or for Kafka to win over batch processing ?