In the example for sf_proximity_count, you state the view covers a 5 minute sliding window, but the WHERE clause does not reference clock_timestamp(). Is 5 minutes an implicit default?
To Be Continuous
51–54 of 54 posts
Re: To Be Continuous
#52Earlier quoted context omitted.
Hey Chad, PipelineDB co-founder here. PipelineDB certainly isn't intended to be the only tool in your data infrastructure. But whenever the same queries are being repeatedly run on granular data, those are the types of situations in which it often makes a lot sense to just compute the condensed result incrementally with a continuous view, because that's the only lens it's ever viewed through anyways (dashboards are a…
So what's the best practice for when you want a real time dashboard but also want the ability to compare data overtime. E.g., ave. bounce rate this month vs last? Is Pipeline still ideal in this case?
Jeff (PipelineDB Co-Founder, here) - Yes, PipelineDB is great for this use case. One powerful aspect of PipelineDB is that it is a fully functional relational database (a superset of PostgreSQL 9.4) in addition to a streaming-SQL engine we have integrated the notion of 'state' into stream processing, for use cases exactly like this.
You can do anything with PipelineDB that you can do with PostgreSQL 9.4, but with the addition of continuous SQL queries, sliding windows, probabilistic data structures, uniques counting, and stream-table JOINs (what you're looking for here, I believe.)
Re: To Be Continuous
#53In the example for sf_proximity_count, you state the view covers a 5 minute sliding window, but the WHERE clause does not reference clock_timestamp(). Is 5 minutes an implicit default?
There was a bug in the example. Fixed, thanks!
Re: To Be Continuous
#54We needed to implement continuous queries in our application code. (It's actually hard to do it right in Postgresql so it's very limited) https://github.com/buremba/rakam/wiki/Postgresql-Backend#con... Since stream processing and real-time analytics are quite hot topics nowadays, I think real-time databases will get much more attention in a near future.
Postgres actually does expose a commit log mechanism since 9.4, expanded with 9.5. http://www.postgresql.org/docs/devel/static/logicaldecoding.... and 9.5's track_commit_timestamp = on.
Sorry for the late response. HN just sent me notification mail.
The documentation states it as "by default". It's required to change wal settings in config file and write a custom logical decoding output plugin in order to take advantage of that feature.
Currently, PaaS provisers such as RDS and Heroku Postgres don't support and it would not be easy to setup it manually. AFAIK, that feature is intended to be to used for backups.