31–40 of 102 posts
We have a requirement of saving 100million data points every 5 mins. What options should we explore for real time system for last 15 days of data and archival system for last 3 years of data?
Might be easiest to discuss more on Slack (https://slack-login.timescale.com/) or email (mike at timescale) if you're interested.
Earlier quoted context omitted.
Ok and if one is partitioning by date and dropping partitions instead of deletes in vanilla postgres how does it compare ?
The delete performance will probably be similar, but standard partitions in postgres have a bunch of current limitations. For example, the insert pipeline is still quite a bit slower, partition creation is still manual, can't do as good constraint exclusion at query time, can't do certain query optimizations we've built in, can't support user-defined triggers, can't handle UPSERTs, doesn't support various constraints…
Could you contrast this with the approaches mentioned in the series of blog posts starting here: https://grisha.org/blog/2015/09/23/storing-time-series-in-po... That blog post grew to be tgres http://github.com/tgres/tgres https://grisha.org/blog/2017/03/22/tgres-0-dot-10-dot-0b-tim...
That's useful in many dashboard-based server monitoring applications, but time-series DB have many other applications (and can benefit from more complex queries even in monitoring).
Tgres and Timescale are actually a bit complementary, and you might even be able to use Timescale as a better backend for Tgres.
The benchmark repo doesn't actually include the performance comparison between Timescale and Postgres: https://github.com/timescale/benchmark-postgres#benchmark-qu...
This blog post (https://blog.timescale.com/timescaledb-vs-6a696248104e) has some query benchmarks and the main benefit it that the hypertable will partition the data smoothly and if we query the table by filtering with timestamp column, it will be fast since Timescale uses partitioning as an indexing method.
Why do you usually advertise the write performance? Let's say that I have "100+ billion rows (the number in your landing page)", how much time it takes to run a simple GROUP BY query? The benchmark repo doesn't actually include the performance comparison between Timescale and Postgres: https://github.com/timescale/benchmark-postgres#benchmark-qu... This blog post ( https://blog.timescale.com/timescaledb-vs-6a69624810…
On the query side, we find that most queries to a time-series DB actually include a time predicate, LIMIT clause, etc. It's pretty rare that you do a full table scan over the 100B rows. (And for these types of broad scans, performance depends on # disks and use of query parallelization.)
Not sure I understand the comment about the benchmark repo doesn't include the performance comparison? That repo is meant to accompany a blog post, which discusses the results (https://blog.timescale.com/timescaledb-vs-6a696248104e), while the repo allows you to replicate our results.
A project I work on has time series stats in postgres--it's essentially an interval, a period, a number of fields that make up the key, and the value. There's a compound index that includes most of the fields except for the value. It works surprisingly well, for tens of thousands of upserts per second on a single postgres instance. Easy app integration and joins are a huge plus. I'm really curious to check this out a…
In a funny bit of coincidence -- we didn't post this link to HN :) -- we just published a blog post today comparing Timescale vs. native Postgres: https://blog.timescale.com/timescaledb-vs-6a696248104e tl;dr: 20x higher inserts at scale, faster queries, 2000x faster deletes, more time-oriented analytical features
We have a requirement of saving 100million data points every 5 mins. What options should we explore for real time system for last 15 days of data and archival system for last 3 years of data?
Earlier quoted context omitted.
In a funny bit of coincidence -- we didn't post this link to HN :) -- we just published a blog post today comparing Timescale vs. native Postgres: https://blog.timescale.com/timescaledb-vs-6a696248104e tl;dr: 20x higher inserts at scale, faster queries, 2000x faster deletes, more time-oriented analytical features
This would be good for a eventsourcing storage? Fast events insert and fast reads?
For event sourcing, just make sure you index on the proper user/session/thing (Docs or Slack for more info).