TimescaleDB Compression: Hypercore and Columnar Storage with up to 98% Ratio in PostgreSQL
Up to 100% of articles using "up to" in the title are complete dross.
How do I get the job of benevolent dictator of title moderation?
How TimescaleDB compresses time-series data
11–20 of 29 posts
Re: How TimescaleDB compresses time-series data
#12TimescaleDB Compression: Hypercore and Columnar Storage with up to 98% Ratio in PostgreSQL Is the full title. Up to 100% of articles using "up to" in the title are complete dross. How do I get the job of benevolent dictator of title moderation?
Okay maybe title is not great but I really achieved this compression on real database where I have devices via MQTT connected.
Re: How TimescaleDB compresses time-series data
#13Old data historians use swinging-door compression algorithms (a lossy compression). Would something like TimescaleDB make this unnecessary for IoT? https://docs.aveva.com/bundle/pi-server-s-da-admin/page/1022...
Based on us keeping a replica of our AVEVA data archive in delta tables which are just parquets; and only compressing the parquets in LZ4 and getting a fuck ton of more space savings. I would say yes. Especially when you're onprem with a dedupe storage appliance.
I think swinging door made sense when storage was slow and scrace, but nowadays storage is cheap (even with these prices)
Re: How TimescaleDB compresses time-series data
#14Old data historians use swinging-door compression algorithms (a lossy compression). Would something like TimescaleDB make this unnecessary for IoT? https://docs.aveva.com/bundle/pi-server-s-da-admin/page/1022...
If I understand your question correctly: For storage, mostly yes. Swinging-door compression went lossy because byte compression on raw floats used to be useless, but with new method like Gorilla you can compress it lossless, so you can keep every sample and still afford it. But there is another point that this Aveva historian adds filter that filters some points so I don't know how it impact ingest. But in general no…
>But in general now trend is to use normal databases from IT world in OT world to overcome some legacy solutions.
Someone tell my company this please ;-;
Re: How TimescaleDB compresses time-series data
#15Compression uses a different license than Apache.
In various distros, the timescaledb installed by the package manager doesn't support other licenses than Apache.
This means that there's a high chance your distro doesn't allow stock timescaledb to use compression. You probably will need to install it manually.
Re: How TimescaleDB compresses time-series data
#16> What does compression do to query performance? That section is the most relevant whenever compression in a DB is discussed. The purpose of a database is to find, aggregate or update data - storage is where the trade-off gets expressed. There are no silver bullets here. Any method of compression which speeds up either filter rejection or scan rate is better than something that only trades off IO for CPU usage. For e…
[dead]
Re: How TimescaleDB compresses time-series data
#17I have been working on another PG extension for timeseries ( https://github.com/xataio/deltax ) for a few months, and trying to make it score as good as possible on ClickBench. This is a project that is simply lot of fun to work on. There are many tricks that can be used to speed-up analytics, besides just type-aware compression: * for each segment you will keep things like max/min/sum, number of distinct values, blo…
I work a lot with databases, but only a couple times have I really messed with time series stuff. I've enjoyed reading your list of tricks and it's gotten me interested in playing around with writing my own toy database to just experiment with some of these ideas.
Re: How TimescaleDB compresses time-series data
#18Is anyone running self hosted timescale? How is it going? I have a use case for it, but don’t know why I’m a bit afraid of using it.
Re: How TimescaleDB compresses time-series data
#19Is anyone running self hosted timescale? How is it going? I have a use case for it, but don’t know why I’m a bit afraid of using it.
I run it on VM the same way as a few separate PostgreSQL instances.
TimescaleDB is an extension to PostgreSQL, so there's a lot of information, guides, and community support online, and Postgres is one of the most widely used databases right now. There are some things to setup but Tiger data documentation is very good.
Re: How TimescaleDB compresses time-series data
#20Is anyone running self hosted timescale? How is it going? I have a use case for it, but don’t know why I’m a bit afraid of using it.
At StackGres [1] we find Timescale to be one of the most used extensions. Timescale is quite a successful project!
StackGres is actually the first solution recommended by Timescale for self-hosting with Kubernetes operators [2].
So if you are into Kubernetes (or if not, consider it, using something like K3s [3] is quite straightforward and lightweight on resources), this is probably a great option to self-host which would avoid you all the hassle of manually doing all the setup involved for production-ready clusters.
[2] https://www.tigerdata.com/docs/get-started/choose-your-path/...
[3] https://k3s.io/