Beringei: A high-performance time series storage engine
code.facebook.com
Beringei: A high-performance time series storage engine
1–10 of 58 posts
Re: Beringei: A high-performance time series storage engine
#2I am very interested in a comparison by prometheus developers. Brian Brazil recently wrote about optimization to prometheus' time series storage.
Re: Beringei: A high-performance time series storage engine
#3[deleted]
Re: Beringei: A high-performance time series storage engine
#4What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp?
We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.
Re: Beringei: A high-performance time series storage engine
#5What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.
One idea might be to write to S3 directly and use Athena for your offline analysis
Re: Beringei: A high-performance time series storage engine
#6What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.
If you want low to no maintenance and maintain some of what you have I'd suggest pushing it into kinesis instead of postgres and then dumping to s3 as you're already doing.
Re: Beringei: A high-performance time series storage engine
#7How would this stack up against InfluxDB?
Re: Beringei: A high-performance time series storage engine
#8What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.
CSV
Same way most web servers log traffic.
Re: Beringei: A high-performance time series storage engine
#9Ok so it's an inmemory product, sharded no less.
They speak about compressing the data before "storing" it.
I don't have a lot of experience with inmemory anything, but are we talking about retaining the compressed format in server memory here? Ie, RAM is your datastore.
Then, at some point, to serve requests/queries for the data don't you have to get it "out of" RAM and uncompress it, also an inmemory operation?
Did I get this right?
Re: Beringei: A high-performance time series storage engine
#10Ok so it's an inmemory product, sharded no less. They speak about compressing the data before "storing" it. I don't have a lot of experience with inmemory anything, but are we talking about retaining the compressed format in server memory here? Ie, RAM is your datastore. Then, at some point, to serve requests/queries for the data don't you have to get it "out of" RAM and uncompress it, also an inmemory operation? Did…
The paper on the algorithm is here: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf
Somebody implemented the algorithm in go based on the paper here: https://github.com/dgryski/go-tsz
A short answer that may work for your question: the bits that are set in RAM are xor values relative to previous values. To provide an answer as to what the value is, a series of read|xor operations are performed.