Live data from Hacker News

Beringei: A high-performance time series storage engine

code.facebook.com

21–30 of 58 posts

Re: Beringei: A high-performance time series storage engine

#21

We're getting good results from http://traildb.io/ and we don't have to grant Facebook a worldwide, royalty-free license to our patent pool in order to use it.

https://github.com/facebookincubator/beringei/blob/master/LI... doesn't even mention patents.

https://github.com/facebookincubator/beringei#license says "We also provide an additional patent grant."

None of this sounds to me like you have to grant Facebook any patents to use Beringei.

Re: Beringei: A high-performance time series storage engine

#22

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

We used InfluxDB along with a few other tools. Check out: https://www.influxdata.com/time-series-platform/

Re: Beringei: A high-performance time series storage engine

#23

We're getting good results from http://traildb.io/ and we don't have to grant Facebook a worldwide, royalty-free license to our patent pool in order to use it.

https://github.com/facebookincubator/beringei/blob/master/LI... doesn't even mention patents. https://github.com/facebookincubator/beringei#license says "We also provide an additional patent grant." None of this sounds to me like you have to grant Facebook any patents to use Beringei.

https://github.com/facebookincubator/beringei/blob/master/PA...

Re: Beringei: A high-performance time series storage engine

#24

We're getting good results from http://traildb.io/ and we don't have to grant Facebook a worldwide, royalty-free license to our patent pool in order to use it.

https://github.com/facebookincubator/beringei/blob/master/LI... doesn't even mention patents. https://github.com/facebookincubator/beringei#license says "We also provide an additional patent grant." None of this sounds to me like you have to grant Facebook any patents to use Beringei.

[deleted]

Re: Beringei: A high-performance time series storage engine

#25

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

If you don't have strong guarantees on that table (usually with high volume comes less requirements, ie. you can loose few rows here and there and it's not the end of the world) then you may want to try unlogged table (new versions of postgres do support it). Also make sure you're using BRIN index (if any) on that timestamp column.

Re: Beringei: A high-performance time series storage engine

#26

We're getting good results from http://traildb.io/ and we don't have to grant Facebook a worldwide, royalty-free license to our patent pool in order to use it.

I don't think TrailDB is comparable to Beringei. TrailDB is built for behavioral analytics queries while Beringei is developed for time-series aggregated metrics.

Re: Beringei: A high-performance time series storage engine

#27
post #26

We're getting good results from http://traildb.io/ and we don't have to grant Facebook a worldwide, royalty-free license to our patent pool in order to use it.

I don't think TrailDB is comparable to Beringei. TrailDB is built for behavioral analytics queries while Beringei is developed for time-series aggregated metrics.

> time-series aggregated metrics

That's exactly what we're using TrailDB for. Works great.

Re: Beringei: A high-performance time series storage engine

#28
post #8

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

CSV Same way most web servers log traffic.

Or perhaps SQLite, since it should perform similar to writing to a CSV file and you get querying out of the box afterwards.

Re: Beringei: A high-performance time series storage engine

#29

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

I think the answer depends on what tool you're using for offline analysis. Reducing the number of intermediate copies and formats would be desirable, right? Start with that, and then find some asynchronous way (to avoid blocking your app) and dump the data straight to that format.

Re: Beringei: A high-performance time series storage engine

#30

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

Given the "archive and offline analysis" requirements, which are special characteristics.

The "hacked" way is to go CSV files, store them on s3. It is very limited but it is also very simple. (Not sure what software can run queries on CSV files directly).

The "doing things right" way is to go for AWS RedShift or Google BigQuery. There is a learning curve at the start but it's really REALLY good and it will pay off by many folds later.

Post reply on HN