Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beringei: A high-performance time series storage engine

code.facebook.com

Post reply on HN
Print view View original

11–20 of 58 posts

Re: Beringei: A high-performance time series storage engine

#11

Post by storyinmemo »

cocoflunchy wrote: post #4

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

Can you elaborate on what it is that slows down your app performance? Is it bottle-necking on writing to the database? Does the database have read activity? Does the application slow down when there is read activity?

With some basic assumptions on my part including that you can have a delay in writing data to the database (since it's archival and analysis), but you don't want the application to be delayed, putting a fast queue like Kafka or RabbitMQ in place could help. It'll buffer when the database is under load, isolating your application from that.

Offline analysis might also skip the database entirely: columnar data formats (parquet) or log structured merge / sorted string tables (rocksdb) could work very well for archival or offline analysis in MR type environments.

Re: Beringei: A high-performance time series storage engine

#12

Post by jdonaldson »

storyinmemo wrote: post #10
crescentfresh wrote: post #9

Ok so it's an inmemory product, sharded no less. They speak about compressing the data before "storing" it. I don't have a lot of experience with inmemory anything, but are we talking about retaining the compressed format in server memory here? Ie, RAM is your datastore. Then, at some point, to serve requests/queries for the data don't you have to get it "out of" RAM and uncompress it, also an inmemory operation? Did…

The paper on the algorithm is here: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf Somebody implemented the algorithm in go based on the paper here: https://github.com/dgryski/go-tsz A short answer that may work for your question: the bits that are set in RAM are xor values relative to previous values. To provide an answer as to what the value is, a series of read|xor operations are performed.

They refer to it as "delta of delta", which implies the encoding handles things like acceleration and momentum. I'm guessing the healthcare data has much more of this sort of behavior than your typical server event time series.

Re: Beringei: A high-performance time series storage engine

#13

Post by robochat »

cocoflunchy wrote: post #4

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

You could try influxdb.

Re: Beringei: A high-performance time series storage engine

#14

Post by ju-st »

pdq wrote: post #8
cocoflunchy wrote: post #4

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

CSV Same way most web servers log traffic.

Considering the use case ("archiving and offline analysis") this is likely the best choice. If you need some buzzwords or want to get fancy, try HDFS

Re: Beringei: A high-performance time series storage engine

#16

Post by Scaevolus »

jdonaldson wrote: post #12
storyinmemo wrote: post #10

Earlier quoted context omitted.

The paper on the algorithm is here: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf Somebody implemented the algorithm in go based on the paper here: https://github.com/dgryski/go-tsz A short answer that may work for your question: the bits that are set in RAM are xor values relative to previous values. To provide an answer as to what the value is, a series of read|xor operations are performed.

They refer to it as "delta of delta", which implies the encoding handles things like acceleration and momentum. I'm guessing the healthcare data has much more of this sort of behavior than your typical server event time series.

Not quite. Delta of deltas means that periodic timestamps turn into a long run of zeros (since the difference between each point is constant), which is very easy to compress. In the Gorilla paper, 96% of timestamps compress down to zero, which is encoded by a single zero bit.

For series without periodic sampling, delta-of-deltas performs similarly to normal deltas.

Re: Beringei: A high-performance time series storage engine

#17

Post by jdwyah »

cocoflunchy wrote: post #4

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

Kinesis firehose to S3 and then query with Athena is pretty great. I've been very happy with the combo.

Re: Beringei: A high-performance time series storage engine

#18

Post by gaius »

cocoflunchy wrote: post #4

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

What would HN suggest to store about 1GB of data per day,

1Gb per day is not a lot - that's 12k/sec which is trivial. Your Postgres must be badly misconfigured!

Re: Beringei: A high-performance time series storage engine

#19

Post by IanCal »

pdq wrote: post #8
cocoflunchy wrote: post #4

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

CSV Same way most web servers log traffic.

Strongly recommend this. Your disks are not going to struggle with 1G/day, it's easy to setup & manage, easy to move data around and easy to process later. Worst case is it's quick to try and see if it suits.

Re: Beringei: A high-performance time series storage engine

#20

Post by rakoo »

jdonaldson wrote: post #12
storyinmemo wrote: post #10

Earlier quoted context omitted.

The paper on the algorithm is here: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf Somebody implemented the algorithm in go based on the paper here: https://github.com/dgryski/go-tsz A short answer that may work for your question: the bits that are set in RAM are xor values relative to previous values. To provide an answer as to what the value is, a series of read|xor operations are performed.

They refer to it as "delta of delta", which implies the encoding handles things like acceleration and momentum. I'm guessing the healthcare data has much more of this sort of behavior than your typical server event time series.

"delta of delta" is only used for timestamps, not for values. In any standard monitoring setup, the acceleration of timestamps will be 0 for a big portion of measurements, and very small for a big portion of the rest, so it makes compression extremely easy and efficient.
Post reply on HN