Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beringei: A high-performance time series storage engine

code.facebook.com

Post reply on HN
Print view View original

31–40 of 58 posts

Re: Beringei: A high-performance time series storage engine

#31

Post by agacera »

cocoflunchy wrote: post #4

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

If you are already invested in aws I suggest you to use the firehose + s3 to store data and use the Athena to query it.

Re: Beringei: A high-performance time series storage engine

#32

Post by throwaway2016a »

jdonaldson wrote: post #12
storyinmemo wrote: post #10

Earlier quoted context omitted.

The paper on the algorithm is here: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf Somebody implemented the algorithm in go based on the paper here: https://github.com/dgryski/go-tsz A short answer that may work for your question: the bits that are set in RAM are xor values relative to previous values. To provide an answer as to what the value is, a series of read|xor operations are performed.

They refer to it as "delta of delta", which implies the encoding handles things like acceleration and momentum. I'm guessing the healthcare data has much more of this sort of behavior than your typical server event time series.

> I'm guessing the healthcare data has much more of this sort of behavior than your typical server event time series.

I thought they were talking about healthcare data too when I first read it but seeing as this is Facebook I realized they are talking about server health not human health.

Re: Beringei: A high-performance time series storage engine

#33

Post by throwaway2016a »

weego wrote: post #6
cocoflunchy wrote: post #4

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

If you want low to no maintenance and maintain some of what you have I'd suggest pushing it into kinesis instead of postgres and then dumping to s3 as you're already doing.

Second this. This use case is exactly what Kinesis was built for in the first place.

Additionally, Dynamodb can be decent for this as long as you only need to sort on time.

Re: Beringei: A high-performance time series storage engine

#34

Post by no0b20 »

erichocean wrote: post #27
buremba wrote: post #26

Earlier quoted context omitted.

I don't think TrailDB is comparable to Beringei. TrailDB is built for behavioral analytics queries while Beringei is developed for time-series aggregated metrics.

> time-series aggregated metrics That's exactly what we're using TrailDB for. Works great.

I'm curious about what kind of write speeds you are using traildb for?

Re: Beringei: A high-performance time series storage engine

#35

Post by erichocean »

no0b20 wrote: post #34
erichocean wrote: post #27

Earlier quoted context omitted.

> time-series aggregated metrics That's exactly what we're using TrailDB for. Works great.

I'm curious about what kind of write speeds you are using traildb for?

We trace every server request with it, across all stages of the request—approx. 10-12 events/timestamps per request—across multiple processes (think Zipkin). The per-request event streams are independent "trails" per process, and then we merge them together later and compute aggregated metrics using hdr_histogram.

Individual events are typically hundreds to thousands of nanoseconds long, with about 20 nanoseconds of overhead to grab a timestamp using the rdtscp instruction.

Hope that helps!

Re: Beringei: A high-performance time series storage engine

#36

Post by cocoflunchy »

storyinmemo wrote: post #11
cocoflunchy wrote: post #4

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

Can you elaborate on what it is that slows down your app performance? Is it bottle-necking on writing to the database? Does the database have read activity? Does the application slow down when there is read activity? With some basic assumptions on my part including that you can have a delay in writing data to the database (since it's archival and analysis), but you don't want the application to be delayed, putting a…

Yes, we only have one DB for everything and it is bottle-necking on writes. Queuing the writes is probably the easiest solution since we already have RabbitMQ setup... thanks for your answer.

Re: Beringei: A high-performance time series storage engine

#37

Post by no0b20 »

erichocean wrote: post #35
no0b20 wrote: post #34

Earlier quoted context omitted.

I'm curious about what kind of write speeds you are using traildb for?

We trace every server request with it, across all stages of the request—approx. 10-12 events/timestamps per request—across multiple processes (think Zipkin). The per-request event streams are independent "trails" per process, and then we merge them together later and compute aggregated metrics using hdr_histogram. Individual events are typically hundreds to thousands of nanoseconds long, with about 20 nanoseconds of…

I'm not sure I follow everything what you wrote. I'm interested in how many inserts are you doing per second to the DB. Is it in 1000s/sec or millions/sec.

Re: Beringei: A high-performance time series storage engine

#38

Post by zo1 »

cocoflunchy wrote: post #4

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

Don't write it to your main transactional database, then.

Have a separate connection to a DB on another machine that is handling the load. Do that within the transaction/request from the client, or pop it into a message queue.

The queue will "persist" the data, but will also schedule it for when your other box/DB can ingest it. At the end of the day, if you're pushing too-much data, you need to have a mechanism in place to let you scale. To me, a simple solution would be a message queue server with plenty of redundant storage space. From there on, you keep adding more consumers.

Re: Beringei: A high-performance time series storage engine

#39

Post by GordonS »

zo1 wrote: post #38
cocoflunchy wrote: post #4

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

Don't write it to your main transactional database, then. Have a separate connection to a DB on another machine that is handling the load. Do that within the transaction/request from the client, or pop it into a message queue. The queue will "persist" the data, but will also schedule it for when your other box/DB can ingest it. At the end of the day, if you're pushing too-much data, you need to have a mechanism in pl…

If the message queue becomes a bottleneck, buffering/batching can also help - instead of sending individual messages to the queue, send them in batches (based on the number, time period, or both).

Re: Beringei: A high-performance time series storage engine

#40

Post by erichocean »

no0b20 wrote: post #37
erichocean wrote: post #35

Earlier quoted context omitted.

We trace every server request with it, across all stages of the request—approx. 10-12 events/timestamps per request—across multiple processes (think Zipkin). The per-request event streams are independent "trails" per process, and then we merge them together later and compute aggregated metrics using hdr_histogram. Individual events are typically hundreds to thousands of nanoseconds long, with about 20 nanoseconds of…

I'm not sure I follow everything what you wrote. I'm interested in how many inserts are you doing per second to the DB. Is it in 1000s/sec or millions/sec.

Millions, not thousands, of inserts per second. TrailDB is hella fast (that's why we use it).
Post reply on HN