Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beringei: A high-performance time series storage engine

code.facebook.com

Post reply on HN
Print view View original

51–58 of 58 posts

Re: Beringei: A high-performance time series storage engine

#51

Post by machineman44 »

jdonaldson wrote: post #12
storyinmemo wrote: post #10

Earlier quoted context omitted.

The paper on the algorithm is here: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf Somebody implemented the algorithm in go based on the paper here: https://github.com/dgryski/go-tsz A short answer that may work for your question: the bits that are set in RAM are xor values relative to previous values. To provide an answer as to what the value is, a series of read|xor operations are performed.

They refer to it as "delta of delta", which implies the encoding handles things like acceleration and momentum. I'm guessing the healthcare data has much more of this sort of behavior than your typical server event time series.

Haha, I thought jdonaldson was making a joke :P

Re: Beringei: A high-performance time series storage engine

#52

Post by erichocean »

sanjayts wrote: post #48
erichocean wrote: post #35

Earlier quoted context omitted.

We trace every server request with it, across all stages of the request—approx. 10-12 events/timestamps per request—across multiple processes (think Zipkin). The per-request event streams are independent "trails" per process, and then we merge them together later and compute aggregated metrics using hdr_histogram. Individual events are typically hundreds to thousands of nanoseconds long, with about 20 nanoseconds of…

If you don't mind asking: how do you uniquely identify the requests? Is it a composite field made up of `client_ip:timestamp` or a random UUID? What does a typical payload sent across to the TSDB look like? Also I'm assuming the services are written in a language like C or C++?

• random UUID

• we track function entry/exit/throw, offset (in nanoseconds) from the initial timestamp, and in the case of a throw, we also capture a stack trace (which is not stored in TrailDB)

• our server is written in C++

Re: Beringei: A high-performance time series storage engine

#53

Post by erichocean »

buremba wrote: post #49
erichocean wrote: post #40

Earlier quoted context omitted.

Millions, not thousands, of inserts per second. TrailDB is hella fast (that's why we use it).

So you have a batching mechanism for creating TrailDB files, right? Then you're able to query historical metrics by processing these TrailDB files.

Correct. We mostly use hdr_histogram right now for the post-processing, since stable latency is what we care about.

We also do some hdr_histogram processing for our dashboard in parallel to storing traces in TrailDB, and then we retain the individual traces to do longer term processing or when we're tracking down issues in production.

Re: Beringei: A high-performance time series storage engine

#54

Post by cocoflunchy »

kvz wrote: post #5
cocoflunchy wrote: post #4

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

One idea might be to write to S3 directly and use Athena for your offline analysis

Thanks for Athena, I hadn't heard about it!

Re: Beringei: A high-performance time series storage engine

#55

Post by cocoflunchy »

weego wrote: post #6
cocoflunchy wrote: post #4

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

If you want low to no maintenance and maintain some of what you have I'd suggest pushing it into kinesis instead of postgres and then dumping to s3 as you're already doing.

Kinesis looks pretty good but it's yet another (proprietary) product to integrate and maintain... I'd rather avoid that if I can for now, we're a very small team and the fewer tools we have to use the better.

Re: Beringei: A high-performance time series storage engine

#56

Post by vegabook »

cocoflunchy wrote: post #4

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

teafiles

Re: Beringei: A high-performance time series storage engine

#58

Post by rodionos »

cocoflunchy wrote: post #4

What would HN suggest to store about 1GB of data per day, mostly for archiving and offline analysis, with less than 10 columns including timestamp? We're currently writing everything to our postgres DB and flushing the table to S3 every few days but it's killing the app performance under high loads. I'm looking for something that is easy to set up and keep running with low to no maintenance.

Try Axibase Time Series Database. We support SQL which should work well for analytics use cases.

SQL docs: https://github.com/axibase/atsd-docs/tree/master/api/sql

Analytics examples: https://github.com/axibase/atsd-use-cases

Post reply on HN