71–80 of 132 posts
Well, all you need to do is to look at KDB+ from kx. All the other products that you mention are for children ;)
Why does every TSD seem so overly engineered and for all the wrong reasons? Why not just use a time decaying ring buffer (multiple buffers could be used), one statistic one file (or more depending on the decay) and offset by a set interval if you have irregular intervals 'smooth' it to fit O(1) for most things. My other issue is (from a glance) looking at some TSD they ignore most research done on how to effectively…
Why does every TSD seem so overly engineered and for all the wrong reasons? Why not just use a time decaying ring buffer (multiple buffers could be used), one statistic one file (or more depending on the decay) and offset by a set interval if you have irregular intervals 'smooth' it to fit O(1) for most things. My other issue is (from a glance) looking at some TSD they ignore most research done on how to effectively…
What you are describing is exactly what Carbon does (The datastore of Graphite). It's a few hundred lines of not-exactly-production-quality python. But if you'd actually RTFA, you'd know that Whisper is hardly the end-all of TSD, that cache to batch is not so trivial as you make it seem. I do believe there's sort of optimal solution that isn't very complex and not far away from many solutions out there. But most defi…
Well, all you need to do is to look at KDB+ from kx. All the other products that you mention are for children ;)
I know the Kx people pretty well, and they are trying to get the word out (have been for years), but it never ceases to amaze how little respect they get in the free software world. They are the leading timeseries database, and yet they don't even get a footnote in the article :(
Yes, I know there is a free version, but limited to 32-bit only (and probably non-commercial?).
EDIT: 32-bit version can be used commercially.
We store our event stream data in Elasticsearch. Two features that made it appealing: * the ingest-side can be scaled up by adding more shards * the query-side can be scaled up by adding more replicas To compute rollup analytics, we make heavy use of Elasticsearch's aggregation framework to compute daily/weekly/monthly/quarterly active users. From my understanding Postgres has many of these features, but the distribu…
That said, one major downside to ES is that it's not schemaless. You can try to use the dynamic mapping system, but it will most likely just bite you eventually, since ES is strict about coercing data types. If your data isn't completely consistent, it will actually refuse to index it. Any changes made to your schema also requires reindexing. (For some reason ES can't do in-place indexing, despite supporting storing all the original data in the "_source" field.)
If your data isn't perfectly consistent, one way to work around the mapping problem is to append a type name to every field. So instead of indexing {"user_id": "3"}, you index {"user_id.string": "3"}. This means that if you get some input data where the user_id is an int, it doesn't conflict because it will stored in "user_id.int". You have to handle the inconsistency on the query end, but it's possibly better than micromanaging the index.
It seems that the vast majority of these open-source TSDBs are focused on fairly technical event stream data arising specifically from IT infrastructure... is anyone using them with success in more business-oriented domains, e.g. energy meter data, stock trading, other telemetry?
0.8 way okay, but very slow and unstable. But in 0.9 the data model is very different and no longer a good fit for this type of analytics (it looks great for devops metrics, though), so we're abandoning InfluxDB altogether.
We're now in the process of migrating to ElasticSearch, which is looking much better. ElasticSearch has its own problems, though, and we will be evaluating the same dataset in PostgreSQL soon.
Well, all you need to do is to look at KDB+ from kx. All the other products that you mention are for children ;)
I know the Kx people pretty well, and they are trying to get the word out (have been for years), but it never ceases to amaze how little respect they get in the free software world. They are the leading timeseries database, and yet they don't even get a footnote in the article :(
I was however focusing on recent Open Source efforts and on the general approaches. Hence, I didn't really discuss in detail my own tsdb.
I also find virtually everything to do with K and Kdb to be be simultaneously impressive and utterly unfathomable:
http://code.kx.com/wsvn/code/kx/kdb%2B/s.k
This is a cheap jab to make, but it makes these systems pretty impenetrable from a source level.
Really? The big boy in the field KDB+ isn't mentioned? Kx's database is pretty much the gold standard for performance in time series, historical and real-time. http://kxcommunity.com/
Part of the reason why it’s hard to learn (unless your job depends on it so you are forced to persist) is that the syntax is very terse. Check out this Java API for example: http://kx.com/q/c/kx/c.java. Yes, that’s the actual code you copy-paste into your Eclipse to get started.