Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time Series, the new shiny?

basho.com

Post reply on HN
Print view View original

61–70 of 82 posts

Re: Time Series, the new shiny?

#61

Post by siculars »

epaulson wrote: post #55

As someone who deals with sensor data, the tricky part is really not the write-rate, but rather dealing with messy data. There's a lot of parallelism in sensor network streams, and for many domains you never look at the sensors from one device against the sensors of another device, so you can put them in entirely different databases and it doesn't matter. (It's not true in every case, of course, but if you're doing t…

>The real pain is handling data that arrives out of order or otherwise very late

Riak TS uses leveldb under the hood. Leveldb is natively sorted. In riak ts that includes the bucket. so the sort order is basically bucket/%PK where PK is your composite PK as defined in your CREATE TABLE statement. See Local Key [0].

[0] http://docs.basho.com/riak/ts/1.3.0/using/planning/

Re: Time Series, the new shiny?

#62

Post by yummyfajitas »

eggy wrote: post #60
yummyfajitas wrote: post #48

Earlier quoted context omitted.

So the file format is a lot better than CSV files, but in principle it's basically just a bunch of files. Maybe a better analogy would have been a big folder of feather/hdf5/etc files. (Incidentally, I'm a big fan of the folder/s3 bucket/etc full of CSV/binary files and use it whenever possible.) I agree - it's absolutely better to use than that, but it's a lot closer to that model than to the Riak model of querying…

kdb+/q/k are used for IOT applications [1], not just fin tec. After all, it is all time series data. The benchmarks given in a response above by srpeck [2], shows spark/shark to be 230 times slower than a k4 query, and using 50GB or RAM vs. 0.2GB RAM for k4. If RiakTS is relying on spark/shark as the in-memory database engine, it is already at a big disadvantage compared to k in terms of speed, and all the RAM that i…

Very cool, I stand corrected. I hope one day I have another opportunity to play with KDB.

As for the speed advantage, you'll have a similar speed advantage with python/pandas/big folder of CSV files. For all of Spark's claims on "speed", it's really just reducing the speed penalty of Hadoop from 500x to 50x. (Here 500x and 50x refer to the performance of loading flat files from a disk.)

Re: Time Series, the new shiny?

#64

Post by svjethani »

bbrazil wrote: post #14

Are there performance numbers available? We're on the look out for suitable remote storage for prometheus.io, and would want to know the hardware that'd be required to handle 1M samples/s and how many bytes a sample takes up. It doesn't support full float64 which we need, but we could workaround by putting it into a 64 bit unsigned number.

We have engaged a 3rd party that will be doing testing that will get published. Currently we have done testing around specific customer use cases.

Re: Time Series, the new shiny?

#65

Post by eggy »

srpeck wrote: post #58
eggy wrote: post #56

Earlier quoted context omitted.

I was inquiring about benchmarks for RiakTS, but your link was perfect. I am a J/APL dabbler, and quite recently learning kdb+/q (I prefer k). As much as I step away from these languages, I always find my way back to them in strange ways. I was studying music, and there was a great J article in Vector magazine written in August 2006 [1] that walks through scales, and other musical concepts in J. A Forth-based music s…

You should definitely check out JohnEarnest/RodgerTheGreat's iKe, built on his open source k interpreter in JS. Fun examples: http://johnearnest.github.io/ok/ike/ike.html?gist=bbab46d613... and http://johnearnest.github.io/ok/ike/ike.html?gist=b741444d04... https://github.com/JohnEarnest/ok/tree/gh-pages/ike https://github.com/JohnEarnest/ok And related APL/J/K subreddit: https://www.reddit.com/r/apljk/

I had stumbled upon John's work before. I am currently dabbling with a stack-based audio language called Sporth [1], and messing with the idea of somehow mashing it up with John's ike project.

See, vector/array languages aren't just for FinTech or Time Series!

[1] https://github.com/PaulBatchelor/Sporth

Re: Time Series, the new shiny?

#66

Post by rdtsc »

gordonguthrie wrote: post #49
rdtsc wrote: post #44

I see SQL support, that is interesting. Isn't Riak the premier NoSQL database. I guess it is a NoNoSQL db now ;-) The implementation of SQL part is so neat. Great work whoever did that. It uses yecc and leex that comes with Erlang and rebar even knows how to compile those. Very cool! https://github.com/basho/riak_ql

[takes a bow]

Awesome work!

Re: Time Series, the new shiny?

#67

Post by siculars »

rch wrote: post #43

Love the simple install for development on a Mac. Thanks for that.

I can't speak for the engineers but a lot of folks at basho are constantly building and tearing down riak single instance or multi instance clusters on redhat/ubuntu virtual machines. I do that and I also have different versions of riak sitting in their own folders on my mac hd. mac osx support +1.

Re: Time Series, the new shiny?

#68

Post by thinkdevcode »

jsonninja wrote: post #50

For the TS experts out there, any real world experience with Influx? ( https://influxdata.com/ )

Fantastic db, but maybe not quite ready for production usage. We are using influx for a small portion of our ingestion engine as well as for storing server metrics. The updates/improvements have been pretty astounding over the last year, but also hard to keep up. I had to fork the nodejs library just to update it from 0.9 to 0.12 [0] because there were a LOT of breaking changes. Pre-0.9 there were many issues we ran into when it came to disk space & performance but they have all been resolved as of 0.9.

Another thing to note (after speaking with them on a few occasions) is that they are only providing cluster support to their enterprise offering which wont be available till this summer. They do offer Relay which is their high availability tool for the open source version.[1] You really won't need clustering unless you're doing an insane amount of writes: single server performance is insane right now. We average 10k writes/sec with bursts up to 5x that and it doesn't break a sweat (on a cheap 2 cpu/7gb ram instance, with ssd block storage).

[0] https://github.com/thinkdevcode/node-influx

[1] https://docs.influxdata.com/influxdb/v0.12/high_availability...

Re: Time Series, the new shiny?

#69

Post by im_down_w_otp »

Might be worth looking into dalmatiner.io (DalmatinerDB) as an alternative to this. It's also built on riak_core to manage cluster membership and the top-level framework for dealing with routing and rebalancing.

Waited for a long time for Riak TS to come out. Tried KairosDB & Cyanite, but the operational overhead of Cassandra wasn't something I wanted to buy into for such a narrow use case (infrastructure metrics store), and then suddenly out of nowhere DalmatinerDB was released. The code is clean, the architecture is solid, and the ops story is simple.

I don't have any affiliation of any kind with the Dataloop folks. I am however a happy end-user. We do currently use Riak KV due to its CRDT support though.

Re: Time Series, the new shiny?

#70

Post by im_down_w_otp »

the_alchemist wrote: post #10

> Riak uses the SHA hash as its distribution mechanism and divides the output range of the SHA hash evenly amongst participating nodes in the cluster. Wait, Riak uses SHA as distribution hash? Why use a cryptographic hash for distribution and not something like Murmur3, if you're talking about high-performant[0] ? [0] http://blog.reverberate.org/2012/01/state-of-hash-functions-...

The hash function in this case is used purely for generating an integer from a small binary blob (bucket/key pair) and that integer is the deterministic artifact that tells Riak Core which machines and hash partitions are supposed to own that data.

The performance impact of that is massively dwarfed (by probably 3+ orders of magnitude) by everything else that's going on in the critical sequential read/write path.

Post reply on HN