Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time Series, the new shiny?

basho.com

Post reply on HN
Print view View original

51–60 of 82 posts

Re: Time Series, the new shiny?

#51

Post by throwaway_xx9 »

rdtsc wrote: post #44

I see SQL support, that is interesting. Isn't Riak the premier NoSQL database. I guess it is a NoNoSQL db now ;-) The implementation of SQL part is so neat. Great work whoever did that. It uses yecc and leex that comes with Erlang and rebar even knows how to compile those. Very cool! https://github.com/basho/riak_ql

All non-trivial NoSQL databases support (or will support) SQL - otherwise programmers can insert data, but end-users can't consume it. No reporting, no revenue.

Cassandra has actually deprecated their original data access API in favor of CQL (Cassandra SQL.)

Re: Time Series, the new shiny?

#52

Post by kybernetikos »

fit2rule wrote: post #18
philh wrote: post #12

Earlier quoted context omitted.

> Time is the only true constant in the universe I'm not sure exactly what you mean by this, but relativity would seem to contradict you.

Have you ever seen Time suddenly stop? It doesn't. It just keeps on ticking. (okay, the theory of relativity may have other things to say about how those ticks propagate through space, however: we're talking about computers and user interfaces here..)

> Have you ever seen Time suddenly stop? It doesn't.

How would you know, since your awareness is embedded in time?

Re: Time Series, the new shiny?

#53

Post by mdigan »

flatwhiskers wrote: post #9

In terms of getting data into RiakTS, would streaming something through Kafka be an option for instance?

Hi, Basho employee here. Yes, Kafka is an option. Here's an example about using Kafka with Riak TS and Spark Streaming: http://docs.basho.com/riak/ts/1.3.0/add-ons/spark-riak-conne...

Re: Time Series, the new shiny?

#55

Post by epaulson »

As someone who deals with sensor data, the tricky part is really not the write-rate, but rather dealing with messy data. There's a lot of parallelism in sensor network streams, and for many domains you never look at the sensors from one device against the sensors of another device, so you can put them in entirely different databases and it doesn't matter. (It's not true in every case, of course, but if you're doing time series/streaming, ask yourself if it's true for you before picking a system)

The real pain is handling data that arrives out of order or otherwise very late, or handling data that never arrives at all, or handling data that's clearly wrong. Worse, you may have streams that are defined/calculated from other streams for some algebra on series, e.g. series C is series A plus series B - so handling new data on A means you need to recalculate/update the view for C.

Oh, and you'd like this all to be mostly declarative so you have some way to migrate between systems if you need to switch for whatever reason.

Apache Beam/Google Dataflow gets a lot of this stuff right: it's not quite as declarative as I'd like but it gets the windowing flexibility right and handles restatements at a data model level.

Re: Time Series, the new shiny?

#56

Post by eggy »

srpeck wrote: post #37
eggy wrote: post #35

Earlier quoted context omitted.

TS enhanced version of riak_core, got it. Most TS, or tick DBs, are columnar-based, memory-mapped, fast and light systems. Are there any benchmarks similar to STAC-M3, which is a year's worth of NYSE data run on different hardware to gauge kdb+'s effectiveness on different hardware configurations [1]? It's a great way to gauge performance and TCO. Does it do both memory (streaming data) and disk-based (historical) st…

You might find these benchmarks interesting: http://kparc.com/q4/readme.txt

I was inquiring about benchmarks for RiakTS, but your link was perfect. I am a J/APL dabbler, and quite recently learning kdb+/q (I prefer k).

As much as I step away from these languages, I always find my way back to them in strange ways. I was studying music, and there was a great J article in Vector magazine written in August 2006 [1] that walks through scales, and other musical concepts in J.

A Forth-based music software called Sporth [2] has a kona ugen in it, so you can generate scales or other musical items in kona, and then use them in the stack-based Sporth audio language.

My interests in kdb+/q, k, J and APL are in applying them to mathematical investigations of music, visuals, doing data analysis, and then just code golfing, or toying around. They're so much fun!

I need more time on large streaming datasets (Time Series data), than large disk-based datasets to really test latencies. I am building a box much better suited for it than my current machine. The goal is to stay in RAM as much as possible.

[1] http://archive.vector.org.uk/art10010610

[2] https://github.com/PaulBatchelor/Sporth

Re: Time Series, the new shiny?

#57

Post by thom_nic »

jsonninja wrote: post #50

For the TS experts out there, any real world experience with Influx? ( https://influxdata.com/ )

I am not an expert but from using InfluxDB I think influx supports more features in terms of aggregation/rollups/ gap filling/ retention/ etc. But I suspect Riak TS would win on certain scalability use cases because it's based on the Dynamo architecture.

Re: Time Series, the new shiny?

#58

Post by srpeck »

eggy wrote: post #56
srpeck wrote: post #37

Earlier quoted context omitted.

You might find these benchmarks interesting: http://kparc.com/q4/readme.txt

I was inquiring about benchmarks for RiakTS, but your link was perfect. I am a J/APL dabbler, and quite recently learning kdb+/q (I prefer k). As much as I step away from these languages, I always find my way back to them in strange ways. I was studying music, and there was a great J article in Vector magazine written in August 2006 [1] that walks through scales, and other musical concepts in J. A Forth-based music s…

You should definitely check out JohnEarnest/RodgerTheGreat's iKe, built on his open source k interpreter in JS. Fun examples: http://johnearnest.github.io/ok/ike/ike.html?gist=bbab46d613... and http://johnearnest.github.io/ok/ike/ike.html?gist=b741444d04...

https://github.com/JohnEarnest/ok/tree/gh-pages/ike

https://github.com/JohnEarnest/ok

And related APL/J/K subreddit: https://www.reddit.com/r/apljk/

Re: Time Series, the new shiny?

#59

Post by anthonybsd »

gricardo99 wrote: post #45
anthonybsd wrote: post #38

Earlier quoted context omitted.

KDB is not distributed and K (APL) is not particularly pleasant to work with. While it has a proven track record in fintech no one I know of is particularly fond of working with this technology. Not to mention the cost of K developers (200K+). Riak is simply offering a free alternative to these systems that is very palatable.

I’m not sure what you mean by "not distributed". With kdb+ you have a lot of flexibility in how to setup the database. You can organize the data to be stored in a distributed fashion (across multiple devices, multiple servers), you can setup query load balancers to distribute work-loads, and you can replicate to multiple servers/devices. You don’t have to use K, you code in q, which most people find far easier to rea…

>I’m not sure what you mean by "not distributed". With kdb+ you have a lot of flexibility in how to setup the database. You can organize the data to be stored in a distributed fashion (across multiple devices, multiple servers.

So I was under impression (feel free to correct me) that kdb horizontal scaling was something akin to Oracle RAC. I.e. horizontal in the name only. I.e. the data is only ever available from one physical instance at a time.

Re: Time Series, the new shiny?

#60

Post by eggy »

yummyfajitas wrote: post #48
gricardo99 wrote: post #47

Earlier quoted context omitted.

> KDB is a single box timeseries It’s not limited to a single box. “single box” is the easiest setup, but by no means the common use case. Kdb+ can be, and is, used across multiple servers and multiple storage devices. > the closest open source analogue would probably be using Pandas + a big folder of CSV files I’m not familiar with Pandas, but I know what “a big folder of CSV files” looks like and that’s pretty far…

So the file format is a lot better than CSV files, but in principle it's basically just a bunch of files. Maybe a better analogy would have been a big folder of feather/hdf5/etc files. (Incidentally, I'm a big fan of the folder/s3 bucket/etc full of CSV/binary files and use it whenever possible.) I agree - it's absolutely better to use than that, but it's a lot closer to that model than to the Riak model of querying…

kdb+/q/k are used for IOT applications [1], not just fin tec. After all, it is all time series data.

The benchmarks given in a response above by srpeck [2], shows spark/shark to be 230 times slower than a k4 query, and using 50GB or RAM vs. 0.2GB RAM for k4. If RiakTS is relying on spark/shark as the in-memory database engine, it is already at a big disadvantage compared to k in terms of speed, and all the RAM that is going to be required on those distributed servers.

I will have to look at the DDL/math functions available in RiakTS too, since that is how you get your work done regardless of speed of access.

[1] http://www.kdnuggets.com/2016/04/kxcon2016-kdb-conference-ma...

[2] http://kparc.com/q4/readme.txt

Post reply on HN