31–40 of 82 posts
Earlier quoted context omitted.
Isn't it more about the number of cycles it takes to generate the hash more than it's size, since you're likely to do that very often in a db context?
Probably. I'd venture to say the reason we use SHA is due to its uniform distribution over its speed (or lack thereof.)
I've always found it quite curious that computer human interfaces have always focused on the noun/verb proposition of describing data, and not the time/place. Time is the only true constant in the universe, and yet computers are set up to track and control it, seemingly, as a second thought. Imagine if instead of having files/folders to (teach,confuse) Grandma, we simply had a time-based system of references. If Time…
https://archive.wired.com/wired/archive/5.02/fflifestreams_p...
http://www.wired.com/2013/02/the-end-of-the-web-computers-an...
Poses the question So what’s the big deal? People have been recording temporally oriented data since we could chisel on tablets. Never answers it, but instead explains how Riak handles large time series. Certainly interesting, but I would like an answer to this question, as I don't understand the big deal.
I think Datomic which has been doing time series data for years now answers the "what’s the big deal" question for me: "Given a value of the database, one can obtain another value of the database as-of, or since, a point in time in the past, or both (creating a windowed view of activity). These database values can be queried ordinarily, without having to make special queries parameterized by time." http://www.datomic…
Datomic is an ok choice in some contexts, but the one detailed here is not one of them, you're likely to reach Datomic limits quickly and be in a world of hurt when you do.
Earlier quoted context omitted.
Isn't it more about the number of cycles it takes to generate the hash more than it's size, since you're likely to do that very often in a db context?
Probably. I'd venture to say the reason we use SHA is due to its uniform distribution over its speed (or lack thereof.)
Earlier quoted context omitted.
Minor quibble: the article is about RiakTS, their time-series enhanced version of riak_core. riak_core's main strength is that it does key-value in a distributed/resilient manner, spreading values in multiple copies (at least 3) all over a cluster of servers. Kill one server, no problem. Need more capacity, add servers and it will rebalance itself. The TS part is just an optimization built on top of that, to make val…
Yes, exactly. (Author)
Most TS, or tick DBs, are columnar-based, memory-mapped, fast and light systems.
Are there any benchmarks similar to STAC-M3, which is a year's worth of NYSE data run on different hardware to gauge kdb+'s effectiveness on different hardware configurations [1]? It's a great way to gauge performance and TCO.
Does it do both memory (streaming data) and disk-based (historical) storage for big data set analytics in realtime?
I'd be interested to see numbers there.
A lot of people think kdb+ is only for finance. There is a conference coming up in May that will have talks on q (the language for the kdb+ database) about natural language processing and machine learning in q to name a few. Another is about using it at a power plant to most efficiently route power based upon realtime data [2].
I only got into kdb+ and q with the free, non-commercial 32-bit version. I usually use J and sometimes APL, which had MapReduce since at least the 80s for APL.Check out this post from 2009 [3]. I guess the 'new shiny' bit threw me in your chosen title.
[1] https://stacresearch.com/news/2014/02/13/stac-reports-intel-...
[2] https://kxcon2016.com/agenda/
[3] http://blog.data-miners.com/2009/04/mapreduce-hadoop-everyth...
> Riak uses the SHA hash as its distribution mechanism and divides the output range of the SHA hash evenly amongst participating nodes in the cluster. Wait, Riak uses SHA as distribution hash? Why use a cryptographic hash for distribution and not something like Murmur3, if you're talking about high-performant[0] ? [0] http://blog.reverberate.org/2012/01/state-of-hash-functions-...
Earlier quoted context omitted.
Yes, exactly. (Author)
TS enhanced version of riak_core, got it. Most TS, or tick DBs, are columnar-based, memory-mapped, fast and light systems. Are there any benchmarks similar to STAC-M3, which is a year's worth of NYSE data run on different hardware to gauge kdb+'s effectiveness on different hardware configurations [1]? It's a great way to gauge performance and TCO. Does it do both memory (streaming data) and disk-based (historical) st…
I don't know Riak, other than its a distributed NoSQL key-value data store. Time series has always been prevalent in the fintec and quantitative finance, and other disciplines for decades. I read a book in the early 1990s on music as time series data, financial tickers, and so on. How is Riak different, or more suited to use than Kdb + q, J with JDB (free), Jd (a commercial J database like Kdb/q)[2], or the new Kerf…
> Riak uses the SHA hash as its distribution mechanism and divides the output range of the SHA hash evenly amongst participating nodes in the cluster. Wait, Riak uses SHA as distribution hash? Why use a cryptographic hash for distribution and not something like Murmur3, if you're talking about high-performant[0] ? [0] http://blog.reverberate.org/2012/01/state-of-hash-functions-...
I don't know Riak, other than its a distributed NoSQL key-value data store. Time series has always been prevalent in the fintec and quantitative finance, and other disciplines for decades. I read a book in the early 1990s on music as time series data, financial tickers, and so on. How is Riak different, or more suited to use than Kdb + q, J with JDB (free), Jd (a commercial J database like Kdb/q)[2], or the new Kerf…
The use case is storing tick data + economic data for all the symbols. I.e., one timeseries per publicly traded company. The primary use case is loading a significant chunk of that data and running some statistical analysis on it.
Also KDB pricing starts at $100k or something like that.
In contrast, Riak's timeseries product is distributed. It could store millions of timeseries if you throw enough boxes at it. You probably won't be loading all the data for analysis, probably you'll be processing some of the data one series at a time, and the rest is just for reference.
The main use case here is sensor networks (aka "internet of things") more than financial data. I.e., one timeseries per rotor on a drone, or per sensor on your phone, sometihng like that.