Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time Series, the new shiny?

basho.com

Post reply on HN
Print view View original

11–20 of 82 posts

Re: Time Series, the new shiny?

#11

Post by x5n1 »

pbowyer wrote: post #8
fit2rule wrote: post #6

I've always found it quite curious that computer human interfaces have always focused on the noun/verb proposition of describing data, and not the time/place. Time is the only true constant in the universe, and yet computers are set up to track and control it, seemingly, as a second thought. Imagine if instead of having files/folders to (teach,confuse) Grandma, we simply had a time-based system of references. If Time…

> Imagine if instead of having files/folders to (teach,confuse) Grandma, we simply had a time-based system of references. If Time was a principle unit of information that a user was required to understand as an abstract concept, I feel that it would result in far better user interfaces. How would it be easier when I, let alone my Grandma, can't remember if I did something last week or the week before last? It seems i…

Yeah I actually don't think the human brain accounts very well for time. Like despite a decade has passed I don't feel as if time has moved very much for me. Despite the fact that people age, it does not seem an inbuilt thing to recognize your age. It just seems to be something that happens to your body.

Thinking a bit more about it I don't think the brain accounts for time in long term memory, it does a better job in short term memory. That's why a musician can use time and we can't use it very well for stuff we stored a week ago.

Re: Time Series, the new shiny?

#12

Post by philh »

fit2rule wrote: post #6

I've always found it quite curious that computer human interfaces have always focused on the noun/verb proposition of describing data, and not the time/place. Time is the only true constant in the universe, and yet computers are set up to track and control it, seemingly, as a second thought. Imagine if instead of having files/folders to (teach,confuse) Grandma, we simply had a time-based system of references. If Time…

> Time is the only true constant in the universe

I'm not sure exactly what you mean by this, but relativity would seem to contradict you.

Re: Time Series, the new shiny?

#13

Post by eggy »

I don't know Riak, other than its a distributed NoSQL key-value data store.

Time series has always been prevalent in the fintec and quantitative finance, and other disciplines for decades. I read a book in the early 1990s on music as time series data, financial tickers, and so on.

How is Riak different, or more suited to use than Kdb + q, J with JDB (free), Jd (a commercial J database like Kdb/q)[2], or the new Kerf lang/db being developed by Kevin Lawler[3]?

Kevin also wrote kona, an opensource version of the "K programming language"[4].

Kdb is very fast at time series analysis on large datasets, and has many years of proven value in the financial industry.

[1] https://kx.com/ [2] http://www.jsoftware.com/jdhelp/overview.html [3] https://github.com/kevinlawler/kerf [4] https://github.com/kevinlawler/kona

Re: Time Series, the new shiny?

#14

Post by bbrazil »

Are there performance numbers available?

We're on the look out for suitable remote storage for prometheus.io, and would want to know the hardware that'd be required to handle 1M samples/s and how many bytes a sample takes up.

It doesn't support full float64 which we need, but we could workaround by putting it into a 64 bit unsigned number.

Re: Time Series, the new shiny?

#15

Post by Twisell »

jholman wrote: post #5
Confusion wrote: post #2

Poses the question So what’s the big deal? People have been recording temporally oriented data since we could chisel on tablets. Never answers it, but instead explains how Riak handles large time series. Certainly interesting, but I would like an answer to this question, as I don't understand the big deal.

I think the big deal is allegedly "we now have so much data that we have a use for distributed databases with high performance", which allows them to go on and say "and our big deal is that we've built a high-performance distributed database tuned especially for time series". I only noticed it because I was shocked that there was an actual comprehensible value proposition in a blogpost about a NoSQL product.

However being system admin of a PostgreSQL database managing a lot of timeseries I was just like "meh use timestamp and good index, what's the deal?". Also for performance sake just cluster the data using the most used index...

But yeah I guess if you really need NoSQL this should be nice. However sharding based on time will probably be one of the easiest approach for horizontal PostgreSQL deployment.

Re: Time Series, the new shiny?

#16

Post by baq »

the_alchemist wrote: post #10

> Riak uses the SHA hash as its distribution mechanism and divides the output range of the SHA hash evenly amongst participating nodes in the cluster. Wait, Riak uses SHA as distribution hash? Why use a cryptographic hash for distribution and not something like Murmur3, if you're talking about high-performant[0] ? [0] http://blog.reverberate.org/2012/01/state-of-hash-functions-...

i'd bet it doesn't matter.

Re: Time Series, the new shiny?

#17

Post by adamneilson »

Confusion wrote: post #2

Poses the question So what’s the big deal? People have been recording temporally oriented data since we could chisel on tablets. Never answers it, but instead explains how Riak handles large time series. Certainly interesting, but I would like an answer to this question, as I don't understand the big deal.

I think Datomic which has been doing time series data for years now answers the "what’s the big deal" question for me:

"Given a value of the database, one can obtain another value of the database as-of, or since, a point in time in the past, or both (creating a windowed view of activity). These database values can be queried ordinarily, without having to make special queries parameterized by time." http://www.datomic.com/rationale.html

So in other words you get to see exactly how the whole database looked at any given moment through history (certain parallels with the blockchain I suppose).

Re: Time Series, the new shiny?

#18

Post by fit2rule »

philh wrote: post #12
fit2rule wrote: post #6

I've always found it quite curious that computer human interfaces have always focused on the noun/verb proposition of describing data, and not the time/place. Time is the only true constant in the universe, and yet computers are set up to track and control it, seemingly, as a second thought. Imagine if instead of having files/folders to (teach,confuse) Grandma, we simply had a time-based system of references. If Time…

> Time is the only true constant in the universe I'm not sure exactly what you mean by this, but relativity would seem to contradict you.

Have you ever seen Time suddenly stop? It doesn't. It just keeps on ticking. (okay, the theory of relativity may have other things to say about how those ticks propagate through space, however: we're talking about computers and user interfaces here..)

Re: Time Series, the new shiny?

#19

Post by fit2rule »

pbowyer wrote: post #8
fit2rule wrote: post #6

I've always found it quite curious that computer human interfaces have always focused on the noun/verb proposition of describing data, and not the time/place. Time is the only true constant in the universe, and yet computers are set up to track and control it, seemingly, as a second thought. Imagine if instead of having files/folders to (teach,confuse) Grandma, we simply had a time-based system of references. If Time…

> Imagine if instead of having files/folders to (teach,confuse) Grandma, we simply had a time-based system of references. If Time was a principle unit of information that a user was required to understand as an abstract concept, I feel that it would result in far better user interfaces. How would it be easier when I, let alone my Grandma, can't remember if I did something last week or the week before last? It seems i…

You think its better to have a situation where you can't find where something is, nor even approximate it, but you can generally approximate when something is, and grope around it? Because I don't think thats the case - if you can't remember where you put it, its not helpful to grope around the location of where you think you might've put it - or at least, lets say this: it might be more helpful to group around time, than space, when dealing with abstract ideas.

A user interface can be derived to help you do that groping with Time. User interfaces have been derived to help you do groping with form or place (Spotlight, cf.), too, but I wager Time-groping is more effective. But that could just be the musician in me...

Re: Time Series, the new shiny?

#20

Post by athenot »

eggy wrote: post #13

I don't know Riak, other than its a distributed NoSQL key-value data store. Time series has always been prevalent in the fintec and quantitative finance, and other disciplines for decades. I read a book in the early 1990s on music as time series data, financial tickers, and so on. How is Riak different, or more suited to use than Kdb + q, J with JDB (free), Jd (a commercial J database like Kdb/q)[2], or the new Kerf…

Minor quibble: the article is about RiakTS, their time-series enhanced version of riak_core.

riak_core's main strength is that it does key-value in a distributed/resilient manner, spreading values in multiple copies (at least 3) all over a cluster of servers. Kill one server, no problem. Need more capacity, add servers and it will rebalance itself.

The TS part is just an optimization built on top of that, to make values which are near each other in time to be near each other in storage, for faster range-base retrieval.

Post reply on HN