For the TS experts out there, any real world experience with Influx? ( https://influxdata.com/ )
Likewise can anyone compare against Druid? So many options, yet it's hard to bet your product/service on a stack...
Time Series, the new shiny?
71–80 of 82 posts
Re: Time Series, the new shiny?
#72Time series does not necessarily have to be about 'huge' data either, just a much greater level of historical precision. Example:
ISP sells a circuit with 95th percentile billing to a customer.
If you poll SNMP data from a router interface on 60 second intervals and store it in an RRA file, you will lose a great deal of precision over time (because RRAs are highly compressed over time). You'll have no ability to go back and pull a query like "We want to see traffic stats for the DDoS this customer took at 9am on February 26th of last year".
with time series statistics you can then feed it into tools such as grafana for visualization.
An implementation such as openTSDB to grab the traffic stats for a particular SNMP OID and store it will allow you to store all traffic data forever and retrieve it as needed later on. The amount of data written per 60 second interval is miniscule, a server with a few hundred GB of SSD storage will be sufficient to store all traffic stats for relevant interfaces on core/agg routers for a fairly large sized ISP for several years.
Re: Time Series, the new shiny?
#73For the TS experts out there, any real world experience with Influx? ( https://influxdata.com/ )
Fantastic db, but maybe not quite ready for production usage. We are using influx for a small portion of our ingestion engine as well as for storing server metrics. The updates/improvements have been pretty astounding over the last year, but also hard to keep up. I had to fork the nodejs library just to update it from 0.9 to 0.12 [0] because there were a LOT of breaking changes. Pre-0.9 there were many issues we ran…
InfluxDB CTO here, thanks for the note! We've definitely had significant improvements even over the last two months. And the breaking API changes will be a thing of the past very soon: https://github.com/influxdata/influxdb/blob/master/CHANGELOG...
0.13 drops Thursday and 1.0 is the next release :)
Re: Time Series, the new shiny?
#74Earlier quoted context omitted.
kdb+/q/k are used for IOT applications [1], not just fin tec. After all, it is all time series data. The benchmarks given in a response above by srpeck [2], shows spark/shark to be 230 times slower than a k4 query, and using 50GB or RAM vs. 0.2GB RAM for k4. If RiakTS is relying on spark/shark as the in-memory database engine, it is already at a big disadvantage compared to k in terms of speed, and all the RAM that i…
Very cool, I stand corrected. I hope one day I have another opportunity to play with KDB. As for the speed advantage, you'll have a similar speed advantage with python/pandas/big folder of CSV files. For all of Spark's claims on "speed", it's really just reducing the speed penalty of Hadoop from 500x to 50x. (Here 500x and 50x refer to the performance of loading flat files from a disk.)
Do you really mean flat CSV text files? I get the simplicity of that, but it seems really expensive (speed and size). But I'm used to tables with more than a dozen columns, and with kdb+ you only be pull in the columns of interest, and the rows of interest (due to on-disk sorting and grouping), which is a smaller subset, often much smaller.
Re: Time Series, the new shiny?
#75I don't know Riak, other than its a distributed NoSQL key-value data store. Time series has always been prevalent in the fintec and quantitative finance, and other disciplines for decades. I read a book in the early 1990s on music as time series data, financial tickers, and so on. How is Riak different, or more suited to use than Kdb + q, J with JDB (free), Jd (a commercial J database like Kdb/q)[2], or the new Kerf…
Time series analysis is also an important component of weather and climate research.
Re: Time Series, the new shiny?
#76Earlier quoted context omitted.
Very cool, I stand corrected. I hope one day I have another opportunity to play with KDB. As for the speed advantage, you'll have a similar speed advantage with python/pandas/big folder of CSV files. For all of Spark's claims on "speed", it's really just reducing the speed penalty of Hadoop from 500x to 50x. (Here 500x and 50x refer to the performance of loading flat files from a disk.)
Do you really mean flat CSV text files? I get the simplicity of that, but it seems really expensive (speed and size). But I'm used to tables with more than a dozen columns, and with kdb+ you only be pull in the columns of interest, and the rows of interest (due to on-disk sorting and grouping), which is a smaller subset, often much smaller.
By number, my data sets are usually in CSV. I could probably get some additional advantage via HDF5, but a gzipped CSV is usually good enough and simpler. By volume (i.e. on my 2 or 3 biggest data sets) I'll probably be mostly HDF5. I haven't tried feather yet but it looks pretty nice.
KDB would probably be better, but don't underestimate what you can do with just a bunch of files.
Re: Time Series, the new shiny?
#77Poses the question So what’s the big deal? People have been recording temporally oriented data since we could chisel on tablets. Never answers it, but instead explains how Riak handles large time series. Certainly interesting, but I would like an answer to this question, as I don't understand the big deal.
Immediately following the part you quote (my emphasis) Well, as it turns out, thanks to the software-eating- the-world thing and the Internet of Things we happen to be amassing *vast quantities* of all sorts of data [...] The demand for systems that are capable of storing and retrieving temporal data on an *ever increasing scale* necessitates systems that are specifically designed for this purpose. Strongly implying…
That's an obstacle that needs to be overcome because you want to get somewhere. A challenging obstacle that has spawned an entire industry, but nevertheless not the goal. It is implied we can get somewhere new and exciting with these vastly larger timeseries. My question is: where?
Re: Time Series, the new shiny?
#78Earlier quoted context omitted.
Have you ever seen Time suddenly stop? It doesn't. It just keeps on ticking. (okay, the theory of relativity may have other things to say about how those ticks propagate through space, however: we're talking about computers and user interfaces here..)
> Have you ever seen Time suddenly stop? It doesn't. How would you know, since your awareness is embedded in time?
What difference does it make since my awareness of the computer user interface is also such?
Re: Time Series, the new shiny?
#79Earlier quoted context omitted.
I’m not sure what you mean by "not distributed". With kdb+ you have a lot of flexibility in how to setup the database. You can organize the data to be stored in a distributed fashion (across multiple devices, multiple servers), you can setup query load balancers to distribute work-loads, and you can replicate to multiple servers/devices. You don’t have to use K, you code in q, which most people find far easier to rea…
>I’m not sure what you mean by "not distributed". With kdb+ you have a lot of flexibility in how to setup the database. You can organize the data to be stored in a distributed fashion (across multiple devices, multiple servers. So I was under impression (feel free to correct me) that kdb horizontal scaling was something akin to Oracle RAC. I.e. horizontal in the name only. I.e. the data is only ever available from on…
Having worked with both Oracle and KDB, the thing to understand is that KDB (or K or Q) is a fully fledged language... you can do whatever you like really. If you want something distributed you just design something yourself and build it. It's not like Oracle where it is all opaque and buried in query plans and the like.
Re: Time Series, the new shiny?
#80Earlier quoted context omitted.
So the file format is a lot better than CSV files, but in principle it's basically just a bunch of files. Maybe a better analogy would have been a big folder of feather/hdf5/etc files. (Incidentally, I'm a big fan of the folder/s3 bucket/etc full of CSV/binary files and use it whenever possible.) I agree - it's absolutely better to use than that, but it's a lot closer to that model than to the Riak model of querying…
kdb+/q/k are used for IOT applications [1], not just fin tec. After all, it is all time series data. The benchmarks given in a response above by srpeck [2], shows spark/shark to be 230 times slower than a k4 query, and using 50GB or RAM vs. 0.2GB RAM for k4. If RiakTS is relying on spark/shark as the in-memory database engine, it is already at a big disadvantage compared to k in terms of speed, and all the RAM that i…
RiakTS does not rely on any external data storage (other than our fork of Google's leveldb) or processing tool, so Spark's performance is irrelevant.