Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Unplugg: An automated Forecasting API for timeseries data

unplu.gg

Post reply on HN
Print view View original

31–35 of 35 posts

Re: Show HN: Unplugg: An automated Forecasting API for timeseries data

#31

Post by keredson »

watty wrote: post #27
keredson wrote: post #12

{ "timestamp": 1458000000, "value": 63.422235 }, dear lord, why? this reminds me of the old "xml binary format" joke: 0 0 1 0 0 1 0 0

I don't get the snark - what's the glaring problem with the format? I work with sensor data at my job and very rarely is it uniformly distributed so we use a similar format.

Because it's horribly inefficient. It's using 49 bytes to encode 8 bytes worth of data. If your data set is a few hundred observations this likely doesn't matter. But most users of timeseries data have millions or billions. (I come from a computational finance background.)

Even if they were wedded to JSON for some reason, they could have just used a list of observations, like:

[1458000000,63.422235],

That would have cut their data costs in half.

Or just use one of the many existing formats for transmitting time series data. It's not a new topic. https://github.com/mobileink/data.frame/wiki/What-is-a-Data-...

Re: Show HN: Unplugg: An automated Forecasting API for timeseries data

#33

Post by watty »

keredson wrote: post #31
watty wrote: post #27

Earlier quoted context omitted.

I don't get the snark - what's the glaring problem with the format? I work with sensor data at my job and very rarely is it uniformly distributed so we use a similar format.

Because it's horribly inefficient. It's using 49 bytes to encode 8 bytes worth of data. If your data set is a few hundred observations this likely doesn't matter. But most users of timeseries data have millions or billions. (I come from a computational finance background.) Even if they were wedded to JSON for some reason, they could have just used a list of observations, like: [1458000000,63.422235], That would have…

This is an API for very small datasets (daily time series data). The goal should be accessibility and readability over saving a few bytes.

I'm not saying it's ideal, I just think the snark is unwarranted considering how common it is. I just checked InfluxDB and they follow a similar model (even more verbose). https://docs.influxdata.com/influxdb/v1.2/guides/querying_da...

Checked a few more and I believe they're the same - Microsoft IoT, Predix (GE), etc.

Re: Show HN: Unplugg: An automated Forecasting API for timeseries data

#34

Post by mr_pink »

ramblenode wrote: post #28
mgontav wrote: post #22

Earlier quoted context omitted.

Well, if we provide confidence intervals it should provide some kind of "safety"... but even so, with forecasting the only assurance you can get is by testing it against known historical values and checking it's accuracy (and even that way, historical results are no guarantee of future performance). What kind of details would you say can be inspected to see if the model is reliable? AR or MA orders, inferred seasonal…

That's a fair answer. Your target audience is interested in prediction rather than inferring parameters. I do think the confidence intervals/prediction intervals should be accessible and should probably be adjustable (e.g. 99%, 95%, 80%).

> I do think the confidence intervals/prediction intervals should be accessible and should probably be adjustable (e.g. 99%, 95%, 80%).

Definitely, this is arguably the most important feature.

Re: Show HN: Unplugg: An automated Forecasting API for timeseries data

#35

Post by keredson »

watty wrote: post #33
keredson wrote: post #31

Earlier quoted context omitted.

Because it's horribly inefficient. It's using 49 bytes to encode 8 bytes worth of data. If your data set is a few hundred observations this likely doesn't matter. But most users of timeseries data have millions or billions. (I come from a computational finance background.) Even if they were wedded to JSON for some reason, they could have just used a list of observations, like: [1458000000,63.422235], That would have…

This is an API for very small datasets (daily time series data). The goal should be accessibility and readability over saving a few bytes. I'm not saying it's ideal, I just think the snark is unwarranted considering how common it is. I just checked InfluxDB and they follow a similar model (even more verbose). https://docs.influxdata.com/influxdb/v1.2/guides/querying_da... Checked a few more and I believe they're the…

that example you give does not follow a similar model. it defines the columns once (not repeated w/ every observation):

                    "columns": [
                        "time",
                        "value"
                    ],
and then the observations as a list of lists:

                    "values": [
                        [
                            "2015-01-29T21:55:43.702900257Z",
                            2
                        ],
                        [
                            "2015-01-29T21:55:43.702900257Z",
                            0.55
                        ],
exactly as i suggested in the "even if they were wedded to JSON for some reason" section of my original explanation.
Post reply on HN