After having looked at what this actually is, I'm a bit disappointed. I was hopping that they wound change the various panels data API so that they would request only the new time slices; ie. instead of querying the last hour of data every minute, query the last minute every minute and reuse the previous 49 data points. But that's not what this is. Instead, you need a specific panel and a specific data source for streaming to work. :-(
Show HN: Grafana Live Streaming Datasource Implementation
11–15 of 15 posts
Re: Show HN: Grafana Live Streaming Datasource Implementation
#12After having looked at what this actually is, I'm a bit disappointed. I was hopping that they wound change the various panels data API so that they would request only the new time slices; ie. instead of querying the last hour of data every minute, query the last minute every minute and reuse the previous 49 data points. But that's not what this is. Instead, you need a specific panel and a specific data source for str…
This datasource doesn't require any custom panels to function.
To your point though, it would be nice if grafana supported incremental time range querying out of the box, since many backends don't support streaming data.
Re: Show HN: Grafana Live Streaming Datasource Implementation
#13After having looked at what this actually is, I'm a bit disappointed. I was hopping that they wound change the various panels data API so that they would request only the new time slices; ie. instead of querying the last hour of data every minute, query the last minute every minute and reuse the previous 49 data points. But that's not what this is. Instead, you need a specific panel and a specific data source for str…
This datasource doesn't require any custom panels to function. To your point though, it would be nice if grafana supported incremental time range querying out of the box, since many backends don't support streaming data.
Something we are looking at.
Problem is complicated when the query applies time series transformations such as moving averages, where requesting only incremental data can be problematic.
Re: Show HN: Grafana Live Streaming Datasource Implementation
#14Earlier quoted context omitted.
Also think about having a dashboard refresh at 5s with 50k datapoints, literally transferring gigabytes of data to every browser looking at that dashboard. If you streamed the data (even at slow rates) it would be a massive traffic savings.
Recently I was looking into Plot.ly Dash ( https://plot.ly/products/dash/ ) for a Python-based real-time updating dashboard. ...until I noticed that it transfers the entire chart and all the datapoints each update. (example: https://dash-gallery.plotly.host/dash-live-model-training/ )
It's funny, we already have a format for transferring graphical data in real time that the web has optimized relentlessly: video!
You'd think someone would just run a daemon local to the data that live-streams a compressed video feed rendering the graph, instead of relentlessly and repeatedly reinventing some notion of lossy-but-useful realtime data transferring technology.
Re: Show HN: Grafana Live Streaming Datasource Implementation
#15Earlier quoted context omitted.
Also think about having a dashboard refresh at 5s with 50k datapoints, literally transferring gigabytes of data to every browser looking at that dashboard. If you streamed the data (even at slow rates) it would be a massive traffic savings.
Recently I was looking into Plot.ly Dash ( https://plot.ly/products/dash/ ) for a Python-based real-time updating dashboard. ...until I noticed that it transfers the entire chart and all the datapoints each update. (example: https://dash-gallery.plotly.host/dash-live-model-training/ )
I'll be honest, before I started working at Plotly I had the same reaction to Dash at first glance, but as I worked with it/on it, I realized that most of the time, this client-server chattiness is not much of a problem from a user perspective, and is a great developer-productivity tradeoff.
More to your point, though, this is something we're trying to address in collaboration with the Dash community, in this PR for example if you want to follow along: https://github.com/plotly/dash-core-components/pull/461