What license is this project using? The repo has a license but besides the provisions listed there I don't see any standard license.
Datashader: turns even the largest data into images, accurately
51–60 of 73 posts
Re: Datashader: turns even the largest data into images, accurately
#52https://anaconda.org/jbednar/gerrymandering/notebook I wish more people were outraged at this kind of election tampering. Great visualizations though! Zoom in on some of those tight masses of black outlines. The shapes are ridiculous. Maryland 3rd? Come on.
[deleted]
Re: Datashader: turns even the largest data into images, accurately
#53Looks like a great project. Contrary to other comments, rendering != visualization. This project seems to have paid attention to lots of the seemingly little but critical details of this type of visualization that are a pain to handle yourself (anti-aliasing of multi-scale data, terrain shading, large- and out-of-core visualization). Any one of these topics can bring a visualization project to a screeching halt, or m…
> anti-aliasing of multi-scale data, terrain shading, large- and out-of-core visualization Webgl will basically do all of that for you, including the out of core if you can stream the data in.
No, it can't. WebGL is at a completely different level of the stack than Datashader. WebGL is even lower level than what most people use for 3D graphics (hence threejs and babylonjs).
Data visualization is at a higher semantic level than rendering; ideally you don't want to deal with pixels and polygons. D3, for instance, binds graphical primitives (usually in SVG) with data representations but requires more programming to do actual data visualization (and that's why a bunch of software layers on top of D3). Bokeh deals with still higher level primitives closer to the data set level (plotting and charts).
And Datashader carves out a niche where there's too much data to have a 1:1 ratio of data element to graphical primitive on the screen. It does that by rasterizing, but then also handling the hard part of mapping backwards from image to data for selection and interactivity (I hope that's right; I got it from watching the 2016 video).
Anyone who has had to do this stuff for a living knows it is hard to do right, and that good modular tools are always welcome.
I don't see how this repeated "it's just like X" line of responses is benefiting the discussion. Datashader is not just like WebGL or basic low-level rendering, any more than D3 is just SVG or web apps are just TCP connections. Completely different levels of abstraction with lots of value add in between (and hard work, I'm sure).
Re: Datashader: turns even the largest data into images, accurately
#54What license is this project using? The repo has a license but besides the provisions listed there I don't see any standard license.
The repo has a standard 3-clause BSD license.
Re: Datashader: turns even the largest data into images, accurately
#55Looks like a great project. Contrary to other comments, rendering != visualization. This project seems to have paid attention to lots of the seemingly little but critical details of this type of visualization that are a pain to handle yourself (anti-aliasing of multi-scale data, terrain shading, large- and out-of-core visualization). Any one of these topics can bring a visualization project to a screeching halt, or m…
> anti-aliasing of multi-scale data, terrain shading, large- and out-of-core visualization Webgl will basically do all of that for you, including the out of core if you can stream the data in.
Datashader is server-side rendering, and thus not in any way comparable to WebGL in its usage. With Datashader, only the final rendered/rasterized image-like object is sent to the client, which lets it handle arbitrarily large datasets (anything your remote servers can process). With WebGL the dataset is sent to the browser for rendering, which has some advantages but is a very different process than what Datashader does.
Re: Datashader: turns even the largest data into images, accurately
#56Earlier quoted context omitted.
Somehow I feel like WebGL in the browser isn't going to just handle it for things like the main image on the project page, where the compressed dataset is > 1 GB already.
Why wouldn't it? There isn't anything special that needs to be done, you can stream in whatever you want and render it to the existing buffer. You can even have it anti aliased practically for free.
> Why wouldn't it? There isn't anything special that needs to be done, you can stream in whatever you want and render it to the existing buffer.
1GB to the browser for rendering... Sure, it's doable. So is eating 1KG of wings for lunch. Doable, but very far from being a good idea.
Re: Datashader: turns even the largest data into images, accurately
#57> Turns even the largest data into images, accurately
The first image, the image of the USA, seems really mis-representative to me. LA and NYC should be way way way more bright in relation to everything else than the entire area east of the Mississippi.
At least to my eyes that map makes it look like parts of Denver, Kansas City, Salt Lake City, Atlanta, and the San Joaquin Valley are just as dense as Manhattan.
Atlanta's population density 630 per square mile
Manhattan's population density 70826 per square mile
It seems like an accurate data image would have Atlanta's brightness 1/100th of Manhattan's. Basically it looks like they saturated out at around 250 people do anything over 250 people is the same brightness.
Re: Datashader: turns even the largest data into images, accurately
#58> Turns even the largest data into images, accurately The first image, the image of the USA, seems really mis-representative to me. LA and NYC should be way way way more bright in relation to everything else than the entire area east of the Mississippi. At least to my eyes that map makes it look like parts of Denver, Kansas City, Salt Lake City, Atlanta, and the San Joaquin Valley are just as dense as Manhattan. Atla…
By default, Datashader accurately conveys the shape of the distribution in a way that the human visual system can process. If you want a linear representation, you can do that easily; see the first plot in http://datashader.org/topics/census.html , but you'll quickly see that the resulting plot completely fails to show that there are any patterns anywhere besides the top few population hotspots, which is highly unrepresentative of the actual patterns in this data. There is no saturation here; what it's doing in the homepage image is basically a rank-order encoding, where the top brightness value is indeed shared by several high-population pixes, the next brightness value is shared by the next batch of populations, etc. Given only 256 possible values, there has to be some grouping, but it's not saturating.
Re: Datashader: turns even the largest data into images, accurately
#59Earlier quoted context omitted.
> anti-aliasing of multi-scale data, terrain shading, large- and out-of-core visualization Webgl will basically do all of that for you, including the out of core if you can stream the data in.
No, it can't. WebGL is at a completely different level of the stack than Datashader. WebGL is even lower level than what most people use for 3D graphics (hence threejs and babylonjs). Data visualization is at a higher semantic level than rendering; ideally you don't want to deal with pixels and polygons. D3, for instance, binds graphical primitives (usually in SVG) with data representations but requires more programm…
I didn't say that webgl would do everything this can do, I said it would do the things I copied from the post I replied to.
Again, my issue is that they seem to put a lot of focus on this having some sort of sophisticated new rendering when it seems to be marketing of trivial techniques. People seem to like what this library does, but they didn't invent new rendering algorithms and their buzzwords and clever names just show a lack of awareness of what they are doing in the rendering department.
Re: Datashader: turns even the largest data into images, accurately
#60Is there anything similar for network graphs?
Perhaps Gephi? It was used for those Graphcore neural network visualizations (millions of edges and nodes) and they look stunning. [1] https://gephi.org/features/ [2] https://www.graphcore.ai/posts/what-does-machine-learning-lo...
I've used both gephi and cytoscape, i have about 2 million nodes with about 3 million edges, it's hard to visualise at once. however i've reduced the graphs to only sub networks with more than 5 connections, the remainder is about 600k, which is still painfully slow to visualise