Live data from Hacker News

Datashader: turns even the largest data into images, accurately

datashader.org

41–50 of 73 posts

Re: Datashader: turns even the largest data into images, accurately

#41

Is there anything similar for network graphs?

Yep -- at https://github.com/graphistry/pygraphistry , we started by making millions of nodes/edges interactive. If you use notebooks, can signup on our site and get going. The trick is we connect GPUs in the browser to GPUs in the cloud, and encapsulate it enough that you can stick to writing standard SQL/pandas/etc. We've been curious about server-side static tile rendering for larger graphs, but has been on the ba…

Geographic maps aren't the primary use for datashader; those are just easy examples that people can appreciate without a lot of explanation. In practice we use it for any large datasets that we don't want to subsample before visualizing them.

Re: Datashader: turns even the largest data into images, accurately

#42
This actually gave me an interesting idea regarding bitcoin passphrase mnemonics.

Instead of text we could use the same algorithm to generate images.

So you could have an index of images and generate them. I'm actually wondering if you could use nouns and verbs to maybe make stories if you could mutate the nouns reliably.

Like 'bird flying' vs 'bird sleeping' ...

This could help to remember long passphrases visually which people seem to be better at.

Re: Datashader: turns even the largest data into images, accurately

#43
post #41

Earlier quoted context omitted.

Yep -- at https://github.com/graphistry/pygraphistry , we started by making millions of nodes/edges interactive. If you use notebooks, can signup on our site and get going. The trick is we connect GPUs in the browser to GPUs in the cloud, and encapsulate it enough that you can stick to writing standard SQL/pandas/etc. We've been curious about server-side static tile rendering for larger graphs, but has been on the ba…

Geographic maps aren't the primary use for datashader; those are just easy examples that people can appreciate without a lot of explanation. In practice we use it for any large datasets that we don't want to subsample before visualizing them.

Yes, definitely more capable. I've seen them primarily used in scatterplots (x, y, maybe z) + maps. Curious where else you're seeing the 80/20 breakdown if not there...

Re: Datashader: turns even the largest data into images, accurately

#44
post #25
post #24

I've used datashader for plotting NGS (Next Generation Sequencing) enrichments. At the time I had to hack together the ability to use the polygon select tool on the data, but it worked and blew my mind. Very elegant solution to a difficult problem (overplotting).

Do you have any examples of this you could point to online? I am looking at different visualization tools for various NGS-based analyses currently.

I do. I remember posting this to the mailing list. I don't have an example calculating enrichments though. We simply group by read and divide using the frequencies and then plot one enrichment vs another. This way we can see how one sequence enriches between conditions. There's more to it than that but this will produce a plot similar to the one in the attachment in the thread linked below.

https://groups.google.com/a/continuum.io/forum/m/#!msg/bokeh...

Edit: here's a link to the plot I refer to: https://10826817673355204906.googlegroups.com/attach/e6e58ad...

Re: Datashader: turns even the largest data into images, accurately

#45
post #33

Earlier quoted context omitted.

> but doing so on large data accurately What do you mean by 'large' or 'accurate' ? Where would accuracy be lost in any approach?

This paper on our original ideas of "Abstract Rendering" talks about the kinds of accuracy problems that plague the visualization of large datasets: https://www.semanticscholar.org/paper/Abstract-rendering%3A-... We renamed from Abstract Rendering to Datashader for affordances of human cognition. This is a great paper from Gordon Kindlmann and Carlos Scheidegger talk about how to gauge the accuracy of a visualization…

There are a lot of red flags in the abstract of that paper alone.

> Rendering techniques are currently a major limiter since they tend to be builtaround central processing with all of the geometric data present.

This is completely untrue - OpenGL and virtually all real time rendering is done using z-buffer techniques that were originally used because they don't need all the geometry present. These techniques date back to the 70s and were some of the first hidden surface rendering algorithms.

> This paper presents Abstract Rendering (AR), a technique for eliminating the cen-tralization requirement while preserving some forms of interactivity.

Interactivity might be novel here so that is what should really be focused on, if anything. I don't think coining a new term and acronym that don't seem to relate to what is happening is a going to be a good choice to communicate the techniques.

> AR is based on the observation that pixelsare fundamentally bins, and that rendering is essentially a binning process on a lattice of bins.

This observation was made in the early 80s and has been the backbone of renderman renderers for almost 40 years. Renderman calls them 'buckets'.

> This approach enables: (1) rendering onlarge datasets without requiring large amounts of working memory,

Renderman originally rendered film resolution images with high resolution textures with only 10MB of memory.

> (3) a direct means of distributing the rendering task across processes,

Giving different threads their own buckets is standard for any non-toy renderer. Distributing buckets across multiple computers is part of many toolsets.

> high-performanceinteraction techniques on large datasets

This is the only part that has a chance of being novel, but paper only shows basic accumulation of density for adjacency matrices. The visualization are timed in the multiple seconds but look extremely simple, and for some reason are rendered 'out-of-core' on a computer with 144GB of memory even though it seems very unclear that these images couldn't be made with z-buffer rendering in opengl.

> This is a great paper from Gordon Kindlmann and Carlos Scheidegger talk about how to gauge the accuracy of a visualization

It looks like that paper is about the transformations of visualizations for higher dimensional data, not rendering accuracy, so these two things are being conflated even though they are completely separate concepts.

Re: Datashader: turns even the largest data into images, accurately

#46
post #36

Earlier quoted context omitted.

I suspect marketing Datashader as an "accumulation buffer" wouldn't have the same effect on its target audience (data visualisation developers) as presenting it simply as a way to "Turn data into images". I'm also curious (as a fledgling graphics programmer) - what leads you to believe that Datashader uses an accumulation buffer internally? I would think that they use some magic to draw all the points in a single dra…

Datashader's approach is a bit different from an accumulation buffer, though similar in principle. It's not 3D rendering, and has no need for a z ordering; instead it's essentially 2D histogramming. For points, it simply takes each point, calculates which pixel it would land in, and aggregates per pixel, without ever storing all the data points per pixel. The key benefit over something like SciPy's histogram2d functi…

> For points, it simply takes each point, calculates which pixel it would land in, and aggregates per pixel, without ever storing all the data points per pixel.

That is literally what opengl does. If you mean a histogram per pixel in depth, that's literally voxels in perspective space.

If there are usability benefits here, that's great, but everything seems to be centered around there being new rendering techniques here, when not only are they not new, they're completely trivial, with solidified names and formalized math.

Re: Datashader: turns even the largest data into images, accurately

#47
post #20

Looks like a great project. Contrary to other comments, rendering != visualization. This project seems to have paid attention to lots of the seemingly little but critical details of this type of visualization that are a pain to handle yourself (anti-aliasing of multi-scale data, terrain shading, large- and out-of-core visualization). Any one of these topics can bring a visualization project to a screeching halt, or m…

> anti-aliasing of multi-scale data, terrain shading, large- and out-of-core visualization

Webgl will basically do all of that for you, including the out of core if you can stream the data in.

Re: Datashader: turns even the largest data into images, accurately

#48
post #47
post #20

Looks like a great project. Contrary to other comments, rendering != visualization. This project seems to have paid attention to lots of the seemingly little but critical details of this type of visualization that are a pain to handle yourself (anti-aliasing of multi-scale data, terrain shading, large- and out-of-core visualization). Any one of these topics can bring a visualization project to a screeching halt, or m…

> anti-aliasing of multi-scale data, terrain shading, large- and out-of-core visualization Webgl will basically do all of that for you, including the out of core if you can stream the data in.

Somehow I feel like WebGL in the browser isn't going to just handle it for things like the main image on the project page, where the compressed dataset is > 1 GB already.

Re: Datashader: turns even the largest data into images, accurately

#49
post #37
post #2

This is basically visualization 101. 'Turning data into images' isn't exactly a new concept. Sometimes I think the next generation of programmers has gotten too good at thinking up fancy names for reinventing the wheel and hasn't spent enough time looking at what has been done in the past 50 years so they can build off of it. Also there are links to 'visualizing big data' and 'visualizing billions of points'. These t…

You're missing the point of this project. It's not about the feasibility of throwing a billion points at a pile of software, to get an image. I can do that with a simple Python script. It's about doing so to create a meaningful and accurate data visualization, and not just a picture of, say, shiny spheres or a scene from Avatar. I actually have a background in 3D computer graphics, and it's precisely because of my de…

> Datashader eats a million points for breakfast.

So does opengl on a decade old computer.

Re: Datashader: turns even the largest data into images, accurately

#50
post #48
post #47

Earlier quoted context omitted.

> anti-aliasing of multi-scale data, terrain shading, large- and out-of-core visualization Webgl will basically do all of that for you, including the out of core if you can stream the data in.

Somehow I feel like WebGL in the browser isn't going to just handle it for things like the main image on the project page, where the compressed dataset is > 1 GB already.

Why wouldn't it? There isn't anything special that needs to be done, you can stream in whatever you want and render it to the existing buffer. You can even have it anti aliased practically for free.
Post reply on HN