Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Datashader: turns even the largest data into images, accurately

datashader.org

Post reply on HN
Print view View original

11–20 of 73 posts

Re: Datashader: turns even the largest data into images, accurately

#11

Post by IanCal »

BubRoss wrote: post #2

This is basically visualization 101. 'Turning data into images' isn't exactly a new concept. Sometimes I think the next generation of programmers has gotten too good at thinking up fancy names for reinventing the wheel and hasn't spent enough time looking at what has been done in the past 50 years so they can build off of it. Also there are links to 'visualizing big data' and 'visualizing billions of points'. These t…

> thinking up fancy names for reinventing the wheel

They're not claiming to have reinvented the wheel, they're just explaining what it is.

> 'Turning data into images' isn't exactly a new concept.

No, but doing so on large data accurately (the last word is important that you cut off) is not something I know I can easily achieve in a different python library faster. I'd like to know if I could.

Re: Datashader: turns even the largest data into images, accurately

#12

Post by johnmarinelli »

Looks like a really cool project. One thing that I would be interested in see would be using Datashader as a dynamic visualisation library - for example, generative art projects. Probably not the main interest of data visualisation practitioners but hey, if you've got a sweet pipeline to render all those points, why not?

Re: Datashader: turns even the largest data into images, accurately

#13

Post by IanCal »

Datashader is a great project. Very fast, very easy to use. You can throw a lot of data at it in a notebook and get back a zoomable interactive pane.

Here's a 2016 talk on it: https://www.youtube.com/watch?v=fB3cUrwxMVY

There's likely a lot of improvements since then, but that should help show some of the core parts and explain why it's a useful tool.

Re: Datashader: turns even the largest data into images, accurately

#14

Post by detaro »

BubRoss wrote: post #9
jameskilton wrote: post #4

Earlier quoted context omitted.

And this is how you push people out of our field. How about instead of starting with an insult (I can't believe you didn't already know this) you instead congratulate them on putting together a full working library, with pretty, easy-to-grasp examples then offer up some research links that they could use to further refine and improve their system. It's our job to teach people, you can't expect everyone to suddenly kn…

My problem isn't that isn't novel or new, it is that it is presented as novel. Visualization is great (and the visualization here looks good), but saying things like 'datashader: turn data into images' when that is literally what rendering is, is a nonsense way to approach a valid topic. Saying 'visualize big data and billions of points' when the buzzwords are just there to sugar coat an accumulation buffer, gets int…

I can't see a single reference to this being somehow something you couldn't do before in the linked page. It describes what it does, it doesn't make claims of superiority over other approaches, ...

Re: Datashader: turns even the largest data into images, accurately

#16

Post by candu »

BubRoss wrote: post #2

This is basically visualization 101. 'Turning data into images' isn't exactly a new concept. Sometimes I think the next generation of programmers has gotten too good at thinking up fancy names for reinventing the wheel and hasn't spent enough time looking at what has been done in the past 50 years so they can build off of it. Also there are links to 'visualizing big data' and 'visualizing billions of points'. These t…

Ah, the good old "MapReduce is basically functional programming 101" trope, usually resulting from a fundamental misunderstanding of the problem the framework / tool in question solves.

Re: Datashader: turns even the largest data into images, accurately

#17

Post by johnmarinelli »

BubRoss wrote: post #9
jameskilton wrote: post #4

Earlier quoted context omitted.

And this is how you push people out of our field. How about instead of starting with an insult (I can't believe you didn't already know this) you instead congratulate them on putting together a full working library, with pretty, easy-to-grasp examples then offer up some research links that they could use to further refine and improve their system. It's our job to teach people, you can't expect everyone to suddenly kn…

My problem isn't that isn't novel or new, it is that it is presented as novel. Visualization is great (and the visualization here looks good), but saying things like 'datashader: turn data into images' when that is literally what rendering is, is a nonsense way to approach a valid topic. Saying 'visualize big data and billions of points' when the buzzwords are just there to sugar coat an accumulation buffer, gets int…

I suspect marketing Datashader as an "accumulation buffer" wouldn't have the same effect on its target audience (data visualisation developers) as presenting it simply as a way to "Turn data into images".

I'm also curious (as a fledgling graphics programmer) - what leads you to believe that Datashader uses an accumulation buffer internally? I would think that they use some magic to draw all the points in a single draw call using instanced rendering, but I am very naive :)

Re: Datashader: turns even the largest data into images, accurately

#18

Post by p49k »

BubRoss wrote: post #9
jameskilton wrote: post #4

Earlier quoted context omitted.

And this is how you push people out of our field. How about instead of starting with an insult (I can't believe you didn't already know this) you instead congratulate them on putting together a full working library, with pretty, easy-to-grasp examples then offer up some research links that they could use to further refine and improve their system. It's our job to teach people, you can't expect everyone to suddenly kn…

My problem isn't that isn't novel or new, it is that it is presented as novel. Visualization is great (and the visualization here looks good), but saying things like 'datashader: turn data into images' when that is literally what rendering is, is a nonsense way to approach a valid topic. Saying 'visualize big data and billions of points' when the buzzwords are just there to sugar coat an accumulation buffer, gets int…

“Turn data into images” is a much better phrase than “rendering.” Anyone who is not intentionally being pedantic for dramatic effect would understand the purpose of the product based on reading that phrase, whereas “Datashader: a renderer” is less clear and could refer to products with an entirely different scope/purpose.

Their product page is well-written and accurate. It sounds like you want them to purposely describe their product as something that is inferior to what it actually is.

Re: Datashader: turns even the largest data into images, accurately

#19

Post by mikepurvis »

BubRoss wrote: post #9
jameskilton wrote: post #4

Earlier quoted context omitted.

And this is how you push people out of our field. How about instead of starting with an insult (I can't believe you didn't already know this) you instead congratulate them on putting together a full working library, with pretty, easy-to-grasp examples then offer up some research links that they could use to further refine and improve their system. It's our job to teach people, you can't expect everyone to suddenly kn…

My problem isn't that isn't novel or new, it is that it is presented as novel. Visualization is great (and the visualization here looks good), but saying things like 'datashader: turn data into images' when that is literally what rendering is, is a nonsense way to approach a valid topic. Saying 'visualize big data and billions of points' when the buzzwords are just there to sugar coat an accumulation buffer, gets int…

It is novel in the sense that it combines the interactivity of a D3/Bokeh/whatever JS based visualization (typically limited to a few thousand points) with the massive data display capability of offline rendering.

Re: Datashader: turns even the largest data into images, accurately

#20

Post by mhalle »

Looks like a great project. Contrary to other comments, rendering != visualization. This project seems to have paid attention to lots of the seemingly little but critical details of this type of visualization that are a pain to handle yourself (anti-aliasing of multi-scale data, terrain shading, large- and out-of-core visualization).

Any one of these topics can bring a visualization project to a screeching halt, or make the results look misleading or bad.

Even better that they built a tool that works with existing libraries, rather than replacing them. Good work!

Post reply on HN