Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Datashader: turns even the largest data into images, accurately

datashader.org

Post reply on HN
Print view View original

21–30 of 73 posts

Re: Datashader: turns even the largest data into images, accurately

#21

Post by CyberDildonics »

IanCal wrote: post #11
BubRoss wrote: post #2

This is basically visualization 101. 'Turning data into images' isn't exactly a new concept. Sometimes I think the next generation of programmers has gotten too good at thinking up fancy names for reinventing the wheel and hasn't spent enough time looking at what has been done in the past 50 years so they can build off of it. Also there are links to 'visualizing big data' and 'visualizing billions of points'. These t…

> thinking up fancy names for reinventing the wheel They're not claiming to have reinvented the wheel, they're just explaining what it is. > 'Turning data into images' isn't exactly a new concept. No, but doing so on large data accurately (the last word is important that you cut off) is not something I know I can easily achieve in a different python library faster. I'd like to know if I could.

> but doing so on large data accurately

What do you mean by 'large' or 'accurate' ? Where would accuracy be lost in any approach?

Re: Datashader: turns even the largest data into images, accurately

#22

Post by TeMPOraL »

candu wrote: post #16
BubRoss wrote: post #2

This is basically visualization 101. 'Turning data into images' isn't exactly a new concept. Sometimes I think the next generation of programmers has gotten too good at thinking up fancy names for reinventing the wheel and hasn't spent enough time looking at what has been done in the past 50 years so they can build off of it. Also there are links to 'visualizing big data' and 'visualizing billions of points'. These t…

Ah, the good old "MapReduce is basically functional programming 101" trope, usually resulting from a fundamental misunderstanding of the problem the framework / tool in question solves.

Well, in that particular case, Google has brought this on themselves by naming their sort-of-a-product after a common FP idiom, and then hyping the hell out of it.

Re: Datashader: turns even the largest data into images, accurately

#23

Post by mhalle »

johnmarinelli wrote: post #17
BubRoss wrote: post #9

Earlier quoted context omitted.

My problem isn't that isn't novel or new, it is that it is presented as novel. Visualization is great (and the visualization here looks good), but saying things like 'datashader: turn data into images' when that is literally what rendering is, is a nonsense way to approach a valid topic. Saying 'visualize big data and billions of points' when the buzzwords are just there to sugar coat an accumulation buffer, gets int…

I suspect marketing Datashader as an "accumulation buffer" wouldn't have the same effect on its target audience (data visualisation developers) as presenting it simply as a way to "Turn data into images". I'm also curious (as a fledgling graphics programmer) - what leads you to believe that Datashader uses an accumulation buffer internally? I would think that they use some magic to draw all the points in a single dra…

If you watch the video, one of the creators explains this point, which makes datashader different from, say, Bokeh. (Pipeline explained around 7:20 in the video)

https://www.youtube.com/watch?v=6m3CFbKmK_c

D3 and Bokeh and other web-based visualization tools, in general, plot HTML or CSS primitives to the browser. This approach works great for smaller datasets, but doesn't scale to millions/billions.

Datashader aggregates (accumulates) graphical representations of data into images, then provides a way to get those to the browser and work well with the other libraries. That high level description leaves out 95% of the critical practical details of visualization, which the creators of datashader handle.

Re: Datashader: turns even the largest data into images, accurately

#24

Post by itodd »

I've used datashader for plotting NGS (Next Generation Sequencing) enrichments. At the time I had to hack together the ability to use the polygon select tool on the data, but it worked and blew my mind.

Very elegant solution to a difficult problem (overplotting).

Re: Datashader: turns even the largest data into images, accurately

#25

Post by abcc8 »

itodd wrote: post #24

I've used datashader for plotting NGS (Next Generation Sequencing) enrichments. At the time I had to hack together the ability to use the polygon select tool on the data, but it worked and blew my mind. Very elegant solution to a difficult problem (overplotting).

Do you have any examples of this you could point to online? I am looking at different visualization tools for various NGS-based analyses currently.

Re: Datashader: turns even the largest data into images, accurately

#26

Post by jointpdf »

simplyinfinity wrote: post #15

Is there anything similar for network graphs?

Perhaps Gephi? It was used for those Graphcore neural network visualizations (millions of edges and nodes) and they look stunning.

[1] https://gephi.org/features/

[2] https://www.graphcore.ai/posts/what-does-machine-learning-lo...

Re: Datashader: turns even the largest data into images, accurately

#27

Post by maliker »

simplyinfinity wrote: post #15

Is there anything similar for network graphs?

My team has had luck rendering an SVG from the graph and sending it to a browser. It works well for about 10k vertices and edges. Above that scale we use datashader, and we're investigating a potential to move to QGIS. We tried Gephi a few years ago and it had trouble at these scales.

Re: Datashader: turns even the largest data into images, accurately

#28

Post by lmeyerov »

simplyinfinity wrote: post #15

Is there anything similar for network graphs?

Yep -- at https://github.com/graphistry/pygraphistry, we started by making millions of nodes/edges interactive. If you use notebooks, can signup on our site and get going. The trick is we connect GPUs in the browser to GPUs in the cloud, and encapsulate it enough that you can stick to writing standard SQL/pandas/etc.

We've been curious about server-side static tile rendering for larger graphs, but has been on the back-burner. (We already connect to GPUs on the server, so not rocket science.) Currently, we're actively increasing how much can be ingested + computed on, such as for finding influencers, communities & rings, etc. However, visualizing that hasn't been an operational priority for our users. More useful to generate the communities, and then either inspect individual ones, or see how communities stitch together: quickly run out of pixels otherwise due to too many edges. Likewise, we're building connectors to gigascale-petascale graph DBs: titan, janus, aws neptune, tigergraph, spark graphx, etc.

We still are interested, but more for when we start supporting geographic maps: you can see that is the primary use for datashader. Also, because data art is fun :)

Re: Datashader: turns even the largest data into images, accurately

#29

Post by 24gttghh »

https://anaconda.org/jbednar/gerrymandering/notebook

I wish more people were outraged at this kind of election tampering. Great visualizations though! Zoom in on some of those tight masses of black outlines. The shapes are ridiculous. Maryland 3rd? Come on.

Re: Datashader: turns even the largest data into images, accurately

#30

Post by IanCal »

CyberDildonics wrote: post #21
IanCal wrote: post #11

Earlier quoted context omitted.

> thinking up fancy names for reinventing the wheel They're not claiming to have reinvented the wheel, they're just explaining what it is. > 'Turning data into images' isn't exactly a new concept. No, but doing so on large data accurately (the last word is important that you cut off) is not something I know I can easily achieve in a different python library faster. I'd like to know if I could.

> but doing so on large data accurately What do you mean by 'large' or 'accurate' ? Where would accuracy be lost in any approach?

If you want to discuss this properly, can you please finish the sentence?

What library should I use to do this faster in python without subsampling the data for a start?

Post reply on HN