Interesting timing: for the last two years I have worked for a research group headed by Sten Linnarsson at the Karolinska Institute[0]. I was specifically hired to build a data browser for a new file format for storing the ever-growing datasets[1][2][3]. The viewer is an SPA specialised in exploring the data on the fly, doing as much as possible client side while minimising the amount of data being transferred, and staying as data-agnostic as possible.
Linnarsson's group just pre-published a paper cataloguing all cell types in the mouse brain, classifying them based on gene expression[4][5]. The whole reason that I was hired was as an "experiment" to see if there was a way to make the enormous amount of data behind it more accessible for quick explorations than raw dumps of data. The viewer uses a lot of recent (as well as slightly-less-recent-but-underused) browser technologies.
Instead of downloading the full data set (which is typically around 28k genes by N cells, where N is in the tens to hundreds of thousands), only the general metadata plus requested genes are downloaded in the form of compressed JSON arrays containing raw numbers or strings. The viewer converts them to Typed Arrays (yes, even with string arrays) and then renders nearly everything on the fly client-side. This also makes it possible to interactively tweak view settings[6]. Because the viewer makes almost no assupmtions of what the data represents, we recently re-used the scatterplot view to display individual cells in a tissue section[7].
Furthermore, this data is stored off-line through IndexedDB, so repeat viewings of the same dataset or specific genes within it does not require re-downloading the (meta)data. This minimises data transfer even further, and makes the whole thing a lot snappier (not to mention cheaper to host, which may matter if you're a small research group). The only reason it isn't completely offline-first is that using service workers is giving me weird interactions with react-router. Being the lone developer I have to prioritise other, more pressing bugs.
In the end however, the viewer is merely a complement to the full catalogue, which is set up with a DocuWiki[8]. No flashy bells and whistles there, but it works. For example, one can look up specific marker genes. it just uses a plugin to create a sortable table, which is established, stable technology that pretty much comes with the DocuWiki[9][10]. The taxonomy tree is a simple static SVG above it. Since the expression data is known client-side to generate the table dynamically, we only need a tiny bit of JavaScript to turn that into an expression heatmap underneath the taxonomy tree. Simple and very effective, and it probably even works in IE8, if not further back. Meanwhile, I got myself into an incredibly complicated mess writing a scatterplotter with low-level sprite rendering and blitting and hand-crafted memoisation to minimise redraws[11].
Personally, I think there isn't enough praise for the pragmatic DocuWiki approach. My contract ends next week. I intend to keep contributing to the viewer, working out the (way too many) rough edges and small bugs that remain, but it won't be full-time. I hope someone will be able to maintain and develop this further. I think the DocuWiki has a better chance of still being on-line and working ten years from now.
[0] http://linnarssonlab.org/
[1] http://loompy.org/
[2] https://github.com/linnarsson-lab/loom-viewer
[3] http://loom.linnarssonlab.org/
[4] https://twitter.com/slinnarsson/status/981919808726892545
[5] https://www.biorxiv.org/content/early/2018/04/05/294918
[6] https://imgur.com/f6GpMZ1
[7] http://loom.linnarssonlab.org/dataset/cells/osmFISH/osmFISH_..., https://i.imgur.com/a7Mjyuu.png
[8] http://mousebrain.org/doku.php?id=start
[9] http://mousebrain.org/doku.php?id=genes:aw551984
[10] http://mousebrain.org/doku.php?id=genes:actb
[11] https://github.com/linnarsson-lab/loom-viewer/blob/master/cl...
