71–80 of 86 posts
I am a huge fan of Observable and I think it has massive potential beyond visualizations. To me it's the holy grail of a development environment, there is no separation between development and output. There is no toolchain separating source and binary, infact, the binary is partially recomputed on changes. On top, it is forkable for others to build upon. It support comments, its reactive. When I explain it to people…
Earlier quoted context omitted.
I agree that Observable is fantastic, but also wish it was more open. I'm building something similar called Starboard Notebook[0] that has a different set of trade-offs. It ends up being something in between Jupyter and Observable: * One of the goals is to build Jupyter how it would have been if it was designed for the web (only). * It's open source [1], plays nice with git (the format is plaintext), and supports loc…
Playing with this a bit. It appears I can't use `pip` with `pyodide` and have to use `micropip` with: micropip.install('torch') Which fails with a message: Couldn't find a pure Python 3 wheel for 'torch' Would you agree that this is a substitute for "pencil and paper" to express ideas around data?
Python works for stuff that was written for it purposefully. Some python libraries work great: numpy, matplotlib, pandas. But many others are not supported directly and can be installed through micropip, but that's quite confusing! These are the issues with Pyodide currently:
* All python code that is executed is synchronous, which means it can not make requests (or call sleep). You can actually make a request using pyodide.open_url('path'), but that makes a synchronous request which isn't really a good idea for anything but small files. I believe asynchronous Python is possible, recent versions of emscripten support it, but it needs someone to put the pieces together (which is not easy!)
* Some packages are huge without being split up. Scipy is actually the only one that's really problematic, I believe it's around 80MB? It should be possible to split it up (into scipy.interpolate, scipy.stats, etc)
* Micropip is asynchronous, so you get a promise when you use micropip.install('my-package'), but you can't "await" it.
* Loading Python initially freezes the browser for a second or two.. not a great user experience.
* Libraries which are not pure python currently need to be manually made compatible with patches - including torch.
Python in the browser (not just in Starboard) needs more love. This is powered by Pyodide[0] which has been making steady progress, but the project is without corporate backing since Mozilla's change of direction. Perhaps Observable can allocate some of their funding towards supporting Python in their notebooks too through this project? Also consider this a call to action for other contributors who want to see Python in the browser become a reality :)
I'm a big fan of Observable, having used it to prototype and learn a number of different visualizations. Note that you don't just have to use D3, but can use other visualization libraries as well (i.e. Vega-Lite, Highcharts). I also want to shout out Mike Bostock, one of the company founders (and creator of D3). I emailed him randomly to ask for some help with a d3 package and he replied the next morning. Busy creato…
He responded to me too, almost a decade ago when I was first getting into the field and looking for mentors. He had some great tips too (basically which was to not do what I was doing privately emailing devs asking for mentorship and instead get involved with the public open communities on via GitHub, Stackoverflow, sharing demos, etc). Obvious advice in hindsight, but took me a while to internalize that.
Earlier quoted context omitted.
We are building a data vis tool ( https://github.com/tracyhenry/kyrix ) to solve this exact problem many web vis tools have. The idea is not surprising: using database indexes and a backend to send the frontend necessary data to render on demand . DB and the backend are containerized so the installation, data loading and authoring are all one command. For creators, we offer D3 and JSON-based declarative primitives th…
Vega-Lite and Falcon author here. Many people ask for scalable visualizations and I think declarative visualization approaches are really promising since systems can transparently optimize data movement and rendering. I have done some more experiments to scale Vega such as https://github.com/vega/scalable-vega but I think what we need is a system that automatically offloads heavy computations to a backend such as htt…
I checked out the Falcon documentation on Github and currently don't have a great understanding of (a) what it would be like to "write in Falcon" and (b) what it's intended use case is and how it differs from existing libraries.
Do you mind clarifying?
Earlier quoted context omitted.
That makes sense! Thank you! Some more questions: 1) Assuming the bulk of folks use jupyter notebooks for analysis, why not integrate with jupyter or use d3.js in jupyter directly? I assume there would be a learning curve for data manipulation on js where analysts/scientists would prefer python 2) Assuming that this isn't even meant for the jupyter audience, who is it meant for within a company?
I mean this is my question as well: how are they making money? As of right now Observable seems like a mix of use cases like Medium-like explanation articles, vis debugging/authoring and interactive journalism that don't seem to generate much money (ofc I can be wrong). I totally agree with you that the DS community is much much larger than the vis community. With the new funding round I think they should start doing…
Given the marketing on the page, it also looks like it's targeted at business use cases (i.e. a bunch of data scientists who want to visualize something in d3 but also want to collaborate).
Earlier quoted context omitted.
Tableau offers templated visualizations. You generally do not need to program to create stuff. Observable is a notebook environment where you can write JS (mostly) to create and tweak highly customized visualizations. It's similar to Jupyter on a very high level (they all have reactive cells for example), but for JS and visualizations.
That makes sense! Thank you! Some more questions: 1) Assuming the bulk of folks use jupyter notebooks for analysis, why not integrate with jupyter or use d3.js in jupyter directly? I assume there would be a learning curve for data manipulation on js where analysts/scientists would prefer python 2) Assuming that this isn't even meant for the jupyter audience, who is it meant for within a company?
Their current target for paid customers seems to be infoviz folks at news outlets (the CTO and co-founder was at NYTimes' data viz team for a number of years): https://observablehq.com/teams
That doesn't seem like a huge market, but perhaps they're starting with this audience segment and will be branching into more DS-type features (i.e. supporting a Python kernel, trying to be a more feature-rich Jupyter).
An issue of visualization on the web I don’t know how to solve is how to load large data.
We are building a data vis tool ( https://github.com/tracyhenry/kyrix ) to solve this exact problem many web vis tools have. The idea is not surprising: using database indexes and a backend to send the frontend necessary data to render on demand . DB and the backend are containerized so the installation, data loading and authoring are all one command. For creators, we offer D3 and JSON-based declarative primitives th…
An interesting thing to me here is the layering of DSLs. E.g., SQL enables user-defined functions like filters, that push down to multi-GPU columnar analytics with the rest of the pipeline... and GPU arrow dataframes for zero copy / streaming to combine it all together. People are posting in this thread about 1M rows, but this stuff is built for 1B+. The DSLs means both analysts and devs work at high levels, and underneath, supercomputing.
Fun historical note wrt JS vs Python for GPU: both have different strengths/weaknesses.. but are basically fine-enough long-term, with tweaking. We started w/ proving out JS GPU dataframes on OpenCL to be more open + viz friendly, and after Nvidia liked using our platform, their RAPIDS.ai team spun up to bring the idea in a more corporate controlled & funded way to Python. That's where the community resources are, so we jumped on board, and every month is now quite a trip. GPU SQL, GPU streaming, etc :) JS does inlining and async better than Python, while Python has the data ecosystem, so I've been eagerly anticipating JS folks stepping up where we had to leave off.
It's exciting to see it all come together -- imo, still early days for what's possible!
Earlier quoted context omitted.
We are building a data vis tool ( https://github.com/tracyhenry/kyrix ) to solve this exact problem many web vis tools have. The idea is not surprising: using database indexes and a backend to send the frontend necessary data to render on demand . DB and the backend are containerized so the installation, data loading and authoring are all one command. For creators, we offer D3 and JSON-based declarative primitives th…
This feels close to a lot of our thinking and why some of those pieces were originally written :) We (the Graphistry team) wrote the original Arrow JS implementation to help us bridge GPU visual analytics components in the browser with real-time GPU clusters in the data center, and our backend is DSLs like dataframes to make it easier to do that. An interesting thing to me here is the layering of DSLs. E.g., SQL enab…
Btw, if you referred to the 1M-row example I posted - we actually can do much larger than that. A recording of a visualization with 1B reddit comments is here: https://youtu.be/ccES97ni_vI Behind the scenes we do indexing with Citus, which is a distributed version of PostgreSQL. Our cloud budget can only afford hosting a small demo 24/7 so that's why. Also, because OP talked about loading data into the web vis tools - 1M can already break many web tools out there.
You can tell our target use case is different than yours, and than OmniSci's. It's great to see solutions being developed in a one-size-doesn't-fit-all world.