Will work great until you try to render 50,000 points and your browser crashes because it's build on d3.
D3 is pretty ok with large datasets but I understand your point. What Shiny does to get around this is natively 'evaluate' the plots on the backend, creating a rasterized PNG file. A similar approach could work for Pyxley (using matplotlib or Seaborn to render the plot, and then sending that image file to the front end) but I fear with so much development time spent on d3 support such an approach would not be nativel…
I don't think that's the case for D3 charts, because that would kill the interactivity that is so great about D3. I use RCharts to inject D3 into my Shiny applications, and have ran into into performance issues with just a couple hundred data points. I think this is because all heavy lifting is done by the client (browser), not the server.