Why Jupyter is data scientists’ computational notebook of choice • Hacker News Board

Re: Why Jupyter is data scientists’ computational notebook of choice

#261

Post by ubernostrum » Wed, Oct 31, 2018, 8:49 AM UTC

gaius wrote: post #259

ubernostrum wrote: post #257

Earlier quoted context omitted.

There is also the entire bend where these are taking off in a way that doesn't make sense. It makes perfect sense. Just not to a lot of HN readers. The average HN reader is approaching this from a perspective of "I am a professional programmer who might occasionally dabble in scientific computing, and therefore I hate this thing because it's not a professional programmer's tool designed by and for professional progra…

They're scientists who increasingly have to do programming as part of their science. And notebooks are a godsend for them 2/3 of scientific research cannot be reproduced by other scientists. But tell us more about why scientists should ignore best practices from other fields.

Show me the reproducible research in programming about the merits of different type systems (murky at best). Or of different approaches to testing. Or software architecture. Or... well, most of the stuff day-to-day working programmers actually do. There are barely even attempts at rigor in most of our practices, let alone the kind of reviewed and reproduced results we demand from the sciences.

Re: Why Jupyter is data scientists’ computational notebook of choice

#262

Post by enriquto » Wed, Oct 31, 2018, 9:50 AM UTC

kortex wrote: post #241

jonnycomputer wrote: post #93

Earlier quoted context omitted.

If you ever have used an R Notebook written in R-Markdown, then its pretty easy to see why Jupyter Notebooks putting everything in JSON is just... infuriatingly wrong-headed. In an R Notebook, I can see my code, I can see my text, everything is exceedingly simple to understand, and I can edit it in any of the fantastic text editors out there (Jupyter's editor is not among them)

The main reason for json, I believe, is that the Jupyter client is separate from the backend. It's actually pretty trivial to run the engine on a beefy box while interacting on a light laptop (on the same subnet). With Jupyter Lab and some fiddling, you can put the server anywhere. It's also trivial to export notebooks to .py files. That said, my goodness do notebooks wreak havoc on git. I hope this in particular get…

> It's also trivial to export notebooks to .py files.

But this is useless if you cannot edit those py files and obtain notebooks from them.

Re: Why Jupyter is data scientists’ computational notebook of choice

#263

Post by mike_ivanov » Wed, Oct 31, 2018, 10:11 AM UTC

wodenokoto wrote: post #157

astral303 wrote: post #117

Earlier quoted context omitted.

RStudio’s Markdown notebooks do not suffer from this and save a separate output file that can be gitignored.

And they pay for this on other accounts: No inline rendering of markdown. Opening an .Rmd file is a lottery to see if rendered graphs and tables still exists. Tables render completely differently in editor, HTML and pdf

> No inline rendering of markdown.

That's incorrect, take a look here -> https://blog.rstudio.com/2016/10/05/r-notebooks

Re: Why Jupyter is data scientists’ computational notebook of choice

#264

Post by kvlr » Wed, Oct 31, 2018, 10:17 AM UTC

thomasfedb wrote: post #73

Jupyter is lovely (and JupyterLab looks delicious ), but the set up required to achieve a reproducible local server with the R kernel and versioned R packages is 100% not. Installing R packages through anaconda is like pulling teeth and the docker images for my Jupyter notebooks push past 6GB and take multiple cups of tea to build. Is there a good solution I'm missing? A good hosted solution perhaps?

I run https://nextjournal.com

We archive full-stack reproducibility by allowing you to install arbitrary software and version these environments using docker. You can reuse these in other articles or pull and use them locally. `xoxo` is a signup code you can you if you want to give it a try.

Re: Why Jupyter is data scientists’ computational notebook of choice

#265

Post by kvlr » Wed, Oct 31, 2018, 10:23 AM UTC

reilly3000 wrote: post #127

Does anybody know of a good hosted solution of JupyterHub? I made a neat notebook that I needed to share with my non-technical team, it was using iPyWidgets to do some interactive modeling, but they each needed to be able to use it independently. It has private data so I couldn't use Binder. I've been following Zepl.com for a long time, but couldn't use them here because Zepplin doesn't support iPyWidgets. Pretty soo…

This isn't exactly JupyterHub but I run https://nextjournal.com which is a fully hosted notebook platform.

Sharing articles with team members and letting them run them is trivial. We automatically version the article, the data and the environment (docker image) and you can remix (fork) other articles. `xoxo` is a signup code you can you if you want to give it a try.

We're not far out (~two weeks) from launching our beta for private research. Here, you'll get your own private data store and docker registry as well as secrets management (stored securely in hashicorps vault).

Re: Why Jupyter is data scientists’ computational notebook of choice

#266

Post by kvlr » Wed, Oct 31, 2018, 10:24 AM UTC

ralmidani wrote: post #92

Out of curiosity, what other options are there?

I run https://nextjournal.com Use `xoxo` as a signup code if you want to try it, thanks!

Re: Why Jupyter is data scientists’ computational notebook of choice

#267

Post by gaius » Wed, Oct 31, 2018, 10:33 AM UTC

red-tea wrote: post #218

zmmmmm wrote: post #204

As with so many things python related (including python itself), I am perplexed by how willing people seem to be to fall in love with solutions that have so many limitations and problems. I find Jupyter just barely usable. I constantly have issues with editing in the cells, diagrams not sizing correctly, cells accidentally displaying huge amounts of data and freezing my browser, complete failure of autocompletion in…

Why should there be something better? Just because you want it? Someone has to make it.

There already are many better things as many commenters have pointed out

Re: Why Jupyter is data scientists’ computational notebook of choice

#268

Post by SiempreViernes » Wed, Oct 31, 2018, 11:21 AM UTC

reilly3000 wrote: post #147

SiempreViernes wrote: post #133

Earlier quoted context omitted.

Do you have some server to host it on? If so, I’d run the notebook on the remote server and just teach them whatever command they need to make a ssh tunnel there. Something like what is described here: https://techtalktone.wordpress.com/2017/03/28/running-jupyte... So they would utter the unknowable incantation and then point their browser at localhost:8000 or whatever and then use their version of the notebook.

I think that would be within reason for most of our users, but we have a couple of Windows users. I'm not particularly keen on telling them to install putty or WSL.

Ah, well they can always borrow their less disadvantaged coworkers computer until they get the sysadmins to install and configure a shortcut on their desktops via active directory I guess.

Re: Why Jupyter is data scientists’ computational notebook of choice

#269

Post by PurpleRamen » Wed, Oct 31, 2018, 11:27 AM UTC

kortex wrote: post #242

taeric wrote: post #220

Earlier quoted context omitted.

That is a complete non-sequitur. Json in no way enables that. Just having a defined format enables that. Emacs org-mode is proof that a simple text format with markup rules is all you really need to support multiple languages in a single file. You lose some of the simplicity of parsing the file, but you gain a ton more.

Json enables you to chuck the notebook to any browser client anywhere on the net. Org-mode is great, but you still have to install emacs.

How is a notebook without proper software to handle it in any way more useful then other structured plaintext-file? Yes, JSON can be prettyprinted in a browser, but what then? It's still a useless mess you can't work with.

Re: Why Jupyter is data scientists’ computational notebook of choice

#270

Post by PurpleRamen » Wed, Oct 31, 2018, 11:35 AM UTC

kortex wrote: post #241

jonnycomputer wrote: post #93

Earlier quoted context omitted.

If you ever have used an R Notebook written in R-Markdown, then its pretty easy to see why Jupyter Notebooks putting everything in JSON is just... infuriatingly wrong-headed. In an R Notebook, I can see my code, I can see my text, everything is exceedingly simple to understand, and I can edit it in any of the fantastic text editors out there (Jupyter's editor is not among them)

The main reason for json, I believe, is that the Jupyter client is separate from the backend. It's actually pretty trivial to run the engine on a beefy box while interacting on a light laptop (on the same subnet). With Jupyter Lab and some fiddling, you can put the server anywhere. It's also trivial to export notebooks to .py files. That said, my goodness do notebooks wreak havoc on git. I hope this in particular get…

Having a proper server available is even more reason to use a proper fileformat. The client doesn't care what the server handles and the server doesn't need to send raw datastructures directly from the storage.

Actually, fixing the fileformat-mess should be very simple. Just change the file-load/save-functions. Use a folder-structure with every cell being a seperate file. Or switch to XML. Or make a generic interface and allow to save in whatever the people want. Saving notebooks in Mongodb or some SQL-Database seems like a good goal for dedicated services.