Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Future of Notebooks: Lessons from JupyterCon

willcrichton.net

Post reply on HN
Print view View original

71–80 of 158 posts

Re: The Future of Notebooks: Lessons from JupyterCon

#71

Post by Tistel »

enriquto wrote: post #68

I love jupyter notebooks, but unfortunately the code editor is unusable to me. I cannot stand the bizarre parenthesis auto-completion, and the autoindentation settings; and there's no easy way to remove them.

I feel the same way. One workaround is that some editors have a plugin. Emacs has a mode called ein that is fantastic. As you poke around getting the snippets to work, you can cut/paste into a different file/buffer. After you start the Jupiter notebook copy the login token, run: ein:notebooklist-login then ein:notebooklist-open and away you go.

Re: The Future of Notebooks: Lessons from JupyterCon

#72

Post by chimprich »

dangirsh wrote: post #42
zmmmmm wrote: post #9

Earlier quoted context omitted.

This is definitely one of my concerns too. Ad hoc code inside these notebooks is almost completely unmanageable from any reasonable software maintenance perspective, and refactoring code out of them is prohibitively difficult as well. I really want something to emerge that combines the best of both worlds of an IDE and notebook development, but there isn't anything close currently.

ob-ipython enables IDE-like editing features within the code cells. It's embedded in a polyglot, git-friendly, literate programming environment called Org-mode. I use it every day and love it. Other goodies: - easily manage multiple kernels (in different languages / machines) in one file - tree-based organization manages complexity better than linear notebooks - no browser in sight (unless you need interactive widget…

Interesting. I use emacs for hacking Python but I've never heard of this tool. Do you actually use this for software development or is it more of a data-science-type exploration tool?

Re: The Future of Notebooks: Lessons from JupyterCon

#73

Post by sheepmullet »

taeric wrote: post #36
aportnoy wrote: post #19

Earlier quoted context omitted.

I use Jupyter for two things: 1. Interactive computation and plotting, run data through a pipeline. 2. Quick prototyping and testing. Once a function/class is ready, it goes into a module, together with some unit tests. Nothing beats Jupyter for that second use case, 90% of the bugs are squashed through interaction and inspection in the notebook cells, not through unit tests.

That some folks can be productive with a tool doesn't say, necessarily, much about the tool. Some could probably make the same two claims for excel. It isn't that the tool should be banned, per se. Just that many practices that have been rather proven in software are much harder to do in this environment. Sounds like what you like is the live coding aspect. Many of the lisp environments of yesteryear would have proba…

> Some could probably make the same two claims for excel

As a tool Excel has provided astronomical real world value.

There are only a small handful of other tools that even come close.

Re: The Future of Notebooks: Lessons from JupyterCon

#74

Post by balaba »

gaius wrote: post #20
ekianjo wrote: post #16

Earlier quoted context omitted.

exactly. I use it for quick data exploration and experimentation but it remains at that level.

People do that in Excel too and the next thing you know a spreadsheet is managing a portfolio or being used as the basis for published science!

everyone knows that excel is a drawing program https://www.thisiscolossal.com/2017/12/tatsuo-horiuchi-excel...

Re: The Future of Notebooks: Lessons from JupyterCon

#75

Post by stevesimmons »

h4b4n3r0 wrote: post #55

Even in its present state, Jupyter is strictly better than plain REPL, especially if you need visualization. It's like Python in general: it was not designed to write 10-100KLOC programs. Python was designed for scripting. Jupyter was designed for interactive data exploration, and to create a record of results which you can view without re-executing the cells. Is it error prone? Yes. But then so is REPL.

> Python was designed for scripting.

It's far more than scripting, and for much bigger programs than 10k LOC... My company's Python codebase is 35m LOC/500k modules, with 25k commits per week and contributions from 2.5k developers per month.

To my mind, Python has the opposite problem. I've been a Python programmer since 2000. Every few years, I think I should devote more time to other languages (first Java, then R, then Haskell, then JS, ...). But the Python ecosystem just keeps getting stronger. My current focus is data science and machine learning. In these domains, there are no good reasons to drop Python.

Re: The Future of Notebooks: Lessons from JupyterCon

#76

Post by jimhefferon »

stevesimmons wrote: post #38

I don't get the people here saying don't use Jupyter notebooks, or they are bad software engineering. So much Python development is trying snippets of code in a REPL as you introspect live objects, then once they're right pasting them into the IDE. All my Jupyter notebooks are like that, where my code starts as cells of a line or two, as I check each output. Then I coalesce them into a function (which avoid the probl…

> I don't get the people here saying don't use Jupyter notebooks, or they are bad software engineering.

I have a python window open on my desk right now. But the model of programming here contrasts sharply, in my mind, with the one discussed yesterday, from HtDP https://news.ycombinator.com/item?id=17826959 .

I absolutely use the REPL, for instance to determine where to put what arguments into a library function. But I have also in the past had it happen that I found a routine of mine that missed an edge case and there were a good number of projects that subsequently used that routine, that now could be wrong. Some of those could have led to a publication, so there could in that sense be a permanance to that error. Oops.

Those of us who have gotten bit, or who teach and routinely see students develop habits we know will lead to them getting bit, worry about systems that are not developed so much as pasted together.

Re: The Future of Notebooks: Lessons from JupyterCon

#77

Post by dnbgfher »

aportnoy wrote: post #28
dnbgfher wrote: post #24

Earlier quoted context omitted.

Maybe I can provide a bit of perspective on this, as I have lots of conflicting feelings about Jupyter. When I'm doing some bit of data wrangling or just exploratory work with data I quite like it - at least at first. As you pointed out, it's really easy to extremely quickly iterate on things and start to get an idea of what's in the data, what techniques work, and which don't. It's great. Until it isn't. I'm probabl…

> Which cells do I need to rerun together? Which cells should I not run again. What the heck is actually in all these variables right now anyway, because I don't remember which order I've run (and rerun) all the cells in. And what was that one approach or parameter that really worked well that one time? I don't remember. Right. Get that all the time. My solution is to try to condense the useful code built up across d…

> My solution is to try to condense the useful code built up across different cells into a reusable block, most often a function. Do that while the 'state' is still fresh in your mind and you remember the order of execution.

Right, that's the sort of thing I was talking about in terms of "using it right." It makes sense and is really a good idea, it just starts eating into the advantages you get from using Jupyter in the first place.

I'm still trying to figure out how best to integrate it in my workflow. So far I've mostly settled on using it to get some rough ideas about the data and libraries I'm using, and then just reworking from scratch in a more stable setting. As needed I'll go back and try things out there, possibly in a new notebook entirely and just do my best to copy/paste the setup over. In getting it working the setup stuff at least gets consolidated. It's... well awful but eh, I haven't really found a better way.

Re: The Future of Notebooks: Lessons from JupyterCon

#78

Post by dnbgfher »

moultano wrote: post #56
dnbgfher wrote: post #24

Earlier quoted context omitted.

Maybe I can provide a bit of perspective on this, as I have lots of conflicting feelings about Jupyter. When I'm doing some bit of data wrangling or just exploratory work with data I quite like it - at least at first. As you pointed out, it's really easy to extremely quickly iterate on things and start to get an idea of what's in the data, what techniques work, and which don't. It's great. Until it isn't. I'm probabl…

I think notebooks could be hugely improved by saving a snapshot of state after each cell, and having the linear order of cells only move forward in time. Accidental variable reuse (especially from a cell that is in the "future" or maybe no longer exists) is a huge source of bugs.

From other comments, it sounds like something called Cocalc might be of interest to you. I haven't even looked at it yet but it sounds like it does at least some of those things.

Re: The Future of Notebooks: Lessons from JupyterCon

#79

Post by stevesimmons »

Ixio wrote: post #66
stevesimmons wrote: post #38

I don't get the people here saying don't use Jupyter notebooks, or they are bad software engineering. So much Python development is trying snippets of code in a REPL as you introspect live objects, then once they're right pasting them into the IDE. All my Jupyter notebooks are like that, where my code starts as cells of a line or two, as I check each output. Then I coalesce them into a function (which avoid the probl…

> The Jupyter notebooks are version controlled like my .py modules. That's interesting, could you give us some more details ? Last time I tried to put a Jupyter notebook under git it was as mess. Do you use another tool than git or have they made tools to help with version control ? Or is it just your workflow that helps, like emptying all cell results before saving ?

Our version control uses a monorepo, with non-branching workflows, development against the head version, and frequent commits/pushes to production.

For Jupyter notebooks, cell results get emptied before saving, unless declared public. That is to ensure data confidentiality, not really to help version control.

Of course, standard diff gets confused with ipynbs. You need a tool like nbdime (notebook diff and merge).

Re: The Future of Notebooks: Lessons from JupyterCon

#80

Post by jdkram »

Ixio wrote: post #66
stevesimmons wrote: post #38

I don't get the people here saying don't use Jupyter notebooks, or they are bad software engineering. So much Python development is trying snippets of code in a REPL as you introspect live objects, then once they're right pasting them into the IDE. All my Jupyter notebooks are like that, where my code starts as cells of a line or two, as I check each output. Then I coalesce them into a function (which avoid the probl…

> The Jupyter notebooks are version controlled like my .py modules. That's interesting, could you give us some more details ? Last time I tried to put a Jupyter notebook under git it was as mess. Do you use another tool than git or have they made tools to help with version control ? Or is it just your workflow that helps, like emptying all cell results before saving ?

Not OP, but I can recommend the handy https://github.com/kynan/nbstripout which acts as a git filter which makes version control ignore cell outputs.

With that approach, though notebooks are clean they're still fairly poor for easily evaluating diffs between versions. If code review / diffs are more important than preserving the notebook, then you could use a post save hook to convert notebook input to a .py file and output to .html:

https://towardsdatascience.com/version-control-for-jupyter-n...

Post reply on HN