Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Future of Notebooks: Lessons from JupyterCon

willcrichton.net

Post reply on HN
Print view View original

121–130 of 158 posts

Re: The Future of Notebooks: Lessons from JupyterCon

#121

Post by analog31 »

yodon wrote: post #116
analog31 wrote: post #97

Earlier quoted context omitted.

I also work in scientific computing, and I love Jupyter. I think out-of-order or unknown-order execution is the biggest source of my own mistakes / bugs. If the order in which your cells are executed affects your result, you've got a bug waiting to bite you or someone else. I've been there. And we tend to run cells more than once, while refining our work. This potentially affects reproducibility, which is a tenet of…

> In my revisionist history, software engineering began when programs got too big to see the whole thing on one screen and intuitively understand. Laughing, politely, at that summary of "history." Software engineering started when there were no screens. Code was written on typewriters (called "terminals") and then printed to cards or long rolls of paper (and occasionally using switches to indicate the 1's and 0's you…

Quite agreed, and it's why I deliberately chose the "revisionist" label.

Before screens, there were pages, before pages there were cards... going back to plugboards, etc.

Each of us got in at a certain time point. When I learned programming, we had a workstation that simulated a card punch, let you see the contents of one card at a time on a little vacuum-fluorescent display, and stored the data on an 8" floppy. You handed the floppy to the computer operator, and received a printout on green-bar paper.

The other way I got into "programming" was staring at a nest of wires, reading my output on an oscilloscope, and making corrections with a soldering iron. ;-)

The common theme was that a program gets too large for somebody to look at and understand it, much less for a team to work on it all at once, unless disciplines are adopted that tend to come from software engineering. When I'm mentoring colleagues who are just beginning to get into scientific programming, some of these simple techniques, such as avoiding globals and creating functions, are in fact an improvement.

Re: The Future of Notebooks: Lessons from JupyterCon

#122

Post by gaius »

kgwgk wrote: post #61
gaius wrote: post #6

Earlier quoted context omitted.

In the mid-90s I was a Mech Eng undergrad using a program called MathCAD which provided a "notebook" interactive computation mixed with text environment by running as a Word plugin. In 2018 I use Jupyter and it's not clear where the progress has been. There are too many compromises trying to make it work in a web browser. For what Jupyter is for I find that RStudio or Spyder are infinitely superior. I do interactive…

Unfortunately RStudio also makes compromises to make it work in a web browser. A native tool could be even better.

RStudio is mainly a native tool, the web version is quite inferior. Linux, Windows and OSX native versions

Re: The Future of Notebooks: Lessons from JupyterCon

#123

Post by m_ke »

yodon wrote: post #117
nojvek wrote: post #94

Earlier quoted context omitted.

Vscode has decent Jupyter support with its python plugin. What I really missed my jupyter notebooks was a world class editor with keyboard shortcuts and code completion. So I went the other route, running jupyter notebooks in vscode. I made a really simple python cli module that would convert from jupyter notebook to python files with runnable cells in vscode, and vice versa. https://github.com/nojvek/vscode-ipynb-py…

Wow! This is insanely cool! You should package this up as a VS Code plugin so more people can use it. At a minimum, please post a license.txt file in your repo so folks can actually use it (MIT or similar is pretty easy and common for something like this, without a license.txt file it's not clear whether others can actually use it and many places that means they can't).

You should check this out https://github.com/nteract/hydrogen

Re: The Future of Notebooks: Lessons from JupyterCon

#124

Post by benrbray »

aluren wrote: post #60

It would seem that notebooks were specifically tailored for my use cases (being in academia and doing data analysis), in practice I found that I drift away from notebooks every time I try them and stick to a good old console (ipython or bash) instead. Here are a few things that irked me: -It's browser based. Yuck. This means you can wait upwards of ten seconds for a session to load up every time you want to start one…

For what it's worth, I love the fact that Jupyter is browser-based.

> It's the entire reason it's so easy to format text / math in a Jupyter notebook. There's Markdown + MathJax for basic interactive formatting, no need to compile or anything. If you need anything more sophisticated (like a box around text, etc.) just throw in a and it'll automatically show up in the notebook. > I only need one Chrome window (or more, if I choose) for my entire workspace. I can have tabs for notebooks AND webpages open at the same time, in the same program. I don't need to worry about having dozens of windows open! I also don't need to deal with some developer's idea of a tabbed workspace--Chrome's tab work exactly how I need them to.

Maybe you could get all the same benefits from an Electron app, but people don't quite like those either :).

Re: The Future of Notebooks: Lessons from JupyterCon

#126

Post by danysdragons »

nojvek wrote: post #94
pacala wrote: post #92

Earlier quoted context omitted.

Jupyter code is draft code. Draft code rarely makes into clean code, because it requires extra tedious effort. Most of us are too overworked to have the time or energy to perform it. Alternatively one can write exploratory code as unit tests and run them inside an IDE. When the exploration is done, three quarters of the tedious cleanup effort is already done. There are already unit tests. There are already APIs exerc…

Vscode has decent Jupyter support with its python plugin. What I really missed my jupyter notebooks was a world class editor with keyboard shortcuts and code completion. So I went the other route, running jupyter notebooks in vscode. I made a really simple python cli module that would convert from jupyter notebook to python files with runnable cells in vscode, and vice versa. https://github.com/nojvek/vscode-ipynb-py…

I only recently learned myself that Jupyter notebooks do have code completion, triggered by hitting the tab button - although possibly you were thinking of something more advanced than that. In any case I'll check out your code on github.

Re: The Future of Notebooks: Lessons from JupyterCon

#127

Post by jpeloquin »

analog31 wrote: post #97
aportnoy wrote: post #11

Most comments here express disbelief and disappointment in Jupyter from the software engineering point of view. What exactly is wrong with it? I use Jupyter daily and find no other Python environment more productive, be it scripts or IPython or IDEs. Granted, I work in scientific computing and use Python for data wrangling and stats. I find immense value in interactivity and iteration speed.

I also work in scientific computing, and I love Jupyter. I think out-of-order or unknown-order execution is the biggest source of my own mistakes / bugs. If the order in which your cells are executed affects your result, you've got a bug waiting to bite you or someone else. I've been there. And we tend to run cells more than once, while refining our work. This potentially affects reproducibility, which is a tenet of…

> I think out-of-order or unknown-order execution is the biggest source of my own mistakes / bugs.

To avoid the state problem, I start each cell (or, in org-mode, each src block) with a command to set up the environment for that cell, usually by importing a script named something like setup_abc123.py. Cells are not allowed to reference code or results from other cells except through the filesystem, which is under the watchful eye of git. In this workflow, cells exist only to write plots or tables into the notebook for human consumption.

This workflow, for me, has completely eliminated errors due to hidden state dependencies. Before I adopted it I made a serious state-related error every couple of days, which was intolerable.

As a bonus, using a single script to set up the analysis environment makes it trivial to start a new analysis, even with an entirely different tool chain.

Re: The Future of Notebooks: Lessons from JupyterCon

#128

Post by [deleted] »

danysdragons wrote: post #126
nojvek wrote: post #94

Earlier quoted context omitted.

Vscode has decent Jupyter support with its python plugin. What I really missed my jupyter notebooks was a world class editor with keyboard shortcuts and code completion. So I went the other route, running jupyter notebooks in vscode. I made a really simple python cli module that would convert from jupyter notebook to python files with runnable cells in vscode, and vice versa. https://github.com/nojvek/vscode-ipynb-py…

I only recently learned myself that Jupyter notebooks do have code completion, triggered by hitting the tab button - although possibly you were thinking of something more advanced than that. In any case I'll check out your code on github.

[deleted]

Re: The Future of Notebooks: Lessons from JupyterCon

#129

Post by yodon »

m_ke wrote: post #123
yodon wrote: post #117

Earlier quoted context omitted.

Wow! This is insanely cool! You should package this up as a VS Code plugin so more people can use it. At a minimum, please post a license.txt file in your repo so folks can actually use it (MIT or similar is pretty easy and common for something like this, without a license.txt file it's not clear whether others can actually use it and many places that means they can't).

You should check this out https://github.com/nteract/hydrogen

Hydrogen looks very cool! Am I right that hydrogen and the https://github.com/DonJayamanne/vscodeJupyter plugin that nojvek's plugin is designed to work with in that hydrogen shows the plots inline (like Jupyter does) but hydrogen doesn't support markdown? [edit: the biggest difference is that hydrogen is a stand-alone atom-like editor not a vscode plugin]

Re: The Future of Notebooks: Lessons from JupyterCon

#130

Post by h4b4n3r0 »

stevesimmons wrote: post #75
h4b4n3r0 wrote: post #55

Even in its present state, Jupyter is strictly better than plain REPL, especially if you need visualization. It's like Python in general: it was not designed to write 10-100KLOC programs. Python was designed for scripting. Jupyter was designed for interactive data exploration, and to create a record of results which you can view without re-executing the cells. Is it error prone? Yes. But then so is REPL.

> Python was designed for scripting. It's far more than scripting, and for much bigger programs than 10k LOC... My company's Python codebase is 35m LOC/500k modules, with 25k commits per week and contributions from 2.5k developers per month. To my mind, Python has the opposite problem. I've been a Python programmer since 2000. Every few years, I think I should devote more time to other languages (first Java, then R,…

JS was also designed for scripting
Post reply on HN