The Future of Notebooks: Lessons from JupyterCon • Hacker News Board

Re: The Future of Notebooks: Lessons from JupyterCon

#131

Post by barrkel » Sat, Aug 25, 2018, 7:00 PM UTC

seanmcdirmid wrote: post #39

stevesimmons wrote: post #38

I don't get the people here saying don't use Jupyter notebooks, or they are bad software engineering. So much Python development is trying snippets of code in a REPL as you introspect live objects, then once they're right pasting them into the IDE. All my Jupyter notebooks are like that, where my code starts as cells of a line or two, as I check each output. Then I coalesce them into a function (which avoid the probl…

> And also more code editing to move from IDEs into the Jupyter Jupyter, and all REPLS in general, rely on a concrete top-level execution context to provide useful feeedback. Much of the code written outside of data science doesn’t really have that, which is why you don’t see notebooks being used for software development very often.

Used all the time in Ruby (repl, not notebook). In fact I think anyone doing Ruby dev who doesn't have a pry session open is missing out on a good 30% productivity boost, unless they know their APIs inside out.

Re: The Future of Notebooks: Lessons from JupyterCon

#132

Post by kgwgk » Sat, Aug 25, 2018, 7:23 PM UTC

gaius wrote: post #122

kgwgk wrote: post #61

Earlier quoted context omitted.

Unfortunately RStudio also makes compromises to make it work in a web browser. A native tool could be even better.

RStudio is mainly a native tool, the web version is quite inferior. Linux, Windows and OSX native versions

One has to strecth the term quite a lot to say that RStudio is a "native" GUI. The interface is just a browser window, the only native part is the menu bar.

https://imgur.com/yszRlMk

Re: The Future of Notebooks: Lessons from JupyterCon

#133

Post by yodon » Sat, Aug 25, 2018, 7:27 PM UTC

analog31 wrote: post #121

yodon wrote: post #116

Earlier quoted context omitted.

> In my revisionist history, software engineering began when programs got too big to see the whole thing on one screen and intuitively understand. Laughing, politely, at that summary of "history." Software engineering started when there were no screens. Code was written on typewriters (called "terminals") and then printed to cards or long rolls of paper (and occasionally using switches to indicate the 1's and 0's you…

Quite agreed, and it's why I deliberately chose the "revisionist" label. Before screens, there were pages, before pages there were cards... going back to plugboards, etc. Each of us got in at a certain time point. When I learned programming, we had a workstation that simulated a card punch, let you see the contents of one card at a time on a little vacuum-fluorescent display, and stored the data on an 8" floppy. You…

It sounds like instead of putting the sentence in past tense and calling it a revisionist history it sounds like what you are really doing is making a present tense generalization based on years of historical observation: software engineering begins when the code gets too big to see the whole thing on one screen and intuitively understand (aka Analog31's law)

Re: The Future of Notebooks: Lessons from JupyterCon

#134

Post by WhATiSCaMeLcaSE » Sat, Aug 25, 2018, 8:03 PM UTC

danysdragons wrote: post #126

nojvek wrote: post #94

Earlier quoted context omitted.

Vscode has decent Jupyter support with its python plugin. What I really missed my jupyter notebooks was a world class editor with keyboard shortcuts and code completion. So I went the other route, running jupyter notebooks in vscode. I made a really simple python cli module that would convert from jupyter notebook to python files with runnable cells in vscode, and vice versa. https://github.com/nojvek/vscode-ipynb-py…

I only recently learned myself that Jupyter notebooks do have code completion, triggered by hitting the tab button - although possibly you were thinking of something more advanced than that. In any case I'll check out your code on github.

Actually, if you activate the 'Hinterland' extension, you don't need to hit tab. And there's a ton of more extensions too. Their Github repo has instructions on how to set up the extensions - https://github.com/ipython-contrib/jupyter_contrib_nbextensi...

Re: The Future of Notebooks: Lessons from JupyterCon

#135

Post by canhascodez » Sat, Aug 25, 2018, 8:05 PM UTC

barrkel wrote: post #131

seanmcdirmid wrote: post #39

Earlier quoted context omitted.

> And also more code editing to move from IDEs into the Jupyter Jupyter, and all REPLS in general, rely on a concrete top-level execution context to provide useful feeedback. Much of the code written outside of data science doesn’t really have that, which is why you don’t see notebooks being used for software development very often.

Used all the time in Ruby (repl, not notebook). In fact I think anyone doing Ruby dev who doesn't have a pry session open is missing out on a good 30% productivity boost, unless they know their APIs inside out.

I would agree with this, although an IDE with really good introspection and an integrated debugger can approximate the experience. Lately I've started using spaceneovim, which is hopefully going to help with the back-and-forth workflow a bit, modulo issues of what-mode-am-I-in-again? The terminal is two keystrokes away, and having all of the normal vim commands available in both contexts is something that I think will be more and more useful.

Re: The Future of Notebooks: Lessons from JupyterCon

#136

Post by analog31 » Sat, Aug 25, 2018, 8:49 PM UTC

yodon wrote: post #133

analog31 wrote: post #121

Earlier quoted context omitted.

Quite agreed, and it's why I deliberately chose the "revisionist" label. Before screens, there were pages, before pages there were cards... going back to plugboards, etc. Each of us got in at a certain time point. When I learned programming, we had a workstation that simulated a card punch, let you see the contents of one card at a time on a little vacuum-fluorescent display, and stored the data on an 8" floppy. You…

It sounds like instead of putting the sentence in past tense and calling it a revisionist history it sounds like what you are really doing is making a present tense generalization based on years of historical observation: software engineering begins when the code gets too big to see the whole thing on one screen and intuitively understand (aka Analog31's law)

Yes, that's perfect.

Re: The Future of Notebooks: Lessons from JupyterCon

#137

Post by CJefferson » Sat, Aug 25, 2018, 9:34 PM UTC

gaius wrote: post #4

When you consider that it is impossible to reuse notebooks in each other... Or unit test them... Or that it is not especially easy to version control them in any sort of branch/merge workflow... Jupyter Notebooks are much closer to Excel spreadsheets than they are to what most people would consider actual programs.

Except Excel is in some ways much better designed -- it automatically recalculate everything if you change a cell, and makes it easy to link to other spreadsheets.

Re: The Future of Notebooks: Lessons from JupyterCon

#138

Post by IgniteTheSun » Sat, Aug 25, 2018, 9:40 PM UTC

geebee wrote: post #118

aportnoy wrote: post #11

Most comments here express disbelief and disappointment in Jupyter from the software engineering point of view. What exactly is wrong with it? I use Jupyter daily and find no other Python environment more productive, be it scripts or IPython or IDEs. Granted, I work in scientific computing and use Python for data wrangling and stats. I find immense value in interactivity and iteration speed.

This is a really important comment. I started my career in scientific computing, went over to software/web development for a while, and am back over to scientific computing. I've been involved in teaching efforts lately, mainly for analysts who want to do data wrangling and stats, and I think it is a good environment for some of what they want to do. I never used Jupyter before getting involved in this kind of coding…

For the times that you mention that you currently use Jupyter, how is it better than simply randomly selecting a small subset of data at the beginning and then writing your code in a full featured IDE with debugging functionality (including the ability to add breakpoints and see variable contents at those points, etc.)? The one line at the beginning to randomly subset the large data set is easy to delete or comment out once you have the code working the way you like - to me it seems more effective to do that in a native IDE like Eclipse+PyDev or in a lightweight IDE like Spyder than to use Jupyter, which requires so many compromises.

Re: The Future of Notebooks: Lessons from JupyterCon

#139

Post by bsder » Sat, Aug 25, 2018, 10:49 PM UTC

stevesimmons wrote: post #38

I don't get the people here saying don't use Jupyter notebooks, or they are bad software engineering. So much Python development is trying snippets of code in a REPL as you introspect live objects, then once they're right pasting them into the IDE. All my Jupyter notebooks are like that, where my code starts as cells of a line or two, as I check each output. Then I coalesce them into a function (which avoid the probl…

> I don't get the people here saying don't use Jupyter notebooks, or they are bad software engineering.

How many businesses rely on bodged up Excel spreadsheets to run their business?

This is that, but on steroids.

For better or worse.

Re: The Future of Notebooks: Lessons from JupyterCon

#140

Post by AdamM12 » Sat, Aug 25, 2018, 11:37 PM UTC

vosper wrote: post #53

seanmcdirmid wrote: post #39

Earlier quoted context omitted.

> And also more code editing to move from IDEs into the Jupyter Jupyter, and all REPLS in general, rely on a concrete top-level execution context to provide useful feeedback. Much of the code written outside of data science doesn’t really have that, which is why you don’t see notebooks being used for software development very often.

I frequently use IPython or REPL.it for software development. It's not my main editor, and I don't write large chunks of my code there, but for rapidly prototyping a small subset of some task I find them invaluable. But you are right about notebooks - I don't find them particularly useful, except as a better REPL, and no-one I know uses them for "normal" (non Data Science) software development.

I think my favorite non "data science" use case is web scraping. Ad-hoc nature of the task lends itself to using a notebook. A lot of times I don't want the full suite and boilerplate Scrapy gives you so I write it out in a notebook with Parsel & Requests. Once done export and clean up.