Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Future of Notebooks: Lessons from JupyterCon

willcrichton.net

Post reply on HN
Print view View original

151–158 of 158 posts

Re: The Future of Notebooks: Lessons from JupyterCon

#151

Post by nojvek »

yodon wrote: post #117
nojvek wrote: post #94

Earlier quoted context omitted.

Vscode has decent Jupyter support with its python plugin. What I really missed my jupyter notebooks was a world class editor with keyboard shortcuts and code completion. So I went the other route, running jupyter notebooks in vscode. I made a really simple python cli module that would convert from jupyter notebook to python files with runnable cells in vscode, and vice versa. https://github.com/nojvek/vscode-ipynb-py…

Wow! This is insanely cool! You should package this up as a VS Code plugin so more people can use it. At a minimum, please post a license.txt file in your repo so folks can actually use it (MIT or similar is pretty easy and common for something like this, without a license.txt file it's not clear whether others can actually use it and many places that means they can't).

I usually mark all my repos as MIT but will explicitly add a LICENSE.txt. Thanks

Re: The Future of Notebooks: Lessons from JupyterCon

#152

Post by PurpleRamen »

LittlePeter wrote: post #146
h4b4n3r0 wrote: post #70

Earlier quoted context omitted.

I don’t provide citations for such obvious things. If you’re interested, you can find the origin story as written by Guido himself in Python FAQ using any of the available search engines.

I was not able to find it. What I was able to find is this: "Python is an interpreted, interactive, object-oriented programming language." https://docs.python.org/3/faq/general.html#what-is-python Yes some people do write scripts in Python. Does that make Python a scripting language? Maybe in your definition of scripting language.

A bit below your section you will find https://docs.python.org/3/faq/general.html#why-was-python-cr...

Notable part: [..]We needed a better way to do system administration[..]

Re: The Future of Notebooks: Lessons from JupyterCon

#153

Post by spappal »

stevesimmons wrote: post #38

I don't get the people here saying don't use Jupyter notebooks, or they are bad software engineering. So much Python development is trying snippets of code in a REPL as you introspect live objects, then once they're right pasting them into the IDE. All my Jupyter notebooks are like that, where my code starts as cells of a line or two, as I check each output. Then I coalesce them into a function (which avoid the probl…

> Over time, I expect editors like VSCode to edit Jupyter notebooks natively.

PyCharm has support for Jupyter notebooks [1], but for now it's not very good [2]. Two weeks ago Jetbrains released this official comment [2]:

Hi, we are aware of the problems our Jupyter support has and we aren't going to fix them, because the current support is going to retire. Currently we are working on another support of Jupyter notebook (even 2!), which hopefully will be better and much more usable. We plan to make it ready for 2018.3 PyCharm release, so stay tuned.

My note: The current PyCharm is version 2.2.

[1] https://www.jetbrains.com/help/pycharm/using-ipython-noteboo...

[2] https://intellij-support.jetbrains.com/hc/en-us/community/po...

Re: The Future of Notebooks: Lessons from JupyterCon

#154

Post by chuckcode »

pyrce wrote: post #111

As the speaker for the scheduling notebook talk that got referenced here, I can probably give a few insights about how we're using notebooks. A lot of the risks and concerns brought up here were talked about in the session. The slides ended up on https://conferences.oreilly.com/jupyter/jup-ny/public/schedu... (and hopefully the talks themselves will get posted soon). In particular I referenced how we treat and emphas…

I really like notebooks as a way to share an analysis or data visualization. For me the biggest benefits are: - Integration. So nice to have access to the compute, data and libraries all in one place. There is a surprising amount of hassle moving data, setting up paths and libraries, etc. Notebooks almost act like for a container for data analysis rather than service. - Sharable and reproducible. My coworkers can reproduce and explore some new idea with almost no effort, especially important when their strengths are more ML or stats than devops. - Literate programming. It is really nice to have plots, markdown and code all it one place when deliverable is a report or analysis rather than code.

Even with these benefits I do think the criticisms about software development are right on point. Notebooks are a step backward in terms of software engineering environment with none of the modern tooling, version control, testing frameworks etc. I think that the folks who dismiss notebooks as a platform though are missing some important benefits that have long been absent in current editors. Larger companies like Facebook and Google are already facing the reality that devops is a pain point even for sophisticated software engineers and have developed remote code editors like cider and nucleotide to try and enable bringing code development to the data and compute rather than doing it from the laptop. R has been working on a long time on integrating analysis results and code in a reproducible package with sweave/knitr and python now has pweave in a similar fashion.

I hear all the issues with jupyter and I'm not particularly married to current form of notebooks. I do think though that the features of remote development, data visualization and support for literate/report programming and sharing ode are first class features that I'll continue to want in the future.

Re: The Future of Notebooks: Lessons from JupyterCon

#155

Post by WorldMaker »

candiodari wrote: post #148
ivansavz wrote: post #54

Earlier quoted context omitted.

Yeah, I hear you about the jupyter-is-for-draft-code. I thought that's how everyone used Jupyter for software dev. I guess this is the MATLAB-style software dev---try something in the REPL, and once you figure out the parameters interactively copy-paste that line/paragraph into your program. As soon as I have one piece of functionality working as a function, I move the code to the "main" file for the project and impo…

I think there is a big misunderstanding here. There's 2 kinds of people working with these things, and their needs and goals are very different. You have scientists. They are trying new things, and while reproducibility is nice, most of the time it doesn't matter. Tests don't matter, to put it mildly, because we're at a stage where it isn't just "not a product", it's not worth to become product, and they are aware of…

> Understanding everything about everything you work with, like an academic would try to do, is not practical for a software engineer.

> So notebooks are not for software developers.

You admit software developers are not a homogeneous group, but these are fascinating blanket statements none-the-less. "Notebook" systems originated from software developers, if you allow the origin to be Knuth's ideas of Literate Programming [1] as the forefather of Notebook languages.

It seems an interesting full circle of sorts that Knuth heavily promoted the idea of Notebooks as a superior programming environment in a time when the languages weren't particularly suited to the task (compiled languages that needed very delicate surgery to convert from human readable order to compilable code) and tools/environments that were effectively suboptimal to the task (text editors with no GUI/WYSIWYG abilities; macro-based compilers instead of live interpreters; etc). Now we finally have the tools to make that work, and at least in a few places developers are starting to make use of it (Netflix's Papermill in the linked article is an interesting example).

Admittedly, you might argue that Knuth was perhaps more on the academic side of the fence than software developers today, but Knuth very succinctly argued that all software development would perhaps be better if we stopped relegating the human/natural language narrative of the project to embedded comments in a programming language, and instead embedded the programming language into our human/natural language narratives.

[1] https://en.wikipedia.org/wiki/Literate_programming

Re: The Future of Notebooks: Lessons from JupyterCon

#156

Post by a_136_chiffa »

aportnoy wrote: post #11

Most comments here express disbelief and disappointment in Jupyter from the software engineering point of view. What exactly is wrong with it? I use Jupyter daily and find no other Python environment more productive, be it scripts or IPython or IDEs. Granted, I work in scientific computing and use Python for data wrangling and stats. I find immense value in interactivity and iteration speed.

I am a computational biologist with a heavy emphasis on the data analysis. I did try Jupyter a couple of years ago and here are my concerns with it, compared to my usual flow (Pycharm + pure python + pickle to store results of heavy processing).

1) Extracting functions is harder 2) Your git commits become completely borked 3) Opening some data-heavy notebooks is neigh impossible once they have been shut down 4) Import of other modules you have in local is pretty non-trivial. 5) Refactoring is pretty hard 6) Sphinx for autodoc extraction is pretty much out of the picture 7) Non-deterministic re-runs - depending on the cell execution order you can get very different results. That's an issue when you are coming back to your code a couple of months later and try to figure what you did to get there.

There are likely work-arounds for most of these problems, but the issue is that with my standard workflow they are non-issues to start with.

In my experience, Jupyter is pretty good if you rely only on existing libraries that you are piecing together, but once you need to do more involved development work, you are screwed.

Re: The Future of Notebooks: Lessons from JupyterCon

#157

Post by sipCom19 »

jakelarkin wrote: post #3

I understand why they became popular, but as a software engineer considering how they work, I am just full of disappointment. we're going to spend the next ten years re-inventing every single software engineering best practice for jupyter's weirdo environment.

Testing, style checking, code coverage...

Re: The Future of Notebooks: Lessons from JupyterCon

#158

Post by chimprich »

dangirsh wrote: post #142
chimprich wrote: post #72

Earlier quoted context omitted.

Interesting. I use emacs for hacking Python but I've never heard of this tool. Do you actually use this for software development or is it more of a data-science-type exploration tool?

I use Org for software development whenever I can, which is currently everyday. Usually, new code starts in cells with some Org-managed context (e.g. a Jupyter kernel in a remote container with some DB/service access). This is done using the :session code cell keyword, which works per subtree . Managing remote sessions like this generally keeps me away from terminals. Surrounding the cell are various mini-dashboards…

Thanks.
Post reply on HN