If Jupyter is not your cup of tea you might like Cauldron, the unnotebook - http://www.unnotebook.com
Interview with the author in May, 2017 - https://www.linkedin.com/pulse/why-cauldron-might-right-data...
The Future of Notebooks: Lessons from JupyterCon
41–50 of 158 posts
Re: The Future of Notebooks: Lessons from JupyterCon
#42I understand why they became popular, but as a software engineer considering how they work, I am just full of disappointment. we're going to spend the next ten years re-inventing every single software engineering best practice for jupyter's weirdo environment.
This is definitely one of my concerns too. Ad hoc code inside these notebooks is almost completely unmanageable from any reasonable software maintenance perspective, and refactoring code out of them is prohibitively difficult as well. I really want something to emerge that combines the best of both worlds of an IDE and notebook development, but there isn't anything close currently.
ob-ipython enables IDE-like editing features within the code cells. It's embedded in a polyglot, git-friendly, literate programming environment called Org-mode. I use it every day and love it. Other goodies:
- easily manage multiple kernels (in different languages / machines) in one file
- tree-based organization manages complexity better than linear notebooks
- no browser in sight (unless you need interactive widgets)
- highly exportable, including to ipynb via ox-ipynb
Downsides:
- Emacs-only
- small user base / limited docs
- can't easily import from ipynb
- async cell execution support is early-stage
See Scimax ipython for examples
Re: The Future of Notebooks: Lessons from JupyterCon
#43As many already said, I think it is a great tool for prototyping and data exploration but when it comes to moving code to production, for me it makes very little sense to use it. Netflix said that if the job breaks they can enter the notebook with the data and see what is wrong. For me it feels like they did development with 0 safe guards and if it breaks they check why. Instead of logging problems and dealing with e…
I'm not sure exactly what Netflix means by "if the job breaks", but I've used jupyter notebook in production to do ML before.
If you consider the notebook as a way to augment your logs with plots, it might make a bit more sense.
Running a jupyter notebook is a nice way to generate a HTML report for a job. For a typical ML pipeline, you first plot some stats about the input data, then train some model, plot some training loss, a confusion matrix, some example of predictions, etc...
If some job gives a strange result (maybe that's what they mean by break), having the notebook rendered as an HTML page with all the plot is a very effective way to do a first round of diagnostics. You can also start the notebook with the same parameters and 'run' through your report, which is a nice way to do interactive debugging.
Also in this case, the notebook itself was quite small in terms of lines of code. All the functions were implemented in modules, so it's really like the notebook is your 20 lines 'main' function. So you need some discipline among your team.
Re: The Future of Notebooks: Lessons from JupyterCon
#44When you consider that it is impossible to reuse notebooks in each other... Or unit test them... Or that it is not especially easy to version control them in any sort of branch/merge workflow... Jupyter Notebooks are much closer to Excel spreadsheets than they are to what most people would consider actual programs.
Spreadsheets have been insanely popular and useful. If the similarity is at all deep, there's opportunity in figuring out version control, unit testing, reusable packages etc, even if they seem hopelessly impossible on the surface.
Re: The Future of Notebooks: Lessons from JupyterCon
#45When you consider that it is impossible to reuse notebooks in each other... Or unit test them... Or that it is not especially easy to version control them in any sort of branch/merge workflow... Jupyter Notebooks are much closer to Excel spreadsheets than they are to what most people would consider actual programs.
>impossible to reuse notebooks in each other... Or unit test them...
It's quite easy to unit test notebooks. Whenever you write a function you want to test, write some unit tests (or even just asserts) in the same cell it's defined, then any time the cell is run to define the function, the tests are run too. Likely you'll be doing some manual tests when you write a block of code cum function, and it's not hard to copy the input crafted from those manual tests to a unit test.
Re: The Future of Notebooks: Lessons from JupyterCon
#46I despise notebooks being used in production for the reasons given early in the article: you can execute code in any order so authors usually end up with spaghetti and long pages without clear flows. I would rather scientists use and learn the tools that have been developed over decades on collaborative code writing using version control. It helps integrate their solutions too.
This is a code style problem not a notebook problem: the same people would probably write spaghetti without clear control flow if writing a Python batch script instead of a Notebook.
Re: The Future of Notebooks: Lessons from JupyterCon
#47It is a disaster that more than half of the research papers do not provide data and the code, required either to reproduce research (if it is a theoretical one) or reproduce results analysis. Wider adoption of JupyterLab (and alike) will help to solve this problem. Using PDF and paper for distributing research results feels like XIX century these days.
Re: The Future of Notebooks: Lessons from JupyterCon
#48Earlier quoted context omitted.
Jupyter is also really useful for figuring out how you want to do something before you copy it to your IDE. Every time I work with a new API or library I try it out in jupyter first.
What are the benefits of this vs just an integrated repl?
There is value when e.g. you need more than simply textual output (think plots, etc.).
Re: The Future of Notebooks: Lessons from JupyterCon
#49Most comments here express disbelief and disappointment in Jupyter from the software engineering point of view. What exactly is wrong with it? I use Jupyter daily and find no other Python environment more productive, be it scripts or IPython or IDEs. Granted, I work in scientific computing and use Python for data wrangling and stats. I find immense value in interactivity and iteration speed.
- hard to use linters
- hard to use formatters like yapf
- can't use the shortcuts from my editor
- easy to screw up the order of execution
- harder to review, generate nice diff, fix merge conflicts
Re: The Future of Notebooks: Lessons from JupyterCon
#50I understand why they became popular, but as a software engineer considering how they work, I am just full of disappointment. we're going to spend the next ten years re-inventing every single software engineering best practice for jupyter's weirdo environment.
In the mid-90s I was a Mech Eng undergrad using a program called MathCAD which provided a "notebook" interactive computation mixed with text environment by running as a Word plugin. In 2018 I use Jupyter and it's not clear where the progress has been. There are too many compromises trying to make it work in a web browser. For what Jupyter is for I find that RStudio or Spyder are infinitely superior. I do interactive…
I too prefer RStudio/RMarkdown. Now we have the Reticulate package that allows running Python inside RMarkdown, for my purposes it's way better than Jupyter. It's just plain text so my editor and VCS and everything plays nicely.
But perhaps the biggest strength is that you go through Pandoc, so you can just click a button and get the output as a Word file for sending to someone who's not a software dev, or you can get it as a LaTeX source file and extend it into a proper formal document like a journal paper, or you can do it as HTML where embedding Bokeh scripts and other interactive things Just Works.
I've even made presentations with it going through the reveal.js framework, where you can put an interactive plot on one of your slides, and show people live "what happens when we change this parameter". That's still semi-witchcraft in 2018, but it's going to become a common thing (hopefully).