91–100 of 158 posts
For anything you want to have graph(e.g. matlab), notebook shall serve you well.
I don't get the people here saying don't use Jupyter notebooks, or they are bad software engineering. So much Python development is trying snippets of code in a REPL as you introspect live objects, then once they're right pasting them into the IDE. All my Jupyter notebooks are like that, where my code starts as cells of a line or two, as I check each output. Then I coalesce them into a function (which avoid the probl…
Alternatively one can write exploratory code as unit tests and run them inside an IDE. When the exploration is done, three quarters of the tedious cleanup effort is already done. There are already unit tests. There are already APIs exercised by the unit tests. The code is in much better shape.
The missing two bits:
* No charting support. To the best of my knowledge, neither PyCharm nor VSCode have an output console with HTML support.
* Caching. Some operations take a lot of time. At the extreme, fast.ai style development, where some cells train a model for hours/days. Re-running such code from scratch on every test run is impractical. An opportunity for a library with explicit save/load/cache/checkpoint support.
I despise notebooks being used in production for the reasons given early in the article: you can execute code in any order so authors usually end up with spaghetti and long pages without clear flows. I would rather scientists use and learn the tools that have been developed over decades on collaborative code writing using version control. It helps integrate their solutions too.
This is a code style problem not a notebook problem: the same people would probably write spaghetti without clear control flow if writing a Python batch script instead of a Notebook.
I don't get the people here saying don't use Jupyter notebooks, or they are bad software engineering. So much Python development is trying snippets of code in a REPL as you introspect live objects, then once they're right pasting them into the IDE. All my Jupyter notebooks are like that, where my code starts as cells of a line or two, as I check each output. Then I coalesce them into a function (which avoid the probl…
Jupyter code is draft code. Draft code rarely makes into clean code, because it requires extra tedious effort. Most of us are too overworked to have the time or energy to perform it. Alternatively one can write exploratory code as unit tests and run them inside an IDE. When the exploration is done, three quarters of the tedious cleanup effort is already done. There are already unit tests. There are already APIs exerc…
I made a really simple python cli module that would convert from jupyter notebook to python files with runnable cells in vscode, and vice versa.
https://github.com/nojvek/vscode-ipynb-py-converter/blob/mas...
Really upped my game while doing the udacity courses.
Vscode does indeed have a html output console.
When you consider that it is impossible to reuse notebooks in each other... Or unit test them... Or that it is not especially easy to version control them in any sort of branch/merge workflow... Jupyter Notebooks are much closer to Excel spreadsheets than they are to what most people would consider actual programs.
>impossible to reuse notebooks in each other... Or unit test them... It's quite easy to unit test notebooks. Whenever you write a function you want to test, write some unit tests (or even just asserts) in the same cell it's defined, then any time the cell is run to define the function, the tests are run too. Likely you'll be doing some manual tests when you write a block of code cum function, and it's not hard to cop…
As many people will do this as presently write tests for their Excel macros or formulas.
Earlier quoted context omitted.
Jupyter notebooks are for data science, mostly because visualization is required. Something you do once, report it and it is done. Itsn't make sense to use Jupyter for other stuff. It doesn't make much sense to use them for training big models of deep learning because there are better tools for that.
What tools can you recommend for machine learning?
Most comments here express disbelief and disappointment in Jupyter from the software engineering point of view. What exactly is wrong with it? I use Jupyter daily and find no other Python environment more productive, be it scripts or IPython or IDEs. Granted, I work in scientific computing and use Python for data wrangling and stats. I find immense value in interactivity and iteration speed.
I think out-of-order or unknown-order execution is the biggest source of my own mistakes / bugs. If the order in which your cells are executed affects your result, you've got a bug waiting to bite you or someone else. I've been there. And we tend to run cells more than once, while refining our work.
This potentially affects reproducibility, which is a tenet of scientific computing. So it's important in our world.
I personally think there might be a couple of ways to deal with this issue. One is to put handcuffs on Jupyter and inadvertently make it more difficult or cumbersome to use. But this might be valuable if the size of your project and importance of your result warrants it.
Another way is for us to recognize that at some point we have to grow up and learn some techniques from the programmers. In my revisionist history, software engineering began when programs got too big to see the whole thing on one screen and intuitively understand.
One of my little habits is that I do a "restart kernel and run all cells" when I'm ready to take a break or close out at the end of the day. If if fails, the causes of the problem tend to be superficial, and nothing to lose sleep over.
Another habit is to take cells that work and are nice, and enclose their innards in a function so that their impact on the global variable space is minimized. This can be done during a sleepy time in the afternoon when you're not really in flow anyway.
"jupyter is the new bash" has to be the most underwhelming hyperbole of the decade
"jupyter is the new bash" has to be the most underwhelming hyperbole of the decade
Are you kidding? The shell is a massively important part of modern computers, and making it more expressive would be a huge win. You might be familiar with TermKit[0] which also tried this. [0]: https://github.com/unconed/TermKit
I love jupyter notebooks, but unfortunately the code editor is unusable to me. I cannot stand the bizarre parenthesis auto-completion, and the autoindentation settings; and there's no easy way to remove them.
I feel the same way. One workaround is that some editors have a plugin. Emacs has a mode called ein that is fantastic. As you poke around getting the snippets to work, you can cut/paste into a different file/buffer. After you start the Jupiter notebook copy the login token, run: ein:notebooklist-login then ein:notebooklist-open and away you go.