51–60 of 158 posts
(The point is otherwise valid, I have seen software engineering in notebooks and it was horrific.)
I don't get the people here saying don't use Jupyter notebooks, or they are bad software engineering. So much Python development is trying snippets of code in a REPL as you introspect live objects, then once they're right pasting them into the IDE. All my Jupyter notebooks are like that, where my code starts as cells of a line or two, as I check each output. Then I coalesce them into a function (which avoid the probl…
I don't get the people here saying don't use Jupyter notebooks, or they are bad software engineering. So much Python development is trying snippets of code in a REPL as you introspect live objects, then once they're right pasting them into the IDE. All my Jupyter notebooks are like that, where my code starts as cells of a line or two, as I check each output. Then I coalesce them into a function (which avoid the probl…
> And also more code editing to move from IDEs into the Jupyter Jupyter, and all REPLS in general, rely on a concrete top-level execution context to provide useful feeedback. Much of the code written outside of data science doesn’t really have that, which is why you don’t see notebooks being used for software development very often.
But you are right about notebooks - I don't find them particularly useful, except as a better REPL, and no-one I know uses them for "normal" (non Data Science) software development.
I don't get the people here saying don't use Jupyter notebooks, or they are bad software engineering. So much Python development is trying snippets of code in a REPL as you introspect live objects, then once they're right pasting them into the IDE. All my Jupyter notebooks are like that, where my code starts as cells of a line or two, as I check each output. Then I coalesce them into a function (which avoid the probl…
As soon as I have one piece of functionality working as a function, I move the code to the "main" file for the project and import the function into the notebook. This means I never have to write more than a few paragraphs of code in the notebook and instead use it for testing and glue code. I don't think of the notebook as something sharable with others, it's more like WIP code... if I keep the notebook around it's usually because it can end up a useful test harness when code needs to be revisited/updated.
As for global state, one thing that I find really helpful is the keyboard shortcut 00. If you focus anywhere outside of a cell and you press zero twice, this restarts the kernel and you can re-run the commands from the beginning in a "clean state" so what you see matches what is. This approach works well when you're editing the source code that gets imported into the notebook---you never have to worry about re-importing since always restarting from scratch.
Most comments here express disbelief and disappointment in Jupyter from the software engineering point of view. What exactly is wrong with it? I use Jupyter daily and find no other Python environment more productive, be it scripts or IPython or IDEs. Granted, I work in scientific computing and use Python for data wrangling and stats. I find immense value in interactivity and iteration speed.
Maybe I can provide a bit of perspective on this, as I have lots of conflicting feelings about Jupyter. When I'm doing some bit of data wrangling or just exploratory work with data I quite like it - at least at first. As you pointed out, it's really easy to extremely quickly iterate on things and start to get an idea of what's in the data, what techniques work, and which don't. It's great. Until it isn't. I'm probabl…
Earlier quoted context omitted.
This is definitely one of my concerns too. Ad hoc code inside these notebooks is almost completely unmanageable from any reasonable software maintenance perspective, and refactoring code out of them is prohibitively difficult as well. I really want something to emerge that combines the best of both worlds of an IDE and notebook development, but there isn't anything close currently.
Some simpler notebook-like environments stay closer to source code in that they basically are source code with interleaved results (e.g. as comments). IMHO they hit a sweet spot between REPLs and those notebook environments inspired by mathematica, maple and similar more mathematically oriented software products.
Earlier quoted context omitted.
People do that in Excel too and the next thing you know a spreadsheet is managing a portfolio or being used as the basis for published science!
Its not the fault of the tools if users dont know any better.
Earlier quoted context omitted.
exactly. I use it for quick data exploration and experimentation but it remains at that level.
People do that in Excel too and the next thing you know a spreadsheet is managing a portfolio or being used as the basis for published science!
-It's browser based. Yuck. This means you can wait upwards of ten seconds for a session to load up every time you want to start one. Also, generally speaking, I deeply dislike browser-based applications that have nothing to do with the internet. It just feels so clunky. My workflow gets interrupted as I have to play between windows and betwen tabs, and whenever I want to switch from a task to another I have to remind myself 'well this is my web browsing instance that also happens to be my notebook', it's not natural and gets in the way. By contrast, a terminal advertises itself as exactly what you want to use it for. It's also very snappy.
-Following up on browser-based awkwardness, my god is this thing slow. Initializing kernels. Interrupting kernels. Restarting kernels. God save you if you want to import more than a couple libraries. Wait you made a mistake and want to stop and restart that one cell? Guess you just have to go through the whole kernel thing again I guess. So you launched something and it turns out it eats way more memory than expected? Well, 'sudo pkill -9 firefox' it is, do not pass go, do not collect autosave. It just does not play nice and lags behind every thought or action you want to take. Again, contrast with a plain terminal that just does the job and does it now.
-Speaking of exiting, these things don't seem to be able to do so gracefully. If something hangs up you just have to kill everything if you want to resume working, instead of good old ctrl-C. The fact that killing everything also tends to kill the web browser you used to do actual web browsing is just icing on the cake.
-Setting things up can be dodgy. Should your notebook fail to start, error messages are extremely cryptic. Documentation for troubleshooting is lackluster. No one ever seems to have any two identical issues on Stackoverflow, which is also plagued with WorksOnMyMachine (tm) syndrome. If you ever have to mess up with both Python 2 and 3, you may be in for a good time.
-The notebook format in which sessions are saved is supposed to be human readable and editable. In practice I have never seen a single human read or edit any of these by hand, for the very simple reason they are horrible to read or edit by hand. Again, why do I have to load a file in a cryptic format in my web browser when I could just %paste my snippets off of my editor or even just use my IDE's builtin capacities (code runner on vscode, F9 in spyder, etc.)? Plus, compatibility issues between versions can make your notebooks suddenly unusable (and hard to salvage due to the aforementioned format issue). This doesn't happen with snippets pasted onto a terminal.
-IDE integration has been sloppy in my experience. I've been told this may be subject to change, however.
Note that all this criticism doesn't mean I don't respect the work of the people behind this - they've made a fantastic tool that just doesn't happens to fit my use cases, for now. Every once in a while I try to work with notebooks, find they're still unusuable, shrug and go back to my terminals. As the experience improves I may make the switch one day, who knows.