61–70 of 153 posts
Some of the worst code I've seen in my life lives in Jupyter notebooks. But that's fine, it's meant to be throwaway code. The problem is a lot of places/people do not use it as such .
One of my family members literally maintains a software platform for test bedding production NASA spacecraft at JPL and it is all on top of Jupyter notebooks.
People are literally on course to build AGI and there’s a good chance that large portions of that work will be done within Jupyter notebooks.
To suggest real work doesn’t get done in notebooks is simply asinine.
As I have embedded further into data science and machine learning teams within my career I have come to learn that a lot of the things software engineers care about around maintainability, reusability etc. are often completely antithetical to data science productivity. In many cases practitioners would consider some of the listed cons of notebooks as features rather than bugs.
There is obviously a time and a place for pulling out those practices in such settings but blindly applying software engineering standards to this kind of work is both all too common and often rather counter productive.
If notebook style dev doesn’t jive with you personally I get that, and god speed. I’ve worked on CV/ML teams with a lot of CS degree grads and they don’t always want Jupyter notebooks. However, I’ve noticed a tendency for folks to extrapolate that to blanket statements about notebooks in general. As someone who has inherited and had to clean up many jumbled/bad notebooks I’m empathetic but I would challenge folks to lean into considering more about the circumstances the previous notebook author was working in and what kinds of productivity outcomes they were responsible and prioritizing rather than casting judgement on said author or their tools of choice.
But I often did.
I agree that notebooks are best reserved for exposition.
Certainly you can shoot yourself in the foot with notebooks, but you can also footgun with many other tools.
Some of the worst code I've seen in my life lives in Jupyter notebooks. But that's fine, it's meant to be throwaway code. The problem is a lot of places/people do not use it as such .
In general anything written by nonprofessional programmers in pursuit of some other goal is terrible code by professional programmer standards. On the other hand, it accomplishes a goal other than getting a programmer paid. So in one important sense is objectively better than probably 80% of the code I've seen people paid to write, no matter how nicely it was constructed.
I spend a good deal of time covertly teaching software engineering best practices to folks who claim they don't want to be "software engineers", yet they are in front of Jupyter most days.
Notebooks are primarily a _teaching_ tool, not code or an experimentation interface. I know that's not how they're used, but it's really bizarre to me that people use them for exploratory data analysis. They're great for teaching and for _documenting_ the results of exploration, but fairly bad for any type of interactive data exploration, let alone actually writing code. Frankly, notebooks are pretty terrible for exp…
I've taught and worked with some of the biggest companies in the world. They were using Jupyter and sound EDA to great effect.
This is a great article. For me, a lot relates to scale. I use notebooks occasionally for the university classes I teach. I like how I can write equations and text to explain ideas, and present interactive graphs to illustrate the ideas. But it's quite a lot of work to set things up, and there can be a problem when components of the work take a long time. In my research work, it is common for a calculation or a graph…
Earlier quoted context omitted.
In general anything written by nonprofessional programmers in pursuit of some other goal is terrible code by professional programmer standards. On the other hand, it accomplishes a goal other than getting a programmer paid. So in one important sense is objectively better than probably 80% of the code I've seen people paid to write, no matter how nicely it was constructed.
This is the crux of a lot of the anti-notebook rhetoric. I spend a good deal of time covertly teaching software engineering best practices to folks who claim they don't want to be "software engineers", yet they are in front of Jupyter most days.
I'm a so called "scientific" programmer, and I use notebooks. I care about good programming practices, and have made an effort to improve my craftsmanship.
And I don't want to be a software engineer. I'm surrounded by SWE's in my work area, and their job doesn't really interest me. Division of labor. I'm glad somebody enjoys it.
I think people may be misusing notebooks. They aren't there to develop software but serve as virtual versions of scientific notebooks (hence the name). They are there to conduct experiments (changing parameters and the like in your code) and to record and plot the results. You don't have to develop the software itself in the notebook.
I can change parameters in a script. What's the advantage?
if your calculation is long running you would not be as productive as could be in notebooks