Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notebooks Are McDonalds of Code

yobibyte.github.io

Post reply on HN
Print view View original

111–120 of 153 posts

Re: Notebooks Are McDonalds of Code

#111

Post by neonsunset »

waldrews wrote: post #87
skydhash wrote: post #77

Earlier quoted context omitted.

I wonder why CL and Smalltalk haven’t beat Python. Is is the languages or just unawareness? The workflow just make more sense there with better updates propagation and state saving.

The Clojure/JVM statistics/scientific computing/now tensor math packages just never got as good as Python, and in Smalltalk they were a non-starter. R is an awkward language, but it's repl-first, has a lot of Lisp's metaprogramming (done in very ad-hoc ways), and Smalltalk's serializable image model -- so a lot of exploratory/experimental statistical methods research happens there, and then gradually makes its way to…

In their defense, F# has https://diffsharp.github.io and .NET has https://ilgpu.net

In terms of NumPy port, for more involved implementations you might be better off using built-in numeric primitives together with upcoming Tensor(kind of like ndarray on steroids) and TensorPrimitives (BLAS).

Re: Notebooks Are McDonalds of Code

#112

Post by whywhywhywhy »

alan-hn wrote: post #5
jhbadger wrote: post #3

I think people may be misusing notebooks. They aren't there to develop software but serve as virtual versions of scientific notebooks (hence the name). They are there to conduct experiments (changing parameters and the like in your code) and to record and plot the results. You don't have to develop the software itself in the notebook.

I can change parameters in a script. What's the advantage?

You can rerun something halfway down a script without re-running the whole thing.

If your script requires loading 12+ GB of ml models into a gpu before running anything at all this is the difference between a few seconds and a minute to see a change also if the output isn't text you can see the image or chart result inline to that code.

Re: Notebooks Are McDonalds of Code

#113

Post by subjectsigma »

jhbadger wrote: post #3

I think people may be misusing notebooks. They aren't there to develop software but serve as virtual versions of scientific notebooks (hence the name). They are there to conduct experiments (changing parameters and the like in your code) and to record and plot the results. You don't have to develop the software itself in the notebook.

I always hated notebooks personally as I found them clunky and inefficient, but never wanted to yuck someone else’s yum.

Then I started mentoring a junior staff member who worked on another project. The lead of that project was a physicist who wrote primarily in Jupyter notebooks. Like, thousands and thousands of lines of code. This junior staff member spent like 90% of her time confused and copying-and-pasting between notebooks. She had no idea what a virtual environment was and her go-to solution for solving import errors was to nuke and re-download the repo and re-run the setup script, or create a new notebook from scratch containing all the prerequisite user functions.

Was top 10 most horrifying things I’ve ever seen. I advised her immediately to stop using notebooks for development and sent her a few Python tutorials. Luckily though she just left the project and got on a better one instead.

Re: Notebooks Are McDonalds of Code

#115

Post by setopt »

Planktonne wrote: post #108
setopt wrote: post #105

Earlier quoted context omitted.

An alternative here is to make a script with `# %%` code cells, so you can send one cell at a time to a REPL while developing the script.

I think you've just reinvented the notebook.

On the contrary, it’s a backport of the (IMO) most useful notebook feature to work also for regular scripts. It’s been ported to most code editors by now (VSCode “Interactive Python”, Vim “hydrogen”, Emacs “code-cells.el”, Sublime “Send Code” extension, Spyder has it built-in, etc.), so a lot of people have found that feature useful.

In contrast to Jupyter, you’re still working with plain text files and not JSON, and you don’t end up saving the cached data in the same file as the script.

In contrast to Quarto, Jupytext, etc., this is still just a code file and not a MarkDown file with code blocks. Not all editors have fully working “go to definition” etc. in MarkDown code blocks, and in any case, many people need a standalone script that can be placed in an HPC job queue after initial local testing is done.

Re: Notebooks Are McDonalds of Code

#117

Post by ahmadmijot »

subjectsigma wrote: post #113
jhbadger wrote: post #3

I think people may be misusing notebooks. They aren't there to develop software but serve as virtual versions of scientific notebooks (hence the name). They are there to conduct experiments (changing parameters and the like in your code) and to record and plot the results. You don't have to develop the software itself in the notebook.

I always hated notebooks personally as I found them clunky and inefficient, but never wanted to yuck someone else’s yum. Then I started mentoring a junior staff member who worked on another project. The lead of that project was a physicist who wrote primarily in Jupyter notebooks. Like, thousands and thousands of lines of code. This junior staff member spent like 90% of her time confused and copying-and-pasting betwe…

This is why spaghetti code is so prevalent in scientific community. We are not really train (especially for non engineering research projects) to become programmers and we do programming just as a way to do research.

Re: Notebooks Are McDonalds of Code

#118

Post by rcxdude »

jofer wrote: post #41

Notebooks are primarily a _teaching_ tool, not code or an experimentation interface. I know that's not how they're used, but it's really bizarre to me that people use them for exploratory data analysis. They're great for teaching and for _documenting_ the results of exploration, but fairly bad for any type of interactive data exploration, let alone actually writing code. Frankly, notebooks are pretty terrible for exp…

The issue with CLI is it's crap for editing the code. It's rare that your data analysis will just be one-liners, it'll be some mess of steps and you want to go edit one in the middle (but not re-run the whole script). That's the first thing that the notebooks give you that you don't tend to get elsewhere.

The second thing is the presentation of the plots: it's really handy to have multiple interactive plots laid out right next to the code block that produced them.

There's nothing that requires that this is done in a web interface, but no-one seems interested in making a non-web version (IIRC there was a QT version in the very very early days of ipython notebook but it's long dead). One big thing that any replacement will need is the easy ability for the code to actually be running on a remote machine, something that's trivial for a web interface but requires active effort for a native client (and makes a lot of interactive chart work more difficult: certainly there's currently no other matplotlib backend which can do it).

I'm fairly sure a better interface than jupyter is possible for data exploration, but I haven't found one yet since there's not really anything else that gives you the two features above.

(one thing people often miss about notebooks and the kind of work that go into them is that the code itself isn't actually really hard code. It's 100% about interacting with the data and the algorithms running on it. Working in one isn't much like software engineering)

Re: Notebooks Are McDonalds of Code

#119

Post by rcxdude »

alan-hn wrote: post #58
pquki4 wrote: post #42

Earlier quoted context omitted.

For a script, you run it from start to end. For a notebook/repl environment, you can create any number of intermediate steps, rerun the previous step with minor modifications and check if the results are better, rinse and repeat. For jupyter notebook specifically, you can visualize data and add markdown inline which are very useful. You won't understand it unless you are already familiar with the workflow.

Sometimes I need to run my code in small pieces for testing and evaluation, I do this with multiple smaller scripts. Data can be saved to files and this accomplishes nearly the same thing as a notebook without needing to have the notebook environment

Sure, if you don't mind the overhead (both development and processing time) of loading/saving state to disk (about 10-30 minutes for a lot of my data). In notebooks you don't have to think about it since it's just the objects in memory (and indeed they don't make it easy to think about it, which is a reasonable criticism, but I don't currently know of a framework or system which gives you the advatages of both).

Re: Notebooks Are McDonalds of Code

#120

Post by rcxdude »

Tarq0n wrote: post #32
eldenring wrote: post #12

Earlier quoted context omitted.

of course, running a script from start to finish is better when it's fast, but if your code is slow to run, each cell acts like an in memory cache of your previous work.

There's so many other ways to achieve the same effect though. I would argue that serializing the result of some expensive computation explicitly is even just better, as it can have a meaningful name and metadata.

Just serialising and deserialising the results can be expensive though: it's quite for jupyter notebooks to have >64GB of data loaded. Even with high-end SSDs that's a fairly significant overhead.
Post reply on HN