Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notebooks Are McDonalds of Code

yobibyte.github.io

Post reply on HN
Print view View original

51–60 of 153 posts

Re: Notebooks Are McDonalds of Code

#51

Post by hot_gril »

YurgenJurgensen wrote: post #28
alan-hn wrote: post #4

I've never been a fan of notebooks, it always felt like a strange way to run my code. What's the real advantage over creating scripts? If I want my code to run somewhere I would need to notebook architecture or just turn it into a normal script anyhow

Notebooks have at least one advantage, but it's one that shouldn't really be an advantage: In a world where every big organisation rushed to adopt all the latest cloud buzzwords, we now have an awful lot of half-baked fontends for AWS and Azure that will certainly be mandated by company policy. Actually getting a computer that can access all the data you want and run in a way you want through EU law, US law, EULAs, c…

Same reason spreadsheets are so powerful.

Re: Notebooks Are McDonalds of Code

#52

Post by hot_gril »

giraffe_lady wrote: post #50
hot_gril wrote: post #43

Earlier quoted context omitted.

In one project, I had a new teammate ask me why almost our entire web backend is a single .js file mostly filled with SQL queries. He wanted to make it his personal project to refactor this into probably 20 different files, adding more layers between the handlers and the DB stuff, using a query builder, and also migrating to TS. When I asked him what's wrong with the current thing, he couldn't answer and gave up on t…

Yeah I think I got this from Sandi Metz a long time ago but it's been so valuable to me: "bad code that you never have to modify is good code." Now when business requirements change and you're constantly sending people into that 1600 line backend function or whatever, it's likely worth it to start doing some refactors. But otherwise if I don't have a good reason to be in there I pretend I don't see it.

I agree with the quote, but this was code that we did modify frequently. It was still ok. 1600 line func is bad, but 1600 line file, well that's what editor splits are for. A lot of app backends don't do much other than shoving stuff into a DB and getting it back.

Re: Notebooks Are McDonalds of Code

#53

Post by bluenose69 »

This is a great article. For me, a lot relates to scale.

I use notebooks occasionally for the university classes I teach. I like how I can write equations and text to explain ideas, and present interactive graphs to illustrate the ideas.

But it's quite a lot of work to set things up, and there can be a problem when components of the work take a long time.

In my research work, it is common for a calculation or a graph to take hours to days. Doing work like that in a notebook is just a non-starter. I use scripts (in various languages) along with Makefiles that "know" if I've changed my code or my data, and only rebuilds a result file or a graph when require. Almost always, I separate code that does analysis from code that creates graphical displays. And of course the text (of a paper, course notes, etc.) is written to incorporate these numerical and graphical results. The whole point is to subdivide complex tasks into smaller tasks that can be executed in an organized way.

I don't see the sense in using one tool (notebooks, say) for simple tasks and other tools (scripts, a strong editor, tmux, unix, etc.) for complex tasks. So, apart from the toy tools that I make for some teaching tasks, I stick to simple unix-style tools for the tricky stuff.

It's important to have many tools in the toolbox. Notebooks can be useful for some things, but I wouldn't want to frame a wall using an awl.

Re: Notebooks Are McDonalds of Code

#54

Post by WalterSear »

hot_gril wrote: post #48
WalterSear wrote: post #38

Earlier quoted context omitted.

https://nbdev.fast.ai/

Oh, that looks like a disaster. Edit: I can't judge without trying, but that's my initial thought.

I have masses of admiration and respect for Jeremy Howard and tried using this to build a bunch of fastapi endpoints.

I still have masses of admiration and respect for him, but no longer so much when it comes to software engineering - due to this, among other things. It seems like a great idea, but is clunky, hard to refactor and to integrate with other things, in my practice.

Wrapping production code in jupyter cruft doesn't work for me. Or, any code, really, that doesn't stay very small and stand alone.

Re: Notebooks Are McDonalds of Code

#55

Post by lelandbatey »

Cacti wrote: post #9
alan-hn wrote: post #5

Earlier quoted context omitted.

I can change parameters in a script. What's the advantage?

It’s a REPL, for starters.

REPL stands for Read–eval–print loop https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93prin...

A REPL, by its name, is a very narrow version of the broader paradigm of interactive computer programming environments. But Notebooks are not REPLs, unless you use REPL to mean "interactive programming environment" and not REPL. Notebooks are much broader than a REPL! In a notebook, you can go back and edit and run individual lines of the notebook without re-running the whole notebook from the start and without re-computing everything that depends on what you just edited. Behavior like this makes it super hard to track the actual state, and super easy to lose track of how things are how they are. That's pretty terrible!

The parent article links this great talk that goes into more detail than the parent post and is much easier to understand: https://www.youtube.com/watch?v=7jiPeIFXb6U

Re: Notebooks Are McDonalds of Code

#56

Post by bitwize »

Notebooks are used for crunching data, for scientific, mathematical, or engineering purposes, and showing your work (revealing to your colleagues exactly how the data was crunched). If you're deploying them to production you get what you f'n deserve.

That said, when I'm writing code, if I want it to be good quality code then I might use org-mode and its literate programming facilities to type up my thoughts, interlineating code as necessary. This is more useful on greenfield and personal stuff, but there is a high probability that I too have undiagnosed ADHD and the use of org-mode helps to organize and clarify my thoughts before I commit them to code. It is a hallmark of the programmer personality to jump into coding without thinking things through beforehand -- or really, sometimes, at all. The management consultant Tim Bryce wrote about a programming class he took in the 70s:

"When given an assignment, the other two members of my team started coding immediately, then keyed up the source code (Yes, we were using punchcards back then), and kept compiling and debugging until they finally got a clean compile. While this was going on, I pulled out my trusty old template, worked out the logic, wrote the code, keyed it up and usually got a clean compile on my first shot. Inevitably, I beat my teammates everytime."

The use of org-mode and LP helps with these organization and planning issues... well, at least it does for me.

Re: Notebooks Are McDonalds of Code

#57

Post by williamcotton »

cqqxo4zV46cp wrote: post #40
giraffe_lady wrote: post #17

Earlier quoted context omitted.

In general anything written by nonprofessional programmers in pursuit of some other goal is terrible code by professional programmer standards. On the other hand, it accomplishes a goal other than getting a programmer paid. So in one important sense is objectively better than probably 80% of the code I've seen people paid to write, no matter how nicely it was constructed.

Amen to that. The confluence of an insulator nature of organisational developer cliques, the culture of self-importance in the field, and the ability for many teams to expend material effort on things that are…at best, tangentially related to business goals (due to cheap money, and frankly, often pulling the wool over management’a eyes), has bred at least a couple of generations of developer where large contingents h…

Preach!

Re: Notebooks Are McDonalds of Code

#58

Post by alan-hn »

pquki4 wrote: post #42
alan-hn wrote: post #4

I've never been a fan of notebooks, it always felt like a strange way to run my code. What's the real advantage over creating scripts? If I want my code to run somewhere I would need to notebook architecture or just turn it into a normal script anyhow

For a script, you run it from start to end. For a notebook/repl environment, you can create any number of intermediate steps, rerun the previous step with minor modifications and check if the results are better, rinse and repeat. For jupyter notebook specifically, you can visualize data and add markdown inline which are very useful. You won't understand it unless you are already familiar with the workflow.

Sometimes I need to run my code in small pieces for testing and evaluation, I do this with multiple smaller scripts. Data can be saved to files and this accomplishes nearly the same thing as a notebook without needing to have the notebook environment

Re: Notebooks Are McDonalds of Code

#59

Post by nikonyrh »

lelandbatey wrote: post #55
Cacti wrote: post #9

Earlier quoted context omitted.

It’s a REPL, for starters.

REPL stands for Read–eval–print loop https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93prin... A REPL, by its name, is a very narrow version of the broader paradigm of interactive computer programming environments. But Notebooks are not REPLs, unless you use REPL to mean "interactive programming environment" and not REPL. Notebooks are much broader than a REPL! In a notebook, you can go back and edit and run in…

"In a notebook, you can go back and edit and run individual lines of the notebook without re-running the whole notebook from the start and without re-computing everything that depends on what you just edited."

Isn't this a standard on REPLs as well? You can select the code you wish to run, and press Ctrl+Enter or what ever. I must admit, I've programmed Python for about 10 years in Spyder and VS Code now, but I haven't used notebooks at any point. Just either ad-hoc scripts or actual source files.

My definition of a "notebook" is an ad-hoc script, split into individual "cells" which are typically run as a whole. On my workflow, I just select the code I wish to run. Sometimes it is one expression, one line, 100 lines or 1000 lines depending what I've changed on the script.

Re: Notebooks Are McDonalds of Code

#60

Post by slowking2 »

vineyardlabs wrote: post #10

I'm an embedded person, so clueless on this stuff. Are notebooks really used in prod? Outside of applications where notebooks are specifically useful to the goal of the system, ie. documentation,blog posts,textbooks with interactive/data driven elements? How would this even work? Can an ipynb file be mechanized in the same way a py file can?

Jupyter notebooks can be executed roughly like scripts by papermill. You can also save a .py version of the notebook without outputs using jupytext. We use these packages together where I work to basically auto-generate the start of notebooks for exploratory work after certain batch jobs. For dashboards only used by a small number of users, we’ve found voila occasionally useful. Voila turns notebooks into dashboards or web apps basically.

You generally shouldn’t put code you want to reuse in notebooks, but we haven’t found this to be much of a problem. And >3/4 of the people who’ve worked on our team don’t have software engineering backgrounds. If you set clear expectations, you can avoid the really bad tar pits even if most of your co-workers are (non-software) engineers or scientists 0-3 years out of college.

Post reply on HN