Advanced computing with IPython • Hacker News Board

Re: Advanced computing with IPython

#41

Post by joshuamorton » Tue, Jun 05, 2018, 3:40 PM UTC

mlthoughts2018 wrote: post #6

If interested, I spent some time on a comment thread a few days ago describing how my experience leads me to believe the Notebook environment (not all of Jupyter / IPython, just the Notebook part) is actually only appropriate for a tiny subset of pedagogical or throw-away situations, and should be avoided most of the time and avoided in most of the cases it’s marketed for (especially anything having to do with ‘repro…

I think most of the complaints you outline can be solved by

1. Having tooling support for source controlling a notebook without all the garbage generated json causing issues. 2. Having tooling support for testing 3. Having style and review guidelines for notebooked code

In other words, treat a notebook as a first class citizen, and as a legitimate part of the workflow, and it stops feeling like a crufty, unreproducible, hacky mess.

Re: Advanced computing with IPython

#42

Post by mlthoughts2018 » Tue, Jun 05, 2018, 3:42 PM UTC

bunderbunder wrote: post #17

mlthoughts2018 wrote: post #6

If interested, I spent some time on a comment thread a few days ago describing how my experience leads me to believe the Notebook environment (not all of Jupyter / IPython, just the Notebook part) is actually only appropriate for a tiny subset of pedagogical or throw-away situations, and should be avoided most of the time and avoided in most of the cases it’s marketed for (especially anything having to do with ‘repro…

You're not wrong, though I think the value of pedagogical and throw-away situations in your work may not be a universal experience. For me, at least, pedagogic and throw-away situations aren't a tiny subset. They're most of what I do. It's exploratory work, figuring out how the data behaves, if the data behaves, where it needs to be cleaned, churning through great heaps of experiments and iterations before hitting on…

> “It's exploratory work, figuring out how the data behaves, if the data behaves, where it needs to be cleaned, churning through great heaps of experiments and iterations before hitting on the ultimate plan”

The thing is though, you should be involving code reviewers even at this stage, to review both the statistical methodology you intend for your experiments, and also the source code you believe implements that methodology. (Even when working alone, but absolutely when part of a team).

Instead of seeing the notebook as a big series of scratch-pad attempts to get something right, you should be using pull requests and code review as that scratch pad.

Additionally, the functions, classes and modules you create to do the work of exploring data fidelity, cleaning pre-treatments, or parameter sweeps through sets of experiment-specific parameter bundles — all that should be written like proper, testable, well-designed code, that lives in separate libraries or packages to facilitate re-using it without reinventing the wheel or copy/pasting from some old notebook, etc.

By that point, the notebook you’d use to explore data behavior or to invoke distributed training across a bunch of parameter values would be a tiny notebook that just imports everything it needs from properly maintained helper libraries you wrote.

And the value of the notebook over the same code just living in an easy-to-review script starts to be extremely questionable.

Re: Advanced computing with IPython

#43

Post by [deleted] » Tue, Jun 05, 2018, 3:44 PM UTC

AdmiralAsshat wrote: post #32

Deep research uses aside, I often prefer to use IPython because it's simply a better shell than the default Python shell. You get basic niceties like tab completion and being able to up-arrow to revise an earlier multi-line command (like a function) without it being an exercise in frustration.

[deleted]

Re: Advanced computing with IPython

#44

Post by kaitai » Tue, Jun 05, 2018, 3:46 PM UTC

w0m wrote: post #29

chrxr wrote: post #11

Earlier quoted context omitted.

It is worth noting that there is an argument that it is a worthwhile task for students to learn how to setup complex computing environments, as it better prepares them for the real world. However, in reality, there just isn't time within a single semester to do this for a class of 100+ students. So implementations such as this one trade-off that learning for a greater focus on computational theory and its implementat…

Agreed. At the beginning of class, walk students through the setup. Then for every project after, let them use the pre-rolled systems.

I'd actually do the opposite. Let them use pre-rolled first, then when they actually know and care about how the system is set up, have them set it up the way they like it.

(I am actually leading a machine learning for high-schoolers camp in 2 weeks and we are using Jupyter notebooks so that all students, with heterogeneous backgrounds, will start in the same place and get to the fun stuff fast. Many will never have used Python and will not know or care about 2.7 vs 3, just to give the most high-level and basic example!)

Re: Advanced computing with IPython

#45

Post by bunderbunder » Tue, Jun 05, 2018, 3:48 PM UTC

wenc wrote: post #28

bunderbunder wrote: post #17

Earlier quoted context omitted.

You're not wrong, though I think the value of pedagogical and throw-away situations in your work may not be a universal experience. For me, at least, pedagogic and throw-away situations aren't a tiny subset. They're most of what I do. It's exploratory work, figuring out how the data behaves, if the data behaves, where it needs to be cleaned, churning through great heaps of experiments and iterations before hitting on…

> At which point, forget Jupyter notebooks, I'm typically not even working in Python anymore for that part of the job. This is what is typically done out there but I suggest it breaks the feedback loop between the scientist roles and the developer roles. In rapidly changing environments those feedback loops could be crucial. It's similar to what Wall Street folks did (still do?)--quants write models in Excel/VBA and…

Well, emphasis on the pronouns there. I'm not doing the proverbial "throw it over the wall to engineering", I'm also writing the production version. I also dislike the "2 teams" approach. Even if you have separate roles for data scientists and software engineers, better to mix them onto a single team than force them to communicate across a partition.

For me it's really down to efficiency. Writing somewhat production-ready code is more expensive and time-consuming than blithely hacking. In the early stages of a new project, I know that almost everything I'm doing will get thrown away. For the most interesting projects, there's even a decent chance that it will be a complete failure and everything gets thrown away. So, at that stage in the game, I'm inclined to say that any extra effort spent on production readiness is just a waste of time and money. Fail fast, YAGNI, etc.

Re: Advanced computing with IPython

#46

Post by mlthoughts2018 » Tue, Jun 05, 2018, 3:54 PM UTC

joshuamorton wrote: post #41

mlthoughts2018 wrote: post #6

If interested, I spent some time on a comment thread a few days ago describing how my experience leads me to believe the Notebook environment (not all of Jupyter / IPython, just the Notebook part) is actually only appropriate for a tiny subset of pedagogical or throw-away situations, and should be avoided most of the time and avoided in most of the cases it’s marketed for (especially anything having to do with ‘repro…

I think most of the complaints you outline can be solved by 1. Having tooling support for source controlling a notebook without all the garbage generated json causing issues. 2. Having tooling support for testing 3. Having style and review guidelines for notebooked code In other words, treat a notebook as a first class citizen, and as a legitimate part of the workflow, and it stops feeling like a crufty, unreproducib…

1. and 2. could alleviate many of the problems, but people overlook how incompatible 1. and 2. are with any basic approach to testing.

For example, notebooks inherently intermix units of display logic with units of implementation logic, but obviously these are separate concerns (in the spirit of e.g. Model-View-Controller), and you shouldn’t be writing “a module” (the notebook) that intermixes them & requires wacky coupling of display concerns for implementation questions.

(This also extends to the formatting of non-code aspects of notebooks too, which should be reviewed as a separate concern... much like how in LaTeX, what content I write is a different concern than how it renders).

Imagine the added strain on e.g. a bunch of pytest code you’ve already written for an underlying implementation library if you now require it to be used for also automatically testing display logic too.

Finally, item 3. is also tricky because the point of a notebook is presentation, so the style guide might rightfully be different. Now if you have some implementation unit (some code block, function body, whatever) it’s suddenly a debate whether it should be styled for presentation or styled according to the team’s source code guidelines .. and you’d need more linting tools that can be surgically used on subsets of the notebook, which seems needlessly complicated in comparison to just factoring out units of implementation logic into a separate helper module in the first place.

For these reasons, it is actually quite hard to treat a notebook as a first-class citizen in any sense beyond mere convention.. which is useless in the automated testing and review side of the issue.

Re: Advanced computing with IPython

#47

Post by happy-go-lucky » Tue, Jun 05, 2018, 3:57 PM UTC

It's a pdf file, which I am trying to open but cannot. Can you guys read it?

Edit: The link to the file: https://hpc.nih.gov/training/handouts/171121_python_in_hpc.p...

Edit: I can read it now.

Re: Advanced computing with IPython

#48

Post by wenc » Tue, Jun 05, 2018, 4:07 PM UTC

mlthoughts2018 wrote: post #39

wenc wrote: post #22

Earlier quoted context omitted.

Agreed. Notebook environments are great for exploration, discovery and pedagogy. They aren't so good for productionizing code. We found this out the hard way when we tried to productionize ML code in Jupyter. We had to export to .py and add boilerplate. This works fine unless there is back and forth iteration between modeling and prod, which there invariably is; our data scientists had to make changes to the notebook…

On that console / IDE point you made, IPython can still be quite good for that if you use the interactive shell. For example, I might make two shell tabs in tmux, and make one a small rectangle towards the bottom of the screen (holds my running IPython session), and a large rectangle above it (holds my Emacs where I’m editing source code). And I might have a third shell tab somewhere that detects any time source file…

Yes, that's definitely a possibility. Though it would be nice if it were more nicely integrated into an environment like RStudio where you can interactively set breakpoints, watch variables, etc. while still maintaining the interactivity.

I do the tmux/vim too, but for exploratory work the experience is less well-integrated than it could be with an Rstudio-like IDE.

Re: Advanced computing with IPython

#49

Post by kennedyp » Tue, Jun 05, 2018, 4:14 PM UTC

bitexploder wrote: post #8

Don't overlook all of the % commands, such as %edit. If you are familiar with emacs keybindings, it has a very good built in editor as well. You can also load snippets from saved files, and save your history to a file. Or individual lines to files using range type expressions. In short it is very easy to get code in and out of IPython. Another great trick. Anywhere you want to debug or play in your scripts run `impor…

You can also leverage embed() within ipdb too and it can be nested. Very useful. Especially when combined with the ultratb or set_trace() from ipdb (or pdb):

syntax is a little different dependant on whether you're in py3 or 2.7 so just google it but the jist of it is you replace the sys.excepthook with this colortb then when your script hits an uncaught exception say, you'll be presented with an ipdb colorised traceback.

You can step up and down the stack frames, embedding at any point with `from IPython import embed` then `embed()` to get a fully functional ipython repl. game changer

Re: Advanced computing with IPython

#50

Post by joshuamorton » Tue, Jun 05, 2018, 4:14 PM UTC

mlthoughts2018 wrote: post #46

joshuamorton wrote: post #41

Earlier quoted context omitted.

I think most of the complaints you outline can be solved by 1. Having tooling support for source controlling a notebook without all the garbage generated json causing issues. 2. Having tooling support for testing 3. Having style and review guidelines for notebooked code In other words, treat a notebook as a first class citizen, and as a legitimate part of the workflow, and it stops feeling like a crufty, unreproducib…

1. and 2. could alleviate many of the problems, but people overlook how incompatible 1. and 2. are with any basic approach to testing. For example, notebooks inherently intermix units of display logic with units of implementation logic, but obviously these are separate concerns (in the spirit of e.g. Model-View-Controller), and you shouldn’t be writing “a module” (the notebook) that intermixes them & requires wacky c…

Generally speaking, the code you write in a notebook isn't too important for presentation, the code you write in a notebook creates something to be presented (a chart, an output value, a pandas dataframe). I'll grant that this isn't true 100% of the time, but I find its normally the case.

So start with the assumption that you have magic tooling (which I know exists) that allows you to ignore diffs in generated output as far as code review and committing is concerned. The impl doesn't really matter. Then you also have magical tooling that allows you to write unit tests at the cell level, either treating a cell as a function container, and testing the single function within or treating a cell an "hermetic" function itself, where you configure globals, run the cell in the context of those globals, and see what the globals look like afterwords.

That's why I'm saying you need dedicated tooling support. And you're still thinking in the realm of "oh I test this normally". No no, I mean truly dedicated libraries that hook into the notebook environment itself. You might structure your tests as pytests, but under the hood they're spinning up a notebook env and doing magic.

>units of implementation logic into a separate helper module in the first place.

Well sure, you need this anyway though. Any kind of shared infrastructure should be factored out into a module, not because of the notebook environment, but because that's good abstraction independent of how you're presenting your models.