Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Advanced computing with IPython

lwn.net

Post reply on HN
Print view View original

51–60 of 108 posts

Re: Advanced computing with IPython

#51

Post by sametmax »

da_chicken wrote: post #13
chrxr wrote: post #10

At Harvard we've built out an infrastructure to allow us to deploy JupyterHub to courses with authentication managed by Canvas. It has allowed us to easily deploy complex set-ups to students so they can do really cool stuff without having to spend hours walking them through setup. Instructors are writing their lectures as IPython notebooks, and distributing them to students, who then work through them in their Jupyte…

I'm torn. On the one hand that's really cool to get everything configured and up and running so students can get to the interesting parts. On the other hand, learning how to configure your own environment is kind of an essential part of working with any tool that forces you to understand at least some of the structure involved.

If you have 100 students, you will have 100 different mistakes to debug in the setup. A setup is not a program, there is rarely an easy way to pinpoint a problem, and so it takes a lot of time to setup just one, let alone a 100.

When the number of hours are limited, it's best to skip it entirely, and just provide a solid paper tutorial.

Re: Advanced computing with IPython

#52

Post by mlthoughts2018 »

wenc wrote: post #48
mlthoughts2018 wrote: post #39

Earlier quoted context omitted.

On that console / IDE point you made, IPython can still be quite good for that if you use the interactive shell. For example, I might make two shell tabs in tmux, and make one a small rectangle towards the bottom of the screen (holds my running IPython session), and a large rectangle above it (holds my Emacs where I’m editing source code). And I might have a third shell tab somewhere that detects any time source file…

Yes, that's definitely a possibility. Though it would be nice if it were more nicely integrated into an environment like RStudio where you can interactively set breakpoints, watch variables, etc. while still maintaining the interactivity. I do the tmux/vim too, but for exploratory work the experience is less well-integrated than it could be with an Rstudio-like IDE.

I agree, and the IDE setups can be very valuable for certain use cases or certain preferences. The equivalent thing in the IPython shell approach, basically using souped-up pudb, is not quite as nicely interactive with the debugging cycle, since setting breakpoints, watchpoints, etc., is either a matter of editing them into the source code and re-running, or becoming a master of specifying them on the command line, both of which require stepping out of the tight iteration workflow slightly (but to be fair, they also offer more power than the preconfigured options availabe in the IDE debugger features).

Re: Advanced computing with IPython

#53

Post by nerpderp83 »

w0m wrote: post #29
chrxr wrote: post #11

Earlier quoted context omitted.

It is worth noting that there is an argument that it is a worthwhile task for students to learn how to setup complex computing environments, as it better prepares them for the real world. However, in reality, there just isn't time within a single semester to do this for a class of 100+ students. So implementations such as this one trade-off that learning for a greater focus on computational theory and its implementat…

Agreed. At the beginning of class, walk students through the setup. Then for every project after, let them use the pre-rolled systems.

Setup is orthogonal to understanding and learning, leave it for extra credit or an optional follow up exercise. It will lose or distract many students.

Re: Advanced computing with IPython

#54

Post by wenc »

bunderbunder wrote: post #45
wenc wrote: post #28

Earlier quoted context omitted.

> At which point, forget Jupyter notebooks, I'm typically not even working in Python anymore for that part of the job. This is what is typically done out there but I suggest it breaks the feedback loop between the scientist roles and the developer roles. In rapidly changing environments those feedback loops could be crucial. It's similar to what Wall Street folks did (still do?)--quants write models in Excel/VBA and…

Well, emphasis on the pronouns there. I'm not doing the proverbial "throw it over the wall to engineering", I'm also writing the production version. I also dislike the "2 teams" approach. Even if you have separate roles for data scientists and software engineers, better to mix them onto a single team than force them to communicate across a partition. For me it's really down to efficiency. Writing somewhat production-…

Oh I understand, I'm one of the few people on my team who does devops + data engineering + data science (some people on my team only do 1 or 2, but not all 3). My point is more about the impedance mismatch between roles and code produced by each role, whether or not they are carried out by the same person. For instance, I find it difficult iterating between my own model code and production code, especially if the model code was conceived in an interactive notebook environment.

I do agree that notebooks are good for writing throwaway code, but of n failed notebooks, typically there's one that we'd like to bring to production. That's typically the one notebook we'd want to be production ready.

When I say production-readiness, I don't mean actually working in production boilerplate in the first iteration (maybe in later iterations...). I mean writing the code in a way that lends itself to easy productionization through observance of certain constraints, e.g. being cognizant of environment/scoping/global state/namespace conflicts, writing model code in modular units (functions or classes depending on the use case) rather than just imperative line-by-line code, etc. These tiny disciplines are almost effortless but can lower the friction of iterating between model and production.

In data science work, the real proof of the pudding is in production, not in unit tests. Most people don't want to admit this but unit testing doesn't work as well in the mathematical modeling world as they do in the software development world -- much of the time our inputs aren't discrete/enumerable, and the state-space is large or infinite. So it's really important to be able to iterate between production and modeling. If I ever need to go back to my interactive environment to experiment and change the logic, there should be an easy path to flow that back into production. Right now notebook environments don't aid in that. I've observed OTOH that IDE environments do.

Re: Advanced computing with IPython

#55

Post by wenc »

mlthoughts2018 wrote: post #52
wenc wrote: post #48

Earlier quoted context omitted.

Yes, that's definitely a possibility. Though it would be nice if it were more nicely integrated into an environment like RStudio where you can interactively set breakpoints, watch variables, etc. while still maintaining the interactivity. I do the tmux/vim too, but for exploratory work the experience is less well-integrated than it could be with an Rstudio-like IDE.

I agree, and the IDE setups can be very valuable for certain use cases or certain preferences. The equivalent thing in the IPython shell approach, basically using souped-up pudb, is not quite as nicely interactive with the debugging cycle, since setting breakpoints, watchpoints, etc., is either a matter of editing them into the source code and re-running, or becoming a master of specifying them on the command line, b…

Yes, my IDE is vim but it's a hard sell to a lot of folks... especially having to map a shortcut key to "import ipdb; ipdb.set_trace()" for breakpoints...

Rodeo [1] was an attempt at an IDE but development died, and now that yhat's been acquired, there's no sign of any further development. I wish the Jupyter folks would push more in this direction (and they are with Jupyter Lab) but I get the sense they are really invested in the notebook paradigm.

[1] https://www.yhat.com/products/rodeo

Re: Advanced computing with IPython

#56

Post by mlthoughts2018 »

joshuamorton wrote: post #50
mlthoughts2018 wrote: post #46

Earlier quoted context omitted.

1. and 2. could alleviate many of the problems, but people overlook how incompatible 1. and 2. are with any basic approach to testing. For example, notebooks inherently intermix units of display logic with units of implementation logic, but obviously these are separate concerns (in the spirit of e.g. Model-View-Controller), and you shouldn’t be writing “a module” (the notebook) that intermixes them & requires wacky c…

Generally speaking, the code you write in a notebook isn't too important for presentation, the code you write in a notebook creates something to be presented (a chart, an output value, a pandas dataframe). I'll grant that this isn't true 100% of the time, but I find its normally the case. So start with the assumption that you have magic tooling (which I know exists) that allows you to ignore diffs in generated output…

I guess I’m saying the problem with testing or linting a notebook is not technical.

Writing a tool that suppresses output cells, infers global parameter blocks, etc., is trivial. Writing a linter with enough configurability to account for notebook presentation styling might be harder, but still straightforward.

Creating the raw tools that can do it is the easy part.

The hard part is that writing code for re-use and testability and with reasonable low-effort best practices at separating concerns and having modularity — all that is antithetical to the whole purpose of the notebook.

So why bother contorting the testing apparatus to accomodate testing something that is created with throw-away design principles from the start?

As soon as you start using the design principles from the beginning of the first prototype or first data exploration plot, then the value of putting them in notebook format goes away, and you’re better off using testing tools that were meant for testing proper modules, than to shoe-horn notebooks into testing with notebook-specific testing tools.

I’d also argue that the benefits of starting out from a craftsmanship-first approach from the beginning, even in exploratory data analysis, has compounding benefits and you quickly reach a state where the extra craftsmanship leads to less time spent debugging, backtracking to understand a plotting error or diagnostic bug, and faster convergence on successful output artifacts, whether it’s a report on model accuracy or production-ready code.

Re: Advanced computing with IPython

#57

Post by joshuamorton »

mlthoughts2018 wrote: post #56
joshuamorton wrote: post #50

Earlier quoted context omitted.

Generally speaking, the code you write in a notebook isn't too important for presentation, the code you write in a notebook creates something to be presented (a chart, an output value, a pandas dataframe). I'll grant that this isn't true 100% of the time, but I find its normally the case. So start with the assumption that you have magic tooling (which I know exists) that allows you to ignore diffs in generated output…

I guess I’m saying the problem with testing or linting a notebook is not technical. Writing a tool that suppresses output cells, infers global parameter blocks, etc., is trivial. Writing a linter with enough configurability to account for notebook presentation styling might be harder, but still straightforward. Creating the raw tools that can do it is the easy part. The hard part is that writing code for re-use and t…

>I’d also argue that the benefits of starting out from a craftsmanship-first approach from the beginning, even in exploratory data analysis, has compounding benefits and you quickly reach a state where the extra craftsmanship leads to less time spent debugging, backtracking to understand a plotting error or diagnostic bug, and faster convergence on successful output artifacts, whether it’s a report on model accuracy or production-ready code

Assuredly, my point is that none of these are incompatible with a notebook-like environment. You can have well crafted, well designed, good code in a notebook, and get the advantages of both craftsmanship and presentation.

Good tooling allows you to focus on the craftsmanship.

>The hard part is that writing code for re-use and testability and with reasonable low-effort best practices at separating concerns and having modularity

These are all hard normally, its just that we have tooling that makes it somewhat less difficult. To be reductive, you're saying "well crafted software is difficult", which I agree with, "and so since we don't have the tooling to make well crafted software as easy in a notebook environment as in the environment we've used for 20-50 years now, we should not use the notebook", which I disagree with since you can also just say "and so we should create the tooling to mature the notebook environment".

Basically, to answer your question:

>So why bother contorting the testing apparatus to accomodate testing something that is created with throw-away design principles from the start?

Don't write notebooks with throw away design principles from the start. Treat them like mature parts of a workflow and in all likelyhood, they'll perform like one. Use good tooling, good design, and good craftsmanship when writing your notebooks (much as you would with any other piece of code you wrote) and they won't be created with "throw-away design principles".

Yes, if you treat notebooks like an unstructured second class citizen you'll get bad results, but that's true of any tool. So don't do that.

Re: Advanced computing with IPython

#58

Post by leephillips »

wenc wrote: post #55
mlthoughts2018 wrote: post #52

Earlier quoted context omitted.

I agree, and the IDE setups can be very valuable for certain use cases or certain preferences. The equivalent thing in the IPython shell approach, basically using souped-up pudb, is not quite as nicely interactive with the debugging cycle, since setting breakpoints, watchpoints, etc., is either a matter of editing them into the source code and re-running, or becoming a master of specifying them on the command line, b…

Yes, my IDE is vim but it's a hard sell to a lot of folks... especially having to map a shortcut key to "import ipdb; ipdb.set_trace()" for breakpoints... Rodeo [1] was an attempt at an IDE but development died, and now that yhat's been acquired, there's no sign of any further development. I wish the Jupyter folks would push more in this direction (and they are with Jupyter Lab) but I get the sense they are really in…

Well, I guess they are invested in it as a component of the JuyterLab toolbox, but JupyterLab tries to integrate it with consoles and editing windows: https://lwn.net/Articles/748937/

Re: Advanced computing with IPython

#59

Post by carreau »

chrxr wrote: post #16
uasnew wrote: post #15

Earlier quoted context omitted.

Is this something that is open source so that other schools could use it?

Most of the implementation is open source. Authentication module is separate as it's part of our Canvas app work, but it will likely be open sourced soon. We've also done implementations which authenticate via GitHub... https://github.com/harvard/cloudJHub

There is a lot of work on the jupyterhub organization that provide custom authenticator (GitHub, laugh) feel free to reach out if you want to migrate your work there. Curious also why the existing GitHub Oauth did not work for you.

Re: Advanced computing with IPython

#60

Post by nightcracker »

wenc wrote: post #22
mlthoughts2018 wrote: post #6

If interested, I spent some time on a comment thread a few days ago describing how my experience leads me to believe the Notebook environment (not all of Jupyter / IPython, just the Notebook part) is actually only appropriate for a tiny subset of pedagogical or throw-away situations, and should be avoided most of the time and avoided in most of the cases it’s marketed for (especially anything having to do with ‘repro…

Agreed. Notebook environments are great for exploration, discovery and pedagogy. They aren't so good for productionizing code. We found this out the hard way when we tried to productionize ML code in Jupyter. We had to export to .py and add boilerplate. This works fine unless there is back and forth iteration between modeling and prod, which there invariably is; our data scientists had to make changes to the notebook…

I solve this in my personal workflow by extracting the important bit to a module, editing in that module, and testing/exploring changes in a notebook by reloading the module.
Post reply on HN