Advanced computing with IPython • Hacker News Board

Re: Advanced computing with IPython

#21

Post by taeric » Tue, Jun 05, 2018, 1:39 PM UTC

bitexploder wrote: post #8

Don't overlook all of the % commands, such as %edit. If you are familiar with emacs keybindings, it has a very good built in editor as well. You can also load snippets from saved files, and save your history to a file. Or individual lines to files using range type expressions. In short it is very easy to get code in and out of IPython. Another great trick. Anywhere you want to debug or play in your scripts run `impor…

I'm not sure I would compare it favorably to emacs in pretty much anything other than mindshare. And, don't get me wrong, that is huge and not intended as a vanity comment.

The keyboard shortcuts it has are superficial. My number one shortcuts in emacs are compile, jump to next error, grep/occur, index, and magic. Just up and down? Obviously u use them a lot, but the arrow keys do work fine. Beginning of line and begging of text are both huge. But really, the screwy tab behavior kills me in Jupyter.

Re: Advanced computing with IPython

#22

Post by wenc » Tue, Jun 05, 2018, 1:41 PM UTC

mlthoughts2018 wrote: post #6

If interested, I spent some time on a comment thread a few days ago describing how my experience leads me to believe the Notebook environment (not all of Jupyter / IPython, just the Notebook part) is actually only appropriate for a tiny subset of pedagogical or throw-away situations, and should be avoided most of the time and avoided in most of the cases it’s marketed for (especially anything having to do with ‘repro…

Agreed. Notebook environments are great for exploration, discovery and pedagogy. They aren't so good for productionizing code.

We found this out the hard way when we tried to productionize ML code in Jupyter. We had to export to .py and add boilerplate. This works fine unless there is back and forth iteration between modeling and prod, which there invariably is; our data scientists had to make changes to the notebook and we had to redo our boilerplate, so the notebook code and production code were constantly out of sync. This could have been alleviated with automation -- but such automation is bespoke and hard to generalize.

PyCharm has a Scientific Mode (similar to RStudio's IDE approach, where you are actually writing code in a text file but are able to statefully/interactively run code by pressing Ctrl-Enter on code blocks). Spyder, Matlab and a bunch of other IDEs implement this idea too.

Unlike notebooks, this is, I feel, a good middle ground between interactive exploration and having production-ready code.

Re: Advanced computing with IPython

#23

Post by da_chicken » Tue, Jun 05, 2018, 1:41 PM UTC

chrxr wrote: post #14

da_chicken wrote: post #13

Earlier quoted context omitted.

I'm torn. On the one hand that's really cool to get everything configured and up and running so students can get to the interesting parts. On the other hand, learning how to configure your own environment is kind of an essential part of working with any tool that forces you to understand at least some of the structure involved.

That is a fair point. Please see my reply to my post.

Ah, that wasn't there when I replied.

I do understand the dilemma. I work at a K-12 and the office next to mine is where they put together the science lab kits for students. It takes a fair bit of understanding to do that correctly sometimes, and that preparation work is some knowledge the students seem to miss out on in order to get to the subject matter. My coworker has mentioned on more than one occasion that with certain modules it feels like she does most of the work and the students just do the final step.

Re: Advanced computing with IPython

#24

Post by [deleted] » Tue, Jun 05, 2018, 1:42 PM UTC

chrxr wrote: post #10

At Harvard we've built out an infrastructure to allow us to deploy JupyterHub to courses with authentication managed by Canvas. It has allowed us to easily deploy complex set-ups to students so they can do really cool stuff without having to spend hours walking them through setup. Instructors are writing their lectures as IPython notebooks, and distributing them to students, who then work through them in their Jupyte…

[deleted]

Re: Advanced computing with IPython

#25

Post by cirgue » Tue, Jun 05, 2018, 1:45 PM UTC

chrxr wrote: post #11

chrxr wrote: post #10

At Harvard we've built out an infrastructure to allow us to deploy JupyterHub to courses with authentication managed by Canvas. It has allowed us to easily deploy complex set-ups to students so they can do really cool stuff without having to spend hours walking them through setup. Instructors are writing their lectures as IPython notebooks, and distributing them to students, who then work through them in their Jupyte…

It is worth noting that there is an argument that it is a worthwhile task for students to learn how to setup complex computing environments, as it better prepares them for the real world. However, in reality, there just isn't time within a single semester to do this for a class of 100+ students. So implementations such as this one trade-off that learning for a greater focus on computational theory and its implementat…

It’s a good experience, but a motivated student who’s good at charging through docs can do it on their own. Having to manage that for a class that’s intended to teach conceptual material would be a big time-sink.

Re: Advanced computing with IPython

#26

Post by na85 » Tue, Jun 05, 2018, 1:47 PM UTC

bunderbunder wrote: post #17

mlthoughts2018 wrote: post #6

If interested, I spent some time on a comment thread a few days ago describing how my experience leads me to believe the Notebook environment (not all of Jupyter / IPython, just the Notebook part) is actually only appropriate for a tiny subset of pedagogical or throw-away situations, and should be avoided most of the time and avoided in most of the cases it’s marketed for (especially anything having to do with ‘repro…

You're not wrong, though I think the value of pedagogical and throw-away situations in your work may not be a universal experience. For me, at least, pedagogic and throw-away situations aren't a tiny subset. They're most of what I do. It's exploratory work, figuring out how the data behaves, if the data behaves, where it needs to be cleaned, churning through great heaps of experiments and iterations before hitting on…

What are you working in at that later part of the job?

Re: Advanced computing with IPython

#27

Post by chrxr » Tue, Jun 05, 2018, 1:47 PM UTC

da_chicken wrote: post #23

chrxr wrote: post #14

Earlier quoted context omitted.

That is a fair point. Please see my reply to my post.

Ah, that wasn't there when I replied. I do understand the dilemma. I work at a K-12 and the office next to mine is where they put together the science lab kits for students. It takes a fair bit of understanding to do that correctly sometimes, and that preparation work is some knowledge the students seem to miss out on in order to get to the subject matter. My coworker has mentioned on more than one occasion that with…

I can appreciate that feeling; my father was an electronics teacher in secondary school in the UK.

It is true that we had to deal with some issues that might not have occurred had students gone through the process of setting up the environment themselves, like have to rebuild the machine of the student who uninstalled Cuda.

Re: Advanced computing with IPython

#28

Post by wenc » Tue, Jun 05, 2018, 1:52 PM UTC

bunderbunder wrote: post #17

mlthoughts2018 wrote: post #6

If interested, I spent some time on a comment thread a few days ago describing how my experience leads me to believe the Notebook environment (not all of Jupyter / IPython, just the Notebook part) is actually only appropriate for a tiny subset of pedagogical or throw-away situations, and should be avoided most of the time and avoided in most of the cases it’s marketed for (especially anything having to do with ‘repro…

You're not wrong, though I think the value of pedagogical and throw-away situations in your work may not be a universal experience. For me, at least, pedagogic and throw-away situations aren't a tiny subset. They're most of what I do. It's exploratory work, figuring out how the data behaves, if the data behaves, where it needs to be cleaned, churning through great heaps of experiments and iterations before hitting on…

> At which point, forget Jupyter notebooks, I'm typically not even working in Python anymore for that part of the job.

This is what is typically done out there but I suggest it breaks the feedback loop between the scientist roles and the developer roles. In rapidly changing environments those feedback loops could be crucial.

It's similar to what Wall Street folks did (still do?)--quants write models in Excel/VBA and pass them over to developers who would rewrite them in Java for production. There's a natural impedance mismatch, and back-and-forths are difficult.

I think a better approach would be for data scientists to write somewhat production-ready code, send it to prod (with the help of devs), get feedback from the production environment as well as get a sense of what tricks are needed for prod, and then iterate on that code. It also helps to remove the insulation between data scientists and the real world.

Re: Advanced computing with IPython

#29

Post by w0m » Tue, Jun 05, 2018, 1:59 PM UTC

chrxr wrote: post #11

chrxr wrote: post #10

At Harvard we've built out an infrastructure to allow us to deploy JupyterHub to courses with authentication managed by Canvas. It has allowed us to easily deploy complex set-ups to students so they can do really cool stuff without having to spend hours walking them through setup. Instructors are writing their lectures as IPython notebooks, and distributing them to students, who then work through them in their Jupyte…

It is worth noting that there is an argument that it is a worthwhile task for students to learn how to setup complex computing environments, as it better prepares them for the real world. However, in reality, there just isn't time within a single semester to do this for a class of 100+ students. So implementations such as this one trade-off that learning for a greater focus on computational theory and its implementat…

Agreed. At the beginning of class, walk students through the setup. Then for every project after, let them use the pre-rolled systems.

Re: Advanced computing with IPython

#30

Post by biridir » Tue, Jun 05, 2018, 2:07 PM UTC

The content in the article is a bit dated. Jupyter + Numba + Dask is the direction scientific computing (in Python) is taking. Ipyparallel is not really scalable in my experience.