Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Advanced computing with IPython

lwn.net

Post reply on HN
Print view View original

61–70 of 108 posts

Re: Advanced computing with IPython

#61

Post by chrxr »

carreau wrote: post #59
chrxr wrote: post #16

Earlier quoted context omitted.

Most of the implementation is open source. Authentication module is separate as it's part of our Canvas app work, but it will likely be open sourced soon. We've also done implementations which authenticate via GitHub... https://github.com/harvard/cloudJHub

There is a lot of work on the jupyterhub organization that provide custom authenticator (GitHub, laugh) feel free to reach out if you want to migrate your work there. Curious also why the existing GitHub Oauth did not work for you.

Yeah, we need to get back in touch with the Jupyter Project folks to see if there is anything we've done that might be useful feeding back into the project in any way. I'm not 100% sure on the Oauth stuff, I am but the lowly PM. I've punted the question over to the dev lead, who may comment further (though not sure if he has an HN account).

Re: Advanced computing with IPython

#62

Post by adamsea »

chrxr wrote: post #11
chrxr wrote: post #10

At Harvard we've built out an infrastructure to allow us to deploy JupyterHub to courses with authentication managed by Canvas. It has allowed us to easily deploy complex set-ups to students so they can do really cool stuff without having to spend hours walking them through setup. Instructors are writing their lectures as IPython notebooks, and distributing them to students, who then work through them in their Jupyte…

It is worth noting that there is an argument that it is a worthwhile task for students to learn how to setup complex computing environments, as it better prepares them for the real world. However, in reality, there just isn't time within a single semester to do this for a class of 100+ students. So implementations such as this one trade-off that learning for a greater focus on computational theory and its implementat…

Depends what that particular course is supposed to teach. Stats, maybe some sort of intro course, or a programming course intended for non-CS majors are all courses where it could make sense to abstract away setting up complex environments -- just like how, in the business world, companies (SAS, etc) make tons of money abstracting away complex environments so businesses can have their employees focus on what provides value in that context.

Re: Advanced computing with IPython

#63

Post by nightski »

sametmax wrote: post #51
da_chicken wrote: post #13

Earlier quoted context omitted.

I'm torn. On the one hand that's really cool to get everything configured and up and running so students can get to the interesting parts. On the other hand, learning how to configure your own environment is kind of an essential part of working with any tool that forces you to understand at least some of the structure involved.

If you have 100 students, you will have 100 different mistakes to debug in the setup. A setup is not a program, there is rarely an easy way to pinpoint a problem, and so it takes a lot of time to setup just one, let alone a 100. When the number of hours are limited, it's best to skip it entirely, and just provide a solid paper tutorial.

I'm so glad my college had class sizes of 10-15 and not 100, what a waste of money.

Re: Advanced computing with IPython

#64

Post by chrxr »

carreau wrote: post #59
chrxr wrote: post #16

Earlier quoted context omitted.

Most of the implementation is open source. Authentication module is separate as it's part of our Canvas app work, but it will likely be open sourced soon. We've also done implementations which authenticate via GitHub... https://github.com/harvard/cloudJHub

There is a lot of work on the jupyterhub organization that provide custom authenticator (GitHub, laugh) feel free to reach out if you want to migrate your work there. Curious also why the existing GitHub Oauth did not work for you.

Dev has no account so relaying this:

"we had an issue with Oauth when we upgraded JupyterHub version 0.7 to version 0.8. The instance spawner we wrote needed to be updated to fix the issue. The case that opened about this in our repository* fixed with the latest update of the instance spawner"

So it seems we could be using GitHub standard OAuth now. But 95% of our implementations utilize Canvas auth reconciling with our university AD.

*https://github.com/harvard/cloudJHub/issues/2

Re: Advanced computing with IPython

#65

Post by semi-extrinsic »

biridir wrote: post #30

The content in the article is a bit dated. Jupyter + Numba + Dask is the direction scientific computing (in Python) is taking. Ipyparallel is not really scalable in my experience.

This is an extremely limited view of "scientific computing" that seems to only focus on analytics, which is a tiiiny part of sci comp.

Your "stack" does nothing for solving/including ODEs, PDEs, DAEs, Fourier analysis, numerical integration, automatic differentiation, linear equation system solvers, preconditioners, nonlinear equation system solvers, the entire field of optimization, inverse problems, statistical methods, Monte Carlo simulations, molecular dynamics, PIC methods, geometric integration, lattice quantum field theory, molecular dynamics, ab initio methods, density functional theory, finite difference/volume/element methods, lattice Boltzmann methods, boundary integral methods, mesh generation methods, error estimation, uncertainty quantification...

Those are just off the top of my head, the list goes on and on.

Re: Advanced computing with IPython

#66

Post by wenc »

nightcracker wrote: post #60
wenc wrote: post #22

Earlier quoted context omitted.

Agreed. Notebook environments are great for exploration, discovery and pedagogy. They aren't so good for productionizing code. We found this out the hard way when we tried to productionize ML code in Jupyter. We had to export to .py and add boilerplate. This works fine unless there is back and forth iteration between modeling and prod, which there invariably is; our data scientists had to make changes to the notebook…

I solve this in my personal workflow by extracting the important bit to a module, editing in that module, and testing/exploring changes in a notebook by reloading the module.

Yes, I tried doing this too but it forces me to flip between the notebook and a separate text editor (for editing that module). It's not that seamless and the context switches were a little expensive (at least for me), but your mileage may vary.

Re: Advanced computing with IPython

#67

Post by pm90 »

chrxr wrote: post #11
chrxr wrote: post #10

At Harvard we've built out an infrastructure to allow us to deploy JupyterHub to courses with authentication managed by Canvas. It has allowed us to easily deploy complex set-ups to students so they can do really cool stuff without having to spend hours walking them through setup. Instructors are writing their lectures as IPython notebooks, and distributing them to students, who then work through them in their Jupyte…

It is worth noting that there is an argument that it is a worthwhile task for students to learn how to setup complex computing environments, as it better prepares them for the real world. However, in reality, there just isn't time within a single semester to do this for a class of 100+ students. So implementations such as this one trade-off that learning for a greater focus on computational theory and its implementat…

Its a valid point to consider that as part of the learning experience. I don't think that students with limited time should be forced to go through that though. If you get them hooked on to computing, their natural curiosity would lead them to explore it further. Bogging them down with unnecessary setup stuff would probably only make them think that this shit takes way too much effort etc.

I've had many courses that were bogged down by software setup issues in college; I would rather than not be the case.

Re: Advanced computing with IPython

#68

Post by xapata »

commandlinefan wrote: post #38
biridir wrote: post #30

The content in the article is a bit dated. Jupyter + Numba + Dask is the direction scientific computing (in Python) is taking. Ipyparallel is not really scalable in my experience.

I thought so, too - enough that I checked the date, but the date on the article is yesterday. Maybe it was written a while ago and just now uploaded. I thought it was strange to refer to IPython as a standalone interpreter outside of Jupyter... I mean, you can, but I don't think anybody does anymore.

> I don't think anybody does anymore

Counter-example: I use IPython in the terminal, outside of Jupyter.

Re: Advanced computing with IPython

#69

Post by Phrodo_00 »

detaro wrote: post #19
Asooka wrote: post #12

Earlier quoted context omitted.

Summary: Jupyter is the evolution of IPython. IPython is deprecated.

Jupyter is the language-agnostic parts of IPython (UI, notebook format, protocols to talk to notebooks, ...) extracted out to be used with many languages. IPython remains as the project maintaining the Python-specific parts of that stack. It's not deprecated, but has been limited in scope.

IPython still makes nicer-repl-for-python. I don't know if it has a name beyond just IPython, which is a bit confusing.

Re: Advanced computing with IPython

#70

Post by leephillips »

xapata wrote: post #68
commandlinefan wrote: post #38

Earlier quoted context omitted.

I thought so, too - enough that I checked the date, but the date on the article is yesterday. Maybe it was written a while ago and just now uploaded. I thought it was strange to refer to IPython as a standalone interpreter outside of Jupyter... I mean, you can, but I don't think anybody does anymore.

> I don't think anybody does anymore Counter-example: I use IPython in the terminal, outside of Jupyter.

So do I, and many others. Nowadays you can type "jupyter console", which gets you almost the same thing; and JupyterLab has console components.
Post reply on HN