Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

JupyterCon: I don't like Notebooks [slides]

docs.google.com

Post reply on HN
Print view View original

61–70 of 112 posts

Re: JupyterCon: I don't like Notebooks [slides]

#61

Post by de_watcher »

sweezyjeezy wrote: post #34
pleasecalllater wrote: post #27

The code written by a couple of "data scientists" I was working with is the worst code I have ever seen. They don't care, they just want to have an experimental results. The problem starts when their experimental "code" needs to be used on production or they are asked to describe how it works. Why cannot we just get good programmers and train them as data scientists?

I've worked with a few people who are software engineers -> data scientists. They were great at bringing good coding / database practices into the team. That said, their lack of formal statistics training was definitely a problem from time to time, and they seemed to show as much disregard for it as data scientists have for engineering practices (at least the ones talked about on this thread). This can be equally dam…

Dealing with code (including tests and versioning) is on the same level as knowing the basic math notation. Should be embarrassing to not to apply the practices.

The only problem is that these good coding practices aren't that exact, and tend to go on and on all the way to infinity.

Re: JupyterCon: I don't like Notebooks [slides]

#62

Post by Mvandenbergh »

I use notebooks a lot but when I've got a piece of code that I re-use more than once I pull it out into a module or two.

My notebooks are usually set up as: 1) a cell to load external libraries 2) a cell to load any of my own modules 3) a cell to set model parameters 4) a cell containing the functions that I use to load data (to be honest these are not always encapsulated in functions in the early stages of analysis) 5) cells containing functions that do stages of analysis that I'm happy with (but unique to this analysis and therefore not turned into modules). 6) A "main" cell that runs the cells in [5] and gets everything ready for 7) working cell[s] where I mess around with new analysis

While I'm working I only execute from [7] onwards unless I need to reload my data.

This is pretty easy to turn into production if needed because when the analysis is done, you can clean up the working cells and incorporate them into your main function.

Re: JupyterCon: I don't like Notebooks [slides]

#63

Post by [deleted] »

j_4 wrote: post #32

I absolutely love Jupyter as a computer science student who uses it a lot to fiddle with new concepts, create visualizations, or write markdown reports with annotated code. I do agree with some points, though. The problems with state felt sort of overstated (heehee), but it obviously is an issue. I always wonder why the menu option Kernel -> Restart & Run All is not a first class citizen, with a big red button at the…

[deleted]

Re: JupyterCon: I don't like Notebooks [slides]

#64

Post by amirathi »

j_4 wrote: post #32

I absolutely love Jupyter as a computer science student who uses it a lot to fiddle with new concepts, create visualizations, or write markdown reports with annotated code. I do agree with some points, though. The problems with state felt sort of overstated (heehee), but it obviously is an issue. I always wonder why the menu option Kernel -> Restart & Run All is not a first class citizen, with a big red button at the…

> I also wish the notebooks diffed better when it comes to SCM, without all the JSON artifacts.

This is one of my biggest qualms as well. I built a DevOps tool [1] that uses Notebooks & not being able to diff and review Notebooks was a pain. I have decided to solve it with a GitHub marketplace app. Wrote more about it here: https://medium.freecodecamp.org/how-to-handle-version-contro...

[1] https://nurtch.com

Re: JupyterCon: I don't like Notebooks [slides]

#65

Post by bitL »

I use notebooks to test initial model performance, add some meaningful stats, descriptions of parts of computation performed and reasoning behind it, links to external (arxiv/github) papers/sources with detailed method descriptions, visualization of preliminary results of initial computation or hyperparameter search; that is a vital piece of information I provide to my clients who can then make an informed decision which way(s) they should move forward or abort completely. Then a production-ready code is developed outside notebook and productionalized including appropriate services (SaaS, serverless etc.). I found this workflow pretty good in convincing clients as they can play with results early in their decision process.

Re: JupyterCon: I don't like Notebooks [slides]

#68

Post by anonu »

For the longest time I was looking for a "highlight and run" environment for python. I've got Spyder IDE connected via ssh forwarded ports to a python kernel on a remote server.

This avoids much of the annoyances that the slides point out about Notebook.

Re: JupyterCon: I don't like Notebooks [slides]

#69

Post by rwasher »

Great presentation, does anyone know if it was recorded?

The strength that kept me coming back to notebooks was their power at iterating on a problem but I was continuously frustrated at the difficulty of extracting my solution / tracking it in git / collaborating with colleagues etc. Also I didn't enjoy the editor experience from a UX point of view.

I've since started using hydrogen [1], a plugin for Atom which (via a Jupyter kernel) seems to get what I wanted from both worlds - it's just a python file but I get most of the notebook fun!

[1] https://github.com/nteract/hydrogen

Re: JupyterCon: I don't like Notebooks [slides]

#70

Post by megamindbrian2 »

FridgeSeal wrote: post #24

I seriously cannot agree more. Jupyter notebooks were fun to use for a bit, then I hit the inevitable wall of "ok, now let's turn this into a real, properly built script, but now everything is breaking for inexplicable reasons". Notebooks are fine for early stage experimenting, but if you've got to the point where you start up you are relying on a notebook for anything, or your workflow consists of "start up notebook…

I wrote a loader for this. And there are tools to split cells for compiling. I wrote a tool to split up cells too and compile in order. Pretty easy, all the code is in notebooks.
Post reply on HN