51–60 of 112 posts
I love notebooks! They are an excellent tool, if used judiciously. They are really practical for situations where you want to play around (er even outright work with) principally not with code, but that code's output. You see, I give lots of trainings. Notebooks offer me an excellent way to mix commands, their output, and explanations into a single document with little effort. I'm able to show my students exactly wha…
I love notebooks! They are an excellent tool, if used judiciously. They are really practical for situations where you want to play around (er even outright work with) principally not with code, but that code's output. You see, I give lots of trainings. Notebooks offer me an excellent way to mix commands, their output, and explanations into a single document with little effort. I'm able to show my students exactly wha…
What do you use to version-control your notebook?
The tool is just a detail. My point was rather that they are very much "code", and deserve to be treated as carefully as any other.
My notebooks also get refactored on occasion.
I'd like to highlight that Notebooks do have use in pre-production while your are exploring data subset and trying out your initial hypothesis. But please, don't throw away decades of software development best practices and reliability for mere development convenience.
We are providing machine learning tooling and our CTO published a blog post talking about the same exact issues we keep encountering with dozens of customers. (https://blog.valohai.com/leveling-up-your-ml-code-from-noteb...)
Notebooks were not created as a way to implement and organize software. You can do that with code files. Jupyter Notebook is a presentation software, for demonstrating something to yourself or others. There is hardly anything comparable to build interactive demonstrations. You'd have to implement a (multi-paged?) GUI application, or a web application. Just plain html output may cut it for certain use cases, but still…
His criticism is valid but the main lesson is to use the appropriate tool for the job. He himself uses VSCode + iPython, which is a better tool more generally for the desired use case of code development.
Notebooks were not created as a way to implement and organize software. You can do that with code files. Jupyter Notebook is a presentation software, for demonstrating something to yourself or others. There is hardly anything comparable to build interactive demonstrations. You'd have to implement a (multi-paged?) GUI application, or a web application. Just plain html output may cut it for certain use cases, but still…
But the problem OP staes is that notebooks encourages unreproducible demonstrations. Markdown + code cells makes data scientists to think a single notebook file is enough to upload to GitHub. Missing dependancy information and etc as OP states make others very hard to reproduce the notebooks. You can see tons of GitHub repos that have notebooks only and impossible to reproduce out of the box.
Notebooks were not created as a way to implement and organize software. You can do that with code files. Jupyter Notebook is a presentation software, for demonstrating something to yourself or others. There is hardly anything comparable to build interactive demonstrations. You'd have to implement a (multi-paged?) GUI application, or a web application. Just plain html output may cut it for certain use cases, but still…
Exactly - it's a classic case of a tool being used for a different purpose ("IDE") than its original intention ("presentation software") and getting criticized for not holding up to misguided expectations. His criticism is valid but the main lesson is to use the appropriate tool for the job. He himself uses VSCode + iPython, which is a better tool more generally for the desired use case of code development.
Is anyone working on bridging the gap between notebook sketches and production ready code?
The code written by a couple of "data scientists" I was working with is the worst code I have ever seen. They don't care, they just want to have an experimental results. The problem starts when their experimental "code" needs to be used on production or they are asked to describe how it works. Why cannot we just get good programmers and train them as data scientists?
I work in data science (came in via a maths background) but I agree 1000%. My software engineering is pretty self taught, but I make an effort to follow best coding practices at all times. That said, I think it goes both ways: people from the sciences tend to be cavalier coders, and people from software background tend to be cavalier about the underlying mathematics. Seems to me that the solution needs to be a strong…
My main point is that too many people in data science don't care at all. They don't care about the repeatable results, about the code quality, even about units (they can even use `mb`, `Mb` and `MB` for megabytes in the same document).
I hope companies will learn that it's really important that if you have code it should be a good quality code, not a randomly gathered set of lines.
Play with your code in the notebook, when you know what the right way to split it is, turn cells into functions. When you have a decent body of functions turn them into a module.
Notebooks excel as ad hoc interfaces to libraries and modules. They don't replace them.
I'm not sure the problems with scientists' code are enabled by notebooks to a significant degree. We used to exclusively use PyCharm and I was still debugging spaghetti scripts with massive global state interacting in weird ways.
Notebooks give people more foot-guns with hidden state. I think they are neutral when it comes to modularity. And I think they are a positive for encouraging documentation.