121–130 of 153 posts
I'm not even a programmer, I'm a data scientist 1 year out of college, and I still know that notebooks should never ever be deployed to production!
The right way to view notebooks is as.. notebooks. Text and visualizations that can run online code. They are for presenting!
You would need a tremendous amount of incompetence and a lack of any code reviews or tech leadership at all to end up deploying a notebook to production.
I think people may be misusing notebooks. They aren't there to develop software but serve as virtual versions of scientific notebooks (hence the name). They are there to conduct experiments (changing parameters and the like in your code) and to record and plot the results. You don't have to develop the software itself in the notebook.
Good SWE practices are 100% not taught in school.
Add a "why I might be wrong" section, or "why this might not apply to you". Just show me that you've thought from more than one side of the issue and I'll be more receptive to your side.
I messed around with paperspace for a while, but I could not get detectron2 working in it so I went back to google colab. And my ML projects, that I have ambitions to become production apps, are still toys so for now the quick and dirty notebook will have to do. But yes, I prefer to be in my element in a traditional IDE, pushing to git, tabbing between files. I certainly have > 10 years of experience doing that, but not really for ML projects if that makes sense.
Recently, since coming across FastAI and nbdev[0], I've been moving increasingly to a more notebook-centric flow. So far it's been better particularly for the exploration aspect where I've primarily used ptpython in the past (and this isn't anywhere ML-related). I think the idea behind nbdev is pretty neat, but it pushes some practices that I'm not a fan of at all. I want to get to the point where I have a mostly com…
I think people may be misusing notebooks. They aren't there to develop software but serve as virtual versions of scientific notebooks (hence the name). They are there to conduct experiments (changing parameters and the like in your code) and to record and plot the results. You don't have to develop the software itself in the notebook.
Academia is also usually a single person trying to get stuff working so they can publish a paper to get tenure/a job. Good SWE practices are 100% not taught in school.
The article should start with more context, what is a notebook ? I know what it is, but the author is particularly bad at introducing his article.
Earlier quoted context omitted.
I can change parameters in a script. What's the advantage?
Another advantage is when you have very slow code, you can use cells as caches, essentially without having to worry about serialization to disk. This often makes it much easier to interactively explore/develop downstream methods without needing to re-run earlier upstream dependencies. This is especially useful with large datasets. Even if serialization is straightforward, if you have enough data (or the data is remot…
It's also useful when one of your cells goes and queries a slow API for a bunch of data — I do this all the time with Datadog.
Earlier quoted context omitted.
Academia is also usually a single person trying to get stuff working so they can publish a paper to get tenure/a job. Good SWE practices are 100% not taught in school.
I still rely heavily on principles acquired from courses on Software Architecture and PL studies as a part of my CS degree - and I can definitely see a difference in how people organise their code with same tenure but no schooling.