191–200 of 223 posts
The FastAI people have been working on a lot of these issues with their NBDev too: https://www.fast.ai/2019/12/02/nbdev/
I don't know what this document is meant to do but you will have to take my Jupyter-lab instance from my dead cold hands. I love notebooks, I work fast, line by line I execute commands and I immediately see the output (dataframes or graphs). For complex code I have an editor open (in jupyter-lab or vscode) for some functions and classes. But the main developing is done in the notebook, anything that ends in a module…
Earlier quoted context omitted.
I just wanted to say thank you. Many of the points in your study strikes a nerve. Part of my responsibility at my last job was to introduce good software engineering practices. What happens? The data scientists go rogue and start running notebooks left and right. How do they productionize their work? Well, they don't. They were academics. All they know is that the models ran fine in their notebooks on their laptops.…
>to introduce good software engineering practices. What happens? The data scientists go rogue and start running notebooks left and right. How do they productionize their work? Well, they don't. They were academics. My background is programming (instead of data analysis & modeling) so I'm sympathetic to your idealistic "software engineering" view... but I'm also sympathetic to the academics' side as explained by Yihui…
Earlier quoted context omitted.
The tone of what you are saying strikes a nerve with me - we had exactly the same issues with Excel in the front office in investment banking. Unknowable ad-hoc, unversioned spreadsheets running much of the capital of the company.
That’s a really good comparison. Excel is often used for storing data and doing analysis because it just plain works. And anyone can use it. Notebooks tend to be the same way. It’s a simple GUI-ish was to do many complex analyses in a quick and dirty way. And many of the arguments for not using Excel are the same as not using notebooks. Each is good at the initial data exploration stage, but are often abused and used…
Earlier quoted context omitted.
That’s a really good comparison. Excel is often used for storing data and doing analysis because it just plain works. And anyone can use it. Notebooks tend to be the same way. It’s a simple GUI-ish was to do many complex analyses in a quick and dirty way. And many of the arguments for not using Excel are the same as not using notebooks. Each is good at the initial data exploration stage, but are often abused and used…
Think of all the damage caused by excel. We replaced one set of avoidable catastrophes for another. But this time there’s no shame.
(For what it's worth, I feel the same way about people who try to send me RDS files with dataframes stored as R objects).
However, I think that whoever decided to name genes "OCT4" and "SEPT7" have to share some of the blame here too...
Earlier quoted context omitted.
> Co-author of the study here. Let me know if you have any questions or how you overcome some of the problems we identified! It's not clear who the audience is. It sounds like most people who complain about them are software people and not researchers/scientists. For someone like me, who once did computational research using MATLAB, and later analyzed data for my job, Jupyter is not worse, and is in most ways superio…
> I honestly have not seen this, and the reason makes no sense. Your browser is not handling the data. The kernel is. I mean yes, if you try to load several GB of data in pandas, it's possible you will have problems if you run out of RAM, but this has nothing to do with notebooks. This is true nowadays with Jupyter, because it is smart about truncating output. But it used to be possible to OOM the browser by e.g. pri…
But I guess if you're manually printing in a for-loop, then I suppose you could make the browser crash - whereas tools like MATLAB wouldn't.
Co-author of the study here. Let me know if you have any questions or how you overcome some of the problems we identified!
I just wanted to say thank you. Many of the points in your study strikes a nerve. Part of my responsibility at my last job was to introduce good software engineering practices. What happens? The data scientists go rogue and start running notebooks left and right. How do they productionize their work? Well, they don't. They were academics. All they know is that the models ran fine in their notebooks on their laptops.…
I am very happy to be out of that business.
Co-author of the study here. Let me know if you have any questions or how you overcome some of the problems we identified!
> Co-author of the study here. Let me know if you have any questions or how you overcome some of the problems we identified! It's not clear who the audience is. It sounds like most people who complain about them are software people and not researchers/scientists. For someone like me, who once did computational research using MATLAB, and later analyzed data for my job, Jupyter is not worse, and is in most ways superio…
People who design and implement features in notebooks. The conclusions in the blog post and research paper are clear that improving these identified problems could improve user experience.
Earlier quoted context omitted.
I just wanted to say thank you. Many of the points in your study strikes a nerve. Part of my responsibility at my last job was to introduce good software engineering practices. What happens? The data scientists go rogue and start running notebooks left and right. How do they productionize their work? Well, they don't. They were academics. All they know is that the models ran fine in their notebooks on their laptops.…
>to introduce good software engineering practices. What happens? The data scientists go rogue and start running notebooks left and right. How do they productionize their work? Well, they don't. They were academics. My background is programming (instead of data analysis & modeling) so I'm sympathetic to your idealistic "software engineering" view... but I'm also sympathetic to the academics' side as explained by Yihui…
I think this is very much off the mark. For sure plenty of scientists are poor programmers, but that isn't the reason they use notebooks. It is because:
They are not attempting to write something that will run everywhere, and often. They are either analyzing some data or doing rapid prototyping. For the latter, it's like criticizing someone who uses a REPL. It's just that the Notebook is much more powerful than a simple REPL that one can safely stick to it. Imagine you will do 40-50 prototypes and only one of those may end up worthy enough to make a product out of, and you don't know which one that will be. If you used a non-notebook environment, you'd give up in frustration by the time you hit the 15th one.
As you said: At the moment, there simply isn't an alternative that allows for rapid prototyping and is production ready. It's a hard problem to solve - there's a reason no one had solved it for decades (well before notebooks were a thing).
Had notebooks not been invented, you would have the same people handing you MATLAB code asking you to productize it.
Claiming they are beginners/novice programmers is off the mark. Peter Norvig started using notebooks for a reason, and no one would call him a novice. I do SW for a living, but when I need to analyze data and visualize it, I'll pick a notebook over "proper" SW tools any day.