111–120 of 223 posts
Co-author of the study here. Let me know if you have any questions or how you overcome some of the problems we identified!
What are your thoughts on the best way to address these things?
Co-author of the study here. Let me know if you have any questions or how you overcome some of the problems we identified!
> Co-author of the study here. Let me know if you have any questions or how you overcome some of the problems we identified! It's not clear who the audience is. It sounds like most people who complain about them are software people and not researchers/scientists. For someone like me, who once did computational research using MATLAB, and later analyzed data for my job, Jupyter is not worse, and is in most ways superio…
Earlier quoted context omitted.
> I think it is a good thing that notebooks separate instructions and re-execute manually by default. The cost of the alternative is just too high. Maybe add a "lock" toggle so a user can block a cell from being automatically executed? The heavy numeric setup tasks could then be gathered in a few cells and locked, leaving the lighter plotting & summary stats cells free to update reactively.
Toggle??? Toggle a whole environment and intrepeter's behavior? Do you know how much architecture that would involve? That's like trying to tell IDLE to be able to both delete or keep your variables on exit, or the JVM to have a toggle switch for memory and garbage management. Why doesn't the developer make themselves useful and simply write a save function that freezes their buffer variable values to a text, json or…
We are still early access, but if you are interested in an invite just let me know. My email is filip at deepnote dot com.
I first saw jupyter notebooks when my sister (physicist, non-programmer) used it for analyzing economical data with pandas. Run-time for the full data set was half a day (and IMHO for that analysis SQL would have been better suited). I understand that as a non-programmer it looks alluring, but once the language proficiency is build up, why not use an IDE and run the code on a shell?
Earlier quoted context omitted.
Doing data science in an IDE would be terrible. With a notebook, you get the chance to load the data, view it, clean it where needed, view it again, analyze it, model it and do anything else you need to it. An IDE means that you can't use the previous output to guide your next operation in a direct fashion like you can with a notebook.
> With a notebook, you get the chance to load the data, view it, clean it where needed, view it again, analyze it, model it and do anything else you need to it. In a good data-oriented IDE like RStudio you get to do all of those things and write code which can be saved as plain text and can be version controlled well under git which you can't do well with Jupyter. R folks have to be the best indicator in this case be…
Notebooks give me some of the interactivity but the experience degrades significantly.
The spyder IDE seem like an okayish replacement but some of the library I use expect you to have html display (within a notebook) to give you full functionalities which is not yet available in spyder.
Co-author of the study here. Let me know if you have any questions or how you overcome some of the problems we identified!
I just wanted to say thank you. Many of the points in your study strikes a nerve. Part of my responsibility at my last job was to introduce good software engineering practices. What happens? The data scientists go rogue and start running notebooks left and right. How do they productionize their work? Well, they don't. They were academics. All they know is that the models ran fine in their notebooks on their laptops.…
Unknowable ad-hoc, unversioned spreadsheets running much of the capital of the company.
The cells can then be individually executed in an ipython shell, or the entire script can be run with the regular python interpreter. This makes it easy to tweak the individual parts without having to re-run everything. In contrast to jupyter notebooks you still end up with a valid python script that can be easily version controlled.
I just wish that I could use vim instead of the Spyder IDE.
Co-author of the study here. Let me know if you have any questions or how you overcome some of the problems we identified!
I just wanted to say thank you. Many of the points in your study strikes a nerve. Part of my responsibility at my last job was to introduce good software engineering practices. What happens? The data scientists go rogue and start running notebooks left and right. How do they productionize their work? Well, they don't. They were academics. All they know is that the models ran fine in their notebooks on their laptops.…
We just store the data tables in the project's database on a Postgres server. Then it's just a matter of pd.read_sql_query()
Earlier quoted context omitted.
My problem with notebooks is that I feel like the natural mental model for them is a spreadsheet mental model, not a REPL mental model. Under that assumption, changing a calculation in the middle means that all of the cells that depend on that calculation would be updated, but instead you need to go and manually re-run the cells after it that depend on that calculation (or re-run the entire notebook) to see the effec…
I keep seeing the complaint about state and I am getting sick of it. JUST HIT "RESTART KERNEL AND RERUN ALL CELLS," HOW HARD CAN IT BE?