Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Visual Debugger for Jupyter

blog.jupyter.org

Post reply on HN
Print view View original

61–70 of 74 posts

Re: A Visual Debugger for Jupyter

#61

Post by zmmmmm »

Myrmornis wrote: post #26

I'm a programmer, mostly not a data scientist nowadays, mostly working with Python. I have tried Jupyter Lab/Notebook on and off over the last 10 years, and I believe I have now firmly settled on my conclusion: Everyone should aim to minimize the amount of work they do in Jupyter Lab / Notebook. It shocks me a bit to find myself saying that, as it is such a beautiful piece of work. Furthermore the people who wrote it…

As someone who flips back and forward b/w data science and software engineering quite a lot, I think they are highly complementary. The benefit of things like Jupyter is the ability to thoroughly explore a problem space without requiring the overhead of things like version control. The way I see it, every piece of code written actually has such an exploratory phase where you try out a few different approaches, test your assumptions, often realise a mistake half way through and rework your algorithm etc.

The danger when that is done in an IDE setting is that because you build up commitment in a solution as you go, a reluctance sets in to rework it. So the final output is nothing like what you would write if you did it from scratch - it's littered with historical quirks of how you arrived at that implementation.

So I actually think that breaking out of the IDE and doing an exploratory phase in something like Jupyter is a really useful way to get your ideas into reasonable shape before you write your "real" code.

Re: A Visual Debugger for Jupyter

#62

Post by Myrmornis »

zmmmmm wrote: post #61
Myrmornis wrote: post #26

I'm a programmer, mostly not a data scientist nowadays, mostly working with Python. I have tried Jupyter Lab/Notebook on and off over the last 10 years, and I believe I have now firmly settled on my conclusion: Everyone should aim to minimize the amount of work they do in Jupyter Lab / Notebook. It shocks me a bit to find myself saying that, as it is such a beautiful piece of work. Furthermore the people who wrote it…

As someone who flips back and forward b/w data science and software engineering quite a lot, I think they are highly complementary. The benefit of things like Jupyter is the ability to thoroughly explore a problem space without requiring the overhead of things like version control. The way I see it, every piece of code written actually has such an exploratory phase where you try out a few different approaches, test y…

With the greatest possible respect, I suspect that you are still on the path towards getting really comfortable with git. I hope this doesn't come across as arrogant or presumptuous. What I suspect, is that you are at a stage which everyone passes through, where the act of "making a commit" feels permanent. You know that there are ways to change history, but it feels like they are going to be a huge distraction from getting work done. It really doesn't help that git uses the word "commit"! I sort of wish they'd used the word "snapshot" or something. In any case, if I am right, what you'll come to see soon is that it really isn't painful to rework commits and there is nothing constraining about git. Just create a copy of your current branch first if you're at all worried about messing something up (git checkout -b mybranch-snapshot-1), and then `git reset $commit_before_your_experimental_throwaway_commits`. There's no need to do anything more complicated than that when it's just your private work on your laptop, that you haven't pushed to a shared remote.

There were a couple of things you wrote that make me think that. Firstly: "without requiring the overhead of things like version control". There is a miniscule overhead to using git in the way I describe: `git init`, `git add`, `git commit`, `git reset` are the only commands you need and they take a second to invoke. Secondly "you build up commitment in a solution as you go": as i said above, I believe that as one gets more comfortable with git, it no longer feels a constraint -- quite the opposite, you feel liberated to experiment because you always know you can get back to any state you wish.

Re: A Visual Debugger for Jupyter

#63

Post by nnq »

Myrmornis wrote: post #26

I'm a programmer, mostly not a data scientist nowadays, mostly working with Python. I have tried Jupyter Lab/Notebook on and off over the last 10 years, and I believe I have now firmly settled on my conclusion: Everyone should aim to minimize the amount of work they do in Jupyter Lab / Notebook. It shocks me a bit to find myself saying that, as it is such a beautiful piece of work. Furthermore the people who wrote it…

I'm a software engineer and I disagree:

I very often use jupyter to "pop open a shell intro a production service and start interactively debugging stuff live" (over an ssh tunnel, no public open ports and other security considerations, mind you).

It's amazing the feel you get the first time when you open a notebook that acts like a live REPL to smth. like a Django app and you start investigating and trying out stuff by just stitching snippets of code together in a notebook that imports your app and uses its db!

Now I code all API services regardless of tech they use so I can easily "pop open a jupyter REPL into a running system importing app code and running it agains its db".

Interactivity and REPL-driven-development-and-debugging is awesome if you have good discipline to contain the chaos and keep your notebooks aggressively short-lived (any useful code will be refactored and copied into its place in the regular codebase, most notebooks get deleted before merging a branch into dev/master).

Re: A Visual Debugger for Jupyter

#64

Post by Myrmornis »

nnq wrote: post #63
Myrmornis wrote: post #26

I'm a programmer, mostly not a data scientist nowadays, mostly working with Python. I have tried Jupyter Lab/Notebook on and off over the last 10 years, and I believe I have now firmly settled on my conclusion: Everyone should aim to minimize the amount of work they do in Jupyter Lab / Notebook. It shocks me a bit to find myself saying that, as it is such a beautiful piece of work. Furthermore the people who wrote it…

I'm a software engineer and I disagree: I very often use jupyter to "pop open a shell intro a production service and start interactively debugging stuff live" (over an ssh tunnel, no public open ports and other security considerations, mind you). It's amazing the feel you get the first time when you open a notebook that acts like a live REPL to smth. like a Django app and you start investigating and trying out stuff…

Yes, absolutely. What you describe is Django's `manage.py shell_plus`, which does not use a notebook.

So not to be argumentative, but to be clear about this discussion, I'm going to say that your comment is 70% irrelevant, since using an interactive REPL is routine in python development.

However, it is 30% relevant, because retrieving and archiving the code you ran is going to be much more convenient in a notebook than by using %history or whatever in a shell-based ipython.

Re: A Visual Debugger for Jupyter

#65

Post by mrwebmaster »

Myrmornis wrote: post #26

I'm a programmer, mostly not a data scientist nowadays, mostly working with Python. I have tried Jupyter Lab/Notebook on and off over the last 10 years, and I believe I have now firmly settled on my conclusion: Everyone should aim to minimize the amount of work they do in Jupyter Lab / Notebook. It shocks me a bit to find myself saying that, as it is such a beautiful piece of work. Furthermore the people who wrote it…

Thanks! I'm learning OOP and data analysis. For OOP I use Pycharm and for data analysis I use Jupyter Notebooks (in Lab or VS Code). Sometimes I write some reusable code in .py modules and call them from my notebooks. Anyway, I didn't fully undertand your proposal to use importlib.reload but will try to research about it. For exploratory data analysis, I guess notebooks are better that IDEs.

Re: A Visual Debugger for Jupyter

#66

Post by afshin »

dorfsmay wrote: post #36
Myrmornis wrote: post #26

I'm a programmer, mostly not a data scientist nowadays, mostly working with Python. I have tried Jupyter Lab/Notebook on and off over the last 10 years, and I believe I have now firmly settled on my conclusion: Everyone should aim to minimize the amount of work they do in Jupyter Lab / Notebook. It shocks me a bit to find myself saying that, as it is such a beautiful piece of work. Furthermore the people who wrote it…

Are there projects to integrate notebooks with version control?

There are a few efforts on this front. Here are two that I know of for JupyterLab:

https://github.com/jupyterlab/jupyterlab-git

https://github.com/elyra-ai/elyra#notebook-versioning-based-...

Re: A Visual Debugger for Jupyter

#67

Post by xkriva11 »

mutant wrote: post #21
arnsholt wrote: post #17

This looks nice and all, but on the other hand it's kind of depressing how people are expending all this effort on Jupyter and surrounding stuff, just to make something that almost, but not quite as good as Smalltalk.

Would you expand on this? I don't know smalltalk, and I'm interested in what you see as better?

It uses the fusion of the IDE and the program you develop. You have vast possibilities of objects visualizations (inside a debugger or in separate windows), you can select some part of the code inside the debugger and let run it in separate debugger etc. See https://pharo.org/features

Re: A Visual Debugger for Jupyter

#68

Post by zmmmmm »

Myrmornis wrote: post #62
zmmmmm wrote: post #61

Earlier quoted context omitted.

As someone who flips back and forward b/w data science and software engineering quite a lot, I think they are highly complementary. The benefit of things like Jupyter is the ability to thoroughly explore a problem space without requiring the overhead of things like version control. The way I see it, every piece of code written actually has such an exploratory phase where you try out a few different approaches, test y…

With the greatest possible respect, I suspect that you are still on the path towards getting really comfortable with git. I hope this doesn't come across as arrogant or presumptuous. What I suspect, is that you are at a stage which everyone passes through, where the act of "making a commit" feels permanent. You know that there are ways to change history, but it feels like they are going to be a huge distraction from…

Thanks ... I take your point .... but it's not really about being comfortable with git. It's more about what kind of activity you are doing. When I'm an exploratory process with Jupyter it's iterative with a feedback cycle that is almost subsecond. I often have 3 - 4 versions of the algorithm I am exploring visible at the same time in different cells. I'm using autocomplete and interactive evaluation continuously to understand what state the algorithm is in and what attributes and methods are available to me and how they behave.

No amount of git or anything else an IDE can do achieves those things.

Re: A Visual Debugger for Jupyter

#69

Post by Myrmornis »

mrwebmaster wrote: post #65
Myrmornis wrote: post #26

I'm a programmer, mostly not a data scientist nowadays, mostly working with Python. I have tried Jupyter Lab/Notebook on and off over the last 10 years, and I believe I have now firmly settled on my conclusion: Everyone should aim to minimize the amount of work they do in Jupyter Lab / Notebook. It shocks me a bit to find myself saying that, as it is such a beautiful piece of work. Furthermore the people who wrote it…

Thanks! I'm learning OOP and data analysis. For OOP I use Pycharm and for data analysis I use Jupyter Notebooks (in Lab or VS Code). Sometimes I write some reusable code in .py modules and call them from my notebooks. Anyway, I didn't fully undertand your proposal to use importlib.reload but will try to research about it. For exploratory data analysis, I guess notebooks are better that IDEs.

Hi, I'm happy to try to help. At its most basic, don't copy paste your code into the notebook! And don't import your code like this `from mymodule import myfunction`. Instead import it like this:

  import mymodule

  mymodule.myfunction()
That allows you to do this to reload your module and pick up the updates to myfunction that you've made:

  from importlib import reload
  reload(mymodule)
However, how do you ensure that python can find your code so that `import mymodule` even works? Don't mess about with PYTHONPATH and sys.path. What you really want to do is house your work in its own python package. So, the milestones you want to get to are (not implying you don't already do these things!):

- Always use a virtualenv when working with python

- Create proper package structure for your python project. This means your directory structure will look like this

  myproject/myproject/__init__.py
  myproject/myproject/mymodule.py
  myproject/setup.py
- Google for how to create a minimal setup.py. Just put what you need in there, it's not much.

- Now, with your virtualenv activated, so that `which pip` resolves to `myvirtualenv/bin/pip`, do this:

  cd myproject
  pip install -e .
- That pip command will execute your setup.py and "install" your library into the virtualenv. But it will install it in such a way that you can edit the code and the edits will be picked up by the installed version (it uses symlinks).

- Now install jupyter in that same virtualenv and start your notebook. You should now be able to do `from myproject import mymodule` and `reload(mymodule)`. And your project is now a real python library so you can create subdirectories, etc e.g. `from myproject/plots import create_boxplot`.

Re: A Visual Debugger for Jupyter

#70

Post by Myrmornis »

Myrmornis wrote: post #64
nnq wrote: post #63

Earlier quoted context omitted.

I'm a software engineer and I disagree: I very often use jupyter to "pop open a shell intro a production service and start interactively debugging stuff live" (over an ssh tunnel, no public open ports and other security considerations, mind you). It's amazing the feel you get the first time when you open a notebook that acts like a live REPL to smth. like a Django app and you start investigating and trying out stuff…

Yes, absolutely. What you describe is Django's `manage.py shell_plus`, which does not use a notebook. So not to be argumentative, but to be clear about this discussion, I'm going to say that your comment is 70% irrelevant, since using an interactive REPL is routine in python development. However, it is 30% relevant, because retrieving and archiving the code you ran is going to be much more convenient in a notebook th…

On reflection, I'm not even sure that the notebook is that much of an advantage over shell-based ipython for archiving the commands. One problem is with a notebook, you have less idea what you've actually executed: you just have a bunch of code sitting in cells in a web app. Whereas the shell UX is extremely simple/linear: if a command is in ipython history, then you executed it.
Post reply on HN