Would this work as my main IDE?
JupyterLab 4.0
91–100 of 199 posts
Re: JupyterLab 4.0
#92Earlier quoted context omitted.
For a while I viewed Jupyter as a toy that is neither here nor there (sitting between the chairs of development and explanation, briefing or visualization and not doing either job great). But about 2 years ago when I changed jobs into a "Jupyter heavy" environment I was forced to learn it and have grown to really like it. I primarily use Jupyter for prototyping: trying ideas, plotting results and sharing notebooks fo…
Are there some usecases where conventional IDE isn't up to the job as the notebooks? I can imagine this to be so for data analysis where pre-loading heavy datasets saves time. Anything else?
For me its the matter of actually liking The notebook environment rather than overcoming the shortconings of IDEs.
Re: JupyterLab 4.0
#93I keep experimenting with Jupyter in the context of telemetry/fault analysis and then hitting a wall with it where: - I get an analysis that I like, but there isn't a good way to share it with others, so I end up just taking screenshots. - There isn't a good way to take the same analysis and plug new data into it, other than to copy-paste the entire notebook. - The process to "promote" fragments of a notebook into be…
> There aren't good boundaries between Jupyter's own Python environment, and that of your notebooks— if you have a dependency which conflicts with one of Jupyter's dependencies, then good luck. It's cumbersome, and I'm not totally sure it's the correct way, but I remember getting around this by creating a virtualenv for my projects and then using that virtualenv's python as Jupyter's "kernel".
Right. I usually `pipx install jupyterlab` and register each project's env with `python -m ipykernel install --user --name MyProject --display-name "My Project"` as per the ipython docs: https://ipython.readthedocs.io/en/stable/install/kernel_inst...
Re: JupyterLab 4.0
#94Earlier quoted context omitted.
Do you have a way to store data directly into the notebook? I do a bunch of device testing and I have master notebooks setup to analyze and condense raw data. I use papermill to evaluate the master notebook to generate a report. But I want to also store/attach intermediates (say a named numpy array) into the notebook for further analysis.
The problem with that is that Jupyter requires a back-end to do any of the actual processing that runs the code in the cells. Only way I can think to make the html file truly portable while fully functional would be to embed a python interpreter and all required libraries as wasm. Could be possible with pyodide? I haven't used it.
I just want selected/declared "final results" to be restorable in a new kernel. Papermill let's you easily parameterize notebooks and execute them. So the notebook feels like a well documented functions that behave like executibles with really fancy debug print logs that can be reviewed for correctness and reproducibility.
Papermill does a great job of recording the input parameters and execution history. But there isn't anything equivalent to the "return foo, bar" part of a function which makes it difficult to build up modules. You don't want to have to digitize a plot to carry on to the next step is what I'm saying.
Re: JupyterLab 4.0
#95Earlier quoted context omitted.
Eh, print/log debugging works fine. Especially in an interactive environment: you've got direct access to the variables and objects, and can easily inspect them directly. At some point I felt like I was a bad dev for not using a debugger, but at this point I think I'm more versatile since I'm less dependent on finicky tooling to figure out what some code is doing... Every language has it's own debugger to learn, but…
Debugging nested dicts and high-dim arrays is a nightmare using print
in python you can turn nested dictionaries and other data structures into json, but only if the data structures doesn't include circular references. I use that a lot.
something like
>>> d = { "a":1, "b":[1,2,3] }
>>> import json
>>> print( json.dumps(d, indent=2) )
{
"a": 1,
"b": [
1,
2,
3,
]
}Re: JupyterLab 4.0
#96Anyone have experience with JetBrain's Data Spell product?
Good enough for me and, clients love it.
Re: JupyterLab 4.0
#97I have been using VSCode notebooks with .ipynb file extensions, this gives me many advantages as I am able to configure things I'm not able in JupyterLab. I also have access to a very rich ecosystem of plugins. If there is anyone aware of VSCode as a solution but keeps using JupyterLab, could they explain why?
I've not been back to full-fledged Jupyter since getting in to VSCode. Most of my analytical work now is done in .py files, broken up into blocks with `#%%`. Real notebooks feel really clunky since adopting the approach.
I wish more data scientists used light percent format notebooks `#%%`. It can be combined now with other powerful tools (linting, formatting and git) that is impossible with the `ipynb` format
Re: JupyterLab 4.0
#98Earlier quoted context omitted.
One thing that I keep running into myself is I want to include data in a notebook as a sort of report or record of an analysis. I really like papermill for creating notebooks that execute and then store results. But you can only include text output or plots. If I wanted to store a numpy array within the notebook for inspection or input into.a next step, there doesn't seem to be a way. I understand it could be difficu…
Ploomber does this kind of proper jupyter notebook pipeline. I've used it, but not stuck with it (yet)
I'll have to check it out. I have looked at it previously but it's somewhat overwhelming to figure out what it actually does. My sense previously was that is it was more like a dependency graph thing similar to a makefile but I'll give it a closer look.
Re: JupyterLab 4.0
#99Earlier quoted context omitted.
> I want to also store/attach intermediates (say a named numpy array) into the notebook for further analysis. Is the issue that you do not want to save the data and report into a folder and distribute that? That is, you want an entirely self-contained notebook? Or is there something else going on here? I'm sure that's possible but it seems kind of wrong to put your binary "data" in with your analysis and presentation…
Somewhat. I want it to be difficult to separate the data from the report. Basically I want the report itself to be ingestible as input to other steps, so it's more of a "documented data" with the analysis results available. The inputs are documented but the final results can be restored without reevaluating the entire notebook. I don't want the entire workspace saved, just the final results. I hoped there was some ma…
It looks like there’s a %store command in Jupyter. I haven’t tried it out, but is this what you are looking for?
https://stackoverflow.com/questions/34342155/how-to-pickle-o...
(I just got it by googling “pickle an object in jupyter,” so sorry if this is something obvious that you’ve already seen and doesn’t quite solve your problem).
Re: JupyterLab 4.0
#100Earlier quoted context omitted.
Somewhat. I want it to be difficult to separate the data from the report. Basically I want the report itself to be ingestible as input to other steps, so it's more of a "documented data" with the analysis results available. The inputs are documented but the final results can be restored without reevaluating the entire notebook. I don't want the entire workspace saved, just the final results. I hoped there was some ma…
It looks like there’s a %store command in Jupyter. I haven’t tried it out, but is this what you are looking for? https://stackoverflow.com/questions/34342155/how-to-pickle-o... (I just got it by googling “pickle an object in jupyter,” so sorry if this is something obvious that you’ve already seen and doesn’t quite solve your problem).
Yeah it got my hopes up when I found it. But when I was testing it, I didn't find the data actually made it into the .ipynb. It turns out that's actually a global storage in your home directory and doesn't go into the notebook at all. So different notebooks each overwrite the value if they use the same variable name.