Live data from Hacker News

Automated Reports with Jupyter Notebooks (Using Jupytext and Papermill)

medium.com

11–16 of 16 posts

Re: Automated Reports with Jupyter Notebooks (Using Jupytext and Papermill)

#11
post #9

What library is the author using, containing the sundial visualization?

Looks like Plotly to me.

UPDATE: Plotly confirmed via looking at the notebook source repo: https://github.com/CFMTech/jupytext_papermill_post/blob/mast...

Re: Automated Reports with Jupyter Notebooks (Using Jupytext and Papermill)

#12
post #8

I don't think Jupyter notebooks should be used for automated jobs. They're great for exploratory stuff but once things are getting fleshed out and cleaned up, one should move to proper python files that can be unit tested and versioned without having to go to crazy lengths...

To be honest, I've always been puzzled with how extensively people use Jupyter notebooks, despite how deficient they can be in many situations. Don't get me wrong, I have used them a lot and I still use them on occasion, but pretty much the only times I would really reach for them are when I want code + documentation to reside together. So if I am prototyping things that also would benefit from Markdown + Latex comme…

The reason Jupyter notebooks are so popular is that people who only learned to code or are learning to code to enable some kind of analysis can easily do it without needing to get much of anything else set up. If all you really need is to read in some data, do some statistical tests, and make some plots, I'd argue it's by far the simplest solution. Many academics don't need much more than this. Similarly, when a data scientist or product analyst is just doing analysis, not much more is needed.

Your comments about what Jupyter notebooks incentivize are definitely true (I'd argue its part of the appeal - people new to programming often don't immediately grok types of state and Jupyter kind of just says "eh, work around it"). I certainly fall victim to it and often wish for a week of "the expectation is that you will go through your notebooks and create modules or packages for them".

I also agree that running automated jobs from Jupyter isn't an optimal solution, but for many companies, if it means a data scientist (whose coding skills are primarily statistical) can get a report into production without a single engineering hour, it's often worth it.

Re: Automated Reports with Jupyter Notebooks (Using Jupytext and Papermill)

#13
post #8

I don't think Jupyter notebooks should be used for automated jobs. They're great for exploratory stuff but once things are getting fleshed out and cleaned up, one should move to proper python files that can be unit tested and versioned without having to go to crazy lengths...

To be honest, I've always been puzzled with how extensively people use Jupyter notebooks, despite how deficient they can be in many situations. Don't get me wrong, I have used them a lot and I still use them on occasion, but pretty much the only times I would really reach for them are when I want code + documentation to reside together. So if I am prototyping things that also would benefit from Markdown + Latex comme…

Notebooks are a souped up repl, basically. So they might make sense anytime you find yourself turning to a repl. Its also nice to have that environment persist, so that you can pick up where you left off from another machine.

They also make nice front ends for spark clusters (and/or other big data engines).

I agree with the OP though. I wouldn't automate notebook execution as a part of any workflow.

Re: Automated Reports with Jupyter Notebooks (Using Jupytext and Papermill)

#14
post #9

What library is the author using, containing the sundial visualization?

Looks like Plotly to me. UPDATE: Plotly confirmed via looking at the notebook source repo: https://github.com/CFMTech/jupytext_papermill_post/blob/mast...

Thank you!

Re: Automated Reports with Jupyter Notebooks (Using Jupytext and Papermill)

#15
post #8

I don't think Jupyter notebooks should be used for automated jobs. They're great for exploratory stuff but once things are getting fleshed out and cleaned up, one should move to proper python files that can be unit tested and versioned without having to go to crazy lengths...

To be honest, I've always been puzzled with how extensively people use Jupyter notebooks, despite how deficient they can be in many situations. Don't get me wrong, I have used them a lot and I still use them on occasion, but pretty much the only times I would really reach for them are when I want code + documentation to reside together. So if I am prototyping things that also would benefit from Markdown + Latex comme…

as someone that learned pycharm below jupyter I always wondered what people used them for. but your comment actually opened my eyes since very often when I'm implementing a numerical algo i want latex comments (rendered). i wonder how hard it would be to do just that in pycharm natively

Re: Automated Reports with Jupyter Notebooks (Using Jupytext and Papermill)

#16
post #4
post #3

I dream about native version of Jupyter, without the need of a browser, JavaScript, HTML, and CSS. Probably a Vulkan-rendered widgets. Would improve performance by an order of magnitude.

I think Visual Studio Code will add that feature

in what sense?
Post reply on HN