Estimating Number of Jupyter Notebooks on Github • Hacker News Board

Re: Estimating Number of Jupyter Notebooks on Github

#61

Post by julienchastang » Wed, May 08, 2019, 5:16 PM UTC

In the same spirit as “Effective Java” and “Effective C++” we need to have a book entitled “Effective Jupyter Notebooks”. Here are some of my items below. Maybe this sub-thread can come up with an outline for this book.

Item #1 Writing a notebook is foremost an exercise in expository writing. Make sure the writing is high quality is the first objective when writing a notebook. This is the Knuth’s literate programming idea where prose takes precedence over code, which is usually the reverse of the way we usually program; code first, comments second.

Item #2 Don't use notebooks for general purpose programming. Notebooks are supposed to have an audience and clearly explain something.

Item #3 Keep code cells simple and clear. If needing a comment in the code block, consider putting that verbiage in a markdown cell instead and elaborating on the idea the notebook is trying to convey.

Item #4 Don't make notebooks a long series of extended code cells, or even worse, just one long cell. Explain what is going on or see Item #2.

Re: Estimating Number of Jupyter Notebooks on Github

#62

Post by kadal » Wed, May 08, 2019, 5:17 PM UTC

tincholio wrote: post #48

If only more people would use org-babel... If you're on emacs and like Jupyter, there's https://github.com/dzop/emacs-jupyter , which is pretty nice. I've been using it for a few days with Julia, and it works really nice. It also allows you to use different kernels from the same org-mode file, though I haven't tried to pass data between them yet (should be possible, though, at least it works in plain org-mode).

I agree. emacs-jupyter is great but I'm eagerly waiting for jupyter notebook server support since my work is on remote clusters now. For local use, it's pretty great.

Re: Estimating Number of Jupyter Notebooks on Github

#63

Post by ertemplin » Wed, May 08, 2019, 5:17 PM UTC

martinzugnoni wrote: post #43

jsilence wrote: post #26

Earlier quoted context omitted.

Wondering how you are planning to keep it free. Also wondering whether you would possibly consider shifting to Sagemath/CoCalc as a service.

We got support of the local university at my city and we got a bunch of free credits at AWS. Costs are very low and we want to keep it that way so we can support the most students we can with a free access.

What happens when your funding and AWS credits run out?

Re: Estimating Number of Jupyter Notebooks on Github

#64

Post by hodder » Wed, May 08, 2019, 5:17 PM UTC

Very cool work here. This is a pretty epic post, so please do not take this the wrong way.

I was under the impression that FB Prophet was optimal for significantly seasonal time series data.

Honestly given the fickle nature of these kind of growth patterns beyond the very near term, an ARIMA with a flat vol or a simple eyeball extrapolation in my experience as a quant would likely generate just as reasonable/reliable results.

While I understand this is likely intended as a standalone project, it would be interesting to run a comparison of ARIMA vs FB Prophet on out of sample trending Github tools/file types, as well as the general performance of these predictions beyond a one year time frame (especially vs the reported confidence intervals in Prophet).

I am not that familiar with how Prophet works, so I am absolutely open to being humbled and corrected. I have a project myself that has a varying seasonal component and I am looking forward to diving into Prophet for a deeper understanding. I am attempting to model an Asian 2 asset spread option with a volume weighted average index price setting mechanism where the underlying exhibits seasonality in the volume traded over the trading time window. I am currently running a Monte Carlo on the valuation with a simple average settlement assumption, as opposed to a volume weighted average assumption, and I was thinking Prophet could help.

Does anyone have experience in financial time series analysis and option valuation who would care to chime in?

Also, what is everyone's thoughts on using prophet non seasonal vol clustering times series?

Re: Estimating Number of Jupyter Notebooks on Github

#65

Post by bawana » Wed, May 08, 2019, 5:20 PM UTC

claytonjy wrote: post #34

kbd wrote: post #30

If you ever put notebooks in source control, you owe it to yourself to try the text-based notebooks supported in Visual Studio Code[1]. They're round-trippable with real (i.e. browser-based) notebooks, yet are much better for collaboration, diffing, and editing. [1] https://code.visualstudio.com/docs/python/jupyter-support

How does this compare to Jupytext? I prefer pipenv to Conda, and I don't like having Jupyter(Lab) installed in each venv separately, so instead I only add `Ipykernel` to each venv and then use my system-level JupyterLab to access per-project kernels; seems like that wouldn't work here?

I've noticed that if I dont install jupyter to each venv, then the pythonpath defaults to the jupyter install. As a result, the jupyter notebook cannot find the packages in the venv(for example, caffe, module not found errors)

Re: Estimating Number of Jupyter Notebooks on Github

#66

Post by pwhitebelt » Wed, May 08, 2019, 5:21 PM UTC

chrisjc wrote: post #60

hyperbovine wrote: post #53

Especially odd because they are so unsuitable for use with git. Someone needs to find a way to fix this.

> because they are so unsuitable for use with git Can you going into a little more depth about this statement?

Diffs primarily I'm guessing - a, it's kinda hard to parse the jsons that you see when you look at a notebook in raw text b, every time I execute a cell, it shows up in the diff as a change. That being said, there are plugins and tools that deal with these issues quite well. check out https://nbdime.readthedocs.io/en/latest/

Re: Estimating Number of Jupyter Notebooks on Github

#67

Post by jimmar » Wed, May 08, 2019, 5:22 PM UTC

julienchastang wrote: post #61

In the same spirit as “Effective Java” and “Effective C++” we need to have a book entitled “Effective Jupyter Notebooks”. Here are some of my items below. Maybe this sub-thread can come up with an outline for this book. Item #1 Writing a notebook is foremost an exercise in expository writing. Make sure the writing is high quality is the first objective when writing a notebook. This is the Knuth’s literate programming…

For #2, I have used notebooks for personal data analysis. I am my own audience and need to remember what I did and why.

Re: Estimating Number of Jupyter Notebooks on Github

#68

Post by arnioxux » Wed, May 08, 2019, 5:27 PM UTC

julienchastang wrote: post #61

In the same spirit as “Effective Java” and “Effective C++” we need to have a book entitled “Effective Jupyter Notebooks”. Here are some of my items below. Maybe this sub-thread can come up with an outline for this book. Item #1 Writing a notebook is foremost an exercise in expository writing. Make sure the writing is high quality is the first objective when writing a notebook. This is the Knuth’s literate programming…

For example Peter Norvig does an exemplary job with jupyter notebooks for expository writings: https://github.com/norvig/pytudes

Re: Estimating Number of Jupyter Notebooks on Github

#69

Post by fathead_glacier » Wed, May 08, 2019, 5:30 PM UTC

kadal wrote: post #59

kbd wrote: post #30

If you ever put notebooks in source control, you owe it to yourself to try the text-based notebooks supported in Visual Studio Code[1]. They're round-trippable with real (i.e. browser-based) notebooks, yet are much better for collaboration, diffing, and editing. [1] https://code.visualstudio.com/docs/python/jupyter-support

Oh man, that's basically org-babel [0] but with working jupyter server support! 0: https://orgmode.org/manual/Working-with-Source-Code.html#Wor...

This sounds so good! I just read that Latex support is also included. If there is also an extension to handle references I can move out of emacs / org-mode / orgref because the maintenance cost is too high.

Cannot wait to give it a try.

Re: Estimating Number of Jupyter Notebooks on Github

#70

Post by martinzugnoni » Wed, May 08, 2019, 5:30 PM UTC

zeptomu wrote: post #52

martinzugnoni wrote: post #20

My two cents: We've been recently working in a FREE hosted version of Jupyter Lab mainly intended for education. Feel free to check it out. https://notebooks.ai/ Would love to hear some feedback.

> We've been recently working in a FREE hosted version [...] Oh no, you make it sound like this is a good thing. It only means I can't take you serious. Don't make it free . That is not a feature for a computation environment, it will only cause headaches on your side and people get wrong (bad) impression about the performance (assuming free accounts get some limited shared instances). I rather pay a monthly fee for…

That's a very good point. We wanted to make it super accessible for students using Jupyter. Also, we only support very small containers which of course don't have GPU. It's not mainly intended to be used for business, but for learning purposes.