Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Estimating Number of Jupyter Notebooks on Github

kyso.io

Post reply on HN
Print view View original

81–90 of 135 posts

Re: Estimating Number of Jupyter Notebooks on Github

#81

Post by alpb »

Why doesn't this just use the GitHub public dataset available on Google BigQuery to have much more accurate data rather than "scraping GitHub web search results"? https://cloud.google.com/bigquery/public-data/

There are a lot of examples of people analyzing public code on GitHub efficiently for patterns and usages with BigQuery and getting pretty accurate data out of it. https://medium.com/google-cloud/analyzing-go-code-with-bigqu...

If you use GitHub on a daily basis, you are unlucky enough to know that web search sadly can't even find words that exist in your repository.

Re: Estimating Number of Jupyter Notebooks on Github

#82

Post by localhost »

julienchastang wrote: post #61

In the same spirit as “Effective Java” and “Effective C++” we need to have a book entitled “Effective Jupyter Notebooks”. Here are some of my items below. Maybe this sub-thread can come up with an outline for this book. Item #1 Writing a notebook is foremost an exercise in expository writing. Make sure the writing is high quality is the first objective when writing a notebook. This is the Knuth’s literate programming…

While I generally agree with everything you say for a particular audience, i.e., someone who is writing something to explain an idea to someone else, I think that at the same time this scoping limits the utility of Jupyter too much.

I believe that there is an audience for Jupyter beyond just "data scientist", or "teacher". I think that what Jupyter encourages is _experimentation_, and there are lots of folks who could benefit by using Jupyter to experiment with ideas. If you consider programmer as another audience, we all at different points in time experiment with things. Think about all of those random console applications that we've all created at one point or another to learn a new API, or to try out an idea. Jupyter can (and does) excel in these scenarios and handles the mundane task of "writing things down". This is why adding a Jupyter experience to a text editor is something that we're experimenting with as well.

Re: Estimating Number of Jupyter Notebooks on Github

#83

Post by rhodin »

julienchastang wrote: post #61

In the same spirit as “Effective Java” and “Effective C++” we need to have a book entitled “Effective Jupyter Notebooks”. Here are some of my items below. Maybe this sub-thread can come up with an outline for this book. Item #1 Writing a notebook is foremost an exercise in expository writing. Make sure the writing is high quality is the first objective when writing a notebook. This is the Knuth’s literate programming…

See also "computational essay"[1]. It is written with Mathematica notebooks in mind, but many of the ideas carry over to Jupyter notebooks.

[1] https://blog.stephenwolfram.com/2017/11/what-is-a-computatio...

Re: Estimating Number of Jupyter Notebooks on Github

#84

Post by airocker »

airocker wrote: post #75
asdfman123 wrote: post #44

Earlier quoted context omitted.

There are a few basic UI problems I ran into immediately. You should focus on that instead of rearchitecting things. It needs to look sleeker, too. Pay some designer to create a new design for you. Like I can't add a project. It says "To create project Please wait till hard drive button turn green." I don't know what that means. You've lost my attention.

Thanks. We are doing some enhancements. Here is some initial screen: https://mondaytest.datacabinet.systems/ system but the page inside is still the same. We have an elaborate design done.

Right now, for VMs we make a hard drive(mkfs, install things) for every user that takes some initial time. We will fix that in the kubernetes version.

Re: Estimating Number of Jupyter Notebooks on Github

#85

Post by KyleOS »

hodder wrote: post #64

Very cool work here. This is a pretty epic post, so please do not take this the wrong way. I was under the impression that FB Prophet was optimal for significantly seasonal time series data. Honestly given the fickle nature of these kind of growth patterns beyond the very near term, an ARIMA with a flat vol or a simple eyeball extrapolation in my experience as a quant would likely generate just as reasonable/reliable…

Hey, I posted the notebook by the OP. Thank you for your feedback! You're correct in saying that FB Prophet is for forecasting time series with strong seasonal effects. FB Prophet was the model used in the original script I found & the main point here was simply to make the notebook more readable on kyso, which has quite a few non-technical readers. I've worked a lot with ARIMAs before for financial/economic data and I like the idea of comparing the results between the two, and maybe even extend the time frame. So 1. I think that'll be my next project and 2. if your project is public I'd love to give it a read when published.

Re: Estimating Number of Jupyter Notebooks on Github

#86

Post by mike_ivanov »

eoinmurray92 wrote: post #55
mike_ivanov wrote: post #36

Earlier quoted context omitted.

Why would you share a notebook with somebody who doesn't understand the code?? What are they supposed to do with them anyways? Reading? Isn't that what PDF is for? Just give them PDFs, C*O people have got piles of other things to worry about besides shared notebooks.

Mostly so you don't need to convert to PDF and so that you can host the reports in a central place where everyone can read them technical or not. Like an internal wiki for a companies data-science where the technical people can communicate their work to the non-technical people with a pretty seamless experience

This is not the experience non-technical senior leadership people are looking for, unless you are a 10-people startup.

Re: Estimating Number of Jupyter Notebooks on Github

#87

Post by d0mine »

kadal wrote: post #59
kbd wrote: post #30

If you ever put notebooks in source control, you owe it to yourself to try the text-based notebooks supported in Visual Studio Code[1]. They're round-trippable with real (i.e. browser-based) notebooks, yet are much better for collaboration, diffing, and editing. [1] https://code.visualstudio.com/docs/python/jupyter-support

Oh man, that's basically org-babel [0] but with working jupyter server support! 0: https://orgmode.org/manual/Working-with-Source-Code.html#Wor...

you can use jupyter with org-babel too https://github.com/dzop/emacs-jupyter/blob/master/README.org...

Re: Estimating Number of Jupyter Notebooks on Github

#88

Post by mike_ivanov »

eoinmurray92 wrote: post #55
mike_ivanov wrote: post #36

Earlier quoted context omitted.

Why would you share a notebook with somebody who doesn't understand the code?? What are they supposed to do with them anyways? Reading? Isn't that what PDF is for? Just give them PDFs, C*O people have got piles of other things to worry about besides shared notebooks.

Mostly so you don't need to convert to PDF and so that you can host the reports in a central place where everyone can read them technical or not. Like an internal wiki for a companies data-science where the technical people can communicate their work to the non-technical people with a pretty seamless experience

Let me elaborate. Delivering information is a job. Either you do this by yourself -- by translating from your datascience-speak (aka notebooks) to the business-speak (aka concise, targeted PDFs) -- or you make stakeholders do that job instead of you.

You effectively say to them "I want you to be proactive and go ahead and grab this food which I prepared for you and placed here and here and over there, and by the way plates and utensils are in that corner, help yourself" instead of "I though you might need this, here it is."

The way you do it works in some orgs, but generally it doesn't.

Re: Estimating Number of Jupyter Notebooks on Github

#89

Post by claytonjy »

bawana wrote: post #65
claytonjy wrote: post #34

Earlier quoted context omitted.

How does this compare to Jupytext? I prefer pipenv to Conda, and I don't like having Jupyter(Lab) installed in each venv separately, so instead I only add `Ipykernel` to each venv and then use my system-level JupyterLab to access per-project kernels; seems like that wouldn't work here?

I've noticed that if I dont install jupyter to each venv, then the pythonpath defaults to the jupyter install. As a result, the jupyter notebook cannot find the packages in the venv(for example, caffe, module not found errors)

huh, I've never used Caffe but wrapping my venvs with pipenv has never caused me this kind of grief.

I tell pipenv to store venv stuff in the project folder (`export PIPENV_VENV_IN_PROJECT=1`), and then do the following to start a new project:

    pipenv --python 3.7
    pipenv install ipykernel
    pipenv run python -m ipykernel install --user --name=`basename $(pipenv run dirname '$VIRTUAL_ENV')
    jupyter lab
(all the magic is in that third command)

Then in the list of kernels, in addition to the usual suspects I'll have one named for the folder I ran the above in. If I started a notebook before all that, I'll have to change it's kernel. Doing more `pipenv install` at the prompt makes new packages immediately available in the running notebook.

Re: Estimating Number of Jupyter Notebooks on Github

#90

Post by dec0dedab0de »

julienchastang wrote: post #61

In the same spirit as “Effective Java” and “Effective C++” we need to have a book entitled “Effective Jupyter Notebooks”. Here are some of my items below. Maybe this sub-thread can come up with an outline for this book. Item #1 Writing a notebook is foremost an exercise in expository writing. Make sure the writing is high quality is the first objective when writing a notebook. This is the Knuth’s literate programming…

I disagree.

While your points are valid for presentations, I believe that notebooks should first and foremost be used for exploratory computing.

Notebooks are my goto tool for whenever I need to do something with a computer and any of the following apply:

* I'm not quite sure what or how.

* It will likely be a one-off

* I need it now.

* Someone is watching me, to learn how I do it.

I would go a step further and say that notebooks should probably not be shared directly most of the time, and that you should wrap up functionality into modules/packages for other people to use in their own notebooks.

Edit: Just thought of a corny pun to make my point: They are notebooks, not textbooks. Notebooks are personal, and while it may be useful/insightful to compare notes, it's not the primary function of a notebook.

Post reply on HN