Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A notebook-free notebook or Jupyter dev env for coders

github.com

Post reply on HN
Print view View original

11–20 of 22 posts

Re: A notebook-free notebook or Jupyter dev env for coders

#11

Post by Jugurtha »

oivey wrote: post #10
xapata wrote: post #5

Earlier quoted context omitted.

Whenever I see a notebook, I think, "Where are the tests?"

Generally speaking, the tests were writing the code in the notebook and looking at the results. At least most of the stuff I use notebooks for is either one off or difficult to test. One off stuff might be some exploratory analysis or glue code to do simple tasks like loading and formatting data. Numerical tasks like ML are frequently very difficult to test because the author doesn’t know exactly what the output shou…

If you're into machine learning and work with notebooks, take a look at the machine learning platform[0] we're making. We've been profitably shipping complete ML products for enterprise clients for many years now. Throughout these years, we've discovered patterns and inefficiencies that slowed us down, threatened the success of our projects, and overall made them more expensive than they ought to be.

It has collaborative notebooks, long-running notebook scheduling that survives network disruptions and closed tabs (you can view them while running without opening Jupyter), automatic experiment tracking for metrics, parameters, and models, and seamless deployment into a REST API. It also enables you to publish a notebook into a parametrized AppBook to allow domain experts to interact with it without being overwhelmed by the notebook interface, and to change parameters without mutating the notebook. Their runs are also tracked.

We're focused on solving actual problems we have faced on paid projects, as opposed to the infinity of features one can build to solve imaginary problems.

- [0]: https://iko.ai

Re: A notebook-free notebook or Jupyter dev env for coders

#12

Post by amingilani »

mellosouls wrote: post #3

Thanks for sharing your experience. One minor critique: the vim stuff looks like a red herring given the topic. I'm personally not particularly interested in vim but am interested in jupyter notebook-alternative setups - I'd either remove it or explain why it's relevant here or contextualise it in some other way.

I second this. I couldn't understand if it was important to the setup or just a nice-to-have feature. There's no context.

Re: A notebook-free notebook or Jupyter dev env for coders

#13

Post by JPKab »

xapata wrote: post #5
Jugurtha wrote: post #2

As long as this works for the author, good for them. What I found out looking at post after post, tool after tool, press release after press release, is that these tools optimize for the workflow of n=1 person working on a toy project with no stakes. There is a disconnect: - Most non-trivial machine learning projects I've seen involve more than one person and have stakes. - Most people do not work on non-trivial mach…

Whenever I see a notebook, I think, "Where are the tests?"

Agreed, but I highly recommend checking out nbdev.

It's used to build all of fastai's libraries, and with nbdev, the notebooks ARE the docs and tests.

I use pytest and standard IDE for most projects, but have used nbdev/jupyter on a few work projects and been astounded at the productivity boost.

Git issues with notebooks, producing pip modules, two way syncing between notebooks and code, if there is a problem that naked jupyter has for development, Jeremy Howard and team have solved it with nbdev.

Check it out.

Re: A notebook-free notebook or Jupyter dev env for coders

#15

Post by Jugurtha »

JPKab wrote: post #13
xapata wrote: post #5

Earlier quoted context omitted.

Whenever I see a notebook, I think, "Where are the tests?"

Agreed, but I highly recommend checking out nbdev. It's used to build all of fastai's libraries, and with nbdev, the notebooks ARE the docs and tests. I use pytest and standard IDE for most projects, but have used nbdev/jupyter on a few work projects and been astounded at the productivity boost. Git issues with notebooks, producing pip modules, two way syncing between notebooks and code, if there is a problem that na…

We're experimenting with `nbdev`[0], especially in our effort to support fast.ai's[1] latest course 'fastbook'[2] on iko.ai[3], to test their notebooks faster. Though scheduling notebooks on our platform is a breeze[4][5] and we could launch the 20 notebooks really fast even manually and check their output while running (some fail that way as they required user interaction for FileUpload or something, and we decided to use fixtures).

- [0]: https://github.com/fastai/nbdev

- [1]: https://www.fast.ai/

- [2]: https://github.com/fastai/fastbook

- [3]: https://iko.ai

- [4]: https://iko.ai/docs/notebook/#long-running-notebooks

- [5]: https://pbs.twimg.com/tweet_video/Entg8COXcAIDdTI.mp4

Re: A notebook-free notebook or Jupyter dev env for coders

#16

Post by 7kmph »

xapata wrote: post #5
Jugurtha wrote: post #2

As long as this works for the author, good for them. What I found out looking at post after post, tool after tool, press release after press release, is that these tools optimize for the workflow of n=1 person working on a toy project with no stakes. There is a disconnect: - Most non-trivial machine learning projects I've seen involve more than one person and have stakes. - Most people do not work on non-trivial mach…

Whenever I see a notebook, I think, "Where are the tests?"

This will be downvoted: Ideally you should not write tests.

Re: A notebook-free notebook or Jupyter dev env for coders

#17

Post by [deleted] »

Jugurtha wrote: post #2

As long as this works for the author, good for them. What I found out looking at post after post, tool after tool, press release after press release, is that these tools optimize for the workflow of n=1 person working on a toy project with no stakes. There is a disconnect: - Most non-trivial machine learning projects I've seen involve more than one person and have stakes. - Most people do not work on non-trivial mach…

[deleted]

Re: A notebook-free notebook or Jupyter dev env for coders

#18

Post by 205g0 »

mellosouls wrote: post #3

Thanks for sharing your experience. One minor critique: the vim stuff looks like a red herring given the topic. I'm personally not particularly interested in vim but am interested in jupyter notebook-alternative setups - I'd either remove it or explain why it's relevant here or contextualise it in some other way.

Good point, I should have explained it. To get into vim takes time and to get muscle memories at least 9-12 months. Something a lot of coders even don't do, so why should a data scientist do this. The result is that you are so much faster because most of the time you don't write code but stare at your code, navigate around and do small changes. So, I think if you deal with code 80% of your day, vim binds are a must and this was my biggest gripe with these notebooks. Google Colab is the only notebook offering painless vim binds. No other notebooks has proper vim support because data scientist are not coders and rather care about math.

Re: A notebook-free notebook or Jupyter dev env for coders

#19

Post by [deleted] »

JPKab wrote: post #13
xapata wrote: post #5

Earlier quoted context omitted.

Whenever I see a notebook, I think, "Where are the tests?"

Agreed, but I highly recommend checking out nbdev. It's used to build all of fastai's libraries, and with nbdev, the notebooks ARE the docs and tests. I use pytest and standard IDE for most projects, but have used nbdev/jupyter on a few work projects and been astounded at the productivity boost. Git issues with notebooks, producing pip modules, two way syncing between notebooks and code, if there is a problem that na…

[deleted]

Re: A notebook-free notebook or Jupyter dev env for coders

#20

Post by xapata »

oivey wrote: post #10
xapata wrote: post #5

Earlier quoted context omitted.

Whenever I see a notebook, I think, "Where are the tests?"

Generally speaking, the tests were writing the code in the notebook and looking at the results. At least most of the stuff I use notebooks for is either one off or difficult to test. One off stuff might be some exploratory analysis or glue code to do simple tasks like loading and formatting data. Numerical tasks like ML are frequently very difficult to test because the author doesn’t know exactly what the output shou…

Unfortunately, much like "prototype" code, notebooks often live and work far beyond their original purpose.
Post reply on HN