Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Teaching Pandas and Jupyter to Northwestern journalism students

californiacivicdata.org

Post reply on HN
Print view View original

41–47 of 47 posts

Re: Teaching Pandas and Jupyter to Northwestern journalism students

#41

Post by [deleted] »

amt528 wrote: post #17
david_eads wrote: post #10

Earlier quoted context omitted.

Exactly this. I'm the author of the post and was a programmer by trade for a long time before I became a journalist. I _don't_ actually find this more intuitive than more explicit and fundamental programming techniques. But my students grokked it immediately, whereas even simple structures like loops seem to be harder to get for them to get their heads around. Given I had ten weeks to cram a lot of material in but di…

Er, do you really think they "grokked" it immediately? Does anyone truly grok anything, especially in programming, immediately?

[deleted]

Re: Teaching Pandas and Jupyter to Northwestern journalism students

#42

Post by Declanomous »

aldanor wrote: post #34

So many people don't realize pandas can be horribly slow if you use it "wrong" -- i.e., for computations that don't vectorize in the way that's native for pandas. Also, working with dataframes that contain millions of rows is like playing a Russian roulette -- there's usually many ways to do the same thing in pandas, if you guessed correct you'll wait a minute or two till the computation's done, if you guessed wrong…

So I've been learning Pandas after mostly using either standard Python, R or VB to do our analysis, and I'm glad I read this because I thought I was going crazy.

I have a data set of about 4 million rows I routinely analyze. I have 32 gb of space on my desktop, and the only time I've really run out is when I write incredibly poor code. In the short while I've been trying to use Pandas run out of memory and get killed by the OOM killer or completely freeze my system for half an hour while processing what I thought were simple operations.

I was honestly beginning to believe I was way worse at programming than I thought due to all of the issues I was having. I wasn't even doing anything particularly complex, I was just loading a dataframe from a sql query and playing around with basic manipulation.

Re: Teaching Pandas and Jupyter to Northwestern journalism students

#43

Post by radarsat1 »

stdbrouw wrote: post #31
thearn4 wrote: post #3

I really like Jupyter, but somehow I'm not in love with it. Like, every time I fire it up to use it for quick data analysis, I seem to inevitably end up back in sublime + bash, sending plots to disk. Am I the odd one out?

You're not the only one. I don't want notebooks, I want my own damned editor. For Atom, there is https://github.com/nteract/hydrogen which embeds Jupyter/iPython output right inside of your editor, not so different from how RStudio works.

There is also a way to embed Jupyter inside emacs: https://github.com/millejoh/emacs-ipython-notebook

Re: Teaching Pandas and Jupyter to Northwestern journalism students

#44

Post by radarsat1 »

thearn4 wrote: post #3

I really like Jupyter, but somehow I'm not in love with it. Like, every time I fire it up to use it for quick data analysis, I seem to inevitably end up back in sublime + bash, sending plots to disk. Am I the odd one out?

My main criticism is the ipynb files. I don't like that it stores input and output in the same file. Ideally I'd like at least an option for it to put the output in a directory, with images stored as normal, separate files. It's commonly known that the current approach is terrible for version control, for one thing.

Re: Teaching Pandas and Jupyter to Northwestern journalism students

#45

Post by radarsat1 »

rjeli wrote: post #22
bsder wrote: post #16

It is hard to overstate just how ferociously bad the experience of getting Jupyter from blank computer to the equivalent of "Hello world" actually is.

I have a strategy that works pretty consistently - close your eyes and ignore the best practices like using Anaconda, Python 3, virtualenv (or venv in py3... oh wait it's a module?) and just install Python 2.7 with pip into default locations (I even run pip with sudo, the horror). It works really well! I run all sorts of CV, ML, deep learning notebooks with no problems.

I agree, I never use virtualenv. I might if I was building a production system, but for my own laptop I feel perfectly capable of remember/tracking/checking what is in my ~/.local. (I always install with `--user`)

If I really need to containerize something, I use Docker.

Re: Teaching Pandas and Jupyter to Northwestern journalism students

#46

Post by david_eads »

gravypod wrote: post #19
david_eads wrote: post #18

Earlier quoted context omitted.

I'm actually so with you on a lot of this. The inability to use Pythonisms with Pandas is insane and I had to do a data analysis where I really, genuinely needed to do some looping and some simple map/reduce and it almost drove me insane. You might like [Agate]( http://agate.readthedocs.io/ ) better. I haven't done a ton of Jupyter in the newsroom yet, but what I've found myself doing is abstracting out the stuff I w…

> The inability to use Pythonisms with Pandas is insane and I had to do a data analysis where I really, genuinely needed to do some looping and some simple map/reduce and it almost drove me insane. I recently started a project that I got to write from the ground up by myself. I was happy with the processing side of things. I was very sad with the data I was getting in and putting out. There's some impedance mismatch…

> I recently started a project that I got to write from the ground up by myself. I was happy with the processing side of things. I was very sad with the data I was getting in and putting out. There's some impedance mismatch that doesn't need to exist.

Impedance mismatch is a great way to put it. For me, if I can deal with that mismatch so that newbies/journalism colleagues don't have to, I'll do it.

> Sadly in my field matplotlib is the professional tool (hah!). The end goal is the matplotlib plots. I'd be all fine for tweaking things in a designing program and putting it up by I'd be upset with myself.

I used to work in science and have found journalism to have better solved many of these issues (at the expense, of course, of specialization and depth -- even a yearlong project isn't quite the same as decades of experience working in a single area). The solutions aren't pure or pretty -- they're more about workflow and held together with duct tape and baling wire. But the competitive pressure to deliver data that has a good user experience on deadline is very powerful and has led to some effective practices.

Post reply on HN