Teaching Pandas and Jupyter to Northwestern journalism students • Hacker News Board

Re: Teaching Pandas and Jupyter to Northwestern journalism students

#21

Post by bigger_cheese » Thu, Jun 08, 2017, 4:28 AM UTC

gravypod wrote: post #15

david_eads wrote: post #10

Earlier quoted context omitted.

Exactly this. I'm the author of the post and was a programmer by trade for a long time before I became a journalist. I _don't_ actually find this more intuitive than more explicit and fundamental programming techniques. But my students grokked it immediately, whereas even simple structures like loops seem to be harder to get for them to get their heads around. Given I had ten weeks to cram a lot of material in but di…

I've been very troubled by coming to this stuff as a programmer. I'm having the same instant dis-satisfactory response that your students are having with looping structures. I've recently started working on some projects where I need to do a lot of data visualization, story telling, and investigation "into the data". As a programmer getting into this stuff is far worse then I expected. Nothing works as I would think…

">Are there any good frameworks that allow for processing, caching, data visualization (layout -> data population -> rendering), then exporting to some format (PNG/PDF/TeX)?"

I use SAS for this in my Day Job it's not a free program but powerful for this type of stuff.

I typically use SQL queries (via SAS's proc sql command) to manipulate and process my data but you can also programatically manipulate your data sets using SAS's "datastep" language.

SAS has support for macro expansions which make some of your examples (like manipulating 10 sensors at once) pretty trivial. But this is getting into programming language territory I would not expect someone new/unfamiliar with programming to grasp all of this intuitively.

edit: Heres some code I have in production that counts how many (of 8) sensors are reading high in a given time frame.

array aads (*) TP_AD1_TOP_STACK_TC1 -- TP_AD1_TOP_STACK_TC8; NO_AD1_TEMPERATURES_HIGH = 0; do j= 1 to dim(aads); if aads(j) gt 160 then NO_AD1_TEMPERATURES_HIGH = NO_AD1_TEMPERATURES_HIGH +1; end;

Downside is that SAS is a commercial package and it is not free I Have heard a lot of good things about "R" which is supposedly quite similar but have not had opportunity to use it myself.

Re: Teaching Pandas and Jupyter to Northwestern journalism students

#22

Post by rjeli » Thu, Jun 08, 2017, 4:31 AM UTC

bsder wrote: post #16

It is hard to overstate just how ferociously bad the experience of getting Jupyter from blank computer to the equivalent of "Hello world" actually is.

I have a strategy that works pretty consistently - close your eyes and ignore the best practices like using Anaconda, Python 3, virtualenv (or venv in py3... oh wait it's a module?) and just install Python 2.7 with pip into default locations (I even run pip with sudo, the horror). It works really well! I run all sorts of CV, ML, deep learning notebooks with no problems.

Re: Teaching Pandas and Jupyter to Northwestern journalism students

#23

Post by boron1006 » Thu, Jun 08, 2017, 4:33 AM UTC

thearn4 wrote: post #3

I really like Jupyter, but somehow I'm not in love with it. Like, every time I fire it up to use it for quick data analysis, I seem to inevitably end up back in sublime + bash, sending plots to disk. Am I the odd one out?

I really like what's offered by Jupyter Lab. It's in alpha right now, but I haven't had too many problems with it. It allows you to open text files, terminals, and notebooks in the interface.

Re: Teaching Pandas and Jupyter to Northwestern journalism students

#24

Post by kinkrtyavimoodh » Thu, Jun 08, 2017, 4:56 AM UTC

gravypod wrote: post #15

david_eads wrote: post #10

Earlier quoted context omitted.

Exactly this. I'm the author of the post and was a programmer by trade for a long time before I became a journalist. I _don't_ actually find this more intuitive than more explicit and fundamental programming techniques. But my students grokked it immediately, whereas even simple structures like loops seem to be harder to get for them to get their heads around. Given I had ten weeks to cram a lot of material in but di…

I've been very troubled by coming to this stuff as a programmer. I'm having the same instant dis-satisfactory response that your students are having with looping structures. I've recently started working on some projects where I need to do a lot of data visualization, story telling, and investigation "into the data". As a programmer getting into this stuff is far worse then I expected. Nothing works as I would think…

I am a programmer and I use Pandas quite a bit. While I agree that it's a little counter-intuitive at times, I have found it to be an extremely useful and important Python package that there is just no reasonable substitute for.

Re: Teaching Pandas and Jupyter to Northwestern journalism students

#25

Post by gravypod » Thu, Jun 08, 2017, 5:18 AM UTC

bigger_cheese wrote: post #21

gravypod wrote: post #15

Earlier quoted context omitted.

I've been very troubled by coming to this stuff as a programmer. I'm having the same instant dis-satisfactory response that your students are having with looping structures. I've recently started working on some projects where I need to do a lot of data visualization, story telling, and investigation "into the data". As a programmer getting into this stuff is far worse then I expected. Nothing works as I would think…

">Are there any good frameworks that allow for processing, caching, data visualization (layout -> data population -> rendering), then exporting to some format (PNG/PDF/TeX)?" I use SAS for this in my Day Job it's not a free program but powerful for this type of stuff. I typically use SQL queries (via SAS's proc sql command) to manipulate and process my data but you can also programatically manipulate your data sets u…

I'd like to get my analysis systems as "inclusive" as possible. I'd be using my internal SQL server and just fall into python for my processing if I didn't care about sharing my work.

SAS looks good though. I've looked at it many times and it is a clean solution if you really are in the "big games".

Re: Teaching Pandas and Jupyter to Northwestern journalism students

#26

Post by bigger_cheese » Thu, Jun 08, 2017, 5:35 AM UTC

gravypod wrote: post #25

bigger_cheese wrote: post #21

Earlier quoted context omitted.

">Are there any good frameworks that allow for processing, caching, data visualization (layout -> data population -> rendering), then exporting to some format (PNG/PDF/TeX)?" I use SAS for this in my Day Job it's not a free program but powerful for this type of stuff. I typically use SQL queries (via SAS's proc sql command) to manipulate and process my data but you can also programatically manipulate your data sets u…

I'd like to get my analysis systems as "inclusive" as possible. I'd be using my internal SQL server and just fall into python for my processing if I didn't care about sharing my work. SAS looks good though. I've looked at it many times and it is a clean solution if you really are in the "big games".

Yeah that is a good point trying to separate analysis from database.

My work is going opposite direction unfortunately we are starting to use Hadoop makes it quite difficult to do things "outside of the database" there is just too much data to work with locally.

Re: Teaching Pandas and Jupyter to Northwestern journalism students

#27

Post by mirimir » Thu, Jun 08, 2017, 5:43 AM UTC

gravypod wrote: post #25

bigger_cheese wrote: post #21

Earlier quoted context omitted.

">Are there any good frameworks that allow for processing, caching, data visualization (layout -> data population -> rendering), then exporting to some format (PNG/PDF/TeX)?" I use SAS for this in my Day Job it's not a free program but powerful for this type of stuff. I typically use SQL queries (via SAS's proc sql command) to manipulate and process my data but you can also programatically manipulate your data sets u…

I'd like to get my analysis systems as "inclusive" as possible. I'd be using my internal SQL server and just fall into python for my processing if I didn't care about sharing my work. SAS looks good though. I've looked at it many times and it is a clean solution if you really are in the "big games".

SQL plus R is a good combo.

Re: Teaching Pandas and Jupyter to Northwestern journalism students

#28

Post by tnecniv » Thu, Jun 08, 2017, 6:00 AM UTC

gravypod wrote: post #15

david_eads wrote: post #10

Earlier quoted context omitted.

Exactly this. I'm the author of the post and was a programmer by trade for a long time before I became a journalist. I _don't_ actually find this more intuitive than more explicit and fundamental programming techniques. But my students grokked it immediately, whereas even simple structures like loops seem to be harder to get for them to get their heads around. Given I had ten weeks to cram a lot of material in but di…

I've been very troubled by coming to this stuff as a programmer. I'm having the same instant dis-satisfactory response that your students are having with looping structures. I've recently started working on some projects where I need to do a lot of data visualization, story telling, and investigation "into the data". As a programmer getting into this stuff is far worse then I expected. Nothing works as I would think…

I feel your pain. You can pretty much blame either MATLAB origins or an relentless pursuit of runtime efficiency for most of these problems

Re: Teaching Pandas and Jupyter to Northwestern journalism students

#29

Post by stdbrouw » Thu, Jun 08, 2017, 6:01 AM UTC

bigger_cheese wrote: post #21

gravypod wrote: post #15

Earlier quoted context omitted.

I've been very troubled by coming to this stuff as a programmer. I'm having the same instant dis-satisfactory response that your students are having with looping structures. I've recently started working on some projects where I need to do a lot of data visualization, story telling, and investigation "into the data". As a programmer getting into this stuff is far worse then I expected. Nothing works as I would think…

">Are there any good frameworks that allow for processing, caching, data visualization (layout -> data population -> rendering), then exporting to some format (PNG/PDF/TeX)?" I use SAS for this in my Day Job it's not a free program but powerful for this type of stuff. I typically use SQL queries (via SAS's proc sql command) to manipulate and process my data but you can also programatically manipulate your data sets u…

As someone who has used SAS for many, many different projects: it is terrible, vastly inferior to Pandas or R, and the only reason to ever use it is when you're forced to. Even simple stuff like functions that operate on data have to be hacked on with macros.

Case in point, your production SAS code could be replaced with this Pandas code (and the R code would look very similar):

  temperatures[[TEMPERATURE_COLUMNS]].apply(lambda t: (t > 160).sum(), axis=1)

or if your data is in proper long form

  data.groupby('time').temperature.gt(160).sum()

Re: Teaching Pandas and Jupyter to Northwestern journalism students

#30

Post by stdbrouw » Thu, Jun 08, 2017, 6:17 AM UTC

gravypod wrote: post #15

david_eads wrote: post #10

Earlier quoted context omitted.

Exactly this. I'm the author of the post and was a programmer by trade for a long time before I became a journalist. I _don't_ actually find this more intuitive than more explicit and fundamental programming techniques. But my students grokked it immediately, whereas even simple structures like loops seem to be harder to get for them to get their heads around. Given I had ten weeks to cram a lot of material in but di…

I've been very troubled by coming to this stuff as a programmer. I'm having the same instant dis-satisfactory response that your students are having with looping structures. I've recently started working on some projects where I need to do a lot of data visualization, story telling, and investigation "into the data". As a programmer getting into this stuff is far worse then I expected. Nothing works as I would think…

Many of the things you list are indeed annoyances when doing data analysis in Python and they make things harder than they should be, but others are typical grievances I see from people new to it, and these do actually go away once you've been working with e.g. Pandas for longer.

> Pandas, one of the biggest "offenders", is trying to be an in-memory database with only one table but ends up having far fewer features and a far clunkier interface (want to do a simple map/reduce? Welcome to chaining a strange combination of '.loc', '&', and ':,' "operators").

What makes Pandas so great is that you can apply arbitrary functions to rows and columns, with the full expressivity of Python. In some cases it might be clunkier (though you should almost never need `.loc` and other indexing methods) but mostly it's just `df.groupby(...).apply(...)` or vectorized methods like `df.column + df.other_column`. This is a huge improvement over having half of your analysis in database queries and half in a programming language.

> Matplotlib is unintuitive and poorly documented

Try https://seaborn.pydata.org/ for statistical graphics.

> Pandas also implements it's own versions of standard python objects! You need to know, and go back and forth between two, ways of doing things.

This sucks but is unavoidable, because Python does not have fast data types with support for missing values built in, so all your columns would have to be of mixed type (the actual type + None) and everything would slow down and simple things like computing the mean of a column with missing values would not work.

Note that you don't actually "need to go back and forth" because Pandas will happily convert plain Python objects to their Numpy equivalents for you.

> 3. All these libraries separate logically grouped concepts.

It's not functional, you're just going to have to deal with that. But split-apply-combine and similar patterns are quite elegant in Pandas: http://pandas.pydata.org/pandas-docs/stable/groupby.html

> 4. Because everything is meaningless lists of numbers there are no ways to reuse code.

A lot of data analysis is throw-away code. Some of it can be abstracted into reusable code, some of it can't.

Lastly, don't forget that Python does have a lot of things going for it when it comes to data analysis, from geospatial tools (http://toblerity.org/shapely/) to Bayesian modeling (http://pymc-devs.github.io/pymc3/index.html), as well as interactive coding with Jupyter and Hydrogen for the Atom editor (https://github.com/nteract/hydrogen).