31–40 of 47 posts
I really like Jupyter, but somehow I'm not in love with it. Like, every time I fire it up to use it for quick data analysis, I seem to inevitably end up back in sublime + bash, sending plots to disk. Am I the odd one out?
But pandas’ magical simplicity makes things like computed columns immediately intuitive: > data['% of total'] = data.amount / data.amount.sum() Is that immediately intuitive? I'm staring at this trying to understand what it's doing. Is the / operator overloaded? data.amount is one particular amount, and data.amount.sum() is the sum of all amounts? Why does the "computed column" property goes on the same data object a…
Earlier quoted context omitted.
Exactly this. I'm the author of the post and was a programmer by trade for a long time before I became a journalist. I _don't_ actually find this more intuitive than more explicit and fundamental programming techniques. But my students grokked it immediately, whereas even simple structures like loops seem to be harder to get for them to get their heads around. Given I had ten weeks to cram a lot of material in but di…
I've been very troubled by coming to this stuff as a programmer. I'm having the same instant dis-satisfactory response that your students are having with looping structures. I've recently started working on some projects where I need to do a lot of data visualization, story telling, and investigation "into the data". As a programmer getting into this stuff is far worse then I expected. Nothing works as I would think…
This part is a gotcha, but it's also a reflection that allowing if checks for things other than empty leads to subtle bugs. (there are long mailing list posts about it and about the bugs that were uncovered). See here for some explanation about why numpy does it: https://github.com/numpy/numpy/issues/8622
For big datasets, I've stopped using pandas myself a few years back for anything other than printing dataframe, datetime index series, doing quick plots, or working with tiny/toy datasets -- in favor of numpy structured/record arrays. It's kind of the same thing, without all the groupby/index fluff, but very fast.
Just last week, I've helped my colleague speed up her code (numerical solver for financial data) by more than 100x, the biggest part of it was ditching pandas entirely and using numpy.
So many people don't realize pandas can be horribly slow if you use it "wrong" -- i.e., for computations that don't vectorize in the way that's native for pandas. Also, working with dataframes that contain millions of rows is like playing a Russian roulette -- there's usually many ways to do the same thing in pandas, if you guessed correct you'll wait a minute or two till the computation's done, if you guessed wrong…
I really like Jupyter, but somehow I'm not in love with it. Like, every time I fire it up to use it for quick data analysis, I seem to inevitably end up back in sublime + bash, sending plots to disk. Am I the odd one out?
1. A variable window where I can browse through the values of each variable (like R Studio) 2. Be able to set breakpoints
So basically something in-between PyCharm and Jupyter.
But pandas’ magical simplicity makes things like computed columns immediately intuitive: > data['% of total'] = data.amount / data.amount.sum() Is that immediately intuitive? I'm staring at this trying to understand what it's doing. Is the / operator overloaded? data.amount is one particular amount, and data.amount.sum() is the sum of all amounts? Why does the "computed column" property goes on the same data object a…
"% of total" : 0.01
I would not expect that to be 1%.
At least, this could easily be the source of an inaccurate calculation elsewhere. This is not a major criticism, but perhaps would be a good point to introduce the idea of testing some of your code, even as a few simple cells that calculate things you expect.
I really like Jupyter, but somehow I'm not in love with it. Like, every time I fire it up to use it for quick data analysis, I seem to inevitably end up back in sublime + bash, sending plots to disk. Am I the odd one out?
I really like what's offered by Jupyter Lab. It's in alpha right now, but I haven't had too many problems with it. It allows you to open text files, terminals, and notebooks in the interface.
Anything to help refactor things into and from file and the notebook is nice.
Earlier quoted context omitted.
I've been very troubled by coming to this stuff as a programmer. I'm having the same instant dis-satisfactory response that your students are having with looping structures. I've recently started working on some projects where I need to do a lot of data visualization, story telling, and investigation "into the data". As a programmer getting into this stuff is far worse then I expected. Nothing works as I would think…
">Are there any good frameworks that allow for processing, caching, data visualization (layout -> data population -> rendering), then exporting to some format (PNG/PDF/TeX)?" I use SAS for this in my Day Job it's not a free program but powerful for this type of stuff. I typically use SQL queries (via SAS's proc sql command) to manipulate and process my data but you can also programatically manipulate your data sets u…
In my former team, we used SAS for a while and once I introduced the team to Pandas, they happily ditched SAS.
It is hard to overstate just how ferociously bad the experience of getting Jupyter from blank computer to the equivalent of "Hello world" actually is.
% mkvirtualenv -p `which python3` notebook
(notebook) % pip install notebook jupyter notebook scipy pandas matplotlib pdbpp ipython