As an R user I noticed a couple of oddities. First, len(df) returns the number of rows rather than the number of columns. This strikes me as a bad idea, because data-frames are better thought of as a collection of columns. Typically you want to loop over the columns of a data-frame and not so much over its rows, which is performance-wise much more costly. Second, the apply method seems totally redundant. Why call a m…
> This strikes me as a bad idea, because data-frames are better thought of as a collection of columns The dataframe is a collection of records then len operator tells you how big the dataset you're dealing with. You also have len(df.columns) and df.shape > Second, the apply method seems totally redundant df.water_year refers to a column. You can certainly use the syntax you wrote, provided you crafted a function that…
My PyData London presentation "Pandas from the Inside" [1, 2] explains in detail how pandas gets its speed from numpy, with benchmarks comparing slow vs fast ways to do common operations. Column-wise operations can be three orders of magnitude faster than iterating by row.
[1] https://www.youtube.com/watch?v=Dr3Hv7aUkmU
[2] https://github.com/SteveSimmons/PyData-PandasFromTheInside