Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Introduction to Scientific Python – Pandas

datadependence.com

Post reply on HN
Print view View original

21–30 of 43 posts

Re: An Introduction to Scientific Python – Pandas

#21

Post by keypusher »

lottin wrote: post #11

As an R user I noticed a couple of oddities. First, len(df) returns the number of rows rather than the number of columns. This strikes me as a bad idea, because data-frames are better thought of as a collection of columns. Typically you want to loop over the columns of a data-frame and not so much over its rows, which is performance-wise much more costly. Second, the apply method seems totally redundant. Why call a m…

Iterating over the rows much more intuitive to me, just like rows in a database. In their example dataframe each row is a year, and columns represent different information about that year. So, if I wanted to compare rain from oct-sep on a yearly basis, I would iterate over the years (rows) and then grab that column by name.

Re: An Introduction to Scientific Python – Pandas

#22

Post by lottin »

nzjrs wrote: post #14
lottin wrote: post #11

As an R user I noticed a couple of oddities. First, len(df) returns the number of rows rather than the number of columns. This strikes me as a bad idea, because data-frames are better thought of as a collection of columns. Typically you want to loop over the columns of a data-frame and not so much over its rows, which is performance-wise much more costly. Second, the apply method seems totally redundant. Why call a m…

As a large pandas user, I don't agree with the len() comment. Can you give an example?

In R a data-frame is a list of vectors (in Python parlance, a dictionary of arrays). Therefore the length of a data-frame is the number of columns and an iteration over a data-frame iterates over its columns. Iterating over the rows can be done but it's generally better avoided because it's highly inefficient. The reason is that since the columns have different types each row has to be represented as a list. This is also true in Python, as far as I know.

Re: An Introduction to Scientific Python – Pandas

#23

Post by lottin »

bicubic wrote: post #16
lottin wrote: post #11

As an R user I noticed a couple of oddities. First, len(df) returns the number of rows rather than the number of columns. This strikes me as a bad idea, because data-frames are better thought of as a collection of columns. Typically you want to loop over the columns of a data-frame and not so much over its rows, which is performance-wise much more costly. Second, the apply method seems totally redundant. Why call a m…

> This strikes me as a bad idea, because data-frames are better thought of as a collection of columns The dataframe is a collection of records then len operator tells you how big the dataset you're dealing with. You also have len(df.columns) and df.shape > Second, the apply method seems totally redundant df.water_year refers to a column. You can certainly use the syntax you wrote, provided you crafted a function that…

Thanks for clearing that up, now it does make sense. In R most functions handle vectors as well as scalars without distinction, so normally one would use the function directly. Whereas if you wanted to process each element of a vector individually then you'd use apply(). It works the other way around.

Re: An Introduction to Scientific Python – Pandas

#24

Post by lottin »

ACow_Adonis wrote: post #19
lottin wrote: post #11

As an R user I noticed a couple of oddities. First, len(df) returns the number of rows rather than the number of columns. This strikes me as a bad idea, because data-frames are better thought of as a collection of columns. Typically you want to loop over the columns of a data-frame and not so much over its rows, which is performance-wise much more costly. Second, the apply method seems totally redundant. Why call a m…

As someone who is, uh, fluent in R (begrudgingly), allow me to retort: While you're right that in R a data frame is essentially a list of columns, this strikes me as a flaw in R. Others coming to R expect to be able to loop over the observations in a data frame, or get number of observations by taking the length of the data structure. Indeed for most of my real world work that's what I actually want to do: iterate ov…

Iterating over variables may seem counter-intuitive but it actually is the right thing to do when you have a data-frame.

The reason is that data-frames are intended for dealing with heterogeneous data. The proper way to loop over observations is to convert the variables to a common data type, e.g. logical or numeric, then you have a matrix and then you can loop over rows.

If recall correctly pandas uses a dictionary to implement data-frames, therefore iterating over rows in pandas has the same performance hit as in R.

Re: An Introduction to Scientific Python – Pandas

#25

Post by filmor »

keypusher wrote: post #21
lottin wrote: post #11

As an R user I noticed a couple of oddities. First, len(df) returns the number of rows rather than the number of columns. This strikes me as a bad idea, because data-frames are better thought of as a collection of columns. Typically you want to loop over the columns of a data-frame and not so much over its rows, which is performance-wise much more costly. Second, the apply method seems totally redundant. Why call a m…

Iterating over the rows much more intuitive to me, just like rows in a database. In their example dataframe each row is a year, and columns represent different information about that year. So, if I wanted to compare rain from oct-sep on a yearly basis, I would iterate over the years (rows) and then grab that column by name.

It's inconsistent though, as iterating over a dataframe like

    for c in df:
will return the column labels. I expect `len(obj)` to return the same as `len([i for i in obj])`.

Re: An Introduction to Scientific Python – Pandas

#26

Post by nzjrs »

lottin wrote: post #24
ACow_Adonis wrote: post #19

Earlier quoted context omitted.

As someone who is, uh, fluent in R (begrudgingly), allow me to retort: While you're right that in R a data frame is essentially a list of columns, this strikes me as a flaw in R. Others coming to R expect to be able to loop over the observations in a data frame, or get number of observations by taking the length of the data structure. Indeed for most of my real world work that's what I actually want to do: iterate ov…

Iterating over variables may seem counter-intuitive but it actually is the right thing to do when you have a data-frame. The reason is that data-frames are intended for dealing with heterogeneous data. The proper way to loop over observations is to convert the variables to a common data type, e.g. logical or numeric, then you have a matrix and then you can loop over rows. If recall correctly pandas uses a dictionary…

> The reason is that data-frames are intended for dealing > with heterogeneous data. The proper way to loop over > observations is to convert the variables to a common data > type, e.g. logical or numeric, then you have a matrix and > then you can loop over rows.

Pandas saves its users the 'proper' step of 'converting the variables to a common data type', and lets me iterate over rows to get the observations. That seems like a win to me, no?

Re: An Introduction to Scientific Python – Pandas

#27

Post by lottin »

nurettin wrote: post #17
lottin wrote: post #11

As an R user I noticed a couple of oddities. First, len(df) returns the number of rows rather than the number of columns. This strikes me as a bad idea, because data-frames are better thought of as a collection of columns. Typically you want to loop over the columns of a data-frame and not so much over its rows, which is performance-wise much more costly. Second, the apply method seems totally redundant. Why call a m…

Sure if your column data is completely independent and you don't need more than one column at a time in a given algorithm, it is natural to iterate over columns instead of rows. However if you need multiple columns (or data properties) at each iteration, which is more likely the case in my experience, then you end up iterating over the rows.

That's what Pandas encourages you to do! In my experience iterating is rarely needed at all if you have functions that operate on arrays.

Re: An Introduction to Scientific Python – Pandas

#28

Post by IndianAstronaut »

visarga wrote: post #10

I usually do this kind of processing by linux pipes, head, tail, cut, sort, uniq, and inline Perl. It is kind of similar to using monads, but you have to handle the formatting to and from text. A few ones of my own creation are a tool for counting and a tool for generating histograms in text. I often chain 5 or 10 of these commands together. My basic data type is similar to CSV, but using "|" instead of comma as sepa…

>but using "|" instead of comma as separator because it tends not to appear in text as much.

I do this as well. Using a comma to separate values seems silly to me, commas appear so frequently in text.

Re: An Introduction to Scientific Python – Pandas

#29

Post by IndianAstronaut »

cavnerj wrote: post #9

pandas is very good for scientific computing and data analysis, but beware, the documentation quite frankly sucks. Stack overflow seems to be the best way to learn things

The scientific Python environment has very erratic documentation. Matplotlib for example has pages and pages of completely disorganized and often hard to decipher documentation. Examples are very sparse.

Re: An Introduction to Scientific Python – Pandas

#30

Post by IndianAstronaut »

mojoe wrote: post #2

Pandas is certainly excellent -- be aware of it's NA type promotion behavior before you start designing data analysis programs, however. I learned this the hard way: http://pandas.pydata.org/pandas-docs/stable/gotchas.html#nan...

Another gotcha is variable type inference. Reading csv files can often produce varying column types. This can be a pain for any consistent data pipeline.
Post reply on HN