Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Introduction to Scientific Python – Pandas

datadependence.com

Post reply on HN
Print view View original

11–20 of 43 posts

Re: An Introduction to Scientific Python – Pandas

#11

Post by lottin »

As an R user I noticed a couple of oddities. First,

  len(df)
returns the number of rows rather than the number of columns. This strikes me as a bad idea, because data-frames are better thought of as a collection of columns. Typically you want to loop over the columns of a data-frame and not so much over its rows, which is performance-wise much more costly.

Second, the apply method seems totally redundant. Why call a method that calls a function when you can simply call the function directly

  df['year'] = base_year(df.water_year)
Probably I'm missing something here.

Re: An Introduction to Scientific Python – Pandas

#12

Post by manish_gill »

cavnerj wrote: post #9

pandas is very good for scientific computing and data analysis, but beware, the documentation quite frankly sucks. Stack overflow seems to be the best way to learn things

Been using Pandas for a few weeks and I...kind of agree. The 10 minute tutorial etc is fine but as soon as you start doing more complicated stuff, you need the API docs. And they leave much to be desired.

Re: An Introduction to Scientific Python – Pandas

#13

Post by jeffwass »

visarga wrote: post #10

I usually do this kind of processing by linux pipes, head, tail, cut, sort, uniq, and inline Perl. It is kind of similar to using monads, but you have to handle the formatting to and from text. A few ones of my own creation are a tool for counting and a tool for generating histograms in text. I often chain 5 or 10 of these commands together. My basic data type is similar to CSV, but using "|" instead of comma as sepa…

It's really too bad that the ASCII codes 29, 30, and 31 (Group, Record, and Unit separators) never took off, as this is exactly what they were designed for.

When implemented, they'd let you include commas, line feeds/carriage returns, etc within your data records.

Re: An Introduction to Scientific Python – Pandas

#14

Post by nzjrs »

lottin wrote: post #11

As an R user I noticed a couple of oddities. First, len(df) returns the number of rows rather than the number of columns. This strikes me as a bad idea, because data-frames are better thought of as a collection of columns. Typically you want to loop over the columns of a data-frame and not so much over its rows, which is performance-wise much more costly. Second, the apply method seems totally redundant. Why call a m…

As a large pandas user, I don't agree with the len() comment. Can you give an example?

Re: An Introduction to Scientific Python – Pandas

#15

Post by kozikow »

My blog post about the most popular pandas methods: https://kozikow.wordpress.com/2016/07/01/top-pandas-function....

Pandas is a big library and it's hard to distinguish between necessary and nice to have methods. I have written 1000s of lines in pandas and I have been doing some things "around" rather than using the proper API call.

Re: An Introduction to Scientific Python – Pandas

#16

Post by bicubic »

lottin wrote: post #11

As an R user I noticed a couple of oddities. First, len(df) returns the number of rows rather than the number of columns. This strikes me as a bad idea, because data-frames are better thought of as a collection of columns. Typically you want to loop over the columns of a data-frame and not so much over its rows, which is performance-wise much more costly. Second, the apply method seems totally redundant. Why call a m…

> This strikes me as a bad idea, because data-frames are better thought of as a collection of columns

The dataframe is a collection of records then len operator tells you how big the dataset you're dealing with. You also have len(df.columns) and df.shape

> Second, the apply method seems totally redundant

df.water_year refers to a column. You can certainly use the syntax you wrote, provided you crafted a function that manipulate a column in some way. E.g. if you had a function that returns the first 2 elements of what was given, passing a column to that function would return a view into that column with only the first 2 rows. Passing the same function into apply would process every element in the (string) column and return the first 2 letters, finally returning a brand new column where each row is the first 2 letters of the corresponding row of the input.

Both of these behaviours make perfect sense if you think about them in terms of expected Python and Numpy which Pandas is built on.

Re: An Introduction to Scientific Python – Pandas

#17

Post by nurettin »

lottin wrote: post #11

As an R user I noticed a couple of oddities. First, len(df) returns the number of rows rather than the number of columns. This strikes me as a bad idea, because data-frames are better thought of as a collection of columns. Typically you want to loop over the columns of a data-frame and not so much over its rows, which is performance-wise much more costly. Second, the apply method seems totally redundant. Why call a m…

Sure if your column data is completely independent and you don't need more than one column at a time in a given algorithm, it is natural to iterate over columns instead of rows. However if you need multiple columns (or data properties) at each iteration, which is more likely the case in my experience, then you end up iterating over the rows.

Re: An Introduction to Scientific Python – Pandas

#18

Post by robochat42 »

manish_gill wrote: post #12
cavnerj wrote: post #9

pandas is very good for scientific computing and data analysis, but beware, the documentation quite frankly sucks. Stack overflow seems to be the best way to learn things

Been using Pandas for a few weeks and I...kind of agree. The 10 minute tutorial etc is fine but as soon as you start doing more complicated stuff, you need the API docs. And they leave much to be desired.

I also use Pandas for some of my data analysis and I found that it took me a long time to learn how to use it. Unlike numpy, I just couldn't remember how to do things and had to keep looking things up. Maybe this is just because Pandas has a lot of functionality. But I might waste half an hour trying to write one line of code although that line would do most of my analysis.

Re: An Introduction to Scientific Python – Pandas

#19

Post by ACow_Adonis »

lottin wrote: post #11

As an R user I noticed a couple of oddities. First, len(df) returns the number of rows rather than the number of columns. This strikes me as a bad idea, because data-frames are better thought of as a collection of columns. Typically you want to loop over the columns of a data-frame and not so much over its rows, which is performance-wise much more costly. Second, the apply method seems totally redundant. Why call a m…

As someone who is, uh, fluent in R (begrudgingly), allow me to retort:

While you're right that in R a data frame is essentially a list of columns, this strikes me as a flaw in R. Others coming to R expect to be able to loop over the observations in a data frame, or get number of observations by taking the length of the data structure. Indeed for most of my real world work that's what I actually want to do: iterate over customers or units that have multiple observations, stored as rows in the df with variables describing characteristics regarding that observation. I assure you, for everyone else coming to R, that is a genuine "WTF" moment when they loop across a data frame and find themselves iterating across variables rather than observations, or that they accidentally took the length of the data frame to be the number of observations rather than the number of variables: and I've got a glorious real world story of a bug caused by that on a 1 x 0 dimension data frame being returned by consultants code...

I have no idea if that's how it's actually implemented in pandas though...

As for the apply thing: I'm guessing that has to do with python syntax and the nature of functions/methods/data frames, but I agree with you it's a bit kludgy to me too. But I guess that's because what you're actually doing is applying a scalar function across a sequence of values, not actually calling a function that takes a sequence as an argument. In your example there, which is very R'y because the function application would be automatically vectorised, in python there's no such (necessary) thing. The reason this "kind of" works "naturally" in R is actually because R is weird and takes an efficiency hit by not having unboxed scalar values at all: even single numbers are actually vectors, as is the result of the returned operations/functions on them, so you actually have no scalar operations at all (but for many applications you don't actually notice:[1] + [1] = [2] is effectively the same as 1 + 1 = 2 in an unvectorised language, barring the R resource hit which is insignificant in smaller examples/problems.

Re: An Introduction to Scientific Python – Pandas

#20

Post by stinos »

jeffwass wrote: post #13
visarga wrote: post #10

I usually do this kind of processing by linux pipes, head, tail, cut, sort, uniq, and inline Perl. It is kind of similar to using monads, but you have to handle the formatting to and from text. A few ones of my own creation are a tool for counting and a tool for generating histograms in text. I often chain 5 or 10 of these commands together. My basic data type is similar to CSV, but using "|" instead of comma as sepa…

It's really too bad that the ASCII codes 29, 30, and 31 (Group, Record, and Unit separators) never took off, as this is exactly what they were designed for. When implemented, they'd let you include commas, line feeds/carriage returns, etc within your data records.

they'd let you include commas, line feeds/carriage returns, etc within your data records

And there would also be less ambiguity as to what seperator to use. I understand the popularity of CSV, but it's really not so nice to share data with. German customers want semicolons as a seperator, the US ones claims they are right 'because after all it is called comma-seperated and else I cannot import it in Excel' (sic). Etc.

Post reply on HN