Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pandas 1.0

pandas.pydata.org

Post reply on HN
Print view View original

11–20 of 80 posts

Re: Pandas 1.0

#11

Post by jzwinck »

jellyksong wrote: post #9
jzwinck wrote: post #3

I am looking forward to a decade of fewer API breaking changes. However, 1.0 introduces a new column type for strings, recommends its use over the old "object" column type, yet says it is "Experimental and may change at any time." How are we supposed to interpret this in light of the promise that there will be no more API breakages until 2.0? It reads as if this promise does not apply to string data, which impacts ra…

It states in the deprecation policy that "API-breaking changes will be made only in major releases (except for experimental features)".

So Pandas 1.0's recommended API for strings is unstable. Seems strange.

Re: Pandas 1.0

#12

Post by alexpetralia »

kmax12 wrote: post #8

I know I'm not the only one, but it's hard to imagine doing my job the last several year without Pandas. Even though Pandas has been used in production by many people as basically a 1.0.0 release for a long time, this an amazing milestone and I think everyone in my office smiled when they saw the release news. I think it's worth it to acknowledge the great stewardship of the community by all the Pandas developers (an…

I basically owe my career to pandas - it made me the go-to resource for any data analysis that couldn't be done in Excel. Once I became useful in that respect, I was strongly encouraged to further develop my programming knowledge.

Re: Pandas 1.0

#13

Post by oarabbus_ »

_coveredInBees wrote: post #7

Great accomplishment and kudos to the dedicated maintainers. That being said, I've always had a love-hate relationship with pandas. It is a very powerful library and does a ton, but yet the API is all over the place and unless you use it regularly for a long period of time, it is almost impossible to get fluent with it. Every time I am away from it for a couple of months, I find even doing the most basic things to be…

Couldn't agree more. Fantastically powerful library, but my goodness the syntax is often unfriendly, inconsistent, etc.

Re: Pandas 1.0

#14

Post by closed »

I've had to dive into the pandas code over the last year for a project [0], and my attitude has shifted dramatically from...

  * old attitude: why does pandas have to make things so hard
  * new attitude: pandas has a crazy difficult job
I think this is most apparent in the functions that decide what "[d]type" a Block--the most basic thing that stores data in pandas--should be.

https://github.com/pandas-dev/pandas/blob/4edcc5541ff3f6470f...

And then, for the ubiquitous Object dtype, often figure out which of the many possible more specific types to cast it to.

If you think that is easy, ask yourself what this outputs:

  import numpy as np
  np.array([np.nan, 'a'])
Lo and behold--it produces an array where the np.nan has been converted to the string "nan".

And yet

  import pandas as pd
  pd.Series([np.nan, "a"])
Knows this, has your back, and does not stringify it.

It also has a pathological fixation on when it tries to convert dtypes, since avoiding all the bad conversion outcomes is a relatively time intensive process (compared to e.g. creating a numpy array).

I realize things could be much easier in pandas user facing interface, but really appreciate the sheer amount of effort that has gone into its dtype wrangling.

[0]: http://github.com/machow/siuba

Re: Pandas 1.0

#15

Post by [deleted] »

closed wrote: post #14

I've had to dive into the pandas code over the last year for a project [0], and my attitude has shifted dramatically from... * old attitude: why does pandas have to make things so hard * new attitude: pandas has a crazy difficult job I think this is most apparent in the functions that decide what "[d]type" a Block--the most basic thing that stores data in pandas--should be. https://github.com/pandas-dev/pandas/blob/4…

[deleted]

Re: Pandas 1.0

#18

Post by lapnitnelav »

Congratulations to the Pandas team. You lot have saved my bacon so many times over the last few years, I owe you many breakfasts.

Long live the King.

Re: Pandas 1.0

#19

Post by _coveredInBees »

closed wrote: post #14

I've had to dive into the pandas code over the last year for a project [0], and my attitude has shifted dramatically from... * old attitude: why does pandas have to make things so hard * new attitude: pandas has a crazy difficult job I think this is most apparent in the functions that decide what "[d]type" a Block--the most basic thing that stores data in pandas--should be. https://github.com/pandas-dev/pandas/blob/4…

I think another area pandas has done a lot of work on is with datetimes. Numpy's datetime objects are pretty deficient when you need to perform computations / data wrangling with them and utilizing python's native datetime objects would slow things down a decent bit. So they have done a lot of work to create their own datetime implementation that helps a lot when dealing with tabular data and performing date/time based arithmetic/manipulations.

Re: Pandas 1.0

#20

Post by geoalchimista »

closed wrote: post #14

I've had to dive into the pandas code over the last year for a project [0], and my attitude has shifted dramatically from... * old attitude: why does pandas have to make things so hard * new attitude: pandas has a crazy difficult job I think this is most apparent in the functions that decide what "[d]type" a Block--the most basic thing that stores data in pandas--should be. https://github.com/pandas-dev/pandas/blob/4…

There are always choices to make. In this case, I would much prefer to let the data be treated as is, i.e., no silent casting of np.nan to the string "nan". When dealing with numerical data, a string "nan" is rarely useful. But when you need it, you can still create a data series with a string "nan" using

    pd.Series([str(np.nan), "a"])
Post reply on HN