Lesser-Known Python Data Analysis Libraries • Hacker News Board

Re: Lesser-Known Python Data Analysis Libraries

#41

Post by __derek__ » Tue, Apr 19, 2016, 4:57 PM UTC

dguaraglia wrote: post #13

My 2 cents: I would not recommend basing any new work on MRjob. As someone who inherited and has been maintaining a bunch of code that depends on it, the library seems to be barely maintained, support for VPC is only partial and not very well documented, the auditing tools stopped working quite a while ago and tracking the progress/status of EMR jobs is extremely painful (to be fair, this is more of an issue with Ela…

It looks like mrjob development has been re-started, but there was a disconcerting period (nearly two years) without a release.[1] I used it for rinky-dink projects, and it seemed fragile at the time, so I can understand your inclination to divest from it.

[1]: https://github.com/Yelp/mrjob/releases

Re: Lesser-Known Python Data Analysis Libraries

#42

Post by ced » Tue, Apr 19, 2016, 5:19 PM UTC

elsherbini wrote: post #18

plotly is a fantastic tool for plotting. It has a python API [0], but also works from R, matlab, and Julia. It also has support for pandas dataframes and jupyter notebook[1], which is by far the fastest way I've found to make attractive plots. plotlyjs[2] is a fantastic wrapper around d3. So I can go all the way from plotting something quickly from a dataframe to building a totally custom chart. [0] https://plot.ly/p…

How does it compare with Bokeh?

Re: Lesser-Known Python Data Analysis Libraries

#43

Post by qopp » Tue, Apr 19, 2016, 5:26 PM UTC

If you're looking for a simple data pipeline, there's pipeless: https://github.com/andychase/pipeless

Also reparse if you want to parse natural language with regular expressions: https://github.com/andychase/reparse

Re: Lesser-Known Python Data Analysis Libraries

#44

Post by solaxun » Tue, Apr 19, 2016, 5:33 PM UTC

daveguy wrote: post #27

herge wrote: post #9

Earlier quoted context omitted.

FWIW, the sort method (and sorted keyword) take a 'key' keyword, where you can pass a function to use to calculate the key to sort the sequence with. So in your file11 case, you can do: sorted(files, key=lambda x: int(x[4:]) , and it will do the right thing. Although with natsort, you don't have to parse the actual strings yourself.

That is a neat trick, but it would be incredibly brittle. Kids, don't try this at home!

Pass in an re.match or re.search based function, i would imagine that would be powerful enough to meet most needs.

import re

x = ['foo12901','fooo900','fooooooo980090']

x =sorted(x,key = lambdax:int(re.search('\d+',x).group()))

print(x)

Re: Lesser-Known Python Data Analysis Libraries

#45

Post by stevejohnson » Tue, Apr 19, 2016, 5:52 PM UTC

__derek__ wrote: post #41

dguaraglia wrote: post #13

My 2 cents: I would not recommend basing any new work on MRjob. As someone who inherited and has been maintaining a bunch of code that depends on it, the library seems to be barely maintained, support for VPC is only partial and not very well documented, the auditing tools stopped working quite a while ago and tracking the progress/status of EMR jobs is extremely painful (to be fair, this is more of an issue with Ela…

It looks like mrjob development has been re-started, but there was a disconcerting period (nearly two years) without a release.[1] I used it for rinky-dink projects, and it seemed fragile at the time, so I can understand your inclination to divest from it. [1]: https://github.com/Yelp/mrjob/releases

In case anyone's curious, what happened was that Dave (@davidmarin) and I (@irskep), the mrjob maintainers, left Yelp within about a month of each other. (There's no story there, just coincidence.) There was never any momentum with new maintainers, going by the release history.

But now Dave is working on mrjob regularly again, hence the pace of recent improvements.

Grandparent is correct about the second-class support for non-EMR production Hadoop usage. Like any open source project, the code only works well if a major stakeholder invests in improving it. Few non-EMR users spend much time contributing, so the situation doesn't improve.

Re: Lesser-Known Python Data Analysis Libraries

#46

Post by dguaraglia » Tue, Apr 19, 2016, 5:55 PM UTC

stevejohnson wrote: post #45

__derek__ wrote: post #41

Earlier quoted context omitted.

It looks like mrjob development has been re-started, but there was a disconcerting period (nearly two years) without a release.[1] I used it for rinky-dink projects, and it seemed fragile at the time, so I can understand your inclination to divest from it. [1]: https://github.com/Yelp/mrjob/releases

In case anyone's curious, what happened was that Dave (@davidmarin) and I (@irskep), the mrjob maintainers, left Yelp within about a month of each other. (There's no story there, just coincidence.) There was never any momentum with new maintainers, going by the release history. But now Dave is working on mrjob regularly again, hence the pace of recent improvements. Grandparent is correct about the second-class suppor…

Hey guys, for what its worth, MRJob has given us around 3 years of working (if sometimes clunky) EMR, so thanks for that :)

Re: Lesser-Known Python Data Analysis Libraries

#47

Post by elsherbini » Tue, Apr 19, 2016, 6:14 PM UTC

ced wrote: post #42

elsherbini wrote: post #18

plotly is a fantastic tool for plotting. It has a python API [0], but also works from R, matlab, and Julia. It also has support for pandas dataframes and jupyter notebook[1], which is by far the fastest way I've found to make attractive plots. plotlyjs[2] is a fantastic wrapper around d3. So I can go all the way from plotting something quickly from a dataframe to building a totally custom chart. [0] https://plot.ly/p…

How does it compare with Bokeh?

I prefer the aesthetic of the defaults in plotly over Bokeh. Also, for most of my tasks I can simply use dataframe.iplot() using the library from [1] above, and I value that simplicity. Lastly, I prefer that plotly is built on top of d3js so I have access to that api if I want to do anything crazy, whereas Bokeh reinvented the wheel a bit with BokehJS.

Re: Lesser-Known Python Data Analysis Libraries

#48

Post by qacek » Tue, Apr 19, 2016, 6:32 PM UTC

elsherbini wrote: post #18

plotly is a fantastic tool for plotting. It has a python API [0], but also works from R, matlab, and Julia. It also has support for pandas dataframes and jupyter notebook[1], which is by far the fastest way I've found to make attractive plots. plotlyjs[2] is a fantastic wrapper around d3. So I can go all the way from plotting something quickly from a dataframe to building a totally custom chart. [0] https://plot.ly/p…

I like plotly as well but I couldn't stand the python api nor cufflinks for that matter so I created my own wrapper. It's not fully featured but it handles 90% of the cases I want.

https://github.com/jwkvam/plotlywrapper

Re: Lesser-Known Python Data Analysis Libraries

#49

Post by jbrambleDC » Tue, Apr 19, 2016, 6:55 PM UTC

Vincent has not been properly maintained in a year. and is broken at this point since the release of Vega 2.0

Re: Lesser-Known Python Data Analysis Libraries

#50

Post by mzs » Tue, Apr 19, 2016, 6:59 PM UTC

teekert wrote: post #4

Natsort is a lifesaver when working with filenames numbered by humans (like file1, file2 ... file11), those will be sorted correctly. Beats asking people to "Please add leading 0's oh and when you suspect you will pass 100, add 2 leading 0's."

I dislike how it changes behavior from release to release, for example foo-1.2, id that foo 1.2 or foo -1.2? Default dpends on release of natsort with new routines to restore previous behavior.