Lesser-Known Python Data Analysis Libraries • Hacker News Board

Re: Lesser-Known Python Data Analysis Libraries

#11

Post by nxzero » Tue, Apr 19, 2016, 2:28 PM UTC

Number of good open source data analysis projects by primary developer of the NumPy package and his company listed here:

https://www.continuum.io/open-source-core-modern-software

Re: Lesser-Known Python Data Analysis Libraries

#12

Post by c17r » Tue, Apr 19, 2016, 2:31 PM UTC

I'll have to check delorean out, I usually use http://crsmithdev.com/arrow/ for python date manipulation. It works a lot like the javascript library moment.

Re: Lesser-Known Python Data Analysis Libraries

#13

Post by dguaraglia » Tue, Apr 19, 2016, 2:32 PM UTC

My 2 cents: I would not recommend basing any new work on MRjob. As someone who inherited and has been maintaining a bunch of code that depends on it, the library seems to be barely maintained, support for VPC is only partial and not very well documented, the auditing tools stopped working quite a while ago and tracking the progress/status of EMR jobs is extremely painful (to be fair, this is more of an issue with Elastic MapReduce than MRJob itself.)

I love the concept and ease of development, but I can't shake the feeling that the infrastructure is so shaky it almost amount to instant technical debt (sorry if this offends anyone, I'm just a dumb customer.)

Re: Lesser-Known Python Data Analysis Libraries

#14

Post by zfrenchee » Tue, Apr 19, 2016, 2:36 PM UTC

dguaraglia wrote: post #13

My 2 cents: I would not recommend basing any new work on MRjob. As someone who inherited and has been maintaining a bunch of code that depends on it, the library seems to be barely maintained, support for VPC is only partial and not very well documented, the auditing tools stopped working quite a while ago and tracking the progress/status of EMR jobs is extremely painful (to be fair, this is more of an issue with Ela…

Do you know of any good alternatives? Any way to write MapReduces in python?

Re: Lesser-Known Python Data Analysis Libraries

#15

Post by micah_chatt » Tue, Apr 19, 2016, 2:38 PM UTC

I'd also add to this list Pandashells https://github.com/robdmc/pandashells - Basically use Pandas in the command line.

Re: Lesser-Known Python Data Analysis Libraries

#16

Post by _dark_matter_ » Tue, Apr 19, 2016, 2:39 PM UTC

zfrenchee wrote: post #14

dguaraglia wrote: post #13

My 2 cents: I would not recommend basing any new work on MRjob. As someone who inherited and has been maintaining a bunch of code that depends on it, the library seems to be barely maintained, support for VPC is only partial and not very well documented, the auditing tools stopped working quite a while ago and tracking the progress/status of EMR jobs is extremely painful (to be fair, this is more of an issue with Ela…

Do you know of any good alternatives? Any way to write MapReduces in python?

https://hadoop.apache.org/docs/r1.2.1/streaming.html

Re: Lesser-Known Python Data Analysis Libraries

#17

Post by elsherbini » Tue, Apr 19, 2016, 2:39 PM UTC

My favorite new (to me) tool is snakemake[0], make files with python 3 support. It allows me to both make my workflow and document it in the same place, hugely helpful for jumping around to different projects or needing to rerun a pipeline with new data. If interested, i recommend taking a look at this tutorial[1] with lots of different snakemake patterns.

[0] https://bitbucket.org/snakemake/snakemake/wiki/Home

[1] https://github.com/leipzig/SandwichesWithSnakemake

Re: Lesser-Known Python Data Analysis Libraries

#18

Post by elsherbini » Tue, Apr 19, 2016, 2:45 PM UTC

plotly is a fantastic tool for plotting. It has a python API [0], but also works from R, matlab, and Julia. It also has support for pandas dataframes and jupyter notebook[1], which is by far the fastest way I've found to make attractive plots. plotlyjs[2] is a fantastic wrapper around d3. So I can go all the way from plotting something quickly from a dataframe to building a totally custom chart.

[0] https://plot.ly/python/

[1] https://plot.ly/ipython-notebooks/cufflinks/

[2] https://plot.ly/javascript/

Re: Lesser-Known Python Data Analysis Libraries

#19

Post by TheLogothete » Tue, Apr 19, 2016, 2:46 PM UTC

I hear a lot of talk about using python for data analysis. I gave up after trying to find a library to do cross tabs. Is there something to make custom tables in python other than prettytables?

Re: Lesser-Known Python Data Analysis Libraries

#20

Post by dguaraglia » Tue, Apr 19, 2016, 2:53 PM UTC

zfrenchee wrote: post #14

dguaraglia wrote: post #13

My 2 cents: I would not recommend basing any new work on MRjob. As someone who inherited and has been maintaining a bunch of code that depends on it, the library seems to be barely maintained, support for VPC is only partial and not very well documented, the auditing tools stopped working quite a while ago and tracking the progress/status of EMR jobs is extremely painful (to be fair, this is more of an issue with Ela…

Do you know of any good alternatives? Any way to write MapReduces in python?

Unfortunately, no. We are slowly moving away to a streaming infrastructure, so I've been mostly trying to "keep it running" until we are done replacing it. Sorry.