Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dataframes – Julia, R, Python

ajkl.github.io

Post reply on HN
Print view View original

11–20 of 39 posts

Re: Dataframes – Julia, R, Python

#11

Post by alecdbrooks »

sebastianavina wrote: post #10

I've seen a lot about Julia on the last months, it seems like a good language (performance and kind of a nice syntax), For me, what makes R a very good choice is because of RStudio. Being able to play there with your data and save it all for later is one of the biggest reasons to use RLang. The Python equivalent would be emacs org-mode, which is great, but not as graphical as RStudio. Julia seems like a good language…

If you're looking for a nice graphical way to play with data in Python, may I suggest IPython Notebook [0]? It's not always easy to configure, but it's maturing fast and lets you have Python code, Markdown, and graphs in one place, not to mention the 21 other languages available natively or as add-ons[1].

[0]: http://ipython.org/notebook.html.

[1]: https://github.com/ipython/ipython/wiki/IPython%20kernels%20... (I'm counting the two Perl kernels as one language and not counting Calico or the example kernel.)

Re: Dataframes – Julia, R, Python

#12

Post by mapcar »

sebastianavina wrote: post #10

I've seen a lot about Julia on the last months, it seems like a good language (performance and kind of a nice syntax), For me, what makes R a very good choice is because of RStudio. Being able to play there with your data and save it all for later is one of the biggest reasons to use RLang. The Python equivalent would be emacs org-mode, which is great, but not as graphical as RStudio. Julia seems like a good language…

How about PyCharm or Eclipse+PyDev (I've personally heard more praise for the former)? I use emacs and ess or python-mode so can't comment on the IDEs too much but being able to use the same platform for both has been convenient for me.

Re: Dataframes – Julia, R, Python

#13

Post by mapcar »

IndianAstronaut wrote: post #8

Interesting. Although after spwnding a lot of time on data frames, I have grown to like the csv parsers in postgres where I can do a lot of the same things as data frames, but with clean sql instead of the sometimes odd data frame syntax. R also stands out because it is so easy to run a wide variety of statistical methods easily on a data frame.

It's amazing to think that R (or S) had data frames since the 70s and only now are other languages implementing them. There are some quirks of course, and pandas introduced some convenient features. But the R community has also provided its own improvements in the way of data.table, and now, dplyr.

Re: Dataframes – Julia, R, Python

#14

Post by hadley »

peatmoss wrote: post #7

For dataframe-like operations, I've started wondering why more languages don't take the dplyr approach and simply default to using something like SQLite under the hood. Granted, I are no super data genius, but every time I start cracking a little into the internals of a dataframe implementation, I get the sinking feeling that SQL databases have already done the hard work of indexes and efficient data structures.

I thought that before writing dplyr, but now I see that there a big differences. Relational databases are designed to work with large datasets on disk, and to accept changes very rapidly. The demands for in memory data analytics are quite differnt. Columnar data stores are a better fit, but it's pretty easy to bang out efficient code for in memory data; it's much harder to work with out of memory data.

Re: Dataframes – Julia, R, Python

#15

Post by cschmidt »

alecdbrooks wrote: post #11
sebastianavina wrote: post #10

I've seen a lot about Julia on the last months, it seems like a good language (performance and kind of a nice syntax), For me, what makes R a very good choice is because of RStudio. Being able to play there with your data and save it all for later is one of the biggest reasons to use RLang. The Python equivalent would be emacs org-mode, which is great, but not as graphical as RStudio. Julia seems like a good language…

If you're looking for a nice graphical way to play with data in Python, may I suggest IPython Notebook [0]? It's not always easy to configure, but it's maturing fast and lets you have Python code, Markdown, and graphs in one place, not to mention the 21 other languages available natively or as add-ons[1]. [0]: http://ipython.org/notebook.html . [1]: https://github.com/ipython/ipython/wiki/IPython%20kernels%20... (I'm…

There is a version of iPython that works with Julia too

Re: Dataframes – Julia, R, Python

#16

Post by peatmoss »

hadley wrote: post #14
peatmoss wrote: post #7

For dataframe-like operations, I've started wondering why more languages don't take the dplyr approach and simply default to using something like SQLite under the hood. Granted, I are no super data genius, but every time I start cracking a little into the internals of a dataframe implementation, I get the sinking feeling that SQL databases have already done the hard work of indexes and efficient data structures.

I thought that before writing dplyr, but now I see that there a big differences. Relational databases are designed to work with large datasets on disk, and to accept changes very rapidly. The demands for in memory data analytics are quite differnt. Columnar data stores are a better fit, but it's pretty easy to bang out efficient code for in memory data; it's much harder to work with out of memory data.

> large datasets on disk

I saw this benchmark a while back comparing Pandas to SQLite in-memory databases. While Pandas did edge out SQLite in several areas, it was by well under an order of magnitude: http://wesmckinney.com/blog/?p=414

Pretty solid performance plus the ability to work with large datasets on disk seemed like a pretty big win to me. I could imagine a set of SQLite extensions (a la spatialite) that could further optimize for various data.frame use cases. As an added bonus, the same libraries would be very portable between different languages--even languages that don't currently have something like dataframes.

EDIT: What I don't know about is memory efficiency. Perhaps SQLite isn't, but I'd not bet against?

Re: Dataframes – Julia, R, Python

#17

Post by IndianAstronaut »

peatmoss wrote: post #16
hadley wrote: post #14

Earlier quoted context omitted.

I thought that before writing dplyr, but now I see that there a big differences. Relational databases are designed to work with large datasets on disk, and to accept changes very rapidly. The demands for in memory data analytics are quite differnt. Columnar data stores are a better fit, but it's pretty easy to bang out efficient code for in memory data; it's much harder to work with out of memory data.

> large datasets on disk I saw this benchmark a while back comparing Pandas to SQLite in-memory databases. While Pandas did edge out SQLite in several areas, it was by well under an order of magnitude: http://wesmckinney.com/blog/?p=414 Pretty solid performance plus the ability to work with large datasets on disk seemed like a pretty big win to me. I could imagine a set of SQLite extensions (a la spatialite) that cou…

I personally switched from Pandas to SQL. While Postgres is a heavy duty database for large production operations, it is fully capable of doing day to day analysis of CSV files with nice SQL syntax.

There were two reasons for the switch. SQL syntax is cleaner and more well understood by others. The second is if you get a dataset bigger than memory, you aren't stuck.

Re: Dataframes – Julia, R, Python

#18

Post by ajinkyakale »

sebastianavina wrote: post #10

I've seen a lot about Julia on the last months, it seems like a good language (performance and kind of a nice syntax), For me, what makes R a very good choice is because of RStudio. Being able to play there with your data and save it all for later is one of the biggest reasons to use RLang. The Python equivalent would be emacs org-mode, which is great, but not as graphical as RStudio. Julia seems like a good language…

I use ijulia which is based on python notebooks. Try juliabox.org which is a notebook (and more) offering by the julia folks. Then there is juno and julia studio if you are inclined towards an rstudio like interface.

Re: Dataframes – Julia, R, Python

#19

Post by ajinkyakale »

mapcar wrote: post #13
IndianAstronaut wrote: post #8

Interesting. Although after spwnding a lot of time on data frames, I have grown to like the csv parsers in postgres where I can do a lot of the same things as data frames, but with clean sql instead of the sometimes odd data frame syntax. R also stands out because it is so easy to run a wide variety of statistical methods easily on a data frame.

It's amazing to think that R (or S) had data frames since the 70s and only now are other languages implementing them. There are some quirks of course, and pandas introduced some convenient features. But the R community has also provided its own improvements in the way of data.table, and now, dplyr.

Data.table package by matt dowle definitely deserves a mention! Its fast and I like the indexing functonalities it provides. The benchmark timings are pretty impressive.

Re: Dataframes – Julia, R, Python

#20

Post by jzwinck »

The example which claims to get "all the rows from 50th row to the 55th row" is broken, since Python is zero-based whereas Julia and R are one-based. The 50th row in Python is at index 49, so the code is not equivalent between the examples.
Post reply on HN