Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-Directed Pandas Crash Course

kellyfoulk.herokuapp.com

Post reply on HN
Print view View original

11–20 of 43 posts

Re: Self-Directed Pandas Crash Course

#11

Post by dehrmann »

systemvoltage wrote: post #3

Pandas is basically impossible for me to use without dozens of Google searches after being familiar with it for over 7 years. Ofcourse, I don't use it daily but its one of those pieces of software that has a very non-intuitive API. Does any one find it difficult to use it or its just me? In particular, I find this answer infuriating [1]. I've come across it so many times. Look, I have a CSV with 200 rows and I need t…

I'm in this boat, and I'm decent at SQL and find the Numpy API to be pretty intuitive, so I'm not sure what it is.

Re: Self-Directed Pandas Crash Course

#12

Post by systemvoltage »

dwohnitmok wrote: post #8
systemvoltage wrote: post #3

Pandas is basically impossible for me to use without dozens of Google searches after being familiar with it for over 7 years. Ofcourse, I don't use it daily but its one of those pieces of software that has a very non-intuitive API. Does any one find it difficult to use it or its just me? In particular, I find this answer infuriating [1]. I've come across it so many times. Look, I have a CSV with 200 rows and I need t…

I understand where you're coming from (yes yes there's a theoretically "good" way of doing this, but come on, why can't I just do the simplest thing), but I also sympathize with the spirit of that SO answer (although I agree that its presentation is wanting). Pandas is really a DSL unto itself and is heavily influenced by R, where the same dynamic happens. Programmers coming from a background where procedural control…

SQL is more natural to me for this sort of declarative DSL. Is there a Pandas-like package that can use in-memory tables and accept SQL queries?

About `pd.read_sql()`. That's totally awesome.

I realized that Pandas is a very useful tool for many thousands of developers, I only have a problem with its interface. Obviously, I've been using it for many years for a reason!

Re: Self-Directed Pandas Crash Course

#13

Post by smabie »

chaps wrote: post #7
systemvoltage wrote: post #3

Pandas is basically impossible for me to use without dozens of Google searches after being familiar with it for over 7 years. Ofcourse, I don't use it daily but its one of those pieces of software that has a very non-intuitive API. Does any one find it difficult to use it or its just me? In particular, I find this answer infuriating [1]. I've come across it so many times. Look, I have a CSV with 200 rows and I need t…

100% my experience whenever I work with it, and I've been working with it on and off for about five years. I get its appeal -- it fits that fuzzy place where a database is too heavy, but vanilla python is cumbersome... but damn, is it tough to work with. With almost no fail, I always seem to just scrap the pandas code in favor of vanilla python, either from usability issues, or from yet another a library's false prom…

Pandas isn't great, but I find it significantly nicer to use than normal Python. Why do you want to write a 100 lines of dumb loopy code when you can probably solve the same problem with a couple lines of Pandas?

If you really just want to use loops and stuff (which I would discourage) just use a list or a dict or something.

Re: Self-Directed Pandas Crash Course

#14

Post by smabie »

jzwinck wrote: post #4
systemvoltage wrote: post #3

Pandas is basically impossible for me to use without dozens of Google searches after being familiar with it for over 7 years. Ofcourse, I don't use it daily but its one of those pieces of software that has a very non-intuitive API. Does any one find it difficult to use it or its just me? In particular, I find this answer infuriating [1]. I've come across it so many times. Look, I have a CSV with 200 rows and I need t…

Here are two options which various people might find intuitive: 1. Make a good old numeric for loop like "for ii in range(len(df))" with df.iloc or df.ix, etc. 2. Use df.apply() to create a new DataFrame with your changes. Both of these are mentioned in brief answers to that SO post. But not in the accepted answer. A lot of the answers focus on the most efficient ways to do things, even though the question was very b…

It's not even about performance, writing vectorized code is just superior. The array language model is very powerful and you can literally write a couple lines of code that would take hundreds of lines of normal Python.

That being said, Pandas isn't a very good array language. I really like kdb+/q and find it better and more expressive than almost any other language I've used.

Re: Self-Directed Pandas Crash Course

#15

Post by smabie »

dwohnitmok wrote: post #8
systemvoltage wrote: post #3

Pandas is basically impossible for me to use without dozens of Google searches after being familiar with it for over 7 years. Ofcourse, I don't use it daily but its one of those pieces of software that has a very non-intuitive API. Does any one find it difficult to use it or its just me? In particular, I find this answer infuriating [1]. I've come across it so many times. Look, I have a CSV with 200 rows and I need t…

I understand where you're coming from (yes yes there's a theoretically "good" way of doing this, but come on, why can't I just do the simplest thing), but I also sympathize with the spirit of that SO answer (although I agree that its presentation is wanting). Pandas is really a DSL unto itself and is heavily influenced by R, where the same dynamic happens. Programmers coming from a background where procedural control…

I actually don't like Pandas but I think it's pretty obvious that the array language style is an incredibly powerful way to manipulate blocks of multiple dimensional data. People here are complaining about not being able to use for loops, but why would you want to use them in the first place? Like let's say you want to add two vectors together. Looping over each index and adding them and assigning them to a third vector element by element is not only I expressive (and like everything else written in Python, computational inefficient), but it is an conceptually inappropriate solution to the problem.

You are operating on n-dimensional arrays, not elements, so your need to write code that expresses that intent.

Anyone who thinks that anything besides writing "a + b" to add two matrices together is a good or simple solution is crazy.

Operate at a higher conceptual level. Don't use loops. Transform and compose your data, not your datums.

Re: Self-Directed Pandas Crash Course

#17

Post by ab111111111 »

smabie wrote: post #13
chaps wrote: post #7

Earlier quoted context omitted.

100% my experience whenever I work with it, and I've been working with it on and off for about five years. I get its appeal -- it fits that fuzzy place where a database is too heavy, but vanilla python is cumbersome... but damn, is it tough to work with. With almost no fail, I always seem to just scrap the pandas code in favor of vanilla python, either from usability issues, or from yet another a library's false prom…

Pandas isn't great, but I find it significantly nicer to use than normal Python. Why do you want to write a 100 lines of dumb loopy code when you can probably solve the same problem with a couple lines of Pandas? If you really just want to use loops and stuff (which I would discourage) just use a list or a dict or something.

As you indicate, Pandas is effectively another language, distinct from regular Python. E.g. it doesn't really use loops. So I think what irritates a lot of people about pandas is that they think "Cool. I can solve this complex problem in a couple of lines of Python code with pandas." then get irritated when they find that the couple of lines of pandas they need to use is "Incomprehensible pandas gibberish" instead of the familiar Python code they were expecting.

Re: Self-Directed Pandas Crash Course

#18

Post by [deleted] »

smabie wrote: post #13
chaps wrote: post #7

Earlier quoted context omitted.

100% my experience whenever I work with it, and I've been working with it on and off for about five years. I get its appeal -- it fits that fuzzy place where a database is too heavy, but vanilla python is cumbersome... but damn, is it tough to work with. With almost no fail, I always seem to just scrap the pandas code in favor of vanilla python, either from usability issues, or from yet another a library's false prom…

Pandas isn't great, but I find it significantly nicer to use than normal Python. Why do you want to write a 100 lines of dumb loopy code when you can probably solve the same problem with a couple lines of Pandas? If you really just want to use loops and stuff (which I would discourage) just use a list or a dict or something.

[deleted]

Re: Self-Directed Pandas Crash Course

#19

Post by chaps »

smabie wrote: post #13
chaps wrote: post #7

Earlier quoted context omitted.

100% my experience whenever I work with it, and I've been working with it on and off for about five years. I get its appeal -- it fits that fuzzy place where a database is too heavy, but vanilla python is cumbersome... but damn, is it tough to work with. With almost no fail, I always seem to just scrap the pandas code in favor of vanilla python, either from usability issues, or from yet another a library's false prom…

Pandas isn't great, but I find it significantly nicer to use than normal Python. Why do you want to write a 100 lines of dumb loopy code when you can probably solve the same problem with a couple lines of Pandas? If you really just want to use loops and stuff (which I would discourage) just use a list or a dict or something.

"If you really just want to use loops and stuff (which I would discourage) just use a list or a dict or something."

Like my post said, I favor vanilla python over pandas... so yes, I use lists, dicts and somethings. FWIW, though, my workflow pushes everything into postgres, and things that would normally go into pandas are just accessed through SQL through and with helper functions.

Re: Self-Directed Pandas Crash Course

#20

Post by carlineng »

systemvoltage wrote: post #12
dwohnitmok wrote: post #8

Earlier quoted context omitted.

I understand where you're coming from (yes yes there's a theoretically "good" way of doing this, but come on, why can't I just do the simplest thing), but I also sympathize with the spirit of that SO answer (although I agree that its presentation is wanting). Pandas is really a DSL unto itself and is heavily influenced by R, where the same dynamic happens. Programmers coming from a background where procedural control…

SQL is more natural to me for this sort of declarative DSL. Is there a Pandas-like package that can use in-memory tables and accept SQL queries? About `pd.read_sql()`. That's totally awesome. I realized that Pandas is a very useful tool for many thousands of developers, I only have a problem with its interface. Obviously, I've been using it for many years for a reason!

DuckDB is a really promising project for just that: https://duckdb.org/
Post reply on HN