Self-Directed Pandas Crash Course
kellyfoulk.herokuapp.com
Self-Directed Pandas Crash Course
1–10 of 43 posts
Re: Self-Directed Pandas Crash Course
#2I like the outline you have here for a crash course on Pandas. I've been tinkering with it myself off and on for a while but have been wanting to to really dig in lately for the same reasons you mention. Just a couple of nitpicks:
1. The style of your website makes it pretty much impossible to tell if a bit of text is linked anywhere. I only figured it out after clicking on the word "here" in the first paragraph and coming back and clicking on essentially everything. None of this happened until after I visited the site on my desktop instead of my mobile and slowed down to read things carefully.
2. It would be great to get a link to the denvergov.org data set, or corresponding area of the site.
Re: Self-Directed Pandas Crash Course
#3Pandas is basically impossible for me to use without dozens of Google searches after being familiar with it for over 7 years. Ofcourse, I don't use it daily but its one of those pieces of software that has a very non-intuitive API. Does any one find it difficult to use it or its just me?
In particular, I find this answer infuriating [1]. I've come across it so many times. Look, I have a CSV with 200 rows and I need to loop through them in the most intuitive way. Sure, its not optimal but I don't want fast code. I have a mental model of how to modify this dataframe. Let me do it, please.
Re: Self-Directed Pandas Crash Course
#4Pandas is basically impossible for me to use without dozens of Google searches after being familiar with it for over 7 years. Ofcourse, I don't use it daily but its one of those pieces of software that has a very non-intuitive API. Does any one find it difficult to use it or its just me? In particular, I find this answer infuriating [1]. I've come across it so many times. Look, I have a CSV with 200 rows and I need t…
Here are two options which various people might find intuitive:
1. Make a good old numeric for loop like "for ii in range(len(df))" with df.iloc or df.ix, etc.
2. Use df.apply() to create a new DataFrame with your changes.
Both of these are mentioned in brief answers to that SO post. But not in the accepted answer. A lot of the answers focus on the most efficient ways to do things, even though the question was very basic and did not imply the data were large.
Re: Self-Directed Pandas Crash Course
#5Pandas is basically impossible for me to use without dozens of Google searches after being familiar with it for over 7 years. Ofcourse, I don't use it daily but its one of those pieces of software that has a very non-intuitive API. Does any one find it difficult to use it or its just me? In particular, I find this answer infuriating [1]. I've come across it so many times. Look, I have a CSV with 200 rows and I need t…
I just read the GitHub repo mentioned in the article and it looks like gibberish to me.
Pandas to me are still big lazy black and white bears eating bamboo, until someone can point me to something more intelligible.
Re: Self-Directed Pandas Crash Course
#6I like the outline you have here for a crash course on Pandas. I've been tinkering with it myself off and on for a while but have been wanting to to really dig in lately for the same reasons you mention. Just a couple of nitpicks: 1. The style of your website makes it pretty much impossible to tell if a bit of text is linked anywhere. I only figured it out after clicking on the word "here" in the first paragraph and…
He links it in the first sentence under Resources. As you mention, it's a "here" link with no styling.
One shouldn't have to, but FWIW the HTML of his page is very clean so you can see all of the links by viewing the source.
Re: Self-Directed Pandas Crash Course
#7Pandas is basically impossible for me to use without dozens of Google searches after being familiar with it for over 7 years. Ofcourse, I don't use it daily but its one of those pieces of software that has a very non-intuitive API. Does any one find it difficult to use it or its just me? In particular, I find this answer infuriating [1]. I've come across it so many times. Look, I have a CSV with 200 rows and I need t…
100% my experience whenever I work with it, and I've been working with it on and off for about five years. I get its appeal -- it fits that fuzzy place where a database is too heavy, but vanilla python is cumbersome... but damn, is it tough to work with. With almost no fail, I always seem to just scrap the pandas code in favor of vanilla python, either from usability issues, or from yet another a library's false promise that it works well with dataframes. So many hours lost in using pandas.
That said, one of its best features, and probably the only thing I use it for these days is, `pd.read_sql(sqlstr, conn).to_csv(fp)`. This is far less cumbersome than using psycopg2.
Edit: for charting these days, similar to OP's visualiztions, I highly recommend vega-lite.
Re: Self-Directed Pandas Crash Course
#8Pandas is basically impossible for me to use without dozens of Google searches after being familiar with it for over 7 years. Ofcourse, I don't use it daily but its one of those pieces of software that has a very non-intuitive API. Does any one find it difficult to use it or its just me? In particular, I find this answer infuriating [1]. I've come across it so many times. Look, I have a CSV with 200 rows and I need t…
I understand where you're coming from (yes yes there's a theoretically "good" way of doing this, but come on, why can't I just do the simplest thing), but I also sympathize with the spirit of that SO answer (although I agree that its presentation is wanting).
Pandas is really a DSL unto itself and is heavily influenced by R, where the same dynamic happens. Programmers coming from a background where procedural control flow constructs are basically second nature bump up against statisticians for whom array-based programming (in the form of overloaded mathematical notation acting on both scalar and vector values) is second nature.
R and pandas are both very array-oriented programming languages (the most extreme example of this might be early-era APL) and it's really going against the grain to implement things with explicit iteration.
It's kind of like trying to program in Python without using loops or list comprehensions and asking just how to do everything in recursion. You can... but someone is bound to point out that doing everything with recursion (and the concomitant trampolines to prevent stack overflows) is not the Pythonic way.
(Also separately @dang, I feel like I'm running into a very minor bug with time stamps, where when I'm composing this reply I get "9 hours ago" for systemvoltage, but in the main thread I get "3 hours ago")
Re: Self-Directed Pandas Crash Course
#9Here's another recent resource: https://www.gormanalysis.com/blog/python-pandas-for-your-gra...
Re: Self-Directed Pandas Crash Course
#10Pandas is basically impossible for me to use without dozens of Google searches after being familiar with it for over 7 years. Ofcourse, I don't use it daily but its one of those pieces of software that has a very non-intuitive API. Does any one find it difficult to use it or its just me? In particular, I find this answer infuriating [1]. I've come across it so many times. Look, I have a CSV with 200 rows and I need t…
Same here. Why re-invent SQL as a weird object system?
I've also recently found that using the sqlite3 command line tool increases my productivity when doing data sciency stuff a lot. It's a super fast, super simple way of making sense of CSV data, especially for the selecting an joining operations that are so unintuitive in pandas. Once that's done, I can dump the results into Jupyter or RStudio for transformation and/or visualization.
Another personal productivity win I've discovered is using two-line python scripts to write really long and repetitive SQL commands (select count a, b, c... from huge_table where d... and e... and f...; etc.) and then running them in SQLite.