Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Loading Data into Pandas: Tips and Tricks You May or May Not Know

dataground.io

Post reply on HN
Print view View original

1–10 of 12 posts

Re: Loading Data into Pandas: Tips and Tricks You May or May Not Know

#2

Post by ashwal »

Some good tips in here, I've find myself reaching for JSON/excel methods often.

Despite using it for years, I still haven't decided if pandas is poorly architected or if the clunkiness (for lack of better of term) is a result of the inherent difficulty of the tasks.

Re: Loading Data into Pandas: Tips and Tricks You May or May Not Know

#3

Post by hansvm »

ashwal wrote: post #2

Some good tips in here, I've find myself reaching for JSON/excel methods often. Despite using it for years, I still haven't decided if pandas is poorly architected or if the clunkiness (for lack of better of term) is a result of the inherent difficulty of the tasks.

I seem to remember the author awhile back writing about how it was their first major project and there were a ton of things they'd learned and would like to change.

Re: Loading Data into Pandas: Tips and Tricks You May or May Not Know

#6

Post by lettergram »

I spent some time working on something called DataProfiler python library

https://github.com/capitalone/DataProfiler

The gist is that you can point to any common dataset and load it directly into pandas.

from dataprofiler import Data

data = Data("your_file.csv") # Auto-Detect & Load: CSV, AVRO, Parquet, JSON, Text, URL

I simply hate dealing with loading data, so it's my go-to.

Re: Loading Data into Pandas: Tips and Tricks You May or May Not Know

#7

Post by Helmut10001 »

Not to taunt about the article, but the most important pandas parameters to me are `iterator=True` and `chunksize=x`, for streamed processing. Here's an example for processing a CSV file with 400 Million latitude and longitude coordinates.[1]

[1]: https://ad.vgiscience.org/twitter-global-preview/00_Twitter_...

Re: Loading Data into Pandas: Tips and Tricks You May or May Not Know

#8

Post by wenc »

I’ve mostly replaced pd.read_csv and pd.read_parquet with duckdb.query(“select * from ‘x.csv’) or duckdb.query(“select * from ‘y/*.parquet’).

It’s much faster because DuckDB is vectorized. The result is a Pandas dataframe.

Querying the Pandas dataframe from DuckDB is faster than querying it with Pandas itself.

Re: Loading Data into Pandas: Tips and Tricks You May or May Not Know

#9

Post by jwilber »

Helmut10001 wrote: post #7

Not to taunt about the article, but the most important pandas parameters to me are `iterator=True` and `chunksize=x`, for streamed processing. Here's an example for processing a CSV file with 400 Million latitude and longitude coordinates.[1] [1]: https://ad.vgiscience.org/twitter-global-preview/00_Twitter_...

This is a great mention, ty!
Post reply on HN