Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pandas Should Go Extinct

eddie.codes

Post reply on HN
Print view View original

21–30 of 105 posts

Re: Pandas Should Go Extinct

#21

Post by fortuitous-frog »

crazysim wrote: post #8

Am I crazy or did the OP swap the contents of the posts around accidentally? https://eddie.codes/posts/pandas-should-go-extinct/ https://eddie.codes/posts/source-code-comments/

Yep. Was very confused about the title and URL slug; reached the end waiting for Pandas to be mentioned.

Re: Pandas Should Go Extinct

#22

Post by __eddie__ »

crazysim wrote: post #8

Am I crazy or did the OP swap the contents of the posts around accidentally? https://eddie.codes/posts/pandas-should-go-extinct/ https://eddie.codes/posts/source-code-comments/

Sorry, fixed it now!

Published two posts at the same time and total PEBCAK

Re: Pandas Should Go Extinct

#26

Post by appplication »

__eddie__ wrote: post #14
sjtrny wrote: post #3

> People typically start with Excel and graduate to Pandas somewhere in the GB range. Pandas serves them well into the 10s of GBs range, and then they start hitting memory issues, slow computation, or become frustrated with Pandas’ baroque API. Assumes that a project moves beyond 10s of GBs. I guess 99.9% of projects that import pandas fall well below this threshold.

True, but also if Polars and DuckDB offer a similar experience with the ability to scale beyond that range, why not use them (for new projects)? This isn't a call to arms to rewrite everything in the new shiny, just consider the new shiny for new shiny things

Not to mention less footguns. I used to spend my days unwinding bad habits DS pick up from years of panda abuse.

Credit to pandas for popularizing dataframes in Python, but polars and duckdb are objectively better APIs in addition to their implementation improvements. Agree that it’s time we let it go.

Re: Pandas Should Go Extinct

#27

Post by stephenlf »

Besides the performance benefits, I use Polars at work because it’s just (subjectively) nicer to work with. The “pl.col” API lets you create arbitrary generated/virtual columns anywhere you want, declaratively. You can throw in these column expressions in wherever without actually computing their values and storing that in memory. Very powerful stuff.

Re: Pandas Should Go Extinct

#28

Post by defrost »

__eddie__ wrote: post #22
crazysim wrote: post #8

Am I crazy or did the OP swap the contents of the posts around accidentally? https://eddie.codes/posts/pandas-should-go-extinct/ https://eddie.codes/posts/source-code-comments/

Sorry, fixed it now! Published two posts at the same time and total PEBCAK

No worries, we all make mistakes, at least some one is thinking of the bamboo: https://www.youtube.com/watch?v=WqX2aqXbwB0

Re: Pandas Should Go Extinct

#29

Post by __eddie__ »

appplication wrote: post #26
__eddie__ wrote: post #14

Earlier quoted context omitted.

True, but also if Polars and DuckDB offer a similar experience with the ability to scale beyond that range, why not use them (for new projects)? This isn't a call to arms to rewrite everything in the new shiny, just consider the new shiny for new shiny things

Not to mention less footguns. I used to spend my days unwinding bad habits DS pick up from years of panda abuse. Credit to pandas for popularizing dataframes in Python, but polars and duckdb are objectively better APIs in addition to their implementation improvements. Agree that it’s time we let it go.

Yes, this was something I emphasised more in the slides / my actual talk. But it's just significantly easier to reason about the APIs for both DuckDB and Polars

Re: Pandas Should Go Extinct

#30

Post by akdor1154 »

Sup Eddie, the actual motivating example is we finally nixed Pandas from Data Ingestion, now reading excel files takes 2 seconds instead of 2 minutes. However unfortunately your

> TODO: rewrite this entire service

remains.

Post reply on HN