Pandas 3.0

Post by mynameisash » Wed, Jan 28, 2026, 2:39 PM UTC

iugtmkbdfil834 wrote: post #29

lvl155 wrote: post #19

Earlier quoted context omitted.

Went from pandas to polars to duckdb. As mentioned elsewhere SQL is the most readable for me and LLM does most of the coding on my end (quant). So I need it at the most readable and rudimentary/step-wise level. OT, but I can’t imagine data science being a job category for too long. It’s got to be one of the first to go in AI age especially since the market is so saturated with mediocre talents.

This is interesting. I wanted to dig into it a little since I am not sure I am following the logic of that statement. Do you mean that AI would take over the field, because by default most people there are already not producing anything that a simple 'talk to data' LLM won't deliver?

Post by Havoc » Wed, Jan 28, 2026, 2:44 PM UTC

OutOfHere wrote: post #9

s/impactfull/impactful

Post by noo_u » Wed, Jan 28, 2026, 2:58 PM UTC

skylurk wrote: post #40

noo_u wrote: post #38

Earlier quoted context omitted.

Polars took a lot of ideas from Pandas and made them better - calling it "inferior in every way" is all sorts of disrespectful :P Unfortunately, there are a lot of third party libraries that work with Pandas that do not work with Polars, so the switch, even for new projects, should be done with that in mind.

Luckily, polars has .to_pandas() so you can still pass pandas dataframes to the libraries that really are still stuck on that interface. I maintain one of those libraries and everything is polars internally.

Post by skylurk » Wed, Jan 28, 2026, 3:00 PM UTC

lairv wrote: post #35

edschofield wrote: post #18

The design of Pandas is inferior in every way to Polars: API, memory use, speed, expressiveness. Pandas has been strictly worse since late 2023 and will never close the gap. Polars is multithreaded by default, written in a low-level language, has a powerful query engine, supports lazy, out-of memory execution, and isn’t constrained by any compatibility concerns with a warty, eager-only API and pre-Arrow data types th…

I would agree if not for the fact that polars is not compatible with Python multiprocessing when using the default fork method, the following script hangs forever (the pandas equivalent runs): import polars as pl from concurrent.futures import ProcessPoolExecutor pl.DataFrame({"a": [1,2,3], "b": [4,5,6]}).write_parquet("test.parquet") def read_parquet(): x = pl.read_parquet("test.parquet") print(x.shape) with Process…

  import polars as pl
  
  pl.DataFrame({"a": [1, 2, 3]}).write_parquet("test.parquet")
  
  
  def print_shape(df: pl.DataFrame) -> pl.DataFrame:
      print(df.shape)
      return df
  
  
  lazy_frames = [
      pl.scan_parquet("test.parquet")
      .map_batches(print_shape)
      for _ in range(100)
  ]
  pl.collect_all(lazy_frames, comm_subplan_elim=False)

Post by ritchie46 » Wed, Jan 28, 2026, 3:02 PM UTC

lairv wrote: post #35

edschofield wrote: post #18

The design of Pandas is inferior in every way to Polars: API, memory use, speed, expressiveness. Pandas has been strictly worse since late 2023 and will never close the gap. Polars is multithreaded by default, written in a low-level language, has a powerful query engine, supports lazy, out-of memory execution, and isn’t constrained by any compatibility concerns with a warty, eager-only API and pre-Arrow data types th…

I would agree if not for the fact that polars is not compatible with Python multiprocessing when using the default fork method, the following script hangs forever (the pandas equivalent runs): import polars as pl from concurrent.futures import ProcessPoolExecutor pl.DataFrame({"a": [1,2,3], "b": [4,5,6]}).write_parquet("test.parquet") def read_parquet(): x = pl.read_parquet("test.parquet") print(x.shape) with Process…