Smallpond – A lightweight data processing framework built on DuckDB and 3FS • Hacker News Board

Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#1

Post by overflowcat » Fri, Feb 28, 2025, 1:56 AM UTC

#2

Post by fastasucan » Fri, Feb 28, 2025, 1:26 PM UTC

What does this do - what is the benefit over DuckDB, Polers etc?

#3

Post by articsputnik » Fri, Feb 28, 2025, 3:06 PM UTC

fastasucan wrote: post #2

What does this do - what is the benefit over DuckDB, Polers etc?

Mehdi just wrote about this. Mainly starting DAGs parallelism using Ray (core) and their filesystem 3FS. See https://mehdio.substack.com/p/duckdb-goes-distributed-deepse....

#4

Post by ilove196884 » Fri, Feb 28, 2025, 5:58 PM UTC

Any benchmark and comparisons?

#5

Post by mritchie712 » Sun, Mar 02, 2025, 2:39 PM UTC

fastasucan wrote: post #2

What does this do - what is the benefit over DuckDB, Polers etc?

I don't think you get any really benefits over duckdb unless your data is 10tb+ or you spin up 3FS (which seem challenging).

#6

Post by HackerThemAll » Sun, Mar 02, 2025, 6:19 PM UTC

DuckDB itself is cool enough, especially when combined with SQLite and/or PostgreSQL, and now this. Thanks DeepSeek!

#7

Post by orlp » Sun, Mar 02, 2025, 6:31 PM UTC

One thing I found peculiar is that for the GraySort benchmark it dispatches to Polars by default to do the actual sorting, not DuckDB: https://github.com/deepseek-ai/smallpond/blob/ed112db42af4d0....

#8

Post by shipp02 » Sun, Mar 02, 2025, 6:53 PM UTC

Is the code written by the deepseek model?

I should probably give up on being a software engineer if it is.

#9

Post by rubenvanwyk » Sun, Mar 02, 2025, 7:08 PM UTC

May Data Engineering content keep on hitting front page HN!

#10

Post by breadwinner » Sun, Mar 02, 2025, 7:09 PM UTC

shipp02 wrote: post #8

Is the code written by the deepseek model? I should probably give up on being a software engineer if it is.

Give up and become what? Most white collar jobs will be automated in the coming years. You think doctors' jobs are safe?