Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Smallpond – A lightweight data processing framework built on DuckDB and 3FS

github.com

Post reply on HN
Print view View original

31–40 of 75 posts

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#31

Post by BobbyJo »

tarruda wrote: post #29
BobbyJo wrote: post #22

Earlier quoted context omitted.

We ain't even solved garbage collection yet, and you think "back end systems" are going to abstracted away in the next few years?

> We ain't even solved garbage collection yet Can you elaborate on that?

People still write in languages that force you to manage your own memory.

Once performance starts to matter (either due to scale or time requirements) abstractions always have tradeoffs you can't accept.

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#32

Post by didntknowyou »

breadwinner wrote: post #10
shipp02 wrote: post #8

Is the code written by the deepseek model? I should probably give up on being a software engineer if it is.

Give up and become what? Most white collar jobs will be automated in the coming years. You think doctors' jobs are safe?

you can already google the information , the majority of a doctor's value is not in their information but their people and technical skills.

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#33

Post by cavisne »

shipp02 wrote: post #8

Is the code written by the deepseek model? I should probably give up on being a software engineer if it is.

There is a chinese blogpost from 2019 about 3FS so it predates deepseek [1]. It will be interesting to see the benchmarks but I suspect without 3FS smallpond is not that useful (the bottleneck would move to the networked file system).

None of the big US clouds support Infiniband broadly (Azure & Oracle have some support) so 3FS itself is not very useful to US companies who want to use public clouds.

[1] https://www.high-flyer.cn/blog/3fs/

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#34

Post by dang »

Related ongoing thread:

Understanding Smallpond and 3FS - https://news.ycombinator.com/item?id=43232410

also:

DuckDB goes distributed? DeepSeek's smallpond takes on Big Data - https://news.ycombinator.com/item?id=43206964 (no comments there, but some people have been recommending that article)

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#35

Post by jamesblonde »

We are seeing more and more specialized query engines. This is a query engine specialized for training pipelines. It is not general purpose - it is for providing batches of training data at workers. It uses Ray for parallelization. The kind of queries you need are random reads (to implement shuffling across epochs), arrow support (zero copy to Pandas DataFrames), and efficient checkpointing.

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#36

Post by rscho »

didntknowyou wrote: post #32
breadwinner wrote: post #10

Earlier quoted context omitted.

Give up and become what? Most white collar jobs will be automated in the coming years. You think doctors' jobs are safe?

you can already google the information , the majority of a doctor's value is not in their information but their people and technical skills.

Well, googling the info is one thing. But today, medicine is still mostly a know-how profession. Residency is there mostly to transmit know-how.

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#37

Post by purplerabbit »

BobbyJo wrote: post #22
lvl155 wrote: post #17

Looking forward to next few years when we can finally abstract away all the back-end techs.

We ain't even solved garbage collection yet, and you think "back end systems" are going to abstracted away in the next few years?

Maybe they just mean for the type of projects they care about

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#38

Post by pyrolistical »

BobbyJo wrote: post #31
tarruda wrote: post #29

Earlier quoted context omitted.

> We ain't even solved garbage collection yet Can you elaborate on that?

People still write in languages that force you to manage your own memory. Once performance starts to matter (either due to scale or time requirements) abstractions always have tradeoffs you can't accept.

So then, how can garbage collection ever be solved if it’s a trade-off

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#39

Post by threeseed »

lvl155 wrote: post #17

Looking forward to next few years when we can finally abstract away all the back-end techs.

We've had this for at least a decade now.

If you use a cloud provider there are managed solutions for data engineering pipelines.

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#40

Post by BobbyJo »

purplerabbit wrote: post #37
BobbyJo wrote: post #22

Earlier quoted context omitted.

We ain't even solved garbage collection yet, and you think "back end systems" are going to abstracted away in the next few years?

Maybe they just mean for the type of projects they care about

Can't you already just use FaaS and managed persistence?
Post reply on HN