Live data from Hacker News

DeepSeek's smallpond: Bringing Distributed Computing to DuckDB

mehdio.substack.com

21–25 of 25 posts

Re: DeepSeek's smallpond: Bringing Distributed Computing to DuckDB

#21
post #15

Isn’t the whole point of DuckDB is that it’s not distributed?

1. Our technology isn't powerful enough, we need to scale by distributing it.

2. The distributed technology is powerful but complex, and most user don't need most of what it offers. Let's build a simple solution.

3. GOTO 1

Re: DeepSeek's smallpond: Bringing Distributed Computing to DuckDB

#22

Deepseek is the real "open " that the world needed. Via these three projects, Deepseek has addressed not only efficient AI but also distributed computing: 1. smallpond: https://github.com/deepseek-ai/smallpond 2. 3fs: https://github.com/deepseek-ai/3FS 3. deepep: https://github.com/deepseek-ai/DeepEP

thank goodness, we’ve had nothing open to do efficient distributed computing with for years!

At least there hasn't been anything for distributed DuckDB before it afaik. For anyone with a substantial DuckDB project, they might now go distributed without having to rewrite it in something else.

Re: DeepSeek's smallpond: Bringing Distributed Computing to DuckDB

#24
post #18

I'm not massively knowledgable about the ins and outs of DeepSeek, but I think I'm in the right place to ask. My understanding is DeepSeek: - Created comparable LLM performance for a fraction of the cost of OpenAI using more off-the-shelf hardware. - Seem to be open sourcing lots of distributed stuff. My question is, are those two things related? Did distributed computing allow the AI model somehow? If so how? Or is…

These type of models need to be trained across thousands of GPUs, which requires distributed engineering on a much higher level than "normal" distributed systems. This is true for DeepSeek as well as for others. There are a few companies giving insights or open-sourcing their approaches, such as Databricks/Mosaic and, well, DeepSeek. The latter also did some particularly clever stuff, but if you look into details so…

Thanks, that's a really great/helpful explanation!
Post reply on HN