Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek's smallpond: Bringing Distributed Computing to DuckDB

mehdio.substack.com

Post reply on HN
Print view View original

11–20 of 25 posts

Re: DeepSeek's smallpond: Bringing Distributed Computing to DuckDB

#11

Post by greenavocado »

antupis wrote: post #3
ogarten wrote: post #2

Looks like we are approaching the "distributed" phase of the distributed-centralized computing cycle :) Not saying this is bad, but it's just interesting to see after being in the industry for 8 years.

Was it already happening when platforms started supporting stuff like Iceberg? But is kinda nice to see things like Snowflake have definitely their place on the ecosystem but too often at margins especially with huge workloads Snowflake creates more issues than solves them

Were you there when we had to work with our data in Teradata and SAS and hundreds of multi hundred MB Excel spreadsheets containing analytical data? 30+ minute queries were the norm. Snowflake was a breath of fresh air.

Re: DeepSeek's smallpond: Bringing Distributed Computing to DuckDB

#12

Post by skeeter2020 »

swyx wrote: post #7
OutOfHere wrote: post #5

Deepseek is the real "open " that the world needed. Via these three projects, Deepseek has addressed not only efficient AI but also distributed computing: 1. smallpond: https://github.com/deepseek-ai/smallpond 2. 3fs: https://github.com/deepseek-ai/3FS 3. deepep: https://github.com/deepseek-ai/DeepEP

how many companies will actually adopt 3FS now that it's open source? not a hater, just know that theres a lot of hurdles to adoption even if something if open source - for example not being an industry standard. i dont know a ton about this space - what is the main alternative?

to me this seems to target a pretty small audience: very big data and specific problem domains, you need killer devops chops, expensive & specialized infrastructure and a desire to build out on bleeding edge architecture. I'd suspect most with these characteristics will stick with what they've got, "medium Big Data" companies should probably go with hsoted services and the rest of use stick with a single node DuckDB.

Re: DeepSeek's smallpond: Bringing Distributed Computing to DuckDB

#13

Post by 0cf8612b2e1e »

skeeter2020 wrote: post #12
swyx wrote: post #7

Earlier quoted context omitted.

how many companies will actually adopt 3FS now that it's open source? not a hater, just know that theres a lot of hurdles to adoption even if something if open source - for example not being an industry standard. i dont know a ton about this space - what is the main alternative?

to me this seems to target a pretty small audience: very big data and specific problem domains, you need killer devops chops, expensive & specialized infrastructure and a desire to build out on bleeding edge architecture. I'd suspect most with these characteristics will stick with what they've got, "medium Big Data" companies should probably go with hsoted services and the rest of use stick with a single node DuckDB.

Bingo. Very few organizations have petabytes of data on which they are trying to efficiently process for machine learning. Such organizations already have personnel and technology in place offering some kind of solution. Maybe this is an improvement, but it is quite unlikely to be offering new capabilities to such teams.

Re: DeepSeek's smallpond: Bringing Distributed Computing to DuckDB

#14

Post by data_marsupial »

greenavocado wrote: post #11
antupis wrote: post #3

Earlier quoted context omitted.

Was it already happening when platforms started supporting stuff like Iceberg? But is kinda nice to see things like Snowflake have definitely their place on the ecosystem but too often at margins especially with huge workloads Snowflake creates more issues than solves them

Were you there when we had to work with our data in Teradata and SAS and hundreds of multi hundred MB Excel spreadsheets containing analytical data? 30+ minute queries were the norm. Snowflake was a breath of fresh air.

I work with Teradata every day and can query years of event data in seconds.

Re: DeepSeek's smallpond: Bringing Distributed Computing to DuckDB

#16

Post by benrutter »

I'm not massively knowledgable about the ins and outs of DeepSeek, but I think I'm in the right place to ask. My understanding is DeepSeek:

- Created comparable LLM performance for a fraction of the cost of OpenAI using more off-the-shelf hardware.

- Seem to be open sourcing lots of distributed stuff.

My question is, are those two things related? Did distributed computing allow the AI model somehow? If so how? Or is it not that simple?

Re: DeepSeek's smallpond: Bringing Distributed Computing to DuckDB

#17

Post by datadrivenangel »

0cf8612b2e1e wrote: post #13
skeeter2020 wrote: post #12

Earlier quoted context omitted.

to me this seems to target a pretty small audience: very big data and specific problem domains, you need killer devops chops, expensive & specialized infrastructure and a desire to build out on bleeding edge architecture. I'd suspect most with these characteristics will stick with what they've got, "medium Big Data" companies should probably go with hsoted services and the rest of use stick with a single node DuckDB.

Bingo. Very few organizations have petabytes of data on which they are trying to efficiently process for machine learning. Such organizations already have personnel and technology in place offering some kind of solution. Maybe this is an improvement, but it is quite unlikely to be offering new capabilities to such teams.

And the organizations that get large enough to be sad with DuckDB performance will have options like MotherDuck for cloud hosting

Re: DeepSeek's smallpond: Bringing Distributed Computing to DuckDB

#18

Post by zwaps »

benrutter wrote: post #16

I'm not massively knowledgable about the ins and outs of DeepSeek, but I think I'm in the right place to ask. My understanding is DeepSeek: - Created comparable LLM performance for a fraction of the cost of OpenAI using more off-the-shelf hardware. - Seem to be open sourcing lots of distributed stuff. My question is, are those two things related? Did distributed computing allow the AI model somehow? If so how? Or is…

These type of models need to be trained across thousands of GPUs, which requires distributed engineering on a much higher level than "normal" distributed systems.

This is true for DeepSeek as well as for others. There are a few companies giving insights or open-sourcing their approaches, such as Databricks/Mosaic and, well, DeepSeek. The latter also did some particularly clever stuff, but if you look into details so did Mosaic.

OpenAI and Anthropic likely have distributed tools of even larger sophistication. They are just not open source.

Post reply on HN