DeepSeek's smallpond: Bringing Distributed Computing to DuckDB
mehdio.substack.com
DeepSeek's smallpond: Bringing Distributed Computing to DuckDB
1–10 of 25 posts
Re: DeepSeek's smallpond: Bringing Distributed Computing to DuckDB
#2Looks like we are approaching the "distributed" phase of the distributed-centralized computing cycle :)
Not saying this is bad, but it's just interesting to see after being in the industry for 8 years.
Re: DeepSeek's smallpond: Bringing Distributed Computing to DuckDB
#3Looks like we are approaching the "distributed" phase of the distributed-centralized computing cycle :) Not saying this is bad, but it's just interesting to see after being in the industry for 8 years.
Was it already happening when platforms started supporting stuff like Iceberg? But is kinda nice to see things like Snowflake have definitely their place on the ecosystem but too often at margins especially with huge workloads Snowflake creates more issues than solves them
Re: DeepSeek's smallpond: Bringing Distributed Computing to DuckDB
#4Looks like we are approaching the "distributed" phase of the distributed-centralized computing cycle :) Not saying this is bad, but it's just interesting to see after being in the industry for 8 years.
Was it already happening when platforms started supporting stuff like Iceberg? But is kinda nice to see things like Snowflake have definitely their place on the ecosystem but too often at margins especially with huge workloads Snowflake creates more issues than solves them
Yes, not saying this is bad at all, just kind of funny. When you think about it it makes sense though. Why wouldn't want someone have a possibility to distribute an efficient engine.
Re: DeepSeek's smallpond: Bringing Distributed Computing to DuckDB
#5Deepseek is the real "open" that the world needed. Via these three projects, Deepseek has addressed not only efficient AI but also distributed computing:
1. smallpond: https://github.com/deepseek-ai/smallpond
2. 3fs: https://github.com/deepseek-ai/3FS
3. deepep: https://github.com/deepseek-ai/DeepEP
Re: DeepSeek's smallpond: Bringing Distributed Computing to DuckDB
#6It was already on HN recently:
Re: DeepSeek's smallpond: Bringing Distributed Computing to DuckDB
#7Deepseek is the real "open " that the world needed. Via these three projects, Deepseek has addressed not only efficient AI but also distributed computing: 1. smallpond: https://github.com/deepseek-ai/smallpond 2. 3fs: https://github.com/deepseek-ai/3FS 3. deepep: https://github.com/deepseek-ai/DeepEP
how many companies will actually adopt 3FS now that it's open source?
not a hater, just know that theres a lot of hurdles to adoption even if something if open source - for example not being an industry standard. i dont know a ton about this space - what is the main alternative?
Re: DeepSeek's smallpond: Bringing Distributed Computing to DuckDB
#8Deepseek is the real "open " that the world needed. Via these three projects, Deepseek has addressed not only efficient AI but also distributed computing: 1. smallpond: https://github.com/deepseek-ai/smallpond 2. 3fs: https://github.com/deepseek-ai/3FS 3. deepep: https://github.com/deepseek-ai/DeepEP
thank goodness, we’ve had nothing open to do efficient distributed computing with for years!
Re: DeepSeek's smallpond: Bringing Distributed Computing to DuckDB
#9spark is getting a bit long in the tooth.. interesting to see duckdb integrated with Ray for data-access partitioning across (currently) 3FS. probably a matter of time before they (or someone) supports S3. It should be noted that duckdb (standalone) actually does a pretty good job scanning s3 parquet on its own.
Re: DeepSeek's smallpond: Bringing Distributed Computing to DuckDB
#10Deepseek is the real "open " that the world needed. Via these three projects, Deepseek has addressed not only efficient AI but also distributed computing: 1. smallpond: https://github.com/deepseek-ai/smallpond 2. 3fs: https://github.com/deepseek-ai/3FS 3. deepep: https://github.com/deepseek-ai/DeepEP
how many companies will actually adopt 3FS now that it's open source? not a hater, just know that theres a lot of hurdles to adoption even if something if open source - for example not being an industry standard. i dont know a ton about this space - what is the main alternative?
For example, in AWS, you can get a similar FSx for Lustre file system for just 11% more cost, which could be worth it to avoid the management costs of running your own storage cluster.