Smallpond – A lightweight data processing framework built on DuckDB and 3FS • Hacker News Board

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#51

Post by sharpshadow » Mon, Mar 03, 2025, 5:02 AM UTC

mdaniel wrote: post #18

rscho wrote: post #12

Earlier quoted context omitted.

Yes, doctors are safe. Because they do things. With their hands. That no one else does.

Also, a hallucination for 'SELECT mising_field FROM borgus_tuble' is one thing, hallucinating that taking a dose of Cl Na O along with CH3 CO2 H will cure covid is another thing entirely

Is it really true that people drank bleach!? It always felt to me as some idiot did it once and it was repeated by the media endlessly, probably for clicks because this story is so dumb. Nonetheless the actual thing which people take is ClO2.

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#52

Post by m2f2 » Mon, Mar 03, 2025, 5:46 AM UTC

threeseed wrote: post #39

lvl155 wrote: post #17

Looking forward to next few years when we can finally abstract away all the back-end techs.

We've had this for at least a decade now. If you use a cloud provider there are managed solutions for data engineering pipelines.

Sure, and it's not cheap.

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#53

Post by holdenk » Mon, Mar 03, 2025, 7:15 AM UTC

mdaniel wrote: post #21

delfinom wrote: post #14

Earlier quoted context omitted.

Nope. Healthcare megacorps are buying up independent practices like crazy. All because doctors can't keep up with the bullshit IT required for insurance, state mandates, etc and that's in addition to the insanity of even renting commercial real estate for an office these days. These megacorps set quotas and push doctors to nickel and dime like crazy. They sure as shit will spend the money to find robots that can give…

Sounds good; if all these pro-AI folks could get it to complete the insurance paperwork that'd be swell. Actually, come to think of it, do that for the paperwork from both sides, doctor and patient, and eliminate and entire class of leaches upon humanity I'm going to laugh if DOGE eliminates the IRS, but also might be thankful

Admittedly just one part of the insurance paperwork but we're working on automating appeals at Fight Health Insurance / Fight Paperwork :) ( https://www.fighthealthinsurance.com/ / https://www.fightpaperwork.com/ )

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#54

Post by dcreater » Mon, Mar 03, 2025, 7:23 AM UTC

HackerThemAll wrote: post #6

DuckDB itself is cool enough, especially when combined with SQLite and/or PostgreSQL, and now this. Thanks DeepSeek!

How is duckdb combined with SQLite? Aren't they alternatives to each other?

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#55

Post by dcreater » Mon, Mar 03, 2025, 7:26 AM UTC

RyanHamilton wrote: post #13

If you want to checkout duckdb try QStudio. It's a free sql client with duckdb integrated: https://www.timestored.com/qstudio/help/duckdb-sql-editor . Disclaimer: I'm the main author.

What's with the win95 ui?

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#56

Post by dcreater » Mon, Mar 03, 2025, 7:28 AM UTC

Confused by the example in the repo? What is the use case for this? Is it a replacement for dask, ray etc? (Not a professional swe)

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#57

Post by agilob » Mon, Mar 03, 2025, 8:23 AM UTC

jamesblonde wrote: post #35

We are seeing more and more specialized query engines. This is a query engine specialized for training pipelines. It is not general purpose - it is for providing batches of training data at workers. It uses Ray for parallelization. The kind of queries you need are random reads (to implement shuffling across epochs), arrow support (zero copy to Pandas DataFrames), and efficient checkpointing.

I'm super impressed how much effort DeepSeek did and how much of it they opensourced.

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#58

Post by agilob » Mon, Mar 03, 2025, 8:27 AM UTC

didntknowyou wrote: post #32

breadwinner wrote: post #10

Earlier quoted context omitted.

Give up and become what? Most white collar jobs will be automated in the coming years. You think doctors' jobs are safe?

you can already google the information , the majority of a doctor's value is not in their information but their people and technical skills.

>the majority of a doctor's value is not in their information but their people and technical skills.

Not even that, in the last years I haven't met a single doctor who would even care. Their value is now a necessary evil, they have the legal powers to recommend you to a hospital and give prescriptions. These legal powers will be much harder to change.

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#59

Post by agilob » Mon, Mar 03, 2025, 8:28 AM UTC

m2f2 wrote: post #52

threeseed wrote: post #39

Earlier quoted context omitted.

We've had this for at least a decade now. If you use a cloud provider there are managed solutions for data engineering pipelines.

Sure, and it's not cheap.

AI isn't going to be any cheaper

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#60

Post by orlp » Mon, Mar 03, 2025, 8:59 AM UTC

tomnipotent wrote: post #47

orlp wrote: post #7

One thing I found peculiar is that for the GraySort benchmark it dispatches to Polars by default to do the actual sorting, not DuckDB: https://github.com/deepseek-ai/smallpond/blob/ed112db42af4d0... .

The function argument defaults to polars, but the actual benchmark code sets duckdb by default. https://github.com/deepseek-ai/smallpond/blob/ed112db42af4d0...

I see, confusing multiple layers of defaults :)