Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Smallpond – A lightweight data processing framework built on DuckDB and 3FS

github.com

Post reply on HN
Print view View original

61–70 of 75 posts

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#61

Post by jitl »

dcreater wrote: post #54
HackerThemAll wrote: post #6

DuckDB itself is cool enough, especially when combined with SQLite and/or PostgreSQL, and now this. Thanks DeepSeek!

How is duckdb combined with SQLite? Aren't they alternatives to each other?

Not sure what the poster meant but DuckDB is an analytics DB, it doesn’t have a btree index - at least not last time I looked. You could consider it the OLAP embedded DB to SQLite’s OLTP embedded db.

DuckDB can read SQLite so you can even imagine using them side by side in the same system, serving point reads and writes from SQLite and using DuckDB for stuff like aggregates and searches that SQLite is slower at.

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#62

Post by RyanHamilton »

dcreater wrote: post #55
RyanHamilton wrote: post #13

If you want to checkout duckdb try QStudio. It's a free sql client with duckdb integrated: https://www.timestored.com/qstudio/help/duckdb-sql-editor . Disclaimer: I'm the main author.

What's with the win95 ui?

There are many themes to choose from. I recorded the demo on that page and I like windows 95. I concede it may not be pretty but I've always found it functional. The default is darcula theme like shown on the main page: https://www.timestored.com/qstudio/

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#63

Post by menaerus »

rscho wrote: post #19
aragonite wrote: post #16

Earlier quoted context omitted.

> Because they do things. With their hands. That no one else does That's only true of surgeons :) What if your specialty is nonsurgical (internal medicine, pediatrics, psychiatry, etc)?

Almost all specialties do various technical procedures that only them really know how to do. The extreme is psychoanalytic psychiatry, which are the only ones really doing nothing with their hands (yes, interventional psychiatry is a thing...). Now, you could argue that 'yes, but most of the times it's done by techs/nurses'. Well, no. When things go south, and in all places where there is noone else to do the stuff (…

Why do you think medical science wrt complexity is any different than applied math, which computer science essentially is? People already can use LLMs to assist them in diagnosing health issues so why would it be hard to believe that the doctors won't be using the same kind of assistance soon too?

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#64

Post by menaerus »

risyachka wrote: post #45
shipp02 wrote: post #8

Is the code written by the deepseek model? I should probably give up on being a software engineer if it is.

Why would you assume that?

It's not impossible to imagine. Not many LoC and it's a lightweight python layer on top of the stack that actually does the heavy lifting - DuckDB and 3FS.

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#65

Post by HackerThemAll »

dcreater wrote: post #54
HackerThemAll wrote: post #6

DuckDB itself is cool enough, especially when combined with SQLite and/or PostgreSQL, and now this. Thanks DeepSeek!

How is duckdb combined with SQLite? Aren't they alternatives to each other?

They are complementary to each other. There's an SQLite extension for use within DuckDB [1], which gives you a power of great transactional capabilities of SQLite and speed of analytical queries within DuckDB's columnar storage engine, all within a single database.

[1] https://duckdb.org/docs/stable/extensions/sqlite.html

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#66

Post by dudeinjapan »

BobbyJo wrote: post #22
lvl155 wrote: post #17

Looking forward to next few years when we can finally abstract away all the back-end techs.

We ain't even solved garbage collection yet, and you think "back end systems" are going to abstracted away in the next few years?

[dead]

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#67

Post by rscho »

menaerus wrote: post #63
rscho wrote: post #19

Earlier quoted context omitted.

Almost all specialties do various technical procedures that only them really know how to do. The extreme is psychoanalytic psychiatry, which are the only ones really doing nothing with their hands (yes, interventional psychiatry is a thing...). Now, you could argue that 'yes, but most of the times it's done by techs/nurses'. Well, no. When things go south, and in all places where there is noone else to do the stuff (…

Why do you think medical science wrt complexity is any different than applied math, which computer science essentially is? People already can use LLMs to assist them in diagnosing health issues so why would it be hard to believe that the doctors won't be using the same kind of assistance soon too?

> Why do you think medical science wrt complexity is any different than applied math

I don't think I wrote that.

Doctors already use tech assistance. I just pointed out that while we've got efficient robots for applied math, we don't have those as agents in the physical world. People who do blue collar jobs are less replaceable. Well, believe it or not, but most doctors are actually blue collar workers.

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#68

Post by nyrikki »

jamesblonde wrote: post #35

We are seeing more and more specialized query engines. This is a query engine specialized for training pipelines. It is not general purpose - it is for providing batches of training data at workers. It uses Ray for parallelization. The kind of queries you need are random reads (to implement shuffling across epochs), arrow support (zero copy to Pandas DataFrames), and efficient checkpointing.

Some of what they are doing is simply what was lost due to the ubiquitous nature of the relational model.

The hierarchical model is applicable to many problems and actually in part why moving off mainframes is challenging because IMS is so much more efficient than the relational model for applications like airline tickets.

There have been several efforts to leverage object stores in the way they did that I am aware of but it was a hard sell.

The hierarchical model really only works for many to one relationships, and it's integrity model differs and is not as DRY.

There are lessons to learn here but it requires some relearning.

When you have a shopping cart, having data local to the server handling the transaction is also a benefit.

Codd's relational model has advantages, but has held back some efforts because we are just use to dealing with the painful parts that we often don't consider other options.

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#69

Post by geodel »

rscho wrote: post #46
skeeter2020 wrote: post #43

Earlier quoted context omitted.

pathologists are some of the highest paid doctors and they are right in the crosshairs of what AI is getting better at performing.

Do you really know what pathologists do ? Apparently not...

IT engineers thought the same. Until finally automation is setting them right.

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#70

Post by rscho »

geodel wrote: post #69
rscho wrote: post #46

Earlier quoted context omitted.

Do you really know what pathologists do ? Apparently not...

IT engineers thought the same. Until finally automation is setting them right.

I'm sure going to be amazed if the LLMs of the future 10y suddenly acquire the ability to physically cut just the right bit of a random surgical piece, with a precise idea of where, when and in what orientation the surgeon dug it out, all that with shitty documentation. Humans will be cheaper for a long time still.
Post reply on HN