Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Smallpond – A lightweight data processing framework built on DuckDB and 3FS

github.com

Post reply on HN
Print view View original

71–75 of 75 posts

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#71

Post by menaerus »

rscho wrote: post #67
menaerus wrote: post #63

Earlier quoted context omitted.

Why do you think medical science wrt complexity is any different than applied math, which computer science essentially is? People already can use LLMs to assist them in diagnosing health issues so why would it be hard to believe that the doctors won't be using the same kind of assistance soon too?

> Why do you think medical science wrt complexity is any different than applied math I don't think I wrote that. Doctors already use tech assistance. I just pointed out that while we've got efficient robots for applied math, we don't have those as agents in the physical world. People who do blue collar jobs are less replaceable. Well, believe it or not, but most doctors are actually blue collar workers.

You sort of implied that with your replies across the thread. And since AI already replaced part of the CS, I was wondering why do you think this would not be the case with doctors. I'm not sure I agree it's a blue collar profession. I can easily see diagnostics being replaced with AI models.

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#72

Post by rscho »

menaerus wrote: post #71
rscho wrote: post #67

Earlier quoted context omitted.

> Why do you think medical science wrt complexity is any different than applied math I don't think I wrote that. Doctors already use tech assistance. I just pointed out that while we've got efficient robots for applied math, we don't have those as agents in the physical world. People who do blue collar jobs are less replaceable. Well, believe it or not, but most doctors are actually blue collar workers.

You sort of implied that with your replies across the thread. And since AI already replaced part of the CS, I was wondering why do you think this would not be the case with doctors. I'm not sure I agree it's a blue collar profession. I can easily see diagnostics being replaced with AI models.

I never wanted to imply that. But here, people frequently assume that because that's what they're used to. Diagnosis is the tip of the iceberg. Most people here aren't sick, so diagnostics are their only focus. If they get ill, they want a diagnosis. But many people are chronically ill already, and doctors spend most of their time treating, not diagnosing. Treating people is made in good part of technical procedures and practical assessments, and you need doctors for that because robots are still far behind for that kind of stuff. People actually have a completely skewed view of what a doctor is.

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#73

Post by menaerus »

rscho wrote: post #72
menaerus wrote: post #71

Earlier quoted context omitted.

You sort of implied that with your replies across the thread. And since AI already replaced part of the CS, I was wondering why do you think this would not be the case with doctors. I'm not sure I agree it's a blue collar profession. I can easily see diagnostics being replaced with AI models.

I never wanted to imply that. But here, people frequently assume that because that's what they're used to. Diagnosis is the tip of the iceberg. Most people here aren't sick, so diagnostics are their only focus. If they get ill, they want a diagnosis. But many people are chronically ill already, and doctors spend most of their time treating, not diagnosing. Treating people is made in good part of technical procedures…

> People actually have a completely skewed view of what a doctor is.

It could be but treating patients also requires continuous diagnostics, result comprehension, and final assessment so this is certainly the part where AI could play the crucial role.

I don't think anyone thinking of the AI consequences on medicine is arguing that it will replace manual labor such as procedure executions or psychological support. This is obviously not possible so when I see people talking about the "AI in medicine" I read that as mostly complementing the existing work with new technology.

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#74

Post by HackerThemAll »

jamesblonde wrote: post #35

We are seeing more and more specialized query engines. This is a query engine specialized for training pipelines. It is not general purpose - it is for providing batches of training data at workers. It uses Ray for parallelization. The kind of queries you need are random reads (to implement shuffling across epochs), arrow support (zero copy to Pandas DataFrames), and efficient checkpointing.

DuckDB is specialized in efficient storage and fast query for analytics (OLAP), using a columnar storage (in contrast to row storage, used by usual RDBMSes doing OLTP processing). It's nothing new, it's been there for couple decades already. But this "distributed" DuckDB can indeed be beneficial for training.

Re: Smallpond – A lightweight data processing framework built on DuckDB and 3FS

#75

Post by lvl155 »

threeseed wrote: post #39
lvl155 wrote: post #17

Looking forward to next few years when we can finally abstract away all the back-end techs.

We've had this for at least a decade now. If you use a cloud provider there are managed solutions for data engineering pipelines.

We did? That’s news to me.
Post reply on HN