Earlier quoted context omitted.
Hey Georg, thanks for posting. I've been working on building a thing with almost the same stack lately and the Dagster integration is my next large-ish step. In the last couple weeks I've kicked around Duck and Clickhouse for the backend. It's been a lot of fun. Ultimately I'd like something that can be run locally or easily installed and run on a PaaS. The only trick with local is that data sets of any appreciable s…
Well local can mean your laptop. But could mean your local server. However, it can also mean a VM on your cloud provider of choice colocated to the object store. In such a case the network transfer (for many cases) can be almost irrelevant - at least for up to medium-ish sized datasets.
Funny thing, I'm trying to get going with Dagster today like I said, and figuring out how to debug is a bit .. annoying. The most helpful post I've come across so far just happens to be https://georgheiler.com/2022/02/02/interactive-dagster-debug...
So thanks!