Viewing profile — abrazensunset

This has loose overlap with: - Materialize - Flink SQL - Arroyo - Readyset - RisingWave - Timeplus - Pathway - Dozer - ReadySet - Snowflake dynamic tables - Native materialized vie…

Agree with this. Snowflake has best-in-class dev experience and performance for Spark-like workloads (so ETL or unconstrained analytics queries). It has close to worst-in-class per…

Please don't do this. If you want an Airflow-ish approach without punishing your future self, pick Prefect. Otherwise go with Temporal. Above all do not adopt Airflow for the use c…

If you use Julia, Makie crushes this use case and comes with great Python interop. https://github.com/holoviz/datashader is a good one in the Python ecosystem.

What distinguishes this from the more well-known OpenMetadata project?

"Lakehouse" usually means a data lake (bunch of files in object storage with some arbitrary structure) that has an open source "table format" making it act like a database. E.g. us…

There is a huge round of "data observability" startups that address exactly this. As a category it was overfunded prior to the VC squeeze. Some of them are actually good. They all …

I think it's more a matter of comparing minivans (cloud "DWH" engines) to sports cars (Clickhouse et al) here. Snowflake's performance characteristics & ops paradigm have always be…

"I want to write my orchestration in Python and I'm comfortable hosting my own compute" -> Prefect (lightweight) or Dagster (heavier but featureful) "My team already knows Airflow …

I'm a heavy Prefect user and was also very confused about the initial rewrite, even after reading several summaries. My best advice is to just try using 2.0 (Orion). Here's how I'd…

Let's say you want to handle populating secrets in multiple places, in an enterprise team context. What do you use today, or what would yo use? Two good examples I've seen are 1Pas…

+1, the Pandas API is somewhere between mediocre and bad, and results in garbage code unless you use it in a carefully constrained way (which is admittedly true of many complete la…

It's worth noting that hydraulic fracturing itself is rarely the problem. Issues come from moving fluid volumes somewhere else during production: subsidence due to extraction from …

+1, Migadu is simple, reliable, easy, and just works. Cost is low enough it might as well be free. I've only ever had one issue (from incorrectly interpreting documentation) and ha…

In that situation (dual usage modes) I think I'd rather have the primary data store be Materialize, and just snapshot Materialize views back to your warehouse (or even just to an o…

In my experience (dependency-heavy data engineering & ML), Poetry is unbearably slow[^1]. Great interface/workflow, though. [1] https://github.com/python-poetry/poetry/issues/2094

Anyone here with practical experience using [Flying Squid]( https://github.com/HazyResearch/flyingsquid ) over the open-source Snorkel library? I'm curious if this platform re-uses…