This sounds very similar to Apache Airflow. How would you compare them?
Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative
61–70 of 88 posts
Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative
#62It's not a DBT alternative if it's not SQL-oriented.
Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative
#63Earlier quoted context omitted.
I don't think SQL can be more brittle than untyped object-oriented Python. The latter is as brittle as it gets, as we all know from ML code. SQL is common for ETL exactly because it's not brittle, (mostly) declarative and easy to test and modularize. What Python give you is more flexibility, at the expensive of exponentially more brittleness.
> SQL is common for ETL exactly because it’s not brittle, SQL is common for ETL because typically at least one, sometimes both, ends of an ETL operation is an RDBMS for which SQL is the standard language. It has nothing to do with lack of brittleness.
I guess it's surprising then that both Hadoop/Hive and Spark, which were the originators of SQL for ETL, typically work on data lakes instead of RDBMSs. In fact, RDBMs support didn't come for a long time. The choice of SQL has nothing to do with RDBMs. It's because SQL is a declarative language that's easy to parse and convert into a physical query plan that can be parallelized and optimized extremely well. Why is that? Because it's not a general-purpose imperative loosely typed brittle language like Python.
Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative
#64The data engineering space feels like it's earning the same reputation front end had/has with the endless stream of new and shiny frameworks.
I couldn’t agree more. Except that the cost for a data tool is incredibly higher than any FE tool.
I manage both a FE team and a data team. The former spends around 1k/month on infra and hosting. While the data team easily spends 20k/month.
The gold rush is data. Build shovels.
Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative
#65Earlier quoted context omitted.
As more and more startups are going open source, source available, open core, etc., I need to figure out how to do Launch HNs without triggering off-topic controversies around the term "open source". My problem is, there's no consensus among HN readers about what the term means. If anyone has a suggestion about how to solve this problem in an accurate and neutral way, I'd like to hear it.
I understand your frustration. IME, HN tends to use the term open source in two senses. It can either refer to: - the license or; - the business model. And we know that licenses exist on a spectrum of permissive to restrictive. So when the community is presented with a for-profit entity in a Launch/Show HN, they tend to dwell on the 2nd sense. If it’s a side project that’s on display, then the 1st sense kicks in. Bas…
(Here is an example from another post on the frontpage where the community is engaging in the 1st sense on a side project: Show HN: Little Rat – Chrome extension monitors network calls of all extensions
Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative
#66Earlier quoted context omitted.
> SQL is common for ETL exactly because it’s not brittle, SQL is common for ETL because typically at least one, sometimes both, ends of an ETL operation is an RDBMS for which SQL is the standard language. It has nothing to do with lack of brittleness.
I guess it's surprising then that both Hadoop/Hive and Spark, which were the originators of SQL for ETL, typically work on data lakes instead of RDBMSs. In fact, RDBMs support didn't come for a long time. The choice of SQL has nothing to do with RDBMs. It's because SQL is a declarative language that's easy to parse and convert into a physical query plan that can be parallelized and optimized extremely well. Why is th…
> Hadoop/Hive and Spark, which were the originators of SQL for ETL
They weren’t.
I guarantee you, before either of those existed, when Data Warehousing was often done with a different version/configuration of the same brand of RDBMS as the transactional store (the latter likely using something closer to a normalized schema, the former using a star or snowflake schema), using SQL for ETL was absolutely normal.
Which is why newer data warehousing / data lake systems support SQL even though they aren’t RDBMSs: a couple decades of RDBMS dominance made it the JavaScript of data storage.
> Because it’s not a general-purpose imperative loosely typed brittle language like Python.
Its not general-purpose or imperative, its just as much “loosely typed” as Python (both Python and SQL are strongly typed.)
Its not clear what concrete meaning “brittle” is supposed to have in this claim, so I can’t evaluate its accuracy.
Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative
#67This sounds very similar to Apache Airflow. How would you compare them?
We see this working more alongside Airflow—we see Airflow mainly centered as an orchestrator/scheduler to chain together your ETL steps after you've written your data transformations and connections. With Serra, we're a flexible dev tool to write these transforms and connectors. I think you could accomplish something similar but the implementation would be unwieldy (ie breaking up 4 SQL script tasks into 40 modular transform blocks), whereas we see ourselves falling into the camp of being able to integrate with Airflow and have great error logs in those 4 SQL tasks that you have simplified into Serra configs.
Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative
#68We’re considering adopting DBT or a similar tool for the orchestration of our data pipelines on Snowflake. But we also explored Snowflake Dynamic Tables, and they make it easy to build a complex DAG without having to describe it. I’m curious if data warehouse features like materialized views or dynamic tables will end up making DBT or the like obsolete? https://docs.snowflake.com/en/user-guide/dynamic-tables-abou...
This is a great point—in terms of dynamic tables/materialized views, the software engineering best practices ie modularization, testing, version control are not as intuitive/straight-forward to apply in comparison to dbt and Serra. We can also add a direct SnowflakeDynamicTableWriter into our framework to work with the best of both worlds!
Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative
#69Earlier quoted context omitted.
I guess it's surprising then that both Hadoop/Hive and Spark, which were the originators of SQL for ETL, typically work on data lakes instead of RDBMSs. In fact, RDBMs support didn't come for a long time. The choice of SQL has nothing to do with RDBMs. It's because SQL is a declarative language that's easy to parse and convert into a physical query plan that can be parallelized and optimized extremely well. Why is th…
> Hadoop/Hive and Spark, which were the originators of SQL for ETL They weren’t. I guarantee you, before either of those existed, when Data Warehousing was often done with a different version/configuration of the same brand of RDBMS as the transactional store (the latter likely using something closer to a normalized schema, the former using a star or snowflake schema), using SQL for ETL was absolutely normal. Which i…
Definitely, I can jump into what we meant by brittle—we mainly meant that SQL scripts are hard to debug/undescriptive, you can't parametrize and customize error messages that you receive from transforms, and you can only execute one complete statement at a time that are often chained together with CTEs (which is a nightmare if its a statement of 400 lines of SQL). Python makes it easier to debug since we turn the approach from a declarative to a procedural one, and that's even the case with breakpoints when you write your actual transformers in Python.
Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative
#70Finally a competitor to dbt. the world needs this!
There is also sqlmesh (https://sqlmesh.com/). Pretty new as well. It introduces some interesting concepts. For smaller dbt projects it could be a drop-in replacement as it allows importing dbt projects.