Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 88 posts

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#51

Post by albertstanley »

mrwnmonm wrote: post #14

> Serra is a low-code, object-oriented ETL framework that allows developers to write PySpark jobs easily—think end-to-end dbt with the benefits of object-oriented Spark. Could you please explain this as if I am three years old? (also, I don't know dbt)

Sure, I’ll clarify some of the terms used in that one-liner in case it’s helpful for anyone else as well.

ETL is the process of extracting transforming and loading data from a source to a destination in a data pipeline. Spark, an engine for large scale data processing, allows us to write code that can work with large amounts of data. dbt is a tool you can use to break up your SQL scripts into smaller “models” - other SQL scripts that can be reused and tested.

We described us as an end to end because we also have extractors and loaders, whereas dbt focuses on the T ( transformation step of ETL ). Each of our steps involved in extraction, transformation and loading correspond to a specific Python object defined in our Python framework. I have also updated the README in our repo to hopefully better explain how the config file links to user defined readers, writers, and transformers.

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#52

Post by Alanhlwang »

esafak wrote: post #42
mrwnmonm wrote: post #14

> Serra is a low-code, object-oriented ETL framework that allows developers to write PySpark jobs easily—think end-to-end dbt with the benefits of object-oriented Spark. Could you please explain this as if I am three years old? (also, I don't know dbt)

If it is really a dbt clone it is an ELT tool not ETL: https://en.wikipedia.org/wiki/Extract,_load,_transform https://en.wikipedia.org/wiki/Data_build_tool It's about (big) data munging.

Thanks for these links! We consider ourselves an ELT and ETL tool—if you run a Serra job in your own warehouse (ie Databricks), you can easily specify extracting from AWS, loading the parquets into your warehouse, then transforming them with our config block approach (ELT).

The same is true for ETL. If you have a spark cluster separate from your warehouse, you can define your config file to run in the order E T L: you can extract from your data source, run the transformations on a separate cluster, then load it to your warehouse.

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#54

Post by nouri »

We’re considering adopting DBT or a similar tool for the orchestration of our data pipelines on Snowflake. But we also explored Snowflake Dynamic Tables, and they make it easy to build a complex DAG without having to describe it.

I’m curious if data warehouse features like materialized views or dynamic tables will end up making DBT or the like obsolete?

https://docs.snowflake.com/en/user-guide/dynamic-tables-abou...

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#55

Post by loveparade »

boxed wrote: post #3

If the selling point is "replacing brittle SQL scripts with object-oriented Python" you should have at least one example of this code in the README!

I don't think SQL can be more brittle than untyped object-oriented Python. The latter is as brittle as it gets, as we all know from ML code. SQL is common for ETL exactly because it's not brittle, (mostly) declarative and easy to test and modularize.

What Python give you is more flexibility, at the expensive of exponentially more brittleness.

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#56

Post by dragonwriter »

loveparade wrote: post #55
boxed wrote: post #3

If the selling point is "replacing brittle SQL scripts with object-oriented Python" you should have at least one example of this code in the README!

I don't think SQL can be more brittle than untyped object-oriented Python. The latter is as brittle as it gets, as we all know from ML code. SQL is common for ETL exactly because it's not brittle, (mostly) declarative and easy to test and modularize. What Python give you is more flexibility, at the expensive of exponentially more brittleness.

> SQL is common for ETL exactly because it’s not brittle,

SQL is common for ETL because typically at least one, sometimes both, ends of an ETL operation is an RDBMS for which SQL is the standard language. It has nothing to do with lack of brittleness.

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#57

Post by sobkas »

dang wrote: post #30
ssddanbrown wrote: post #18

Earlier quoted context omitted.

Thanks dang, "source available" is pretty common for licenses like the ELv2 used here.

As more and more startups are going open source, source available, open core, etc., I need to figure out how to do Launch HNs without triggering off-topic controversies around the term "open source". My problem is, there's no consensus among HN readers about what the term means. If anyone has a suggestion about how to solve this problem in an accurate and neutral way, I'd like to hear it.

First if you don't know what open source is, don't add that phrase to titles at random. Fact that it's launched with YC help doesn't actually help your case. Adding feel good phrase "open source" to benefit hand that feeds you is pathetic. Because right now damage is already done, many people saw and will associate Serra with open source, which it isn't.

Second if you are adding it, add definition you are using(not supported? then someone should implement it). Make sure that definition you are using factually describes license used. Don't use some fringe bs. as your definition, so you can crowbar it in every time you find it convenient.

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#58

Post by dang »

sobkas wrote: post #57
dang wrote: post #30

Earlier quoted context omitted.

As more and more startups are going open source, source available, open core, etc., I need to figure out how to do Launch HNs without triggering off-topic controversies around the term "open source". My problem is, there's no consensus among HN readers about what the term means. If anyone has a suggestion about how to solve this problem in an accurate and neutral way, I'd like to hear it.

First if you don't know what open source is, don't add that phrase to titles at random. Fact that it's launched with YC help doesn't actually help your case. Adding feel good phrase "open source" to benefit hand that feeds you is pathetic. Because right now damage is already done, many people saw and will associate Serra with open source, which it isn't. Second if you are adding it, add definition you are using(not s…

No one "knows" what the meaning of a disputed term is; it's disputed.

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#59

Post by alex-korr »

Maybe I am missing something but would there ever be a scenario where taking a single albeit large sql statement and rewriting it as several pyspark scripts would result in faster runtime for your data pipeline? In most cases, this will be much much slower.
Post reply on HN