Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

news.ycombinator.com

Post reply on HN
Print view View original

81–88 of 88 posts

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#81

Post by adammarples »

I'm not sure about this. The big SQL script is annoying but when broken down into 4 parts it's very easy to understand. This is precisely the strength of dbt. This would decompose to 4 dbt models, which could be deployed as views, tables, or ctes etc. Instead it seems that you've developed your own DSL for data transforms to take its place, in the form of special classes declared in yaml.

It might be better? But sql is so well understood and covers so much functionality I'd expect that it would be a long time before you ever hit parity with it.

It would be nice if dbt could interface with buckets etc but if they're wrapped in an external table or whatever then that problem goes away.

One thing I noticed is that it (the example) misses a killer feature of dbt, you're specifying your database targets right in the class config. The killer feature of dbt is that you just specify the transforms and then point it at different environments using a target flag and a profiles file, deploy to different envs with ease. I would definitely separate location/env/credential config from transform logic or make it variable.

Given that sql is a totally valid language for declaring transforms in spark, I would probably rather see spark as a materialization backend to dbt somehow rather than an entirely new thing.

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#83

Post by 0cf8612b2e1e »

alex-korr wrote: post #59

Maybe I am missing something but would there ever be a scenario where taking a single albeit large sql statement and rewriting it as several pyspark scripts would result in faster runtime for your data pipeline? In most cases, this will be much much slower.

Greatly depends on your environment. I am thankfully in an area where there are very modest timeliness requirements. Improving the speed of a job means little to me. However, improving debugability or checkpointing when things go wrong is always valuable.

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#84

Post by mrwnmonm »

albertstanley wrote: post #51
mrwnmonm wrote: post #14

> Serra is a low-code, object-oriented ETL framework that allows developers to write PySpark jobs easily—think end-to-end dbt with the benefits of object-oriented Spark. Could you please explain this as if I am three years old? (also, I don't know dbt)

Sure, I’ll clarify some of the terms used in that one-liner in case it’s helpful for anyone else as well. ETL is the process of extracting transforming and loading data from a source to a destination in a data pipeline. Spark, an engine for large scale data processing, allows us to write code that can work with large amounts of data. dbt is a tool you can use to break up your SQL scripts into smaller “models” - other…

Thanks

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#85

Post by kermatt »

smcin wrote: post #73
kermatt wrote: post #26

Earlier quoted context omitted.

Moving between Spark and Pandas can cause type casting as well. For example the range of allowable dates in Pandas is much smaller than in Spark. We completely abandoned Pandas in favor of PySpark for this reason. It seems unnecessary to use multiple dataframe implementations when Spark is already in play.

Are you referring to pandas.Timestamp.max being 2262-04-11 23:47:16.854775807 ? https://pandas.pydata.org/docs/reference/api/pandas.Timestam... (pandas design choice was to support nanosecond times, for financial data.)

Yes. Unfortunately I’m dealing with an app that likes to use multiple magic dates way past the Pandas range.

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#86

Post by smcin »

kermatt wrote: post #85
smcin wrote: post #73

Earlier quoted context omitted.

Are you referring to pandas.Timestamp.max being 2262-04-11 23:47:16.854775807 ? https://pandas.pydata.org/docs/reference/api/pandas.Timestam... (pandas design choice was to support nanosecond times, for financial data.)

Yes. Unfortunately I’m dealing with an app that likes to use multiple magic dates way past the Pandas range.

"much smaller range" seems disingenuous without saying that you mean "not beyond 2262". And you said those aren't real dates, only magic dates or sentinels. So that's a totally artificial requirement. And you could fix the magic dates up at conversion with a simple replacement script.

* MS-DOS supports dates from 1/1/1980 to 12/31/2099

* 32b Linux (or Windows 7) supported timestamps up to 2038

* 64b timestamps fixed all thia already, and presumably OSes will be using 128b datetimes well before 2099 if not sooner.

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#87

Post by kermatt »

smcin wrote: post #86
kermatt wrote: post #85

Earlier quoted context omitted.

Yes. Unfortunately I’m dealing with an app that likes to use multiple magic dates way past the Pandas range.

"much smaller range" seems disingenuous without saying that you mean "not beyond 2262". And you said those aren't real dates, only magic dates or sentinels. So that's a totally artificial requirement. And you could fix the magic dates up at conversion with a simple replacement script. * MS-DOS supports dates from 1/1/1980 to 12/31/2099 * 32b Linux (or Windows 7) supported timestamps up to 2038 * 64b timestamps fixed…

The RDBMs in this case accepts 9999-12-31 as a valid date. Pandas does not. This is where the issue came in, and switching to PySpark meant we needed no date manipulation to handle the data supplied by the upstream.

Magic dates suck, but they exist in the wild. There are also valid cases where data is not tied to the lifetimes of humans currently writing code.

The range of values for date values in PostgreSQL is 4713 BC to 5874897 AD:

https://www.postgresql.org/docs/current/datatype-datetime.ht...

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#88

Post by smcin »

kermatt wrote: post #87
smcin wrote: post #86

Earlier quoted context omitted.

"much smaller range" seems disingenuous without saying that you mean "not beyond 2262". And you said those aren't real dates, only magic dates or sentinels. So that's a totally artificial requirement. And you could fix the magic dates up at conversion with a simple replacement script. * MS-DOS supports dates from 1/1/1980 to 12/31/2099 * 32b Linux (or Windows 7) supported timestamps up to 2038 * 64b timestamps fixed…

The RDBMs in this case accepts 9999-12-31 as a valid date. Pandas does not. This is where the issue came in, and switching to PySpark meant we needed no date manipulation to handle the data supplied by the upstream. Magic dates suck, but they exist in the wild. There are also valid cases where data is not tied to the lifetimes of humans currently writing code. The range of values for date values in PostgreSQL is 4713…

Ah I see your point. Yeah I noticed SQL goes up to 9999.
Post reply on HN