Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 88 posts

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#73

Post by smcin »

kermatt wrote: post #26
khaledh wrote: post #8

The pattern of reading from data sources to a Pandas DataFrame first defeats the whole point of using Spark[1]. Maybe it's ok for small tables, but you'll probably run out of memory on large tables. [1] https://github.com/Serra-Technologies/serra/blob/a7a80c77af5...

Moving between Spark and Pandas can cause type casting as well. For example the range of allowable dates in Pandas is much smaller than in Spark. We completely abandoned Pandas in favor of PySpark for this reason. It seems unnecessary to use multiple dataframe implementations when Spark is already in play.

Are you referring to pandas.Timestamp.max being 2262-04-11 23:47:16.854775807 ?

https://pandas.pydata.org/docs/reference/api/pandas.Timestam...

(pandas design choice was to support nanosecond times, for financial data.)

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#74

Post by camgunz »

applgo443 wrote: post #53

Is ETL/ELT same as writing SQL scripts and periodically executing them? I assumed there's more to it.

Sometimes there is more to it, like pulling data from external services or running ML, but other than that, yeah it's SQL, DAGs, and cronjobs

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#78

Post by noel_gomez »

nouri wrote: post #54

We’re considering adopting DBT or a similar tool for the orchestration of our data pipelines on Snowflake. But we also explored Snowflake Dynamic Tables, and they make it easy to build a complex DAG without having to describe it. I’m curious if data warehouse features like materialized views or dynamic tables will end up making DBT or the like obsolete? https://docs.snowflake.com/en/user-guide/dynamic-tables-abou...

It is likely that these will make it into dbt since there are many dbt users who also use snowflake. The main thing to consider when deciding to some something other than dbt is:

* can you apply software dev best practices, CI/CD etc * is it proprietary or can you use it with other dbs * is there a large community behind it e.g. dbt packages and dbt python libraries * will you also get docs, dq, lineage or will you need additional tools\ing * will you need to orchestrate other aspects of your data flow e.g. EL, then T, then activation, etc

Databricks also has delta live tables and for the reasons above I usually suggest people consider all of these and not just go all-in with one vendor

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#79

Post by nailer »

whoknowswhat11 wrote: post #12

A quick note that the "open source" license they use requires activation and license keys that block feature activations in the "open source" software be preserved. This license was popular w folks like apollo who arguably hijacked nearly 700 contributors efforts w a license like this. Because they are using it from start at least that won't be as bad

It's not an open source license. Maybe that's why the quotes but let's not indulge them in their lies.

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#80

Post by nailer »

dang wrote: post #30
ssddanbrown wrote: post #18

Earlier quoted context omitted.

Thanks dang, "source available" is pretty common for licenses like the ELv2 used here.

As more and more startups are going open source, source available, open core, etc., I need to figure out how to do Launch HNs without triggering off-topic controversies around the term "open source". My problem is, there's no consensus among HN readers about what the term means. If anyone has a suggestion about how to solve this problem in an accurate and neutral way, I'd like to hear it.

> My problem is, there's no consensus among HN readers about what the term means.

Open Source is defined here, by the people that invented the term: https://opensource.org/osd/

The vast majority of HN readers would support this over any other definition.

Anything that provides access to source but doesn't allow forks, commercial use, competition, removal of advertising, etc is 100% not open source.

Post reply on HN