Live data from Hacker News

Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

news.ycombinator.com

41–50 of 88 posts

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#42

> Serra is a low-code, object-oriented ETL framework that allows developers to write PySpark jobs easily—think end-to-end dbt with the benefits of object-oriented Spark. Could you please explain this as if I am three years old? (also, I don't know dbt)

If it is really a dbt clone it is an ELT tool not ETL:

https://en.wikipedia.org/wiki/Extract,_load,_transform

https://en.wikipedia.org/wiki/Data_build_tool

It's about (big) data munging.

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#43
post #38

The selling point was replacing brittle SQL with Python. Sounds great, but I'm not seeing where Python comes into play from that demo? Was it not shown in the video?

Yep we only showed our configuration file which instantiates the said Python objects. Definitely would've made the demo clearer, and if you want to look at the actual objects that our config is referencing, definitely check out our repo!

Thanks! I'll check it out, cheers

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#45
post #36
post #31

Earlier quoted context omitted.

That was my mistake, not the founders', and I've changed the term to source available (edit: now open core) to try to avoid misunderstanding. I assure you there was no attempt at fraud!

Thanks! Do not make this mistake again!

I will try! but it is not so easy, because there's no consensus on what these terms mean.

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#46
post #35
post #30

Earlier quoted context omitted.

As more and more startups are going open source, source available, open core, etc., I need to figure out how to do Launch HNs without triggering off-topic controversies around the term "open source". My problem is, there's no consensus among HN readers about what the term means. If anyone has a suggestion about how to solve this problem in an accurate and neutral way, I'd like to hear it.

I understand your frustration. IME, HN tends to use the term open source in two senses. It can either refer to: - the license or; - the business model. And we know that licenses exist on a spectrum of permissive to restrictive. So when the community is presented with a for-profit entity in a Launch/Show HN, they tend to dwell on the 2nd sense. If it’s a side project that’s on display, then the 1st sense kicks in. Bas…

Thanks! that's helpful. I've changed the wording to "open core" above.

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#47
post #11

Earlier quoted context omitted.

https://opensource.org/osd/ > Introduction > Open source doesn’t just mean access to the source code. The distribution terms of open-source software must comply with the following criteria There’s a definition. This isn’t open source.

And how does the OSI derive it's legitimacy as the steward for all things open source? As far as I am concerned, it is just one body with its own private viewpoint, not a universal lawmaker for all open source devs. In general, while I appreciate the work of the OSI, I believe that they are too idealistic in their viewpoint, derived from the world of Linux and early OS. In my view, if we want to maintain a healthy an…

> And how does the OSI derive it's legitimacy as the steward for all things open source?

We give it to them.

OSI isn't an entity that's existed since the beginning and the original definition doesn't come from them.

I agree that it's not a space that doesn't change.

Having said that, change must come from the community. It can't be just a couple corporate entities defining their own license, calling it open source, and going against the established definition.

> In my view, if we want to maintain a healthy and growing open source ecosystem, we must allow the makers of great OSS to be sustainable and monetize their creation. I don't believe that that's an inherent conflict with the spirit of OSS.

I don't mind monetization don't get me wrong.

Should it change? I'm not the authority on it. I'm just saying it's not the current definition.

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#48
post #32

> easy by replacing brittle SQL scripts with object-oriented Python There is a lot to unpack here. Can you explain this in more detail?

Sure, our approach is to define Python classes to handle reusable steps for reading, transforming or loading data. For example, we have a MapTransformer, CastColumnsTransformer, GeoDistanceTransformer.

Each class specifies some configuration needed for the "step" and can then be used in the config file to construct a full ETL job. You can write unit tests for custom transformers you create as we have shown in the tests/ directory.

I have also updated the README in our repo to hopefully provide a better explanation of how our config file connects to specific Python objects.

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#49

Congrats on the launch! Interesting project in a space that I am pretty certain is going to change a lot in the coming years. Here is a bit of random feedback and questions. * Some of your messaging related to python vs yaml is a bit confusing, which results in me not being immediately clear on the value prop. After digging through docs and code I now understand that the yaml is a declarative pipeline calling the und…

This is super insightful thanks a ton for this gold mine.

On the python vs yaml part—definitely could've made that way more clear in the demo. Right now are framework lets you call these python objects in your yaml file, but we are working on just a python-centric implementation as well for those that do not want to interact with yamls.

On the loom and licensing choice—that's a great point. One of the main issues we ran into is getting adoption as we originally just tried licensing out the framework (mega fail ofc)—found out the hard way that no dev wants to buy something to try it out. We're definitely flexible on our license and will take all this feedback into account.

On the barrier of entry—also super insightful. We're working on a local UI offering that will be a 'config' block builder that will be free for all installs. We're implementing a DAG view similar to Airflow on the transform level. We also want to make it super easy to see your code and preview how it changes with this local UI (and have a list of all the params you need for your spark objects without having to go through the docs). We also want to flesh out more features especially on the translate side, as well as host on the cloud.

With the complexity issue that's something I ran into Disney as well! As the product grows we definitely want to flesh out our transformers based on the scripts we see. For now, the developer can make one-off transformers—we actually have a catch all "SQL transformer" for cases where you want to just pass in your sql (similar to a dbt model) and run it that way. That way it's a fail safe for if you have one specific transform that you feel is super hard to break down, you can fall back on dbt's way of just modularizing the SQL into a transform, and reference it however many times you want as an input block later on.

Thanks so much for the congrats, will definitely reach out and would love to have further discussions in the thread as will.

Re: Launch HN: Serra (YC S23) – Open-core, Python-based dbt alternative

#50
post #22

I like that you improve on the underlying database error messages, as they are really unhelpful, and I think this is a great place to add value. I've been keeping track of a few dbt alternatives. Dbt have opened up the market to this use case, while only partially solving the business model and maturity side. Here are the more interesting ones: sdf.com (ex Meta team) sqlmesh.com (relatively new) paradime.io (more an…

These are great links, we'll take a look
Post reply on HN