Launch HN: Grai (YC S22) – Open-Source Data Observability Platform • Hacker News Board

Re: Launch HN: Grai (YC S22) – Open-Source Data Observability Platform

#21

Post by ssddanbrown » Mon, Jul 17, 2023, 9:03 PM UTC

jagtstronaut wrote: post #19

ssddanbrown wrote: post #17

Earlier quoted context omitted.

I personally don't have anything against the license you've chosen, and I respect your right to protect your efforts against usage you don't desire. I just think it's better to avoid using "open source" if going down the ELv2 path, and using something like "source available" or "fair code" instead to prevent confusion in misrepresenting this as, what is commonly considered, open source. If you'd like further detail i…

This question is for my education alone, but since you seem quite passionate I am curious. I just read a super long article about licensing to understand your comment as well as the article you wrote. Under these "source available" licenses, I can still sell the software within some kind of package correct? Like if I create my own PR linter I can use Grai and still sell it? I just can't host grai with some observabil…

Just to be clear for my responses, I am not a legal expert in any way.

> Under these "source available" licenses, I can still sell the software within some kind of package correct? Like if I create my own PR linter I can use Grai and still sell it?

"Source available" means the source is accessible. Whether you can sell the software depends on the license. In the case of the Elastic License v2 as used here, I believe you could re-sell the works but you cannot re-license and the original limitations will remain which include providing as a hosted/managed service. There are other limitations too, the limitations around license keys functionality could be a significant hindrance depending on specific use and implementation.

> I just can't host grai with some observability and sell it? Or am I misunderstanding?

That is kind of the most significant limitation, but ultimately you are subject to the detail of all limitations:

~~

>> You may not provide the software to third parties as a hosted or managed service, where the service provides users with access to any substantial set of the features or functionality of the software.

>> You may not move, change, disable, or circumvent the license key functionality in the software, and you may not remove or obscure any functionality in the software that is protected by the license key.

>> You may not alter, remove, or obscure any licensing, copyright, or other notices of the licensor in the software. Any use of the licensor’s trademarks is subject to applicable law.

~~

Note that there's nothing about selling at all. Also think about how widely that first limitation could cover different types of use-case. And, as touched on above, that second limitation could be used in quite a protective/combative way to make significant parts of the software unusable in re-use.

Re: Launch HN: Grai (YC S22) – Open-Source Data Observability Platform

#22

Post by jedberg » Mon, Jul 17, 2023, 9:10 PM UTC

ersatz_username wrote: post #12

ssddanbrown wrote: post #8

The license chosen [1] (Elastic License 2.0) is one that isn't considered open source by many, due to not being OSD [2] compatible. Were you aware of this before marketing as open source and, out of interest, does the license & usage of "open source" come into conversation when going through the YC process? [1] https://github.com/grai-io/grai-core/blob/master/LICENSE [2] https://opensource.org/osd/

We are pretty open to feedback on licensing and have gone back and forth internally because, frankly, we'd rather use a copy-left license. We believe a project like this needs financial backing and a dedicated team driving development along but therein lies the tension. The common monetization paths either feature-lock critical self-hosted capabilities like SSO behind a paywall and/or monetize behind a cloud hosted o…

> We believe a project like this needs financial backing and a dedicated team driving development

What benefits do you get from being open source other than the OS stamp of approval?

Perhaps the solution is to just go closed source. I'm all for open source, but I'm not the biggest fan of open core or source available. All it does it hurt the business with little benefit to me. I'd rather you make more money and support me or go full altruistic and make it truly open source.

Re: Launch HN: Grai (YC S22) – Open-Source Data Observability Platform

#23

Post by ersatz_username » Mon, Jul 17, 2023, 9:16 PM UTC

jedberg wrote: post #22

ersatz_username wrote: post #12

Earlier quoted context omitted.

We are pretty open to feedback on licensing and have gone back and forth internally because, frankly, we'd rather use a copy-left license. We believe a project like this needs financial backing and a dedicated team driving development along but therein lies the tension. The common monetization paths either feature-lock critical self-hosted capabilities like SSO behind a paywall and/or monetize behind a cloud hosted o…

> We believe a project like this needs financial backing and a dedicated team driving development What benefits do you get from being open source other than the OS stamp of approval? Perhaps the solution is to just go closed source. I'm all for open source, but I'm not the biggest fan of open core or source available. All it does it hurt the business with little benefit to me. I'd rather you make more money and suppo…

We aren't open source because we want to get anything out of it is the short answer. Of course to each their own but I've personally gotten a ton of value from open core tools in the past.

Re: Launch HN: Grai (YC S22) – Open-Source Data Observability Platform

#24

Post by pjot » Mon, Jul 17, 2023, 9:28 PM UTC

I recently demoed an observability platform with another company, and one of my biggest gripes was that we weren’t able to “observe” the error before it actually made it to the open.

And that it took 2+ weeks to train their models with the table metadata - so time to value for my team was always “in two weeks”.

Glad to see y’all going against that trend!

Re: Launch HN: Grai (YC S22) – Open-Source Data Observability Platform

#25

Post by kevinmershon » Mon, Jul 17, 2023, 9:31 PM UTC

boredemployee wrote: post #9

kevinmershon wrote: post #5

Your intro video started with the assumption, I think, that a team already has some infra relating to this called DBT. It would be nice to have a video for onboarding from scratch assuming there's no data prior effort toward data observability.

Just a side note, DBT is being required everywhere these days

I believe you w.r.t. tech-first companies. I work in a tiny software dept in a small service company and we have no infrastructure like this at all. It would be nice to know how I can go from zero-to-Grai.

Re: Launch HN: Grai (YC S22) – Open-Source Data Observability Platform

#26

Post by swordsmith8 » Mon, Jul 17, 2023, 9:59 PM UTC

Thanks for sharing! Seems like this is a dbt-centric lineage tool that surfaces failed tests in the lineage itself?

Unlike a data observability platform like Monte Carlo which proactively monitors data, am I correct in assuming that your solution is less focused on data observability (i.e. monitoring production data and conducting root cause analysis / impact analysis) and more on ensuring reliable CI/CD?

Re: Launch HN: Grai (YC S22) – Open-Source Data Observability Platform

#27

Post by whytai » Mon, Jul 17, 2023, 10:11 PM UTC

How do you guys do the static analysis on the queries? I notice you support dbt, bigquery etc, but all of our companies pipelines are in airflow. That makes the static analysis difficult because we're dealing with arbitrary python code that programmatically generates queries :).

Any plans to support airflow in the future? Would love to have something like this for our companies 500k+ airflow jobs.

Re: Launch HN: Grai (YC S22) – Open-Source Data Observability Platform

#28

Post by ersatz_username » Mon, Jul 17, 2023, 11:24 PM UTC

pjot wrote: post #24

I recently demoed an observability platform with another company, and one of my biggest gripes was that we weren’t able to “observe” the error before it actually made it to the open. And that it took 2+ weeks to train their models with the table metadata - so time to value for my team was always “in two weeks”. Glad to see y’all going against that trend!

Really appreciate the kind words. If you don't mind my asking, what sort of issues were y'all experiencing that prompted you to start looking for solutions now?

Re: Launch HN: Grai (YC S22) – Open-Source Data Observability Platform

#29

Post by ersatz_username » Mon, Jul 17, 2023, 11:39 PM UTC

whytai wrote: post #27

How do you guys do the static analysis on the queries? I notice you support dbt, bigquery etc, but all of our companies pipelines are in airflow. That makes the static analysis difficult because we're dealing with arbitrary python code that programmatically generates queries :). Any plans to support airflow in the future? Would love to have something like this for our companies 500k+ airflow jobs.

It depends a bit on your stack. Out of the box it does a lot with the metadata produced by the tools your using. With something like dbt we can do things like extract your test assertions while for postgres we might use database constraints.

More generally we can embed the transformation logic of each stage of your data pipelines into the edge between nodes (like two columns). Like you said, in the case of SQL there are lots of ways to statically analyze that pipeline but it becomes much more complicated with something like pure python.

As an intermediate solution you can manually curate data contracts or assertions about application behavior into Grai but these inevitably fall out of sync with the code.

Airflow has a really great API for exposing task level lineage but we've held off integrating it because we weren't sure how to convert that into robust column or field level lineage as well. How are y'all handling testing / observability at the moment?

Re: Launch HN: Grai (YC S22) – Open-Source Data Observability Platform

#30

Post by ersatz_username » Mon, Jul 17, 2023, 11:45 PM UTC

swordsmith8 wrote: post #26

Thanks for sharing! Seems like this is a dbt-centric lineage tool that surfaces failed tests in the lineage itself? Unlike a data observability platform like Monte Carlo which proactively monitors data, am I correct in assuming that your solution is less focused on data observability (i.e. monitoring production data and conducting root cause analysis / impact analysis) and more on ensuring reliable CI/CD?

I wouldn't personally draw such a bright line between monitoring and reliable CI/CD. That division definitely exists but partly as a product of the complexity introduced by fragmented data systems. In some ways an ideal world is one where the need for extraordinarily complex monitoring tools is actually pretty limited because we had tools to validate end to end data pipelines before making code changes if that makes sense.

We actually already do data monitoring as well although we haven't built the specific alerting features of Monte Carlo. There are quite a few tools that do that really well so it's not our focus at the moment.