What does features does Grai have that makes it superior to DataHub?
As I see it, DataHub already gives you a data lineage. Is that not enough?
Launch HN: Grai (YC S22) – Open-Source Data Observability Platform
41–46 of 46 posts
Re: Launch HN: Grai (YC S22) – Open-Source Data Observability Platform
#42I think the space of DQ is pretty competitive. Theres datafold [1], databend (acquired by ibm), atlan, greatexpectations to name a few, doing very similar things. Just looking at the video I could not figure out what's differentiated. I hope you have success in the space. [1] https://www.datafold.com
A lot of these tools are very, very different from each other so it's hard to address each individually. Just by way of example, databend is a full on datawarehouse while greatexpectations is a testing framework evaluating data assertions (i.e. "I see there are nulls, but you wrote a test which says there shouldn't be"). Here are some things we think are really important though 1. Data quality testing ideally happens…
My bad, I was actually referring to databand [1]
Re: Launch HN: Grai (YC S22) – Open-Source Data Observability Platform
#43Earlier quoted context omitted.
> We believe a project like this needs financial backing and a dedicated team driving development What benefits do you get from being open source other than the OS stamp of approval? Perhaps the solution is to just go closed source. I'm all for open source, but I'm not the biggest fan of open core or source available. All it does it hurt the business with little benefit to me. I'd rather you make more money and suppo…
Don't its customers get the benefit of being able to self-host and modify for their own internal use? Seems like a big benefit to me...
My point is if it's a commercial entity, I'd rather pay them to make the modifications and then maintain it than pay my own engineers to do it.
Re: Launch HN: Grai (YC S22) – Open-Source Data Observability Platform
#44How do you guys do the static analysis on the queries? I notice you support dbt, bigquery etc, but all of our companies pipelines are in airflow. That makes the static analysis difficult because we're dealing with arbitrary python code that programmatically generates queries :). Any plans to support airflow in the future? Would love to have something like this for our companies 500k+ airflow jobs.
It depends a bit on your stack. Out of the box it does a lot with the metadata produced by the tools your using. With something like dbt we can do things like extract your test assertions while for postgres we might use database constraints. More generally we can embed the transformation logic of each stage of your data pipelines into the edge between nodes (like two columns). Like you said, in the case of SQL there…
For testing:
- we have a dedicated dev environment for analysts to experience a dev/test loop. None of the pipelines can be run locally unfortunately.
- we have CI jobs and unit tests that are run on all pipelines
Observability:
- we have data quality checks for each dataset, organized by tier. This also integrates with our alerting system to send pagers when data quality dips.
- Airflow and our query engines hive/spark/presto each integrate with our in-house lineage service. We have a lineage graph that shows which pipelines produce/consume which assets but it doesn't work at the column level because our internal version of Hive doesn't support that.
- we have a service that essential surfaces observability metrics for pipelines in a nice ui
- our airflow is integrated with pagerduty to send pagers to owning teams when pipelines fail.
We'd like to do more, but nobody has really put in the work to make a good static analysis system for airflow/python. Couple that with the lack of support for column level lineage OOTB and it's easy to get into a mess. For large migrations (airflow/infra/python/dependecy changes) we still end up doing adhoc analysis to make sure things go right, and we often miss important things.
Happy to talk more about this if you're interested.
Re: Launch HN: Grai (YC S22) – Open-Source Data Observability Platform
#45Earlier quoted context omitted.
Don't its customers get the benefit of being able to self-host and modify for their own internal use? Seems like a big benefit to me...
My point is if it's a commercial entity, I'd rather pay them to make the modifications and then maintain it than pay my own engineers to do it.
Yes, but if I want to make larger modifications than would make sense for the core project, I'd like to have the ability to self host my modified version (and ideally have a support contract as well, if they're into that).
So you asked what the point of doing this is for them, from a business perspective. I think the point is marketing / smoothing the sales process. I feel much better about using SaaS products that I know I can self host if necessary, even if I'm unlikely to actually do so.
Frankly, it's just the same reason I prefer any of my tools to be open source. I don't like using proprietary programming languages or frameworks, because I can't fix things that are broken even if I want to. This remains true even though I can count the number of times I've actually done this on one hand.
Re: Launch HN: Grai (YC S22) – Open-Source Data Observability Platform
#46Earlier quoted context omitted.
Thanks for the link. Some personal thoughts: I think the effort to standardize what is meant by a term like "open source" is generally good, but I also think the meaning of language is always up for debate, and the OSI's definitions are only right if they are useful. Of the two clauses you pulled out of the EL2 license, the first one - "You may not provide the software to third parties as a hosted or managed service…
> It seems totally reasonable and not violating the spirit of "open source" to say you can modify and self-host for any purpose, but you can't re-sell. Personally I would heavily disagree with that, and that statement is something I see as against the spirit of open source. In my view, open source and free software mainly intend to use licensing to put the freedoms and rights of the code & it's users in front of thos…
It's just an honest disagreement. I don't think your opinion is the way, but I don't begrudge it.
I'm just not an ideologue. What matters to me is having as much software tooling that is as useful to me as possible. I consider tools that I can modify and run myself to be more useful than those that are proprietary. But I don't require or demand the ability to re-sell someone else's software; that isn't a capability that is useful to me. That capability is pretty much entirely only useful to Amazon and Google, and that's just not something I care about optimizing for.