Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Elementary (YC W22) – Open-source data observability

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 41 posts

Re: Launch HN: Elementary (YC W22) – Open-source data observability

#21

Post by dang »

tutuca wrote: post #19

What's with this comment section? there's a lot of dead comments...

Someone sent out a link or something and it led to a bunch of booster comments. That's not allowed on HN, so I killed the comments and emailed the founders asking them to make it stop.

The guidelines say not to do it: https://news.ycombinator.com/newsguidelines.html, the FAQ says not to do it: https://news.ycombinator.com/newsfaq.html, and the Launch HN advice we give to YC startups says to "make sure" (in bold!) not to do it: https://news.ycombinator.com/yli.html. But people who aren't familiar with HN's conventions still end up doing it sometimes.

We make a distinction between innocent mistakes (which are usually obvious) and repeat offenses (where people usually try to cover their tracks). The former isn't a big deal, the latter we ban accounts for.

Re: Launch HN: Elementary (YC W22) – Open-source data observability

#22

Post by mateuszklimek »

windsquirrel wrote: post #18
mateuszklimek wrote: post #13

Hey Maayan and Or, Nice project, at re_data we just got over a lot of your new updates and it seems a quite large part of your project is "inspired" by code from our library https://github.com/re-data/re-data . Even with parts, we are not especially proud of ;) If you decide to copy not only ideas but a big part of internal implementation, I think you should include that information in your LICENSE. Cheers

Is the idea here that it's inspired by re_data due to using dbt transformations underneath or because it's reposted looking nearly the same? (or both?) Looks like much of the lineage code is also largely a wrapper around this library: https://github.com/reata/sqllineage Would be curious to understand the project's purpose and unique contributions vs. the underlying dependencies powering it as there seems to be some a…

It's "inspired" the dbt transformation part by using the same models and logic/part of code of generating them. We, for example, had a funny thing of computing metrics in 4 threads via multiple dbt models, and this is also done in elementary in a very similar way :)

The lineage part is independent (re_data uses lineage from dbt), so I haven't looked into that much.

Re: Launch HN: Elementary (YC W22) – Open-source data observability

#24

Post by idomi »

mateuszklimek wrote: post #13

Hey Maayan and Or, Nice project, at re_data we just got over a lot of your new updates and it seems a quite large part of your project is "inspired" by code from our library https://github.com/re-data/re-data . Even with parts, we are not especially proud of ;) If you decide to copy not only ideas but a big part of internal implementation, I think you should include that information in your LICENSE. Cheers

Pretty strong accusation, are you sure re-data isn't "inspired" from Monte Carlo? :)

Re: Launch HN: Elementary (YC W22) – Open-source data observability

#25

Post by edublancas »

Congrats on the launch! As a former data scientist, I suffered from bad data on a daily basis. Can you provide some details on how anomalies are detected? Is it some kind of threshold-based approach defined by the user or are you running statistical analysis on user's data? Curious to learn more!

Re: Launch HN: Elementary (YC W22) – Open-source data observability

#26

Post by windsquirrel »

mateuszklimek wrote: post #22
windsquirrel wrote: post #18

Earlier quoted context omitted.

Is the idea here that it's inspired by re_data due to using dbt transformations underneath or because it's reposted looking nearly the same? (or both?) Looks like much of the lineage code is also largely a wrapper around this library: https://github.com/reata/sqllineage Would be curious to understand the project's purpose and unique contributions vs. the underlying dependencies powering it as there seems to be some a…

It's "inspired" the dbt transformation part by using the same models and logic/part of code of generating them. We, for example, had a funny thing of computing metrics in 4 threads via multiple dbt models, and this is also done in elementary in a very similar way :) The lineage part is independent (re_data uses lineage from dbt), so I haven't looked into that much.

Gotcha - I can see what you mean, appreciate the clarification

Re: Launch HN: Elementary (YC W22) – Open-source data observability

#27

Post by mateuszklimek »

idomi wrote: post #24
mateuszklimek wrote: post #13

Hey Maayan and Or, Nice project, at re_data we just got over a lot of your new updates and it seems a quite large part of your project is "inspired" by code from our library https://github.com/re-data/re-data . Even with parts, we are not especially proud of ;) If you decide to copy not only ideas but a big part of internal implementation, I think you should include that information in your LICENSE. Cheers

Pretty strong accusation, are you sure re-data isn't "inspired" from Monte Carlo? :)

It is! But it doesn't have Monte Carlo code in it :)

And it's open-source so it's generally okay to do that, but it should be reflected in the LICENSE.

Re: Launch HN: Elementary (YC W22) – Open-source data observability

#30

Post by Maayansa »

mateuszklimek wrote: post #22
windsquirrel wrote: post #18

Earlier quoted context omitted.

Is the idea here that it's inspired by re_data due to using dbt transformations underneath or because it's reposted looking nearly the same? (or both?) Looks like much of the lineage code is also largely a wrapper around this library: https://github.com/reata/sqllineage Would be curious to understand the project's purpose and unique contributions vs. the underlying dependencies powering it as there seems to be some a…

It's "inspired" the dbt transformation part by using the same models and logic/part of code of generating them. We, for example, had a funny thing of computing metrics in 4 threads via multiple dbt models, and this is also done in elementary in a very similar way :) The lineage part is independent (re_data uses lineage from dbt), so I haven't looked into that much.

While writing our dbt project we looked into more than 60 dbt projects to learn from prior work while developing Elementary, and have been inspired by different things in different places. You're right that we were inspired by a couple of techniques you used, one being that creative way to improve performance (though the 4 thread setting itself is the dbt recommendation in their docs). Another is using z-score for anomaly detection, which we saw in a number of related projects and it's widely used in the industry.

In terms of the lineage, you can see in the code that we mostly rely on query and access history that exist in Snowflake and Bigquery to parse the queries and learn about the connection between nodes in the graph. We use other python libraries like sqlfluff and sqllineage as low level parsers for some specific use cases which we extend and solve many things on top of them. Actually we're heavy open source users, depending on around 20 libraries, all MIT or Apache.

Post reply on HN