31–40 of 41 posts
Great job Elementary team! Does this in essence similar to the aws deeque project but fancier and more inclusive of edge cases, common scenarios? ( https://github.com/awslabs/deequ )
Hey Maayan and Or, Nice project, at re_data we just got over a lot of your new updates and it seems a quite large part of your project is "inspired" by code from our library https://github.com/re-data/re-data . Even with parts, we are not especially proud of ;) If you decide to copy not only ideas but a big part of internal implementation, I think you should include that information in your LICENSE. Cheers
Congrats on the launch! As a former data scientist, I suffered from bad data on a daily basis. Can you provide some details on how anomalies are detected? Is it some kind of threshold-based approach defined by the user or are you running statistical analysis on user's data? Curious to learn more!
Earlier quoted context omitted.
It's "inspired" the dbt transformation part by using the same models and logic/part of code of generating them. We, for example, had a funny thing of computing metrics in 4 threads via multiple dbt models, and this is also done in elementary in a very similar way :) The lineage part is independent (re_data uses lineage from dbt), so I haven't looked into that much.
While writing our dbt project we looked into more than 60 dbt projects to learn from prior work while developing Elementary, and have been inspired by different things in different places. You're right that we were inspired by a couple of techniques you used, one being that creative way to improve performance (though the 4 thread setting itself is the dbt recommendation in their docs). Another is using z-score for an…
Hey Maayan and Or, Nice project, at re_data we just got over a lot of your new updates and it seems a quite large part of your project is "inspired" by code from our library https://github.com/re-data/re-data . Even with parts, we are not especially proud of ;) If you decide to copy not only ideas but a big part of internal implementation, I think you should include that information in your LICENSE. Cheers
If you're going to make an accusation like this on HN, you should provide line by line evidence. Saying "you copied us" without any examples makes you incredible.
Elementary models like data_monitors_thread1, data_monitors_thread2, data_monitors_thread3, data_monitors_thread4, data_monitoring_metrics, latest_metrics, metrics_stats_for_anomalies, z_score, anomaly_detection, schema_schenages, etc. Existed before in re_data, are doing the same things and specifically for *_thread4 are not similar to anything you normally do in dbt.
And these similarities are also visible in code, for example here: the same usage of the undocumented dbt context feature.
# elementary
{% macro get_monitor_macro(monitor) %}
{%- set macro_name = monitor + '_monitor' -%}
{%- if context['elementary'].get(macro_name) -%}
{%- set monitor_macro = context['elementary'][macro_name] -%}
{%- else -%}
{%- set monitor_macro = context['elementary']['no_monitor'] -%}
{%- endif -%}
{{- return(monitor_macro) -}}
# re_data
{%- macro get_metric_macro(metric_name) %}
{% set macro_name = 're_data_metric' + '_' + metric_name %}
{% if context['re_data'].get(macro_name) %}
{% set metric_macro = context['re_data'][macro_name] %}
{%- else %}
{% set metric_macro = context[project_name][macro_name] %}
{% endif %}
{{ return (metric_macro) }}
Earlier quoted context omitted.
While writing our dbt project we looked into more than 60 dbt projects to learn from prior work while developing Elementary, and have been inspired by different things in different places. You're right that we were inspired by a couple of techniques you used, one being that creative way to improve performance (though the 4 thread setting itself is the dbt recommendation in their docs). Another is using z-score for an…
I think mateuszklimek is pointing out that the MIT license requires you to include the redata copyright in your source.
> Copyright [yyyy] [name of copyright owner]
https://github.com/elementary-data/elementary/blob/master/LI...