Viewing profile — hichkaker

Thank you for mentioning Data Diff! Founder of Datafold here. We built Data Diff to solve a variety of problems that we encountered as data engineers: (A) Testing SQL code changes …

As a data engineer, validating your SQL code as you develop pipelines is hard. It requires running the code to materialize a "development" dataset, then auditing the resulting data…

I did data engineering for 6 years and am building a company to automate SQL validation for dbt users. First, by “testing SQL pipelines”, I assume you mean testing changes to SQL c…

Hi HN: We at Datafold are excited to announce a new release of data-diff (https://github.com/datafold/data-diff), an open-source tool that efficiently compares tables within or acr…

As a data engineer myself, totally agree about the abuse of the word “data”. What we strived for when naming was to make it self-describing as much as possible. Since the tool does…

Gleb, Alex, Erez and Simon here – we are building an open-source tool for comparing data within and across databases at any scale. The repo is at https://github.com/datafold/data-d…

Datafold automates data pipeline testing for data engineers. With Datafold, data engineers can deal with data quality issues in the pull request by seeing how a change to source co…

We actually built a tool exactly for that: https://www.datafold.com/data-diff The core use case is helping data engineers building analytical pipelines test the changes to their co…

Datafold automates data pipeline testing for data engineers. With Datafold, data engineers can deal with data quality issues in the pull request by seeing how a change to source co…

Datafold (YC S20) | https://www.datafold.com Datafold automates data pipeline testing for data engineers. With Datafold, data engineers can deal with data quality issues in the pul…

We've been using Tella for product demos. A huge upgrade from hacking together QuickTime recordings!

Thank you for sharing! I assume we are talking about analytical, not transactional data: > Diff'ing data is one of the weakest and most cumbersome ways to verify correctness. It de…

Thank you for pointing it out, I've never noticed but now that you brought it up will do a few more iterations on our design to avoid the collision.

Support for MS SQL is in progress. We can deploy in your DC using with Docker or K8S.

If diffing datasets within the same physical database, generate SQL, execute in the database, analyze and render results. If diffing datasets across physically different databases,…

Totally fair! We offer free trials for on-prem. Please ping us at hn@datafold.com to set it up.

You raised a great point. Data testing methods can perhaps be broken down to two main categories: 1. "Unit testing" – validating assumptions about the data that you define explicit…

Thank you for the feedback! Agree with you about the niche. Diff is our first tool that helps test changes in the ETL code , and the impact is correlated with the size and complexi…

Thank you for the suggestion! We're leaning towards Option 1: free diffing for datasets < 1M rows. Option 2 seems a bit tricker since we are in a way creating a new tool category a…

If you can materialize those .bcp files as tables, then yes, absolutely.

Thank you for asking deep questions and providing specific examples! The degree of scale and complexity varies significantly between companies, so we definitely can't claim that ev…

> But will this problem be big enough for VC investment is the question? That's a great question. Thinking about where problems arise in data pipelines, there are fundamentally two…