Looks like yet another ETL/ELT tool.
We have been using composable with quite a bit of success: https://dev.composable.ai
Launch HN: Dataform (YC W18) – Build Reliable SQL Data Pipelines as a Team
31–39 of 39 posts
Re: Launch HN: Dataform (YC W18) – Build Reliable SQL Data Pipelines as a Team
#32Looks like yet another ETL/ELT tool. We have been using composable with quite a bit of success: https://dev.composable.ai
I'm routinely confused about the role these sort of tools play. I conceptually understand what they're doing but weren't these problems solved a long time ago with materialized views and foreign data wrappers?
The hard part of ETL for me has always been gracefully handling the outliers, the data that you need to look at within the context of multiple rows (e.g. duplicate rows that aren't exactly the same so it's not a simple SELECT DISTINCT) in order to make the correct decision, or the entity matching/mapping/categorization that often is necessary. Having the lookup tables that often need manual oversight when new entries show up that haven't yet been properly tagged. Or if you're ingesting address data and you want to normalize it through a geocoder, where exactly does that occur in these SQL-templating "pipelines"?
I feel like these are basically focused on moving data between different DBs and generating group-by queries to populate some rollup tables?
Re: Launch HN: Dataform (YC W18) – Build Reliable SQL Data Pipelines as a Team
#33What's the core transformation technology running the SQL transforms? Are you relying on the cloud data warehouse itself (i.e. BigQuery, Amazon Redshift or Snowflake) to run the SQL transforms? Asking because the transform workload usually dominates the query workload if they're not properly separated.
Just from the docs, it appears to rely on the warehouse itself. It’s basically SQL + js templates make data engineers/analysts jobs easier.
Yup, the warehouse or query engine does the heavy lifting, we do the orchestration and make easier to develop these pipelines.
Re: Launch HN: Dataform (YC W18) – Build Reliable SQL Data Pipelines as a Team
#34Does it work cross databases? Ie, if data is stored in multiple databases and needs to be combined for analytics? Currently, we are getting all data to S3 via DMS and running a script on top of this to combine data
Dataform still requires you to move your data to a single place where it can be queried together, whether that's S3 with Presto, Redshift spectrum, or something else.
There are lots of great products for moving data like this, Stitch, Fivetran are both great and we've used them ourselves, but they don't help you manage complex data transformation processes.
Re: Launch HN: Dataform (YC W18) – Build Reliable SQL Data Pipelines as a Team
#35>In Dataform, you build data pipelines in SQL, and our open source framework lets you seamlessly define dependencies, build incremental tables and reuse code across scripts. Do you have any plan to add a visual interface? Unfortunately many analysts I work with are not really capable in SQL, or don't fully grok SQL features. Also, how you plan on creating a moat against cloud companies putting the functionality of yo…
A UI for building data transformations (SQl queries) isn't on our immediate roadmap, but we have thought about it.
We are taking a bit of a bet here that the industry is moving towards SQL, and we see a growing interesting in analysts wanting to learn it.
Moat: Always a concern! Short term we think most of our unique value comes from the collaborative web development experience which is not OSS currently. Also staying platform agnostic is important to us. Longer term, we have some plans which without going into details mostly focuses on building value around a community and our SQL ecosystem, but not ready to get into details just yet.
Re: Launch HN: Dataform (YC W18) – Build Reliable SQL Data Pipelines as a Team
#36Looks like yet another ETL/ELT tool. We have been using composable with quite a bit of success: https://dev.composable.ai
Indeed. Dataform however only does the T (transformation) in ELT. Most businesses are centralising an exploding diversity of raw data in the data warehouse and Dataform helps them manage the data there.
One other main difference with a product like composable is that Dataform is built with software engineering best practices in mind. The transformations are written in code (SQL) rather than a GUI. It enables version control, reusable components for example and makes it easier to manage a large number of interdependent transformations.
Re: Launch HN: Dataform (YC W18) – Build Reliable SQL Data Pipelines as a Team
#37Does this add any value beyond what PDTs in Looker provide?
Definitely! We see that PDT doesn't scale well with the the amount of data complexity you usually have to deal with in the warehouse. On top of that, the results of PDTs are only accessible to Looker. As your data team matures, there will be many other users and applications that need to take advantage of your transformed data in the warehouse. (e.g. modelling in Python) Dataform also brings other features not availa…
I much agree with this response. The term that I've heard to summarize all of the above is "semantic layer". That layer will run in the warehouse, vs. some external dashboard tool.
Re: Launch HN: Dataform (YC W18) – Build Reliable SQL Data Pipelines as a Team
#38Looks like yet another ETL/ELT tool. We have been using composable with quite a bit of success: https://dev.composable.ai
Indeed. Dataform however only does the T (transformation) in ELT. Most businesses are centralising an exploding diversity of raw data in the data warehouse and Dataform helps them manage the data there. One other main difference with a product like composable is that Dataform is built with software engineering best practices in mind. The transformations are written in code (SQL) rather than a GUI. It enables version…
To be fair, these other tools do this as well.
Re: Launch HN: Dataform (YC W18) – Build Reliable SQL Data Pipelines as a Team
#39Any success getting through the VRM process at a financial services firm? Your product looks very exciting.
Thank you! We haven't been through VRM yet but that's about to come up. If you have any tips, would love to hear your thoughts. I am at ghh@[domain]