Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Dataform (YC W18) – Build Reliable SQL Data Pipelines as a Team

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 39 posts

Re: Launch HN: Dataform (YC W18) – Build Reliable SQL Data Pipelines as a Team

#21

Post by thekhatribharat »

G2H wrote: post #14
thekhatribharat wrote: post #11

1. What do you think of NoETL and SQL++? Ref: https://www.couchbase.com/products/analytics 2. Do you think adopting a tool like Dataform makes more sense at mid-stage companies? Ref: https://thinkgrowth.org/the-startup-founders-guide-to-analyt...

1. I think NoETL products are complementary solutions. NoETL can be a great option for ad hoc queries and when performing a one-off analysis. But for your company's analytics, you want to build a suite of well defined, reliable and documented datasets that power you dashboards and your regular analytics. Another way to say it is that with NoETL, you write one query to answer one complex question. With ETL (actually E…

> well defined, reliable and documented datasets

Can you elaborate this?

NoETL targets semi-structured and schema-less data models prevalent in NoSQL/HTAP data stores, so if well defined relates to well defined schemas, it maybe doesn't apply here.

As far as reliability (aka reproducibility) and documentation is concerned, I guess it's more about building tools that promote sound engineering processes like version control, modules (aka reusable snippets), documentation (could be Jupyter notebooks, etc.) and these tools could be built for NoETL systems too.

Re: Launch HN: Dataform (YC W18) – Build Reliable SQL Data Pipelines as a Team

#22

Post by devbug »

1ewish wrote: post #7
thom wrote: post #6

I take it everything is based on schedules, there’s no runtime magic to handle incremental updates in a timely or efficient way? I’ve been watching @frankmcsherry’s Materialize.io with interest, for example.

Possibly controversial, but I don't believe there are that many situations in data analytics where you really need realtime/streaming analytics. It usually comes at the cost of significant additional complexity and maintenance, that ultimately hurts reproducibility and agility. Having said that, we are working on triggering table builds from watching changes to consumed data sets, which paired with an incremental tab…

I disagree.

Near realtime (say sub 1-minute) feeds can be leveraged to extract a lot of value out of the data you're collecting. Perhaps maybe not for your typical SaaS startup, but for free-to-play video games you end up leaving so much value on the table if you can't quickly (automatically) respond to spend or churn indicators.

I think a good rule of thumb for this is looking at how fast your decisions are made. If you're making decisions daily then refreshing your data every 24 hours is probably good enough. If you're making decisions every hour, then every 60 minutes is probably good enough.

Another factor is scale. When you're dealing with 6-figure CCUs and are trying to optimize conversion or retention through split-testing, figuring out which variants are anomalously poorly performing quickly can save you a whole lot of money.

I reckon there's at least a 50 titles that can benefit from streaming analytics, which immediately affects anywhere from 100-500 employees (analysts, engineers) and likely influences over 1000 (broader company). That's a significant portion of the field.

Re: Launch HN: Dataform (YC W18) – Build Reliable SQL Data Pipelines as a Team

#23

Post by cgio »

devbug wrote: post #22
1ewish wrote: post #7

Earlier quoted context omitted.

Possibly controversial, but I don't believe there are that many situations in data analytics where you really need realtime/streaming analytics. It usually comes at the cost of significant additional complexity and maintenance, that ultimately hurts reproducibility and agility. Having said that, we are working on triggering table builds from watching changes to consumed data sets, which paired with an incremental tab…

I disagree. Near realtime (say sub 1-minute) feeds can be leveraged to extract a lot of value out of the data you're collecting. Perhaps maybe not for your typical SaaS startup, but for free-to-play video games you end up leaving so much value on the table if you can't quickly (automatically) respond to spend or churn indicators. I think a good rule of thumb for this is looking at how fast your decisions are made. If…

That’s why I don’t like the term real time. Decision time should be the criterion.

Re: Launch HN: Dataform (YC W18) – Build Reliable SQL Data Pipelines as a Team

#24

Post by opportune »

>In Dataform, you build data pipelines in SQL, and our open source framework lets you seamlessly define dependencies, build incremental tables and reuse code across scripts.

Do you have any plan to add a visual interface? Unfortunately many analysts I work with are not really capable in SQL, or don't fully grok SQL features.

Also, how you plan on creating a moat against cloud companies putting the functionality of your project in their own proprietary solutions? For example, in Azure you can already use the Azure Data Lake stack of ADL Storage + Analytics + Data Factory to do what you are doing. These have IDE integrations, CICD integrationos, git integrations, are first class citizens in their cloud interfaces, etc.

Re: Launch HN: Dataform (YC W18) – Build Reliable SQL Data Pipelines as a Team

#25

Post by buremba »

1ewish wrote: post #20
buremba wrote: post #19

Earlier quoted context omitted.

What’s the rationale behind using JS instead of Python/Jinja? Any insights on that?

Great question :) 1. Speed. We wanted compilation to really, really fast. 2. In all honesty we just weren't big fans of Jinja, having used it for quite a while. JS templating is OK out the box, and we are considering React like syntax in the future. 3. Love it or hate it, NPM packages are pretty easy to work with and we are working on a number of packages at the moment. 4. When you start to look at things like UDFs a…

I see your point but I think that data analysts are usually familiar with Python, not JS. The UI is cool and intuitive, it looks like you copied most of the concepts except Jinja from DBT. That's perfectly OK though, let's see where it goes!

Re: Launch HN: Dataform (YC W18) – Build Reliable SQL Data Pipelines as a Team

#27

Post by khaledh »

What's the core transformation technology running the SQL transforms? Are you relying on the cloud data warehouse itself (i.e. BigQuery, Amazon Redshift or Snowflake) to run the SQL transforms? Asking because the transform workload usually dominates the query workload if they're not properly separated.

Re: Launch HN: Dataform (YC W18) – Build Reliable SQL Data Pipelines as a Team

#28

Post by justjico »

khaledh wrote: post #27

What's the core transformation technology running the SQL transforms? Are you relying on the cloud data warehouse itself (i.e. BigQuery, Amazon Redshift or Snowflake) to run the SQL transforms? Asking because the transform workload usually dominates the query workload if they're not properly separated.

Just from the docs, it appears to rely on the warehouse itself. It’s basically SQL + js templates make data engineers/analysts jobs easier.

Re: Launch HN: Dataform (YC W18) – Build Reliable SQL Data Pipelines as a Team

#30

Post by thundergolfer »

opportune wrote: post #24

>In Dataform, you build data pipelines in SQL, and our open source framework lets you seamlessly define dependencies, build incremental tables and reuse code across scripts. Do you have any plan to add a visual interface? Unfortunately many analysts I work with are not really capable in SQL, or don't fully grok SQL features. Also, how you plan on creating a moat against cloud companies putting the functionality of yo…

Check out the demo. There’s a significant UI available.
Post reply on HN