what about the latency of data warehouses? how do you get around that?
Launch HN: Hightouch (YC S19) – Sync data from data warehouses to SaaS tools
21–30 of 51 posts
Re: Launch HN: Hightouch (YC S19) – Sync data from data warehouses to SaaS tools
#22Really, really love that it's just SQL. How is data mapped to the target API from the SQL projection? Are the columns themselves the actual API contract?
Cofounder here! Not quite. In Hightouch, you define your model (SQL) and create a sync (point-and-click or JSON/YAML). The syncs are declarative, not imperative. They don't map 1:1 to API calls by design. You tell us what you want the destination to look like, and we figure out how :). Kinda like how a database creates the best plan for your SQL query before executing it. Here's an example - https://i.imgur.com/05T5i…
What if the state of IsHireable changes in the system of record (SFDC)? Will Hightouch overwrite OLTP data with stale warehouse data?
Re: Launch HN: Hightouch (YC S19) – Sync data from data warehouses to SaaS tools
#23Interesting. How does Reverse ETL differ from ETL/ELT tools like Fivetran?
Re: Launch HN: Hightouch (YC S19) – Sync data from data warehouses to SaaS tools
#24what about the latency of data warehouses? how do you get around that?
Good callout. Sometimes, I joke that warehouse ingestion latency is the bane of my existence, but it's improving...
Our average customer runs Hightouch syncs roughly every hour, but we can actually run syncs up to every minute! HT has a lot of optimizations like only sending changes to destinations instead of all data every run.
On the warehouse side, we're seeing a lot of improvements. BigQuery has streaming insert APIs [0] implemented with a parallel database on the backend that's joined at read time. Combined with timestamp partitioned tables (sortable) and our in-warehouse diff'ing, you can actually create a streaming pipeline in Hightouch. Some companies like JetBlue are doing cool stuff with lambda views on top of Snowflake [1]. Our power users at Hightouch are running syncs as fast as every minute.
For wider context, we find 90%+ of business use cases to be just fine in batch. It's amazing to see how many people are still replacing... manual CSV workflows... with Hightouch :)
That said, there are some use cases for truly real-time workflows (e.g. a post-checkout email), and for that, customers either implement outside of Hightouch or lately, we've been fiddling around with letting customers plug directly into streams like Kafka, Kinesis, PubSub - though they lose the power of SQL aggregations _for now_.
Streaming SQL databases like Materialize [2] will fix this fundamentally, and Hightouch can connect to them. Email hello@hightouch.io if you want to try any of the new stuff!
[0]: https://cloud.google.com/bigquery/docs/write-api [1]: https://discourse.getdbt.com/t/how-to-create-near-real-time-... [2]: https://materialize.com/
Re: Launch HN: Hightouch (YC S19) – Sync data from data warehouses to SaaS tools
#25Earlier quoted context omitted.
Cofounder here! Not quite. In Hightouch, you define your model (SQL) and create a sync (point-and-click or JSON/YAML). The syncs are declarative, not imperative. They don't map 1:1 to API calls by design. You tell us what you want the destination to look like, and we figure out how :). Kinda like how a database creates the best plan for your SQL query before executing it. Here's an example - https://i.imgur.com/05T5i…
> You tell us what you want the destination to look like Implicit mapping between SQL to target is great, but how does the SQL author know what SQL to write in the first place? I've done no shortage of integrations like this, and there is no avoiding reading the target SaaS documentation to know what their schema looks like so I can shape data accordingly. Without that step, I can't even start writing SQL.
Not implicit but - declarative! Our goal is to provide enough context in our docs, app (e.g. autocomplete, automatic schema discovery, etc.), and resources to guide users through this and then recipes on top for common workflows!
We do a lot of validation upfront (at both the schema & data layer), and I think it's still early days there... this is a big opportunity IMO. Great callout.
We find people start with a simple SQL model + sync and then bounce back and forth and edit their queries as they explore our columns.
Re: Launch HN: Hightouch (YC S19) – Sync data from data warehouses to SaaS tools
#26If you don't mind me asking, how much time did it take to reach this maturity or quality as a product? Did you start in 2019 or before that?
Kiss your designer for me.
Re: Launch HN: Hightouch (YC S19) – Sync data from data warehouses to SaaS tools
#27Interesting. How does Reverse ETL differ from ETL/ELT tools like Fivetran?
On a high level, ETL/ELT is about sending data from your SaaS tools into your data warehouse (you are reading from different tools). Reverse ETL is about getting data from your warehouse into tools (writing into different tools). Building ELT is a fundamentally different technical challenge than building Reverse ETL. Aspects like types, rate limits, and destination state (knowing whether data already exists in a destination) are unique to Reverse ETL. Visibility becomes challenging too as some destinations have unique quirks, like API contracts where you write to them but you don’t know if the write was successful or completed until later. Writing to tools also requires references between objects (foreign keys onto existing data, like mapping Companies and Opportunities in Salesforce) that aren’t necessary in the ELT world.
From a product perspective, the UX is very different as well. Reverse ETL requires a lot more user input (ex: mapping which fields to update in a tool), whereas ELT typically mirrors data using a standard schema (without much user customization involved).
We are close partners with ELT tools like Fivetran, and you can see our partnership post here: https://fivetran.com/blog/fivetran-partners-with-hightouch-t...
Re: Launch HN: Hightouch (YC S19) – Sync data from data warehouses to SaaS tools
#28Earlier quoted context omitted.
Cofounder here! Not quite. In Hightouch, you define your model (SQL) and create a sync (point-and-click or JSON/YAML). The syncs are declarative, not imperative. They don't map 1:1 to API calls by design. You tell us what you want the destination to look like, and we figure out how :). Kinda like how a database creates the best plan for your SQL query before executing it. Here's an example - https://i.imgur.com/05T5i…
What if the state of IsHireable changes in the system of record (SFDC)? Will Hightouch overwrite OLTP data with stale warehouse data?
Short answer is yes, but here's why:
By syncing data to a particular field in Salesforce, you're effectively saying that the source of truth for that field is the warehouse, and not Salesforce. If you expect a human to update a field, then Salesforce is the source of truth for that field, and Hightouch shouldn't write to it!
What we typically see is that Salesforce contains data that's expected to be updated and maintained in the tool, and then other "read-only" fields coming from Hightouch.
Re: Launch HN: Hightouch (YC S19) – Sync data from data warehouses to SaaS tools
#29If you don't mind me asking, how much time did it take to reach this maturity or quality as a product? Did you start in 2019 or before that? Kiss your designer for me.
we started building this one August 2020 but honestly just had a lot of fun working on the design and UX! Conveying your feedback to him now!
Re: Launch HN: Hightouch (YC S19) – Sync data from data warehouses to SaaS tools
#30If you don't mind me asking, how much time did it take to reach this maturity or quality as a product? Did you start in 2019 or before that? Kiss your designer for me.
[deleted]