Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Hubble (YC S20) – Monitor data quality inside data warehouses

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 44 posts

Re: Launch HN: Hubble (YC S20) – Monitor data quality inside data warehouses

#31

Post by oliver101 »

iblaine wrote: post #30

What does the tech stack look like? Is there any caching for those situations where you may read the same historical data over & over?

Yes, we store the historical value of each test so you can always scroll back through time and see the state of the data warehouse at any given point.

For example, if you have a test that counts the number of rows "COUNT(*)" - that value will be recorded. So you can look back an hour/day/week and see how many rows the table had without executing any SQL. These values are stored in a time series db, so querying history is fast.

Our tech stack: monolith backend in python + postgres + react. The test themselves are all SQL queries and run in the data warehouse.

Re: Launch HN: Hubble (YC S20) – Monitor data quality inside data warehouses

#32

Post by sails »

oliver101 wrote: post #16
verhey wrote: post #12

How does hubble compare to Great Expectations or DBT for pipeline testing? It looks like more emphasis on automated profiling than "having to write and maintain lots of individual tests" and obviously hubble being a saas offering is the big difference? Also any plans to profile and test file-based stores as well? There's a lot that can go wrong in a pipeline before data even reaches BigQuery or Snowflake, and you may…

Thanks! We love DBT and take a lot of inspiration from their work. We’re putting a lot of effort into suggesting the right tests based on the data types, sources, and field names. A lot of these tests are pretty repetitive to write so we want to make it easy to spin them up. We’ve also found that keeping a history of the state of the warehouse over time is really useful context for determining whether a test has fail…

Very interesting tool, I am trying to do this with Dataform/Looker, and feel like some kind of inference like below would be great.

> this table tends to update every 30-40 minutes so we’ll set a threshold at an hour

Can you achieve these tests with metadata or do you need 100% read access to the database?

I also wonder if this would work as part of a Analytics Engineering CICD process? Something like how dbt cloud will block pull requests that fail certain criteria.

Re: Launch HN: Hubble (YC S20) – Monitor data quality inside data warehouses

#33

Post by sails »

texasbigdata wrote: post #19
scapecast wrote: post #10

Co-founder of intermix.io here (which we sold in March). We came more from the performance monitoring angle (specifically for Redshift), but then shifted to a product that works horizontally across all warehouses, to track usage, workflows and user engagement. "Shift to Data Products" was the narrative we started using in Q4 2019. If you read the copy on the current intermix.io website, I think you'll find yourself n…

Fantastic blog post, thanks for sharing. So I guess if you had to pick arbitrary revenue/data/fte cutoffs, do you see the org chart of these adopters as you’ve described looking a certain way? Let me try to rephrase that. Do you think there’s a step function of “here you need one DBA who is a holy librarian” and “here we need a gitlab styled data team with SLAs and the data equivalent of HR business partners who get…

Where is the blog post?

Re: Launch HN: Hubble (YC S20) – Monitor data quality inside data warehouses

#34

Post by sails »

scapecast wrote: post #10

Co-founder of intermix.io here (which we sold in March). We came more from the performance monitoring angle (specifically for Redshift), but then shifted to a product that works horizontally across all warehouses, to track usage, workflows and user engagement. "Shift to Data Products" was the narrative we started using in Q4 2019. If you read the copy on the current intermix.io website, I think you'll find yourself n…

> we got bought by a small PE Fund that is rolling the product into Xplenty

I'm interested to hear more about your experience building data warehouse related products, and perhaps learnings you've had along the way. I guess selling to PE wasn't the initial goal, but I'd imagine your product is very well suited to the Redshift space.

I've been working on Snowflake related products, and their adoption speaks to a world of new problems being created, similar to your product with Redshift. I suppose the risk is being squashed by Snowflake building the feature, or businesses migrating to something new (perhaps Redshift products have suffered because of Snowflake)

Basically, what do the battle scars look like :D

Re: Launch HN: Hubble (YC S20) – Monitor data quality inside data warehouses

#35

Post by oliver101 »

sails wrote: post #32
oliver101 wrote: post #16

Earlier quoted context omitted.

Thanks! We love DBT and take a lot of inspiration from their work. We’re putting a lot of effort into suggesting the right tests based on the data types, sources, and field names. A lot of these tests are pretty repetitive to write so we want to make it easy to spin them up. We’ve also found that keeping a history of the state of the warehouse over time is really useful context for determining whether a test has fail…

Very interesting tool, I am trying to do this with Dataform/Looker, and feel like some kind of inference like below would be great. > this table tends to update every 30-40 minutes so we’ll set a threshold at an hour Can you achieve these tests with metadata or do you need 100% read access to the database? I also wonder if this would work as part of a Analytics Engineering CICD process? Something like how dbt cloud w…

Metadata is a valuable place for finding information like load times, rows inserted / updated. Currently we just rely on read-access and raw SQL. A common way users are doing this now (and we are internally for our analytics data) is using, for example, the Fivetran logs table to monitor ingestion times and inserted rows, rather than querying the raw tables.

For CICD, absolutely we want to support this as well as stopping/conditional execution in DAGs (e.g. airflow). We’re launching webhooks very soon

Re: Launch HN: Hubble (YC S20) – Monitor data quality inside data warehouses

#36

Post by scapecast »

sails wrote: post #34
scapecast wrote: post #10

Co-founder of intermix.io here (which we sold in March). We came more from the performance monitoring angle (specifically for Redshift), but then shifted to a product that works horizontally across all warehouses, to track usage, workflows and user engagement. "Shift to Data Products" was the narrative we started using in Q4 2019. If you read the copy on the current intermix.io website, I think you'll find yourself n…

> we got bought by a small PE Fund that is rolling the product into Xplenty I'm interested to hear more about your experience building data warehouse related products, and perhaps learnings you've had along the way. I guess selling to PE wasn't the initial goal, but I'd imagine your product is very well suited to the Redshift space. I've been working on Snowflake related products, and their adoption speaks to a world…

there are always things the warehouses can't build themselves.

For example, with intermix.io, it was the tracing we had built for other tools like Looker and dbt. The insight was that the result of a DAG involves many different calculations across different tables. The metadata only tells you that the steps happened, but doesn't tell you in what sequence they happened, where the "hiccup", latency, etc.

Redshift is clearly suffering from Snowflake. I wrote about that in my post-mortem. That post also has a few battle scars:

https://medium.com/@larskamp/why-we-sold-intermix-io-to-priv...

Ping me on LinkedIn if you care to hear more :-)

Re: Launch HN: Hubble (YC S20) – Monitor data quality inside data warehouses

#37

Post by scapecast »

hamzahc wrote: post #25
scapecast wrote: post #10

Co-founder of intermix.io here (which we sold in March). We came more from the performance monitoring angle (specifically for Redshift), but then shifted to a product that works horizontally across all warehouses, to track usage, workflows and user engagement. "Shift to Data Products" was the narrative we started using in Q4 2019. If you read the copy on the current intermix.io website, I think you'll find yourself n…

> My experience is that monitoring data quality is still an under-appreciated discipline. We agree with this a lot, we found there are often a lot of unknown unknowns that drive data issues, and a lot of teams aren’t sure of where to start. It’s why we’re spending so much time on trying to make relevant tests in Hubble that are easy to set up and use (and then let users create custom tests once they get the hang of i…

Awesome - just followed up on your ping!

Re: Launch HN: Hubble (YC S20) – Monitor data quality inside data warehouses

#38

Post by Kye »

oliver101 wrote: post #28
_Microft wrote: post #2

Have you considered picking a different name? Searching for "Hubble" for whatever reason is going to return millions of irrelevant results for your customers.

Yeah we called this project hubble long before we were worried about SEO. Actually, the name does relate back to Edwin Hubble. We previously worked together on an internal data tool called Telescope (it was used for annotating medical images for computer vision). The telescope project slowly evolved into the product we have today. So we changed the name to our favourite telescope. I have a fondness for the Hubble tel…

The main thing is to be mindful of keywords you target. Don't do as another commenter suggested and target hubble data[0] unless you apply what you make to actual Hubble data. Like AWS did with its Open Data thing that comes up for that keyword.

The telescope is older than the web and is what every single person on the planet with some access to space-related media thinks of when they think of Hubble. Think long tail, not one or two keywords. Hubble data is out unless you go with a telescope-related project, but you already rank indirectly for hubble data warehouse.

[0] https://news.ycombinator.com/item?id=24229880

Re: Launch HN: Hubble (YC S20) – Monitor data quality inside data warehouses

#39

Post by gk1 »

Kye wrote: post #29
switz wrote: post #23

Earlier quoted context omitted.

The product literally just launched -- give it a few weeks, it'll show up.

I don't know who's advising you on SEO, but you will not ever outrank STSCI, NASA, ESA, AWS Open Data's HSTS archive, The Planetary Society, the National Academy of Sciences, or the ESO on "hubble data" as long as Hubble is still what people think of when they hear Hubble. The telescope and related sites/agencies/organizations have a 22 year head start building a relevant link profile in Google. And if you did, Googl…

You're getting carried away with the example "hubble data." The point is that people will modify their search terms until they find the company they're looking for. If they don't know the company they're looking for then they will search by use case (eg, "detect data drift"), in which case the search results for the company name don't matter.

Re: Launch HN: Hubble (YC S20) – Monitor data quality inside data warehouses

#40

Post by gk1 »

Kye wrote: post #38
oliver101 wrote: post #28

Earlier quoted context omitted.

Yeah we called this project hubble long before we were worried about SEO. Actually, the name does relate back to Edwin Hubble. We previously worked together on an internal data tool called Telescope (it was used for annotating medical images for computer vision). The telescope project slowly evolved into the product we have today. So we changed the name to our favourite telescope. I have a fondness for the Hubble tel…

The main thing is to be mindful of keywords you target. Don't do as another commenter suggested and target hubble data [0] unless you apply what you make to actual Hubble data. Like AWS did with its Open Data thing that comes up for that keyword. The telescope is older than the web and is what every single person on the planet with some access to space-related media thinks of when they think of Hubble. Think long tai…

As the person you may be referring to, I'd like to clarify that I was not in any way suggesting they target "hubble data." It was just an example of how a user might modify their search if they were looking for this company but found telescope content instead.

There's no sense in doing SEO for your company name, unless you're at the point where competitors are trying to outrank you for your own company name. (Which is a pretty good tactic, actually: https://www.gkogan.co/blog/alternative-pages/.) So don't target "hubble," don't target "hubble data," don't target "hubble the YC company I saw on HN a while back," don't worry about it. Try and catch the people searching for use cases or solutions instead.

Post reply on HN