11–20 of 44 posts
Have you considered picking a different name? Searching for "Hubble" for whatever reason is going to return millions of irrelevant results for your customers.
Also any plans to profile and test file-based stores as well? There's a lot that can go wrong in a pipeline before data even reaches BigQuery or Snowflake, and you may help your customers save money if you could profile data in S3 before it goes through a potentially expensive transform process.
Best of luck, though! Data testing is a very real need in most data organizations I've been in, and I'm glad more and more tools seem to be popping up recently to help with it.
this is interesting! running tests on data is certainly a pain point for me, and there doesn't seem to be nearly the kind of infrastructure available as for, say, tests for code functionality. Is this open source? Sending my data to a third party is a no-go, as is having a third-party connect to the database. Something part of a managed hosting service, though, or an add-on to an existing trusted hosted service that…
It's not open source but we can deploy on-prem (or cloud-prem more accurately) pretty easily. We’re also going to setup as an add-on available through AWS marketplace. Feel free to shoot me an email if you want to see if this can work for you hamzah[at]gethubble.io
Have you considered picking a different name? Searching for "Hubble" for whatever reason is going to return millions of irrelevant results for your customers.
I can't think of a worse name for SEO purposes. You'd have to fight through a well loved and well known space telescope, the astronomer it was named after, and Hubble contact lenses, which has raised ~74MM.
> See irrelevant results
> Search for "hubble data"
Problem solved. People are smart enough to modify their search if the initial results are about telescopes and not data pipelines.
One of my clients had a similar name to a global pizza chain. It hasn't been an issue at all, besides having to hear the same pizza puns over and over.
How does hubble compare to Great Expectations or DBT for pipeline testing? It looks like more emphasis on automated profiling than "having to write and maintain lots of individual tests" and obviously hubble being a saas offering is the big difference? Also any plans to profile and test file-based stores as well? There's a lot that can go wrong in a pipeline before data even reaches BigQuery or Snowflake, and you may…
We’ve also found that keeping a history of the state of the warehouse over time is really useful context for determining whether a test has failed (example: this table tends to update every 30-40 minutes so we’ll set a threshold at an hour).
We also handle the scheduling, which is surprisingly annoying to manage (we built a couple of internal tools for this in the past). That’s something we really missed with great expectations (you get this with DBT cloud). Testing files is an interesting use case, to an extent we support this using Athena or Bigquery external tables for json/csv/parquet. We’re intentionally limiting it to SQL for now.
Have you considered picking a different name? Searching for "Hubble" for whatever reason is going to return millions of irrelevant results for your customers.
I can't think of a worse name for SEO purposes. You'd have to fight through a well loved and well known space telescope, the astronomer it was named after, and Hubble contact lenses, which has raised ~74MM.
In the grand scheme of problems a new company has, this is so trivially minor that I can't fathom this having any tangible effect on the success of a company. It's one thing if there's another data warehousing company called "hubble", but that's not the case you're making.
Earlier quoted context omitted.
I can't think of a worse name for SEO purposes. You'd have to fight through a well loved and well known space telescope, the astronomer it was named after, and Hubble contact lenses, which has raised ~74MM.
If a customer is looking for you specifically, they will find you (e.g. "hubble data" as stated above). If they are looking for a "data quality monitor" then the SEO will need to reflect that. The name is largely irrelevant at that point, it's merely a moniker. In the grand scheme of problems a new company has, this is so trivially minor that I can't fathom this having any tangible effect on the success of a company.…
Co-founder of intermix.io here (which we sold in March). We came more from the performance monitoring angle (specifically for Redshift), but then shifted to a product that works horizontally across all warehouses, to track usage, workflows and user engagement. "Shift to Data Products" was the narrative we started using in Q4 2019. If you read the copy on the current intermix.io website, I think you'll find yourself n…
So I guess if you had to pick arbitrary revenue/data/fte cutoffs, do you see the org chart of these adopters as you’ve described looking a certain way? Let me try to rephrase that.
Do you think there’s a step function of “here you need one DBA who is a holy librarian” and “here we need a gitlab styled data team with SLAs and the data equivalent of HR business partners who get assigned to the BU”?
Tangential to your comment but curious if you believe the human side scales akin to the infrastructure side.
Running a full table scan on BigQuery every hour can get quite expensive. Do you support some sort of deltas? I signed up. Unlike the video, I do not see Redshift as an option. Any idea when Redshift will be supported? How does billing per user make sense here? What prevents me monitoring thousands of tables under single user? Your workload costs will be higher than $200 here, no? Do you have a set of fixed IPs you'r…
Full table scans can get expensive. We’re adding support for incremental tests so for append-only tables you’ll only test the recent rows. This is especially useful if you use partitioned tables in bigquery. Actually in the first version of the product we automatically tested every column in every table. The tests are more selective now, which is partially due to cost and partially because nobody wants to navigate th…
BTW, you don't need to navigate 10K tests... you only need to navigate the failing ones.