161–170 of 181 posts
Earlier quoted context omitted.
We’ve been battling Redshift scaling problems so it’s fluctuated a lot over the last 6 months (both up and down). $15k per month to an order of magnitude (not counting Looker licensing).
That seems like a lot. Snowflake typically cuts Redshift bills in half if you are hitting limits with RS
Earlier quoted context omitted.
> and attach model data Can you elaborate on this? Do you mean enriching events and/or windowing/aggregating/summarizing? Are you using KSQL?
Yeah, enriching the events as they pass through - so we have a Kafka Connect JDBC source polling the source of truth DB every X seconds for model changes (we looked into Debezium for streaming model changes, but the additional complexity didn't bring us much benefit based on how often our model changes), and then we pull that into a GlobalKTable and join the KStream as appropriate. We also use Kafka streaming with a…
At Grove, we are: Airflow -> S3 -> DBT with Spark/EMR or Redshift/Spectrum -> Redshift data marts -> Looker At least, that’s the way we like our pipelines to work. In practice we have a couple of extractions that land directly in Redshift (we extract Zendesk data, for instance, with Stitch Data). We use Snowplow for click stream analytics. And we’ll likely move from Redshift to Snowflake (or mayybbbeee Bigtable) in Q…
Earlier quoted context omitted.
Yeah, enriching the events as they pass through - so we have a Kafka Connect JDBC source polling the source of truth DB every X seconds for model changes (we looked into Debezium for streaming model changes, but the additional complexity didn't bring us much benefit based on how often our model changes), and then we pull that into a GlobalKTable and join the KStream as appropriate. We also use Kafka streaming with a…
Mind if I ask you what you used to capture source changes if Debezium didn't work out for you? Is there a specific Kafka Connector for JDBC that you're using?
If we had a need for near instantaenous model updates, then I'd definitely go Debezium. We didn't use it because we didn't need it - a generic KC JDBC source with a poll period of 5 seconds met our needs.s
Earlier quoted context omitted.
I don't want to derail the thread, but I recently started a company ( https://syndetic.co ) that's working on this problem. We've been focusing on the external data dictionary use case (how does a data-as-a-service company explain a dataset to their prospective customers) but we've been encountering a number of companies that are evaluating data catalogs and other internal tools for their data science teams. I would…
Was just thinking this would make a great business opportunity. Thanks, we'll check it out!
The legacy players are a mess, the new cloud native offerings are overly Engineering focused and immature, and the few startups in this space are recreating the legacy solutions in a SAAS formats
Go forth and claim the significant checks that I and others plan to write for these mediocre offerings.
Earlier quoted context omitted.
Can you please compare it with Superset?
Worlds apart, superset provides powerful visualisation capabilities, but is very difficult to use/setup/maintain (although they just launched as a company [0], so perhaps that will change). Metabase is a very easy to use data exploration, data vis and reporting tool designed to give insights into data in an easy/self service way for mostly non-technical users. I've POC'd both, and would recommend Metabase or redash u…
As a data scientist (startup / Yahoo) turned product manager (4 startups), I've used a variety of stacks in the past - from plain SQL, to Mode, to Mixpanel, Looker, Interana, and Hive. Recently we started using PopSQL ( https://popsql.com ) and love it. When I don't have a dedicated data team, my philosophy is: 1) Make it difficult to get wrong answers - Don't use Google Analytics. It's too easy to generate incorrect…
Earlier quoted context omitted.
Worlds apart, superset provides powerful visualisation capabilities, but is very difficult to use/setup/maintain (although they just launched as a company [0], so perhaps that will change). Metabase is a very easy to use data exploration, data vis and reporting tool designed to give insights into data in an easy/self service way for mostly non-technical users. I've POC'd both, and would recommend Metabase or redash u…
Thank you! Any idea how does it compare with redash then?
I didn't demo redash so can't comment on the actual use comparison.
Postgres -> Metabase I believe this is the best combination of cheap/powerful for early-stage startups. My very non-technical cofounder is able to use metabase's simple GUI interface to create graphs/insights (even joining and aggregating across tables!), and for anything complex I can step in a give a helper SQL query. We have around 10M records we aggregate around for daily insights. Metabase can also be run as a p…
UX guy from Metabase here. Glad to hear it's working well for you. Happy to answer any questions folks might have.