Why confluent instead of something like MSK, Redpanda or one of the new leaderless, direct to S3 Kafka implementations?
I think they do mean Kafka. Anyways theres connectors from Kafka to a bunch of things so I think its a reasonable choice.
We built a modern data stack from scratch and reduced our bill by 70%
31–40 of 60 posts
Re: We built a modern data stack from scratch and reduced our bill by 70%
#32When working with ETL, it really helps to not conflate the letters or worry about them in the wrong order. A lot of the most insane complexity comes out of moving too quickly with data.
If you don't have good staging data after running extraction (i.e., a 1:1 view of the source system data available in your database), there is nothing you can do to help with this downstream. You should stop right there and keep digging.
Extracting the data should be the most challenging aspect of an ETL pipeline. It can make a lot of sense to write custom software to handle this part. It is worth the investment because if you do the extraction really well, the transform & load stages can happen as a combined afterthought [0,1,2,3] in many situations.
This also tends to be one of the fastest ways to deal with gigantic amounts of data. If you are doing things like pulling 2 different tables and joining them in code as part of your T/L stages, you are really missing out on the power of views, CTEs, TVFs, merge statements, etc.
[0] https://learn.microsoft.com/en-us/sql/t-sql/statements/merge...
[1] https://www.postgresql.org/docs/current/sql-merge.html
[2] https://docs.oracle.com/database/121/SQLRF/statements_9017.h...
[3] https://www.ibm.com/docs/en/db2/12.1?topic=statements-merge
Re: We built a modern data stack from scratch and reduced our bill by 70%
#33So you saved just $20k per year? Not sure the context of your company but I’m not sure if this turns out to be a net win given the cost of engineering resources to produce this infra gain
If it's only for cost savings it's a hard sell. But generally rolling your own has other benefits.
> But generally rolling your own has other benefits.
Not for startups it doesn't. The only rolling-your-own they should be doing is their main product.
Once you get bigger with many hundreds of employees, and existing software starts becoming a measurable blocker, then you can gradually build your own stuff once the tradeoffs actually make sense. But it generally takes a while for that to become the case. For startups, premature optimization isn't the root of all evil, but it's the root of a lot of it.
Re: We built a modern data stack from scratch and reduced our bill by 70%
#34Earlier quoted context omitted.
If it's only for cost savings it's a hard sell. But generally rolling your own has other benefits.
> But generally rolling your own has other benefits. Not for startups it doesn't. The only rolling-your-own they should be doing is their main product. Once you get bigger with many hundreds of employees, and existing software starts becoming a measurable blocker, then you can gradually build your own stuff once the tradeoffs actually make sense. But it generally takes a while for that to become the case. For startup…
You might argue that it is never worth it to roll anything on your own (which is already an extreme proposition), but to argue that it has no benefits (other than cost?), I think it's either bad reading comprehension or overzeal to jump into the keyboard and type dogma that you should never roll your own and that you should download 1000 dependencies and a dependency manager to manage all those version conflicts.
Re: We built a modern data stack from scratch and reduced our bill by 70%
#35Earlier quoted context omitted.
If it's only for cost savings it's a hard sell. But generally rolling your own has other benefits.
Rolling your own generally has mainly downsides in the context they are in. 1. This is clearly a small team with very little spend 2. Tomorrow someone leaves and next engineer will have to manage all of this. 3. I don't think they realize that they actually increased cost of this service not decreased it. Now they need to manage their own Kafka monthly. Engineering time is expensive.
Yes, rolling your own X has downsides and it has upsides.
Welcome to tradeoff engineering.
Re: We built a modern data stack from scratch and reduced our bill by 70%
#36How much did this cost in engineering time and how much will it cost to maintain? How about when you need to add a new feature? Seems like you saved roughly 1.5k per month which pays for a couple days of engineering time (ignoring product,mgmt and costs related to maintaining the software)
No idea how many hours to build, but I maintain something similar, (different stack though) and its so trivial I don't even count the hours, so it's probably about 1/2 a day to maintain that every 3 months? Even if you needed to invent a new feature, you could invent a months worth of features every year and still save money.
Keep components up to date & dealing with schema changes will easily take more than a half day every 3 months.
Re: We built a modern data stack from scratch and reduced our bill by 70%
#37When working with ETL, it really helps to not conflate the letters or worry about them in the wrong order. A lot of the most insane complexity comes out of moving too quickly with data. If you don't have good staging data after running extraction (i.e., a 1:1 view of the source system data available in your database), there is nothing you can do to help with this downstream. You should stop right there and keep diggi…
> Extracting the data should be the most challenging aspect of an ETL pipeline.
Why should this be difficult? It’s the easiest part. You run SELECT * and you’re done.
The difficult part is transforming all the disparate upstream systems and their evolving schemas into a useful analytical model for decision support.
Re: We built a modern data stack from scratch and reduced our bill by 70%
#38Is Debezium the only good CDC tool out there? I have a fairly simple data stack and am looking at integrating a CDC solution but I really dont want to touch Kafka just for this. Are there any easier alternatives?
Redpanda Connect is a yaml file that has a cdc input and you can process the data and send it anywhere, no Kafka required.
I’ve seen postgres cdc directly being written to Snowflake.
https://docs.redpanda.com/redpanda-connect/components/inputs...
Re: We built a modern data stack from scratch and reduced our bill by 70%
#39Earlier quoted context omitted.
> But generally rolling your own has other benefits. Not for startups it doesn't. The only rolling-your-own they should be doing is their main product. Once you get bigger with many hundreds of employees, and existing software starts becoming a measurable blocker, then you can gradually build your own stuff once the tradeoffs actually make sense. But it generally takes a while for that to become the case. For startup…
You might argue that it is never worth it to roll anything on your own (which is already an extreme proposition), but to argue that it has no benefits (other than cost?), I think it's either bad reading comprehension or overzeal to jump into the keyboard and type dogma that you should never roll your own and that you should download 1000 dependencies and a dependency manager to manage all those version conflicts.
It's not being overzealous.
If you're a startup and you need something and it already exists, why would you write your own instead? Even if it's clunky, it saves you time.
And for startups, speed and runway is everything. Rolling your own is not a luxury you have.
Especially when you consider how much of the startup experience is changing requirements, pivoting, throwing out code, etc.
And I've never seen 1,000 dependencies in my life. Sure maybe you have 50. That's entirely manageable. Think how much time you'd lose writing and testing all that stuff yourself...
Re: We built a modern data stack from scratch and reduced our bill by 70%
#40How much did this cost in engineering time and how much will it cost to maintain? How about when you need to add a new feature? Seems like you saved roughly 1.5k per month which pays for a couple days of engineering time (ignoring product,mgmt and costs related to maintaining the software)
I know this is off the topic of the actual post but I'm confused as to why I've been downvoted, seems other people have since made similar comments but my comment has been downvoted. Help me out here, why?