Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We built a modern data stack from scratch and reduced our bill by 70%

jchandra.com

Post reply on HN
Print view View original

21–30 of 60 posts

Re: We built a modern data stack from scratch and reduced our bill by 70%

#21

Post by moltar »

snake_doc wrote: post #7

These just seems like over engineered solutions trying to guarantee their job security. When the dataflows are so straight forward, just replicate into pick your OLAP, and transform there.

I came from traditional engineering into data engineering by accident and had a similar view. But every time I tried to make a pipeline from first principles it always eventually turned out something like this for a reason. This is especially true when trying to bridge many teams and skillsets - everyone wants their favourite tool.

Re: We built a modern data stack from scratch and reduced our bill by 70%

#22

Post by nxm »

moltar wrote: post #19
tacker2000 wrote: post #18

Is Debezium the only good CDC tool out there? I have a fairly simple data stack and am looking at integrating a CDC solution but I really dont want to touch Kafka just for this. Are there any easier alternatives?

AWS has DMS with a serverless option which is great for CDC

Except when a new column is added upstream and DMS inserts the new column not at the end but before the dms timestamp fields, ruining the index based loads of DMS files

Re: We built a modern data stack from scratch and reduced our bill by 70%

#23

Post by vood »

TZubiri wrote: post #16
ripped_britches wrote: post #10

So you saved just $20k per year? Not sure the context of your company but I’m not sure if this turns out to be a net win given the cost of engineering resources to produce this infra gain

If it's only for cost savings it's a hard sell. But generally rolling your own has other benefits.

Rolling your own generally has mainly downsides in the context they are in. 1. This is clearly a small team with very little spend 2. Tomorrow someone leaves and next engineer will have to manage all of this. 3. I don't think they realize that they actually increased cost of this service not decreased it. Now they need to manage their own Kafka monthly. Engineering time is expensive.

Re: We built a modern data stack from scratch and reduced our bill by 70%

#24

Post by grayhatter »

reillyse wrote: post #20

How much did this cost in engineering time and how much will it cost to maintain? How about when you need to add a new feature? Seems like you saved roughly 1.5k per month which pays for a couple days of engineering time (ignoring product,mgmt and costs related to maintaining the software)

No idea how many hours to build, but I maintain something similar, (different stack though) and its so trivial I don't even count the hours, so it's probably about 1/2 a day to maintain that every 3 months?

Even if you needed to invent a new feature, you could invent a months worth of features every year and still save money.

Re: We built a modern data stack from scratch and reduced our bill by 70%

#25

Post by polskibus »

snake_doc wrote: post #7

These just seems like over engineered solutions trying to guarantee their job security. When the dataflows are so straight forward, just replicate into pick your OLAP, and transform there.

What is the current state of the art (open source) when doing oltp to olap pipelines in these days? I don’t mean a one-off etl style load at night but a continuous process with relatively low latency?

Re: We built a modern data stack from scratch and reduced our bill by 70%

#26

Post by mattbillenstein »

SkyPuncher wrote: post #11

I know it's easy to be critical, but I'm having trouble seeing the ROI on this. This is a $20k/year savings. Perhaps, I'm not aware of the pricing in the Indian market (where this startup is), but that simply doesn't seem like a good use of time. There's an actual cost of doing these implementations. Both in hard financial dollars (salaries of the people doing the work) and the trade-offs of de prioritizing other oth…

Yeah, but you can always make this argument and build nothing - dealing with all the problems of every 3rd party SaaS / PaaS under the sun. Sometimes it's much easier to just build the thing and then you know where its' limitations are and you can address them over time.

Re: We built a modern data stack from scratch and reduced our bill by 70%

#27

Post by moandcompany »

> "We are a fintech startup helping SMEs raise capital from our platform where we provide diverse financial products ranging from Term Loan, Revenue Based Financing to Syndication, we face one unique data challenge: Our data comes from everywhere."

Reading carefully: the result of this work yields an expected $21,000 USD in annual operating cost savings for infrastructure services.

Is this resume driven development?

What was the opportunity cost of this work? Is the resulting system more or less maintainable by future employees/teammates?

Re: We built a modern data stack from scratch and reduced our bill by 70%

#28

Post by nchmy »

tacker2000 wrote: post #18

Is Debezium the only good CDC tool out there? I have a fairly simple data stack and am looking at integrating a CDC solution but I really dont want to touch Kafka just for this. Are there any easier alternatives?

Conduit.io is where it's at. FAR more source and destination connectors, easier to deploy etc... Pair it with NATS to replace Kafka's mess

Re: We built a modern data stack from scratch and reduced our bill by 70%

#29

Post by thecleaner »

This is a little bit of a word soup. Its hard to see why the various redesigns were done without a set of requirements. I dont get why you'd trigger Airflow workflows for doing CDC. These things were designed for large scale batch jobs rather than doing CDC on a some Google sheets. Either way without scale numbers its hard to why PG was used or why the shift to BigQuery. Anyways the site uses Hugo, which actually sticks out for me.

Re: We built a modern data stack from scratch and reduced our bill by 70%

#30

Post by williamdclt »

polskibus wrote: post #25
snake_doc wrote: post #7

These just seems like over engineered solutions trying to guarantee their job security. When the dataflows are so straight forward, just replicate into pick your OLAP, and transform there.

What is the current state of the art (open source) when doing oltp to olap pipelines in these days? I don’t mean a one-off etl style load at night but a continuous process with relatively low latency?

Idk what the state of the art is, but I’ve used change data capture with Debezium and Kafka, sink’d into Snowflake. Not sure Kafka is the right tool as you don’t need persistence, and having replication slots makes a lot of operations (eg DB engine upgrade) a lot harder though.
Post reply on HN