51–60 of 66 posts
I should have known it was Coinbase. I know that Coinbase used to spend $35,000 a month to back up the data directory of ETH nodes.
They paid backups to whom? Who was vendor....I'm interested.
What problems does Datadog solve that you can't solve with cheaper solutions?
I believe a much more useful question to ask is just “is this the highest and best use of my finite attention and time?” It is much easier to find $100,000 a year of budget than it is to find an additional $50,000 worth of skilled[1] developer time.
[1] This skilled part is critical because if you have some flunky create your “SaaS alternative” you are in for an even worse time.
I should have known it was Coinbase. I know that Coinbase used to spend $35,000 a month to back up the data directory of ETH nodes.
Earlier quoted context omitted.
the visible cost of burning runway on a bill is very often far less than the invisible cost of burning engineer time rebuilding undifferentiated heavy lifting rather than working on product/customer needs
This is very well stated.
I should have known it was Coinbase. I know that Coinbase used to spend $35,000 a month to back up the data directory of ETH nodes.
>I know that Coinbase used to spend $35,000 a month to back up the data directory of ETH nodes. They paid backups to whom? Who was vendor....I'm interested.
Earlier quoted context omitted.
>I know that Coinbase used to spend $35,000 a month to back up the data directory of ETH nodes. They paid backups to whom? Who was vendor....I'm interested.
Amazon. They put it on an EBS volume and kept it around.
Earlier quoted context omitted.
>I wonder how much that no-expense-spared, money-is-no-object attitude to buying SaaS impacts an engineers ability to make sensible decisions around infra and architecture I saw this a lot at a previous company. Being able to just "have more Lambdas scale up to handle it" got some very mediocre engineers past challenges they encountered. But it did so at the cost of wasting VAST amounts of money and saddling themselv…
It is also not a very hard skill. You do a back of the envelope calculation and if your proposed architecture is crazy expensive for your reasonable load, then you have to figure out if you are a special snowflake or just doing it wrong.
Earlier quoted context omitted.
>I wonder how much that no-expense-spared, money-is-no-object attitude to buying SaaS impacts an engineers ability to make sensible decisions around infra and architecture I saw this a lot at a previous company. Being able to just "have more Lambdas scale up to handle it" got some very mediocre engineers past challenges they encountered. But it did so at the cost of wasting VAST amounts of money and saddling themselv…
What's also frustrating is that a lot of times, costs are hidden from engineering. I don't know if I would call them mediocre, but without a feedback loop its hard to get engineers to agree whether it's worth time reviewing the code to make it faster compared to just making the db one size larger.
Yeah one of my big pet peeves was when engineering teams build platforms to run things on that obscure the cost. There have been times where they said "hey we made this big platform for analytics, just ship your stuff as configuration changes and it's deployed!" Then when I did it with very simple small cases, some unoptimized stuff on their end (a lot of what I talked about before) resulted in runaway costs that they, of course, tagged to my team.
Ultimately, you can only control what's in your scope and anything else you will need to hope that management can take that runaway cost feedback and make the correct team optimize it away.
>I don't know if I would call them mediocre, but without a feedback loop its hard to get engineers to agree whether it's worth time reviewing the code to make it faster compared to just making the db one size larger.
This started in the mid 2010s, by which point they should understand that you don't put terabytes of data into S3 in 100kb files. And if not, they should be willing to take some very simple steps to address it (literally just bundling them all in 100mb files with an index file containing the byte offsets of the individual ones would have solved a lot of their problems). There was a feedback loop. There just happened to be big egos more interested in their next fun project of reinventing another solution to another solved problem. I learned there that engineering driven companies sometimes wind up in situations in which the staff engineers love fun new database and infrastructure projects like that more than they enjoy improving their existing product.