Sounds like a whole lot of stuff for a startup. Maybe start with a simple stack until there's market fit. Even Amazon didn't start this way.
Almost every infrastructure decision I endorse or regret
431–440 of 644 posts
Re: Almost every infrastructure decision I endorse or regret
#432Earlier quoted context omitted.
Discount rates are actually much better too on the bigger instances. Therefore the "sticker price" that people compare on the public site is no where close to a fair comparison. We technically aren't supposed to talk about pricing publically, but I'm just going to say that we run a few 8XL and 12Xl RDS instances and we pay ~40% off the sticker price. If you switch to Aurora engine the pricing is absurdly complex (its…
> but I'm just going to say that we run a few 8XL and 12Xl RDS instances and we pay ~40% off the sticker price. Genuinely curious, how do you that? We pay a couple of million dollars per year and the biggest spend is RDS. The bulk of those are 8xl and 12xl as you mention and we have a lot of these. We do have savings plans, but those are nowhere near 40%.
Yeah 40% seems like a pipedream. I was at a Fortune 500 defense firm and we couldn't get any cloud provider to even offer us anything close to that discount if we agreed to move to them for 3-4 years minimum. That org ended up not migrating because it was significantly cheaper to buy land and build datacenters from scratch than to rent in the cloud.
Re: Almost every infrastructure decision I endorse or regret
#433I’m imagining a developer in the 90s/00s reading this list and being baffled by the complexity/terminology
We had FB up to 6 figures in servers and a billion MAUs (conservatively) before even tinkering with containers. The “control plane” was ZooKeeper. Everything had bindings to it, Thrift/Protobuf goes in a znode fine. List of servers for FooService? znode. The packaging system was a little more complicated than a tarball, but it was spiritually a tarball. Static link everything. Dependency hell: gone. Docker: redundant…
> We had FB up to 6 figures in servers and a billion MAUs (conservatively) before even tinkering with containers.
that would have been around the time when containers entered the public/developer consciousness, no?
Re: Almost every infrastructure decision I endorse or regret
#434Earlier quoted context omitted.
That’s just another way of saying the opportunity cost isn’t worth paying to do the migration. Optionality and flexibility are extremely valuable, and that is why cloud compute continues to be popular, especially for rapidly/burstily growing businesses like startups.
I don't mean to pick on your specific comments, but I find these analysis almost always lack a crucial perspective: level of knowledge. This is the single biggest factor, and it's the hardest one to be honest about. No one wants to say "RDS is a good choice . . . because I don't know how nor have I ever self managed a database." If you want a different opportunity cost, get people with different experience. If RDS is…
I disagree here. This falls apart when you zoom out one step. I'm perfectly capable of managing a database. I'm also capable of maintaining load balancers, redis, container orchestrators, Jenkins, perforce, grafana, Loki, Oncall, individually. But each of those has the high chance of being a distraction from what our software actually does.
Its about tradeoffs, and some tradeoffs are often more applicable than others - getting a ping at 7am on a Sunday because your ec2 instance filled it's drive up with logs and your log rotation script failed because it didn't have a long enough retey is a problem I'm happy to outsource when I should be focusing on the actual app.
Re: Almost every infrastructure decision I endorse or regret
#435Earlier quoted context omitted.
Context: I build internal tools and platforms. Traffic on them varies, but some of them are quite active. My nasty little secret is for single server databases I have zero fear of over provisioning disk iops and running it on SQLite or making a single RDBMS server in a container. I've never actually run into an issue with this. It surprises me the number of internal tools I see that depend on large RDS installations…
>making a single RDBMS server in a container On what disk is the actual data written? How do you do backups, if you do?
In most setups like this, it’s going to be spinning rust with mdadm, and MySQL dumps that get created via cron and sent to another location.
Re: Almost every infrastructure decision I endorse or regret
#436Earlier quoted context omitted.
Reading further > Endorse-ish: Schema migration by Diff Well that explains it... What a terrible approach to migrations for data integrity.
Genuinely curious (I don't have much experiences with DBs), how is schema migration done 'properly' these days?
Incremental forward-only migrations (non-state based).
Then, for the How and When, it mostly depends of your constraints and sizes. There's no silver bullet, it's hard, it require constant thinking, it's a slow and often multi step process.
I never saw a successful fully automated one-way-of-doing process.
Re: Almost every infrastructure decision I endorse or regret
#437Earlier quoted context omitted.
>By the time your db is big enough to cost an unreasonable amount on RDS, you’re likely a big enough team to have options. By the time your db is big enough to cost an unreasonable amount on RDS, you've likely got so much momentum that getting off is nearly impossible as you bleed cash. You can buy a used server and find colocation space and still be pennies on the dollar for even the smallest database. If you're doi…
In the small SaaS startup case, I’d say the production database is typically the most critical single piece of infra, so self hosting is just not a compelling proposition unless you have a strong technical reason where having super powerful database hardware is important, or a team with multiple people who have sysadmin or DBA experience. I think both of those cases are unusual. I’ve been the guy managing a critical…
I agree, but I also classify some of these as "learn them once and you're all set".
Maybe it takes you a month the first time around and a week the 10th time around. First product suffers, the other products not so much. Now it just takes a week of your time and does not require you to pay large AWS fees, which means you are not bleeding money
I like to set up scrappy products that do not rack up large monthly fees. This means I can let them run unprofitable for longer and I don't have to seek an investor early, which would light up a large fire under everyone's butts and start influencing timelines because now they have the money and want a return asap.
I'll launch a week later - no biggie usually. I could have come up with the idea a month later, so I'm still 3 weeks early ;)
It doesn't work for all projects, obviously, but I've seen plenty of SaaS start out with a shopping spree, then pay monthly fees and purchase licenses for stuff that they could have set up for free if they put some (usually not a lot) effort into it. When times get rough, the shorter runway becomes a hard fact of life. Maybe they wouldn't have needed a VC and could have bootstrapped and also survived for longer.
Re: Almost every infrastructure decision I endorse or regret
#438Reading this I couldn’t help but think: yeah all of these points make sense in isolation, but if you look at the big picture, this is an absurd level of complexity. Why do we need entire teams making 1000s of micro decisions to deploy our app? I’m hungry for a simpler way, and I doubt I’m alone in this.
It smells like ZIRP is not over yet. VCs are still burning money in the AWS fire pit.
Re: Almost every infrastructure decision I endorse or regret
#439Earlier quoted context omitted.
For big companies such as banks this cost comparison is not as straight forward. They have whole data centres just sitting there for disaster recovery. They periodically do switchovers to test DR. All of this expense goes away when they migrate to cloud.
> All of this expense goes away when they migrate to cloud. Just to pay someone else enough money to provide the same service and make a profit while do it
That's how nearly every aspect of every business works; would you you start a bakery by learning construction and building it yourself?
Re: Almost every infrastructure decision I endorse or regret
#440Disagree on the point and reasoning about the single database. Sounds like they experienced badly managed and badly constrained database. The described fks and relations: that's what the key constraints and other guard rails and cascades are for - so that you are able to manage a schema. That's exactly how you do it: add in new tables that reference old data. I think the regret is actually not managing the database,…
Reading further > Endorse-ish: Schema migration by Diff Well that explains it... What a terrible approach to migrations for data integrity.
Can you explain?
Having a tool to detect changes and create a migration doesn’t sound bad?
In a nutshell thats how django migrations work as well, which works really well.