stuff like this makes me want to experiment with going back to just one huge $100k server and running it all on one box in a server rack.
Almost every infrastructure decision I endorse or regret
221–230 of 644 posts
Re: Almost every infrastructure decision I endorse or regret
#222> The markup cost of using RDS (or any managed database) is worth it. Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a…
Some orgs are looking at moving back to on prem because they're figuring this out. For a while it was vogue to go from capex to opex costs, and C suite people were incentivized to do that via comp structures, hence "digital transformation" ie: migration to public cloud infrastructure. Now, those same orgs are realizing that renting computers actually costs more than owning them, when you're utilizing them to a signif…
Context: I build internal tools and platforms. Traffic on them varies, but some of them are quite active.
My nasty little secret is for single server databases I have zero fear of over provisioning disk iops and running it on SQLite or making a single RDBMS server in a container. I've never actually run into an issue with this. It surprises me the number of internal tools I see that depend on large RDS installations that have piddly requirements.
Re: Almost every infrastructure decision I endorse or regret
#223Earlier quoted context omitted.
On AWS, at least, there are alternatives such as ECS and even plain old EC2 auto scaling groups. Teams can have the autonomy to run their infrastructure however they like (subject to whatever corporate policy and compliance regime requirements they might have to adhere to). Kubernetes is appealing to many, but it is not 100% frictionless. There are upgrades to manage, control plane limits, leaky abstractions, differe…
The iceberg is fine, but using ECS doesn't absolve you from needing to care about monitoring, affinity, audit logging, OS upgrades, authentication/IAM, etc. That's generally why organizations choose to have infrastructure teams, or to not have infrastructure at all. I have seen people rewrite Kubernetes in CloudFormation. You can do it! But it certainly isn't problem-free.
ECS Fargate does manage the security of the node up to and including the container runtime. Patches are often applied behind the scenes, without many folks even knowing, and for those that require interruption, a restart of the task will land it on a patched node.
You’re right that if you use a cloud provider, IAM is something that has to be reckoned with. But the question is, how many implementations of IAM and policy mechanisms do I want to deal with?
Re: Almost every infrastructure decision I endorse or regret
#224So by and large I agree with the things in this article. It's interesting that the points I disagree with the author on are all SaaS products: > Moving off JIRA onto linear I don't get the hype. Linear is fine and all but I constantly find things I either can't or don't know how to do. How do I make different ticket types with different sets of fields? No clue. > Not using Terraform Cloud No Regrets I generally recom…
DatDog is a freaking beast. NY wife works in workday (a huge employee management system) and they have a very large number of tutorials, videos, "working hours" and other tools to ensure their customers are making the best use of it. Datadog on the other side... their "DD University" is a shame and we as paying customers are overwhelmed and with no real guidance. DD should assign some time for integration for new cus…
[deleted]
Re: Almost every infrastructure decision I endorse or regret
#225Earlier quoted context omitted.
The closest thing to what you're describing is Prisma in Node. It generates a Typescript file from your schema so you get code completion on your data. And it exists somewhere between a query builder and a traditional ORM. I have worked in many languages with many ORMs and this has been my personal favorite.
Until Prisma can manage JOINs [0] there is no way I can recommend it. [0]: https://github.com/prisma/prisma/discussions/12715
The support for JOINs is coming, currently under a feature flag [0]
[0]: https://github.com/prisma/prisma/issues/5184#issuecomment-18...
Re: Almost every infrastructure decision I endorse or regret
#226I was hoping there would be a section for Search Engines. It's one of those things you tend to get locked in to, and it's hard to clearly know your requirements well enough early on.
Any references to something like this with a Search slant would be greatly appreciated.
Re: Almost every infrastructure decision I endorse or regret
#227Awesome writeup! Just had a couple comments/questions.
> Not adopting an identity platform early on
The reason for not adopting an IDP early is because almost every vendor price gouges for SAML SSO integration. Would you say it's worth the cost even when you're a 3-5 person startup?
> Datadog
What would you recommend as an alternative? Cloudwatch? I love everything about Datadog, except for their pricing....
> Nginx load balancer for EKS ingress
Any reason for doing this instead of an Application Load Balancer? Or even HA Proxy?
Re: Almost every infrastructure decision I endorse or regret
#228Great post. I do wonder - what are the simplest K8s alternatives? Many say in the database world, "use Postgres", or "use sqlite." Similarly there are those databases that are robust that no one has heard of, but are very limited like FoundationDB. Or things that are specialized and generally respected like Clickhouse. What are the equivalents of above for Kubernetes?
It’s mainly running your own control plane that is complex. Managed k8s (EKS, AKS, GKE) is not difficult at all. Don’t listen to all the haters. It’s the same crowd who think they can replace systemd with self hacked init scripts written in bash, because they don’t trust abstractions and need to see everything the computer does step-by-step.
I also stayed away for a long time due to all the fear spread here, after taking the leap, I’m not looking back.
The lightweight “simpler” alternative is docker-compose. I put simpler in quotes because once you factor in all the auxiliary software needed to operate the compose files in a professional way (IaC, Ansible, monitoring, auth, VM provisioning, ...), you will accumulate the same complexity yourself, only difference is you are doing it with tools that may be more familiar to what you are used to. Kubernetes gives you a single point of control plane for all this. Does it come with a learning curve? Yes, but once you get over it there is nothing inherent about it that makes it unnecessary complex. You don’t need autoscaler, replicasets and those more advanced features just because you are on k8s.
If you want to go even simpler, the clouds have offerings to just run a container, serverless, no fuzz around. I have to warn everyone though that using ACI on Azure was the biggest mistake of my career. Conceptually it sounds like a good idea but Azures execution of it is just a joke. Updating a very small container image taking upwards of 20-30 minutes, no logs on startup crashes, randomly stops serving traffic, bad integration with storage.
Re: Almost every infrastructure decision I endorse or regret
#229Earlier quoted context omitted.
> Cost is the most common complaint and it's almost always from people who don't have it configured correctly (which to be fair Datadog makes it far too easy to misconfigure things and blow up costs). I loved Datadog 10 years ago when I joined a company that already used it where I never once had to think about pricing. It was at the top of my list when evaluating monitoring tools for my company last year, until I go…
I’m a big fan of Datadog from multiple angles. Their pricing setup is evil. Breaking out by SKUs and having 10+ SKUs is fine, trialing services with “spot” prices before committing to reserved capacity is also fine. But (for some SKUs, at least) they make it really difficult to be confident that the reserved capacity you’re purchasing will cover your spot use cases. Then, they make you contact a sales rep to lower yo…
Another annoying thing is that the billing dashboards do not map clearly to what's on the pricing pages / in the contract. Good luck figuring out the extras for RUM when you have multiple orgs.
Then they have things that I wanted to try for a long time, but... support doesn't care? Repeated "would you like to use this? / very likely, can we try it out? / (silence)". I love their product, but they are so annoying to deal with at the billing level.
Re: Almost every infrastructure decision I endorse or regret
#230Great post. I do wonder - what are the simplest K8s alternatives? Many say in the database world, "use Postgres", or "use sqlite." Similarly there are those databases that are robust that no one has heard of, but are very limited like FoundationDB. Or things that are specialized and generally respected like Clickhouse. What are the equivalents of above for Kubernetes?
Kubernetes aren't like that. It's just that, you should start with a handful of backed-up pet servers. Then manually automate their deployment when you need it. And only then go for a tool that abstracts the automated deployment when you need it. But I fear the simplest option on the Kubernetes area is Kubernetes.
This is good advice, if you haven't experienced the pain of doing it yourself, you won't know what the framework does for you. There are limits to this reasoning of course, we don't reimplement everything on the stack just for the learning experience. But starting with just docker might be a good idea.