Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost every infrastructure decision I endorse or regret

cep.dev

Post reply on HN
Print view View original

551–560 of 644 posts

Re: Almost every infrastructure decision I endorse or regret

#551

Post by raffraffraff »

electroly wrote: post #6

> The markup cost of using RDS (or any managed database) is worth it. Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a…

I agree that RDS is stupidly expensive and not worth it provided that the company actually hires at least 2x full-time database owners who monitor, configure, scale and back up databases. Most startups will just save the money and let developers "own" their own databases or "be responsible for" uptime and backups.

Re: Almost every infrastructure decision I endorse or regret

#552

Post by icedchai »

tpetry wrote: post #499
thelastparadise wrote: post #418

Earlier quoted context omitted.

> $600k/year operational against sub-$100k capital cost pays DBAs, backups, etc with money to spare. One of these is not like the others (DBAs are not capex.) Have you ever considered that if a company can get the same result for the same price ($100K opex for RDS vs same for human DBA), it actually makes much more sense to go the route that takes the human out of the loop? The human shows up hungover, goes crazy, gr…

And when you have performance issues you still need a DBA. Because RDS only runs your database. It is up to you to make it fast.

You'll need an engineer with database skills, not a dedicated DBA. I haven't seen a small company with a full time DBA in well over a decade. If you can learn a programming language, you can learn about indexes and basic tuning parameters (buffer pool, cache, etc.)

Re: Almost every infrastructure decision I endorse or regret

#555

Post by icedchai »

ffsm8 wrote: post #313
davedx wrote: post #303

Earlier quoted context omitted.

All that infra doesn’t integrate itself. Everywhere I’ve worked that had this kind of stack employed at least one if not a team of DevOps people to maintain it all, full time, the year round. Automating a database backup and testing it works takes half a day unless you’re doing something weird

> Automating a database backup and testing it works takes half a day unless you’re doing something weird True story bro I'm sure that's possible if you're storing the backup on the same server you're restoring on and everything is on top of the line nvme storage. Otherwise your backup just started to run and will need another few days to finish. And that's only if you're running single master. You're massively undere…

I worked at a place with its own colo where they ran several multi TB MySQL database servers. We did weekly backups and it could take days. Our backups were stored on external USB disks. The I/O performance was abysmal. Taking a filesystem snapshot and copying it to USB could take days. The disks would occasionally lock up and someone would have to power cycle them. Total clown show.

I would rather pay for RDS. Databases are the one thing you don't want to screw up.

Re: Almost every infrastructure decision I endorse or regret

#556

Post by rr808 »

raffraffraff wrote: post #551
electroly wrote: post #6

> The markup cost of using RDS (or any managed database) is worth it. Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a…

I agree that RDS is stupidly expensive and not worth it provided that the company actually hires at least 2x full-time database owners who monitor, configure, scale and back up databases. Most startups will just save the money and let developers "own" their own databases or "be responsible for" uptime and backups.

For a couple hundred grand you can get a team of 20 fully trained people working full time in most parts of the world.

Re: Almost every infrastructure decision I endorse or regret

#557

Post by mardifoufs »

YoshiRulz wrote: post #384
mardifoufs wrote: post #175

Earlier quoted context omitted.

Why gitlab? GitHub actions are a mess but gitlab online's ci cd is not much better at all, and for self hosted it opens a whole different can of worms. At least with GitHub actions you have a plugin ecosystem that makes the super janky underlying platform a bit more bearable.

I've found GitLab CI's "DAG of jobs" model has made maintenance and, crucially for us, optimisation relatively easy. Then I look into GitHub Actions and... where are the abstraction tools? How do I cache just part of my "workflow"? Plugins be damned. GitLab CI is so good that I'm willing to overlook vendor lock-in and YAML, and use it for our GitHub project even without proper integration. (Frankly the rest of GitLab…

Mhmm that's actually a good point!! I didn't realize that I couldn't do that with GitHub, I never really used partial caching. I just had a lot (a looot) of issues with our kubernetes runner (which I even made sure to be as close to the vanilla docs example as possible). I guess the grass is always greener on the other side :)

Re: Almost every infrastructure decision I endorse or regret

#558

Post by kevinslin »

CSMastermind wrote: post #10

So by and large I agree with the things in this article. It's interesting that the points I disagree with the author on are all SaaS products: > Moving off JIRA onto linear I don't get the hype. Linear is fine and all but I constantly find things I either can't or don't know how to do. How do I make different ticket types with different sets of fields? No clue. > Not using Terraform Cloud No Regrets I generally recom…

In terms of Datadog - the per host pricing on infrastructure in a k8/microservices world is perhaps the most egregious of pricing models across all datadog services. Triply true if you use spot instances for short lived workloads.

For folks running k8 at any sort of scale, I generally recommend aggregating metrics BEFORE sending them to datadog, either on a per deployment or per cluster level. Individual host metrics tend to also matter less once you have a large fleet.

You can use opensource tools like veneur (https://github.com/stripe/veneur) to do this. And if you don't want to set this up yourself, third party services like Nimbus (https://nimbus.dev/) can do this for you automatically (note that this is currently a preview feature). Disclaimer also that I'm the founder of Nimbus (we help companies cut datadog costs by over 60%) and have a dog in this fight.

Re: Almost every infrastructure decision I endorse or regret

#559

Post by kevinslin »

thesurlydev wrote: post #511

I've seen a lot of comments about how bad DataDog is because of cost but surprisingly I haven't seen open-source alternatives like OpenTelemetry/Prometheus/Grafana/Tempa mentioned. Is it because most people are willing to pay someone else to manage monitoring infrastructure or other reasons?

the way I think of datadog is that datadog it provides a second to none DX combined with a wide suite of product offerings that is good enough for most companies most of the time. does it have opaque pricing that can be 100x more expensive than alternatives? absolutely! will people continue to use it? yes!

something to keep in mind is that most companies are not like the folks in this thread. they might not have the expertise, time or bandwidth to build invest in observability.

the vast majority of companies just want something that basically works and doesn’t take a lot of training to use. I think of Datadog as the Apple of observability vendors - it doesn’t offer everything and there are real limitations (and price tags) for more precise use cases but in the general case, it just works (especially if you stay within its ecosystem)

Re: Almost every infrastructure decision I endorse or regret

#560

Post by kevinslin »

brycelarkin wrote: post #227

Awesome writeup! Just had a couple comments/questions. > Not adopting an identity platform early on The reason for not adopting an IDP early is because almost every vendor price gouges for SAML SSO integration. Would you say it's worth the cost even when you're a 3-5 person startup? > Datadog What would you recommend as an alternative? Cloudwatch? I love everything about Datadog, except for their pricing.... > Nginx…

For datadog, unfortunately there's no obvious altnernative despite many companies trying to take marketshare. This is to say, datadog both has second to none DX and a wide breadth of services.

Grafana Labs comes closest in terms of breadth but their DX is abysmal (I say this as a heavy grafana/prometheus user) Same comments about new relic though they have better dx than grafana. Chronosphere has some nice DX around prometheus based metrics but lack the full product suite. I could go on but essentially, all vendors either lack breadth, DX, or both.

Post reply on HN