Almost every infrastructure decision I endorse or regret • Hacker News Board

Re: Almost every infrastructure decision I endorse or regret

#471

Post by cedws » Sat, Feb 10, 2024, 2:43 PM UTC

mardifoufs wrote: post #172

cedws wrote: post #91

Earlier quoted context omitted.

K8S is a disastrous complexity bomb. You need millions upon millions of lines of code just to build a usable platform. Securing Kubernetes is a nightmare. And lock-in never really went away because it's all coupled with cloud specific stuff anyway. Many of the core concepts of Kubernetes should be taken to build a new alternative without all the footguns. Security should be baked in, not an afterthought when you need…

Millions upon millions of lines of code?! What? Can you specify what you were trying to do with it?

Argo CD, Argo Rollouts, Vault, External Secrets, Cert Manager, Envoy, Velero, plus countless operators, plus a service mesh if you need it, the list goes on. If you're providing Kubernetes as a platform at any sort of scale you're going to need most of this stuff or some alternatives. This sums up to at least multiple million LOC. Then you have Kubernetes itself, containerd, etcd...

Re: Almost every infrastructure decision I endorse or regret

#472

Post by sgarland » Sat, Feb 10, 2024, 2:44 PM UTC

rswail wrote: post #251

sgarland wrote: post #29

Earlier quoted context omitted.

> Most large SaaS companies can run off of $5k / m or cheaper RDS Hard disagree. An r6i.12xl Multi-AZ with 7500 IOPS / 500 GiB io1 books at $10K/month on its own. Add a read replica, even Single-AZ at a smaller size, and you’re half that again. And this is without the infra required to run a load balancer / connection pooler. I don’t know what your definition of “large” is, but the described would be adequate at best…

If you're not using GP3 storage that provides 12K minimum IOPS without requiring provisioned IOPS for >400GB storage, as well as 4 volume striping, then you're overpaying. If you don't have a reserved instance, then you're giving up potentially a 50% discount on on-demand pricing. An r6i.12xl is a huge instance. There are other equivalents in the range of instances available (and you can change them as required, with…

> GP3... as well as 4 volume striping

For MySQL and Postgres, RDS stripes across four volumes once you hit 400 GiB. Doesn't matter the type.

The latency variation on gp3 is abysmal [0], and the average [1] isn't great either. It's probably fine if you have low demands, or if your working set fits into memory and you can risk the performance hit when you get an uncached query.

12K IOPS sounds nice until you add latency into it. If you have 2 msec latency, then (ignoring various other overheads, and kernel or EBS command merging) the maximum a single thread can accomplish in one second is (1000 msec / 1 sec / 2 msec) = 500 I/O. Depending on your needs that may be fine, of course.

> If you don't have a reserved instance, then you're giving up potentially a 50% discount on on-demand pricing.

True, of course. Large customers also don't pay retail.

> An r6i.12xl is a huge instance.

I mean, it goes well past that to .32xl, so I wouldn't say it's huge. I work with DBs with 1 TiB of RAM, and I'm positive there are people here who think those are toys. The original comment I replied to said, "large SaaS," and a .12xl, as I said, would be roughly adequate for ~100K QPS, assuming no absurdly bad queries.

[0]: https://www.percona.com/blog/performance-of-various-ebs-stor...

[1]: https://silashansen.medium.com/looking-into-the-new-ebs-gp3-...

Re: Almost every infrastructure decision I endorse or regret

#473

Post by xmcqdpt2 » Sat, Feb 10, 2024, 2:46 PM UTC

fcarraldo wrote: post #461

interroboink wrote: post #340

Earlier quoted context omitted.

My (peripheral) experience is that it is much cheaper to get data in than to get data out. When you have the amount of data being discussed — "Enterprise data. Going back years." — that can get very costly. It's the amount of data where it makes more sense to put hard drives on a truck and drive across the country rather than send it over a network, where this becomes an issue (actually, probably a bit before then).

AWS actually has a service for this - Snowmobile, a storage datacenter inside of a shipping container, which is driven to you on a semi truck. https://aws.amazon.com/snowmobile/

They do not!

> Q: Can I export data from AWS with Snowmobile? > > Snowmobile does not support data export. It is designed to let you quickly, easily, and more securely migrate exabytes of data to AWS. When you need to export data from AWS, you can use AWS Snowball Edge to quickly export up to 100TB per appliance and run multiple export jobs in parallel as necessary. Visit the Snowball Edge FAQs to learn more.

https://aws.amazon.com/snowmobile/faqs/?nc2=h_mo-lang

Why would they make it convenient to leave?

Re: Almost every infrastructure decision I endorse or regret

#474

Post by mijoharas » Sat, Feb 10, 2024, 2:50 PM UTC

hhsectech wrote: post #319

smitty1e wrote: post #307

Earlier quoted context omitted.

My funny story is built on the idea that AWS is Hotel California for your data. A customer had an interest in merging the data from an older account into a new one, just to simplify matters. Enterprise data. Going back years. Not even leaving the region. The AWS rep in the meeting kinda pauses, says: "We'll get back to you on the cost to do that." The sticker shock was enough that the customer simply inherited the ol…

Eh? I've never had a problem moving data out of AWS. Have people lost the ability to write export and backup scripts?

There's a cost for data egress (but not ingress)

Re: Almost every infrastructure decision I endorse or regret

#475

Post by marcosdumay » Sat, Feb 10, 2024, 2:50 PM UTC

ildjarn wrote: post #414

marcosdumay wrote: post #109

Earlier quoted context omitted.

Serial integers always work better than any uuid as PKs, but the thing with uuid4 is that it disrupts any kind of index or physical ordering you decide to put on your data. Uuids are really for external communication, not in-system organization.

Serial index forces a synchronisation point on every entity that can create records. If this is only ever a single database that’s fine but plenty of apps can’t scale this way.

They don't. Clustered databases deal with parallel generation of them just fine.

They require periodic synchronization. What isn't a big deal at all and is required by many other database features.

Re: Almost every infrastructure decision I endorse or regret

#476

Post by pavel_lishin » Sat, Feb 10, 2024, 2:52 PM UTC

> Discourage private messages and encourage public channels.

I wish my current company did this. It's infuriating. The other day, I asked a question about how to set something up, and a manager linked me to a channel where they'd discussed that very topic - but it was private, and apparently I don't warrant an invite, so instead I have to go bother some other engineers (one of whom is on vacation.)

Private channels should be for sensitive topics (legal, finance, etc) or for "cozy spaces" - a team should have a private channel that feels like their own area, but for things like projects and anything that should be searchable, please keep things public.

Re: Almost every infrastructure decision I endorse or regret

#477

Post by brickteacup » Sat, Feb 10, 2024, 2:52 PM UTC

fcarraldo wrote: post #461

interroboink wrote: post #340

Earlier quoted context omitted.

My (peripheral) experience is that it is much cheaper to get data in than to get data out. When you have the amount of data being discussed — "Enterprise data. Going back years." — that can get very costly. It's the amount of data where it makes more sense to put hard drives on a truck and drive across the country rather than send it over a network, where this becomes an issue (actually, probably a bit before then).

AWS actually has a service for this - Snowmobile, a storage datacenter inside of a shipping container, which is driven to you on a semi truck. https://aws.amazon.com/snowmobile/

That's only for data into AWS though, not data out

Re: Almost every infrastructure decision I endorse or regret

#478

Post by marcosdumay » Sat, Feb 10, 2024, 2:54 PM UTC

gregw2 wrote: post #428

marcosdumay wrote: post #105

Earlier quoted context omitted.

> not to fuck up schema/db design The neat thing is, you don't. Nobody ever avoids fucking up db design. The best you can do is decide what is really important to get right, and not fuck that part up.

Wow, what an astute comment! Thank you! P.S. to the original person concerned about this though… for your own sake and your successors, please keep trying.

Assuming that was sarcastic, you are free to try, I guess everyone needs to try it once.

Just do the exercise of deciding what is really important first, so you can make sure you succeed for that stuff.

Re: Almost every infrastructure decision I endorse or regret

#479

Post by AaronM » Sat, Feb 10, 2024, 2:55 PM UTC

thelastparadise wrote: post #418

steveBK123 wrote: post #11

Earlier quoted context omitted.

RDS pricing is deranged at the scales I've seen too. $60k/year for something I could run on just a slice of one of my on-prem $20k servers. This is something we would have run 10s of. $600k/year operational against sub-$100k capital cost pays DBAs, backups, etc with money to spare. Sure, maybe if you are some sort of SaaS with a need for a small single DB, that also needs to be resilient, backed up, rock solid bullet…

> $600k/year operational against sub-$100k capital cost pays DBAs, backups, etc with money to spare. One of these is not like the others (DBAs are not capex.) Have you ever considered that if a company can get the same result for the same price ($100K opex for RDS vs same for human DBA), it actually makes much more sense to go the route that takes the human out of the loop? The human shows up hungover, goes crazy, gr…

Not only that, you can't just have one DBA. You need a team a them, otherwise that person is going to be on call 24/7, can never take a vacation, etc. Your probably looking at a minimum of 3.

Re: Almost every infrastructure decision I endorse or regret

#480

Post by cedws » Sat, Feb 10, 2024, 2:57 PM UTC

foofie wrote: post #184

cedws wrote: post #91

Earlier quoted context omitted.

K8S is a disastrous complexity bomb. You need millions upon millions of lines of code just to build a usable platform. Securing Kubernetes is a nightmare. And lock-in never really went away because it's all coupled with cloud specific stuff anyway. Many of the core concepts of Kubernetes should be taken to build a new alternative without all the footguns. Security should be baked in, not an afterthought when you need…

> K8S is a disastrous complexity bomb. You need millions upon millions of lines of code just to build a usable platform. I don't know what you have been doing with Kubernetes, but I run a few web apps out of my own Kubernetes cluster and the full extent of my lines of code are the two dozen or so LoC kustomize scripts I use to run each app.

If you're using a K8S cluster just to deploy a few web apps then it's not really a platform that you could provide to an engineering team within a medium-large company. You could probably run your stuff on ECS.