Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost every infrastructure decision I endorse or regret

cep.dev

Post reply on HN
Print view View original

401–410 of 644 posts

Re: Almost every infrastructure decision I endorse or regret

#401

Post by Sankozi »

f549abd0 wrote: post #341

Disagree on the point and reasoning about the single database. Sounds like they experienced badly managed and badly constrained database. The described fks and relations: that's what the key constraints and other guard rails and cascades are for - so that you are able to manage a schema. That's exactly how you do it: add in new tables that reference old data. I think the regret is actually not managing the database,…

> How about "database is used by everyone, it becomes cared for by everyone".

So every one needs to know every use case of that database? Seems very unlikely if there are multiple teams using same DB.

FKs? Unique constraints? Not null colums? If not added at the creation of the table they will never be added - the moment DB is part of a public API you cannot do a lot of things safely.

The only moment when you want to share DB is when you really need to squeeze every last bit of performance - and even then, you want to have one owner and severly limited user accounts (with white list of accessible views and stored procedures).

Re: Almost every infrastructure decision I endorse or regret

#402

Post by maccard »

yellow_lead wrote: post #264
maccard wrote: post #103

Earlier quoted context omitted.

Because I can go from main.go to a load balanced, autoscaling app with rolling deploys, segeregated environments, logging & monitoring in about 30 minutes, and never need to touch _any_ of that again. Plus, if I leave, the guy who comes after me can look at a helm chart, terraform module + pipeline.yml and figure out how it works. Meanwhile, our janq shell script based task scheduler craps out on something new every…

If you can do all that in 30 minutes (or even a few hours), I would love to read an article/post about your setup, or any resources you might recommend.

I've just done it a dozen times at this point. Hello world from gin-gonic [0], terraform file with a DO K8s cluster [1] and load balancer, and CI/CD [2] on deploy. There's even time to make a cuppa when you run terraform.

We use this for our internal services at work, and the last time I touched the infra was in 2022 according to git

[0] https://github.com/gin-gonic/gin

[1] https://gist.github.com/donalmacc/0efbb0b377533232da3f776c60....

[2] https://docs.digitalocean.com/products/kubernetes/how-to/dep...

Re: Almost every infrastructure decision I endorse or regret

#404

Post by storyinmemo »

mciancia wrote: post #380
LadyCailin wrote: post #329

Earlier quoted context omitted.

It’s the cost of data egress, which isn’t free.

But there is no paid egress when we are moving data between account within one region, rigth?

There is. You pay a price for any cross-VPC traffic.

Re: Almost every infrastructure decision I endorse or regret

#405

Post by jgalt212 »

andreif wrote: post #339
alentred wrote: post #308

Earlier quoted context omitted.

BigQuery - Athena Pub/Sub - Kinesis Cloud CDN - CloudFront Cloud Domains - Route 53 ...

I thought you meant API and parameters. Blaming them for product names is weird to me.

It's nice when things do what they say on the tin. That being said, it's hard to build a "brand" when you start out with a generic name.

Re: Almost every infrastructure decision I endorse or regret

#406

Post by varjag »

Draiken wrote: post #328
hhsectech wrote: post #319

Earlier quoted context omitted.

Eh? I've never had a problem moving data out of AWS. Have people lost the ability to write export and backup scripts?

The ingress/egress cost is ridiculously high. Some companies don't care, but it is there and I've seen it catch people off guard multiple times.

Oh come on from the description both accounts could be sitting on the same datacenter LAN.

Re: Almost every infrastructure decision I endorse or regret

#407

Post by Sankozi »

maccard wrote: post #96
Scubabear68 wrote: post #27

Earlier quoted context omitted.

Not a downside to me. Each team maintains their own DB and pays for their own choices. In the kitchen sink model all teams are tied together for performance and scalability, and some bad apple applications can ruin the party for everyone. Seen this countless times doing due diligence on startups. The universal kitchen sink DB is almost always one of the major tech debt items.

> Not a downside to me. Each team maintains their own DB and pays for their own choices. This is how you end up with the infamous "jira and confluence have two different markdown flavors" issue.

I don't think Jira and Confluence different markdown setup is due to them not sharing their databases. It is just poor product management from Attlasian.

Re: Almost every infrastructure decision I endorse or regret

#408

Post by benreesman »

ndjshe3838 wrote: post #78

I’m imagining a developer in the 90s/00s reading this list and being baffled by the complexity/terminology

We had FB up to 6 figures in servers and a billion MAUs (conservatively) before even tinkering with containers.

The “control plane” was ZooKeeper. Everything had bindings to it, Thrift/Protobuf goes in a znode fine. List of servers for FooService? znode.

The packaging system was a little more complicated than a tarball, but it was spiritually a tarball.

Static link everything. Dependency hell: gone. Docker: redundant.

The deployment pipeline used hypershell to drop the packages and kick the processes over.

There were hundreds of services and dozens of clusters of them, but every single one was a service because it needed a different SKU (read: instance type), or needed to be in Java or C++, or some engineering reason. If it didn’t have a real reason, it goes in the monolith.

This was dramatically less painful than any of the two dozen server type shops I’ve consulted for using kube and shit. It’s not that I can’t use Kubernetes, I know the k9s shortcuts blindfolded. But it’s no fun. And pros built these deployments and did it well, serious Kubernetes people can do everything right and it’s complicated.

After 4 years of hundreds of elite SWEs and PEs (SRE) building a Borg-alike, we’d hit parity with the bash and ZK stuff. And it ultimately got to be a clear win.

But we had an engineering reason to use containers: we were on bare metal, containers can make a lot of sense on bare metal.

In a hyperscaler that has a zillion SKUs on-demand? Kubernetes/Docker/OCI/runc/blah is the friggin Bezos tax. You’re already virtualized!

Some of the new stuff is hot shit, I’m glad I don’t ssh into prod boxes anymore, let alone run a command on 10k at the same time. I’m glad there are good UIs for fleet management in the browser and TUI/CLI, and stuff like TailScale where mortals can do some network stuff without a guaranteed zero day. I’m glad there are layers on top of lock servers for service discovery now. There’s a lot to keep from the last ten years.

But this yo dawg I heard you like virtual containers in your virtual machines so you can virtualize while you virtualize shit is overdue for its CORBA/XML/microservice/many-many-many repos moment.

You want reproducibility. Statically link. Save Docker for a CI/CD SaaS or something.

You want pros handing the datacenter because pets are for petting: pay the EC2 markup.

You can’t take risks with customer data: RDS is a very sane place to splurge.

Half this stuff is awesome, let’s keep it. The other half is job security and AWS profits.

Re: Almost every infrastructure decision I endorse or regret

#409

Post by thelastparadise »

hhsectech wrote: post #317
jumploops wrote: post #280

Earlier quoted context omitted.

Funny story time. I was once part of an acquisition from a much larger corporate entity. The new parent company was in the middle of a huge cloud migration, and as part of our integration into their org, we were required to migrate our services to the cloud. Our calculations said it would cost 3x as much to run our infra on the cloud. We pushed back, and were greenlit on creating a hybrid architecture that allowed us…

There are two possible scenarios here. Firstly, they can't find the talent to support what you implemented...or more likely, your docs suck! I've made a career out of inheriting other peoples whacky setups and supporting them (as well as fixing them) and almost always its documentation that has prevented the client getting anywhere. I personally dont care if the docs are crap because usually the first thing I do is u…

> the first thing I do is update / actually write the docs to make them usable.

OK so the docs are in sync for a single point of time when you finish. Plus you get to have the context in your head (bus factor of 1, job security for you, bad for the org.)

How about if we just write clean infra configs/code, stick to well known systems like docker, ansible, k8s, etc.

Then we can make this infra code available to an on prem LLM and ask it questions as needed without it drifting out of sync overtime as your docs surely will.

Wrong documentation is worse than no documentation.

Re: Almost every infrastructure decision I endorse or regret

#410

Post by danpalmer »

lysecret wrote: post #366
erostrate wrote: post #322

The author leads infrastructure at Cresta. Cresta is a customer service automation company. His first point is about how happy he is to have picked AWS and their human-based customer service, versus Google's robot-based customer service. I'm not saying there's anything wrong, and I'm oversimplifying a bit, but I still find this amusing.

Haha very good catch. I prefer GCP but I will admit any day of the week that their support is bad. Makes sense that they would value good support highly.

We used to use AWS and GCP at my previous company. GCP support was fine, and I never saw anything from AWS support that GCP didn't also do. I've heard horror stories about both, including some security support horror stories from AWS that are quite troubling.
Post reply on HN