Almost every infrastructure decision I endorse or regret • Hacker News Board

Re: Almost every infrastructure decision I endorse or regret

#501

Post by Grimm665 » Sat, Feb 10, 2024, 4:11 PM UTC

jq-r wrote: post #392

morsecodist wrote: post #131

> Picking AWS over Google Cloud I know this is an unpopular opinion but I think google cloud is amazing compared to AWS. I use google cloud run and it works like a dream. I have never found an easier way to get a docker container running in the cloud. The services all have sensible names, there are fewer more important services compared to the mess of AWS services, and the UI is more intuitive. The only downside I ha…

> "regular cadence meetings with our AWS account manager" and I am not sure what could be discusse. As being on a number of those calls, its just a bunch of crap where they talk like a scripted bot reading from corporate buzzword bingo card over a slideshow. Their real intention is two fold. To sell you even more AWS complexity/services, and to provide "value" to their person of contact (which is person working in yo…

This rings so true from experience it hurts.

Re: Almost every infrastructure decision I endorse or regret

#502

Post by layer8 » Sat, Feb 10, 2024, 4:17 PM UTC

Sankozi wrote: post #401

f549abd0 wrote: post #341

Disagree on the point and reasoning about the single database. Sounds like they experienced badly managed and badly constrained database. The described fks and relations: that's what the key constraints and other guard rails and cascades are for - so that you are able to manage a schema. That's exactly how you do it: add in new tables that reference old data. I think the regret is actually not managing the database,…

> How about "database is used by everyone, it becomes cared for by everyone". So every one needs to know every use case of that database? Seems very unlikely if there are multiple teams using same DB. FKs? Unique constraints? Not null colums? If not added at the creation of the table they will never be added - the moment DB is part of a public API you cannot do a lot of things safely. The only moment when you want to…

The database should never ever become part of a public API.

You don’t share a DB for performance reasons (rather the opposite), you do it to ensure data integrity and consistency.

And no, not everyone needs to know every use case. But every team needs to have someone who coordinates any overlapping schema concerns with the other teams. This needs to be managed, but it’s also not rocket science.

Re: Almost every infrastructure decision I endorse or regret

#503

Post by Cacti » Sat, Feb 10, 2024, 4:20 PM UTC

mixmastamyk wrote: post #274

Cacti wrote: post #191

Earlier quoted context omitted.

Using Jira feels like using IBM enterprise web software from 2005, and I am simply not going to make my teams put up with that amount of inanity.

Found the person who never used Lotus Notes haha.

I was blown away when I found out a couple years ago that there were major corporations still using that as their primary communication platform.

Re: Almost every infrastructure decision I endorse or regret

#504

Post by fcarraldo » Sat, Feb 10, 2024, 4:30 PM UTC

xmcqdpt2 wrote: post #473

fcarraldo wrote: post #461

Earlier quoted context omitted.

AWS actually has a service for this - Snowmobile, a storage datacenter inside of a shipping container, which is driven to you on a semi truck. https://aws.amazon.com/snowmobile/

They do not! > Q: Can I export data from AWS with Snowmobile? > > Snowmobile does not support data export. It is designed to let you quickly, easily, and more securely migrate exabytes of data to AWS. When you need to export data from AWS, you can use AWS Snowball Edge to quickly export up to 100TB per appliance and run multiple export jobs in parallel as necessary. Visit the Snowball Edge FAQs to learn more. https:/…

Oh, TIL! Thanks for correcting me.

Re: Almost every infrastructure decision I endorse or regret

#505

Post by fulafel » Sat, Feb 10, 2024, 4:34 PM UTC

paulgb wrote: post #115

sroussey wrote: post #51

If you are startup that can can’t afford a DBA, then why why why are you using Kubernetes?

I think a lot of startups have a set of requirements that is something like: - I want to spin up multiple redundant instances of some set of services - I want to load balance over those services - I want some form of rolling deploy so that I don’t have downtime when I deploy - I want some form of declarative infrastructure, not click-ops Given these requirements, I can’t think of an alternative to managed k8s that is…

AWS Copilot (if you're on AWS). It's a bit like the older Elastic Beanstalk for EC2.

Re: Almost every infrastructure decision I endorse or regret

#506

Post by dvfjsdhgfv » Sat, Feb 10, 2024, 4:35 PM UTC

oooyay wrote: post #222

nyc_data_geek wrote: post #86

Earlier quoted context omitted.

Some orgs are looking at moving back to on prem because they're figuring this out. For a while it was vogue to go from capex to opex costs, and C suite people were incentivized to do that via comp structures, hence "digital transformation" ie: migration to public cloud infrastructure. Now, those same orgs are realizing that renting computers actually costs more than owning them, when you're utilizing them to a signif…

Context: I build internal tools and platforms. Traffic on them varies, but some of them are quite active. My nasty little secret is for single server databases I have zero fear of over provisioning disk iops and running it on SQLite or making a single RDBMS server in a container. I've never actually run into an issue with this. It surprises me the number of internal tools I see that depend on large RDS installations…

The problem with single instance is that while performance-wise it's best (at least on bare metal), there comes a moment when you simply have too much data and one machine can't handle. Your your scenario, it may never come up, but many organizations face this problem sooner or later.

Re: Almost every infrastructure decision I endorse or regret

#507

Post by dvfjsdhgfv » Sat, Feb 10, 2024, 4:39 PM UTC

tormeh wrote: post #389

jwr wrote: post #266

Earlier quoted context omitted.

I do that. In fact I've been doing it for years, because every time I do the math, AWS is unreasonably expensive and my solo-founder SaaS would much rather keep the extra money. I think there is an unreasonable fear of "doing the routing and everything". I run vpncloud, my server clusters are managed using ansible, and can be set up from either a list of static IPs or from a terraform-prepared configuration. The same…

When talking about Hetzner pricing, please don’t change the subject to AWS pricing. The two have nothing in common, and intuition derived from one does not transfer to the other.

> please don’t change the subject to AWS pricing

Why? The only reason I'm using Hetzner and not AWS for several of my own projects (even though I know AWS much better since this is what I use at work) is an enormous price difference in each aspect (compute, storage, traffic).

Re: Almost every infrastructure decision I endorse or regret

#508

Post by dalyons » Sat, Feb 10, 2024, 4:44 PM UTC

eduction wrote: post #489

dalyons wrote: post #236

Earlier quoted context omitted.

That’s just not true though, I’ve worked at decent sized companies without shared RDBMs, so you don’t have to. You DO have to share data in other ways, usually datawarehouse or services, but that is not the same thing.

To me this is semantics. So it’s a data warehouse rather than a database. Ok. Or we share data from a common source via “services” - ok but that’s another word for a database and a client (using http to do the talking doesn’t really change anything). I’m not saying literally every source of data has to be shared and centrally managed. I’m also not saying “rdbms accessed via traditional client and queried via sql” whe…

Ok, but the OP and the article are talking specifically about a directly shared rdbms scenario, not some nebulous concept of shared data.

Also, operationally it’s not “semantics” at all. You don’t get into (many) operational problems with analysts sharing a datawarehouse. You absolutely do with online apps sharing a rdbms, they aren’t the same thing.

Re: Almost every infrastructure decision I endorse or regret

#509

Post by sgarland » Sat, Feb 10, 2024, 4:45 PM UTC

Sankozi wrote: post #417

sgarland wrote: post #37

Earlier quoted context omitted.

But then the DB Team – if you have one – is responsible for 50 databases, each full of their own unique problems. This will undoubtedly go over poorly, but honestly I think every data decision should be gated through the DB Team (again, if you have them). Your proposed schema isn’t normalized? Straight to jail. You don’t want to learn SQL? Also straight to jail. You want to use a UUIDv4 as a primary key? Believe it o…

No single team should not be responsible for all databases. If such team exists they will either become bottleneck for every other team (by auditing carefully each schema change) or become bloated and not utilized 90% of time, or (most common) they will become nearly useless or even harmful - they will not be really responsible and they will act as dumb proxy - they will introduce latency to the schema updates, but t…

> If such team exists they will either become bottleneck for every other team (by auditing carefully each schema change)

That’s the point. Would you send a backend code review to a frontend team? Why do DBs not deserve domain expertise, especially when the entire company depends on them?

> they are not responsible for the whole product, just for the database

I assure you, that’s a lot to be responsible for at scale.

> DB team could act as an auditor and expert support, but they should never be fully responsible for DB layer.

Again, the issue here is when the DB gets borked enough that a SME is required to fix it, they effectively do become responsible, because no CTO is going to accept, “sorry, we’ll be down for a couple of days because our team doesn’t really know how this thing works.”

And if your answer is, “AWS Premium Support,” they’ll just tell you to upsize the instance. Every time. That is not a long-term strategy.

Re: Almost every infrastructure decision I endorse or regret

#510

Post by ndriscoll » Sat, Feb 10, 2024, 4:45 PM UTC

sgarland wrote: post #30

renewiltord wrote: post #9

Earlier quoted context omitted.

You don't get the higher end machines on AWS unless you're a big guy. We have Epyc 9684X on-prem. Cannot match that at the price on AWS. That's just about making the choices. Most companies are not DB-primary.

I think most people who’ve never experienced native NVMe for a DB are also unaware of just how blindingly fast it is. Even io2 Block Express isn’t the same.

Funny enough, the easiest way to experience this is probably to do some performance experimentation on the machine you code on. If it's a laptop made in the last few years, the performance you can get out of it knowing that it's sipping on a 45W power brick with probably not great cooling will make you very skeptical of when people talk about "scale".