Almost every infrastructure decision I endorse or regret • Hacker News Board

Re: Almost every infrastructure decision I endorse or regret

#151

Post by klooney » Sat, Feb 10, 2024, 4:37 AM UTC

otterley wrote: post #82

jrockway wrote: post #71

Earlier quoted context omitted.

Why wouldn't you use Kubernetes? There are basically 3 classes of deployments: 1) We don't have any software, so we don't have a prod environment. 2) We have 1 team that makes 1 thing, so we just launch it out of systemd. 3) We have between 2 and 1000 teams that make things and want to self-manage when stuff gets rolled out. Kubernetes is case 3. Like it or not, teams that don't coordinate with each other is how star…

On AWS, at least, there are alternatives such as ECS and even plain old EC2 auto scaling groups. Teams can have the autonomy to run their infrastructure however they like (subject to whatever corporate policy and compliance regime requirements they might have to adhere to). Kubernetes is appealing to many, but it is not 100% frictionless. There are upgrades to manage, control plane limits, leaky abstractions, differe…

K8S has a credible local development and testing story, ECS and ASGs do not. The fact that there's a generic interface for load-balancer like things, and then you can have a different implementation on your laptop, in the datacenter, and in AWS, and everything ports, is huge.

Also, you can bundle your load balancer config and application config together. No written description of the load balancer config + an RPM file to a disinterested different team.

Re: Almost every infrastructure decision I endorse or regret

#152

Post by fshbbdssbbgdd » Sat, Feb 10, 2024, 4:39 AM UTC

morsecodist wrote: post #131

> Picking AWS over Google Cloud I know this is an unpopular opinion but I think google cloud is amazing compared to AWS. I use google cloud run and it works like a dream. I have never found an easier way to get a docker container running in the cloud. The services all have sensible names, there are fewer more important services compared to the mess of AWS services, and the UI is more intuitive. The only downside I ha…

I have had the experience of an AWS account manager helping me by getting something fixed (working at a big client). But more commonly, I think the account manager’s job at AWS or any cloud or SAAS is to create a reality distortion field and distract you from how much they are charging you.

Re: Almost every infrastructure decision I endorse or regret

#153

Post by jacurtis » Sat, Feb 10, 2024, 4:41 AM UTC

macNchz wrote: post #54

CSMastermind wrote: post #10

So by and large I agree with the things in this article. It's interesting that the points I disagree with the author on are all SaaS products: > Moving off JIRA onto linear I don't get the hype. Linear is fine and all but I constantly find things I either can't or don't know how to do. How do I make different ticket types with different sets of fields? No clue. > Not using Terraform Cloud No Regrets I generally recom…

> Cost is the most common complaint and it's almost always from people who don't have it configured correctly (which to be fair Datadog makes it far too easy to misconfigure things and blow up costs). I loved Datadog 10 years ago when I joined a company that already used it where I never once had to think about pricing. It was at the top of my list when evaluating monitoring tools for my company last year, until I go…

> Datadog makes it far too easy to misconfigure things and blow up costs

I'll give you a fun example. It's fresh in my mind because i just got reamed out about it this week.

In our last contract with DataDog, they convinced us to try out the CloudSIEM product, we put in a small $600/mo committment to it to try it out. Well, we never really set it up and it sat on autopilot for many months. We fell under our contract rate for it for almost a year.

Then last month we had some crazy stuff happen and we were spamming logs into DataDog for a variety of reasons. I knew I didn't want to pay for these billions of logs to be indexed, so I made an exclusion filter to keep them out of our log indexes so we didn't have a crazy bill for log indexing.

So our rep emailed me last week and said "Hey just a heads up you have $6,500 in on-demand costs for CloudSIEM, I hope that was expected". No, it was NOT expected. Turns out excluding logs from indexing does not exclude them from CloudSIEM. Fun fact, you will not find any documented way to exclude logs from CloudSIEM ingestion. It is technically possible, but only through their API and it isn't documented. Anyway, I didn't do or know this, so now i had $6,500 of on-demand costs plus $400-500 misc on-demand costs that I had to explain to the CTO.

I should mention my annual review/pay raise is also next week (I report to the CTO), so this will now be fresh in their mind for that experience.

Re: Almost every infrastructure decision I endorse or regret

#154

Post by IggleSniggle » Sat, Feb 10, 2024, 4:41 AM UTC

eduction wrote: post #76

Scubabear68 wrote: post #3

The kitchen sink database used by everybody is such a common problem, yet it is repeated over and over again. If you grow it becomes significant tech debt and a performance bottleneck. Fortunately, with managed DBs like RDS it is really easy to run individual DB clusters per major app.

Management problem masquerading as a tech problem. Being shared between applications is literally what databases were invented to do. That’s why you learn a special dsl to query and update them instead of just doing it in the same language as your application. The problem is that data is a shared resource. The database is where multiple groups in an organization come together to get something they all need. So it nee…

...I worked at a large software organization where larger teams had their own bank account, and there was a lot of internal billing, etc, mixed with plenty of funny-money to go along with it. That's not a contradiction, though, it perfectly illustrated your point for me.

Re: Almost every infrastructure decision I endorse or regret

#155

Post by solatic » Sat, Feb 10, 2024, 4:42 AM UTC

compumike wrote: post #69

colechristensen wrote: post #53

Earlier quoted context omitted.

PagerDuty’s cheapest plan is $21 per user month OpsGenie’s cheapest is $9 per user month but arbitrarily crippled, the plan anybody would want to use is $19 per user month So instead of a factor of ten it’s ten percent cheaper. And i just kind of expect Atlassian to suck. Datadog is ridiculously expensive and on several occasions I’ve run into problems where an obvious cause for an incident was hidden by bad behavior…

Heii On-Call is $32 per month total for your team — not per user. https://heiioncall.com/ (Full disclosure: part of the team building it)

Looks super interesting, and that $3/month for hobbyists is just low enough to meet my budget for hobby services, but please, for on-call stuff, you gotta have alerts that make phone calls. Nothing else is going to wake me in the middle of the night. This is the #1 feature I expect from an on-call service - you're on-call because you will be called.

Re: Almost every infrastructure decision I endorse or regret

#156

Post by piotrkaminski » Sat, Feb 10, 2024, 4:44 AM UTC

morsecodist wrote: post #131

> Picking AWS over Google Cloud I know this is an unpopular opinion but I think google cloud is amazing compared to AWS. I use google cloud run and it works like a dream. I have never found an easier way to get a docker container running in the cloud. The services all have sensible names, there are fewer more important services compared to the mess of AWS services, and the UI is more intuitive. The only downside I ha…

Heartily seconded. Also don't forget the docs: Google Cloud docs are generally fairly sane and often even useful, whereas my stomach churns whenever I have to dive into AWS's labyrinth of semi-outdated, nigh-unreadable crap.

Re: Almost every infrastructure decision I endorse or regret

#157

Post by sroussey » Sat, Feb 10, 2024, 4:44 AM UTC

maccard wrote: post #103

sroussey wrote: post #51

If you are startup that can can’t afford a DBA, then why why why are you using Kubernetes?

Because I can go from main.go to a load balanced, autoscaling app with rolling deploys, segeregated environments, logging & monitoring in about 30 minutes, and never need to touch _any_ of that again. Plus, if I leave, the guy who comes after me can look at a helm chart, terraform module + pipeline.yml and figure out how it works. Meanwhile, our janq shell script based task scheduler craps out on something new every…

I spent zero hours on a MySQL server on bare hardware for seven years.

Admittedly, I was afraid of ever restarting as I wasn’t sure it would reboot. But still…

Re: Almost every infrastructure decision I endorse or regret

#158

Post by lanstin » Sat, Feb 10, 2024, 4:46 AM UTC

layoric wrote: post #139

nextos wrote: post #128

Earlier quoted context omitted.

Same experience here. As a small organization, the quotes we got from cloud providers have always been prohibitively expensive compared to running things locally, even when we accounted for geographical redundancy, generous labor costs, etc. Plus, we get to keep know how and avoid lock-in, which are extremely important things in the long term. Besides, running things locally can be refreshingly simple if you are just…

> Plus, we get to keep know how and avoid lock-in, which are extremely important things in the long term. So much this. The "keep know how" has been so greatly avoided over the past 10 years, I hope people with these skills start getting paid more as more companies realize the cost difference.

When I started working in the 1980s (as a teenager but getting paid) there was a sort of battle between the (genuinely cool and impressive) closed technology of IBM and the open world of open standards/interop like TCP/IP and Unix, SMTP, PCs, even Novell sort of, etc. There was a species of expert that knew the whole product offering of IBM, all the model numbers and recommended solution packages and so on. And the technology was good - I had an opportunity to program a 3093K(?) CM/VMS monster with APL and rexx and so on. Later on I had a job working with AS/400 and SNADS and token ring and all that, and it was interesting; thing is they couldn't keep up and the more open, less greedy, hobbyists and experts working on Linux and NFS and DNS etc. completely won the field. For decades, open source, open standards, and interoperability dominated and one could pick the best thing for each part of the technology stack, and be pretty sure that the resultant systems would be good. Now however, the Amazon cloud stacks are like IBM in the 1980s - amazingly high quality, but not open; the cloud architects master the arcane set of product offerings and can design a bespoke AWS "solution" to any problems. But where is the openness? Is this a pendulum that goes back and forth (and many IBM folks left IBM in the 1990s and built great open technologies on the internet) or was it a brief dawn of freedom that will be put down by the capital requirements of modern compute and networking stacks?

My money is on openness continuing to grow and more and more pieces of the stack being completely owned by openness (kernels anyone?) but one doesn't know.

Re: Almost every infrastructure decision I endorse or regret

#159

Post by sroussey » Sat, Feb 10, 2024, 4:46 AM UTC

paulgb wrote: post #115

sroussey wrote: post #51

If you are startup that can can’t afford a DBA, then why why why are you using Kubernetes?

I think a lot of startups have a set of requirements that is something like: - I want to spin up multiple redundant instances of some set of services - I want to load balance over those services - I want some form of rolling deploy so that I don’t have downtime when I deploy - I want some form of declarative infrastructure, not click-ops Given these requirements, I can’t think of an alternative to managed k8s that is…

A startup with no DBA does not need redundant anything. Too small.

Re: Almost every infrastructure decision I endorse or regret

#160

Post by jacurtis » Sat, Feb 10, 2024, 4:47 AM UTC

tootie wrote: post #22

CSMastermind wrote: post #10

So by and large I agree with the things in this article. It's interesting that the points I disagree with the author on are all SaaS products: > Moving off JIRA onto linear I don't get the hype. Linear is fine and all but I constantly find things I either can't or don't know how to do. How do I make different ticket types with different sets of fields? No clue. > Not using Terraform Cloud No Regrets I generally recom…

Interesting. Atlassian also just launched an integration with OpsGenie. I have the same opinion of JIRA. I've tried many competitors (not Linear so far) and regretted it every time.

I'm not sure they just launched anything. OpsGenie has been an Atlassian product for 5 or more years now. I've been using it for 3-4 myself and its been integrated with Jira the whole time.

In fact, OpsGenie has mostly been on Auto-pilot for a few years now.