Live data from Hacker News

The costs of microservices (2020)

robertovitillo.com

331–340 of 410 posts

Re: The costs of microservices (2020)

#331
post #330

The article touches on it a bit, but in my experience microservices multiply operational problems (especially at startups where you don't have big, dedicated infrastructure teams). All of a sudden you have 5-10x things getting built in CI and deployed. You need some way to debug issues so usually distributed tracing comes up. Now that you have 10x as many of everything, you obviously want to try to centralize things…

A typical startup with 20k DAU: "We need scale, microservices, k8s, CDNs, etc!".

Stackoverflow: "We run a single .NET-based multi-tenant web app running across just nine web servers, at 5% to 10% of capacity" [1].

Hacker News: "HN still runs on one core, at least the part that serves logged-in requests, and yes this will all get better someday...it kills me that this isn't done yet but one day you will all see." [2]. HN had ~12M MAU by the end of 2022.

[1]: https://www.datacenterdynamics.com/en/news/stack-overflow-st...

[2]: https://news.ycombinator.com/item?id=35157344

Re: The costs of microservices (2020)

#332
post #331
post #330

The article touches on it a bit, but in my experience microservices multiply operational problems (especially at startups where you don't have big, dedicated infrastructure teams). All of a sudden you have 5-10x things getting built in CI and deployed. You need some way to debug issues so usually distributed tracing comes up. Now that you have 10x as many of everything, you obviously want to try to centralize things…

A typical startup with 20k DAU: "We need scale, microservices, k8s, CDNs, etc!". Stackoverflow: "We run a single .NET-based multi-tenant web app running across just nine web servers, at 5% to 10% of capacity" [1]. Hacker News: "HN still runs on one core, at least the part that serves logged-in requests, and yes this will all get better someday...it kills me that this isn't done yet but one day you will all see." [2].…

Does HN have the SLA and uptime that the startup signed with their clients and will half an hour of downtime sink HN?

Re: The costs of microservices (2020)

#333
post #300

Earlier quoted context omitted.

Does .net not have a way to run simple private repository like Nexus? Over in JVM-land that's a basic thing that you'd be doing anyway.

It's the difference between drag-dropping a reference between "projects" or whatnot in your monolith project, or coming up with a publish pipeline after which your merged code gets into the repo so that other people can use it, if you have three projects in three checkouts that all have to be coordinated and merged in order so that you don't accidentally break your environment and worry about security and and and....…

> It's the difference between drag-dropping a reference between "projects" or whatnot in your monolith project, or coming up with a publish pipeline after which your merged code gets into the repo so that other people can use it

You already have that pipeline for release/deployment though, don't you? (And it's already hooked up to your SSO or what have you).

> if you have three projects in three checkouts that all have to be coordinated and merged in order so that you don't accidentally break your environment and worry about security and and and....

It's the same as any other library dependency though, which is a completely normal thing to deal with. The cost of separating your pieces enough that you can use different versions of a library in different services is that you can use different versions of a library in different services.

I'm skeptical about microservices as a deployment model, but I'm absolutely convinced that code-level modularisation and independent release cycles for things that deploy independently are worthwhile, at least if your language ecosystem has decent dependency management.

Re: The costs of microservices (2020)

#334

Earlier quoted context omitted.

Really man. I have almost two decades developing software and yet, I feel a lot more comfortable having all my code reviewed. If anything I get annoyed by junior developers in my team when they just rub-stamp my PRs because supposedly I am this super senior guy that can't err. Code Reviews are supposed to give you peace of mind, not being a hassle. During all this time, I've seen plenty of "small changes" having comp…

At 30 years of coding professionally in great engineering-focused organizations, and that on top of nearly a lifetime of having coded for myself, I’ve concluded code reviews barely work. I agree with everything you say here, but honestly it’s quite ineffective. I wish we had found something better than unit tests (often misleading and fragile) and the ability of a qualified reviewer to maintain attention and context…

IMO catching bugs is a nice side-effect of code reviews.

The primary value I've seen across teams has been more on having shared team context across a codebase, if something goes bump in the night you've got a sense on how that part of the codebase works. It's also a great opportunity for other engineers to ask "why" and explain parts that aren't obvious or other context that's relevant. We'll find the occasional architectural mismatch(although we like to catch those much earlier in the design process) and certainly prevented bugs from shipping but if that's the primary focus I think a team is missing a lot of the value from regular code reviews.

Re: The costs of microservices (2020)

#335

Earlier quoted context omitted.

Without getting tied up in the whole "every language except assembly, Python and possibly JavaScript already solved this problem by forcing people to adhere to module-level APIs" argument, I think the crux of the issue is that the article just defines microservice architecture as any architecture consisting of multiple services, and explicitly states "there doesn’t have to be anything micro about the services". Which…

I think we should start calling this Pendulum Blindness. We just go from 'one' to 'too many' as equally unworkable solutions to all of our problems, and each side (and each person currently subscribed to that 'side') knows the other side is wrong. The assumption is that this means their side is right instead of reality, which is nobody is right. The moderates are right, but their answers are wiggly and they're too bu…

You might like "Software Architecture: The Hard Parts." Though you already describe some of the points of the book. There isn't a magic bullet and every decision to split something apart or which parts to combine has various trade-offs.

The book isn't perfect. The use of afferent and efferent terminology and some of the arbitrary methods to put numbers on decisions weren't ideal. Most of the concepts are sound. The fact that almost every decision has cost/benefit and real world implications for a living product was refreshing. That a monolith can't be cut over instantly with zero effort to a perfect system is absolutely true.

It's good food for thought for anyone considering slicing up a monolith, but maybe don't follow it to the letter.

Re: The costs of microservices (2020)

#336

Earlier quoted context omitted.

Still trying to unlearn that one. Turns out, most decisions are cheap to revert or backtrack on, while delaying them until Last Responsible Moment often ends in shooting past that moment.

Depends who you are, mostly the ones making the decisions aren't usually listening to their developers (maybe by choice maybe because they are at the whim of a customer), so their cost functions are calibrated towards course changing being more expensive than less. By the time your devs are saying "this sucks" you've long overshot.

That's a good point. The (estimate of the) cost function is key, it determines whether delaying decision is better or worse than making it eagerly and reverting if it turns out to be wrong. You give a good case for when delaying is a better choice.

In my case however, I ended up applying the "LRM" strategy to my own work, where I'm both the decision maker and the sole implementer. This is where I see my mistake. In my defense, the software development books that argued for delaying decisions did not warn that this applies to larger decisions in projects developed by teams, and may not apply to small-scale design decisions made by an individual contributor or a small team in the scope of a small piece of work. It took me way too long to realize that, for most of my day-to-day choices, the cost function is pretty much flat.

Re: The costs of microservices (2020)

#337
post #300

Earlier quoted context omitted.

Does .net not have a way to run simple private repository like Nexus? Over in JVM-land that's a basic thing that you'd be doing anyway.

It's the difference between drag-dropping a reference between "projects" or whatnot in your monolith project, or coming up with a publish pipeline after which your merged code gets into the repo so that other people can use it, if you have three projects in three checkouts that all have to be coordinated and merged in order so that you don't accidentally break your environment and worry about security and and and....…

The only way package feed complexity works -- and really microservices in general -- is to be absolutely fastidious about backwards compatibility of your packages and as open as possible with transitive dependency versions.

Nuget does provide mechanisms for obsoleting packages, so it's reasonable to enforce that new packages should allow for a few versions worth of backwards compatibility before deprecation and finally pulling the plug.

Re: The costs of microservices (2020)

#338

> And think of the sheer number of libraries - one for each language adopted - that need to be supported to provide common functionality that all services need, like logging. This is the #1 reason we quit the microservices game. It is simply a complete waste of mental bandwidth to worry about with the kind of tooling we have in 2023 (pure cloud native / infiniscale FaaS), especially when you have a customer base (e.g…

We have 2 supported langs across our cloud teams and our core libraries are dual-lang'd where applicable; meaning they're both Ruby Gems and Pypi packages (Python3) in one repo with unified APIs and backed by a shared static config where applicable (capability definitions etc.) Each dual lib is released simultaneously with matching SemVer versions to our various artifactory instances & S3 buckets (for our lambda "layers"), automatically on every push to mainline by CI/CD.

It works surprisingly well. We're evaluating a 3rd language but won't make that choice lightly (if it happens at all.)

We have 14+ micro services, and it's fairly easy to "rewrite the shit pile" when you actually follow the micro designation. One of our services was originally in perl and we, quite mechanically, rewrote it in ruby in a sprint to align with our other services.

Speaking from personal experience, when monoliths and the teams working on them get big enough, you start having "action at a distance" problems, where seemingly benign changes affect completely unrelated flows, often to catastrophic effect.

You make an innocuous looking resource update in the monolith, like an update to a CSS stylesheet that fixes a bug in a flow your team owns, now breaks 10 others flows owned by teams you never heard of because they were using the existing structure for selenium tests or some js that now fails to traverse the dom because some order of selectors changed, etc.

Microservices are as much a team organizational tool as they are a code one. The idea being those that work on the service "know it", and all it does. They can wrap their head around the whole thing. I think some orgs don't really get this point and _start_ with microservices, completely unnecessarily, for the stage they're at as a company. You always start with a monolith and if you get to the point where everyone is stepping on each other's toes from the lack of enforceable boundaries in the code, you do the obvious and start to create those boundaries.

Microservices aren't the only way to do this of course. Any way of dividing up your service with enforceable contracts will work. Modules get designated with codeowners, assigned to various teams. Resources that were once shared get split up to align with the team structures better. Many frameworks allow multiple "apps" or distinct collections of APIs, so you can still ship your one-binary without splitting out the collections into different services. As soon as you have to independently scale one set of APIs but not another, you can state looking at service boundaries again. For the majority, that day will never come.

Re: The costs of microservices (2020)

#339
post #331

Earlier quoted context omitted.

A typical startup with 20k DAU: "We need scale, microservices, k8s, CDNs, etc!". Stackoverflow: "We run a single .NET-based multi-tenant web app running across just nine web servers, at 5% to 10% of capacity" [1]. Hacker News: "HN still runs on one core, at least the part that serves logged-in requests, and yes this will all get better someday...it kills me that this isn't done yet but one day you will all see." [2].…

Does HN have the SLA and uptime that the startup signed with their clients and will half an hour of downtime sink HN?

No. A number of startups don't either. I hazard to say that even giants like Tinder or Uber likely can have a 30 min outage and lose some revenue and goodwill, but not be hit by some exorbitant liabilities.

Also, microservices add both resilience (by running many copies) and fragility (many loosely coupled moving parts). Which effect prevails, depends on many factors.

Re: The costs of microservices (2020)

#340
post #147

Earlier quoted context omitted.

For some reason, most of the people I've worked with recently are either fully into monoliths or lots of fine grained, interdependent microservices. They don't seem to understand there's a useful middleground of adding fewer, larger data services, etc. It's like SOA isn't a hot topic so people aren't aware of it.

I've been using the term 'macro-services' to describe this middle ground.

We used to just call them, "services"
Post reply on HN