Earlier quoted context omitted.
I agree completely. The current FAA outage is a good example of this. What if the system responsible for NOTAMs was the same system responsible for sending system outage messages. They weren’t so that status could be communicated while mitigation was done on a system with unrelated concerns. At the same time, a few person startup should probably focus on what allows them to deliver the fastest. I’ve seen that work wi…
We had a major cockup at work a few years ago caused by bad organizational choices and Conway’s Law. We had a disk array go sideways, which is when we learned that some dumb motherfucker had put our wiki on the same SAN with production traffic. You know, the wiki where you keep all your run books for solving production issues? Everyone was furious and that team lost some prestige that day. How dumb do you have to be?
Microservices are hard
261–270 of 356 posts
Re: Microservices are hard
#262I really do not understand the debate on monoliths and microservice anymore. Context matters so much. Should you have absolutely everything in 1 system. No. And I think no one thinks that anymore. Should you spilt your system into as many pieces as possible? No, of course not. You are prop. storing files one place and have a data in a database another place. And most likely none are on the webserver receiving request…
> enough pieces so that; 1. your engineers … are not afraid of doing changes. I don’t really understand that argument, and I don’t really feel safer making a change to a microservice inside a large system, as opposed to making a change to a monolith — the consequences of a mistake are equal in both cases (although harder to observe/debug in a microservice architecture) — am I missing something?
From the Amazon book description:
Quite simply, test-driven development is meant to eliminate fear in application development. While some fear is healthy (often viewed as a conscience that tells programmers to "be careful!"), the author believes that byproducts of fear include tentative, grumpy, and uncommunicative programmers who are unable to absorb constructive criticism.
Re: Microservices are hard
#263No, they aren't. The entire point of the big ball of mud is that there are no meaningful divisions in the code. Everything uses everything willy-nilly, at the smallest possible level of abstraction. There is, metaphorically if not always entirely literally, not a single line of code in the system that you can change without fear of bringing something else down that you may not have even known they existed. Microservi…
> It may be the sloppiest, crappiest definition every, with dynamic types Funny you said that since a microservice API is always dynamically typed and its usage cannot be checked by compiler. And the more microservices you use the more dynamically typed the whole project gets overall. While you can opt into a free for all everything importing everything, all languages do also support creating modules which define API…
Not true at all, I have a build system in place that when changes are made to Typescript, the JSON Schema on the endpoints is updated, and client libraries are updated.
Types are validated at both compile time and runtime.
This is just one of many solutions to the problem, there are a lot of ways to get type safety for service endpoints, at both runtime and compile time.
Re: Microservices are hard
#264Is my setup "microservices"? Well, maybe, depending on your definition, but, in truth I don't really care - it works pretty well.
We also do "DDD" with it and have multiple AWS accounts with these marking the domain borders. Comms between the accounts is via eventbridge or (very rarely) inter-account API invocation.
This allows many of the benefits of microservices, without the pain of dealing with k8s. Clean separation of domains, reduced cognitive loads for teams, each of which looks after all the stuff in a single account (so-called feature teams, where each team designs/manages and runs everything in that account/domain).
The hard bit was defining the domain borders, and the inter-domain protocols/interactions, but, once this is well defined, things work pretty well.
Having come from a k8s world, this setup feels so much nicer, and lighter, and easier to get stuff built in a both fast and performant way.
It would be interesting to see trends away from _classical_ "cloud-native" (k8s) setups with microservices, to true serverless setups. I wonder how much of k8s's lunch serverless has managed to eat so far.
Re: Microservices are hard
#265Earlier quoted context omitted.
Microservices is not the same, though.
I don't see that distinction being made. The counter argument is things like libraries vs microservices. Those articles ignore the common case that probably is true for 99% of the apps - we live in a service based architecture. The Internet is service based. Everything an app talks to is service based. So quit arguing about it. Microservices don't address any of the issues I ever faced or people on Hacker News talks…
I am currently getting a hard intro into a microservice first architecture. Where customers are managed in one service, users are managed in a different one, user operations are in a third one. All have their own databases and much of the data is copied between them. There are microservices, that are used by only one other microservice.
That's where we are now - arguing about the validity of "microservices are the best/first" approach.
Microservices are defined as self contained, highly granular, separately developed and deployed horizontally scalable systems.
Think about the debates about Linux vs Hurd, that's where we are.
Re: Microservices are hard
#266No, they aren't. The entire point of the big ball of mud is that there are no meaningful divisions in the code. Everything uses everything willy-nilly, at the smallest possible level of abstraction. There is, metaphorically if not always entirely literally, not a single line of code in the system that you can change without fear of bringing something else down that you may not have even known they existed. Microservi…
Do they? The code itself may be in entirely separate repos but still be tightly coupled. Monoliths can have cleanly separated libraries/modules, those modules built from separate repos or at the very least, different namespaces.
The "macroservices" I've been seeing are many separate containers all sharing at least one data store. So they have all of the disadvantage of the "ball of mud" monolith combined with all of the disadvantage of much more complicated infrastructure. Yet the people working on them think they're "doing microservices" because k8!
The microservice separation is not just code in separate repos. It's also everything else behind the kimono - keep that kimono clasped tightly!
Re: Microservices are hard
#267Having run k8s and _classical_ microservices before, I am now in a much happier place just using the AWS serverless suite (lambda, API GW, CF, SNS, SQS, eventbridge, dynamodb, etc). Is my setup "microservices"? Well, maybe, depending on your definition, but, in truth I don't really care - it works pretty well. We also do "DDD" with it and have multiple AWS accounts with these marking the domain borders. Comms between…
Re: Microservices are hard
#268(True) Microservices have no dependencies on anything but unstructured text data. They do not couple to a database, the business understanding of what it's doing, a domain model, or anything else. They perform a simple, idempotent, business task that can never fail although it can create various error chains.
Programming at scale is tough. There's no free ride here. All you've done is turn the traditional model of coding "inside-out" and now you've got a ton of work doing all of the wiring.
But if you keep your microservice doing one simple useful business thing, then all of that inside-out work become business decisions. What do we do if the sign-up fails? How do we move IMPORTANT-THING to those other guys to use? You still have business coupling: things change and you have to adapt. But you're not coupled at the _coding_ level. If there's any magic, that's it. Your business should be able to wander all over the place and your microservices hold up just fine. The old way, where we may have coupled every little business need or want with every piece of code in the system, was not only a pain, more importantly it was impossible to keep organized in any one person's head and aligned with everyone else at scale.
What I see is a lot of drift. Folks start coupling things up, perhaps by trying to create one domain model to rule them all. They start creating microservices to do _system_ activities, like flushing a cache. There should be a one-to-one correspondence between your microservices and interesting business conversations. That's a hella discipline to maintain. It may force a lot of conversations you thought you could avoid by hiding them in a class hierarchy somewhere. Once you start drifting, pretty soon you're writing essays like this. And then here we are/
Re: Microservices are hard
#269Earlier quoted context omitted.
We had a major cockup at work a few years ago caused by bad organizational choices and Conway’s Law. We had a disk array go sideways, which is when we learned that some dumb motherfucker had put our wiki on the same SAN with production traffic. You know, the wiki where you keep all your run books for solving production issues? Everyone was furious and that team lost some prestige that day. How dumb do you have to be?
Seems a little harsh. We all overlook things like this. Things like storage are so reliable we expect them to always be available. When you lay it out like you did, it does sound silly.
Re: Microservices are hard
#270If your microservices are hard to test you've designed shitty microservices. Period.
You microservice can be easily tested. But then you may end up with - is your whole system testable and how do you do good integration tests? Right now I work in a pure microservice architecture, where there are no proper system or integration tests. And we had minor bug fixes break things spectacularly.