Earlier quoted context omitted.
I'm going to disagree heavily here. The world of cloud computing, microservices, and hosted/managed services has made the analyst and data engineers job easier than ever. If the software team builds a new dynamodb table, they simple give the AWS account for the analytics team the appropriate IAM permissions and the analytics team will set-up an off-peak bulk extract. A single analyst can easily run an entire data war…
What/where do you run this mythical one-analyst pipeline, though? Is that in cloud services too? Airflow? Kubeflow? Apache Beam? It sounds like you're just pushing the problem around.
Monoliths Are the Future
201–210 of 567 posts
Re: Monoliths Are the Future
#202Earlier quoted context omitted.
> But you do not simply go and poke your reporting fingers into individual service databases. This is why I distrust all of the monolith folks. Yes, it's easier to get your data, but in the long run you create unmaintainable spaghetti that can't ever change without breaking things you can't easily surface. Monoliths are undisciplined and encourage unhealthy and unsustainable engineering. Microservices enforce separat…
You're saying "monoliths encourage unhealthy engineering" and then in the next sentence say "when executed correctly" for microservices. That sounds like a having/eating cake type situation.
Re: Monoliths Are the Future
#203Earlier quoted context omitted.
https://prestosql.io/ It can access all those different databases. You can also make your own connectors that make your services appear as tables, which you can query with SQL in the normal way. So if the new accounts micro-service doesn't have a database, or the team won't let your analysts access the database behind it, you can always go in through the front-door e.g. the rest/graphql/grpc/thrift/buzzword api it ex…
Do issues arise when trying to match types across different persistence layers?
Re: Monoliths Are the Future
#204Re: Monoliths Are the Future
#205My employer adopted microservices for a very specific reason: it became nearly impossible to deploy the monolith. With hundreds of commits trying to go out every day, probability that at least one would break something approached 1. Then everything had to be rolled back. Getting unrelated concerns into separate deployable artifacts rescued our velocity. It came with many of its own challenges, too! A great deal of in…
Our company migrated for very similar reasons - I'm surprised that people aren't mentioning this pain point more. Maybe the overall execution of microservices is poor, so people generally would rather go back to the old thing.
It worked great and I was honestly shocked when I worked for other companies that plan their releases like they're shipping shrink-wrapped CDs in 1999.
Re: Monoliths Are the Future
#206Earlier quoted context omitted.
No, Java has its own share of problems that it didn't inherit from anywhere. No custom value types, no operator overloading and the distaste for AOT compilation, to name a few. Also culture of code generation instead of using some kind of macro system.
It's like a museum of problems. Waves of solutions that became problems after. The culture of code generation before the culture of XML configuration.
They are there for a reason.
Re: Monoliths Are the Future
#207Earlier quoted context omitted.
> Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting? One of the key concepts in microservice architecture is data sovereignity. It doesn't matter how/where the data is stored. The only thing that cares about the details of the data storage is the service itself. If you need some data the service operates on for reporting purposes, make an API that gets you this data and…
> But you do not simply go and poke your reporting fingers into individual service databases. This is why I distrust all of the monolith folks. Yes, it's easier to get your data, but in the long run you create unmaintainable spaghetti that can't ever change without breaking things you can't easily surface. Monoliths are undisciplined and encourage unhealthy and unsustainable engineering. Microservices enforce separat…
How does creating a tangle of microservices (effectively distributed objects) really solve the problem?
Re: Monoliths Are the Future
#208Pretty much microservices happened because of managers. Managers are promoted to high level jobs such as Director or VP based on number of direct reports. This won't happen with a small team of devs. So they need an infrastructure team, a devops team, a SRE team, a QA team, a microservices core team, an access management team, network engineering team, AWS integration team, and the list goes on. What was once a four…
Re: Monoliths Are the Future
#209There are two big reasons to go to microservices (note that the exact definition of microservice can vary a lot). 1. Organizational streamlining. If the team working on the monolith becomes to large, then coordinating and pushing out changes quickly can become incredibly difficult. One rule of thumb I've heard is the two pizzas rule. If two pizzas can't feed the team working on a system, it's time to break up the sys…
You build a monolithic application. Everyone works on the same code base. Things are broken up into modules/classes/packages. From the programmers point of view it's just like working on a standard Java project or something similar.
The magic happens at the method and module boundaries. When the application is first started everything works normally. Methods call other methods using addresses. As the application runs some parts of it become hotter than other parts. At some trigger point an included process starts that spins up 1+ cloud instances. Only the hot code is deployed to the instances. If necessary the instance is load balanced on multiple nodes. You configure the triggers and whatnot as part of the applications config. The framework/language would either come with support for popular cloud services or allow you to create whatever system you need to create the instances.
My hypothetical language/framework would proxy all method calls and remap object instances to the new instance(s). If the extracted code cools down enough it is integrated back into the main monolith. At that point proxying is turned off and the methods use address again.
Using this approach you get the all the advantages of a monolith (interface compatibility checked by compiler, not needed EVERY service writing their own http code, etc). Of course you can't optimize latency as easily and merging is harder with monoliths. There's undoubtedly a hundred other reasons why this is a terrible idea.
Re: Monoliths Are the Future
#210My employer adopted microservices for a very specific reason: it became nearly impossible to deploy the monolith. With hundreds of commits trying to go out every day, probability that at least one would break something approached 1. Then everything had to be rolled back. Getting unrelated concerns into separate deployable artifacts rescued our velocity. It came with many of its own challenges, too! A great deal of in…
My company is currently moving to microservices, but for different reasons. The problem you raise was in fact fixed years ago in our org simply by properly decoupling our "monolithic" app into modules. The top-level build was simply a collection of all pre-built modules. After each dependency update, an automated regression would run, and if pass rate was less than X%, the change was not pushed upstream. Really, micr…