No Comment...
Monoliths Are the Future
321–330 of 567 posts
Re: Monoliths Are the Future
#322Earlier quoted context omitted.
Sorry, but "making an API that gets you this data" is the wrong answer. Most APIs are glorified wrappers around individual record-level operations like- get me this user- or constrained searches that return a portion of the data, maybe paginated. Reporting needs to see all the data. This is a completely different query and service delivery pattern. What happens to your API service written in a memory managed/garbage-…
>It goes into GC hell Can you exapand on this a little? Or a paper that I can read?
The service will need to read all its data and put it into objects, then extract the data from the objects to report it, then garbage collect all of that. For every single record in its entire data set.
You could say but oh, why not just return the underlying data without making objects? Well now you are exposing the underlying data format, which is what we’re trying to avoid by giving this job to the service.
Re: Monoliths Are the Future
#323Earlier quoted context omitted.
Not exactly. It's hard to tell from the outside if a monolith was architecture well or is about the fall over. In a microservice architecture it's harder to pretend you're doing it right.
I disagree, "doing it wrong" just looks different there.
The argument (which I sort of buy) is that microservices provide rails that keep people from doing certain stupid things like N clients depending on the data schema (making the schema a de-facto public interface).
The trick with microservices is that the ecosystem is maturing and there are still lots of ways to screw up other things that are harder to screw up with monoliths. In time 95% of those will go away (my specific prediction is that one day we will write programs that express concurrency and the compiler/toolchain will work out distributing the program across Cloud services--although "Cloud" will be an antiquated term by then--including stitching together the relevant logs, etc and possibly even a coherent distributed debugger experience).
Re: Monoliths Are the Future
#324Earlier quoted context omitted.
My previous company went this direction. I wouldn't recommend it. Say you version each module and pull in specified versions. It'll work fine, right up until two modules both try to pull different versions of a third module. In practice, you have to update multiple modules at once to avoid conflicts, which, in turn, can require updating other team's code. It also turns out some tools like Maven don't prevent conflict…
That's what semver is for, right? Breaking changes go in a major, same major means that the latest is always compatible. You'll have the same issue with microservices if you introduce breaking changes.
Yeah, but even with semver library owners have problems. If you need to make a critical change (e.g. a security update), you can either (a) wait until everyone does a version bump on their own schedule, or (b) do the version bump yourself, which means you're deploying every app that relies on your library. (a) might not be feasible for critical things, and (b) means that you might be deploying changes that the app isn't ready for yet.
For example, if master is v1.5.0, and I'm an app that uses v1.1.0, then if the library owner bumps to 1.5.1 for a critical update, I need to go from 1.1.0 -> 1.5.1, which might involve changes I'm not ready for yet. I better have phenomenal integration testing to make sure the update is safe to do.
Re: Monoliths Are the Future
#325I'm a database guy, so the question I get from clients is, "We're thinking about breaking up our monolith into a bunch of microservices, and we want to use best-of-breed persistence layers for each microservice. Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting?" Analysts expect to be able to connect to one system, see their data, and write queries for it. They were neve…
just recreate the database you broke apart into something less flexible
Re: Monoliths Are the Future
#326Earlier quoted context omitted.
If you are "export[ing] data in its raw, natural state" then haven't you lost the isolation benefits of microservices? Now you have external systems dependent on your implementation details, and changing your schema will break them.
Changing the schema of an upstream data source almost always breaks or requires updates to the downstream analytics system. It's an unavoidable problem whether its a microservice or a monolith; you just get to choose where you put the pain. Consider: Source Data -> Data Lake -> ETL Process -> Reporting DataWarehouse(s)/DataMart(s) -> User Queries vs Source Data -> Data Lake -> User Queries vs MonolithDB -> User queri…
If the benefit of microservices is primarily an organizational one around ownership, Conway's Law, and so on, then Example 1 still seems problematic, because it's likely a different team that has to deal with the fallout.
Another strategy is that the service has an explicit API or report specification; that way, the team that owns the services also owns the problem of continuing to support that while changing their internal implementation.
Of course, whether the benefits are worth the cost is probably organization specific, just like microservices in general.
Re: Monoliths Are the Future
#327Earlier quoted context omitted.
I came here simply to echo this statement! Design a reporting solution that is responsible for ingesting data from these micro services' persistence layers. Analysts should only ever be querying this reporting solution and should not be allowed to connect directly to any micro service persistence layer or API. We have a whole industry around Analytics and Data and the tools and processes to build this reporting layer…
This is seriously why my company still has monoliths. Our databases are open to way too many people. What's worse, they are multi tenant making refactoring really hard.
Why monoliths? Everyone still wants to to have OLAP and OLTP systems where analytics are done on OLAP. Where having this separation you can get data from multiple sources to put into your analytics.
I cannot imagine people not doing that and having need to have stats in real time. For most shopping/banking stuff you can get away with once in 24 hours dumps and then analytics can be done on that.
Re: Monoliths Are the Future
#328Earlier quoted context omitted.
> Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting? One of the key concepts in microservice architecture is data sovereignity. It doesn't matter how/where the data is stored. The only thing that cares about the details of the data storage is the service itself. If you need some data the service operates on for reporting purposes, make an API that gets you this data and…
Sorry, but "making an API that gets you this data" is the wrong answer. Most APIs are glorified wrappers around individual record-level operations like- get me this user- or constrained searches that return a portion of the data, maybe paginated. Reporting needs to see all the data. This is a completely different query and service delivery pattern. What happens to your API service written in a memory managed/garbage-…
Wait, I would assume that the people in need of reporting would have a pretty good idea of what those reports should look like. That means you know exactly what data needs to be read from a data store optimized for reporting. Each micro-service contributes their share of data to a data store optimized for reading. This is a text-book use case for a non-relational document store. I'm really not seeing what's so difficult about building such a process.
Re: Monoliths Are the Future
#329I'm a database guy, so the question I get from clients is, "We're thinking about breaking up our monolith into a bunch of microservices, and we want to use best-of-breed persistence layers for each microservice. Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting?" Analysts expect to be able to connect to one system, see their data, and write queries for it. They were neve…
In my last job I was a DevOps guy in a Data Eng. team and we used microservice (actually serverless) extensively to the point that none of our ETL relied on servers (they were all serverless; AWS lambda). Now databases themselves are different stories, they are the persistence/data layer that microservices themselves use . But it's actually doable and I'd even say much easier to use microservices/serverless for ETL b…
Is there a book or course to buy toward getting an understanding of the more philosophical concepts underneath your approach?
Re: Monoliths Are the Future
#330Earlier quoted context omitted.
It used to be that people wrote Fortran in any language. Now they're rewriting Java in every language.
The bad reputation that enterprise Java gets, comes from enterprise C and C++ codebases. https://en.wikipedia.org/wiki/Structured_systems_analysis_an... https://en.wikipedia.org/wiki/Booch_method Any language that gets into enterprise architect hands, with projects spread around multiple development sites with several consulting agencies, gets their FactoryFactories and such.
Spring, Hibernate and other "enterprise" monsters came from Java alone. FactoryFactory123.