Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monoliths Are the Future

changelog.com

Post reply on HN
Print view View original

311–320 of 567 posts

Re: Monoliths Are the Future

#311

Post by sidlls »

satyrnein wrote: post #275
jlmorton wrote: post #255

Earlier quoted context omitted.

> Spend 12 months building ETLs Okay, sounds reasonable enough for a complex enterprise. > to feed a torrent of raw data to your lake Well, there's the problem. Why is it taking a year to export data in its raw, natural state? The entire point of a data lake is that there is no transformation of the data. There's no need to verify the data is accurate. There's no need to make sure it's performant. It's just data expo…

If you are "export[ing] data in its raw, natural state" then haven't you lost the isolation benefits of microservices? Now you have external systems dependent on your implementation details, and changing your schema will break them.

One way to avoid this is to have the microservices to publish data changes to some other (monolith) system, like an MQ system with a specified scheme for the payload.

On the other hand, the notion that “microservices == completely independent of everything else” is an unrealistic one to hold.

Re: Monoliths Are the Future

#312

Post by disintegore »

js8 wrote: post #118
MosheZada wrote: post #43

Why do we need to choose one of monolith and microservices? What about simply "services"? Monolith doesn't have to be split into 50 microservices, it can be split to 3 services

> What about simply "services"? Let's call them "tiers". I think 3 is a reasonable number..

"Tiers" imply some sort of vertical partitioning. These "non-micro" services may coexist on a layer, transcend boundaries, overlap, etc.

In any case, there's already such a term. It's called SOA: https://en.wikipedia.org/wiki/Service-oriented_architecture. Microservices arguably evolved out of this.

Re: Monoliths Are the Future

#313

Post by SahAssar »

jmngomes wrote: post #75
sharadov wrote: post #25

Earlier quoted context omitted.

No one has solved that problem and it sucks, and what ends up happening is you end up again porting that data from those disparate SQL and NoSQL databases either to a warehouse which is RDBMS or you put it into a datalake. That's again possible if you somehow manage to find all the drivers. You're doubly screwed if you have a hybrid - cloud and on-prem setup.

>> No one has solved that problem (...) you end up again porting that data from those disparate SQL and NoSQL databases either to a warehouse That's exactly how that problem has been solved successfully for the past 20 years.

Well, it's debatable if that is "solving" the problem or just hiding or mitigating it.

Re: Monoliths Are the Future

#314

Post by edmundsauto »

TeMPOraL wrote: post #289
smoe wrote: post #162

Earlier quoted context omitted.

> It can be done wrong, but when executed correctly [...] Quite the self-fulfilling prophecy there. > Yes, it's easier to get your data, but in the long run [...] Systems can and should be evolved and adapted over time. E.g. deploying components of the monolith as separate services. You can't easily predict what the requirements for your software going to be in say 10 years. And depending on the stage a company is, e…

> easy access to data for business decisions outweighs engineering idealism I think there are different levels of sophistication of "engineering idealism". GP talks about "data ownership", and I get the desire to keep the data a microservice is responsible for locked in tightly with it. But let's be precise why it's good: because isolating responsibility reduces complexity. Not because code has some innate right to p…

Isolating responsibility reduces complexity for that piece of code. It increases complexity for assembling the whole thing into a holistic package, which is usually what analytics primary need is.

YMMV, but the tradeoff is less complexity at the SWE/prod department, and more at the analytics team.

Re: Monoliths Are the Future

#315

Post by solarengineer »

abhishekjha wrote: post #304
jonahbenton wrote: post #271

Earlier quoted context omitted.

Sorry, but "making an API that gets you this data" is the wrong answer. Most APIs are glorified wrappers around individual record-level operations like- get me this user- or constrained searches that return a portion of the data, maybe paginated. Reporting needs to see all the data. This is a completely different query and service delivery pattern. What happens to your API service written in a memory managed/garbage-…

>It goes into GC hell Can you exapand on this a little? Or a paper that I can read?

The parent comment said “is asked for all records..GC hell “.

Since a micro service deals with only its own data and reporting is then across services, we’d need to query across services to get data and make sense of it. If we’d ever need to query all records, then such records would become domain objects in the micro services first before being passed along. A large number of domain objects would require a large amount of memory. Processing and releasing domain objects will result in GC on the released objects.

Re: Monoliths Are the Future

#316

Post by WoahNoun »

satyrnein wrote: post #275
jlmorton wrote: post #255

Earlier quoted context omitted.

> Spend 12 months building ETLs Okay, sounds reasonable enough for a complex enterprise. > to feed a torrent of raw data to your lake Well, there's the problem. Why is it taking a year to export data in its raw, natural state? The entire point of a data lake is that there is no transformation of the data. There's no need to verify the data is accurate. There's no need to make sure it's performant. It's just data expo…

If you are "export[ing] data in its raw, natural state" then haven't you lost the isolation benefits of microservices? Now you have external systems dependent on your implementation details, and changing your schema will break them.

Changing the schema of an upstream data source almost always breaks or requires updates to the downstream analytics system. It's an unavoidable problem whether its a microservice or a monolith; you just get to choose where you put the pain.

Consider:

Source Data -> Data Lake -> ETL Process -> Reporting DataWarehouse(s)/DataMart(s) -> User Queries

vs

Source Data -> Data Lake -> User Queries

vs

MonolithDB -> User queries

vs

MonolithDB -> ETL Process -> Reporting DataWarehouse(s)/DataMart(s) -> User Queries

A schema change in the source data should be easily updated in the ETL process in example 1. Most changes are minimal (adding, removing, renaming columns). And for a complete schema redesign in the source data, a new entry in the data lake should be created and the owners of the ETL process should decide if the new schema should be mangled to fit their existing reporting tables or to build new ones. Across the four models I outlined above, the first is by far the easiest to update and maintain, IMO.

Re: Monoliths Are the Future

#317

Post by Jupe »

All designs have trade-offs. When trade-offs appear, you either accept them or mitigate them...

If it's important to know how many blue widgets are bought at night in Europe, vs. how many blue watcha-ma-call-its are bought in the evening in the US, and your location, orders and product data are in separate micro-services, you are kinda out of luck.

And, as mentioned by others, replication and API wrappers on micro-services suck for reporting.

If you built an eventing system, you'd be better off tapping into that to update the central reporting data store (warehouse/lake/etc.) I've used this myself to "good" effect. (some chance of failures, a little behind the times, etc.)

The central database may be "monolithic" in nature, but at least you'd be able to report on the data. If you expect to modify data in the feeding micro-service's databases, then yes, you do have a monolith. But, if it's "just" for reporting, it's like a dynamically-updated replica of the pertinent data for your reporting.

Re: Monoliths Are the Future

#318

Post by cc81 »

tflinton wrote: post #280

Distributed monoliths (or micro services) do have some advantages: 1. Easier for users to see "who owns what" (albeit a module pattern could fix this as well). 2. Different hardware resources or scaling for different parts of a monolith really isn't possible. If one module requires 16GB then everytime you scale a horizontally you must have at least 16GB, you're at the mercy of your worst module in the monolith. 3. De…

>3. Deploys are very difficult, and as you scale to over 10 developers increasingly becomes difficult to push up (it takes one persons bad commit to hold everyone in the organization from deploying).

I feel that is a way too low number. If don't have good enough engineering practices with branching, pull requests/code reviews, unit/integration tests to handle 10 people then microservices will be painful as well.

I would say like 3+ teams at least to really justify it? You can do it earlier but I don't see it as a necessary benefit.

>7. Microservices can be cheaper when you scale, you don't have the GCD of memory/CPU/disk requirements as you do in a monolith.

Yes, but by default they are more expensive until you reach a certain scale and it needs to be a specific type of scaling.

Re: Monoliths Are the Future

#319

Post by throwaway894345 »

BrentOzar wrote: post #8

I'm a database guy, so the question I get from clients is, "We're thinking about breaking up our monolith into a bunch of microservices, and we want to use best-of-breed persistence layers for each microservice. Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting?" Analysts expect to be able to connect to one system, see their data, and write queries for it. They were neve…

> Reporting on a bunch of different databases is a hard nut to crack.

Maybe, but your business analyst already needs to connect to N other databases/data-sources anyway (marketing data, web analytics, salesforce, etc, etc), so you already need the infrastructure to connect to N data sources. N+1 isn't much worse.

Re: Monoliths Are the Future

#320

Post by silasb »

sytse wrote: post #39

At GitLab a couple of years ago we had to fight the temptation to split the application up in hundreds of micro-services. I'm glad we did and today GitLab has a big monolith but also a ton of services working together https://docs.gitlab.com/ee/development/architecture.html#com... I did an interview about this yesterday https://www.youtube.com/watch?v=WDqGaPGBZ9Y

Your architecture is similar to something that we want to get towards. How do you handle standing up all these pieces (or even a subset of these pieces) in the dev environment?
Post reply on HN