Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monoliths Are the Future

changelog.com

Post reply on HN
Print view View original

41–50 of 567 posts

Re: Monoliths Are the Future

#41

Post by BrentOzar »

bcrosby95 wrote: post #29
BrentOzar wrote: post #8

I'm a database guy, so the question I get from clients is, "We're thinking about breaking up our monolith into a bunch of microservices, and we want to use best-of-breed persistence layers for each microservice. Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting?" Analysts expect to be able to connect to one system, see their data, and write queries for it. They were neve…

Microservices are primarily about silo'ing different engineering teams from eachother. If you have a singular reporting database that a singular engineering team manages I'm not sure its a big deal. Reporting might be a "monolith" but the system as a whole isn't. Teams can still deploy their services and change their database schemas without stepping on eachother's toes.

> Teams can still deploy their services and change their database schemas

No, because as soon as you change your schema, you have to plan ahead with the reporting team for starters. The reports still have to be able to work the same way, which means the data needs to be in the same format, or else the reports need to be rewritten to handle the changes in tables/structures.

Re: Monoliths Are the Future

#42

Post by andrew_n »

Spot on. The metaphor I typically use here is cleaning up a mess vs spreading it around. If you have a really big mess and spend a year or two rearranging it into dozens or hundreds of smaller messes, yes the big obvious mess is gone, but the overall amount of mess has likely gone up and by segregating everything you’ve probably made it much harder to someday get to a clean state.

If you’re moving to microservices because the number of people working on a project is growing too large to manage and you need independent teams, great. If you’re refactoring to microservices because “we’re going to do everything right this time,” this is just big-rewrite-in-disguise.

Whatever engineering quality improvements you’re trying to make—tech stack modernization, test automation, extracting common components, improved reliability, better encapsulation—you’re probably a lot better off picking one problem at a time and tackling it directly, measuring progress and adjusting course, rather than expecting a microservices rewrite to magically solve a bunch of these problems all at once.

Re: Monoliths Are the Future

#44

Post by manigandham »

keithb- wrote: post #33
andy_ppp wrote: post #17

Earlier quoted context omitted.

> Reporting on a bunch of different databases is a hard nut to crack. It's not necessarily a bad idea though :-/

I thought that's what solutions like Calcite[1] were for: running queries across disparate data sources. Yes, you still need to have adapters for each source to normalize access semantics. No, not all sources will have the same schema. But if you're trying to combine Postgres and DynamoDB into a single query, you would narrow your view to something that exists in both places, e.g. customer keys, meta data, etc. Maybe…

Calcite is only a SQL parser and planner. It doesn't execute anything and is meant to be a component in a larger database system.

You need to use something like Spark, Presto or Drill if you want to run queries across different data sources.

Re: Monoliths Are the Future

#45

Post by BrentOzar »

gowld wrote: post #26
BrentOzar wrote: post #8

I'm a database guy, so the question I get from clients is, "We're thinking about breaking up our monolith into a bunch of microservices, and we want to use best-of-breed persistence layers for each microservice. Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting?" Analysts expect to be able to connect to one system, see their data, and write queries for it. They were neve…

Build a reporting database that maintains a copy of data from other data stores. Simple enough. Surely you wouldn't run analytics directly on your prod serving database, and risk a bad query taking down your whole system?

> Surely you wouldn't run analytics directly on your prod serving database, and risk a bad query taking down your whole system?

Uhh, yep, that's exactly how a lot of businesses work.

There are defenses at the database layer. For example, in Microsoft SQL Server, we've got Resource Governor which lets you cap how much CPU/memory/etc that any one department or user can get.

Re: Monoliths Are the Future

#46

Post by bcrosby95 »

rjkennedy98 wrote: post #24
reading-at-work wrote: post #6

I think the underlying point, as expressed by the author, is that trendy new architecture patterns will never be a panacea for bad engineering, though that's often how they're implicitly sold as ideas.

It is actually sold as a way to fix bad engineering? I've literally never heard that before. I think almost everyone knows that microservices are hard. On the other hand they are often sold as a way to increase developer velocity. And I do sometimes wonder if that is the case (based on personal experience).

Yes. I see people push microservices all the time because their monolith is poorly designed. If your monolith is messy, it's on you, not monoliths.

Re: Monoliths Are the Future

#47

Post by BrentOzar »

1MoreThing wrote: post #38
BrentOzar wrote: post #8

I'm a database guy, so the question I get from clients is, "We're thinking about breaking up our monolith into a bunch of microservices, and we want to use best-of-breed persistence layers for each microservice. Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting?" Analysts expect to be able to connect to one system, see their data, and write queries for it. They were neve…

Isn't this the whole point of a data lake?

> Isn't this the whole point of a data lake?

Yes, but data lakes don't fill themselves. Each team has to be responsible for exporting every transaction to the lake, either in real time or delayed, and then the reporting systems have to be able to combine the different sources/formats. If each microservice team expects to be able to change their formats in the data lake willy-nilly, bam, there breaks the report again.

Re: Monoliths Are the Future

#48

Post by BrentOzar »

rjkennedy98 wrote: post #19
BrentOzar wrote: post #8

I'm a database guy, so the question I get from clients is, "We're thinking about breaking up our monolith into a bunch of microservices, and we want to use best-of-breed persistence layers for each microservice. Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting?" Analysts expect to be able to connect to one system, see their data, and write queries for it. They were neve…

This is what Kafka is for. You put Kafka on top of your database to expose data and events. Now BI can take the events put them into their system as they want.

> You put Kafka on top of your database to expose data and events. Now BI can take the events put them into their system as they want.

Right, that's the data warehouse method that I described. "Put them into their system" is a lot harder work than just typing in "put Kafka on top of your database."

Re: Monoliths Are the Future

#49

Post by dmos62 »

reading-at-work wrote: post #6

I think the underlying point, as expressed by the author, is that trendy new architecture patterns will never be a panacea for bad engineering, though that's often how they're implicitly sold as ideas.

> architecture patterns will never be a panacea for bad engineering

That's undeniably true; but, it's too easy to slip from that to "good engineering doesn't require good architecture, good technology, etc." I think it has to be stressed that adapting the big picture features of a solution is as important as adapting the finer features.

Re: Monoliths Are the Future

#50

Post by 9dev »

holoduke wrote: post #3

Designing an application from scratch where pure microservices is implemented is in my opinion the same as over engineering possible future performance issues. Splitting up your application in many services requires a lot of thinking and designing. Challenges with syncing, communication etc are not always easy to deal with. That's why I agree to start as a monolith but with architectural principles to still have mult…

I'm totally with you here, but from my experience with a chaotic team, people end up doing weird things when they have access to the full database from every microservice: say you have an authorization service and several applications with a public API that rely on tokens signed by that service. Suddenly we get a requirement to automatically generate a user account when something happens in another application. I didn't look what my devs were doing for a second, and suddenly the offending application does an insert on the users table. The developer even went the extra mile to copy-paste the token generation method into their code.

I ended up restricting database access, but wouldn't have thought that was necessary. To me it was obvious all services should only communicate via their public API, but I guess that's not so much of a no-brainer as I thought.

Post reply on HN