Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monoliths Are the Future

changelog.com

Post reply on HN
Print view View original

191–200 of 567 posts

Re: Monoliths Are the Future

#191

Post by Swinx43 »

MadWombat wrote: post #131
BrentOzar wrote: post #8

I'm a database guy, so the question I get from clients is, "We're thinking about breaking up our monolith into a bunch of microservices, and we want to use best-of-breed persistence layers for each microservice. Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting?" Analysts expect to be able to connect to one system, see their data, and write queries for it. They were neve…

> Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting? One of the key concepts in microservice architecture is data sovereignity. It doesn't matter how/where the data is stored. The only thing that cares about the details of the data storage is the service itself. If you need some data the service operates on for reporting purposes, make an API that gets you this data and…

I came here simply to echo this statement! Design a reporting solution that is responsible for ingesting data from these micro services' persistence layers. Analysts should only ever be querying this reporting solution and should not be allowed to connect directly to any micro service persistence layer or API.

We have a whole industry around Analytics and Data and the tools and processes to build this reporting layer is well established and proven.

Nothing will give you as many nightmares as letting your analysts loose on your micro service persistence layers!

Re: Monoliths Are the Future

#192

Post by rantwasp »

crimsonalucard wrote: post #171
rantwasp wrote: post #89

as always the truth is somewhere in the middle. monoliths make a lot of sense when you're starting out and you can see all your code in one place and you can build, test and deploy everything together. as the service grows there are arguments to be made around splitting it (based on usage patterns, loads, etc). the things that most people don't get is that: microservices are not free (now you're doing all this devops…

>as always the truth is somewhere in the middle This is the greatest illusion of our time. We tend to think of all things in the world as "middle" like apples and oranges. If this is your viewpoint then you are biased, the truth has equal probability in being in all extremes just as well as the middle.

fair point. the likelihood of it being 100% oranges(or apples) is 0. so not true middle but in between

Re: Monoliths Are the Future

#193

Post by StavrosK »

echelon wrote: post #140
MadWombat wrote: post #131

Earlier quoted context omitted.

> Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting? One of the key concepts in microservice architecture is data sovereignity. It doesn't matter how/where the data is stored. The only thing that cares about the details of the data storage is the service itself. If you need some data the service operates on for reporting purposes, make an API that gets you this data and…

> But you do not simply go and poke your reporting fingers into individual service databases. This is why I distrust all of the monolith folks. Yes, it's easier to get your data, but in the long run you create unmaintainable spaghetti that can't ever change without breaking things you can't easily surface. Monoliths are undisciplined and encourage unhealthy and unsustainable engineering. Microservices enforce separat…

You're saying "monoliths encourage unhealthy engineering" and then in the next sentence say "when executed correctly" for microservices. That sounds like a having/eating cake type situation.

Re: Monoliths Are the Future

#194

Post by sharadov »

jandrewrogers wrote: post #127
jmngomes wrote: post #75

Earlier quoted context omitted.

>> No one has solved that problem (...) you end up again porting that data from those disparate SQL and NoSQL databases either to a warehouse That's exactly how that problem has been solved successfully for the past 20 years.

The high latency between operational data and that data being reflected in reporting using traditional data warehouse pipelines makes it difficult for companies to make effective business decisions in a fast-paced business environment. Even in competent execution, that latency is frequently measured in weeks for big businesses. In the last few years, I've been approached by a number of traditional big businesses look…

Our data is near real time. But I can see how latency can be an issue based on size of the data and transformations are needed on the data before it hits the DW. But there are solutions - kafka being one.

Re: Monoliths Are the Future

#195

Post by bob1029 »

We wound up taking a similar journey:

Monolith => Microservices => Monolith

I wouldn't say the journey was completely pointless, because the fact that we had to deploy 10+ services to make a single environment whole required us to build extremely powerful CI/CD management tools that we happen to be able to re-use in the (new) monolith case today. This journey was also a really good growth and learning opportunity for the team. Everyone who has touched this project and has seen both ends of the distributedmonolith spectrum is now radicalized towards preferring the monolith approach.

On the trip back into a monolith, we didn't just stop with the binary outputs of our codebase. We also made the entire codebase a monorepo. We have a single solution (VS2019) within that monorepo which tracks all of our projects. Prior, we had upwards of 15 different repositories to keep track of. Being able to right-click on a type, select "View all References" and legitimately get every possible reference to that type across the entire enterprise is the most powerful thing I have yet to see in my career.

Re: Monoliths Are the Future

#196

Post by SigmundA »

speleding wrote: post #90
BrentOzar wrote: post #45

Earlier quoted context omitted.

> Surely you wouldn't run analytics directly on your prod serving database, and risk a bad query taking down your whole system? Uhh, yep, that's exactly how a lot of businesses work. There are defenses at the database layer. For example, in Microsoft SQL Server, we've got Resource Governor which lets you cap how much CPU/memory/etc that any one department or user can get.

That's not a good idea. The usage patterns for a production database and one that runs reporting are very different. Reporting has long running queries with complex joins, production has many parallel short queries. If you start mixing the two, you can no longer reliably tune your database. For example you would want the alert for slow queries set set to a different time out on production and on reporting. Also, you…

Pretty trival to setup read replicas using log shipping in MSSQL and postgresql to offload analytical loads to secondary servers.

I work on a monolith that does this, but its usually not even necessary a single db server on modern hardware with proper resource governing can handle quite a bit.

Re: Monoliths Are the Future

#197

Post by freewilly1040 »

willvarfar wrote: post #190
BrentOzar wrote: post #8

I'm a database guy, so the question I get from clients is, "We're thinking about breaking up our monolith into a bunch of microservices, and we want to use best-of-breed persistence layers for each microservice. Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting?" Analysts expect to be able to connect to one system, see their data, and write queries for it. They were neve…

https://prestosql.io/ It can access all those different databases. You can also make your own connectors that make your services appear as tables, which you can query with SQL in the normal way. So if the new accounts micro-service doesn't have a database, or the team won't let your analysts access the database behind it, you can always go in through the front-door e.g. the rest/graphql/grpc/thrift/buzzword api it ex…

Do issues arise when trying to match types across different persistence layers?

Re: Monoliths Are the Future

#198

Post by segmondy »

derision wrote: post #119
segmondy wrote: post #111

Earlier quoted context omitted.

This is a solved problem, "Data Engineering" teams solve this by building a data pipeline. It's not for all orgs, but for a large org, this is worth doing right.

"data pipeline" is just the new trendy phrase for ETL, which the GP mentioned. just because a solution exists, does not make it a solved problem. it's not the right solution for everyone

no it's not.

ETL is really database focused and batch focused , Extract, Transform, Load.

Data pipeline, is a combination of streams and batch. For example, you can implement a Data Capture using something like https://debezium.io/

Here's how Netflix solves it https://netflixtechblog.com/dblog-a-generic-change-data-capt...

Overview

"Change-Data-Capture (CDC) allows capturing committed changes from a database in real-time and propagating those changes to downstream consumers [1][2]. CDC is becoming increasingly popular for use cases that require keeping multiple heterogeneous datastores in sync (like MySQL and ElasticSearch) and addresses challenges that exist with traditional techniques like dual-writes and distributed transactions [3][4]."

Re: Monoliths Are the Future

#199

Post by davnicwil »

eternalny1 wrote: post #35

What is old is new again. I've been a software engineer for over 30 years and have dealt with companies always trying to jump on the next bandwagon. One company I worked with tried to move our entire monolith application, which was well architected and worked fine, over to a microservices-based architecture and the result was an unstable, complex mess. Sometimes, if it's not broke, don't try to "fix" it. I can say th…

> people are trying to replicate stuff that works fine in other languages in JavaScript. Mostly because they are only familiar with JavaScript and don't realize this stuff already exists and doesn't need to be in JavaScript.

Do you have a concrete example to illustate this, and what issues it causes?

On the surface I'm not sure I agree, if what you're saying is people wanting a certain feature should switch languages to get it rather than build it into the language they already use.

Re: Monoliths Are the Future

#200

Post by jasonlotito »

You know, at some point, I get tired of these rehashed ideas. This isn't a new thought, this isn't a unique perspective. Yes, we fully realize that you shouldn't jump on the newest fad just because you want to. Microservices aren't a silver bullet. All the things that have been talked about for years/decades now.

Monoliths aren't the future. They never left. Rather, they are still an option, along with microservices.

Blindly adopting anything is silly and error prone.

It gets tiring hearing the same advice preached every few years about a new techonology.

Post reply on HN