Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monoliths Are the Future

changelog.com

Post reply on HN
Print view View original

331–340 of 567 posts

Re: Monoliths Are the Future

#331

Post by simonw »

MadWombat wrote: post #131
BrentOzar wrote: post #8

I'm a database guy, so the question I get from clients is, "We're thinking about breaking up our monolith into a bunch of microservices, and we want to use best-of-breed persistence layers for each microservice. Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting?" Analysts expect to be able to connect to one system, see their data, and write queries for it. They were neve…

> Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting? One of the key concepts in microservice architecture is data sovereignity. It doesn't matter how/where the data is stored. The only thing that cares about the details of the data storage is the service itself. If you need some data the service operates on for reporting purposes, make an API that gets you this data and…

A co-worker had a smart solution for this: your service's representation in a reporting system (a data warehouse for example) is part of its API. Your team should document it, and should ensure that when that representation changes information about the changes is available to the people who need to know it.

This really makes sense to me. I love the idea that part of a microservice team's responsibility is ensuring that a sensible subset of the data is copied over to the reporting systems in such a way that it can be used for analysis without risk of other teams writing queries that depend on undocumented internal details.

Re: Monoliths Are the Future

#332

Post by te_chris »

jlmorton wrote: post #301
satyrnein wrote: post #275

Earlier quoted context omitted.

If you are "export[ing] data in its raw, natural state" then haven't you lost the isolation benefits of microservices? Now you have external systems dependent on your implementation details, and changing your schema will break them.

That's a problem for the future data engineers to deal with. The data lake is an insurance policy so you only need to think about these problems if you later want the data. If you already know you want to analyze the data, then a data lake is not a good choice. Yes, it makes life harder for the data engineers in the future, but it might turn out that analysts only ever need 5% of the data in the lake, and dealing wit…

The data lake is also a real, live GDPR PII time bomb if you worked out how to get the data in but not take it out

Re: Monoliths Are the Future

#333

Post by timwaagh »

tootie wrote: post #241
therealdrag0 wrote: post #234

Earlier quoted context omitted.

My favorite project to work on was a modular monolith. It was a single deployable but each component (vertical) had its own maven-module.

When developers complain about being verbose and not a great language for coders, I counter that it doesn't exist to solve programming problems, but rather organizational problems. The killer feature that launched Java wasn't crap like checked exceptions, it was javadoc. Strict, self-documenting APIs are 10X more valuable than any intrinsic language feature.

i dont care about it not being great. its good enough as a language. but maven... but websphere... java is a hellish platform that ordinarily would not win from interpreted languages or those which focus on fast compilation. But it runs literally everywhere, including your toaster, but more importantly on mainframes which also run the real mvp aka Cobol. run once, run anywhere remains a killer feature no other platform has replicated as well.

Re: Monoliths Are the Future

#334

Post by Polylactic_acid »

jonahbenton wrote: post #271
MadWombat wrote: post #131

Earlier quoted context omitted.

> Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting? One of the key concepts in microservice architecture is data sovereignity. It doesn't matter how/where the data is stored. The only thing that cares about the details of the data storage is the service itself. If you need some data the service operates on for reporting purposes, make an API that gets you this data and…

Sorry, but "making an API that gets you this data" is the wrong answer. Most APIs are glorified wrappers around individual record-level operations like- get me this user- or constrained searches that return a portion of the data, maybe paginated. Reporting needs to see all the data. This is a completely different query and service delivery pattern. What happens to your API service written in a memory managed/garbage-…

>It is also true that analysts should not be given access to service databases, because the schema and semantics are likely to change out from under them.

Its not perfect but what we do is create a bunch of table views that represent each of the core data types in the system. We can then do all of the complex joins to collect the data analysts want in to an easy to query table as well as trying to keep the views consistent even as the db changes.

Re: Monoliths Are the Future

#335

Post by hnick »

satyrnein wrote: post #275
jlmorton wrote: post #255

Earlier quoted context omitted.

> Spend 12 months building ETLs Okay, sounds reasonable enough for a complex enterprise. > to feed a torrent of raw data to your lake Well, there's the problem. Why is it taking a year to export data in its raw, natural state? The entire point of a data lake is that there is no transformation of the data. There's no need to verify the data is accurate. There's no need to make sure it's performant. It's just data expo…

If you are "export[ing] data in its raw, natural state" then haven't you lost the isolation benefits of microservices? Now you have external systems dependent on your implementation details, and changing your schema will break them.

You also lose benefits of microservices if they have to stop and change the data exporting system all the time too, slowing down development.

The main benefit of a dumb copy is that the production service is not impacted by reporting, only a copy is. This relates to performance (large queries) but also implementation time.

Re: Monoliths Are the Future

#336

Post by disintegore »

MockObject wrote: post #73
eof wrote: post #40

As someone in the middle of destroying a monolith I hope the title is not true. Of course distributed-yet-still-deeply-coupled systems are possible they are at least harder to create. For me though the most important thing is grokability. Our monolith is to a point literally no one on earth can understand the whole thing. Even if the system is complex, the individual deployables being fully understood by some number…

> For me though the most important thing is grokability. Our monolith is to a point literally no one on earth can understand the whole thing. Well, then you're really going to love it when the concerns are spread across different codebases connected by APIs!

Introducing boundaries between problem domains should drastically reduce cross-cutting concerns. It also makes issues easier to find.

Not that this isn't possible in a "well engineered" monolithic system, but design constraints are usually better than hoping for engineering discipline.

Re: Monoliths Are the Future

#337

Post by cc81 »

LennyWhiteJr wrote: post #4

One thing a microservice architecture does really well is enforce bounded contexts. Oh you want to access that data? Well you need to go through the public API because it exists in a separate process. In a monolith it's all too easy to just 'call this piece of code and grab what I need' (no one will ever know). Project isolation can help but it's not a silver bullet. The author makes good points though, there are man…

That is true and when you are at big enough scale it probably works.

But when you have chosen a cool new microservice architecture for your team to implement and you grab that small user story that spans 3-4 different services things suddenly went from. "Hey, easy implementation and refactor and the compiler will tell me if I fucked up" to something much more time consuming and error prone.

In an ideal world that would not happen of course. Just like it in an ideal world a monolith is built correctly as well.

Re: Monoliths Are the Future

#338

Post by markbnj »

I often agree with Kelsey Hightower, but there are so many things he doesn't mention here. For example, being able to independently deploy components frees up certain kinds of development worklows. Distributed components also scale and fail independently, and you can use nifty things like message queues between them to provide resilience and soak up load spikes. I'm sure the pattern has often been applied in the wrong use cases, and that many people have over-applied it, but "the monolith is the future" seems just as wrong as "microservices are the future." We are nowhere near the size of a large bank... or even a small bank, and yet we've benefited from a distributed set of independently deployable and scalable components. You can call them microservices, or not. I can think of ways we could restructure on a monolithic backend, but just noodling on the idea leaves me with more constraints than benefits. Idk, it's a thought-provoking statement at least, but I sort of wish we'd stop reacting to fads with anti-fads.

Re: Monoliths Are the Future

#339

Post by edw »

lostmsu wrote: post #295
tootie wrote: post #214

Earlier quoted context omitted.

This is what gave rise to data lakes. The typical data lake maturity model I see in enterprise is: 1. Pay a ton of money to Microsoft for Azure Data Lake, Power BI, etc. 2. Spend 12 months building ETLs from all your microservices to feed a torrent of raw data to your lake. 3. Start to think about what KPIs you want to measure. 4. Sign up for a free Google Analytics account and use that instead.

What does Google Analytics have to do with datalakes? Are you talking about some specific scenario?

Google Analytics is associated with gleaning useful, actionable insights from your users' behavior on your web sites and in your apps, which was what the guy who sold you on the concept of a data lake was promising.

Re: Monoliths Are the Future

#340

Post by majormajor »

jayd16 wrote: post #268
StavrosK wrote: post #193

Earlier quoted context omitted.

You're saying "monoliths encourage unhealthy engineering" and then in the next sentence say "when executed correctly" for microservices. That sounds like a having/eating cake type situation.

Not exactly. It's hard to tell from the outside if a monolith was architecture well or is about the fall over. In a microservice architecture it's harder to pretend you're doing it right.

> In a microservice architecture it's harder to pretend you're doing it right.

After seeing a few of them, I'd say: "it's less embarrassingly obvious that you're doing it wrong."

But dig into the code for a few endpoints and it usually don't take long to find the crazy spaghetti and the poorly-carved-out separation of responsibilities breaches.

Post reply on HN