Monoliths Are the Future • Hacker News Board

Re: Monoliths Are the Future

#271

Post by jonahbenton » Thu, Jan 30, 2020, 9:31 PM UTC

MadWombat wrote: post #131

BrentOzar wrote: post #8

I'm a database guy, so the question I get from clients is, "We're thinking about breaking up our monolith into a bunch of microservices, and we want to use best-of-breed persistence layers for each microservice. Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting?" Analysts expect to be able to connect to one system, see their data, and write queries for it. They were neve…

> Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting? One of the key concepts in microservice architecture is data sovereignity. It doesn't matter how/where the data is stored. The only thing that cares about the details of the data storage is the service itself. If you need some data the service operates on for reporting purposes, make an API that gets you this data and…

Sorry, but "making an API that gets you this data" is the wrong answer.

Most APIs are glorified wrappers around individual record-level operations like- get me this user- or constrained searches that return a portion of the data, maybe paginated. Reporting needs to see all the data. This is a completely different query and service delivery pattern.

What happens to your API service written in a memory managed/garbage-collected language when you ask it to pull all the data from its bespoke database, pass it through its memory space, then send it back down the caller? It goes into GC hell, is what.

What happens when your API service when it issues queries for a consistent view of all the data and winds up forcing the database to lock tables? It stops working for users, is what.

There are so many ways to fail when your microservice starts pretending it is a database. It is not. Databases are dedicated services, not libraries, for a reason.

It is also true that analysts should not be given access to service databases, because the schema and semantics are likely to change out from under them.

The least bad solution? The engineering team is responsible for delivering either semantic events or doing the batch transformation themselves into a model that the data team can consume. It's a data delivery format, not an API.

Re: Monoliths Are the Future

#272

Post by jonathannorris » Thu, Jan 30, 2020, 9:31 PM UTC

The problem he is describing is solved with a Mono-repo, not a Monolith.

We've seen great success with a Mono-repo that enables the sharing of code across Micro-services, and enforcement of code and deployment processes.

Re: Monoliths Are the Future

#273

Post by StavrosK » Thu, Jan 30, 2020, 9:32 PM UTC

jayd16 wrote: post #268

StavrosK wrote: post #193

Earlier quoted context omitted.

You're saying "monoliths encourage unhealthy engineering" and then in the next sentence say "when executed correctly" for microservices. That sounds like a having/eating cake type situation.

Not exactly. It's hard to tell from the outside if a monolith was architecture well or is about the fall over. In a microservice architecture it's harder to pretend you're doing it right.

I disagree, "doing it wrong" just looks different there.

Re: Monoliths Are the Future

#274

Post by [deleted] » Thu, Jan 30, 2020, 9:36 PM UTC

NicoJuicy wrote: post #166

echelon wrote: post #140

Earlier quoted context omitted.

> But you do not simply go and poke your reporting fingers into individual service databases. This is why I distrust all of the monolith folks. Yes, it's easier to get your data, but in the long run you create unmaintainable spaghetti that can't ever change without breaking things you can't easily surface. Monoliths are undisciplined and encourage unhealthy and unsustainable engineering. Microservices enforce separat…

DDD enforces seperation also. It's about code quality, microservices are easy replaceable. Modules are too. With both systems, the core part ( eg. mesh, Infrastructure, ... ) Is crucial. I think experienced developers can see this, the ones that actually delivered products and had big code changes. The ones that handled their "legacy" code. Microservices are just a way to enforce it, there are others. None are perfec…

[deleted]

Re: Monoliths Are the Future

#275

Post by satyrnein » Thu, Jan 30, 2020, 9:37 PM UTC

jlmorton wrote: post #255

tootie wrote: post #214

Earlier quoted context omitted.

This is what gave rise to data lakes. The typical data lake maturity model I see in enterprise is: 1. Pay a ton of money to Microsoft for Azure Data Lake, Power BI, etc. 2. Spend 12 months building ETLs from all your microservices to feed a torrent of raw data to your lake. 3. Start to think about what KPIs you want to measure. 4. Sign up for a free Google Analytics account and use that instead.

> Spend 12 months building ETLs Okay, sounds reasonable enough for a complex enterprise. > to feed a torrent of raw data to your lake Well, there's the problem. Why is it taking a year to export data in its raw, natural state? The entire point of a data lake is that there is no transformation of the data. There's no need to verify the data is accurate. There's no need to make sure it's performant. It's just data expo…

If you are "export[ing] data in its raw, natural state" then haven't you lost the isolation benefits of microservices? Now you have external systems dependent on your implementation details, and changing your schema will break them.

Re: Monoliths Are the Future

#276

Post by daxfohl » Thu, Jan 30, 2020, 9:38 PM UTC

keanebean86 wrote: post #209

tynpeddler wrote: post #181

There are two big reasons to go to microservices (note that the exact definition of microservice can vary a lot). 1. Organizational streamlining. If the team working on the monolith becomes to large, then coordinating and pushing out changes quickly can become incredibly difficult. One rule of thumb I've heard is the two pizzas rule. If two pizzas can't feed the team working on a system, it's time to break up the sys…

I have this idea for a new framework/language. I'm sure if it either already exists or it's a dumb idea in practice but anyways. You build a monolithic application. Everyone works on the same code base. Things are broken up into modules/classes/packages. From the programmers point of view it's just like working on a standard Java project or something similar. The magic happens at the method and module boundaries. Whe…

I've gone through this thought process too, but where I arrived is that it's kind of the tail wagging the dog. The whole point of microservices (or at least one major one) is so that teams can operate independently without having to use the same framework. Each service provides a fairly static API and other services write code against it, but under the covers each service is relatively free to implement that contract however they want, whatever language, whatever database, whatever scaling strategy, whatever version control, whatever deployment cadence, whatever hardware SKUs, all specific to the service.

So with a hypothetical framework like this, yeah it would make microservices easier (in theory, though there are lots of technical problems too) in terms of "look ma, I made microservices", but it wouldn't actually address any of the problem microservice architectures actually try to solve. So, tail wagging the dog.

Some of the technical problems stem from memory working different from API calls: they can fail, the overhead is much higher (shouldn't call in a big loop), can't pass pointers, global state may differ on remote machine. So an application model that tries to abstract away those differences is bound to have problems. Also deployment: remotes will be temporarily broken if the API changes, until the deployment has fully propagated; a team running a microservice takes a different mindset with respect to API versioning than does a compilation unit. And that's really the crux of the whole thing: microservicing requires an entirely different mindset than monolithing, and approaching one with the mindset of another will cause problems.

Re: Monoliths Are the Future

#277

Post by cleong » Thu, Jan 30, 2020, 9:39 PM UTC

segmondy wrote: post #198

derision wrote: post #119

Earlier quoted context omitted.

"data pipeline" is just the new trendy phrase for ETL, which the GP mentioned. just because a solution exists, does not make it a solved problem. it's not the right solution for everyone

no it's not. ETL is really database focused and batch focused , Extract, Transform, Load. Data pipeline, is a combination of streams and batch. For example, you can implement a Data Capture using something like https://debezium.io/ Here's how Netflix solves it https://netflixtechblog.com/dblog-a-generic-change-data-capt... Overview "Change-Data-Capture (CDC) allows capturing committed changes from a database in real-…

What your saying applys more to ELT, which is an efficient method when using batches. There's no reason why you can't stream ETL.

Re: Monoliths Are the Future

#278

Post by hello_moto » Thu, Jan 30, 2020, 9:40 PM UTC

willvarfar wrote: post #190

BrentOzar wrote: post #8

I'm a database guy, so the question I get from clients is, "We're thinking about breaking up our monolith into a bunch of microservices, and we want to use best-of-breed persistence layers for each microservice. Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting?" Analysts expect to be able to connect to one system, see their data, and write queries for it. They were neve…

https://prestosql.io/ It can access all those different databases. You can also make your own connectors that make your services appear as tables, which you can query with SQL in the normal way. So if the new accounts micro-service doesn't have a database, or the team won't let your analysts access the database behind it, you can always go in through the front-door e.g. the rest/graphql/grpc/thrift/buzzword api it ex…

Presto can't deal with ElasticSearch. It can query basic data but it's not optimized to translate SQL to native ES query (SQL ES query is for paying customer).

Re: Monoliths Are the Future

#279

Post by arkh » Thu, Jan 30, 2020, 9:42 PM UTC

pjmlp wrote: post #206

bernawil wrote: post #177

Earlier quoted context omitted.

It's like a museum of problems. Waves of solutions that became problems after. The culture of code generation before the culture of XML configuration.

I love my XML, with comments and static type validation, making my graphical tooling and IDE based editing quite comfortable, while observing YAML and JSON formats catching up on XML features. They are there for a reason.

The joys of JSONPath. Its documentation. Its implementations. Its capabilities.

I'm waiting for more fun with JSONSLT the day someone has the idea to make a first limited and poorly documented implementation.

Re: Monoliths Are the Future

#280

Post by tflinton » Thu, Jan 30, 2020, 9:43 PM UTC

Distributed monoliths (or micro services) do have some advantages:

1. Easier for users to see "who owns what" (albeit a module pattern could fix this as well).

2. Different hardware resources or scaling for different parts of a monolith really isn't possible. If one module requires 16GB then everytime you scale a horizontally you must have at least 16GB, you're at the mercy of your worst module in the monolith.

3. Deploys are very difficult, and as you scale to over 10 developers increasingly becomes difficult to push up (it takes one persons bad commit to hold everyone in the organization from deploying).

4. Security boundaries are easier to define, each "module" in a monolith effectively has access to all resources for all modules.

5. Poly-languages are easier, albeit depending on the base language, you could do a lot of transpiling on a monolith but.. ew.

6. HTTP status codes and request paths can give you a clear view of how calls are happening in your system; in a monolith you'll only get stack traces generally on errors, not on successes, usually you need to invest more in static analysis and APM stuff for a monolith.

7. Microservices can be cheaper when you scale, you don't have the GCD of memory/CPU/disk requirements as you do in a monolith.

8. GCD of implementation details, if one request requires a sticky session, all of your requests require stick sessions...

9. More complex and long builds, most monoliths have component-based hot reloads, but even those can take 30s to a minute in my experience, and a full build, that's at least 20.

10. Harder to unit test, this can vary by language but without clear boundaries and resource definitions monoliths can be very tricky to unit test, microservices/distributed monoliths are inherently smaller with clearly declared resources so it becomes easier to find where and how data flows in them.