Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monoliths Are the Future

changelog.com

Post reply on HN
Print view View original

241–250 of 567 posts

Re: Monoliths Are the Future

#241

Post by tootie »

therealdrag0 wrote: post #234
tootie wrote: post #220

Earlier quoted context omitted.

The solution to this is just writing modular code and using an artifact repository. It's a model I've rarely seen attempted even though it's much easier than microservices and serves the same purpose. You can have individual dev teams, with their own repo ,backlogs, own stakeholders, etc all working at their own paces. They build modules (jars, nuget packages, npm modules) and deploy semver versioned artifacts to a r…

My favorite project to work on was a modular monolith. It was a single deployable but each component (vertical) had its own maven-module.

When developers complain about being verbose and not a great language for coders, I counter that it doesn't exist to solve programming problems, but rather organizational problems. The killer feature that launched Java wasn't crap like checked exceptions, it was javadoc. Strict, self-documenting APIs are 10X more valuable than any intrinsic language feature.

Re: Monoliths Are the Future

#242

Post by mrits »

MadWombat wrote: post #131
BrentOzar wrote: post #8

I'm a database guy, so the question I get from clients is, "We're thinking about breaking up our monolith into a bunch of microservices, and we want to use best-of-breed persistence layers for each microservice. Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting?" Analysts expect to be able to connect to one system, see their data, and write queries for it. They were neve…

> Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting? One of the key concepts in microservice architecture is data sovereignity. It doesn't matter how/where the data is stored. The only thing that cares about the details of the data storage is the service itself. If you need some data the service operates on for reporting purposes, make an API that gets you this data and…

What you are describing certainly isn't unique to microservices.

Re: Monoliths Are the Future

#243

Post by chrisjc »

gunnarmorling wrote: post #170
rjkennedy98 wrote: post #19

Earlier quoted context omitted.

This is what Kafka is for. You put Kafka on top of your database to expose data and events. Now BI can take the events put them into their system as they want.

Shameless plug: feeding data from operational databases to DWH is one main use case of change data capture as e.g. implemented by Debezium ( https://debezium.io/ ) for MySQL, Postgres, MongoDB, and more, using Apache Kafka as the messaging layer. Disclaimer: working on Debezium

Data Warehouses are (usually) not optimized for individual inserts, updates, deletes (DMLs). Loading data into DWHs is usually done through copy/ingestion commands where aggregated/batched data is copied from blob storage (s3, azure, etc...) to a staging table in the DWH.

In a non-append-only scenario, Debezium tracks each source operation (insert, update, delete) from the replication log (oplog, binlog, etc) as an individual operation that's emitted into a kafka topic. How does one efficiently replicate this to a Data Warehouse in an efficient manner?

I have not been able to use Debezium as way to replicate to a Data Warehouse for this very reason. At least not without having to resort to very complicated data warehouse merge strategies.

Note, there exist Data Warehouses that allow tables to be created in either OLAP or OLTP flavor. I understand that Debezium could easily replicate to an OLTP staging table. But are there any solutions if this isn't an option?

Re: Monoliths Are the Future

#244

Post by mstump »

I call B.S.. He's conflating two things. One is a deployment infrastructure, another is an application pattern. I can deploy a monolith in k8s and I can deploy micro-services on a single server or a fleet of on-prem servers using any of the legacy dev-ops deployment automation tools. Sure k8s makes doing micro-services easier because it’s got a lot of the raw building blocks necessary to handle the CD process but in no way are the two concepts related. They’re tackling different problems. He's making the same mistake that he's criticizing others for by equating the two. What he is really describing is an organization doing a “big bang” refactor with poor planning and execution.

Re: Monoliths Are the Future

#245

Post by dmitrig01 »

joefkelley wrote: post #212
willvarfar wrote: post #190

Earlier quoted context omitted.

https://prestosql.io/ It can access all those different databases. You can also make your own connectors that make your services appear as tables, which you can query with SQL in the normal way. So if the new accounts micro-service doesn't have a database, or the team won't let your analysts access the database behind it, you can always go in through the front-door e.g. the rest/graphql/grpc/thrift/buzzword api it ex…

It's been about 4 years since I've been in this world, but I remember there being several products all doing a very similar thing: Presto, Hive, SparkSQL, Impala, perhaps some more I'm forgetting. Is the situation still the same? Or has Presto "won out" in any sense?

Hive and Impala are databases.

Presto and SparkSQL are SQL interfaces to many different datasources, including Hive and Impala, but also any SQL database such as Postgres/Redis/etc, and many other types of databases, such as Cassandra and Redis; the SQL tools can query all these different types of databases with a unified SQL interface, and even do joins across them.

The difference between Presto and SparkSQL is that Presto is run on a multi-tenant cluster with automatic resource allocation. SparkSQL jobs tend to have to be allocated with a specific resource allocation ahead of time. This makes Presto is (in my experience) a little more user-friendly. On the other hand, SparkSQL has better support for writing data to different datasources, whereas Presto pretty much only supports collecting results from a client or writing data into Hive.

Re: Monoliths Are the Future

#246

Post by [deleted] »

echelon wrote: post #140
MadWombat wrote: post #131

Earlier quoted context omitted.

> Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting? One of the key concepts in microservice architecture is data sovereignity. It doesn't matter how/where the data is stored. The only thing that cares about the details of the data storage is the service itself. If you need some data the service operates on for reporting purposes, make an API that gets you this data and…

> But you do not simply go and poke your reporting fingers into individual service databases. This is why I distrust all of the monolith folks. Yes, it's easier to get your data, but in the long run you create unmaintainable spaghetti that can't ever change without breaking things you can't easily surface. Monoliths are undisciplined and encourage unhealthy and unsustainable engineering. Microservices enforce separat…

[deleted]

Re: Monoliths Are the Future

#247

Post by tsimionescu »

mypalmike wrote: post #210
simiones wrote: post #135

Earlier quoted context omitted.

My company is currently moving to microservices, but for different reasons. The problem you raise was in fact fixed years ago in our org simply by properly decoupling our "monolithic" app into modules. The top-level build was simply a collection of all pre-built modules. After each dependency update, an automated regression would run, and if pass rate was less than X%, the change was not pushed upstream. Really, micr…

Modules help some, but being tied to a deployment cadence with say, hundreds of other developers, can be very painful. Having to delay my team's features because another team broke the build doesn't help my customers. Isolating CI pipelines by team can help that somewhat, but you still end up sharing fate with teams with whom you might have no real interdependencies.

I guess this also depends a lot on the type of product you build. For me, we are building a piece of software that we ship to customers, so the release cadence is limited by many factors outside engineering anyway.

But I can see how on a live service-type product, microservices would help a lot more on this front.

Re: Monoliths Are the Future

#248

Post by ukyrgf »

sequoia wrote: post #189
MadWombat wrote: post #131

Earlier quoted context omitted.

> Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting? One of the key concepts in microservice architecture is data sovereignity. It doesn't matter how/where the data is stored. The only thing that cares about the details of the data storage is the service itself. If you need some data the service operates on for reporting purposes, make an API that gets you this data and…

> you do not simply go and poke your reporting fingers into individual service databases Side point: This is a needlessly hostile and unprofessional way to refer to a colleague. Remember that you and the reporting/analytics people at your company are working towards the same goals (the company's business goals). You are collaborators, not combatants. You can express your same point by saying something like "The habit…

I think I prefer the poking around analogy. I can immediately visualize why it's bad, and it doesn't have the word "tantamount".

Re: Monoliths Are the Future

#249

Post by geggam »

rriepe wrote: post #202
StavrosK wrote: post #193

Earlier quoted context omitted.

You're saying "monoliths encourage unhealthy engineering" and then in the next sentence say "when executed correctly" for microservices. That sounds like a having/eating cake type situation.

To be fair, this is how I've seen tech decisions presented at most big tech companies.

Quit talking about what is behind the curtain
Post reply on HN