Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monoliths Are the Future

changelog.com

Post reply on HN
Print view View original

411–420 of 567 posts

Re: Monoliths Are the Future

#411

Post by JamesBarney »

satyrnein wrote: post #275
jlmorton wrote: post #255

Earlier quoted context omitted.

> Spend 12 months building ETLs Okay, sounds reasonable enough for a complex enterprise. > to feed a torrent of raw data to your lake Well, there's the problem. Why is it taking a year to export data in its raw, natural state? The entire point of a data lake is that there is no transformation of the data. There's no need to verify the data is accurate. There's no need to make sure it's performant. It's just data expo…

If you are "export[ing] data in its raw, natural state" then haven't you lost the isolation benefits of microservices? Now you have external systems dependent on your implementation details, and changing your schema will break them.

The key insight and definition of a data lake was pulling data and storing in it's raw form form across the enterprise.

https://en.wikipedia.org/wiki/Data_lake

The reasons for it are hard to explain succinctly in HN comment but if you look up data lake there will be a lot of explanation. But it basically comes down to "is it better for the data integrators or the data consumers to massage the data?" And data lakes was the insight it's really great when the consumers decide how to massage the data.

Re: Monoliths Are the Future

#412

Post by bcrosby95 »

ahnick wrote: post #370
bcrosby95 wrote: post #282

Earlier quoted context omitted.

Why exactly do you want to independently scale a hot code path? If its hot, its already using most of the resources in your monolith. Take your monolith, distribute it to more servers, and it will reduce the load of your other servers regardless of which parts of your code are causing the load.

Sure you can do that, but it's not as efficient as deploying only the code that is getting hit. Also, it might be taking up most of your resources, but it might not be as well. For example, let's say in an eCommerce application that the shipping calculator is getting hit a lot. You'd like to be able to scale this independently as a service, so you can handle all the requests without also having to replicate all of th…

> For example, let's say in an eCommerce application that the shipping calculator is getting hit a lot. You'd like to be able to scale this independently as a service, so you can handle all the requests without also having to replicate all of the other resources, such as the cart persistence, user sessions, etc. that are a lot more memory intensive.

Assuming you allocate different resources for it. If you're using the same instance type for all your microservices, you aren't benefiting from this. In fact, you're paying for resources you aren't using.

You might even be paying more - allocating high memory instances to high memory services, and regular instances to low memory services that don't use that memory. You might be able to get by with only regular instances if you distributed your services in a monolithic fashion.

In my experience most small teams I've seen aren't that specific with their resources. Unless something is obviously super high memory, like a cache, they tend to just use default instances.

Re: Monoliths Are the Future

#413

Post by austincheney »

ritchiea wrote: post #397
austincheney wrote: post #121

Earlier quoted context omitted.

He/She cannot do that, at least at their employer, because people would throw up at the sight of the code and cry until its rewritten. As an example slip some DOM methods into your code and watch people go into convulsions like an angry zombie on cocaine.

There are very real reasons not to use DOM methods assuming you're using a front end framework. Not only is the coding style more imperative than say React, but frontend frameworks have been designed to efficiently update the DOM and have diffing algorithms that can check for necessary state changes. Is there really a compelling reason for writing your own DOM manipulation in a sizeable frontend codebase in 2020?

> Is there really a compelling reason for writing your own DOM manipulation in a sizeable frontend codebase in 2020?

It is several orders of magnitude faster. It’s also how I prefer to code because managing state isn’t challenging and I am not hopelessly paralyzed by imperative code.

Re: Monoliths Are the Future

#414

Post by mfDjB »

Perhaps there is a problem where people are splitting a perfectly good monolith into microservices, but I do wonder how do you deal with large scale machine learning without microservices? I am a rather small operation and I still have models taking several gigabytes worth of memory and ANN indexes of about the same size, which clearly couldn't operate in a monolith unless it was a massive machine, and even if it could not every request would necessitate such power. How does a dogmatic monolith approach solve these problems?

Re: Monoliths Are the Future

#415

Post by yesbabyyes »

te_chris wrote: post #332
jlmorton wrote: post #301

Earlier quoted context omitted.

That's a problem for the future data engineers to deal with. The data lake is an insurance policy so you only need to think about these problems if you later want the data. If you already know you want to analyze the data, then a data lake is not a good choice. Yes, it makes life harder for the data engineers in the future, but it might turn out that analysts only ever need 5% of the data in the lake, and dealing wit…

The data lake is also a real, live GDPR PII time bomb if you worked out how to get the data in but not take it out

What I think you'd typically do is put different data under different keys/paths, so that red is personally identifiable data, yellow contains pointers to such data, and green is just regular data. You could have a structure like s3://my-data-lake/{red|yellow|green}/{raw|intermediate}/year={year}/month={month}/day={day}/source={system}/dataset={table}

Then you just don't keep red data for longer than 30 days.

Re: Monoliths Are the Future

#416

Post by tristador »

JamesBarney wrote: post #401
yes_man wrote: post #374

Earlier quoted context omitted.

I don't want advocate one way or another (micro vs. monoliths) because tomato tomato. However here are a few arguments in defense of microservices regarding these three signs you commented: 1. Microservices do not have some inherent property of having to duplicate data. You can have data in single source and deliver that data to anyone who needs it through an API. There are infinitely many caching solutions if you ar…

You're right that micro-services avoid a lot of the pain of micro-services if they have one consolidated "Data service" that sits on top of their data repositories. But a micro-services architecture with a consolidated data service is similar to an airplane on the ground, it's true it can't fall out of the sky, but it's as useful as a car with awful gas mileage. Once you add in this consolidated data service, every o…

I'm not following. How do things work better for the non-microservice approach?

Re. teams: For any project above a certain size, you'll have teams. If that's a network boundary, a process boundary, or a library boundary doesn't change that you'll have multiple teams for a large project.

I'm not sure I get the resiliency point. I worked on a project where the dependent data service was offline painfully frequently. We used async tasks and caching to keep things running and were able to let the users do many tasks. For us our tool was still fairly useful when dependencies went down. If we used monolith then everything would be down, right? That doesn't sound better.

Re: Monoliths Are the Future

#417

Post by tomphoolery »

tootie wrote: post #220
closeparen wrote: post #57

My employer adopted microservices for a very specific reason: it became nearly impossible to deploy the monolith. With hundreds of commits trying to go out every day, probability that at least one would break something approached 1. Then everything had to be rolled back. Getting unrelated concerns into separate deployable artifacts rescued our velocity. It came with many of its own challenges, too! A great deal of in…

The solution to this is just writing modular code and using an artifact repository. It's a model I've rarely seen attempted even though it's much easier than microservices and serves the same purpose. You can have individual dev teams, with their own repo ,backlogs, own stakeholders, etc all working at their own paces. They build modules (jars, nuget packages, npm modules) and deploy semver versioned artifacts to a r…

My company does this in a sense. We build a (mostly) open-source eCommerce platform with almost 100 public repos and a lot more proprietary ones for larger retailers. It's not "microservices", but it's definitely a lot of packages to manage, and we have frequently had trouble keeping them all organized and in sync with the latest changes to the core platform. The core platform is designed internally with a modular, domain-driven architecture, and the applications made with the platforms are monoliths, but the features of the platform are distributed amongst a couple hundred RubyGems. It's a lot to keep track of, but it's still a great compromise between a modular, plug-and-play architecture, and a monolithic application with all those features installed at once (which would be really hard to maintain). We're looking for ways to reduce our workload in that sense, because our team is rather small (about 6 people) and as I was saying, it's a little difficult to manage all those repos. So we've been thinking about keeping the modular architecture but keeping everything in a monorepo, so it's easier to release new versions and figure out what's changed (or what needs to change).

Re: Monoliths Are the Future

#418

Post by balfirevic »

rumanator wrote: post #366
rubyn00bie wrote: post #347

I couldn't agree more with an article. Most people think a micro-service architecture is a panacea because "look at how simple X is," but it's not that simple. It's now a distributed system, and very likely, it's a the worst-of-the-worst a distributed monolith. Distributed system are hard, I know, I do it. Three signs you have a distributed monolith: 1. You're duplicating the tables (information), without transformin…

> Most people think a micro-service architecture is a panacea because "look at how simple X is," but it's not that simple. It's now a distributed system, and very likely, it's a the worst-of-the-worst a distributed monolith. Distributed system are hard, I know, I do it. This line of argument fails to take into consideration any of the reasons why in general microservices are the right tool for the right job. Yes, it'…

> What would you have to do to achieve the same goal with a monolith?

You would take Celery, get a message broker up and running and launch some worker instances.

Re: Monoliths Are the Future

#419

Post by yesbabyyes »

tootie wrote: post #214
BrentOzar wrote: post #8

I'm a database guy, so the question I get from clients is, "We're thinking about breaking up our monolith into a bunch of microservices, and we want to use best-of-breed persistence layers for each microservice. Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting?" Analysts expect to be able to connect to one system, see their data, and write queries for it. They were neve…

This is what gave rise to data lakes. The typical data lake maturity model I see in enterprise is: 1. Pay a ton of money to Microsoft for Azure Data Lake, Power BI, etc. 2. Spend 12 months building ETLs from all your microservices to feed a torrent of raw data to your lake. 3. Start to think about what KPIs you want to measure. 4. Sign up for a free Google Analytics account and use that instead.

Where I was a bit more than a year ago they hired a real expensive consultant who did a data lake project which wasn't finished by the time my team were told to use it, so we rolled our own according to that team's instructions and best practices (best practices were a huge deal at this company, everything we did was best practices). We exported data s3 under a particular structure and I built an ETL system around that and spotify's Luigi. Noone else on my team knew what ETL was, which made me feel old. We spent two, maybe three months on this. The BI team got their data and so did the marketing automation team.

But yeah, it's funny how these projects get complicated in larger organizations. Personally I would have rolled something even simpler on gnu/posix tools and scripts, in rather less than a month.

Re: Monoliths Are the Future

#420

Post by franciscop »

mfDjB wrote: post #414

Perhaps there is a problem where people are splitting a perfectly good monolith into microservices, but I do wonder how do you deal with large scale machine learning without microservices? I am a rather small operation and I still have models taking several gigabytes worth of memory and ANN indexes of about the same size, which clearly couldn't operate in a monolith unless it was a massive machine, and even if it cou…

Is there any problem with separating the ML into a different service/machine, and everything else together? Then you can treat the ML in the same way you treat an external service, or your DB or Redis (if external). While no longer a pure monolith that certainly doesn't qualify as a microservices architecture.

Note: no idea what I'm talking about, I'm genuinely curious if that's a valid solution.

Post reply on HN