Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monoliths Are the Future

changelog.com

Post reply on HN
Print view View original

301–310 of 567 posts

Re: Monoliths Are the Future

#301

Post by jlmorton »

satyrnein wrote: post #275
jlmorton wrote: post #255

Earlier quoted context omitted.

> Spend 12 months building ETLs Okay, sounds reasonable enough for a complex enterprise. > to feed a torrent of raw data to your lake Well, there's the problem. Why is it taking a year to export data in its raw, natural state? The entire point of a data lake is that there is no transformation of the data. There's no need to verify the data is accurate. There's no need to make sure it's performant. It's just data expo…

If you are "export[ing] data in its raw, natural state" then haven't you lost the isolation benefits of microservices? Now you have external systems dependent on your implementation details, and changing your schema will break them.

That's a problem for the future data engineers to deal with. The data lake is an insurance policy so you only need to think about these problems if you later want the data. If you already know you want to analyze the data, then a data lake is not a good choice.

Yes, it makes life harder for the data engineers in the future, but it might turn out that analysts only ever need 5% of the data in the lake, and dealing with these schema changes for 5% of the data is easier than carefully planning a public schema for 100% of it.

It can be helpful to include some small amount of metadata in the export though, with things like the source system name, date & time, # of records, and a schema version. Schema version could easily be the latest migration revision, or something like that.

Re: Monoliths Are the Future

#302

Post by skybrian »

BrentOzar wrote: post #8

I'm a database guy, so the question I get from clients is, "We're thinking about breaking up our monolith into a bunch of microservices, and we want to use best-of-breed persistence layers for each microservice. Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting?" Analysts expect to be able to connect to one system, see their data, and write queries for it. They were neve…

It seems like interacting with customers and enforcing business rules is one job, and observing what's happening is a different concern. Observing means collecting a lot of logs to a reporting database.

Re: Monoliths Are the Future

#303

Post by oneplane »

cogman10 wrote: post #236
Swinx43 wrote: post #191

Earlier quoted context omitted.

I came here simply to echo this statement! Design a reporting solution that is responsible for ingesting data from these micro services' persistence layers. Analysts should only ever be querying this reporting solution and should not be allowed to connect directly to any micro service persistence layer or API. We have a whole industry around Analytics and Data and the tools and processes to build this reporting layer…

This is seriously why my company still has monoliths. Our databases are open to way too many people. What's worse, they are multi tenant making refactoring really hard.

Having more than one schema owner is practically a death sentence for development and engineering...

We used to have a few of those, especially on exadata clusters. Finally carted them out of the local dc after moving to RDS Aurora databases with strict policies. Might have caused 3 or 4 people to quit, but totally worth it for the 500+ people that stayed who now can own their data, schema and development (and be held responsible for it! -- another issue of multi-db-access, it's always someone else's fault). Went from deploying once a day with a 'heads up' message to no-message deploying multiple times per hour.

Re: Monoliths Are the Future

#304

Post by abhishekjha »

jonahbenton wrote: post #271
MadWombat wrote: post #131

Earlier quoted context omitted.

> Some data belongs in Postgres, some in DynamoDB, some in JSON files. Now, how do we do reporting? One of the key concepts in microservice architecture is data sovereignity. It doesn't matter how/where the data is stored. The only thing that cares about the details of the data storage is the service itself. If you need some data the service operates on for reporting purposes, make an API that gets you this data and…

Sorry, but "making an API that gets you this data" is the wrong answer. Most APIs are glorified wrappers around individual record-level operations like- get me this user- or constrained searches that return a portion of the data, maybe paginated. Reporting needs to see all the data. This is a completely different query and service delivery pattern. What happens to your API service written in a memory managed/garbage-…

>It goes into GC hell

Can you exapand on this a little? Or a paper that I can read?

Re: Monoliths Are the Future

#305

Post by WoahNoun »

geggam wrote: post #258
WoahNoun wrote: post #201

Earlier quoted context omitted.

AWS data pipelines and AWS lambda. It's cloud services the whole way down. https://aws.amazon.com/datapipeline/

I saved a company 20k a month by creating a job server in AWS. Lambda isnt cheap when you start using it hard

Lambda is mostly used for it's trigger functionality for data or artifacts that are created at irregular intervals. Eg, an object is uploaded to s3 which triggers a lambda which runs a copy command for that object into redshift. The kind of stuff that's well below the threshold for leaving the free tier.

Re: Monoliths Are the Future

#306

Post by jorblumesea »

tynpeddler wrote: post #181

There are two big reasons to go to microservices (note that the exact definition of microservice can vary a lot). 1. Organizational streamlining. If the team working on the monolith becomes to large, then coordinating and pushing out changes quickly can become incredibly difficult. One rule of thumb I've heard is the two pizzas rule. If two pizzas can't feed the team working on a system, it's time to break up the sys…

I would also add cost savings and ops excellence. Having individual business processes broken up into separate services can often lead to being able to tune individual services and allocate just the resources it needs. Especially when used with containers. It's also easier to spot offending commits.

A monolith is hard to tune and often ends up being a money pit.

Re: Monoliths Are the Future

#307

Post by mikepk »

I'm tempted to write a blog post... I bristle a little when microservices are described as "best practice". Monolith vs microservice is really about _people_ and _organizations_. Monoliths make sense in some contexts and microservices in others, but the deciding factor is really the size of the team and number of people working on different functional contexts.

The best analog I can come up with is monoliths in larger organizations are like a manifestation of Amdahl's law. The overhead of communication and synchronization reduces your development throughput. Each additional person does not add one persons worth of throughput when you cross a critical individual count threshold (mythical man month and all that).

I'm not describing this clearly so I should probably actually commit to writing out my thoughts on this in a post describing my experience with this.

Re: Monoliths Are the Future

#308

Post by pmoleri »

nitwit005 wrote: post #290
tootie wrote: post #220

Earlier quoted context omitted.

The solution to this is just writing modular code and using an artifact repository. It's a model I've rarely seen attempted even though it's much easier than microservices and serves the same purpose. You can have individual dev teams, with their own repo ,backlogs, own stakeholders, etc all working at their own paces. They build modules (jars, nuget packages, npm modules) and deploy semver versioned artifacts to a r…

My previous company went this direction. I wouldn't recommend it. Say you version each module and pull in specified versions. It'll work fine, right up until two modules both try to pull different versions of a third module. In practice, you have to update multiple modules at once to avoid conflicts, which, in turn, can require updating other team's code. It also turns out some tools like Maven don't prevent conflict…

That's what semver is for, right? Breaking changes go in a major, same major means that the latest is always compatible. You'll have the same issue with microservices if you introduce breaking changes.

Re: Monoliths Are the Future

#309

Post by p1necone »

sequoia wrote: post #296
cheese4242 wrote: post #239

Earlier quoted context omitted.

This is an incredible overreaction to an entirely innocuous comment.

If you think telling colleagues not to "simply go and poke your reporting fingers into" things won't insult them or put them on a defensive footing, I encourage you to try it and closely note the reception you receive. In my experience, people do not appreciate being spoken to like that.

They didn't tell their colleagues to do that, they made a slightly humorous comment on a hacker news thread.
Post reply on HN