Monoliths Are the Future • Hacker News Board

Re: Monoliths Are the Future

#461

Post by zerd » Fri, Jan 31, 2020, 4:14 AM UTC

WoahNoun wrote: post #142

dfabulich wrote: post #139

Earlier quoted context omitted.

Cloud computing helps here, but microservices still make this harder. Some of the data is in Dynamo, some of it is in Aurora, some of it is in MySQL RDS, some of it is in S3, and nobody knows where all of it is at once.

From a project management perspective, each data source should have some requirements behind it from the business team. Those requirements should be prioritized meaning you can prioritize which data source to tackle first. You automate the process in AWS data pipelines for that data source, write the documentation for the next analyst, and move on to the next data source. The complexity you and the OP seem to be desc…

Also one of the service teams might need to change their schema, which the reporting team needs to adjust their process to handle that. That's fine, but they need to know that in advance, and then they might have a backlog of other things that they need to do, and then some other teams schema changed without notice, so now they always have to play catch-up.

Re: Monoliths Are the Future

#462

Post by tootie » Fri, Jan 31, 2020, 4:26 AM UTC

lostmsu wrote: post #295

tootie wrote: post #214

Earlier quoted context omitted.

This is what gave rise to data lakes. The typical data lake maturity model I see in enterprise is: 1. Pay a ton of money to Microsoft for Azure Data Lake, Power BI, etc. 2. Spend 12 months building ETLs from all your microservices to feed a torrent of raw data to your lake. 3. Start to think about what KPIs you want to measure. 4. Sign up for a free Google Analytics account and use that instead.

What does Google Analytics have to do with datalakes? Are you talking about some specific scenario?

I'm mostly being sarcastic but I'm partly describing an exact scenario that I'm witnessing right now. Business wants "analytics". IT starts spending a load of money. Business has no idea what it's for and buys their own thing to just track website visits which is what they wanted.

Re: Monoliths Are the Future

#463

Post by zerd » Fri, Jan 31, 2020, 4:37 AM UTC

default-kramer wrote: post #372

projektfu wrote: post #130

Earlier quoted context omitted.

It's a little sad because originally, people thought there would be a shared data base (now one word) for the whole organization. Data administrators would write rules for the data as a whole and keep applications in line so that they operated on that data base appropriately. A lot of DBMS features are meant to support this concept of shared use by diverse applications. What ended up happening is each application use…

Do you know why the shared database vision didn't work out? Because I still think it would be the best approach for many companies. Most companies are small enough that they could spend less than $10k/month for an extremely powerful cloud DB. Then you could replace most microservices with views or stored procs. What could be simpler? I think one reason to avoid this approach is because SQL and other DB languages are…

I've worked on a service like this. 800k lines of PL/SQL and Java stored procesures (running inside the database, so you could call Java from PL/SQL and vice versa), powered by triggers.

* Testing is god-awful. To test a simple thing you had to know how the whole application worked, because there's validation in triggers, which triggers other triggers, which require things to be in a certain state. This made refactoring really hard/risky so it rarely got done.

* There's a performance ceiling, and when you hit that, you're done. We did hit a ceiling, did months of performance tuning, then upgraded to the biggest available box at the time, 96 cores, 2TB ram, which helped, but next time the upgrade won't be big enough. You're limited in what one box can do (and due to stored procedures being tied to the transaction there's limits to what you can do concurrently as well)

Re: Monoliths Are the Future

#464

Post by tonfreed » Fri, Jan 31, 2020, 4:38 AM UTC

Yep. I call this resume oriented development, which I think is in the Gotime podcast he links at the bottom here. Martin Fowler wrote a good article about the same thing https://www.martinfowler.com/bliki/MonolithFirst.html

tl;dr: If you don't understand the problem domain, build a monolith following sensible engineering principles to get going ASAP and then split it out when you understand where the functional lines actually are.

Re: Monoliths Are the Future

#465

Post by averros » Fri, Jan 31, 2020, 4:42 AM UTC

WoahNoun wrote: post #316

satyrnein wrote: post #275

Earlier quoted context omitted.

If you are "export[ing] data in its raw, natural state" then haven't you lost the isolation benefits of microservices? Now you have external systems dependent on your implementation details, and changing your schema will break them.

Changing the schema of an upstream data source almost always breaks or requires updates to the downstream analytics system. It's an unavoidable problem whether its a microservice or a monolith; you just get to choose where you put the pain. Consider: Source Data -> Data Lake -> ETL Process -> Reporting DataWarehouse(s)/DataMart(s) -> User Queries vs Source Data -> Data Lake -> User Queries vs MonolithDB -> User queri…

This statement is so 20th century.

Since then analytical data warehouses (such as Snowflake) got ability to handle schema-less semi-structured data. No more ETL, it's ELT (where transformation, if it is needed, is done in SQL at query time or at intermediate transform time).

Re: Monoliths Are the Future

#466

Post by ajxs » Fri, Jan 31, 2020, 4:48 AM UTC

satyrnein wrote: post #275

jlmorton wrote: post #255

Earlier quoted context omitted.

> Spend 12 months building ETLs Okay, sounds reasonable enough for a complex enterprise. > to feed a torrent of raw data to your lake Well, there's the problem. Why is it taking a year to export data in its raw, natural state? The entire point of a data lake is that there is no transformation of the data. There's no need to verify the data is accurate. There's no need to make sure it's performant. It's just data expo…

If you are "export[ing] data in its raw, natural state" then haven't you lost the isolation benefits of microservices? Now you have external systems dependent on your implementation details, and changing your schema will break them.

This implies that the engineers who lobbied so hard for microservices were even all that concerned with these benefits to begin with, and took this into account when designing the architecture of the system. More often than not, in my experience, the developers involved are more concerned with code ownership than genuine architectural concerns.

Re: Monoliths Are the Future

#467

Post by sequoia » Fri, Jan 31, 2020, 4:56 AM UTC

p1necone wrote: post #309

sequoia wrote: post #296

Earlier quoted context omitted.

If you think telling colleagues not to "simply go and poke your reporting fingers into" things won't insult them or put them on a defensive footing, I encourage you to try it and closely note the reception you receive. In my experience, people do not appreciate being spoken to like that.

They didn't tell their colleagues to do that, they made a slightly humorous comment on a hacker news thread.

We’re colleagues by virtue of the fact that we’re members of the same profession.

Anyway, what’s the reason not to treat people on hackernews with the same respect you’d treat a coworker with?

Re: Monoliths Are the Future

#468

Post by closeparen » Fri, Jan 31, 2020, 4:58 AM UTC

titanomachy wrote: post #454

jrochkind1 wrote: post #350

Earlier quoted context omitted.

Interesting. Ie, just the way the larger community (ie open source) ecosystem works, but just inside your company instead. That makes an obvious kind of sense, when your number of contributors have grown too large to operate like a monolith... why not operate using models well-established for very large inter-entity communities? I wonder why more very large companies don't do this, if they don't.

I work at a very large company where each team builds and deploys its own production artifacts, but we always build everything off of head rather than choosing versions of each dependency and upgrading on our own schedule. The choose-your-own-adventure approach seems like it might be nice until you write (e.g) a critical security patch and you have to go nag N teams to upgrade their version of your library. With our…

Absolutely, "go nag N teams" is a huge problem in a microservices environment. It's mitigated somewhat by building all the services from a monorepo.

Re: Monoliths Are the Future

#469

Post by vlttnv » Fri, Jan 31, 2020, 4:59 AM UTC

From what I understand the author is against a bad implementation of microservices. I don't see a good argument against microservices but rather "We just chopped up a monolith and things are not better therefore microservices are bad". Am I missing something?

Re: Monoliths Are the Future

#470

Post by paracorp » Fri, Jan 31, 2020, 5:08 AM UTC

This blog post and the top response is an example of an echo chamber.

Microservices, when done right (driven by well defined bounded contexts) are simpler to develop and iterate against; but that's not why we do Microservices!

You should not do Microservices without considerable experience in authoring integration tests, a clear understanding of the domain, observability tools, and a team that can handle debugging distributed system.

Bonus: You do not need a distributed system if you are working out of a single data center. You should not do Microservices if you think they're cool. You should not title your blog post claiming Monoliths are the future. If your future has a horizon of never scaling out then yes I guess they are ...