Earlier quoted context omitted.
I know it's off-topic but maybe "MAANG level app" >> "MANGA" if we want go there?
But if F->M, then why not G->A? MAAAN.
Referring to the tech powers that be as "The MAAAN" is pretty fitting.
Reducing complexity by integrating through the database
21–30 of 45 posts
Re: Reducing complexity by integrating through the database
#22Re: Reducing complexity by integrating through the database
#23While they quickly acknowledge that integrating by means of a database is widely regarded as an anti-pattern, the rest of the article doesn't really address why this wouldn't be an anti-pattern, other than to pretend that the major reasons for avoiding this pattern are deployment complexity and furnishing multi-region services. > A customer pattern we see solves this problem, and it’s the integration database pattern…
Not related to the original article but I had a go at articulating the rationale here:
https://hexdocs.pm/integratedb/intro.html
(This project isn’t ready for prime time but it was a sketch of an approach to mitigating the downsides of integrating through the database, in order to unlock the upsides).
Re: Reducing complexity by integrating through the database
#24My first two programming jobs out of college back in the early 2010's took the approach in this article. Albeit, with older technology. This is bringing back old (bad) memories of the times where I was debugging stored procedures that called triggers that called the same stored procedures. A lot of that was due to poor design and bad choices. Some of that was due to developers trying to fit processes and patterns int…
I remember doing the same almost 30 years back, copying data from an Oracle database to an MSSQL database.
It was a fairly limiting approach, performance wasn't great, and diagnosing problems was a total PITA.
Afterwards I did an integration using an antiquated version of BizTalk, which was all based on COM and to this day remains my most loathed software ever. It was way too complex, the UI was crap, it was flakey as hell, and ridiculously slow. Later versions got better, but it was a very low bar.
Later I used a Java-based Apache project to do an integration - I forget the name. It was actually really good, easy to use but also allowing full control.
Later still, I did some integrations using custom C# code based around WWF (Windows Workflow Foundation) - the UI in Visual Studio was horrendously slow and flakey, it was too complex to build your own pipeline segments, and difficult to diagnose issues in production. But it did work fairly well for simple stuff.
And then later still I used PowerCenter - I was the architect, so wasn't that hands on, but the PowerCenter guys liked it, and we ended up adopting it as our standard integrations tool, with hundreds of integrations across the company.
Re: Reducing complexity by integrating through the database
#25Use the right tool for the job. What is the most deployed database? SQLite3. In what pattern is it most often deployed? One language from one environment accessing one database with full read/write. Low call volume, high data complexity, and embedded (tightly application-coupled) use. This is observably the normal use case for a database and the simplest mode of implementation. Problems occur when software people start solving problems that don't exist: typically performance, future scalability, dubious theories of security and future language/database migrations. KISS.
Re: Reducing complexity by integrating through the database
#26I think anyone with experience on this type of system will recoil in horror at the thought of building it intentionally today.
Tell me about it. This is the current system I maintain. While new products have (mostly) started using non-integrated DBs, all the legacy systems we have use a giant integrated db. It SUCKS. Let me count the ways. - You can't update a table schema without updating and deploying a bunch of applications in sync (even if that's part of a stored proc). Which for something like this, means that getting out of this situat…
I mean, yeah, this is why people stopped using this pattern. But these problems are getting solved, especially in Fauna:
1. Schemaless/document/schema-on-need databases like Fauna don't mandate the application breakage on every change that SQL does
2. It's hard to reason about if its not transparent, but it can be transparent now, see below
3. Fauna is a temporal database, which acts like version control on your stored procedures, so you can easily check and revert any change
4. Fauna is serverless and horizontally scalable without consistency/latency impact
5. This was definitely a problem when you were occupying precious CPU cores on a vertically scaled RDBMS with business logic, but compute in Fauna or in serverless lambdas scales horizontally indefinitely
Re: Reducing complexity by integrating through the database
#27My first two programming jobs out of college back in the early 2010's took the approach in this article. Albeit, with older technology. This is bringing back old (bad) memories of the times where I was debugging stored procedures that called triggers that called the same stored procedures. A lot of that was due to poor design and bad choices. Some of that was due to developers trying to fit processes and patterns int…
I remember doing the same almost 30 years back, copying data from an Oracle database to an MSSQL database. It was a fairly limiting approach, performance wasn't great, and diagnosing problems was a total PITA. Afterwards I did an integration using an antiquated version of BizTalk, which was all based on COM and to this day remains my most loathed software ever . It was way too complex, the UI was crap, it was flakey…
Was it Apache Camel? I used that a while back and liked how low hassle it was.
Re: Reducing complexity by integrating through the database
#28I wouldn't integrate through a database but I wouldn't refuse to integrate through a distributed cache.
Anyway using REST for inter services communication decreases performance and increases latency and 99% of projects still do it.
Re: Reducing complexity by integrating through the database
#29I wouldn't integrate through a database but I wouldn't refuse to integrate through a distributed cache. Anyway using REST for inter services communication decreases performance and increases latency and 99% of projects still do it.
Curious what makes a cache better to you.
Re: Reducing complexity by integrating through the database
#30If I'm reading this correctly the crux of the argument appears to be if your serverless database engine provides cross-region data replication and consistency and provides a GraphQL interface then applications can use GraphQL to go directly to the database, thus solving a lot of the problems we endured in the past when using SQL to go to the database. It's an interesting idea that at first glance appears to be worth looking into further, though I still have a uneasy feeling because I remember all the pain in the past!