Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reducing complexity by integrating through the database

fauna.com

Post reply on HN
Print view View original

31–40 of 45 posts

Re: Reducing complexity by integrating through the database

#31

Post by mumblemumble »

nescioquid wrote: post #10

While they quickly acknowledge that integrating by means of a database is widely regarded as an anti-pattern, the rest of the article doesn't really address why this wouldn't be an anti-pattern, other than to pretend that the major reasons for avoiding this pattern are deployment complexity and furnishing multi-region services. > A customer pattern we see solves this problem, and it’s the integration database pattern…

Also not the author, but I have seen some success here when the database is used as another way to build a service with an API, and not just a simple data store. What that means in practice is that applications interact exclusively through stored procedures, which serve as a sort of RPC API.

The big downside of this approach are that the human factors are tricky nowadays. Developers expect to be allowed to poke at the tables directly, and don't necessarily take kindly to alternative ways of framing things. Especially if they've been raised on a diet of code-first object relational mapping. And there's not really a great way to enforce this boundary, because, unlike HTTP, an ODBC session will generally let you pretty much anything you want to the remote service's implementation details.

Re: Reducing complexity by integrating through the database

#32

Post by tlarkworthy »

At Firebase it was sometimes called the client-database-server architecture. The pattern was documented in 2013 [1]

If you use Firebase as an ephemeral message bus it's a great pattern. It has problems if you use it like a traditional database because migrations are very tricky. DBs that support views (or GraphQL) can make migrations much easier

[1] https://firebase.googleblog.com/2013/03/where-does-firebase-...

Re: Reducing complexity by integrating through the database

#33

Post by gunnarmorling »

No matter how you integrate different applications, be it via APIs, messaging, or a database, it's vital to separate your application's internal data model from models which it exposes. If you don't do that, you're in for a never-ending story of upstream services unknowingly breaking downstream services, or upstream services not being able to evolve in any meaningful way.

So if they mean directly exposing a service's data model from the database to other services, I'm very skeptical. If they mean providing that access by means of some abstraction, e.g. database views, it can be an option in some cases.

You'll still loose lots of flexibility you'd gain by putting some middleware in between, e.g. ability to scale out compute independently from storage, ability to merge data from multiple sources into one API response, ability to implement arbitrarily complex business logic e.g. in Java, etc.

Re: Reducing complexity by integrating through the database

#34

Post by DeathArrow »

evanweaver wrote: post #29
DeathArrow wrote: post #28

I wouldn't integrate through a database but I wouldn't refuse to integrate through a distributed cache. Anyway using REST for inter services communication decreases performance and increases latency and 99% of projects still do it.

Curious what makes a cache better to you.

It would be faster and I still get to keep microservices from accessing the data they don't need or shouldn't access.

But I rather use RPC for communication.

Re: Reducing complexity by integrating through the database

#35

Post by dennisy »

gunnarmorling wrote: post #33

No matter how you integrate different applications, be it via APIs, messaging, or a database, it's vital to separate your application's internal data model from models which it exposes. If you don't do that, you're in for a never-ending story of upstream services unknowingly breaking downstream services, or upstream services not being able to evolve in any meaningful way. So if they mean directly exposing a service's…

I agree, but would you mind providing a concrete example of:

> No matter how you integrate different applications, be it via APIs, messaging, or a database, it's vital to separate your application's internal data model from models which it exposes.

Especially in respect to this product.

Re: Reducing complexity by integrating through the database

#37

Post by GordonS »

meepmorp wrote: post #27
GordonS wrote: post #24

Earlier quoted context omitted.

I remember doing the same almost 30 years back, copying data from an Oracle database to an MSSQL database. It was a fairly limiting approach, performance wasn't great, and diagnosing problems was a total PITA. Afterwards I did an integration using an antiquated version of BizTalk, which was all based on COM and to this day remains my most loathed software ever . It was way too complex, the UI was crap, it was flakey…

Was it Apache Camel? I used that a while back and liked how low hassle it was.

Hmm, name doesn't sound familiar, it was a very long time ago tho.

Re: Reducing complexity by integrating through the database

#38

Post by treeman79 »

dennisy wrote: post #35
gunnarmorling wrote: post #33

No matter how you integrate different applications, be it via APIs, messaging, or a database, it's vital to separate your application's internal data model from models which it exposes. If you don't do that, you're in for a never-ending story of upstream services unknowingly breaking downstream services, or upstream services not being able to evolve in any meaningful way. So if they mean directly exposing a service's…

I agree, but would you mind providing a concrete example of: > No matter how you integrate different applications, be it via APIs, messaging, or a database, it's vital to separate your application's internal data model from models which it exposes. Especially in respect to this product.

A field you rely on changed form integer to string. Or goes away. Or was unique but a duplicate shows up. Or the database disappears for a few minutes every hour.

S

Re: Reducing complexity by integrating through the database

#39

Post by joshuanapoli »

dennisy wrote: post #35
gunnarmorling wrote: post #33

No matter how you integrate different applications, be it via APIs, messaging, or a database, it's vital to separate your application's internal data model from models which it exposes. If you don't do that, you're in for a never-ending story of upstream services unknowingly breaking downstream services, or upstream services not being able to evolve in any meaningful way. So if they mean directly exposing a service's…

I agree, but would you mind providing a concrete example of: > No matter how you integrate different applications, be it via APIs, messaging, or a database, it's vital to separate your application's internal data model from models which it exposes. Especially in respect to this product.

Over time, the internal data model might need to change to more accurately model the world. For consistency and efficiency, we usually do not want to maintain the combination of the original low-fidelity model and new high-fidelity model within the internal database.

The internal data model might need to be reorganized to improve the efficiency of part of the internal application.

A client might not need a higher fidelity model. The internal reorganization of the data model might not be pertinent to the client's interface. So we normally have some data-mapping in the application to help provide a stable interface for clients.

It's possible to providing these compatibility mappings within the database through views, but this is usually considered to be harder to control, test and scale.

Maybe in a big application, the application-layer mapping and caching eventually get complicated enough to be something like a custom-made database. And so we might end up with an "integration database" but call it something different.

Re: Reducing complexity by integrating through the database

#40

Post by ttymck »

dennisy wrote: post #35
gunnarmorling wrote: post #33

No matter how you integrate different applications, be it via APIs, messaging, or a database, it's vital to separate your application's internal data model from models which it exposes. If you don't do that, you're in for a never-ending story of upstream services unknowingly breaking downstream services, or upstream services not being able to evolve in any meaningful way. So if they mean directly exposing a service's…

I agree, but would you mind providing a concrete example of: > No matter how you integrate different applications, be it via APIs, messaging, or a database, it's vital to separate your application's internal data model from models which it exposes. Especially in respect to this product.

Are you asking for an example of separating the data models or an example of what happens when you don't.

This is a good example of what separation enables you to do:

https://www.troyhunt.com/your-api-versioning-is-wrong-which-...

(it is presented from the perspective of how to version an API, but the examples of what it looks like are there)

Post reply on HN