Live data from Hacker News

Does anyone integrate with their customers' DB directly?

news.ycombinator.com

31–40 of 48 posts

Re: Does anyone integrate with their customers' DB directly?

#31
post #4

> there's nothing wrong with querying the fintech's DB as long as we're treating infosec seriously This is an unfortunate attitude. From a software architecture point of view, you're hard-coupling your software to that database which can create brittleness: You're at the mercy of the original database schema which may not be optimal, carries it's own tech debt, may require translations to your own implementation, and…

> You're at the mercy of the original database schema SQL has views. As long as you can represent the underlying data through a view you should be fine. > and may cause mistakes in interpretation in anything of scale All APIs have this concern. > direct access to a database is an unnecessary security risk You can constrain access to be read-only and to only access the particular view or views necessary. > has found p…

> SQL has views.

This, and it doesn't have to be SQL, a number of other databases have similar concepts.

If you have a separate schema with views (in SQL terms), and the rest is either entirely walled off or available on a "you have been warned" basis, then this is simply an API with a clear contract, just a flexible one (not entirely unlike a GraphQL endpoint) and available over a less common (for APIs) protocol.

It can be risky for the API provider, though, unless it's a proper multi-tenant database. Given that those are less common, and, I guess, potentially harder to manage, it's probably why database access APIs are rare.

Re: Does anyone integrate with their customers' DB directly?

#32
The only reasonable way to do this safely is by querying a read replica. You could take down your customer's systems very, very easily by running queries without proper indexes. Dealing with that is probably trickier than it sounds, because a DB of appreciable size just won't be queryable.

You might go live quicker. But the integration will break every time your customer makes a change or needs to upgrade. Pray there's documentation. You'll spend unending hours debugging your customers' weird data models. Issues will appear when logic changes but the data model doesn't.

I won't tell you not to do this because you've been so light on details of what you're actually doing (maybe it is easier this way!), but it would not fly on my watch.

Re: Does anyone integrate with their customers' DB directly?

#33

Something boring but workable is using CSV files, dropped to a location on customers end through SFTP containing your transaction IDs that you are interested in. If its MS SQL Server on the other side, then a SQL agent on that server using some SSIS can read the CSV as input and prepare your required output as a CSV again for you to pickup from their SFTP Server.

This works in some situations but the OPs use case seems to suggest near real-time access to records that would only have recently been created. I'm unsure if CSV could work well here.

Re: Does anyone integrate with their customers' DB directly?

#34

The only reasonable way to do this safely is by querying a read replica. You could take down your customer's systems very, very easily by running queries without proper indexes. Dealing with that is probably trickier than it sounds, because a DB of appreciable size just won't be queryable. You might go live quicker. But the integration will break every time your customer makes a change or needs to upgrade. Pray there…

Exactly this. It's very easy to accidentally write a query that takes 20 minutes to complete rather than 20 milliseconds. A few of those and boom, your server is dead.

There is no situation in which I'd let a customer write their own queries against the main shared production database, for performance reasons alone. It's a complete non-starter.

(Although like you say, if they want to pay for their own dedicated private read replica, and deal with breaking schema changes whenever they arise, let them go at it. That's sure not something I'd want as a customer though.)

Re: Does anyone integrate with their customers' DB directly?

#36
It's really funny seeing some of the outraged, self-righteous responses here from developers who probably take themselves too seriously.

The OP already said that their customer didn't have the resource to even connect to an API, so what on earth makes you think they're gonna be able to create some highly-abstrated API layer or some other "bEsT PrAcTiCe" way of exposing the data?

And even if they did create a view or API for the database instead of giving direct access, what makes you think they'd keep that view or API up-to-date? It's just as likely to break as a changing database schema is.

Re: Does anyone integrate with their customers' DB directly?

#37

Data visualization products do this regularly

Are there any good open source/free ones? I tried Apache Superset a few months ago which seemed promising but was appalling bad when it came to trying to use it.

I like metabase

Re: Does anyone integrate with their customers' DB directly?

#38
post #4

> there's nothing wrong with querying the fintech's DB as long as we're treating infosec seriously This is an unfortunate attitude. From a software architecture point of view, you're hard-coupling your software to that database which can create brittleness: You're at the mercy of the original database schema which may not be optimal, carries it's own tech debt, may require translations to your own implementation, and…

Your scorn should be addressed at the OP's customer, not OP themselves. If the customer doesn't have the resource to implement an API or whatever then OP's choice becomes either "awkwardly refuse the job" or "use the credentials they're given".

Re: Does anyone integrate with their customers' DB directly?

#39
If they don’t have staff to hit an api, tying to their database sounds like a one-way trip to a lawsuit. The second something goes wrong they’re going to blame you and won’t have the technical expertise to understand your explanation of why it’s not your fault.

Re: Does anyone integrate with their customers' DB directly?

#40
First off, this is creative and you did a great job of writing it up and explaining the benefits to your approach. And I think it is a good idea in extremely early stages of a software product because as you mention, it’s a quick way to get something up and running.

Unfortunately, the longer into the project you are, the more brittle that connection becomes. You’re tightly coupling a process to the state of the database at time of build. When you do that, you create a situation where your customers can kill your code.

So fundamentally, when you integrate directly, you create a situation where your customers’ engineers control how reliable your software can be. They likely won’t fess up when they break your tool so the problems will come back to you.

I know that sounds a lot like typing ‘npm install’ or ‘building an API’ but in this case, you’re handing the keys over to your customers. At best, it will cause reliability problems. At worst, it will add friction to a sales process. As companies scale and sales gains power, that can become a career issue for you. As a rule, it’s better to have difficult technical decisions when you’re employed than difficult financial decisions when you’re not.

So good writing and excellent analysis. But if you choose to go down that path, at some point in the future, I feel like that integration will become someone’s headache.

If you have to do it, it would be worth documenting that the customer in this case won’t provide proper access through an API. That’s a good reason to do something like this because that’s quite unreasonable of the customer. If they’re unwilling to provide API access, I feel like they’re just as unlikely to provide a read replica so aside from the integration’s brittleness, your queries have to be rock solid or you could slow down prod.

It’s all risky and you have a big decision to make. Good luck and have fun.

Post reply on HN