Live data from Hacker News

Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

news.ycombinator.com

231–240 of 264 posts

Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

#231
post #108

My startup currently does just this 'at scale', which is for us ~150 b2b customers with a total database footprint of ~500 GB. We are using Rails and the Apartment gem to do mutli-tenancy via unique databases per account with a single master database holding some top-level tables. This architecture decisions is one of my biggest regrets, and we are currently in the process of rebuilding into a single database model.…

Echoing this as well, I worked for Influitive and was one of the original authours of apartment (sorry!) There are a lot of headaches involved with the "tenant per schema" approach. Certainly it was nice to never have to worry about the "customer is seeing data from another customer" bug (a death knell if you're in enterprisish B2B software), but it added so many problems: - Migrations become a very expensive and tim…

Oh I want to add, I created an small tool called octoelephant, basically an akka client to parallel issue and merge a query result in thousands of DB. It queries let's say 2000 DBs in 10 secs for regular queries. Throw me a line if you're interested

Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

#232
post #214
post #66

Earlier quoted context omitted.

It's a terrible idea in the same way that using PHP instead of Rust to build a production large scale application is a terrible idea (i.e. it's actually a great idea but it's not "cool").

It’s not a cool factor issue. It’s an issue of bloating the system catalogs, inability to use the buffer pool, and having to run database migrations for each and every separate schema or maintaining concurrent versions of application code to deal with different schema versions. It’s be just as dumb to do it in Rust as it would be in PHP.

As you can see now that the thread has matured, there are a lot of proponents of this architecture that have production experience with it, so it's likely not as dumb as you assume.

Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

#233
post #64
post #50

Earlier quoted context omitted.

Do you not have to have a large dev ops team to handle upgrades or changes for each customer? I'd be against it to do more with fewer people, especially for an unproven < 5 year old startup.

Actually, I think conventional multitenant (shared database with tenant_id on database rows) is a bad idea (as well as premature optimization) in the previously described context. One of the companies I am thinking of had no dedicated ops team until 300+ customers, running on 300+ instances of the setup. Probably 40+ employees at the time the first dedicated sysadmin was hired. The trick is that since the production…

For context, we implemented your exact model at my job prior to switching to "conventional multi-tenant". I agree with your premise that there are a lot of off the shelf tools to help this, but in enterprise I feel like we hit dozens of other issues and our customers didn't even care if they shared the same database as someone else.

If the production environment is extremely similar to dev, the problems are numerous, how do you ensure that a bug in one app server doesn't let users access other request data? We solved this via multiple app servers, but then how do you solve deploying to hundreds of individual app servers? We have 12 services per customer, replicated to 4 physical locations, that means 124500(customers) 24,000 containers to roll out any upgrade completely. (99.99% SLA)

Also some customers get different services depending on their tier, so now we need to bake in some form of Salesforce integration into our deployment so we know which services to start up(we didn't do this unfortunately, so all services started up and some didn't do anything).

If you split into many physical database servers vs many "logical" databases, how do you monitor that an individual customers performance is adequate? What about APM? One NewRelic instance per customer? One ELK cluster per customer? How do your developers / customer support engineers get access to a specific customer instances both in the software, in the database and APM? I cannot fathom having to manage 500 ELK clusters, even with AWS managed Elastic search.

How do you enable ETL and reporting solutions that support all these databases. What if you have data aggregation rights and need to combine different customer's data? What happens when your product team needs to report across all customers to determine what features to work on (gotta be data driven)?

Maybe this is only related to healthcare (where I work), but in our industry I cannot imagine our health systems and manufacturers not wanting answers to these. These answers are infinitely simpler in our new multi-tenant model vs our old multiple host model. Writing the processes and documentation around guaranteeing segmentation took less than 8 hours and is only 3 pages of our entire security model. Hi-trust specifically calls out many of these points, and I am sure SOC-2 does as well.

In my experience, not a single one of our health systems or auditors has had an opinion either way on this. They only care when you aren't doing APM properly, or taking the necessary steps to prevent attacks both from external parties and by internal parties. We do in-depth third party audits that are shared with our customers. Ensuring good processes for managing risk is always more important than wishful thinking that a design pattern will make your enterprise model work.

Our total lines of code for logical segmentation takes up less than 300 lines in all of our services combined.

Finally, given that Postgres10+ now natively supports row level security, I fail to see why anyone starting a new project would choose the multiple database option when you can bake in authorization straight into the DB at the lowest level.

Sorry this is really long, but I wanted to share my experience & thoughts.

Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

#234
I worked on a project with PostgreSQL schemas per tenant (almost like databases). Also worked on another one with "normal" account_id field separation.

I documented how they compare in a blogpost: https://blog.arkency.com/comparison-of-approaches-to-multite... - funnily it was waiting unpublished for some time, stumbling on your post made me finally publish it.

Looking forward to go through this comments question and see what others have experienced.

I have another draft in the making - about some of the pitfalls of PostgreSQL schemas approach, should be released soon.

Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

#236
post #175
post #108

My startup currently does just this 'at scale', which is for us ~150 b2b customers with a total database footprint of ~500 GB. We are using Rails and the Apartment gem to do mutli-tenancy via unique databases per account with a single master database holding some top-level tables. This architecture decisions is one of my biggest regrets, and we are currently in the process of rebuilding into a single database model.…

Can confirm, here be dragons. I did a DB per tenant for a local franchise retailer and it was the worst design mistake I ever made, which of course seemed justified at the time (different tax rules, what not), but we never managed to get off it and I spent a significant amount of time working around it, building ETL sync processes to suck everything into one big DB, and so on. Instead of a DB per tenant, or a table p…

I do both.

Have a tenant_id column in every table.

This gives me flexibility to either host each client separately or club them together.

Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

#237
post #232
post #214

Earlier quoted context omitted.

It’s not a cool factor issue. It’s an issue of bloating the system catalogs, inability to use the buffer pool, and having to run database migrations for each and every separate schema or maintaining concurrent versions of application code to deal with different schema versions. It’s be just as dumb to do it in Rust as it would be in PHP.

As you can see now that the thread has matured, there are a lot of proponents of this architecture that have production experience with it, so it's likely not as dumb as you assume.

> As you can see now that the thread has matured, there are a lot of proponents of this architecture that have production experience with it, ...

Skimming through the updated comments I do not see many claiming it was a good idea or successful at scale. It may work fine for 10s or even 100s of customers, but it quickly grows out of control. Trying to maintain 100,000 customer schemas and running database migrations across all of them is a serious headache.

> ...so it's likely not as dumb as you assume.

I'm not just assuming, I've tried out some of the ideas proposed in this thread and know first hand they do not work at scale. Index page caching in particular is a killer as you lose most benefits of a centralized BTREE structure when each customer has their own top level pages. Also, writing dynamic SQL to perform 100K "... UNION ALL SELECT * FROM customer_12345.widget" is both incredibly annoying and painfully slow.

Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

#238
post #117

Earlier quoted context omitted.

Somewhat tangetial: If we use something like bigquery which handles multitenancy well, there still doesn't appear to be a good way to expose it to a customer directly (say for a BI tool). Like with a simple username/pwd. Any pointers?

In GCP you can grant permissions to entities outside of your org, either to users or to an external service account. This is how things like Cloud Build and other services that require Google to have access to resources inside your projects work.

Doesn't it still need the end users to have some kind of google account? At the very least a google email account?

Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

#239
post #29

Earlier quoted context omitted.

> In your design, every time a user hits your service, a new connection, specific to that user, will have to be made. This will incur network traffic and the overhead of logging in to the DBMS. If connecting to your DB is significantly increasing your page times, you've got seriously fast pages. Even back when I was working with a MySQL database regularly in 2010, connect + login was 5 ms at maximum (and I think it w…

> If connecting to your DB is significantly increasing your page times, you've got seriously fast pages. Even back when I was working with a MySQL database regularly in 2010, connect + login was 5 ms at maximum (and I think it was much less, I just don't remember that far). One of the major ways I helped Mastodon was improving their connection pooling situation. If you haven't encountered connection size issues with…

My experience was with Apache + mod_php, so there was no option to pool connections between workers, and you would set the Apache connection limits such that they summed up to less than the MySQL connection limits (unless you had a lot of traffic that didn't hit the database... then sizing would be tricky)

Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

#240
post #117

Earlier quoted context omitted.

In GCP you can grant permissions to entities outside of your org, either to users or to an external service account. This is how things like Cloud Build and other services that require Google to have access to resources inside your projects work.

Doesn't it still need the end users to have some kind of google account? At the very least a google email account?

[deleted]
Post reply on HN