My startup currently does just this 'at scale', which is for us ~150 b2b customers with a total database footprint of ~500 GB. We are using Rails and the Apartment gem to do mutli-tenancy via unique databases per account with a single master database holding some top-level tables. This architecture decisions is one of my biggest regrets, and we are currently in the process of rebuilding into a single database model.…
Echoing this as well, I worked for Influitive and was one of the original authours of apartment (sorry!) There are a lot of headaches involved with the "tenant per schema" approach. Certainly it was nice to never have to worry about the "customer is seeing data from another customer" bug (a death knell if you're in enterprisish B2B software), but it added so many problems: - Migrations become a very expensive and tim…
Ask HN: Has anybody shipped a web app at scale with 1 DB per account?
231–240 of 264 posts
Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?
#232Earlier quoted context omitted.
It's a terrible idea in the same way that using PHP instead of Rust to build a production large scale application is a terrible idea (i.e. it's actually a great idea but it's not "cool").
It’s not a cool factor issue. It’s an issue of bloating the system catalogs, inability to use the buffer pool, and having to run database migrations for each and every separate schema or maintaining concurrent versions of application code to deal with different schema versions. It’s be just as dumb to do it in Rust as it would be in PHP.
Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?
#233Earlier quoted context omitted.
Do you not have to have a large dev ops team to handle upgrades or changes for each customer? I'd be against it to do more with fewer people, especially for an unproven < 5 year old startup.
Actually, I think conventional multitenant (shared database with tenant_id on database rows) is a bad idea (as well as premature optimization) in the previously described context. One of the companies I am thinking of had no dedicated ops team until 300+ customers, running on 300+ instances of the setup. Probably 40+ employees at the time the first dedicated sysadmin was hired. The trick is that since the production…
If the production environment is extremely similar to dev, the problems are numerous, how do you ensure that a bug in one app server doesn't let users access other request data? We solved this via multiple app servers, but then how do you solve deploying to hundreds of individual app servers? We have 12 services per customer, replicated to 4 physical locations, that means 124500(customers) 24,000 containers to roll out any upgrade completely. (99.99% SLA)
Also some customers get different services depending on their tier, so now we need to bake in some form of Salesforce integration into our deployment so we know which services to start up(we didn't do this unfortunately, so all services started up and some didn't do anything).
If you split into many physical database servers vs many "logical" databases, how do you monitor that an individual customers performance is adequate? What about APM? One NewRelic instance per customer? One ELK cluster per customer? How do your developers / customer support engineers get access to a specific customer instances both in the software, in the database and APM? I cannot fathom having to manage 500 ELK clusters, even with AWS managed Elastic search.
How do you enable ETL and reporting solutions that support all these databases. What if you have data aggregation rights and need to combine different customer's data? What happens when your product team needs to report across all customers to determine what features to work on (gotta be data driven)?
Maybe this is only related to healthcare (where I work), but in our industry I cannot imagine our health systems and manufacturers not wanting answers to these. These answers are infinitely simpler in our new multi-tenant model vs our old multiple host model. Writing the processes and documentation around guaranteeing segmentation took less than 8 hours and is only 3 pages of our entire security model. Hi-trust specifically calls out many of these points, and I am sure SOC-2 does as well.
In my experience, not a single one of our health systems or auditors has had an opinion either way on this. They only care when you aren't doing APM properly, or taking the necessary steps to prevent attacks both from external parties and by internal parties. We do in-depth third party audits that are shared with our customers. Ensuring good processes for managing risk is always more important than wishful thinking that a design pattern will make your enterprise model work.
Our total lines of code for logical segmentation takes up less than 300 lines in all of our services combined.
Finally, given that Postgres10+ now natively supports row level security, I fail to see why anyone starting a new project would choose the multiple database option when you can bake in authorization straight into the DB at the lowest level.
Sorry this is really long, but I wanted to share my experience & thoughts.
Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?
#234I documented how they compare in a blogpost: https://blog.arkency.com/comparison-of-approaches-to-multite... - funnily it was waiting unpublished for some time, stumbling on your post made me finally publish it.
Looking forward to go through this comments question and see what others have experienced.
I have another draft in the making - about some of the pitfalls of PostgreSQL schemas approach, should be released soon.
Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?
#235Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?
#236My startup currently does just this 'at scale', which is for us ~150 b2b customers with a total database footprint of ~500 GB. We are using Rails and the Apartment gem to do mutli-tenancy via unique databases per account with a single master database holding some top-level tables. This architecture decisions is one of my biggest regrets, and we are currently in the process of rebuilding into a single database model.…
Can confirm, here be dragons. I did a DB per tenant for a local franchise retailer and it was the worst design mistake I ever made, which of course seemed justified at the time (different tax rules, what not), but we never managed to get off it and I spent a significant amount of time working around it, building ETL sync processes to suck everything into one big DB, and so on. Instead of a DB per tenant, or a table p…
Have a tenant_id column in every table.
This gives me flexibility to either host each client separately or club them together.
Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?
#237Earlier quoted context omitted.
It’s not a cool factor issue. It’s an issue of bloating the system catalogs, inability to use the buffer pool, and having to run database migrations for each and every separate schema or maintaining concurrent versions of application code to deal with different schema versions. It’s be just as dumb to do it in Rust as it would be in PHP.
As you can see now that the thread has matured, there are a lot of proponents of this architecture that have production experience with it, so it's likely not as dumb as you assume.
Skimming through the updated comments I do not see many claiming it was a good idea or successful at scale. It may work fine for 10s or even 100s of customers, but it quickly grows out of control. Trying to maintain 100,000 customer schemas and running database migrations across all of them is a serious headache.
> ...so it's likely not as dumb as you assume.
I'm not just assuming, I've tried out some of the ideas proposed in this thread and know first hand they do not work at scale. Index page caching in particular is a killer as you lose most benefits of a centralized BTREE structure when each customer has their own top level pages. Also, writing dynamic SQL to perform 100K "... UNION ALL SELECT * FROM customer_12345.widget" is both incredibly annoying and painfully slow.
Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?
#238Earlier quoted context omitted.
Somewhat tangetial: If we use something like bigquery which handles multitenancy well, there still doesn't appear to be a good way to expose it to a customer directly (say for a BI tool). Like with a simple username/pwd. Any pointers?
In GCP you can grant permissions to entities outside of your org, either to users or to an external service account. This is how things like Cloud Build and other services that require Google to have access to resources inside your projects work.
Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?
#239Earlier quoted context omitted.
> In your design, every time a user hits your service, a new connection, specific to that user, will have to be made. This will incur network traffic and the overhead of logging in to the DBMS. If connecting to your DB is significantly increasing your page times, you've got seriously fast pages. Even back when I was working with a MySQL database regularly in 2010, connect + login was 5 ms at maximum (and I think it w…
> If connecting to your DB is significantly increasing your page times, you've got seriously fast pages. Even back when I was working with a MySQL database regularly in 2010, connect + login was 5 ms at maximum (and I think it was much less, I just don't remember that far). One of the major ways I helped Mastodon was improving their connection pooling situation. If you haven't encountered connection size issues with…
Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?
#240Earlier quoted context omitted.
In GCP you can grant permissions to entities outside of your org, either to users or to an external service account. This is how things like Cloud Build and other services that require Google to have access to resources inside your projects work.
Doesn't it still need the end users to have some kind of google account? At the very least a google email account?