Live data from Hacker News

Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

news.ycombinator.com

1–10 of 264 posts

Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

#1
A common way of deploying a web application database at scale is to setup a MySQL or Postgres server, create one table for all customers, and have an account_id or owner_if field and let the application code handle security. This makes it easier to run database migrations and upgrade code per customer all at once.

I’m curious if anybody has taken the approach of provisioning one database per account? This means you’d have to run migrations per account and keep track of all the migration versions and statuses somewhere. Additionally, if an application has custom fields or columns, the differences would have to be tracked somehow and name space collisions managed.

Has anybody done this? Particularly with Rails? What kinda of tools or processes did you learn when you did it? Would you do it again? What are some interesting trade offs between the two approaches?

Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

#2
Seems impractical and slow at scale to manage even a few hundred separate databases. You lose all the advantages of the relational model — asking simple questions like “Which customers ordered more than $100 last month” require more application code. You might as well store the customer info in separate files on disk, each with a different possible format and version.

Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

#3
This will be very inefficient due to the way DBMS commonly lay out data in pages. And if you want to do any kind of aggregate queries (e.g. analytics) you're probably in for some royal pain.

If you want to do this for security, why not layer the DB behind some system that requires and verifies the users access tokens for each request?

The only situation where such a setup might make sense is when you actually need per-user migrations to cater to specific customer's needs, but then you'll make it very hard to work with all customer's data through a generic interface.

Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

#4
This talk may be helpful[1]. It's given by Jeremy Evans, the maintainer of Sequel, and it's about how he's made Roda (a Rack-based web framework) more secure than your average web framework does by using database security and some of the features of databases that all too often are overlooked by app developers. You could possibly use Roda for the authentication phase of a Rails app (among other things) but the insights will be helpful regardless.

[1] https://www.youtube.com/watch?v=z3HZZHXXo3I

Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

#5
I’ve seen this done for performance. At one point, it was more common than you might think. These days, most data stores have a way to indicate how to store the data on disk, or one is using SSDs so random access/seeks are less expensive than they were with spinning disks.

As one example, New Relic had a table per (hour, customer) pair for a long time. From http://highscalability.com/blog/2011/7/18/new-relic-architec... (2011):

> Within each server we have individual tables per customer to keep the customer data close together on disk and to keep the total number of rows per table down.

In a situation like that, all queries and table operations are customer- and time-specific anyway. At the time, dropping an entire table took less I/O than deleting specific rows in a multi-customer table (for MyISAM tables, this may still be true: https://mariadb.com/kb/en/big-deletes/). Also, there was no risk from locking the table.

https://www.slideshare.net/newrelic/how-to-build-a-saas-app-... has a bit more. I think Lew Cirne gave that presentation in 2011 but I can’t find a video of it.

In the example you gave, if the goal is to support customer-defined fields, I don't think most people would map the customer-defined fields directly to SQL columns. Consider something like Postgres hstore (with indices as needed) or the many similar implementations in other data stores.

Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

#7
My multi-tenant web app does this but I don't know if you'd call 100 unique users a day "at scale".

I believe it will be helpful if it's necessary to separate customers into "pods" as we grow.

The main advantage I feel we get however is that it was quite easy to write a wrapper around mysqldump to retrieve data for development purposes.

I worked at a company that stored all customer data in a single database. The performance was comparable but the agility was poor. Firstly, you had to download all customer data to get a copy to debug a problem. This was a security concern I had, and eventually we had to build a serialisation format to retrieve the "slice" of data we needed. This tool needed frequent updating as new tables were added.

You might argue that we should just try to imagine the bug and recreate it but we have some pretty complicated data structures which can make investigations very hard.

Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

#8
The inability to reuse database connections would be a huge performance hit.

In a traditional webapp backend, you have a pool of connections to the database. User01 hits your service, and grabs a connection off the pool. User02 does the same, and so on. These connections get put back in the pool for reuse once a user is done with them.

In your design, every time a user hits your service, a new connection, specific to that user, will have to be made. This will incur network traffic and the overhead of logging in to the DBMS.

If you're thinking about using something like SQLite, you will hit a hard wall when the OS isn't able to open any more file descriptors, as well.

Like you said, DB administration will be a huge pain in the ass. Rather than having Flyway or Liquidbase or whatever run a migration on one database, you'll have to run it on thousands of databases. There will be significant downtime when your databases are not in a consistent state with one another. There will also be bloat from the migration tool's record keeping, which will be duplicated for every user, rather than every database.

A lot of the tools a database gives you for free will also need to be implemented in application logic, instead. For example, you might want to run a query that says "show me every user using more than 1GB of storage," but under your schema, you'll have to log into every user's database individually, determine storage used, and add it to an in-memory list.

If you ever want to allow users to collaborate, you will end up replicating the owner_id field type metadata anyway, and the entire benefit of this schema will evaporate.

Most frameworks are not set up to handle this style of database access, either. I don't use Rails, but Spring Boot would fight you every step of the way if you tried to do this.

Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

#9
One company in SF using MeteorJS deploys a whole container with DB and everything per customer.

I think their primary reasoning is that Meteor doesn't scale super easy, so it was easier to just "shard" the whole stack per customer.

Personally, it's a lot of work. It depends on what you're doing to make the tradeoffs worthwhile.

I see this as an optimization. Build everything so you can deploy once for all your customers. If you need to shard by customer later, it's just an infrastructure problem.

Re: Ask HN: Has anybody shipped a web app at scale with 1 DB per account?

#10

The inability to reuse database connections would be a huge performance hit. In a traditional webapp backend, you have a pool of connections to the database. User01 hits your service, and grabs a connection off the pool. User02 does the same, and so on. These connections get put back in the pool for reuse once a user is done with them. In your design, every time a user hits your service, a new connection, specific to…

You can use the same connection across multiple databases without any problem.
Post reply on HN