Live data from Hacker News

Sharding data models

citusdata.com

1–10 of 20 posts

Re: Sharding data models

#3
post #2

Citus is a pretty powerful tool. We recently used Citus to help scale one of our databases and wrote a blog post about it: https://hipmunk.github.io/posts/2017/Aug/16/a-fare-cache-in-...

What were the before/after performance and load numbers? Your post mentions that data-refresh frequency was an issue, but doesn't mention what sort of problems it caused. Would love to hear how the move affected query perf

Re: Sharding data models

#6
post #4

Then there's the 0 model for sharding - don't shard, just replicate everything, everywhere, with eventual consistency and MVCC

Author of the post here, couldn't agree more that if you don't have to shard then don't. Scaling out using replicas is an option to certain scale, you do have to ensure no long running queries and make sure your replication is up to date. Those are another option of things to manage, and at an intermediate stage very viable ones, at certain scale it all changes a bit. All that said if you're at a small data level I wouldn't encourage sharding for the sake of it.. if you know you're going to have to scale beyond a single node or approaching limits where you're frequently scaling up it's good to know your options and plan ahead.

Re: Sharding data models

#8
For financial transaction services I can recommend sharding first by customer, then by ledger. As a result, instead of enforcing double-entry book-keeping standards within a single database, do it at an application-specific middleware server layer to enforce only the guarantees you need.

As with all decisions, there are tradeoffs. For a little more up-front complexity and a tiny nominal performance hit, this allows maintenance, encryption, non-uniform storage paradigms to suit individual ledgers, rolling upgrades, storage location migration, and other cool stuff which is typically painful with traditional all-or-nothing RDBMS architectures.

Re: Sharding data models

#9

What ORMs out there natively support sharding well? I use Django the most and whenever I look into Django sharding, I don't see very many options being kept up to date...

I use ServiceStack's OrmLite for .NET, and it supports multi-nested database connections for Master/Shard setup: https://github.com/ServiceStack/ServiceStack.OrmLite#multi-n...

Re: Sharding data models

#10

What ORMs out there natively support sharding well? I use Django the most and whenever I look into Django sharding, I don't see very many options being kept up to date...

I'm not sure any do completely out of the box but many are very close.

We've created a library at Citus to make rails sharding turn-key, activerecord-multi-tenant. We have a Django one in the works, if you want to take a look and give us any feedback on it please drop us a note. From what I've seen hibernate has some out of the box support, but we only have a few customers leveraging it so not as familiar with it.

Post reply on HN