Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Breaking Through Scaling Barriers with Bigtable

remesh.blog

Post reply on HN
Print view View original

11–20 of 21 posts

Re: Breaking Through Scaling Barriers with Bigtable

#11

Post by stangles1 »

jpgvm wrote: post #4

Not sure I would have opted for Bigtable here over sharded PostgreSQL because it unduly limits flexibility but within this constrained use case it works fine. The big thing to remember which is covered by this article is your only real performant option to return multiple rows for BT is range scans so your keys should be setup to support this. If you need more than 1 index you are essentially shit out of luck - hence…

Good points - sharded Postgres is more likely to be a better choice in most instances. I wouldn't be surprised even if sharded Postgres would have worked well for us, but as you mentioned, for this constrained use case, Bigtable works fine.

Re: Breaking Through Scaling Barriers with Bigtable

#13

Post by bgm1975 »

I’d be curious to know if cockroachdb was evaluated as a potential candidate. The various literature bits I’ve read seem to indicate its whole reason for existence is to solve these kinds of problems at scale while still providing some semblance of ACID compliance.

Re: Breaking Through Scaling Barriers with Bigtable

#15

Post by ntietz »

jvolkman wrote: post #3

Surprised they didn't mention Cloud Spanner at all.

I dug back through our design docs from when we designed this, and we largely chose not to use Cloud Spanner due to unknowns. We were more confident we could predict the read and write performance in BigTable (especially due to the constraints you get when you drop relational features).

Re: Breaking Through Scaling Barriers with Bigtable

#16

Post by stangles1 »

bbulkow wrote: post #8
stangles1 wrote: post #6

Earlier quoted context omitted.

To be honest, it never even made it onto our radar, not for any particular reason though :) IIRC, Spanner relies on precise timing to make certain guarantees, which is definitely relevant to our use case. I wonder how its write performance would stack up against Postgres and Bigtable.

Why would it matter how they do it under the covers? You see correct transactions. The interesting tech in this space is aws redshift, in my testing a few years ago they dominated in price performance when you used the then new node types.

>Why would it matter how they do it under the covers?

It doesn't really matter, I suppose I just got nerd-sniped recalling the details of Spanner's internals and their (somewhat superficial) relevance to the issues of timekeeping mentioned in the blog post :)

Re: Breaking Through Scaling Barriers with Bigtable

#17

Post by ntietz »

jpgvm wrote: post #4

Not sure I would have opted for Bigtable here over sharded PostgreSQL because it unduly limits flexibility but within this constrained use case it works fine. The big thing to remember which is covered by this article is your only real performant option to return multiple rows for BT is range scans so your keys should be setup to support this. If you need more than 1 index you are essentially shit out of luck - hence…

We did consider sharding PostgreSQL and opted for something that would be fully managed for us to minimize our management overhead. Managing sharded databases can be complicated business and that wasn't something we wanted to take on if we could avoid it.

Re: Breaking Through Scaling Barriers with Bigtable

#18

Post by bigdubs »

bgm1975 wrote: post #13

I’d be curious to know if cockroachdb was evaluated as a potential candidate. The various literature bits I’ve read seem to indicate its whole reason for existence is to solve these kinds of problems at scale while still providing some semblance of ACID compliance.

CockroachDB targets a different workload, namely, lots of reads and writes of individual records, versus returning large chunks of even larger tables.

So it might help with inserts but would struggle with larger queries.

Re: Breaking Through Scaling Barriers with Bigtable

#19

Post by ithkuil »

ntietz wrote: post #17
jpgvm wrote: post #4

Not sure I would have opted for Bigtable here over sharded PostgreSQL because it unduly limits flexibility but within this constrained use case it works fine. The big thing to remember which is covered by this article is your only real performant option to return multiple rows for BT is range scans so your keys should be setup to support this. If you need more than 1 index you are essentially shit out of luck - hence…

We did consider sharding PostgreSQL and opted for something that would be fully managed for us to minimize our management overhead. Managing sharded databases can be complicated business and that wasn't something we wanted to take on if we could avoid it.

Planetscale does a good job at that, but for MySQL

Re: Breaking Through Scaling Barriers with Bigtable

#20

Post by sitkack »

stangles1 wrote: post #11
jpgvm wrote: post #4

Not sure I would have opted for Bigtable here over sharded PostgreSQL because it unduly limits flexibility but within this constrained use case it works fine. The big thing to remember which is covered by this article is your only real performant option to return multiple rows for BT is range scans so your keys should be setup to support this. If you need more than 1 index you are essentially shit out of luck - hence…

Good points - sharded Postgres is more likely to be a better choice in most instances. I wouldn't be surprised even if sharded Postgres would have worked well for us, but as you mentioned, for this constrained use case, Bigtable works fine.

What is the recommended way to manage and interact with a shared Postgres cluster? Is it a fully connected mesh where every node running business logic talks to every Postgres node?

Would you use chain replication or a hash ring?

Post reply on HN