Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NewSQL databases fail to guarantee consistency and I blame Spanner

dbmsmusings.blogspot.com

Post reply on HN
Print view View original

301–310 of 319 posts

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#301

Post by cbsmith »

pmart123 wrote: post #299
cbsmith wrote: post #265

Earlier quoted context omitted.

> With Python, in practice most systems end up based on Celery, with huge long-running tasks. Oh dear... Yeah, that's a terrible distributed system. Interestingly, all the distributed systems I've worked on with Python haven't had Celery as any kind of core component. It's just poorly suited for the job, as it is more of a task queue. A task queue is really not a good spine for a distributed system. There are a lot o…

Well, Celery uses Rabbit-MQ and typically Redis underneath. Rabbit-MQ to pass messages, and Redis to store results. You can scale up web servers to handle more requests, which then uses Celery to offload jobs to different clusters.

Yeah, but fundamentally Celery is a task queue. You don't build a distributed system around that.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#302

Post by ultraluminous »

joemag wrote: post #231
ultraluminous wrote: post #218

Earlier quoted context omitted.

There's a substantial difference between High Availability (HA) clustering and Load Sharing (LS) clustering. In an HA cluster a production database can run on a single active node while replicating to a separate standby node. Network partitions will cause a lag in replication but will not present an external consistency issue as there's still only one active node at any time. When the partition is resolved the replic…

Agreed that such setups can be simpler than multi-master replication. Yet, there's still a hard distributed systems challenge even in the active/passive setup like you describe. When a failure occurs, the clients need to agree which host is the master and which host is the slave. And in the presence of network partitions, it can be impossible to say if the old master is dead, or partitioned away, and hence still happ…

You can also just configure the client to point to a single db node and have an applicative monitoring based update to the db connection string upon failover. Then you're not actually dependent on your client loadbanacing logic. It's very robust and if you follow CQRS also highly available.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#303

Post by joemag »

ultraluminous wrote: post #302
joemag wrote: post #231

Earlier quoted context omitted.

Agreed that such setups can be simpler than multi-master replication. Yet, there's still a hard distributed systems challenge even in the active/passive setup like you describe. When a failure occurs, the clients need to agree which host is the master and which host is the slave. And in the presence of network partitions, it can be impossible to say if the old master is dead, or partitioned away, and hence still happ…

You can also just configure the client to point to a single db node and have an applicative monitoring based update to the db connection string upon failover. Then you're not actually dependent on your client loadbanacing logic. It's very robust and if you follow CQRS also highly available.

Except propagating a configuration update to multiple clients, without a split brain is a hard distributed systems problem :). Especially in the presence of those pesky network partitions.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#304

Post by jules »

voidmain wrote: post #272
uluyol wrote: post #250

Earlier quoted context omitted.

Without knowing exactly how reads and writes are replicated, I am skeptical. I have gone through the technical documentation and haven't found many details. For reference, I've done work in storage and consensus algorithms and I can tell you for a fact that without using a consensus algorithm for either reconfiguration or request propagation, you will have consistency violations. I would love to be proven wrong, as m…

FoundationDB uses a consensus algorithm for reconfiguration, but not in the (happy path) transaction pipeline. It provides (by default) strict serializability (i.e. serializability and external consistency/linearizability) for arbitrary, ad hoc, interactive transactions, and it's expected to provide excellent performance when every single transaction is cross node (so e.g. indexes can be efficiently updated this way)…

How does FoundationDB do distributed transactions without consensus?

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#305

Post by AtlasBarfed »

The context of this article still seems to constrain consistency to guarantees for single atomic units of data (atomic at the node level).

But the second you have multiple-node transactions (your data after all in a truly distributed system will exist on mulitple nodes), then the single-node reliability become dependent on multiple data exchanges across networks for confirmations. Then your network partition exposure starts to snowball.

Same thing for joins. Great if your joins are perfectly served by a node-local sharding strategy, but when you retrieve from multiple nodes, again your network partitioning risk starts to compound.

In AWS you will see noisy neighbor networks and network unreliability. Your VMs get yanked. Your EBS might experience an IO pause. GC pauses. Or simply one of your nodes might get swamped by a query spike.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#306

Post by AtlasBarfed »

Quekid5 wrote: post #78
nemothekid wrote: post #48

> Systems that guarantee consistency only experience a necessary reduction in availability in the event of a network partition. As networks become more redundant, partitions become an increasingly rare event. And even if there is a partition, it is still possible for the majority partition to be available In my experience, yes network partitions are incredibly rare. However 99% of my distributed ststem partitions hav…

I may be a bit of an old fart, but this is the exact reasoning behind my decision to never go with "distributed X" if there's a "single-machine X" where you can just vertically scale. If you can afford 3-5 machines/VMs for a cluster you can almost certainly afford a single machine/VM with 2-4x the resources/CPU and chances are that it'll perform just as well (or better) because it doesn't have network latency to cont…

Then you have a huge single point of failure. AP distributed systems assume you will lose nodes and force you to deal with the reality of them happening.

So you are either accepting the downtime or planting your head in the ground in denial.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#307

Post by tssva »

placebo wrote: post #271
dvirsky wrote: post #255

Earlier quoted context omitted.

Luxury! My first computer (Sinclair ZX Spectrum) had 48KB RAM total, and IIRC 8KB of BASIC RAM available.

Guess I win this round, unless there are older farts than me - Commodore VIC-20 with 3.5KB RAM which was plenty for me to do really cool things with in BASIC and 6502 machine language)

My first computer was a TRS-80 Model 1 with Level 1 BASIC. 4KB of RAM and 4KB of ROM.

It predated the VIC-20 by just under 3 years. You might have had 3.5KB free for BASIC programs but the VIC-20 had a comparatively spacious 5KB of RAM and 20KB of ROM.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#308

Post by guillon »

wikibob wrote: post #151
CydeWeys wrote: post #42

It's definitely true that putting the burden of consistency on developers (instead of on the DB) results in a lot more tricky work for developers. On my project, which started six years ago, we use Cloud Datastore, because Cloud Spanner hadn't come out yet. It results in complicated, painful code that would be completely unnecessary with stronger transactional guarantees. Some examples: https://github.com/google/nomu…

I’m guessing that Nomulus is your project? I ran across it before and just wanted to say it’s really cool that this is open sourced.

Any idea if any company is planning to create an offer dedicates To .BRAND new gTLDs?

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#309

Post by rtpg »

e12e wrote: post #289
rtpg wrote: post #260

Earlier quoted context omitted.

It unfortunately doesn’t adequately solve “tweak and update the database software in the middle of the day without requiring downtime” situation very well We do rolling releases of software all the time but it’s pretty hard for us to do much optimisation of our DB setup without doing it in the middle of the night because of how all this stuff works.

I would assume you have a staging system to test fixes/updates on, and that rolling out updates would take the form of updating a read slave, then doing a cutover? Or are you talking about other kinds of tweaks?

The issue we have is that the cutover is super hard to do in a zero-downtime way because we almost always have active connections and you can’t have multiple primaries. There’s tricks like read connections on the replicas but it’s still really hard to coordinate.

Given that Postgres has built in synchronous replication, I feel like it should also have some support for multiple primaries (during a cutover window) to allow better HA support

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#310

Post by placebo »

tssva wrote: post #307
placebo wrote: post #271

Earlier quoted context omitted.

Guess I win this round, unless there are older farts than me - Commodore VIC-20 with 3.5KB RAM which was plenty for me to do really cool things with in BASIC and 6502 machine language)

My first computer was a TRS-80 Model 1 with Level 1 BASIC. 4KB of RAM and 4KB of ROM. It predated the VIC-20 by just under 3 years. You might have had 3.5KB free for BASIC programs but the VIC-20 had a comparatively spacious 5KB of RAM and 20KB of ROM.

I did start earlier than the VIC-20 (with TRS-80 Model 2 at school) but you definitely take the title :)
Post reply on HN