Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NewSQL databases fail to guarantee consistency and I blame Spanner

dbmsmusings.blogspot.com

Post reply on HN
Print view View original

311–319 of 319 posts

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#311

Post by CydeWeys »

mvijaykarthik wrote: post #282
CydeWeys wrote: post #193

Earlier quoted context omitted.

Spanner does guarantee consistency, thanks to its use of hardware atomic clocks and GPS. It's alternatives like CockroachDB that don't have this dedicated hardware that can fail to guarantee consistency if clocks get of sync (a problem that can't happen in Spanner). Spanner is really fast and massively parallelizable.

We recently open-sourced https://github.com/rubrikinc/kronos for the exact same problem. Coincidentally I shared that on Show HN just today: https://news.ycombinator.com/item?id=18037609

Neat. The obvious question would be, how does kronos compare with ntpd? Do they work together, is kronos a replacement, do they solve different problems, etc.? Is it expected that all the servers in a cluster synced by kronos are already running ntpd, and that kronos provides an additional level of reduced skew on top of that?

It'd be great if you could address this somewhere in the top level README on GitHub.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#312

Post by lamontcg »

rtpg wrote: post #260
lamontcg wrote: post #144

Earlier quoted context omitted.

Also an "old fart", I would recommend postgresql first over any of these other databases. Solve the big data scaling problems when you actually have them. One database server with replication and failover is going to still solve 95-98% or more of the use cases on the web.

It unfortunately doesn’t adequately solve “tweak and update the database software in the middle of the day without requiring downtime” situation very well We do rolling releases of software all the time but it’s pretty hard for us to do much optimisation of our DB setup without doing it in the middle of the night because of how all this stuff works.

Where I worked in the past they had a rule to only hire DBAs that could manage schema changes and such without downtime. Proper planning up front can mitigate the need to take a lot of those outages.

They still took a window a year for changes where there was no alternative. That is still seems preferable to me to the kinds of ongoing problems that distributed eventually consistent databases produce. You might have better availability numbers, but if customer service has to spend money to fix mistakes caused by data errors I don't know that is better for the business.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#313

Post by amaccuish »

e12e wrote: post #290
amaccuish wrote: post #249

Earlier quoted context omitted.

Time shouldn't be a massive issue for AD no? It's a vector clock, not a UTC clock. The UTC clock is only used to solve conflicts no?

Why do you say that? I thought kerberos depended on timestamps +/- drift? https://tools.ietf.org/html/rfc4120#section-5.2.3 Or do you mean some other part of AD?

It defaults to 5 minutes. I perhaps wrongly assumed the issue was within 5 minutes.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#315

Post by mvijaykarthik »

CydeWeys wrote: post #311
mvijaykarthik wrote: post #282

Earlier quoted context omitted.

We recently open-sourced https://github.com/rubrikinc/kronos for the exact same problem. Coincidentally I shared that on Show HN just today: https://news.ycombinator.com/item?id=18037609

Neat. The obvious question would be, how does kronos compare with ntpd? Do they work together, is kronos a replacement, do they solve different problems, etc.? Is it expected that all the servers in a cluster synced by kronos are already running ntpd, and that kronos provides an additional level of reduced skew on top of that? It'd be great if you could address this somewhere in the top level README on GitHub.

I've added a section in the README for this. It works in conjunction with NTPD and the time provided by this library has some extra guarantees like monotonicity, immunity to large clock jumps etc (more info in the README).

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#316

Post by cdcfa78156ae5 »

cbsmith wrote: post #265
cdcfa78156ae5 wrote: post #213

Earlier quoted context omitted.

> If the code is written in a distributed fashion from the start then it can be designed so it's one python/node/R process per core. That's a really glib dismissal of how hard the problem is. Python and node have pretty terrible support for building distributed systems. With Python, in practice most systems end up based on Celery, with huge long-running tasks. This configuration basically boils down to using Celery,…

> With Python, in practice most systems end up based on Celery, with huge long-running tasks. Oh dear... Yeah, that's a terrible distributed system. Interestingly, all the distributed systems I've worked on with Python haven't had Celery as any kind of core component. It's just poorly suited for the job, as it is more of a task queue. A task queue is really not a good spine for a distributed system. There are a lot o…

> There are a lot of python distributed systems built around in memory stores, like pycos or dask, or built around existing distributed systems like Akka, Casandra, even Redis.

Cassandra and Redis just mean that you have a database-backed application. How do you schedule Python jobs? Either you build your own scheduler that pulls things out of the database, or you use an existing scheduler. I once worked on a Python system that scheduled tasks using Celery, used Redis for some synchronization flags, and Cassandra for the shared store (also for the main database). Building a custom scheduler for that system would have been a waste of time.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#317

Post by cdcfa78156ae5 »

mercer wrote: post #238
cdcfa78156ae5 wrote: post #213

Earlier quoted context omitted.

> If the code is written in a distributed fashion from the start then it can be designed so it's one python/node/R process per core. That's a really glib dismissal of how hard the problem is. Python and node have pretty terrible support for building distributed systems. With Python, in practice most systems end up based on Celery, with huge long-running tasks. This configuration basically boils down to using Celery,…

Am I correct in assuming Elixir/Erlang does a much better job at this compared to Node/Python/etc., putting aside (what I understand to be) the rather big problem of their relative weakness for computation?

Yes. Erlang is built around distributed message sending and serialization. Python does not have any such things; even some libraries like Celery punt on it by having you configure different messaging mechanisms ("backends" like RabbitMQ, Redis, etc.) and different serialization mechanisms (because built-in pickle sucks for different uses in different ways). Node.js does not come with distributed message sending and serialization either.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#318

Post by ADefenestrator »

nemothekid wrote: post #48

> Systems that guarantee consistency only experience a necessary reduction in availability in the event of a network partition. As networks become more redundant, partitions become an increasingly rare event. And even if there is a partition, it is still possible for the majority partition to be available In my experience, yes network partitions are incredibly rare. However 99% of my distributed ststem partitions hav…

5. Your network isn't partitioned exactly, but someone bumped an ethernet cable and the packet loss has reduced the goodput on that link to a level too low to sustain the throughput you need. With most congestion control algorithms (basically, not-BBR), at 10Gb even 1% packet loss is devastating.

6. Well, basically the hundred other reasons the machine could brown-out enough that things start timing out even though it's sporadically online. Bad drive, rogue process, loose heatsink, etc.

Dead hosts are easy. Half-dead hosts suck.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#319

Post by cbsmith »

cdcfa78156ae5 wrote: post #316
cbsmith wrote: post #265

Earlier quoted context omitted.

> With Python, in practice most systems end up based on Celery, with huge long-running tasks. Oh dear... Yeah, that's a terrible distributed system. Interestingly, all the distributed systems I've worked on with Python haven't had Celery as any kind of core component. It's just poorly suited for the job, as it is more of a task queue. A task queue is really not a good spine for a distributed system. There are a lot o…

> There are a lot of python distributed systems built around in memory stores, like pycos or dask, or built around existing distributed systems like Akka, Casandra, even Redis. Cassandra and Redis just mean that you have a database-backed application. How do you schedule Python jobs? Either you build your own scheduler that pulls things out of the database, or you use an existing scheduler. I once worked on a Python…

> Cassandra and Redis just mean that you have a database-backed application.

Oh there's a lot more to it than that. CRDT's... for example.

Post reply on HN