Probably a dumb question but if you need all servers to talk to each other to achieve consensus, how do you deal with network latency? Do people just keep data in separate zones so they aren't globally replicated?
NewSQL databases fail to guarantee consistency and I blame Spanner
261–270 of 319 posts
Re: NewSQL databases fail to guarantee consistency and I blame Spanner
#262Earlier quoted context omitted.
Link? It depends what is meant by "commodity" I guess. The largest server AWS offers is only 64 physical cores (128 logical) and less than 4 TB RAM.
http://www.supermicro.com/products/system/4U/8048/SYS-8048B-... Quad socket R1 (LGA 2011) supports Intel® Xeon® processor E7-8800 v4/v3, E7-4800 v4/v3 family (up to 24-Core) Up to 12TB DDR4 (128GB 3DS LRDIMM); 96x DIMM slots (8x memory module boards: X10QBi-MEM2) AFAIK the latest Skylake Xeons ("Scalable" - Platinum/Gold/Silver/Bronze) have regressed to 1.5TB support, see https://ark.intel.com/products/93794/Intel-Xe…
That's a previous-gen CPU system. Current is:
https://www.supermicro.com/products/system/7U/7089/SYS-7089P...
Server itself is might only be $25k but those 224 cores could add another $90k, so the total would be close to $400k.
The previous-gen version, SYS-7088B-TR4FT (link in my comment upthread), has 192 DIMM slots, so if you don't need CPU horsepower, you cn get the lower-density modules and still have 12TB (or the max of 24TB for the price premium!).
Re: NewSQL databases fail to guarantee consistency and I blame Spanner
#263Earlier quoted context omitted.
I think part of this is that most of the common knowledge about scaling is hard fought from the 90s/2000 era. eBay got bigger and bigger Sun boxes to run Oracle, until they couldn't get anything bigger -- then they had a problem and had to shard their listings into categories, etc. In the last few Intel cpu generations, computation performance has had small gains, but addressable memory has doubled about every other…
My experience is that scaling up can lead to owning pets instead of cattle. It's never fun having a 1TB ATS instance that's acting funky but you're terrified of bouncing it. That's more a devil's advocate anecdote than an argument against scaling up.
I think you can (and clearly should) be ready for your big boxes to go down. And it is much more painful to bounce a large box than a small box. But that doesn't mean having say 4x [1] big boxes is strictly worse than having a whole bunch of smaller boxes.
[1] one pair in two locations, basically the minimum amount, unless you're really latency insensitive so you could have one in three locations.
Re: NewSQL databases fail to guarantee consistency and I blame Spanner
#264Earlier quoted context omitted.
I may be a bit of an old fart, but this is the exact reasoning behind my decision to never go with "distributed X" if there's a "single-machine X" where you can just vertically scale. If you can afford 3-5 machines/VMs for a cluster you can almost certainly afford a single machine/VM with 2-4x the resources/CPU and chances are that it'll perform just as well (or better) because it doesn't have network latency to cont…
Does running on 3-5 machines give you the advantage of being able to apply kernel patches and restart machines without bringing down the whole system? That might matter to people with paying customers, even if in theory they could've replaced their 3 machines with one bigger one.
I'm not sure anyone would ever actually have only one machine, without a failover option, and failing over had better not bring down the whole system.
Re: NewSQL databases fail to guarantee consistency and I blame Spanner
#265Earlier quoted context omitted.
> Common languages like node.js, Python, R, OCaml, etc. do NOT let you use all your cores "by default" If the code is written in a distributed fashion from the start then it can be designed so it's one python/node/R process per core. And that makes the transition to distributed servers simpler anyway.
> If the code is written in a distributed fashion from the start then it can be designed so it's one python/node/R process per core. That's a really glib dismissal of how hard the problem is. Python and node have pretty terrible support for building distributed systems. With Python, in practice most systems end up based on Celery, with huge long-running tasks. This configuration basically boils down to using Celery,…
> With Python, in practice most systems end up based on Celery, with huge long-running tasks.
Oh dear... Yeah, that's a terrible distributed system. Interestingly, all the distributed systems I've worked on with Python haven't had Celery as any kind of core component. It's just poorly suited for the job, as it is more of a task queue. A task queue is really not a good spine for a distributed system.
There are a lot of python distributed systems built around in memory stores, like pycos or dask, or built around existing distributed systems like Akka, Casandra, even Redis.
Re: NewSQL databases fail to guarantee consistency and I blame Spanner
#266Earlier quoted context omitted.
CockroachDB's performance is dependent on time synchronization. If a node detects it's too far behind, it will commit suicide. However, before it detects it, there is a possibility of stale reads. https://www.cockroachlabs.com/docs/stable/recommended-produc...
Thanks for pointing that out - I have yet to rtfm and dive deep. I wonder how frequently time sync problems occur in virtual environments after ntp syncing - I've seen pretty erratic behavior on virtual active directory domain controllers even after syncing with hyper-v and vmware.
Here’s their blog post on how they manage to live without atomic clocks. I’ve found at least one assumption of thiers that i don’t agree with, but notwithstanding that, it’s a good read.
https://www.cockroachlabs.com/blog/living-without-atomic-clo...
Re: NewSQL databases fail to guarantee consistency and I blame Spanner
#267Earlier quoted context omitted.
Does running on 3-5 machines give you the advantage of being able to apply kernel patches and restart machines without bringing down the whole system? That might matter to people with paying customers, even if in theory they could've replaced their 3 machines with one bigger one.
I'm not sure anyone would ever actually have only one machine, without a failover option, and failing over had better not bring down the whole system.
But now we're back to a distributed system
Re: NewSQL databases fail to guarantee consistency and I blame Spanner
#268Earlier quoted context omitted.
Spanner's approach requires help from hardware and several full time employees maintaining and ensuring the uncertainty guarantees. This increases the cost of the maintaining the system, which for Cloud Spanner is partially passed on to the end users. If you can build a system that doesn't require time synchronization, yet doesn't have any significant drawbacks relative to what Spanner provides, you'd be better off u…
You clearly have an agenda here, the blog post contains little information while spreading FUD. Never heard of calvin and will now avoid it.
Calvin is a research database, described in a paper and also provided as an open source project, but last I checked, it was pure research, not usable for real work. The "production version" of Calvin is arguably FaunaDB, which is a hosted SaaS product.
Abadi is a very well known and, as far as I know, respected database researcher. No need to avoid his work.
Re: NewSQL databases fail to guarantee consistency and I blame Spanner
#269Earlier quoted context omitted.
Interesting, though I don't see the price or any other purchasing details (at least on that page).
Pretty sure once you're talking about 12TB of memory it goes to "call for pricing".
That seems more like enterprise built and sold specially, and less like commodity hardware.
Re: NewSQL databases fail to guarantee consistency and I blame Spanner
#270Earlier quoted context omitted.
> If the code is written in a distributed fashion from the start then it can be designed so it's one python/node/R process per core. That's a really glib dismissal of how hard the problem is. Python and node have pretty terrible support for building distributed systems. With Python, in practice most systems end up based on Celery, with huge long-running tasks. This configuration basically boils down to using Celery,…
> With Python, in practice most systems end up based on Celery, with huge long-running tasks. Oh dear... Yeah, that's a terrible distributed system. Interestingly, all the distributed systems I've worked on with Python haven't had Celery as any kind of core component. It's just poorly suited for the job, as it is more of a task queue. A task queue is really not a good spine for a distributed system. There are a lot o…
[deleted]