Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NewSQL databases fail to guarantee consistency and I blame Spanner

dbmsmusings.blogspot.com

Post reply on HN
Print view View original

281–290 of 319 posts

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#281

Post by mmt »

chx wrote: post #278
mmt wrote: post #262

Earlier quoted context omitted.

That's a previous-gen CPU system. Current is: https://www.supermicro.com/products/system/7U/7089/SYS-7089P... Server itself is might only be $25k but those 224 cores could add another $90k, so the total would be close to $400k. The previous-gen version, SYS-7088B-TR4FT (link in my comment upthread), has 192 DIMM slots, so if you don't need CPU horsepower, you cn get the lower-density modules and still have 12TB (or t…

I edited my answer to indicate that current gen 3.06TB RAM supporting CPUs are not out (yet?)

Right, which is why the current generation is limited to 12TiB for 8S systems, compared to 24TiB for the generation you quoted.

Even previous gen, if 12TiB main memory is your goal (NUMA concerns aside), it's probably worth going for the 8S system instead of the 4S one, since that's a savings of $144k, and 8 slower/fewer-core CPUs might even be cheaper than 4 that have twice the performance.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#282

Post by mvijaykarthik »

CydeWeys wrote: post #193
mehrdadn wrote: post #88

Earlier quoted context omitted.

I would understand if the complaint was that Spanner is too slow without expensively accurate clocks and synchronization. But the complaint is that Spanner fails to guarantee consistency, which doesn't make sense to me. The requirements clearly include giving a valid clock bound, so if you give an invalid clock bound, it's clearly your fault for getting incorrect results, not Spanner's!

Spanner does guarantee consistency, thanks to its use of hardware atomic clocks and GPS. It's alternatives like CockroachDB that don't have this dedicated hardware that can fail to guarantee consistency if clocks get of sync (a problem that can't happen in Spanner). Spanner is really fast and massively parallelizable.

We recently open-sourced https://github.com/rubrikinc/kronos for the exact same problem. Coincidentally I shared that on Show HN just today: https://news.ycombinator.com/item?id=18037609

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#283

Post by qaq »

paulddraper wrote: post #269
jandrese wrote: post #192

Earlier quoted context omitted.

Pretty sure once you're talking about 12TB of memory it goes to "call for pricing".

That seems more like enterprise built and sold specially, and less like commodity hardware.

To me things available from grey box x86 vendors like supermicro or lenovo look like commodity hardware.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#284

Post by bostik »

wting wrote: post #246
bostik wrote: post #217

Earlier quoted context omitted.

> what is my service level objective? There are environments where flat time distribution for SLO calculation is not acceptable. ( cough betting exchange) If your traffic patterns are extremely spiky, such as weekly peaks hitting 15-20x of your base load, and where a big chunk of your business can come from those peaks, then most normal calculations don't apply. Let's say your main system that accepts writes is 10 mi…

Time based SLOs definitely have their limitations, but in this instance isn't it fairly easy to redefine the SLO in terms of requests rather than time?

I wish it was that easy - our teams have their targets for p99 and p995 ratios but they cannot capture the overall user experience. For us it's not just the ratio of failed requests, but closer to a four-tuple of:

  * maximum number of users affected
  * maximum time of unavailability
  * maximum observed latency 
  * highest ratio of failed requests over a sequence of relatively tight measurement windows
Those are demanding constraints, but such is reality when peak trading activity can take place within just a few minutes. If users can not place their trades during those short windows, they will quickly lose confidence and take their business elsewhere.

So yes, request ratio is certainly a good part of the overall SLO but covers only a portion of the spectrum.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#285

Post by e12e »

xapata wrote: post #38
dgreensp wrote: post #21

This is the clearest presentation of why CAP is misleading that I’ve ever seen! Wonderful. If the app programmer knows they don’t have global consistency, just consistency per partition, I wonder if in practice there are ways to achieve the necessary application-level guarantees such as in the photo-sharing case (not that it sounds that appealing to need to do so).

> application-level guarantees If your database doesn't "guarantee" consistency, but instead provides eventual consistency, your application needs to be aware of how long "eventually" is. In effect, you'd be coding something similar to Spanner's timestamp uncertainty estimation. In the photo sharing example, it'd be reasonable to put a delay on sharing new photos with other users until after permissions changes would…

> delay on sharing new photos with other users until after permissions changes would have propagated.

And then you'd maybe have to map the ux of changing permissions on an existing permission to a three-state: user makes change;ui shows change in progress;change takes effect and is reflected in ui.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#286

Post by e12e »

mercer wrote: post #238
cdcfa78156ae5 wrote: post #213

Earlier quoted context omitted.

> If the code is written in a distributed fashion from the start then it can be designed so it's one python/node/R process per core. That's a really glib dismissal of how hard the problem is. Python and node have pretty terrible support for building distributed systems. With Python, in practice most systems end up based on Celery, with huge long-running tasks. This configuration basically boils down to using Celery,…

Am I correct in assuming Elixir/Erlang does a much better job at this compared to Node/Python/etc., putting aside (what I understand to be) the rather big problem of their relative weakness for computation?

I believe with the combination of native functions ("NIFs") in rust and some work on the nif interface (to avoid making it so easy to take down the whole beam VM on errors) - you might get more of best of both worlds today - than you used to. As you say erlang itself is rather slow wrt compute.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#287

Post by e12e »

qaq wrote: post #154
toast0 wrote: post #101

Earlier quoted context omitted.

I think part of this is that most of the common knowledge about scaling is hard fought from the 90s/2000 era. eBay got bigger and bigger Sun boxes to run Oracle, until they couldn't get anything bigger -- then they had a problem and had to shard their listings into categories, etc. In the last few Intel cpu generations, computation performance has had small gains, but addressable memory has doubled about every other…

You can get commodity x86 server with 12TB RAM and 224 cores.

Anyone have experience with mainframes when it comes to this level of cost/performance? I suppose 500k USD is still too "cheap" to consider more special hw/systems - but where's the cut-off when getting a monster running db2 and Code in a Linux VM or something?

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#288

Post by mercer »

e12e wrote: post #286
mercer wrote: post #238

Earlier quoted context omitted.

Am I correct in assuming Elixir/Erlang does a much better job at this compared to Node/Python/etc., putting aside (what I understand to be) the rather big problem of their relative weakness for computation?

I believe with the combination of native functions ("NIFs") in rust and some work on the nif interface (to avoid making it so easy to take down the whole beam VM on errors) - you might get more of best of both worlds today - than you used to. As you say erlang itself is rather slow wrt compute.

Thankfully it's not an issue for me. Elixir/Erlang is pretty much perfect for most of my use-cases :). But I foresee a few projects where NIFs or perhaps using Elixir to 'orchestrate' NumPy stuff might be useful. Most of my work would remain on the Elixir side though.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#289

Post by e12e »

rtpg wrote: post #260
lamontcg wrote: post #144

Earlier quoted context omitted.

Also an "old fart", I would recommend postgresql first over any of these other databases. Solve the big data scaling problems when you actually have them. One database server with replication and failover is going to still solve 95-98% or more of the use cases on the web.

It unfortunately doesn’t adequately solve “tweak and update the database software in the middle of the day without requiring downtime” situation very well We do rolling releases of software all the time but it’s pretty hard for us to do much optimisation of our DB setup without doing it in the middle of the night because of how all this stuff works.

I would assume you have a staging system to test fixes/updates on, and that rolling out updates would take the form of updating a read slave, then doing a cutover?

Or are you talking about other kinds of tweaks?

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#290

Post by e12e »

amaccuish wrote: post #249
jgalentine007 wrote: post #57

Earlier quoted context omitted.

Thanks for pointing that out - I have yet to rtfm and dive deep. I wonder how frequently time sync problems occur in virtual environments after ntp syncing - I've seen pretty erratic behavior on virtual active directory domain controllers even after syncing with hyper-v and vmware.

Time shouldn't be a massive issue for AD no? It's a vector clock, not a UTC clock. The UTC clock is only used to solve conflicts no?

Why do you say that? I thought kerberos depended on timestamps +/- drift?

https://tools.ietf.org/html/rfc4120#section-5.2.3

Or do you mean some other part of AD?

Post reply on HN