Live data from Hacker News

Introducing Cloud Spanner, a Global Database Service

cloudplatform.googleblog.com

181–190 of 456 posts

Re: Introducing Cloud Spanner, a Global Database Service

#181

Earlier quoted context omitted.

> It will be interesting if Amazon and Microsoft choose to compete with Google on this service. If we get to the point where you have databases, compute, storage, and connectivity services from those three at equal scale, well that would be a lot of choice for the developers! There are also plenty of choices evolving for developers who aren't looking for hosted solutions (which can sometimes be a showstopper for ente…

As someone who works (in part) in the MS SQL field, is it irrational to be a bit worried about the effects some of these platform advances might have one one's career? For example, being a MSSQL performance tuning expert requires years of experience and probably pays very well, but just the other day I read an anecdotal story where someone switched a large BI database to use columnar indexes, allowing them to replace…

Microsoft has a history of sales and support that will allow them a certain longevity. They also have less "brand hate" than Oracle. I dont think MSSQL is going to be like Sybase any time soon, but I probably wouldn't focus on that stack starting now if you are into the startup or california scene. For many places in the USA, MS is the way to go.

EDIT: Also, most DB users don't need global-scale databases.

Re: Introducing Cloud Spanner, a Global Database Service

#182

Earlier quoted context omitted.

Except that A in CAP has nothing to do with overall system's availability over time and using it as such is just confusing.

But it makes sense in this case. The system guarantees CP. But as customer it looks like you're getting CA as well, because A is so high. If you drink the kool-aid, you get C & A & P. The kool-aid isn't too bad, though if they can measurably guarantee A > 99.999999%, I'm happy to round off to 100% and call it CAP.

The availability is "only" 99.999%, which IMO is still really high!

(I work for Google Cloud)

Re: Introducing Cloud Spanner, a Global Database Service

#183

Earlier quoted context omitted.

The parent comment doesn't seem to specify "consumer level" and the loss of businesses having their own infrastructure is equally troubling. Everyone is putting a lot of eggs in a very small number of baskets.

I would disagree about the character of the situation. This isn't about people putting eggs in a few baskets, it's that it's more efficient to have centralized chicken coops instead of every family in the world owning their own chickens. Now, you could play with that analogy further and see some issues as well, but I don't think the issue here is centralized failure; all these data centers/"clouds" are at least good.…

Monocultures are efficient, but not healthy ecosystems in the long term.

Re: Introducing Cloud Spanner, a Global Database Service

#184

For those trying to compare this with AWS Aurora, Aurora is more a regular database (MySQL / Postgres) engine with a custom data storage plugin that's AWS/ELB/SSD/EFS-aware. Because of this the database engine can make AWS specific decisions and optimizations that greatly boost performance. It supports master-master replication in the same region, master-slave across regions. Global Spanner looks like a different bea…

> Aurora... supports master-master replication in the same region

I don't believe that's true, but I could be mistaken?

(Work at Google, not on Cloud Spanner)

Re: Introducing Cloud Spanner, a Global Database Service

#185
post #148

Earlier quoted context omitted.

The "old way" was sacrificing functionality such as transactions and joins to get scalability (BigTable, DynamoDB). Google tried that a decade ago and found it lacking, this is why Spanner exists in the first place.

Well, I'd say the "old way" is SQL with joins and schemas and transactions, and the "new way" is KV with eventual consistency.

Chronologically, we have: SQL -> NoSQL -> NewSQL

You're both right.

Re: Introducing Cloud Spanner, a Global Database Service

#186
post #95
post #92

Earlier quoted context omitted.

Amazon's Aurora databases seem to be solving the same problem, and are MySQL or Postgres compatible to boot.

Aurora is a 'better MySQL mousetrap', IMO. This is a globally-available, nearly-CAP-beating datastore that powers one of the biggest websites on the internet. It's not quite apples and oranges, but this is definitely a different problem they are solving.

That's vague. AWS also powers huge websites and Amazon is recommending Aurora as the "default choice" for most workloads.[1] There are certainly significant architectural differences but I would say we can definitely make a direct practical comparison.

[1] http://www.computerworld.com/article/2953299/cloud-computing...

Re: Introducing Cloud Spanner, a Global Database Service

#187

Earlier quoted context omitted.

> It will be interesting if Amazon and Microsoft choose to compete with Google on this service. If we get to the point where you have databases, compute, storage, and connectivity services from those three at equal scale, well that would be a lot of choice for the developers! There are also plenty of choices evolving for developers who aren't looking for hosted solutions (which can sometimes be a showstopper for ente…

As someone who works (in part) in the MS SQL field, is it irrational to be a bit worried about the effects some of these platform advances might have one one's career? For example, being a MSSQL performance tuning expert requires years of experience and probably pays very well, but just the other day I read an anecdotal story where someone switched a large BI database to use columnar indexes, allowing them to replace…

I write software as a developer. This is how I earn my livelihood.

Four years ago, I determined that while development work might seem to be near the top of the food chain, there will at some point where my work will be replaced by AIs.

This is not so different from how word processors replaced the specialist job of typesetters. Word processors make "good enough" typesetting. You can still find typesetters practicing their craft; the rest of us use word processors and don't even think about it.

At the time, I was learning to put the Buddhist ideals of emptiness and impermanence to practice, and to become more emotionally aware: the _main_ reason I had thought I would never be replaced by an AI writing software has more to do with wishful thinking and attachment than any clear-sighted look at this.

I also made a decision to work on the technologies to accelerate this. Rather than becoming intoxicated by the worry, anxiety, and existential anguish, I decided try to face it. Fears are inherently irrational, but just because they are irrational does not mean it is not what you are experiencing. Fears are not so easily banished by labeling them as irrational. Denial is a form of willful ignorance.

Now, having said all that, whether our tech base will come to that, who can say?

Since then, I have been tracking things like:

Viv - a chat assistant that can write it's own queries

DeepMind's demonstration of creating a Turing-complete machine with deep learning using a memory module.

I watched a tech enthusiast write a chat bot. He does not write software professionally. Talking with him over the months when he tinkers with in his spare time, I realized that in the future, you won't have as many software engineers writing code; you would learn how to _train_ AIs when they become sufficiently accessible to the masses. Skills in coaching, negotiation, and management becomes more important then some of the fundamental skills supporting software engineering. And like typesetting, I can see development work being pushed down the eco-ladder.

It's not surprising to me to see that Wired article about how coding becoming blue collar work. And even that will eventually be pushed down even further.

It's not surprising to me about Google's site-reliability engineering book, branding, and approach. I have done system admin work in the past, and I can already see traditional, manual sysadmin work being replaced.

It's easy to get nihilistic about this, but that isn't my point here either. I know the human potential is incredible, but I think we have to let go of our self-serving narratives first.

Re: Introducing Cloud Spanner, a Global Database Service

#188

Congratulations to the Spanner team for becoming part of the Google public cloud! And for those wondering, this is why Oracle wants billions of dollars from Google for "Java Copyright Infringement" because the only growth market for Oracle right now is their hosted database service, and whoops Google has a better one now. It will be interesting if Amazon and Microsoft choose to compete with Google on this service. If…

>well that would be a lot of choice for the developers! A sad choice though. The centralization of computation is likely not a good thing in the long run.

> The centralization of computation is likely not a good thing in the long run.

I agree. It only makes sense if you need special data for statistics, AI training, etc.

In all other cases the classic way of programming on pc and notebook is smarter. If you do everything in the cloud, what if you lose Internet connection? I had that experience several times over the last years.

Re: Introducing Cloud Spanner, a Global Database Service

#189

I wonder how this will affect adoption of CockroachDB [1], which was inspired by Spanner and supposedly an open source equivalent. I'd imagine that Spanner is a rather compelling choice, since they don't have to host it themselves. As far as I know, CockroachDB currently does not support providing CockroachDB as a service (but it is on their roadmap) [2]. [1] https://www.cockroachlabs.com/docs/frequently-asked-questi…

(Cockroach Labs CTO here)

Google launching Spanner is generally a positive thing for our industry and our product. It's more proof that what we're aiming for is possible and that there's demand for it. We expect that in five years, all tech companies will be deploying technology like ours.

One of the big differences is that Spanner only uses SQL for read-only operations, with a custom API for writes. We use standard SQL for both reads and writes, which means we also work with major ORMs like GORM, SQLAlchemy, and Hibernate (docs should be live today or tomorrow). Spanner's custom write API will make it difficult to work with existing frameworks, or to convert an existing application to Spanner.

Cloud Spanner only works on Google Cloud and is a black-box managed service. CockroachDB is open source and can be run on-prem or in any cloud on commodity hardware. (We don't offer CockroachDB as a service yet, but may in the future)

At this point, both products are still in beta and are still missing features like back-up and restore (according to the Quizlet blog post). We plan to launch CockroachDB 1.0 with back-up / restore enabled.

* For anyone wanting to know more about how we make CockroachDB work without TrueTime, see our blog post: https://www.cockroachlabs.com/blog/living-without-atomic-clo...

Re: Introducing Cloud Spanner, a Global Database Service

#190
"What if you could have a fully managed database service that's consistent, scales horizontally across data centers and speaks SQL?"

Looks like Google forgot to mention one central requirement: latency.

This is a hosted version of Spanner and F1. Since both systems are published, we know a lot about their trade-offs:

Spanner (see OSDI'12 and TODS'13 papers) evolved from the observation that Megastore guarantees - though useful - come at performance penalty that is prohibitive for some applications. Spanner is a multi-version database system that unlike Megastore (the system behind the Google Cloud Datastore) provides general-purpose transactions. The authors argue: We believe it is better to have application programmers deal with performance problems due to overuse of transactions as bottlenecks arise, rather than always coding around the lack of transactions. Spanner automatically groups data into partitions (tablets) that are synchronously replicated across sites via Paxos and stored in Colossus, the successor of the Google File System (GFS). Transactions in Spanner are based on two-phase locking (2PL) and two-phase commits (2PC) executed over the leaders for each partition involved in the transaction. In order for transactions to be serialized according to their global commit times, Spanner introduces TrueTime, an API for high precision timestamps with uncertainty bounds based on atomic clocks and GPS. Each transaction is assigned a commit timestamp from TrueTime and using the uncertainty bounds, the leader can wait until the transaction is guaranteed to be visible at all sites before releasing locks. This also enables efficient read-only transactions that can read a consistent snapshot for a certain timestamp across all data centers without any locking.

F1 (see VLDB'13 paper) builds on Spanner to support SQL-based access for Google's advertising business. To this end, F1 introduces a hierarchical schema based on Protobuf, a rich data encoding format similar to Avro and Thrift. To support both OLTP and OLAP queries, it uses Spanner's abstractions to provide consistent indexing. A lazy protocol for schema changes allows non-blocking schema evolution. Besides pessimistic Spanner transactions, F1 supports optimistic transactions. Each row bears a version timestamp that used at commit time to perform a short-lived pessimistic transaction to validate a transaction's read set. Optimistic transactions in F1 suffer from the abort rate problem of optimistic concurrency control, as the read phase is latency-bound and the commit requires slow, distributed Spanner transactions, increasing the vulnerability window for potential conflicts.

While Spanner and F1 are highly influential system designs, they do come at a cost Google does not tell in its marketing: high latency. Consistent geo-replication is expensive even for single operations. Both optimistic and pessimistic transactions even increase these latencies.

It will be very interesting to see first benchmarks. My guess is that operation latencies will be in the order of 80-120ms and therefore much slower than what can be achieved on database clusters distributed only over local replicas.

Post reply on HN