Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud Spanner is now half the cost of Amazon DynamoDB

cloud.google.com

Post reply on HN
Print view View original

351–360 of 371 posts

Re: Google Cloud Spanner is now half the cost of Amazon DynamoDB

#351

Post by jopsen »

airstrike wrote: post #107
nameless912 wrote: post #15

And for many projects, Postgres is still cheaper than both. Having used both, I would much, much rather do the work to fit my project in Postgres/CockroachDB than use either Spanner or DynamoDB, which have WAY more footguns. Not to mention sudden cost spikes, vendor lock in, and god knows what else. AWS and GCP (and Azure, and Oracle cloud, and bare Kubernetes via an operator, and...) support Postgres really well. Ju…

Except for projects for which NoSQL is a better fit than a RDBMS, no? If I'm writing a chat app with millions of messages and very little in the way of "relationships", should I use Postgres or some flavor of NoSQL? Honest question.

> If I'm writing a chat app with millions of messages...

Once you have millions of messages, maybe consider moving the data intensive parts out if postgres, if necessary.

The criticism is often that people look for big data solutions, before they have big data.

If you scale out of postgres, you probably have enough users and money that you can fix it :)

But moving to a NoSQL before you have to, might just slow down development velocity -- also you haven't yet learned what patterns users have.

Re: Google Cloud Spanner is now half the cost of Amazon DynamoDB

#352

Post by spiffytech »

CommanderData wrote: post #323
sparc24 wrote: post #322

Earlier quoted context omitted.

It's amazing how people complain about GCP. We run a massive deployment across 100+ regions cross-cloud GCP, Azure, AWS and oh boy. GCP has good support if you are big enough. Azure though which has a much bigger share than GCP is horrendous. Absolutely garbage all around. Good luck ever getting anyone in Engineering even if you are paying for support. AWS on the other hand - Amazing. We have Ent Support so those guy…

Can you give Azure specifics, as you know Azure has a massive offering. My experience has been the opposite though not without issues, Azure has some of the best corporate and security features of any cloud and it's only getting better. The zero trust model fits in so nicely with their identity platforms it's a sight to behold compared to other cloud providers which likely use some form of AAD or AD DS anyway. Their…

A memorable example is when we ran a heavy Azure Functions workload on our App Service Plan, the hosts would devour themselves.

Functions use containers under the hood. Each invocation created a new container, and when enough of them ran long enough, the host disk would fill up. (Pretty sure our workload wrote almost nothing to disk.)

An internal Azure disk clean-up routine kicked in, which deleted image layers for running Functions. This deleted the filesystems for containers that were still running, yanking them out from underneath the running processes. It also meant the host couldn't launch new instances of our Functions.

At this point the host was poisoned and couldn't launch any new work, even after the workload was reduced, It had to be terminated and replaced, after we detected the problem manually.

Azure support never weemed to take the problem seriously, and after we migrated our workload off of Functions they decided the problem must be resolved since we weren't complaining anymore.

Re: Google Cloud Spanner is now half the cost of Amazon DynamoDB

#353

Post by nameless912 »

endorphine wrote: post #160
nameless912 wrote: post #150

Earlier quoted context omitted.

I don't think the API you interface with fundamentally changes the point that Spanner is hard to recommend from an engineering perspective at anything except the absolute most massive of scales, and even then it will create nearly as many problems as it solves. I'm not saying spanner is _wrong_ or shouldn't exist, but it's very difficult to be in the position where Spanner is the critical key to your application's su…

Care to elaborate on what are the problems that it will create? Honest question.

Sure. Spanner is expensive, and your primary job as an engineer (if you work for an enterprise like most of us do) is to generate business value. So, if nothing else, you will run into the cost problems of Spanner. There are also other problems; iirc both DynamoDB and Spanner shard their key spaces, and each shard gets the same quota, and the key space shards all have to be the same size. This means that even though you might have paid for 1000rps, for example, that RPS volume is divided across all your shards, so if you have one part of the key space that gets way more volume than another you end up eating up the fractional capacity of that shard way faster than you intend and you have to either overprovision or queue requests, both of which are not ideal.

At a previous job, we ended up creating a very complicated write through cache system in front of spanner that dynamically added memory/CPU capacity as needed to prevent hot shards; our application was extremely read heavy, and writes were relatively low RPS, so this ended up working OK, but we were paying tens of thousands of dollars a month for Spanner plus tens of thousands of dollars a month for all the compute sitting in front of it. I don't think we ended up doing much better than if we had bitten the bullet and run clustered Postgres because our write volume ended up being just a few hundred RPS, even though the read volume was 1000x that. Postgres behind this cache system would have handled the load just as well and cost less than half as much.

The other thing that frustrates me personally about Spanner is that Google's docs are incomplete (as usual); there are lots of performance gotchas like this that exist throughout the entire service, and they aren't clearly documented (unlike, to their credit, AWS with Dynamo, who explains this entire problem very clearly and has an [expensive] prebuilt solution for it in the form of the DynamoDB accelerator).

Re: Google Cloud Spanner is now half the cost of Amazon DynamoDB

#354

Post by nameless912 »

GabeWeiss_ wrote: post #220
jpgvm wrote: post #148

Earlier quoted context omitted.

Golden Rule of data: Use PostgreSQL unless you have an extremely good reason not to. PostgreSQL is extremely good at append-mostly data, i.e like a chat log and has powerful partitioning features that allow you to keep said chat logs for quite some time (with some caveats) while keeping queries fast. Generally speaking though PostgreSQL has powerful features for pretty much every workload, hence the Golden Rule.

100% this, and even though I work for Google I absolutely agree. BUT, for the folks that need it, PostgreSQL just DOESN'T cut it, so it's why we have databases like DynamoDB, Spanner, etc. Arguing that we should "Just use PG" is kinda a moot point.

I think I said this in another comment, but I'm not shitting on Spanner or DDB's right to exist here. Obviously, there are _some_ problems for which a globally distributed ACID compliant SQL-compatible database are useful. However, those problems are few and far between, and many/most of them exist at companies like Google. The fact is your average small to medium size enterprise doesn't need and doesn't benefit from DDB/Spanner, but "enterprise architects" love to push them for some ungodly reason.

Re: Google Cloud Spanner is now half the cost of Amazon DynamoDB

#355

Post by RedlineTriad »

bennyg wrote: post #243
iot_devs wrote: post #233

Earlier quoted context omitted.

No need to guess when you can measure. I am running https://cloud-canary.com a service where I monitor AWS primary services for latency and availability. It comes with a lot of data. For instance this is the latency I see doing operations against Dynamo. https://cloudcanary.grafana.net/public-dashboards/c53e2092d6...

Nice dash - if you don't mind a drive-by recommendation: I use Grafana for work a lot and it's nice to see a table legend with min, max, mean, and last metrics for these kinds of dashboards. Really makes it easy to grok without hovering over data points and guessing.

What is more important for me when using Grafana (though a summary is as well) is actually units, to know if it's second, millisecond, microsecond, and also if 0.5 is a quantile or what.

Numbers without units are dangerous in my opinion.

Re: Google Cloud Spanner is now half the cost of Amazon DynamoDB

#356

Post by michellegienow »

benmanns wrote: post #10

I wish I could play around with Spanner for personal/side projects, but a production ready instance starts at $65/mo. DynamoDB can run for ~$0.00/month with per-request pricing.

if you are interested in spanner, you might take a look at cockroachdb. esp the production ready serverless offering which is pay for consumption only. crdb architecture is essentially spanner under the bonnet, GIFFE

https://www.cockroachlabs.com/get-started-cockroachdb/

Re: Google Cloud Spanner is now half the cost of Amazon DynamoDB

#357

Post by michellegienow »

jdmg94 wrote: post #244
dchest wrote: post #141

"Organizations of all sizes and across all industries are increasingly looking to accelerate digital transformation and power AI-driven innovation." How did Google become like this?

They removed that line about not being evil

^^^ this

Re: Google Cloud Spanner is now half the cost of Amazon DynamoDB

#358

Post by planckscnst »

rkeene2 wrote: post #78
joshuanapoli wrote: post #33

Earlier quoted context omitted.

Exactly: I have to provision for peak throughput on Spanner. Average throughput is much lower than the peak throughput, so I'm doubtful of seeing savings no Spanner. (But I bet that Spanner is much easier than DynamoDB to develop with...)

You can scale Spanner up/down based on demand although there is a lag time with it. I built a system that relies on a high-performance database and tested with both AWS DynamoDB and Google Cloud Spanner (see disclaimer) and was able to scale Google Cloud Spanner much higher than AWS DynamoDB. DynamoDB is limited to 1000 WRUs per node, and there isn't an obvious way to get more than 100 nodes per table, so you're limi…

Was that test before this announcement?

https://aws.amazon.com/about-aws/whats-new/2019/05/amazon-dy...

Re: Google Cloud Spanner is now half the cost of Amazon DynamoDB

#359

Post by bananapub »

nerpderp82 wrote: post #260
falcolas wrote: post #246

Earlier quoted context omitted.

We have an old "quiet part out loud" corporate story. It's about how one arm of Google using our service and wondering why it had so much downtime, only for us to point at their GAE arm and say "when they're down, we're down". They went and talked to GAE and - funny enough - were able to correlate the downtime they observed with GAE downtime. GAE uptime improved, for a little while. Yeah, we're on AWS now too.

Does Google run on GCP?

no

Re: Google Cloud Spanner is now half the cost of Amazon DynamoDB

#360

Post by bananapub »

GabeWeiss_ wrote: post #265
nerpderp82 wrote: post #260

Earlier quoted context omitted.

Does Google run on GCP?

Parts, yes. In reference to the specifics mentioned in here though, those services run on Infra Spanner, not Cloud Spanner, but they're the same stack. The main reason things like Gmail, Ads, etc haven't swapped into GCP is because of the internal tooling that's built up around the infra spanner relating to those services specific to Google that don't make sense in Cloud Spanner.

this is maybe the most ill-informed post about google I've ever seen on this site, wow
Post reply on HN