Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Spotify moves its back end to Google Cloud

news.spotify.com

Post reply on HN
Print view View original

141–150 of 397 posts

Re: Spotify moves its back end to Google Cloud

#141

Post by nanoojaboo »

latchkey wrote: post #70
thibaut_barrere wrote: post #60

Earlier quoted context omitted.

I've been relying on multiple Cloud offerings (including PaaS, IaaS etc) outside Google for many years, but for some unclear reason, I cannot manage to consider Google a reliable partner for a type of hosting or another, at this point. I would love to be proven wrong really, because I often look at their GC offering, and I /really/ want to dive in, yet I feel that I would risk having to migrate quickly in 2 weeks at…

I should add, I've been on top of AppEngine for about 4 years now. This is my second business on it.

but what's your businesss (that you're hiring for)

Re: Spotify moves its back end to Google Cloud

#142

Post by latchkey »

simonebrunozzi wrote: post #63
latchkey wrote: post #35

What Google Cloud needs is a better sales story. AWS consistently beats out Google in this respect. This story is welcomed and there should be more success stories published like this. My business is a heavy GC user (AppEngine, Datastore, CloudStorage, BQ, ComputeEngine, ManagedVM) and I couldn't imagine achieving what we have achieved in such a short amount of time on any other platform. We (myself and another guy)…

You could have done the same on AWS (source: I have worked at AWS for 6 years, and know GCP very well)

AWS doesn't have AppEngine or the Datastore. Yes, there is similar products, but they are absolutely not 'the same'.

Re: Spotify moves its back end to Google Cloud

#143

Post by jamesblonde »

jkestelyn wrote: post #132
jamesblonde wrote: post #78

Earlier quoted context omitted.

You can bet your bottom dollar Spotify are getting heavy, heavy discounts. Google are ramping up here in Stockholm, partly to get this account. This has been in the works since before last summer, so they've taken time before announcing it. The precedent for Spotify was switching from Cloudera to Hortonworks a couple of years ago. Rumours are that they got their licenses for close to nothing from Hortonworks, they we…

Just with respect to some details: Spotify changed from being a self-supporting (free) user of Cloudera's open source platform to being a paid support customer of Hortonworks. So, it was never actually a Cloudera customer (despite what was widely "reported"). Disclaimer: Cloudera emp here.

Nice, reinforces the point that Spotify likes and Spotify gets discounts on infrastructure :)

Re: Spotify moves its back end to Google Cloud

#144

Post by nrh »

jamesblonde wrote: post #119

The big point of this move is the hit it will have on on-premises Hadoop offerings: Cloudera, Hortonworks, MapR. It's a massive vote of no-confidence in their offerings. Spotify are effectively ditching a 2000 server, 90 PB Hadoop cluster to go GCE. As Spotify say themselves, that's big news....

Spotifier here. This is an important point. I have nothing bad to say about Cloudera or HWX (disclaimer: we're an HWX customer - we've had a pretty good experience), but I don't really see a compelling reason at this stage to manage your own cluster(s) (HIPAA/regulatory constraints, maybe?)

Getting shared-storage and indepedently operated/scaled compute clusters on top of that storage isn't easily achievable with the standard Hadoop stack, and building that on top of HDFS is non-trivial.

Re: Spotify moves its back end to Google Cloud

#145

Post by tn13 »

nrh wrote: post #92
tn13 wrote: post #37

Earlier quoted context omitted.

One of the reasons I avoid Google is because they have taken lot of pain to make their customer care totally suck. I mean even IRS and Comcast support does better compared to Google. I published a very critical release to my Android app few weeks back and realized that it was not getting published at all. It took me a month to get the problem resolved. I don't want to buy a cloud service where the support might be no…

Spotifier here. One of the big risks that we identified with the Google partnership was exactly this: Google isn't exactly known for awesome customer service. We've been very pleasantly surprised. The cloud team has been pretty awesome to work with, including lots of engineer engineer contact, walk-throughs of systems and code for critical dependencies, and solid support and collaboration. Exceeded our expectations.

I am not surprised. Spotify is a big company. The question is will it work exactly same for Joe's Inc.

Re: Spotify moves its back end to Google Cloud

#146

Post by ariwilson »

QuercusMax wrote: post #137
stickfigure wrote: post #110

Earlier quoted context omitted.

Also worth noting that Google built an automated tool to handle the M/S to HRD datastore migration for you. It was remarkably painless.

Painless eventually; the first versions didn't work at all, at least on the project I was involved with. Fortunately by the time you had to switch, it worked very well.

Being first to migrate to the hot new infrastructure has never worked out for me in any project.

Re: Spotify moves its back end to Google Cloud

#147

Post by rohansingh »

KaiserPro wrote: post #130

Sorry but spotify's infrastructure is not a shiny beacon of correctness. They've compounded mistake upon "Oooshiny lets use that" here is a post where someone dissects spotify's infrastructure bit by bit: http://www.secretbatcave.co.uk/software/docker-and-you/ This isn't an anti google compute, its got some really great features. But, spotify's backend infrastructure ain't one Yes, I appreciate that that post is from…

To be fair, the presentation that that article is using as a reference for Spotify's infrastructure was from 3 years ago, and during most of the presentation I talked about what was wrong with how we were doing things.

How we do things is quite different now. Check out my DockerCon 2014 presentation or our tech blog for details.

Re: Spotify moves its back end to Google Cloud

#148

Post by tn13 »

latchkey wrote: post #41
tn13 wrote: post #37

Earlier quoted context omitted.

One of the reasons I avoid Google is because they have taken lot of pain to make their customer care totally suck. I mean even IRS and Comcast support does better compared to Google. I published a very critical release to my Android app few weeks back and realized that it was not getting published at all. It took me a month to get the problem resolved. I don't want to buy a cloud service where the support might be no…

They have a paid support service that is actually pretty good. It sounds like you're avoiding the generic Google in the same way you'd not buy something from Amazon vs. using AWS.

As I said Google's reputation is what made me not even bother to try. We all know that AWS and AZURE both are state of art and very competitively priced. If Google wants to beat them they have to win the perception war too. (OR reduce prices substantially).

Re: Spotify moves its back end to Google Cloud

#149

Post by nkw »

boulos wrote: post #76
thibaut_barrere wrote: post #60

Earlier quoted context omitted.

I've been relying on multiple Cloud offerings (including PaaS, IaaS etc) outside Google for many years, but for some unclear reason, I cannot manage to consider Google a reliable partner for a type of hosting or another, at this point. I would love to be proven wrong really, because I often look at their GC offering, and I /really/ want to dive in, yet I feel that I would risk having to migrate quickly in 2 weeks at…

For Google Cloud products that have made it to General Availability (GA), we have a one-year deprecation policy ( https://cloud.google.com/terms/ , Section 7.2). That is, if we're going to make some big change, we give you a 1 year heads up. Disclaimer: I work on Compute Engine, but I'm not a lawyer ;).

Promises to do something for a year which you can change or withdraw after 30 days are really promises for 30 days:

"Google may make changes to this Agreement, including pricing (and any linked documents) from time to time. Unless otherwise noted by Google, material changes to the Agreement will become effective 30 days after they are posted, except if the changes apply to new functionality in which case they will be effective immediately. If Customer does not agree to the revised Agreement, please stop using the Services." https://cloud.google.com/terms/, Section 1.7b

Also, Sections 13.1-13.3 means even though Google promises nt to do something, you really don't have any recourse if they do it anyway and cause you damage.

Although in the grand scheme of things, not the worst terms of service document I have read. The mere fact the drafters addressed the issue is worth ... something.

Disclaimer: IAMALBIANYL.

Re: Spotify moves its back end to Google Cloud

#150

Post by nrh »

shockzzz wrote: post #87
vgt wrote: post #36

Earlier quoted context omitted.

I think if Spotify dressed down Google Cloud Platform competition, no matter the merit, there would be many more complaints (ex. see comments on this technical post https://news.ycombinator.com/item?id=11029032 ). Spotify did say that they really enjoy specific aspects of Google Cloud Platform, through their own engineering blog. Spotify engineers have also been independently vocal on Twitter. Netflix has been very s…

Right, I don't think they have to directly attack anyone. It's just a matter of tradeoffs - here are the things the Google Cloud gets us, here's a specific example that hones it in, and here's why that's critical to our business. At the moment, the blog post just says, "Yay we moved to the cloud" and threw out some marketing buzzwords. It should be beyond a shadow of doubt that they got a lower price as a result. Tha…

Spotifier here. Frankly, price is not the biggest factor in a decision like this. If we were going for the lowest cost cloud option, it probably wouldn't be either AWS or Google - there are other providers who are hungrier for business that would be willing to do deep cuts at our scale.

The way we think about this is that there are basically two classes of cloud services: commodities and differentiated services. Commodities are storage/network/compute, and the big players are going to compete on price and quality on these for the foreseeable future (as with most commodities).

The differentiated services stuff is a bit more interesting. Different players have different strengths and weaknesses here - AWS has way, way better capabilities when it comes to administration and access control and identity management, for example (which is actually pretty important when trying to do this in a large org). The places were Google is strong (data platform) are the places that are most important for us as a business.

Compelling: dataproc+gcs, bigquery, pubsub, dataflow Made it safe: high-enough quality, cheap enough.

What more would you like to know?

Post reply on HN