11–20 of 20 posts
"We also assigned each server a static unique identifier in the form of a woman’s name – a shrinking namespace with thousands of servers." Let me fix that for you; stop gendering your servers.
> While we heavily utilise Helios for container-based continuous integration and deployment (CI/CD) each machine typically has a single role – i.e. most machines run a single instance of a microservice. It's strange to me that this is still so common. My theory is that the "one machine one port" philosophy is still built into a lot of software (monitoring, the ELB, etc). Another is that this is the philosophy we've a…
However given Spotify's business position our priority has yet to shift from providing engineers compute capacity as fast as possible to optimising our usage of said compute capacity. It's all now somewhat of a moot point as we move away from our own hardware into Google's cloud.
What happened to your DNS data? Did you switch to dynamic DNS based upon database data? You talk about how much a burden the manual DNS information was, but then you don't specify how you actually solved it using "automation." Is it all dynamic? Everything use SRV records that have TTL's or are added and removed? Sorry for so many questions, but you made a big deal about how manual "DNS curation" was a bad thing, the…
The greatest public DNS feature since sliced bread is Joyent's new CNS. Tag instances and they are available instantly through a CNAME. It's like the public equivalent of running Hashicorp's Consul. Freaking fantastic and makes me really glad I've stuck with the JPC for my infrastructure.
One has to wonder why they would opt for this. The entire story is a textbook example of where using a cloud would have been immensely better. Instead of leveraging mature pubic cloud offerings, they chose a path that evidently required huge amounts of developer time and resulted in a tremendous amount of pain/wasted time for downstream developers, only to scrap it in the end when they finally realized there's no point in trying to re-implement AWS/GCE. Think clouds are expensive? I'd love to quantify the number of wasted developer-hours resulting from this decision to use physical servers and see how it would stack up against even a very expensive AWS bill.
> Spotify has historically opted to run our core infrastructure on our own private fleet of physical servers (aka machines) rather than leveraging a public cloud One has to wonder why they would opt for this. The entire story is a textbook example of where using a cloud would have been immensely better. Instead of leveraging mature pubic cloud offerings, they chose a path that evidently required huge amounts of devel…
> Spotify has historically opted to run our core infrastructure on our own private fleet of physical servers (aka machines) rather than leveraging a public cloud One has to wonder why they would opt for this. The entire story is a textbook example of where using a cloud would have been immensely better. Instead of leveraging mature pubic cloud offerings, they chose a path that evidently required huge amounts of devel…
Would be good to hear their perspective on this.
> Spotify has historically opted to run our core infrastructure on our own private fleet of physical servers (aka machines) rather than leveraging a public cloud One has to wonder why they would opt for this. The entire story is a textbook example of where using a cloud would have been immensely better. Instead of leveraging mature pubic cloud offerings, they chose a path that evidently required huge amounts of devel…
Depending on their workloads, running their own datacenter(s) might save them millions a month. I know it does in my own case. That said you give up flexibility for the $ savings. They may think the additional flexibility is worth the cost differential at this point. Would be good to hear their perspective on this.
I believe we started building this platform when AWS was very new, and hadn't seen a compelling reason to transition from it to the cloud until now. There's a couple of posts with more details behind our decision to go to GCP, but primarily it was to leverage their data tooling.
What happened to your DNS data? Did you switch to dynamic DNS based upon database data? You talk about how much a burden the manual DNS information was, but then you don't specify how you actually solved it using "automation." Is it all dynamic? Everything use SRV records that have TTL's or are added and removed? Sorry for so many questions, but you made a big deal about how manual "DNS curation" was a bad thing, the…
Look at the section "DNS Pushes".
It sounds like they went to another method for service discovery, then created DNS entries from a DB either dynamically by registering in a zone or just a regular trigger pulled on DB update. Either way, it sounds like they moved the scary stuff to another level/service in the stack.
Also, linters exist for DNS and can be automated even with manual edits. Jenkins + gerrit makes easy work for this.
> Spotify has historically opted to run our core infrastructure on our own private fleet of physical servers (aka machines) rather than leveraging a public cloud One has to wonder why they would opt for this. The entire story is a textbook example of where using a cloud would have been immensely better. Instead of leveraging mature pubic cloud offerings, they chose a path that evidently required huge amounts of devel…