Automating Datacenter Operations at Dropbox
blogs.dropbox.com
Automating Datacenter Operations at Dropbox
1–10 of 26 posts
Re: Automating Datacenter Operations at Dropbox
#2A theory based on the article is it seems Pirlo may be written in Python (going off the fact that is leverages SQLAlchemy) - which is interesting given most providers are writing new infrastructure code largely in Go (Spinnaker is another Python exception).
I'm guessing that Python is still heavily used inside Dropbox, but does anyone know if they have published any style guides or tooling to managing Python codebases at their scale?
Re: Automating Datacenter Operations at Dropbox
#3A theory based on the article is it seems Pirlo may be written in Python (going off the fact that is leverages SQLAlchemy) - which is interesting given most providers are writing new infrastructure code largely in Go (Spinnaker is another Python exception). I'm guessing that Python is still heavily used inside Dropbox, but does anyone know if they have published any style guides or tooling to managing Python codebase…
Dropbox hired the creator of Python to help migrate their huge Python 2 codebase to Python 3. It would make sense for them to continue with their investment in Python.
Re: Automating Datacenter Operations at Dropbox
#4A theory based on the article is it seems Pirlo may be written in Python (going off the fact that is leverages SQLAlchemy) - which is interesting given most providers are writing new infrastructure code largely in Go (Spinnaker is another Python exception). I'm guessing that Python is still heavily used inside Dropbox, but does anyone know if they have published any style guides or tooling to managing Python codebase…
Dropbox sponsors most of the developers of http://mypy-lang.org/ , an optional static typing system for Python.
I've found it to be hugely useful in safely working in a large Python codebase :)
(NB: I work at Dropbox)
Re: Automating Datacenter Operations at Dropbox
#5A theory based on the article is it seems Pirlo may be written in Python (going off the fact that is leverages SQLAlchemy) - which is interesting given most providers are writing new infrastructure code largely in Go (Spinnaker is another Python exception). I'm guessing that Python is still heavily used inside Dropbox, but does anyone know if they have published any style guides or tooling to managing Python codebase…
Unless I'm mistaken, Guido still works for Dropbox. And as far as I know they still mostly do Python.
Re: Automating Datacenter Operations at Dropbox
#6A theory based on the article is it seems Pirlo may be written in Python (going off the fact that is leverages SQLAlchemy) - which is interesting given most providers are writing new infrastructure code largely in Go (Spinnaker is another Python exception). I'm guessing that Python is still heavily used inside Dropbox, but does anyone know if they have published any style guides or tooling to managing Python codebase…
Dropbox sponsors most of the developers of http://mypy-lang.org/ , an optional static typing system for Python. I've found it to be hugely useful in safely working in a large Python codebase :) (NB: I work at Dropbox)
Curious if you’ve ever tried Cython out. Not quite the same, but a semi-similar end goal to a degree. I started looking into the other day and it might provide some nice improvements to heavier applications.
Re: Automating Datacenter Operations at Dropbox
#7A theory based on the article is it seems Pirlo may be written in Python (going off the fact that is leverages SQLAlchemy) - which is interesting given most providers are writing new infrastructure code largely in Go (Spinnaker is another Python exception). I'm guessing that Python is still heavily used inside Dropbox, but does anyone know if they have published any style guides or tooling to managing Python codebase…
Spinnaker is written in Java
Re: Automating Datacenter Operations at Dropbox
#8"While there are some excellent job queue systems such as Celery, we didn’t need the whole feature set, nor the complexity of a third-party tool. Leveraging in-house primitives gave us more flexibility in the design and allows us to both develop and operate the Pirlo service with a very small group of SREs."
I don't know any more than this paragraph or two explains, but it sounds like NIH syndrome. If you chose to write your own solution just because a 3rd party one was complicated or expensive, you've underestimated the complexity and expense of developing and supporting new software. Not only do you have software developers developing your business products, but now you have software developers developing the IT tools that support the software developers writing the business products.
"Using the network database and configuration tool developed by our Network Reliability Engineering (NRE) team,"
Another custom tool? Network inventory and config management tools do exist already...
"Rather than having engineers manually running tests using playbooks, Pirlo performed an automated sequential battery of tests that reduced the need for hands-on attention and concurrently increased diagnostic accuracy."
Or you could, like, install Jenkins, write your tests, and do all this without writing your own distributed job queue system.
Re: Automating Datacenter Operations at Dropbox
#9"While there are some excellent job queue systems such as Celery, we didn’t need the whole feature set, nor the complexity of a third-party tool. Leveraging in-house primitives gave us more flexibility in the design and allows us to both develop and operate the Pirlo service with a very small group of SREs." I don't know any more than this paragraph or two explains, but it sounds like NIH syndrome. If you chose to wr…
Developers at these big companies get bored & assume they know better than a mature open source solution so like to reinvent a perfectly good wheel basically.
Re: Automating Datacenter Operations at Dropbox
#10A theory based on the article is it seems Pirlo may be written in Python (going off the fact that is leverages SQLAlchemy) - which is interesting given most providers are writing new infrastructure code largely in Go (Spinnaker is another Python exception). I'm guessing that Python is still heavily used inside Dropbox, but does anyone know if they have published any style guides or tooling to managing Python codebase…
> A theory based on the article is it seems Pirlo may be written in Python (going off the fact that is leverages SQLAlchemy)
Sounds like a reasonable guess.
> which is interesting given most providers are writing new infrastructure code largely in Go
Dropbox apparently uses Go a lot, see e.g. https://about.sourcegraph.com/go/go-reliability-and-durabili...
They also use Rust for some performance-critical stuff, e.g. https://news.ycombinator.com/item?id=11282948
That being said, AFAIK originally Dropbox was written mostly in python. Probably there's still lots of that left.