11–20 of 26 posts
"While there are some excellent job queue systems such as Celery, we didn’t need the whole feature set, nor the complexity of a third-party tool. Leveraging in-house primitives gave us more flexibility in the design and allows us to both develop and operate the Pirlo service with a very small group of SREs." I don't know any more than this paragraph or two explains, but it sounds like NIH syndrome. If you chose to wr…
That's a more pressing topic, I think.
"While there are some excellent job queue systems such as Celery, we didn’t need the whole feature set, nor the complexity of a third-party tool. Leveraging in-house primitives gave us more flexibility in the design and allows us to both develop and operate the Pirlo service with a very small group of SREs." I don't know any more than this paragraph or two explains, but it sounds like NIH syndrome. If you chose to wr…
Developers at these big companies get bored & assume they know better than a mature open source solution so like to reinvent a perfectly good wheel basically.
Most recently I've been rebuilding my own CI stack and never really gave much thought to going back to Jenkins. So I've been asking around and one of the only common complaints I've heard so far is that getting the initial configuration done is painful and generally orthogonal to automation. Plus the documentation is atrocious.
No off-the-shelf product will be a perfect fit, but with CI software I was truly surprised at just how large the gaps were.
So, sure, if you're Dropbox and you want to automate everything Jenkins is almost certainly not the right tool for the job. If Dropbox already had a supported, mature in-house job queue system, why not use it?
Conversely at megacorp, they spent 4+ years claiming to work on deploying a Jenkins (CloudBees) cluster and still came up with bupkis. Our own internal job and message queuing systems were astoundingly bad (and support for internal tools was almost entirely forbidden).
At megacorp I absolutely decried any sort of home grown solution. But if Dropbox were to actually tackle the problems of internal testing and support and come up with mature solutions, why shouldn't they use them?
"While there are some excellent job queue systems such as Celery, we didn’t need the whole feature set, nor the complexity of a third-party tool. Leveraging in-house primitives gave us more flexibility in the design and allows us to both develop and operate the Pirlo service with a very small group of SREs." I don't know any more than this paragraph or two explains, but it sounds like NIH syndrome. If you chose to wr…
I think the part "Leveraging in-house primitives" is the important bit here. It sounds like they have an in-house task queue, so introducing another external one like Celery might add a lot of operational complexity rather than piggy backing on existing infrastructure. As for the reason for the existing infrastructure not being Celery, I suspect at Dropbox's scale tools like Celery start to break down.
> Or you could, like, install Jenkins, write your tests, and do all this without writing your own distributed job queue system.
They likely haven't written their own job queue, they've just used an existing one. Again, the operational overhead of Jenkins is unfortunately very non trivial.
A theory based on the article is it seems Pirlo may be written in Python (going off the fact that is leverages SQLAlchemy) - which is interesting given most providers are writing new infrastructure code largely in Go (Spinnaker is another Python exception). I'm guessing that Python is still heavily used inside Dropbox, but does anyone know if they have published any style guides or tooling to managing Python codebase…
I'm not sure this is true. There's a lot of Java around, Scala is pretty popular too, in many industries C/C++ are the norm still for this sort of code.
Go might be the trend in open source infrastructure projects right now, but a significant amount of that is likely to be inertia from Docker and Kubernetes.
Earlier quoted context omitted.
Developers at these big companies get bored & assume they know better than a mature open source solution so like to reinvent a perfectly good wheel basically.
Or they've actually used Jenkins (or any of these other suggested alternatives). I've used Jenkins personally and professionally. Most recently I've been rebuilding my own CI stack and never really gave much thought to going back to Jenkins. So I've been asking around and one of the only common complaints I've heard so far is that getting the initial configuration done is painful and generally orthogonal to automatio…
"While there are some excellent job queue systems such as Celery, we didn’t need the whole feature set, nor the complexity of a third-party tool. Leveraging in-house primitives gave us more flexibility in the design and allows us to both develop and operate the Pirlo service with a very small group of SREs." I don't know any more than this paragraph or two explains, but it sounds like NIH syndrome. If you chose to wr…
Developers at these big companies get bored & assume they know better than a mature open source solution so like to reinvent a perfectly good wheel basically.
Maybe, just maybe, the engineers in question considered mature open source tools and decided they didn't serve their needs.
"While there are some excellent job queue systems such as Celery, we didn’t need the whole feature set, nor the complexity of a third-party tool. Leveraging in-house primitives gave us more flexibility in the design and allows us to both develop and operate the Pirlo service with a very small group of SREs." I don't know any more than this paragraph or two explains, but it sounds like NIH syndrome. If you chose to wr…
> I don't know any more than this paragraph or two explains, but it sounds like NIH syndrome. If you chose to write your own solution just because a 3rd party one was complicated or expensive, you've underestimated the complexity and expense of developing and supporting new software. Not only do you have software developers developing your business products, but now you have software developers developing the IT tool…
Not to mention the overhead of running Jenkins at the kind of scale they're talking about here. We're not talking about small numbers here.
"While there are some excellent job queue systems such as Celery, we didn’t need the whole feature set, nor the complexity of a third-party tool. Leveraging in-house primitives gave us more flexibility in the design and allows us to both develop and operate the Pirlo service with a very small group of SREs." I don't know any more than this paragraph or two explains, but it sounds like NIH syndrome. If you chose to wr…
^ perfect band name
Earlier quoted context omitted.
> I don't know any more than this paragraph or two explains, but it sounds like NIH syndrome. If you chose to write your own solution just because a 3rd party one was complicated or expensive, you've underestimated the complexity and expense of developing and supporting new software. Not only do you have software developers developing your business products, but now you have software developers developing the IT tool…
> Again, the operational overhead of Jenkins is unfortunately very non trivial. Not to mention the overhead of running Jenkins at the kind of scale they're talking about here. We're not talking about small numbers here.
Jenkins X, for example, is inherently cloud-based and dynamically scaling. If you need a generic job queue to grow with the organization, that's what it does. And you can even pay someone to set you up, reducing lead time.
But there's also a dozen similar systems out there, so you're not even limited to this one. There's lots of choices and really no need to make a new one.
Re-using an existing component to make a new tool is still an independent software development project, which will almost never be as cheap as integrating a finished tool.