Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why do some companies build their own CICD orchestrators?

news.ycombinator.com

Post reply on HN
Print view View original

11–18 of 18 posts

Re: Ask HN: Why do some companies build their own CICD orchestrators?

#11

Post by derefnull »

theamk wrote: post #3

Checkpoint is a deploy system, not a CICD. A quick Google finds a nice description of it: https://slack.engineering/deploys-at-slack/ That said, once you step away from webapp world, a lot of times all existing CI/CD systems are pretty terrible. For example we have pre-merge tests that need a few tens of cpu-hours spread across a dozen machines, and running natively (no docker). This rules out pretty much every CI/CD…

Did you consider leveraging buildbot (buildbot.net) when writing your infrastructure? It has some features that should be helpful.

I'm currently setting up a ci/cd build infrastructure for myself. While I've been exploring different ways to perform the build/test cycle, I expect it will be beneficial to wrap the whole thing in buildbot in the end.

Re: Ask HN: Why do some companies build their own CICD orchestrators?

#12

Post by hansvm »

CICD can get out of hand pretty quickly. If an off-the-shelf solution works for you with minimal changes to your code and workflow then that's fantastic. Ofttimes they're simultaneously clunky and full of unnecessary (to you) features while not supporting the few things you care about. Writing a general purpose CICD probably isn't a good idea for most organizations IMO, but writing a CICD that handles just your use cases is super easy and can take less time than writing the workarounds necessary otherwise. Plus, when things inevitably go wrong the code is right there for you to fix.

The second you notice your engineers are writing anything like custom batching logic to get around CICD limits, custom change detection because the general-purpose solution can't appropriately recognize transitive dependencies, ways out of the CICD sandbox to interact with some raw hardware device with vendored blobs that only work on XP, or anything else fancy then that's a strong indication that you would have been better off doing it yourself from the start.

Re: Ask HN: Why do some companies build their own CICD orchestrators?

#13

Post by theamk »

thrwawy74 wrote: post #6
theamk wrote: post #3

Checkpoint is a deploy system, not a CICD. A quick Google finds a nice description of it: https://slack.engineering/deploys-at-slack/ That said, once you step away from webapp world, a lot of times all existing CI/CD systems are pretty terrible. For example we have pre-merge tests that need a few tens of cpu-hours spread across a dozen machines, and running natively (no docker). This rules out pretty much every CI/CD…

I would think running something within a Docker container is running it natively?

Test build is dozens of gigabytes and millions of files (yeah, we have some suboptimality in how we set up bazel...), so even simple "rm -rf" can take 30+ minutes. Docker really has too high overhead for this, especially in the cleanup stage.

Re: Ask HN: Why do some companies build their own CICD orchestrators?

#14

Post by theamk »

longcommonname wrote: post #7
theamk wrote: post #3

Checkpoint is a deploy system, not a CICD. A quick Google finds a nice description of it: https://slack.engineering/deploys-at-slack/ That said, once you step away from webapp world, a lot of times all existing CI/CD systems are pretty terrible. For example we have pre-merge tests that need a few tens of cpu-hours spread across a dozen machines, and running natively (no docker). This rules out pretty much every CI/CD…

This is possible with github actions. Have one job create a matrix of other jobs for each of your runners. Use self hosted runners if you are worried about costs or have special requirements.

Test build is dozens of gigabytes and millions of files and takes 20+ minutes.

When setting up github actions matrix, we have a choice of either (1) build before every test (+20 minutes per test) or (2) making a huge docker image and upload it, and download it for each worker, which also is 20+ minutes or more.

You can use github actions if you just use github to allocate nodes, and do all the work with your custom test scheduler, but then you are starting on the road of rewriting the CI/CD for yourself.

Re: Ask HN: Why do some companies build their own CICD orchestrators?

#15

Post by cloudfive »

theamk wrote: post #3

Checkpoint is a deploy system, not a CICD. A quick Google finds a nice description of it: https://slack.engineering/deploys-at-slack/ That said, once you step away from webapp world, a lot of times all existing CI/CD systems are pretty terrible. For example we have pre-merge tests that need a few tens of cpu-hours spread across a dozen machines, and running natively (no docker). This rules out pretty much every CI/CD…

There are tons of systems that setup environments on-demand (EaaS) to facilitate "pre-merge" tests. Im sure your problem can be solved without building it all yourself.

Also noteworthy: Slack is going to explain this in more depth next month:

https://www.linkedin.com/video/event/urn:li:ugcPost:69626375...

Re: Ask HN: Why do some companies build their own CICD orchestrators?

#16

Post by cloudfive »

Gravityloss wrote: post #10

It has been tried, for example making diagrams in Rational Rose and generating C++ code from that. Then I assume filling in the code "details" manually. Probably someone else can comment on how that has worked. That was before agile or TDD were such big things.

I was around when UML took off - but the only real success I have seen from automatically generating code (from diagrams or otherwise) has been in the API space recently since those contracts are easy to describe and generate with tools like Swagger

Re: Ask HN: Why do some companies build their own CICD orchestrators?

#17

Post by cloudfive »

Maybe the question is really "why do some companies build a new in-house technology rather than use an existing one"?

1. Some companies are so slow and some teams so agile that creating a new working project takes less time than submitting a ticket for permission to install and demo software and get the required permission to purchase it. 2. Developers hate to deal with salespeople, so unless the CICD platform is OpenSource / free the developer might choose to build something simple. 3. Developers might be unaware that a solution exists to solve their specific problem. 4. The commercial solution to a small developer's problem might be too expensive.

The developer estimates of time and money for building something in-house often neglect the "hidden costs" ... (including the long tail). The hidden costs often include : 1. opportunity costs - a developer can't do two things at once, so if they are writing a tool, they are not releasing new features or fixing defects. 2. Maintenance costs - keeping the thing running (when the host OS is upgraded, the network changes, the repository credentials expire, etc) 3. support costs - if this touches any other outside teams or customers, someone will have issues and need support. Maybe a new team has to be created to handle the Zendesk support tickets, provide Slack help, etc.

Re: Ask HN: Why do some companies build their own CICD orchestrators?

#18

Post by Mandatum »

cloudfive wrote: post #15
theamk wrote: post #3

Checkpoint is a deploy system, not a CICD. A quick Google finds a nice description of it: https://slack.engineering/deploys-at-slack/ That said, once you step away from webapp world, a lot of times all existing CI/CD systems are pretty terrible. For example we have pre-merge tests that need a few tens of cpu-hours spread across a dozen machines, and running natively (no docker). This rules out pretty much every CI/CD…

There are tons of systems that setup environments on-demand (EaaS) to facilitate "pre-merge" tests. Im sure your problem can be solved without building it all yourself. Also noteworthy: Slack is going to explain this in more depth next month: https://www.linkedin.com/video/event/urn:li:ugcPost:69626375...

Or you can read the article: https://slack.engineering/remote-development-at-slack/
Post reply on HN