I've had similar issues with Microsofts CI implementation. I'll describe some of the problems I've had here, note that the last time I've used it is about half a year ago, maybe things have changed though I doubt it. This was with VisualStudio TeamServices on an instance hosted by Microsoft itself. I deployed self-hosted build agents as they were orders of magnitude faster.
Probably the biggest problem I've had was that build agents are stateful. The agent checksout a repository when the build starts, and removes the directory when it's done. It doesn't however clean up other changes made on the system, such as cache directories for tooling and other data you might move out of your build directory. I believe that builds must be reproducible, and therefore must start with a clean and consistent environment. You don't want builds to fail down the line because a precious job broke the agents environment, or even worse, having jobs succeed only on the agent because some dependent files are left on the agent. Other CI solutions use Docker containers for this to build in, which is great, as they are immutable (or stateless, rather) by default. This is in my opinion the single worst 'feature', making these CI builds distrustful.
Secondly, automating these agents is hard. If you want to automatically scale these self-hosted build agents horizontally based on demand, good luck! When starting an agent for the first time, you've to go through some cumbersome configuration wizard to set it up. There is no simple static configuration file available to use instead, or at least, I wasn't able to find it at the time. To mitigate the previous problem with stateful agents I created a Docker container having an agent in it, which was in fact quite difficult to properly achieve. I automatically restarted these agent containers multiple times a day in order to clean-up any garbage that was unintentionally left by a job. This is a horrible hack to slightly improve their (in my opinion) broken system, but it had the bonus of making these agents more easily scalable.
Another thing is the build configuration. I prefer having a configuration in a simple YAML file in the same repository, to have it as _close_ to the code as possible. Although Microsoft does provide such functionality it doesn't seem to be common to do it this way, and if I remember correctly it's much more difficult to properly set-up and use than for example, a Travis CI configuration. They do provide a graphical interface to configure these build steps though, which is useful when getting started with the system so you can get a feel of what is possible. It provides some graphical blocks/steps to achieve various thighs, such as doing an FTP upload or running a CLI command, with all kinds of weird properties. Maybe that's because Microsofts things are generally less scriptable. But yet again, I prefer a simple script in an in-repository configuration file, because it feels somewhat constrained and limited.
Sadly, it doesn't stop here:
- sometimes jobs just didn't want to stop, and no, bashing the abort button a million times didn't fix things. In some situations it took an hour before jobs were properly killed.
- the interface (VisualStudo TeamServices) is horrible to navigate, have had to wild-click many times to find what I need. I'm sure this will improve once you use it for a longer period though.
- and there are many other inconsistencies and little problems, such as build logs loading half of the time.
Maybe I'm grumpy. Maybe I'm ranting about nothing. I don't know. Maybe things have changed, or will change, at least I hope it will. I didn't have a great time with their CI system, my experience was horrible as you can probably tell by now. It feels like an unfinished product, that was released to quickly just to satisfy customers, so they didn't have to switch to some other platform not owned by Microsoft for their CI needs.
What amazes me is that there are quite a few free alternatives available which are much simpler and more elegant, such as GitLab, Travis and Circle CI. They are easier to navigate, are usually quite quick, don't have the issues I've (and the article have) described. And greatest of all, you can get it up and running in minutes without having to worry about it afterwards.