Recommendations for keeping build servers updated
As someone who constantly switches between QA, builds, and operations, I’ve been having trouble with how to handle OS updates on build servers. On Windows, Linux, MacOS, or any other operating system that can be self-updated via the internet, the dichotomy is the same:
- The QA team wants the build server to remain intact from the beginning to the end of the product release cycle, as installing updates can destabilize the server and mean that subsequent builds are not based on the same baseline.
- Operations teams want to deploy software on systems with all the latest security patches installed; This could mean that the software is not deployed on exactly the same version as the operating system on which it was built.
I usually mitigate this by taking release candidate builds and installing them on a test server with a completely up-to-date operating system, repeating automated tests run on the build server and doing some additional system-level testing to make sure everything looks fine before deployment. However, this seems inefficient to me; Does anyone have a better way?
Solution
Personally, I don’t think you have much of a problem here – just apply the latest update to the build server. The main reason I say this is:
It’s unlikely that any dependencies on your code or build server will be so tightly coupled to the OS version that installing regular updates can affect anything, let alone break it. There may be subtle differences between Windows versions, such as window messages, etc., but these differences are few and far between, and are generally well documented on the internet. If you’re using a managed technology stack like WPF/Silverlight or ASP.Net, or even primarily Winforms, you’ll be isolated from these changes—they will only affect you if you use WinAPI directly to create or draw your windows. Button.
It’s a good practice to always design your product for the latest version of the operating system, because you need to encourage your customers to implement these updates as well – IOW You shouldn’t be in a must-say position to tell your clients not to install update xyz because your application won’t run against it – especially if the update is a critical security update
Testing differences between OS versions should be done by the QA team and should be independent of what’s on the build server
You don’t want your build server to go into a state where it’s so isolated from the company update process that when you finally apply them, it vomits and spits molten silicon everywhere. IOW, the longer you wait for updates, the higher the risk of something going wrong with catastrophic consequences. Small and frequent/incremental updates are less risky than massive updates once every decade 🙂
You must be careful that build server updates are third-party control or library updates – they often contain breaking changes or significantly altered behavior. They really should be arranged and then run a round of testing to look for any changes.