Companies have turned to internal solutions and more scripting to help their team members achieve more consistent deployments and keep step with the increasing demands of competitive markets. However, like many homegrown solutions, the expense ratio measured in both time and dollars is often high: Most organizations aren’t in the business of providing process automation and need to borrow resources so as to style, create, implement and maintain an automation initiative. Often, even after initial success, the answer doesn’t stand the test of time – changes in practices, like the introduction of Agile, require an entire rewrite of the initiative.
Manual vs automated deployments
Using the rough-and-ready definition of manual deployments, it’s easy to ascertain where they fail. It is safe to say that any deployment is manual if it is:
- Slow and inconsistent
- Prone to error and failure
- Lacking in visibility and traceability Keeping this extended definition of a deployment in mind may help as we identify a pattern of practices (described below) that show why manual deployments are slow, error prone and black boxes. These bad practices will provide insight into what’s needed from a system that will operate quickly, be highly successful and supply high traceability.
Development, testing and operations work differently
Developers see manually deploying to a personal test environment slowly and precisely as an inefficient and burdensome practice. Therefore, they will often create shortcuts or scripts to deploy. If the operations team has scripts, they have a tendency to be created by operations for operations. Test teams want to maneuver quickly like developers. Between their desire to avoid false bugs and test the deployment process, they also want to mirror the deliberate, controlled production deployments. Unfortunately, test teams have little pull on developers or production support and sometimes invent a third way of deploying. Because inconsistency across environments is built into the process, repeated failures, on top of the duplication of labor, are inevitable.
Complicated and large deployment steps
Complicated, manual deployments require careful instructions – often in large deployment documents. These documents take many forms, from a Microsoft Word manual to a spreadsheet to a wiki. Often, there’ll be dozens, hundreds or even thousands of steps that must be assigned to someone on the team and then precisely executed. It is very difficult to properly document and record many steps. Likewise, executing those steps correctly and in exactly the right order is difficult and time consuming. The method inevitably becomes slow and susceptible to failure when the deployment instructions become lengthy.
Production access for developers
Many of these patterns have caused production deployment failures, which result in another failure. When there’s trouble in production, the top priority of the deployment team is to repair production. Developers who know the system best are often brought in to explore it and find out which of the steps within the large document were skipped or what tweak wasn’t communicated. More often than expected, the developers are simply given the production passwords so that they can work more quickly. After all, fixing production is of the utmost importance. If asked whether developers have passwords for production, usually almost everyone will say no, as this would be absurd and violate audit requirements. And yet, one developer may claim to have access to production to repair broken deployments. Because the passwords are handed over during a stressful time and typically late at night, the deployment team often forgets they gave out the password. The result is a breakdown in control, audit capability and traceability.
Why are manual deployments so common?
Manual deployments are slow, painful and fail in production; yet, they’re still extremely common within the industry. Why? An easy answer could be that until recently, powerful deployment automation tooling wasn’t available. However, even today, there’s an excellent deal of resistance to automating deployments. Because they’re at the keyboard executing the manual steps, deployment teams feel on top of things. They immediately see all the output and may answer problems appropriately. Operators' feeling of control, alongside organizational inertia, prolongs the utilization of manual deployments, even when those not performing the deployment see serious problems. Also common is the notion that “discipline” and “consistency” can remove the patterns of failure related to manual deployments – but this is often not the case. The issues that we see with manual deployments are often exposed by human error. If discipline and consistency were the answers, then there would be no reason to communicate on an “as broken” basis if every change process change was correctly documented. However, even the most disciplined organizations believe in this method. Humans aren’t the best at performing rote activities. Likewise, executing dozens of steps correctly each time wouldn’t be a problem – that is, if proper attention were paid. The truth is that regardless of how attentive an individual is, they’re going to eventually make an error.
Automated solutions
Manual deployments are broken and cannot be saved by more disciplined deployment engineers. The rote work of executing a deployment should be delegated to an automated system that can execute a process consistently.
Putting automated processes in place
The first element of a successful solution is to translate the massive document containing the deployment steps into an automated system. The same basic process for deployment must be used throughout all the test environments and production. Likewise, only the automated system should be used for deployments, and deployment errors should be corrected by fixing the automation and performing the deployment again. Using the same process in all environments and only fixing failures through redeployments ensures that when a change to the method is required in an early test environment, the automation is updated. Tweaks are not any longer communicated on an “as broken” basis but are instead immediately incorporated. The deployment plan is usually up to date. Because a machine is executing the plan, it’s always followed precisely and with no lag time between steps.
Environment variables
Using the same process for all environments requires factoring out all environment-specific information. This element is critical. A proper definition of an environment can capture key information, including:
- The servers each component of the appliance gets deployed to
- Preconditions for deployment, like approval requirements
- Parameter variances, like passwords, should be handled securely, while others are more visible
Master repository of deployable artifacts
Traceability is vital, and deploying whatever happens to be on the file share is a method that lacks traceability and increases risk. One must know not only that the deployment process worked but that it also worked with the files that are close to being deployed. One or more authoritative repositories should be used to supply a definitive source of uniquely addressable file versions. These repositories should be integrated into, or natively accessible from, the deployment automation system. Example repositories include better build servers, maven repositories, asset management systems or even more robust deployment tools.
Inventory management
When the deployment automation framework is the sole deployer of an application, it knows what versions of every application are in each environment. This information is often leveraged to supply clear information about which build is where, as well as to optimize system deployments, skipping deployments of components already present within the environment.
Access control
In manual deployments, deployment access control is the equivalent of server access control. With an automatic system in situ, this is no longer true. Access to the automation provides deployment rights. The automation systems must themselves be secure and supply role-based security. This approach provides new capabilities. An environment owner, who may have once only approved manual deployments but lacked the skill or permissions to execute them, can now be provided self-service deployment capabilities. Likewise, it could be appropriate to revoke the permissions of some users who were granted broad permissions to servers to perform deployments but were instructed to not otherwise modify the servers.
Maximizing the scope of automation
An automated deployment infrastructure is something the team can build on. Smoke tests are often integrated so that rollbacks occur automatically on test failure. Likewise, application deployment is often paired with automated updates to applications and application server configuration. Deployments to test environments are often automatic based on standing schedules or events. Once deployment automation is in place, these kinds of additional practices can be layered on top of each other. The deployment infrastructure provides the mechanics and traceability. People provide the creative insight to understand what would most benefit their organizations.
Automated deployment solution checklist
- Coordinates deployments of multi-tiered applications
- Models full application version across versioned components
- Tracks which servers are in which environment
- Tracks server roles in the environment (e.g., database server vs web server)
- Per environment parameters (e.g., database password in development vs production)
- Secure management and filtering of passwords/secrets
- Integrated artifact repository
- Integrations with third-party artifact repositories
- Integrates with many or all of your deployment targets
- Extensible with the ability to create custom steps
- Integration with authentication technologies (e.g., LDAP)
- Role-based security
- Maintains logs of all commands executed in deployments
- Inventory tracks which version is where
- Tracks who runs deployments
Business insights
According to IBM, organizations are finding it difficult to release more quickly without adding more people and incurring additional risk. They have described the limitations of manual deployments, common failure patterns, the benefits of automation and a base set of features an automated system should provide:
- Manual deployments are inherently slow and error prone.
- Deployment automation used only in development or only in operations may help one silo but leads to a hand-off where changes to the process may be insufficiently communicated.
- Automated deployments provide superior audit trails.
- An automated deployment infrastructure provides a framework to build upon. Additional activities, such as automated functional testing, can leverage the deployment infrastructure.
- Deployments standardized across environments must still take into account environmental differences. The environment configuration is a key concern.
Conclusion
A single deployment performed by an automated deployment mechanism has low overhead. A release process with low overhead is one that can be repeated frequently. Frequent releases are desirable for many reasons, but the crux of the matter is that frequent releases promote truly Agile software development. Teams that release frequently can deliver valuable features to their users more often and in incremental steps. In doing so, they can gather continuous feedback from these users on the software they are creating and adapt their approach in response. This feedback can be the difference between a product delighting its target audience or missing them completely.
Useful links and resources
- Here’s an article on how to maintain your automated build pipelines.
- Here’s an article on what Mobile DevOps is and why you should care.
- Here’s an article on DevOps testing tools in 2021.
- For discussions, learning and support, join the Codemagic Slack Community.
