Avoid 911 Production Upgrade Failures: Painless Upgrade Tests

Upgrade Tests with Ravello

Software product upgrades are nothing new. But with the spread of continuous development and continuous integration they are becoming more and more frequent.  Typically, the upgrade process takes one or more specific components and upgrades them to a new version, for example, to a newer code or configuration version.  Whenever there is a need to perform an upgrade, there are several factors to consider to ensure that the product works correctly.

Potential vulnerabilities

One key consideration is downtime – Is the downtime resulting from the upgrade process acceptable?  Upgrades can involve some downtime.  If availability is critical, as is the case for most online applications today, then it is important that the upgrade process be seamless and errorless.  Even in cases where some downtime is acceptable, it is important that it be capped and kept to a minimum.

There are also resiliency issues that need to be addressed. For example, if a request is issued exactly when the upgrade is in initiated, how is the interrupted request handled? Is it picked up later on?  If not, what are implications for the system?

Backward compatibility  can also be an issue, for example, if a system exposes some kind of API.  It is important to make sure that the API does not “break”, otherwise external users of the API might not be able to continue to use it.  This can create problems for system integration.

Data migration is a particularly painful issue. If the upgrade includes changes in the data schema, then the data itself must also undergo migration.  Data migration tends to be prone to errors, and the successful migration of data should be carefully verified.

Learn more: Cloud-based Dev & Test For Enterprises: Agile and Elastic

Testing as a tool for smoother migration

The overall functionality and stability need to be verified before upgrading your production environment. If something isn’t quite right with even a single component, the functionality and availability of the entire system might be compromised.

Manual testing can be very time consuming. For this reason, tests should be automated. Upgrade tests should be performed as  part of the Continuous Integration process so that any problems are detected as early as possible in the development process.

Best practice is to ensure seamless upgrade with zero downtime is to make sure that all application servers are stateless (instead of stateful). The best way to discover stateful servers is through upgrade tests, purely functional tests do not detect this kind of issue.

For upgrade testing to be reliable, it needs to be run on an environment that closely mirrors the production environment. There are a lot of environment-sensitive components that affect upgrading, for example, load balancing and database configuration.

4 Tips for planning your upgrade testing

In planning for your next upgrade, consider these stages in developing your upgrade test efforts:

  1. Identify the relevant upgrade scenarios
  2. Set up the test environment, with simulated data
  3. Develop the test infrastructure and the actual tests
  4. Run the test on the test environment as part of continuous integration

Ravello’s ability to replicate your production environment provides a feasible option for creating a cloned environment for testing.  In a future post, we will describe how we perform migration tests at Ravello.

Instant Dev/Test Environments – Learn more

Instantly create dev & test environments for your
complex VMware based enterprise applications in AWS
Avoid 911 Production Upgrade Failures: Painless Upgrade Tests
Gil Hoffer

By Gil Hoffer

Gil Hoffer is the VP of R&D at Ravello Systems, where he plays a central role in Ravello’s day-to-day development activities, as well as in the grander long-term journey to redefine cloud computing. Prior to joining Ravello Systems, Gil served as CTO of one of the development branches in an elite technological unit in the Israel Defense Forces intelligence corps. Gil holds a B.A. from the Technion IIT, and a M.Sc. from Tel Aviv University, both in computer science.
More posts from Gil Hoffer