Upgrading a managed Database Plan in Cloud Foundry

Using application services in Cloud Foundry is a great way to free yourself from the hassle of administering your own infrastructure. Need a MySQL database, a MongoDB or both?

Application services come in different sizes, which means that they will scale up to a certain amount of load. This is how you can transition an application to an upgraded database plan with no downtime.

Preparing the application

Going from a single database server to a cluster involves a few configuration steps in the application. For example, the application needs to correctly set its read preferences for MongoDB. After setting up the client with a test environment, they were quick to implement the required changes.

For the transition to the new database clusters, the customer decided it was acceptable for the application to stop accepting write requests during the transition period. Their engineering team implemented a read-only mode in the application, activated by a feature toggle. To set the feature-toggle, the application looks for a special environment variable at runtime. While the application is in read-only mode, customers can continue using the application but certain actions that require updating records will be disabled. Some of the worlds largest websites like Stackoverflow.com successfully use this approach for failovers.

The migration plan

With these preparations out of the way, the migration plan we developed together with our customer works as follows:

1. On the new database clusters, provision new service instances for the application
2. Activate read-only mode in the application
3. Take a backup of the databases
4. Restore the backup on the newly provisioned database instances
5. Change the service bindings of the application, disable readonly-mode and restart

To minimize any chance of downtime for the application, we tested the migration scenario in a test project. Thanks to Cloud Foundry’s excellent support for infrastructure as code, we we’re able to spin up a full copy of the customer’s application in less than 15 minutes.

Restaging and Restarting Apps in Cloud Foundry without downtime

One of the things we learned from our test set up is that updating the application’s environment variables requires a full restage and restart of the application. However, a cf restage or cf restart command will cause Cloud Foundry to take down all instances of an application simultaneously. If you’d like to learn more about this, here’s why.

We figured that we could use something similar to the blue-green deployment technique that we are already using to deploy updated versions of an application to Cloud Foundry. Sure enough, we were not the first to face this problem. Turns out that Orange (yup, one of the world’s largest Telco’s) runs Cloud Foundry and maintains an arsenal of useful open source tools on their github page. One of those is bg-restage, a plugin for the cf cli that will do exactly what we need: blue-green restage an application.

Migrating to a larger service plan

With that out of the way, our migration could begin. Here’s the exact scripts (minus credentials of course) that we used.

0) Disable CI/CD Pipeline

Make sure that no deploys accidentally interfere with the update. Disable the Continuous Delivery pipeline.

1) Provision new database instances

$ cf create-service MySQL XL db-mysql
$ cf create-service MongoDB XL db-mongodb

Create service keys so that we can remotely access the new database instances. We will later use the credentials contained in the service key to import the database backup. Make sure to save the output of these commands.

$ cf create-service-key db-mysql remote
$ cf service-key db-mysql remote
Getting key remote for service instance db-mysql as jrudolph@meshcloud.io...

{
"database": "27c9f710-5cfa-40e7-bd68-5eda075bdf1f",
"host": "217.26.224.15",
...

$ cf create-service-key db-mongodb remote
$ cf service-key db-mongodb remote

Getting key remote for service instance db-mysql as jrudolph@meshcloud.io...

{
"database": "700a1921-f1d0-409c-8cf0-9f02db7c6291",
"host": "217.26.224.15",
...

You will also need service-keys for the old database instances to take the backup.

2) Activate read-only modeThe customer’s application allows activating readonly mode when the environment-variable web.maintenance_mode is set to true. Hence we instruct Cloud Foundry to set this environment variable for the application deployment and restage the application without downtime using bg-restage.

$ cf set-env application web.maintenance_mode true
$ cf bg-restage application

3) Create backup of database

Once readonly mode is active, take a backup of the database. Using the credentials from service-keys on the "old" database instances, we can take a simple data-export using these commands:

$ mysqldump -h -P -u -p > mysql_backup.sql
$ mongodump --host= --port= --username= --password= --db= --gzip --archive=mongodb_backup.agz

4) Restore Backup on new database instances

Next, we need to restore these backups in the new database instances. Using the credentials from service-keys on the "new" database instances, we can use these commands to restore the backups. Note that for mongodb we will also have to adjust the namespaces of exported objects so that they will live in the correct namespace on the database instance.

$ mysql -u -p -h -P --database < mysql_backup.sql

$ mongorestore --host= --port= --username= --password= --authenticationDatabase= --nsFrom=".*" --nsTo=".*" --gzip --archive=mongodb_backup.agz

5) Update ApplicationWith the databases restored on the new database instances, we need to update the application’s service bindings to use the new database instances:

cf unbind-service application old-mysql
cf bind-service application db-mysql
cf unbind-service application old-mongodb
cf bind-service application db-mongodb

Using cf unset-env we also clear the environment variable that activates readonly mode for the application. The new bindings as well as the unset readonly mode will become active on the next restage of the application.

cf unset-env application web.maintenance_mode
cf bg-restage application

That’s it. By now the application is successfully running on the new database instances. Before we can reactivate the CI/CD pipeline, we also need to make sure to update the application’s manifests.yml to use the new service instances.

A successful migration

Using the plan above we were able to quickly migrate the customer’s database. We chose to run the import/export from a VM running in the same datacenter as Cloud Foundry to speed up the import/export process using the internal datacenter network instead of going through the internet. That’s one of the big benefits of the meshcloud cloud platform stack (currently OpenStack IaaS and Cloud Foundry PaaS), which allows you to easily have IaaS and PaaS services hosted in the same datacenter. We were able to backup and restore the databases in less than 15 minutes. The process was very quick and with minimal disruption of service for the users of the application.