Airflow Upgradability

Problem Statement : While working with Apache Airflow, we need to navigate between the various versions (1.8, 1.9, 1.10 etc) of it. For that we need to define the process to upgrade the Apache Airflow from one version to another version.

Upgrade Types 

As we are having multiple clusters in terms of multiple airflow servers and schedulers. So we have mainly 2 kinds of upgrade based on the client’s requirements.

Full Upgrade

In this we need to upgrade all the clusters defined in the CI-CD pipeline for the deployment

Upgrade Approaches

1. By taking down-time

   1. First kill the airflow containers like server, scheduler and workers.
      1. One way is to wait for all the running tasks to either complete as success or fail to kill the container and then disable all the tasks. But it will introduce the longer down-time based on the behaviour of task completion.  In this way we will have to maintain a map of user disabled tasks and script disabled tasks.
      2. Other way is to directly kill the containers. As soon as the new container comes up, all the running tasks should be completed as success or fail based on it’s nature.
   2. Take the backup of airflow metadata.
   3. Make the necessary setting for the new airflow version as a part of docker build args .
   4. Install the new airflow version on all the containers.
   5. In case, installation fails, restore the DB to it’s previous backed up state.
   6. Execute the airflow upgradedb command to make the necessary changes in airflow metadata required for airflow server and scheduler both.
   7. Again, if installation fails, restore the DB to it’s previous backed up state.
   8. Start all the containers to boot the service.

2. With out taking down-time

   1. Define the NEW_VERSION_NAME as run-time build args in the service yml file and this will be applicable for all the clusters defined as a part of  pipeline
   2. Re-onboard the service again with this NEW_VERSION_NAME.
   3. Redirect all the current cluster’s (running containers) traffic to the dummy DB to avoid inconsistency in the current Airflow metadata.
   4. Take the backup of airflow metadata.
   5. Build the docker images with this NEW_VERSION_NAME which will be applicable for all the clusters.
   6. Make the necessary setting for the new airflow version as a part of docker build args.
   7. Execute the airflow upgradedb command to make the necessary changes in airflow metadata required for airflow server and scheduler both.
   8. If installation fails, restore the DB to it’s previous backed up state.
   9. Start all the containers to boot the service which will point to the original metadata.

Partial Upgrade

In this, we may need to upgrade the specific cluster based on the client’s requirement.

• By installing the required binary at deploy step

So there are following steps which are needed to perform as a part of deployment –

1. Define a Environment variable like NEW_VERSION_NAME = 1.10.1 in the required cluster.
2. Install the airflow new version by comparing the airflow current version and NEW_VERSION_NAME through the run file which is executed as a part of execution of the desired docker image on the given cluster(container) at deploy step. In addition to it, we need to maintain additional requirement_NEW_VERSION_NAME.txt to download the additional dependencies as a part of NEW_VERSION_NAME and these will also take care by the run file.
3.  In this case, we need to invoke the airflow upgradedb to make the appropriate schema changes for Mysql required for apache airflow NEW_VERSION_NAME but run file always executes the initdb and upgradedb is the part of that command.
4. In case of any failure, we need to take the backup of existing scheme so that we can rollback the same and that will cause some downtime in the service.
5. Now, container is started and make sure there is no inconsistency in the system.

• By creating multiple virtual environments

   In this, we need to perform the following steps as a part of deployment –

1. By default, one virtual environment is already created inside the container and that environment is used to run the default airflow binary 1.9 on all the clusters.
2. To suffice the use case of partial upgrade, we need to create more virtual environments based on the binary needed on the clusters.
3. Create the virtual environments and install the different versions of apache-airflow in those environments at build step.
4. Now, based on the defined NEW_VERSION_NAME as a part of env variable, we need to switch the virtual environment on that specific cluster.
5. The problem with this approach is to create multiple environments and install the desired binary in that. As soon as number of binaries are increased, this becomes bottleneck and it will be memory intensive.Note* : To avoid initdb execution everytime when run script is called at the time of deploy step, I have controlled it through the Env variable called INITDB_FLAG. Once any new cluster is setup, we set the value of this variable otherwise this is not set in Env variable.

Hope this will help you to upgrade your airflow clusters.