Statement

In my previous post, I had mentioned how to upgrade your system with airflow from 1.9 to 1.10. With the increasing popularity and maturity of apache-airflow, it releases it’s version very frequently. So as we are moving ahead, later than sooner we realise the need of upgrading apache airflow. Last time we did the upgrade from 1.9 to 1.10.2 and now it’s the time to upgrade from 1.10.2 to 1.10.5.

Recommendation

As we all know, Currently Orchestration Engines uses python 2.7 and python 2.x is going to end of life soon. So to perform airflow upgrade, we need to make the system compatible with python 3.x as well as apache-airflow’s version onwards 1.10.3 supports python 3x fully.

Change in Airflow Upgrade Process

Right now, airflow metadata is created through orchestration Server. So last time, we did the airflow upgrade (1.9 to 1.10) from Orchestration Server only and there we found few issues. The major issue was timeout from Orchestration server while upgrading the metadata and in turn health check fails. So this time, we thought of doing it through Orchestration Scheduler because this is the http application and here we can by pass the health check.

Airflow Upgrade Flow-Chart

Note* Here Orca indicates the docker container on which server and scheduler is running.

Results

I was able to upgrade from 1.10.2 to 1.10.5 on my local MAC machine keeping our Service in mind.
I was able to run the existing compute DAG and plugins successfully.