Statement : The sole purpose of this post is to learn how to run a cron job which I have demonstrated in my previous post and access the environment variable inside the Docker container.
Prerequisites – Please follow the previous post steps to install and the whole process of building and running the cron job inside the Docker container.
Using environment variables : Here the goal is to read the environment variable inside the script file. If we don’t inject the env variable using the below approach, we won’t be able to access the env variable. With out injecting the env variable, if we do echo $ENV_DAG_NAME inside the script, then it will give us the output as empty String. If we do the echo on the command prompt, then it will give us the right output.
Steps : Please follow the below steps to implement the whole task one by one –
- Dockerfile includes the base image and contains the dependencies required to build and run the image –
FROM ubuntu:16.04 RUN apt-get update && apt install -y build-essential libmysqlclient-dev python-dev libapr1-dev libsvn-dev wget libcurl4-nss-dev libsasl2-dev libsasl2-modules zlib1g-dev curl cron zip && DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends rsync && apt-get clean autoclean && apt-get autoremove -y && rm -rf /var/lib/apt/lists/* // Either Hardcode the environment variable here or pass these through docker run command using -e flag (ex. "-e ENV_DAG_CONTAINER=test") ENV ENV_DAG_CONTAINER=test COPY crontab /tmp/crontab COPY run-crond.sh /run-crond.sh COPY script.sh /script.sh DD dag_script.sh /dag_script.sh RUN chmod +x /dag_script.sh && chmod +x /script.sh && touch /var/log/cron.log CMD ["/run-crond.sh"]
- *Important Step : The below line shows how to grep environment variable and use them in the script. In this, first I grep the env variable (ENV_DAG_CONTAINER) and then moved this variable into the temp file. Finally added this at the top of the script so that we can use it.
$ env | egrep '^ENV_DAG' | cat - /dag_script.sh > temp && mv temp /dag_script.sh
- Add env variable to the docker run command using the below command or set them inside the Docker file (shown in the Dockerfile) if you don’t need to change them at run time.
$ docker run -it -e ENV_DAG_CONTAINER=tanuj docker-cron-example
- Finally, the entry point script is like below (run-crond.sh) –
#!/bin/sh crontab /tmp/crontab #The below line shows how to grep environment variable aur use them in the script. In this, first I have greped the env variable (ENV_DAG_CONTAINER) and then moved this variable into the temp file. Finally added this at the top of the script so that we can use it. env | egrep '^ENV_DAG' | cat - /dag_script.sh > temp && mv temp /dag_script.sh #To start the cron service inside the container. If below is not working then use "cron/crond -L /var/log/cron.log" service cron start
- crontab contains the list of cron jobs to be scheduled for the specific time. In the below crontab, I have shown how to run any script with in the interval of seconds using cron job (See the 5th line to echo the message in the internal of 5 seconds once the cron demean has been triggered). **Basically cron job has the granularity of 1 minute to run any job. So initially it takes minimum one minute to boot the job when we don’t initialise any time ( like this * * * * *) after that it executes based on the script (I ran the script in the interval of 5 seconds).
# In this crontab file, multiple lines are added for the testing purose. Please use them based on your need. * * * * * /script.sh * * * * * /dag_script.sh >> /var/log/cron/cron.log 2>&1 #* * * * * ( sleep 5 && echo "Hello world" >> /var/log/cron/cron.log 2>&1 ) * * * * * while true; do echo "Hello world" >> /var/log/cron/cron.log 2>&1 & sleep 1; done #* * * * * sleep 10; echo "Hello world" >> /var/log/cron/cron.log 2>&1 #*/1 * * * * rclone sync remote:test /tmp/azure/local && rsync -avc /tmp/azure/local /tmp/azure/dag #* * * * * while true; do rclone sync -v remote:test /tmp/azure/local/dag && rsync -avc /tmp/azure/local/dag/* /usr/tanuj/dag & sleep 5; done #* * * * * while true; do rclone sync -v remote:test /tmp/azure/local/plugin && rsync -avc /tmp/azure/local/plugin/* /usr/tanuj/plugin & sleep 5; done # Don't remove the empty line at the end of this file. It is required to run the cron job
- Write the script files to be executed with cron job. Below is the example of dag_script.sh file –
// This below line will be appeneded through the run-scrond.sh file once the container is started. I have used it here for the testing purpose. // This below line will be appeneded through the run-scrond.sh file once the container is started. I have used it here for the testing purpose. ENV_DAG_CONTAINER=test echo "$(date): executed script" >> /var/log/cron.log 2>&1 if [ -n "$ENV_DAG_CONTAINER" ] then echo "rclone process is started" while true; do rclone sync -v remote:$ENV_DAG_CONTAINER /tmp/azure/local/dags && rsync -avc /tmp/azure/local/dags/* /usr/local/airflow/dags & sleep 5; done echo "rclone and rsync process is ended" fi
So, this is the way to run the cron job inside the docker container and learn how to access the env variable inside the same. Hope you enjoy it and find the full source code of the above implementation from the git. Happy Reading 🙂