I am going to prepare a live deploy of a web app built with Flask. The app will be hosted as a subdomain for an already existing website that runs a WordPress installation on a LAMP stack exposed to the web via an Nginx reverse proxy.
The Flask app (svm-demo) will be served by a Gunicorn service also via Nginx. The web app will be available at this URL: svm-demo.singularaspect .com
WSGI Entry Point
Since the app is served by Gunicorn, it will require an entry point. For that I create a file under a name wsgi.py with this content:
|
1 2 3 4 5 |
from app import create_app app = create_app() if __name__ == "__main__": app.run() |
Here, it is important that the module contains a variable called app or application. My web app uses the app factory pattern, so I have to import the function that produces the app instance.
The path to this entry point will be used later when I define a service that will run Gunicorn for my website.
Deploying the App
The future deployment of the app will be done via continuous integration (CI) service set up with Jenkins. But the first step is to prepare the deployment script and the physical location on the target server.
Preparing the Destination
The file system location of the future app will be within /var/sites directory. The project will be deployed into:
|
1 |
drwxrwxr-x 3 ownername ownergroup 4096 Aug 19 18:02 svm-demo/ |
The folder structure looks like this:
|
1 2 3 4 5 6 7 8 9 |
/var/ sites/ svm-demo/ master/ app -> /var/sites/svm-demo/master/releases/1566286138.888088/ releases/ 1566210232.5667802/ 1566230574.4630783/ 1566286138.888088/ |
In this location, the deploy script will create subdirectories that correspond to the types of deployment. The default type is master that I will use for the production (live) deployment. I could provide other types (stage or testing) for environments other than production. Important is that the deployment type label must have a corresponding branch in the repo (master, staging or testing) because the deploy script will pull that branch each time it runs.
Within the master (or any other deployment type subdirectory) the script will create a releases folder. There, the script will clone the repository. Each tome a clone will be placed under a timestamp-named directory. After each deployment, the most recent such directory will be symlinked to /var/sites/master/app. The Ngnix will serve the website from this symlink’s location. The older directories will be kept for a fallback in the case of a catastrophic deploy. There will be at most 2 releases directory at a time: one active and one fallback. Older directories will be deleted after each deployment.
Deploy Script
I use a fabric script to run the deployment. First, I need to do the required imports and set up some constants:
|
1 2 3 4 5 6 7 8 9 |
import time from fabric.contrib.files import exists from fabric.api import cd, run project_name = 'svm-demo' REPO_URL = "git@github.com:varinen/svm_loss.git" INSTALLATION_PATH = '/var/sites/' + project_name + '/' release = time.time() releases_dir = 'releases' |
The main function’s content is this:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
def deploy(dep_type='master', use_key=None): """ :param dep_type: must be 'master', 'stage' or 'testing'. The repository must have respective branches :param use_key: specify optional path to the github key on the target machine, i.e., ~/.ssh/id_rsa :return: """ print('Executing deploy for: ', dep_type) app_folder = f'{INSTALLATION_PATH}{dep_type}' cur_release_dir = app_folder + '/' + releases_dir + '/' + str(release) run(f'mkdir -p {cur_release_dir}') with cd(cur_release_dir): _git_clone(dep_type, use_key) print('Setting up the app') _setup_app() _run_tests() _create_symlink(cur_release_dir, app_folder) with cd(app_folder + '/' + releases_dir): _remove_older_releases() |
This function takes the deployment type and (optionally) the path to the private key used to access the git repository. The function creates the target path, clones there the web-app repository, and checks out the branch respective to the deployment type. This is done by calling the _git_clone function:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
def _git_clone(branch, use_key): """ Clones the repo and switches to the specified branch :param branch: :return: """ run('echo $PWD') if use_key is not None: run( f'ssh-agent bash -c \'ssh-add {use_key}; git clone {REPO_URL} . --quiet\'') else: run(f'git clone {REPO_URL} . --quiet') run(f'git checkout {branch} --quiet') |
In the next step, it calls the _setup_app function that creates a virtual environment and installs requirements:
|
1 2 3 4 5 6 7 |
def _setup_app(): """ Runs the software installation and marks the installation as remote :return: """ if not exists('venv/bin/pip'): run(f'python3.6 -m venv venv') run('./venv/bin/pip install -r requirements.txt') |
The unit tests are run in the next step by the _run_tests function:
|
1 2 3 4 5 |
def _run_tests(): """ Runs the unittests :return: """ run(f'./venv/bin/python -m pytest tests') |
If nothing is broken at this stage, it is safe to update the symlink to the live code:
|
1 2 3 4 5 6 7 8 |
def _create_symlink(cur_release_dir, app_folder): """ Creates a symlink to the current release location """ with cd(app_folder): link = app_folder + '/app' if exists(link): run(f'rm -rf {link}') run(f'ln -s {cur_release_dir} {link}') |
Finally, the script deletes the older release directories:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
def _remove_older_releases(): """ Makes sure that the releases dir does not contain more than 2 latest release directories. Deletes all but the latest 2. Creates a new release dir :return: """ print('Removing older releases') dirs = run(" ls -tr | tr '\n' ','") print(dirs) dirs = dirs.split(',') rev_dirs = dirs[::-1] print(' '.join(rev_dirs)) to_delete = rev_dirs[4:] if len(to_delete) > 0: print("Removing older releases: ", ' '.join(to_delete)) for delete_me in to_delete: run(f'rm -rf {delete_me}') |
The deployment script requires Fabric 3. I place the script into the project’s subdirectory deploy_tools. To run the deploy, I switch to this directory and run the following command:
|
1 |
fab deploy:host=USERNAME@HOSTNAME,dep_type=TYPE -i /PATH_TO_PRIVATE_KEY |
Setting Up Nginx
I create a new file under /etc/nginx/sites-available/svm-demo.singularaspect.com with the following content:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
server { listen 80; server_name svm-demo.singularaspect.com; location /static { alias /var/sites/svm-demo/master/app/app/static; } location / { proxy_pass http://unix:/tmp/svm-demo.singularaspect.com.socket; proxy_set_header Host $host; } } |
The website code will be served via a socket that will be created by a service that runs Gunicorn for the website. The static files are delivered directly by Nginx.
After the file is created, I add a symlink to it under /etc/nginx/sites-enabled:
|
1 |
sudo ln -s /etc/nginx/sites-available/svm-demo.singularaspect.com /etc/nginx/sites-enabled/svm-demo.singularaspect.com |
I test the Nginx configuration by running:
|
1 |
sudo nginx -c /etc/nginx/nginx.conf -t |
If everything is OK, I reload the Nginx server:
|
1 |
sudo systemctl reload nginx |
Setting Up Gunicorn
For this website, I want to run Gunicorn as a service bound to a socket. Under /etc/systemd/system , I create a file gunicorn-svm-demo.singularaspect.com.service with the following content:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
[Unit] Description=Gunicorn server for the demo app for the DOMAIN, USERNAME [Service] Restart=on-failure User=USERNAME WorkingDirectory=/var/sites/svm-demo/master/app/ EnvironmentFile=/var/sites/svm-demo/master/app/.env ExecStart=/var/sites/svm-demo/master/app/venv/bin/gunicorn --bind \ unix:/tmp/DOMAIN.socket wsgi:app [Install] WantedBy=multi-user.target |
I replace DOMAIN with svm-demo.singularaspect.com and USERNAME with the user I am going to run the service under. It is the same user that runs the deploy script.
To enable this configuration I run:
|
1 2 3 |
sudo systemctl daemon-reload sudo systemctl enable gunicorn-svm-demo.singularaspect.com.service sudo systemctl start gunicorn-svm-demo.singularaspect.com.service |
I check the status of the service by running:
|
1 |
sudo systemctl status gunicorn-svm-demo.singularaspect.com.service |
If something goes wrong I can check either the Nginx log or the service log by running:
|
1 |
sudo journalctl -u gunicorn-svm-demo.singularaspect.com.service |
Continuous Integration
I want to have continuous integration set up for this project so that it would automatically pull any changes from the production branch (master) and perform all the required checks and deploy it to the production server. Alternatively, I would like to be able to manually start the deployment process from the continuous integration server.
Setting up Jenkins
I set up a Jenkins project. The general settings are these:
I specify Git as the source code control and point to the repository and branch to pull from. The repository is accessed with credentials that I’ve set up previously for this purpose.
I set up the repository polling for every hour. If changes in the master branch are detected, the project will be built and deployed.
I set up the build environment to use an SSH-Agent configured with the credentials I’ve set up to access the production server.
I set up two build steps. The first step is the virtualenv Builder that runs the pip installation and tests on the CI server and the fabric deploy on the production server.
After the fabric deploy is done, Jenkins must restart the gunicorn service.
Dealing with User Privileges
For the last step to execute, I need to allow the user that logs in with the SSH plugin to run systemd’s systemctl component to restart the gunicorn service by executing this command:
|
1 |
systemctl start gunicorn-svm-demo.singularaspect.com.service |
When I run this command manually I must do it with sudo and type in my password. To automate this I need to do the following steps.
First, I create a user group whose members will have access to commands starting, stopping, and restarting the service:
|
1 |
sudo groupadd appadmin |
Second, I add the user that will run the service and that Jenkins will use to SSH connect to this group:
|
1 |
sudo usermod -a -G appadmin USERNAME |
Third, I edit /etc/sudoers file using the sudo visudo command to add a command alias for the service control commands and give passwordless access to this alias to the user. I add these lines first:
|
1 2 |
# Cmnd alias specification Cmnd_Alias SVM_DEMO_CMNDS = /bin/systemctl start gunicorn-svm-demo.singularaspect.com.service, /bin/systemctl stop gunicorn-svm-demo.singularaspect.com.service, /bin/systemctl restart gunicorn-svm-demo.singularaspect.com.service |
To the end of the file I add these:
|
1 2 |
# Allow members of the specified group to execute commands %appadmin ALL=NOPASSWD: SVM_DEMO_CMNDS |
With this, Jenkins can deploy the fresh code and restart the service to put it live.
