DevOps Guide

Environment “testenv”

This environment allow developers and power users to run additional testing such as automated system tests and manual tests. One of its main purposes is to uncover bugs and defects not detected by the CI.

One host machine acting as a GitLab runner runs the instructions specified in the deploy job of the GitLab CI. There’s only one specific machine allowed to pick and run the deploy job so that environment testenv is always deployed to the same machine. The app running in testenv is exposed to the WWW as reflectivity-test.sns.gov.

Currently, the database in this environment is not persistent, meaning a new deployment will erase whatever data has been stored since the last deployment. Also, testenv is currently listening and writing to livedata.sns.gov, as well as listening to oncat.ornl.gov. Development of a test environment for the web monitor will bring about test substitutes for both servers.

Database Management

We rely on Django’s manage.py for dumping the source database and loading it into the recipient database. Django’s manage.py allows for a fine control of what tables to dump and the command is agnostic of the database flavor (mysql, postgresql, sqlite) for both the source and recipient databases.

Dumping the Old Database

The database from reflectivity.sns.gov has quite a different schema than the database of the modernized application because models for the app and its dependencies have evolved.

Login to reflectivity.sns.gov and then:

cd /var/www/web_reflectivity/app
dumpfile=/tmp/webreflect_$(date +%F).json  # e.g. /tmp/webreflect_2022-05-01.json
python manage.py dumpdata --verbosity 3 --natural-foreign --natural-primary -e contenttypes -e auth.Permission -e django_auth_ldap -e django_celery_results --indent 2 --database default --traceback > ${dumpfile}

For loading the resulting JSON file into the recipient database, jump to Loading onto the Modernized Database.

Dumping the Modernized Database

It is assumed that the container running the web_reflectivity app, as well as the container running the database are up and running.

The name of the container running the web_reflectivity app should be test_webref_1 if running in the TEST environment. One can make sure by listing the running containers:

$> docker container ls
CONTAINER ID  MAGE                                                                           COMMAND               CREATED      STATUS               PORTS              NAMES
e71b9b6c4a4e  code.ornl.gov:4567/reflectometry/web_reflectivity/web_reflectivity:latest-dev  /usr/bin/docker-ent…  6 hours ago  Up 6 hours (healthy) 22/tcp, 8000/tcp   test_webref_1

In this particular case, the name of the container is test_webref_1, and we can use the CONTAINER ID e71b9b6c4a4e in place of this name.

Open a shell to the container running the web_reflectivity app and execute the dumpdata make target:

$> docker exec -it test_webref_1 bash
(webrefl)$ make dumpdata  # e.g. creates /tmp/webreflect_2022-05-20.json

A JSON dump file /tmp/webreflect_$(date +%F).json is generated in the container’s /tmp directory. An easy way to make it available to the host machine is to move this file to directory /var/log/ because this directory is mounted in the host machine as directory /tmp/log/web_reflectivity/web/

Loading onto the Modernized Database

We need to make the JSON dump accessible from within the container running the app. An easy way is to place the file in the host machine directory /tmp/log/web_reflectivity/web/ because is bind-mounted to container’s directory /var/log/.

Assuming we have file /tmp/log/web_reflectivity/web/webreflect_2022-05-01.json in the host machine, we need to open a shell in the running container servicing the application and execute the make loaddata target.

Details on how to find out the name of the running container are laid out in the previous section Dumping the Modernized Database.

docker exec -it test_webref_1 bash
(webrefl)$ make fixturefile=/var/log/webreflect_2022-05-01.json loaddata

This will update the recipient database. The command takes minutes to execute because it translates the JSON file into a large set of python objects. These objects are in turn translated into a long list of postgres commands to be executed on the recipient database.

Deployment for Testing

Deployment for the testing environment is handled by the web-relectivity-deploy repo. For details on how to deploy, read the Guide for the deployment to Testing Environment.

Access to the Test Environment

Access to the test server reflectivity-test.sns.gov via SSH is granted on a per user and per client-machine basis. Contact the DevOps engineer assigned to maintenance of the server. After granted access, SSH to the machine as user cloud

ssh cloud@reflectivity-test.sns.gov

The goal of accessing the test server is to troubleshoot any problems by directly manipulating the deployed application.

Deployment in Production

This deployment is scheduled to happen after a new version of the software is released and thoroughly tested in deployment testenv. For details on how to deploy, read the Guide for the deployment to Production Environment.