Skip to content

How to set up continuous integration / deployment for a new project

Continuous integration is the set of services that automatically runs tests from within GitLab, so that new feature branches and merge requests are tested automatically. Continuous deployment is the set of services that automatically deploys your code to a development or live server, directly from GitLab. This helps us to avoid a situation where we might merge or deploy code which has not been fully tested, or code that does not compile or build successfully. Because the results of each set of tests is available on GitLab and Slack, everyone working on the project should be clear about the current status of each branch.

By default, all Beautiful Canoe projects run their tests / builds on a dedicated test server, and are deployed to servers that have been set up specifically for that project.

Using the GitLab multi-runners

To run your build and test process in your test, staging or production environments, GitLab needs to be able to contact the relevant server and deploy code to it. GitLab provides runners which do this. As a rule, we do not use shared runners at Beautiful Canoe. Instead, you will need to install a GitLab multi-runner on the relevant VM. This page contains the information you need to do this.

Once you have the runner(s) installed, go to Settings -> CI/CD on your GitLab project page, and expand the Runners section, then activate the runners you wish to use (remember, these should normally be running on BC servers).

Getting started with automatic deployment scripts

Our deployment framework is a collection of scripts aimed at implementing most of the deployment logic for Beautiful Canoe projects, reducing the need for duplicated code between repositories. These scripts are aimed at utilising GitLab's Continuous Integration features to drive testing, building and deployment of code to environments.

This framework implements the common deployment logic required by most projects, whilst delegating project-specific portions to projects. These scripts also enforce a directory and environment structure to ensure that Beautiful Canoe projects are consistent.

Currently, this framework is compatible with Tomcat and Apache-based projects. If further project types are needed, the deployment portion of the codebase will need to be modified.

For a current working example, please see the ACTSS project.

Passwords and other sensitive data

At no point should any sensitive data, such as usernames or passwords, be committed to your repository. All sensitive data should be stored securely elsewhere. If needed for deployment, data should be held as a GitLab Variable.

Pre-requisites

  1. You have a develop, staging and/or production environment. Each environment should be stand-alone, in that a change in one environment should have no impact on the other, e.g. separate user accounts, databases, etc.
  2. Each environment (e.g. staging or production) should have its own GitLab Runner, pinned to your project and tagged appropriately. Speak to the CTO about getting this set up.
    1. Tags should be set when registering each runner, or edited via the runners section of the project once registered. Generally we tag staging servers as demo,staging,test and production servers as live,production.
    2. Runners must be set up to only accept tagged jobs. A runner should never accept untagged jobs, otherwise development code may end up on the production server!
  3. You are using GitLab merge requests to create feature branches as described here.

Naming environments

For legacy reasons, the deployment framework calls staging environments demo, production environments live, and all other environments dev.

Set-up

You will need to set-up the deployment repository as a Git submodule in the root of your repo. You should be managing these changes from dev -> demo -> live, so this should initially be done via a merge request targeting the develop branch, and then progressed as usual.

Firstly, ensure that you do not have a directory called deployment in the root of your repository. Then from the command line, execute:

git submodule add git@git.aston.ac.uk:beautifulcanoe/deployment.git deployment
git submodule sync deployment
git submodule update --init --recursive

Then modify .gitmodules so that the deployment submodule looks as follows:

[submodule "deployment"]
    path = deployment
    url = ../deployment.git

The project.sh script

In a new feature branch, create a new bash script called project.sh. This script will contain deployment attributes and functions that define how your project is build and deployed onto the target server. Please see deployment vars and functions for the variables and functions your project.sh file should implement. An example file can be found in examples/project.sh.example. This may be copied into your repo and modified (rename as project.sh), or you can start from scratch.

The document deployment vars and functions contains an explanation of the available variables and functions in project.sh files.

Running unit tests and lints

In the build_artefact() function within your project.sh file, you should ensure that you run the unit tests and relevant lints for your project. How you do this will depend on the programming languages and frameworks you are using on your particular project, so be sure to try out your tests and linters on the command line before you raise a merge request to run them automatically.

As an example, in PHP projects, we would expect to have tests using the phpunit framework, and to use the Code sniffer lint.

To do this, we would need the relevant packages added to composer.json. This example comes from a PHP5 project at Beautiful Canoe:

...
    "require-dev": {
        "fzaninotto/faker": "~1.4",
        "mockery/mockery": "0.9.*",
        "phpunit/phpunit": "~5.7",
        "squizlabs/php_codesniffer": "^3.3"
    },
...

and then the relevant config files will need to be committed to the repository. Here is an example phpunit.xml file from a Beautiful Canoe project:

<?xml version="1.0" encoding="UTF-8"?>
<phpunit backupGlobals="false"
         backupStaticAttributes="false"
         bootstrap="bootstrap/autoload.php"
         colors="true"
         convertErrorsToExceptions="true"
         convertNoticesToExceptions="true"
         convertWarningsToExceptions="true"
         processIsolation="false"
         stopOnFailure="false">
    <testsuites>
        <testsuite name="Feature">
            <directory suffix="Test.php">./tests/Feature</directory>
        </testsuite>

        <testsuite name="Unit">
            <directory suffix="Test.php">./tests/Unit</directory>
        </testsuite>
    </testsuites>
    <filter>
        <whitelist processUncoveredFilesFromWhitelist="true">
            <directory suffix=".php">./app</directory>
        </whitelist>
    </filter>
    <php>
        <env name="APP_ENV" value="testing"/>
        <env name="DB_DATABASE" value=":memory:"/>
        <env name="CACHE_DRIVER" value="array"/>
        <env name="SESSION_DRIVER" value="array"/>
        <env name="QUEUE_DRIVER" value="sync"/>
        <env name="DB_CONNECTION" value="sqlite_testing"/>
    </php>
</phpunit>

and an example phpcs.xml file:

<?xml version="1.0"?>
<ruleset name="ASH2">
<description>The ASH2 coding standard.</description>
    <rule ref="PSR2"/>
    <file>app/</file>
    <file>tests/</file>
    <exclude-pattern>vendor</exclude-pattern>
</ruleset>

Lastly, the project.sh script needs to run the relevant tests and lints in its build_artefact() function, for example:

function build_artefact() {
    ...
    $composer validate --no-check-all --strict
    vendor/bin/phpunit
    vendor/bin/phpcs
}

Notice that before we run the tests and lint, we also check that the composer JSON is valid. You may want to add other, project-specific lints and tests as your project matures.

The gitlab-ci.yml config file

Copy examples/.gitlab-ci.yml.example into the root of your repo as .gitlab-ci.yml. This is the configuration file that GitLab will use to run your builds and deployments, and the example configuration conforms to Beautiful Canoe standards.

You will need to define your environments, which look like this in the YAML file:

deploy:staging:
  stage: deploy
  environment:
    name: staging
    url: https://SUBDOMAIN.beautifulcanoe.com
  ...
  tags:
    - staging
    - demo

Environments define deployments, and having these recorded in the configuration allows GitLab to track them. If you go to Operations->Environments in your GitLab repository, you will see a list of deployments, their status (e.g. Running), which commit is currently at the HEAD of the code which is deployed, and options to view or re-deploy the environment.

Scheduled production check pipelines

Pipelines can be triggered (e.g. by a git push or a merge) or scheduled. When we create production deployments, we also create a weekly scheduled pipeline called Weekly production deploy check. This should be set to run the default pipeline on the production branch every Monday at 2am.

The intention here is to ensure that if a client project falls out of contract, and we do not deploy frequently, that any problems with the production environment are picked up via a pipeline failure.

If you are expecting to create a production deployment in the future, please be sure to raise an issue to add the scheduled pipeline, to ensure that this job does not get forgotten.

Commit and push your new deployment set-up

As usual, you should commit your new files on a feature branch, push them and raise a merge request. Your merge request will target the develop branch, but when you are happy that everything is working you should raise a second MR to merge develop into main. Keep an eye on your merge requests -- GitLab will build your code each time you push to a feature branch. If a pipeline fails, that means that a test/lint has failed or your code could not be built. By default, you will not be able to merge an MR until the pipeline passes, which should give you some confidence that your code is behaving as you expect it to.

Change your merge request settings

Once you have CI/CD working, you should change the merge request settings for your repository, or, if you don't have the rights to do this, ask the CTO to do it for you on Slack.

Go to Settings->General->Merge requests, and check Only allow merge requests to be merged if the pipeline succeeds:

Merge request settings

Keeping the CI/CD framework up to date

Once the project is set up with this framework, it must be kept up to date. Please see how to update the deployment framework for more information.