How to create a static website¶

Most websites that Beautiful Canoe creates for clients are dynamic sites with some sort of datastore, usually based on the Laravel framework. Like the (static) company website these are bespoke, and each site has its own styling, according to the needs of the client.

However, we have a small number of static sites for internal use, such as this site and our brand guidelines. Less frequently, we sometimes need to create static sites for clients (most likely to publicise a piece of research software) such as the Traffic3D site.

For these static sites, we do not use raw HTML (like the company website does), partly to save time, but also to avoid inconsistencies in styling and content. Instead, we have standardised our static sites on the mkdocs package for Python, with the mkdocs material theme which we use because it is clean, fully responsive and based on Google's material design guidelines. In the past we have been able to put together and deploy a whole new site within a couple of hours.

Starting a new repository: tweaks to our usual advice¶

Each static website should be in its own repository, even if the intention of the site is to publicise a specific software product. This helps us to separate out issues and merge requests for each piece of work we do, and to easily use CI/CD to test documentation and code separately. However, it does mean that sometimes you will need to change some software, then raise a separate issue in another repository to document that change.

Tip

Your repository should normally have the same name as its public URL. For example, this repository is beautifulcanoe/peopleops/docs.beautifulcanoe.com.

If you are starting a new repository for a static site, please read through and follow the start a new project HOWTO and related documentation.

For static sites, though, there are a few important tweaks to our usual advice:

You will not need a develop branch in your static website repository
When you create a .gitignore file, use the standard Python .gitignore file but add these lines to the file:

# Virtual environment.
venv/*

# Files generated by mkdocs.
site/*

When you create a Slack integration for the new repository, use one of the bc-SUBGROUP-gitlab channels if you are creating a site for internal Beautiful Canoe use, and use the PROJECT-gitlab channel if you are creating a client site for an existing client.

Warning

The rest of this HOWTO assumes that you have created an issue (to add the initial version of the site), started an MR and that you are working in the feature branch for that MR. Hopefully, if you have followed the instructions above, you will already be in a feature branch, where you have committed your .gitignore file and README.md.

Creating a new URL¶

By the time you get to the end of this HOWTO, your site will be public, so ask @snim2 or @a.garcia-dominguez to create a new URL for you as soon as possible. If your static site will be for internal company use, the URL will be SOMETHING.beautifulcanoe.com, otherwise this will depend on your project and client.

Writing Markdown¶

The contents of your site will be in Markdown, not HTML. This will save you a lot of time in terms of styling and formatting, but it is important that you are familiar with Markdown syntax.

Tip

All Beautiful Canoe Markdown should have a newline at the end of every sentence. This means that when you raise a merge request, the diff will only consist of sentences that have changed. If this seems confusing, have a look at the way documentation for this site is formatted.

Creating and using a Python virtualenv¶

To run mkdocs you will need to install virtualenv and create a new virtual environment. This is a new environment with its own version of Python and its own package manager, so we can ensure that whatever we install here is exactly as it will in the production environment:

sudo apt-get install python-virtualenv
virtualenv --python=python3.7 venv
. venv/bin/activate

Your command line prompt should now look like this:

(venv) $

If you need to get out of the virtual environment, run the deactivate command.

Next, create a file called requirements.txt containing the following text:

certifi==2019.6.16
chardet==3.0.4
Click==7.0
htmlmin==0.1.12
idna==2.8
Jinja2==2.10.1
jsmin==2.2.2
livereload==2.6.1
Markdown==3.1.1
MarkupSafe==1.1.1
mkdocs==1.0.4
mkdocs-material==4.4.0
mkdocs-minify-plugin==0.2.1
Pygments==2.4.2
pymdown-extensions==6.0
python-gitlab==1.10.0
PyYAML==5.1.2
requests==2.22.0
six==1.12.0
tornado==6.0.3
urllib3==1.25.3

then install the requirements:

pip install -r requirements.txt

At this stage, it is a good idea to add requirements.txt and commit it.

Create an empty site¶

Create a new, empty site with this command:

(venv) $ mkdocs new .
INFO    -  Writing config file: ./mkdocs.yml
INFO    -  Writing initial docs: ./docs/index.md
(venv) $

Now you can build the HTML, and serve it locally:

(venv) $ mkdocs build
INFO    -  Cleaning site directory
INFO    -  Building documentation to directory: .../REPO/site

(venv) $ mkdocs serve
INFO    -  Building documentation...
INFO    -  Cleaning site directory
[I 191119 15:13:16 server:296] Serving on http://127.0.0.1:8000
[I 191119 15:13:16 handlers:62] Start watching changes
[I 191119 15:13:16 handlers:64] Start detecting changes
[I 191119 15:13:19 handlers:135] Browser Connected: http://127.0.0.1:8000/

And open your browser at http://127.0.0.1:8000 to see something like this:

Brand new mkdocs site

At this point you should commit the new mkdocs.yml and docs/index.md files.

Standard `mkdocs` configuration¶

Next, you will want to reconfigure the new site. You will need to tell mkdocs to use the Material theme, set some details about the site and add a logo, a favicon, and so on. Below is a standard configuration from one of our current sites. Notice that the words in CAPITAL LETTERS will need to be replaced with something specific to your site.

Note

If your static site is for internal Beautiful Canoe use, please use Brown and Deep Orange for your brand colours, and Didact Gothic and Source Code Pro for your fonts, as seen below. If your site is not for internal company use, you will want to change the social media links to something appropriate. Beautiful Canoe logos and favicons can be found on our brand guidelines site.

site_name: 'SITE NAME'
site_url: 'PUBLIC URL'
dev_addr: '127.0.0.1:PORT'
site_author: 'Beautiful Canoe'
site_description: 'DESCRIPTION'
theme:
    name: 'material'
    logo: 'assets/images/logo.png'  # File: docs/assets/images/logo.png
    favicon: 'assets/images/favicon.ico'  # File: docs/assets/images/favicon.ico
    palette:
        primary: 'Brown'
        accent: 'Deep Orange'
    font:
        text: 'Didact Gothic'
        code: 'Source Code Pro'
# Links in the footer to social media sites, the type field determines
# which Font Awesome icon is used.
extra:
    social:
        - type: 'twitter'
          link: 'https://twitter.com/beautifulcanoe'
        - type: 'facebook'
          link: 'https://www.facebook.com/BeautifulCanoe/'
        - type: 'linkedin'
          link: 'https://www.linkedin.com/company/10626063'
# Provide a link to the repo for the documentation site.
repo_url: 'https://gitlab.com/beautifulcanoe/projects/REPO'
repo_name: 'REPO'
copyright: 'Copyright &copy; YEAR Beautiful Canoe'
# These extensions include code highlighting for PHP, they ensure
# that every heading has a permalink and add admonitions:
# https://squidfunk.github.io/mkdocs-material/extensions/admonition/
markdown_extensions:
    - admonition
    - pymdownx.highlight:
          css_class: codehilite
          extend_pygments_lang:
              - name: php-inline
                lang: php
                options:
                    startinline: true
    - pymdownx.superfences:
    - pymdownx.inlinehilite:
    - toc:
          permalink: true
strict: false
# Site navigation structure. The docs/ prefix to path names is not needed.
nav:
    - Home: index.md  # File docs/index.md
    ...

Test your new configuration manually, and then commit it.

Linting and testing your new site¶

To make sure that your Markdown is valid, please use the mdl Markdown lint. To install the lint on Debian-like machines, use the Rubygems package manager:

sudo apt-get install gem
sudo gem install mdl

Because we keep each sentence on a separate line, you will want to suppress spurious MD013 Line length reports by configuring mdl. Create a file called .mdl.rb to hold configuration for the lint, and copy this text into it:

# Enable all rules by default
all

# Extend line length, since each sentence should be on a separate line.
rule 'MD013', :line_length => 99999

# Allow multiple headers with same content.
exclude_rule 'MD024'

# Allow inline HTML
exclude_rule 'MD033'

# Allow trailing punctuation (e.g. question marks) in headers.
exclude_rule 'MD026'

# Nested lists should be indented with four spaces.
rule 'MD007', :indent => 4

Make sure you commit the .mdl.rb file. To use the style configuration, pass it as a parameter to mdl on the command line:

mdl -s .mdl.rb HOWTO_DOCUMENT.md

If you want to run mdl from your IDE or editor, you will either need to configure it, or find a plugin, such as this one for Sublime Text.

To check for broken links, we use Linkchecker and (if using the git hook) webfsd. First, install these on your development machine:

sudo apt-get install linkchecker webfs

Next, to run the linkchecker locally, serve the site (if you aren't doing so already):

mkdocs serve

and, in a separate terminal, run the linkchecker:

$ linkchecker http://127.0.0.1:8000/

INFO linkcheck.cmdline 2019-09-03 20:53:02,231 MainThread Checking intern URLs only; use --check-extern to check extern URLs.
LinkChecker 9.4.0              Copyright (C) 2000-2014 Bastian Kleineidam
LinkChecker comes with ABSOLUTELY NO WARRANTY!
This is free software, and you are welcome to redistribute it
under certain conditions. Look at the file `LICENSE` within this
distribution.
Get the newest version at http://wummel.github.io/linkchecker/
Write comments and bugs to https://github.com/wummel/linkchecker/issues
Support this project at http://wummel.github.io/linkchecker/donations.html

Start checking at 2019-09-03 20:53:02+001
10 threads active,    24 links queued,   24 links in  58 URLs checked, runtime 1 seconds
10 threads active,    91 links queued,   92 links in 193 URLs checked, runtime 6 seconds
10 threads active,    79 links queued,  104 links in 193 URLs checked, runtime 11 seconds
10 threads active,    65 links queued,  118 links in 193 URLs checked, runtime 16 seconds
10 threads active,    50 links queued,  133 links in 193 URLs checked, runtime 21 seconds
10 threads active,    35 links queued,  148 links in 193 URLs checked, runtime 26 seconds
10 threads active,    22 links queued,  161 links in 193 URLs checked, runtime 31 seconds
10 threads active,     8 links queued,  175 links in 193 URLs checked, runtime 36 seconds
 3 threads active,     0 links queued,  190 links in 193 URLs checked, runtime 41 seconds

Statistics:
Downloaded: 268.34KB.
Content types: 104 image, 8 text, 0 video, 0 audio, 7 application, 1 mail and 73 other.
URL lengths: min=18, max=106, avg=48.

That's it. 193 links in 193 URLs checked. 0 warnings found. 0 errors found.
Stopped checking at 2019-09-03 20:53:43+001 (41 seconds)

Create git hooks¶

To prevent developers from committing and pushing documentation that will fail the CI/CD pipeline, it is a good idea to run the lint and linkchecker automatically from Git. Git uses hooks to automate these processes, and we want to make sure that standard hooks that we want all developers to use are checked into source and versioned.

Firstly, create a file called bin/create-hook-symlinks with this contents:

#!/bin/sh

#
# Install all git hooks in the hooks/ directory into the local repository.
#

GIT_DIR=$(git rev-parse --show-toplevel)
HOOK_DIR="$GIT_DIR"/.git/hooks

for FILE in hooks/* ; do
    ln -s "$GIT_DIR/$FILE" "$HOOK_DIR/${FILE##*/}"
done

and make it executable:

chmod +x bin/create-hook-symlinks

Next, create a file called hooks/pre-commit with this contents:

#!/bin/bash

set -e

#
# To use this pre-commit hook you will need to install Markdown Lint (mdl).
# Please see the mdl project pages for details:
#
#    https://github.com/markdownlint/markdownlint
#
# Once mdl is installed, run this file to check that it works, the run
# ./bin/create-hook-symlinks to install this hook.
#

mdl --style .mdl.rb docs

and a file called hooks/pre-push with this contents:

#!/bin/sh

mkdocs build -c -d public
cd public
# mkdocs uses port 8889 (set in mkdocs.yml), so we use 8899 to avoid a clash.
webfsd -p 8899
linkchecker http://localhost:8899/
# Only kill the most recent webfsd process.
SERVER_PID="$(pidof webfsd | cut -d' ' -f1)"
kill -9 "$SERVER_PID"
cd ..
rm -Rf public

make them both executable:

chmod +x hooks/pre-*

Install the hooks yourself:

./bin/create-hook-symlinks

Now, commit these three files (in one commit) and push to the branch on origin to check that the hooks are working.

Note

We separate out the lint (which runs quickly) and the link checker (which runs very slowly). Normally, you will be committing often and pushing rarely, and this separation should help speed up your development time.

Add a CONTRIBUTING.md file¶

The CONTRIBUTING.md file should be in the root of your repository, and should describe how a new developer can start contributing to your documentation. You should describe how to clone your repository, how to serve the site locally, and how to install and run the lint and link checker. See brand.beautifulcanoe.com/blob/master/CONTRIBUTING.md for an example. Be sure to commit your file once you are done.

Deploying with GitLab pages¶

Typically, we deploy all static sites on GitLab Pages, rather than our own servers. We also use GitLab servers to run a CI/CD pipeline to check Markdown formatting and broken links.

In the instructions above you installed some extra packages in your operating system, and to make mdl and linkchecker run on the GitLab servers you will need to install the packages there, also. Doing this every time you run a pipeline is time consuming, so to speed the pipelines up, we store an OS image specific to the repository in the GitLab Container Registry, and use that to run our CI/CD pipeline. This blog post explains the process in more detail.

First create a file called .meta/Dockerfile with this contents:

FROM ruby:2.5

RUN gem install mdl && \
    apt-get update -qq && \
    apt-get install -y -qq webfs linkchecker && \
    apt-get install -y -qq python3-pip && \
    pip3 --version && \
    pip3 install virtualenv

Next create a file called .gitlab-ci.yml with the following contents, noting that the words IN CAPITAL LETTERS need to be replaced by something specific to your repository, but variables such as $CI_PROJECT_DIR are provided automatically in the GitLab environment, and should be left as they are:

# Change pip's cache directory to be inside the project directory since we can
# only cache local items.
variables:
    PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

# Pip's cache doesn't store the python packages:
#         https://pip.pypa.io/en/stable/reference/pip_install/#caching
# To cache the installed packages, we install them in a virtualenv.
cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
        - .cache/pip
        - venv/

meta-build-image:
    stage: prepare
    image: docker:stable
    services:
        - docker:dind
    script:
        - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
        - cd .meta
        - docker build -t $CI_REGISTRY/beautifulcanoe/projects/REPO/webtest:latest .
        - docker push $CI_REGISTRY/beautifulcanoe/projects/REPO/webtest:latest
    only:
        changes:
            - .meta/Dockerfile

build:
    stage: build
    image: registry.gitlab.com/beautifulcanoe/projects/REPO/webtest
    before_script:
        - virtualenv venv
        - source venv/bin/activate
        - which python
        - python --version
        - pip3 install -r requirements.txt
    script:
        - mdl --style .mdl.rb docs
        - mkdocs build
        - cd site && webfsd -p 4000
        - linkchecker http://localhost:4000/
    except:
        - master

pages:
    stage: deploy
    image: python:3.7-alpine
    before_script:
        - pip install virtualenv
        - virtualenv venv
        - pip install -r requirements.txt
    script:
        - mkdocs build
        - mv site public
    environment:
        name: production
        url: PUBLIC_URL
    artifacts:
        paths:
            - public
    only:
        - master

stages:
    - prepare
    - build
    - deploy

Commit and push both new files, and keep an eye on the pipeline that should be triggered once GitLab has received your code. Once your pipeline succeeds, put your MR up for review.