How to create a static website¶

Most websites that Beautiful Canoe creates for clients are dynamic sites with some sort of datastore, usually based on the Laravel framework. Like the (static) company website these are bespoke, and each site has its own styling, according to the needs of the client.

However, we have a small number of static sites for internal use, such as this site and our brand guidelines. Less frequently, we sometimes need to create static sites for clients (most likely to publicise a piece of research software) such as the Traffic3D site.

For these static sites, we do not use raw HTML (like the company website does), partly to save time, but also to avoid inconsistencies in styling and content. Instead, we have standardised our static sites on the mkdocs package for Python, with the mkdocs material theme which we use because it is clean, fully responsive and based on Google's material design guidelines. In the past we have been able to put together and deploy a whole new site within a couple of hours.

Starting a new repository: tweaks to our usual advice¶

Each static website should be in its own repository, even if the intention of the site is to publicise a specific software product. This helps us to separate out issues and merge requests for each piece of work we do, and to easily use CI/CD to test documentation and code separately. However, it does mean that sometimes you will need to change some software, then raise a separate issue in another repository to document that change.

Tip

Your repository should normally have the same name as its public URL. For example, this repository is beautifulcanoe/peopleops/docs.beautifulcanoe.com.

If you are starting a new repository for a static site, please read through and follow the start a new project HOWTO and related documentation.

For static sites, though, there are a few important tweaks to our usual advice:

You will not need a develop branch in your static website repository
When you create a .gitignore file, use the standard Python .gitignore file but add these lines to the file:

# Virtual environment.
venv/*

# Files generated by mkdocs.
site/*

When you create a Slack integration for the new repository, use one of the bc-SUBGROUP-gitlab channels if you are creating a site for internal Beautiful Canoe use, and use the PROJECT-gitlab channel if you are creating a client site for an existing client.

Warning

The rest of this HOWTO assumes that you have created an issue (to add the initial version of the site), started an MR and that you are working in the feature branch for that MR. Hopefully, if you have followed the instructions above, you will already be in a feature branch, where you have committed your .gitignore file and README.md.

Creating a new URL¶

By the time you get to the end of this HOWTO, your site will be public, so ask the CTO to create a new URL for you as soon as possible. If your static site will be for internal company use, the URL will be SOMETHING.beautifulcanoe.com, otherwise this will depend on your project and client.

Writing Markdown¶

The contents of your site will be in Markdown, not HTML. This will save you a lot of time in terms of styling and formatting, but it is important that you are familiar with Markdown syntax.

Tip

All Beautiful Canoe Markdown should have a newline at the end of every sentence. This means that when you raise a merge request, the diff will only consist of sentences that have changed. If this seems confusing, have a look at the way documentation for this site is formatted.

Using `mkdocs-material`¶

We do not recommend that you install Python packages globally on your development system, because you may need different versions of the same package for different projects. However, there are several ways to install Python packages locally. In this section, we cover two development workflows:

Developing with a mkdocs running in a Docker container (recommended)
Installing mkdocs-material locally with a Python virtualenv.

Using Docker means that you will be able to use exactly the same environment and dependencies on your development machine and your production deployment environment. It also means that you only need to install Docker locally.

`mkdocs-material` with Docker¶

If you are working on Ubuntu, this should work for you:

sudo apt-get install docker docker.io docker-compose

It is also a good idea to add yourself to the docker group, so that you can avoid using sudo:

sudo usermod -aG docker ${USER}

Before usermod takes effect, you will need to log out and log back in again. After that, please check that this has worked by running id -nG and checking that docker is listed in your groups.

For other operating systems, or for more details about Docker on Ubuntu, see the official Docker documentation:

Create a new, empty site with this command:

$ docker run --rm -it --user $(id -u):$(id -g) -v /etc/passwd:/etc/passwd -v ${PWD}:/docs squidfunk/mkdocs-material new
INFO    -  Writing config file: ./mkdocs.yml
INFO    -  Writing initial docs: ./docs/index.md
(venv) $

Note that, by default, the Docker container will create files as the root user, rather than the user currently logged in. In the line above, we use the /etc/passwd file on disk and the current user name $(id -n) and primary group $(id -g) to ensure that the files have the right ownership.

If you are not on a UNIX based system, you can remove these options, but you should check the ownership of the files in site/:

You can now build and serve the site with mkdocs serve running on your container:

$ docker run --rm -it -p 8888:8000 -v ${PWD}:/docs squidfunk/mkdocs-material

INFO    -  Building documentation...
WARNING -  Config value: 'dev_addr'. Warning: The use of the IP address '0.0.0.0' suggests a production environment or the use of a proxy to connect to the MkDocs server.
However, the MkDocs' server is intended for local development purposes only. Please use a third party production-ready server instead.
INFO    -  Cleaning site directory
INFO    -  Documentation built in 0.56 seconds
[I 210315 13:12:02 server:335] Serving on http://0.0.0.0:8000
INFO    -  Serving on http://0.0.0.0:8000
[I 210315 13:12:02 handlers:62] Start watching changes
INFO    -  Start watching changes
[I 210315 13:12:02 handlers:64] Start detecting changes
INFO    -  Start detecting changes
[I 210315 13:12:14 handlers:135] Browser Connected: http://127.0.0.1:8888/#bibliography
INFO    -  Browser Connected: http://127.0.0.1:8888/#bibliography
...

And open your browser at http://127.0.0.1:8888.

Note that you may need to change -p 8888:8000 in the invocation above if you have changed the dev_addr in your mkdocs.yml file.

It is unlikely that you will need to build the site without also viewing it in a browser. However, if you do, and you are using Linux, MacOS or another UNIX system you can do this via Docker:

docker run --rm -it --user $(id -u):$(id -g) -v /etc/passwd:/etc/passwd -v ${PWD}:/docs squidfunk/mkdocs-material build

This will build the HTML files and write them to a directory called site. If you wish to use a different directory name, add the switch -d DIRECTORY_NAME at the end of that line.

docker run --rm -it -v ${PWD}:/docs squidfunk/mkdocs-material build

`mkdocs-material` installed natively¶

To run mkdocs directly from your OS, you will need to install virtualenv and create a new virtual environment. This is a new environment with its own version of Python and its own package manager, so that you can install Python packages in a subdirectory of your repository:

sudo apt-get install python-virtualenv
virtualenv --python=python3 venv
. venv/bin/activate

Your command line prompt should now look like this:

(venv) $

If you need to get out of the virtual environment, run the deactivate command.

Next, create a file called requirements.txt containing the dependencies that you need:

pip install mkdocs-material
pip freeze >requirements.txt

At this stage, it is a good idea to add requirements.txt and commit it. You will need to run pip freeze and re-commit requirements.txt whenever you add a new dependency to the repository.

Create a new, empty site with this command:

(venv) $ mkdocs new .
INFO    -  Writing config file: ./mkdocs.yml
INFO    -  Writing initial docs: ./docs/index.md
(venv) $

Now you can build the HTML, and serve it locally:

(venv) $ mkdocs build
INFO    -  Cleaning site directory
INFO    -  Building documentation to directory: .../REPO/site

(venv) $ mkdocs serve
INFO    -  Building documentation...
INFO    -  Cleaning site directory
[I 191119 15:13:16 server:296] Serving on http://127.0.0.1:8000
[I 191119 15:13:16 handlers:62] Start watching changes
[I 191119 15:13:16 handlers:64] Start detecting changes
[I 191119 15:13:19 handlers:135] Browser Connected: http://127.0.0.1:8000/

And open your browser at http://127.0.0.1:8000 to see something like this:

Brand new mkdocs site

At this point you should commit the new mkdocs.yml and docs/index.md files.

Standard `mkdocs` configuration¶

Next, you will want to reconfigure the new site. You will need to tell mkdocs to use the Material theme, set some details about the site and add a logo, a favicon, and so on.

It would be a good idea to start with a mkdocs.yml file from one of the existing repositories, and adapt it to your needs.

In general, it is a good idea to have these extensions:

markdown_extensions:
    - admonition
    - pymdownx.emoji:
          emoji_index: !!python/name:materialx.emoji.twemoji
          emoji_generator: !!python/name:materialx.emoji.to_svg
    - pymdownx.highlight:
          css_class: codehilite
          extend_pygments_lang:
              - name: php-inline
                lang: php
                options:
                    startinline: true
    - pymdownx.superfences:
          custom_fences:
              - name: mermaid
                class: mermaid
                format: !!python/name:pymdownx.superfences.fence_div_format
    - pymdownx.inlinehilite:
    - toc:
          permalink: true

These enable:

The permalink: true setting in toc ensures that every section heading has a permalink that the reader can copy.

Note

If your static site is for internal Beautiful Canoe use, please do not use any of the default colour schemes. Instead, please add an extra CSS file called brand.css which should be a copy of the one in this site. You will want to change the social media links to something appropriate and use logos and favicons from our brand guidelines site. Again, you can take this customisation from the mkdocs.yml file in this repository.

Test your new configuration manually, and then commit it.

Linting and testing your new site¶

To make sure that your Markdown is valid, please use the mdl Markdown lint. You should use our standard mdl configuration.

To check for broken links, we use LinkChecker In general, we only use LinkChecker in GitLab CI pipelines, but you may sometimes wish to run it in your development environment, in which case the instructions here should be enough to start you off. However, we would generally recommend just reading the output of the LinkChecker tool in the pipeline logs on GitLab.

First, build the site using mkdocs build. This will generate HTML files in a directory called site/.

Then run LinkChecker from Docker:

docker run --rm -it  -v "$PWD"/site:/mnt ghcr.io/linkchecker/linkchecker index.html

You should see something like this:

$ docker run --rm -it  -v "$PWD"/site:/mnt ghcr.io/linkchecker/linkchecker index.html
WARNING linkcheck.check 2021-03-15 16:11:31,107 MainThread Running as root user; dropping privileges by changing user to nobody.
INFO linkcheck.cmdline 2021-03-15 16:11:31,107 MainThread Checking intern URLs only; use --check-extern to check extern URLs.
LinkChecker 9.4.0              Copyright (C) 2000-2014 Bastian Kleineidam
LinkChecker comes with ABSOLUTELY NO WARRANTY!
This is free software, and you are welcome to redistribute it
under certain conditions. Look at the file `LICENSE' within this
distribution.
Get the newest version at https://linkchecker.github.io/linkchecker/
Write comments and bugs to https://github.com/linkchecker/linkchecker/issues

Start checking at 2021-03-15 16:11:31+000

Statistics:
Downloaded: 367.56KB.
Content types: 50 image, 16 text, 0 video, 0 audio, 43 application, 0 mail and 93 other.
URL lengths: min=8, max=899, avg=79.

That's it. 202 links in 202 URLs checked. 0 warnings found. 0 errors found.
Stopped checking at 2021-03-15 16:11:31+000 (0.72 seconds)

$

Or, if you have a broken link:

$ docker run --rm -it  -v "$PWD"/site:/mnt linkchecker/linkchecker index.html
WARNING linkcheck.check 2021-03-15 16:10:20,326 MainThread Running as root user; dropping privileges by changing user to nobody.
INFO linkcheck.cmdline 2021-03-15 16:10:20,326 MainThread Checking intern URLs only; use --check-extern to check extern URLs.
LinkChecker 9.4.0              Copyright (C) 2000-2014 Bastian Kleineidam
LinkChecker comes with ABSOLUTELY NO WARRANTY!
This is free software, and you are welcome to redistribute it
under certain conditions. Look at the file `LICENSE' within this
distribution.
Get the newest version at https://linkchecker.github.io/linkchecker/
Write comments and bugs to https://github.com/linkchecker/linkchecker/issues

Start checking at 2021-03-15 16:10:20+000

URL        `assets/images/sdadaFrontPageImage.jpg'
Name       `Traffic3D Front Page Image'
Parent URL file:///mnt/index.html, line 618, col 4
Real URL   file:///mnt/assets/images/sdadaFrontPageImage.jpg
Check time 0.000 seconds
Result     Error: URLError: <urlopen error [Errno 2] No such file or directory: '/mnt/assets/images/sdadaFrontPageImage.jpg'>

Statistics:
Downloaded: 367.56KB.
Content types: 50 image, 16 text, 0 video, 0 audio, 43 application, 0 mail and 94 other.
URL lengths: min=8, max=899, avg=79.

That's it. 203 links in 203 URLs checked. 0 warnings found. 1 error found.
Stopped checking at 2021-03-15 16:10:21+000 (0.67 seconds)

$

Create git hooks¶

To prevent developers from committing and pushing documentation that will fail the CI/CD pipeline, it is a good idea to run the lint automatically from Git. Git uses hooks to automate these processes, and we want to make sure that standard hooks that we want all developers to use are checked into source control.

You should use our standard bin/create-hook-symlinks script, and add your hooks to a new hooks/ directory. For mkdocs-material sites we usually have these hooks:

A pre-commit hook that runs mdl to lint all Markdown files in the repository.
A pre-push hook that checks that the site builds without error.

We usually do not run LinkChecker in a hook, as it runs quite slowly.

You should start by copying Git hooks from one of our existing mkdocs-material projects and add them to the repository in a one or more commits.

Add a CONTRIBUTING.md file¶

The CONTRIBUTING.md file should be in the root of your repository, and should describe how a new developer can start contributing to your documentation. You should describe how to clone your repository, how to serve the site locally, and how to install and run the lint and link checker. See the CONTRIBUTING.md in this repository for an example. Be sure to commit your file once you are done.

Deploying with GitLab pages¶

Typically, we deploy all static sites on GitLab Pages, rather than our own servers. We also use GitLab servers to run a CI/CD pipeline to check Markdown formatting and broken links. It would be a good idea to start by coping the .gitlab-ci.yml file from one of our existing mkdocs-material sites.

Rather than creating a custom Docker container with all the required packages, each job or stage should use the Docker configuration it requires. Wherever possible, this should mean using the "official" Docker images relating to the necessary packaeges. For example, there is an official package for mkdocs-material and LinkChecker.

Once you have a working .gitlab-ci.yml file and your pipeline succeeds, put your MR up for review.

Monitoring¶

Please add your new site to the company dashboard on uptimerobot for live monitoring, and ensure that alerts go to alert@beautifulcanoe.com and to the relevant -project Slack channel (or bc-online if your project is internal). For the Slack alert, you will need to create a new webhook in our Slack app. If you don't have access to our uptimerobot account, or our Slack app, please ask the CTO.

If your site has an SSL certificate, please also raise an issue and an MR to add it to our SSL validation script in the cron repository.