This tutorial is based on a GitLab instance installed using the Omnibus deployment method, on Rocky Linux 8.6 - other deployment types may require additional steps.
The self-hosted version of GitLab is a popular tool for privacy-conscious developers, open-source projects, and organisations looking to keep full control of their source code (like us!).
As an organisation operating a GitLab instance internally, one of our key considerations is ensuring we store our GitLab backups in a secure manner. While GitLab provides a suite of functionality allowing backups to be stored on Cloud object storage, we were keen to protect backups of the underlying Ionburst Cloud source code with Ionburst Cloud, while also minimising our configuration overhead.
In this tutorial, we will be covering how to use IBC S6 secure object storage and IonFS CLI, to backup self-hosted GitLab instances.
When backing up a GitLab instance, there are two main data sources to consider:
the application data - database, repositories etc.
the configuration and secrets data - stored within /etc/gitlab
GitLab application backups are typically performed with the gitlab-backup tool. Assuming no additional backup options have been added to the GitLab configuration file, this tool creates a tar archive of all GitLab application data and saves it in a well-known directory: /var/opt/gitlab/backups/.
Depending on how the GitLab instance is used, this archive can end up extremely large (10s of GB), typically when CI/CD build artifacts and the container registry are included. The GitLab backup tool allows aspects of the GitLab application to be skipped when backing up, using the SKIP environment variable. To minimise the amount of data stored, this tutorial will skip the artifacts stage.
By default, the GitLab backup process automatically generates a filename for the backup archive using the current timestamp and version of GitLab installed. This generated filename allows GitLab to automatically manage backup archives stored locally.
However, as we will be transferring the backups to IBC S6, and to make the automation process easier, we will override this automatic name using the BACKUP environment variable.
To begin, we need a user account setup on the underlying operating system of the GitLab instance, with sudo access - it is not recommended to use the root account.
IonFS CLI will also need to be installed on the GitLab instance, and configured with an IBC S6 data repository and Ionburst credentials file. For the purposes of this tutorial, we will use a local metadata repository stored in our user account's home directory.
2022-10-14 20:04:29 +0100 -- Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
2022-10-14 20:04:29 +0100 -- Backup ionfs-example is done.
The filesystem location used by GitLab is locked down to the git user only, so before we can upload our backup to IBC S6, we need to move the archive to an accessible location and change the ownership to our user account:
[josh@git ~]$ ls -lah /tmp/ionfs-example_gitlab_backup.tar
-rw-------. 1 josh josh 244M Oct 14 20:04 /tmp/ionfs-example_gitlab_backup.tar
As noted in the application backup output, the GitLab configuration files have not been included in the backup. We can create a configuration backup with the following:
sudo tar -cf /tmp/gitlab-config.tar /etc/gitlab/
sudo chown josh:josh /tmp/gitlab-config.tar
ls -lah /tmp/gitlab-config.tar
Example output:
[josh@git ~]$ sudo tar -cf /tmp/gitlab-config.tar /etc/gitlab/
Now that we've gone through the backup steps manually, we can have a look at wrapping the steps in a simple bash script that can then be used to automatically backup GitLab to IBC S6. We'll also add in some extra logic to add dates and other useful context to the backup filenames.
In this tutorial we've covered some background on the self-hosted GitLab backups, how to create them, and how to upload them to IBC S6 with the IonFS CLI. Finally we wrapped the steps in a simple bash script to allow the process to be automated.
All the steps covered in this tutorial, and the backup script, are currently used by Ionburst Cloud to backup and protect our internal GitLab instance. To keep up with the latest developments on the backup script, please checkout out our examples repository on GitHub.
