Skip to main content

Protecting CI/CD artifacts with IBC S6 and IonFS CLI

· 6 min read
Josh Fraser
info

Prerequisites - before you begin, please ensure:

Overview​

As covered previously in our GitLab backup tutorial, we (Ionburst) use IBC S6 to protect the backups from our internal GitLab instance. In this tutorial, we will look at the other way we use IBC S6 with GitLab; as artifact storage for our GitLab CI/CD builds.

For this tutorial, we'll be using a GitLab repository containing the base code for our namegen utility. We'll add a GitLab CI/CD config file to build the application, then use IonFS CLI to store the build artifact in IBC S6. Finally, we'll download the artifact to the local machine using IonFS CLI, and execute it.

One key point to note for this tutorial: we will be using an IonFS metadata repository set up in Amazon S3. As our GitLab CI/CD pipelines will be running in a containerised, ephemeral environment, this ensures the IonFS metadata is persisted elsewhere. It will also allow the build artifact to be downloaded to the local machine at the end of the tutorial.

Shared Responsibility Model Breakdown​

Customer Responsibility​

  • You, the customer, are responsible for the secure management of the Ionburst Cloud credentials used by ionfs.
  • You, the customer, are responsible for the security of ionfs metadata repositories and the metadata stored in them.
  • You, the customer, are responsible for the security of the GitLab application and underlying instance - if self-hosting GitLab.
  • You, the customer, are responsible for the security of the GitLab projects used in conjunction with this tutorial.

Ionburst Cloud Responsibility​

  • We are responsible for the security of GitLab backup data stored in IBC S6 using ionfs.
  • We are responsible for the underlying security and availability of the Ionburst Cloud platform.

GitLab CI/CD​

To enable GitLab CI/CD for a project, a file, gitlab-ci.yml, is added to the root of the project. Within this file, the concepts of stages and jobs are used to define the tasks needed to build, test, and deploy an application or piece of software.

For this tutorial, we will be creating a single build stage, with a job that will compile our namegen application, then upload the binary to IBC S6.

Setting up the repository​

First, we'll add a .gitlab-ci.yml file to the root of our project.

image: jishf/golang-runner-1.19
stages:
- build
compile_linux_amd64:
stage: build
before_script:
- export PATH=$PATH:/usr/local/go/bin
- mkdir -p ~/.ionfs
- cp $IONFS_CONFIG ~/.ionfs/appsettings.json
- mkdir -p build
- export RELEASE_VERSION=0.1.0
- go get -v -d
script:
- GOOS=linux GOARCH=amd64 go build -o ./build/namegen-linux-amd64-$RELEASE_VERSION -ldflags="-X=main.appVersion=$RELEASE_VERSION"
- ionfs put build/namegen-linux-amd64-$RELEASE_VERSION ion://builds/
- ionfs ls ion://builds/

To break this file down:

  • image is the Docker image we're going to run our pipeline with, the defined image has both Go and ionfs installed.
  • stages defines the different pipeline stages, we're only specifying a build stage.
  • compile_linux_amd64:
    • this is our configured job, running in the build stage
    • the before_script is ensuring the go command is available in the $PATH, creating the ~/.ionfs config directory and file, creating a build directory, then setting an environment variable with semantic version for our build.
    • the script is building our namegen binary, and embedding our version in the build, then uploading the binary to IBC S6, before listing the IonFS metadata repository.

Before we commit the .gitlab-ci.yml file to our repository, we first need to add some environment variables to our repository CI/CD configuration:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • AWS_REGION
  • IONBURST_ID
  • IONBURST_KEY
  • IONBURST_URI
  • IONFS_CONFIG

The IONFS_CONFIG variable should be of type "file" and look like the following:

{
"IonFS": {
"MaxSize": "50000000",
"Verbose": "false",
"DefaultClassification": "Restricted",
"Repositories": [
{
"Name": "builds",
"Usage": "Data",
"Class": "Ionburst.Apps.IonFS.Repo.S3.MetadataS3",
"Assembly": "Ionburst.Apps.IonFS.Repo.S3",
"DataStore": "ibc-example"
}
],
"DefaultRepository": "builds"
}
}

Once these environment variables have been configured, we can commit the gitlab-ci.yml to our repository to kick off the first pipeline, which looks something like:

Running with gitlab-runner 15.5.0 (0d4137b8)
on ionburst-runner-public vyQm-zNw
Preparing the "docker" executor 00:02
Using Docker executor with image jishf/golang-runner-1.19 ...
Pulling docker image jishf/golang-runner-1.19 ...
Using docker image sha256:cb41b40fca0d36126a9eccf01b1b0e6d6fd4b55380314543b9450b8fd1ba9142 for jishf/golang-runner-1.19 with digest jishf/golang-runner-1.19@sha256:6730a3a0603d91780e15a297a2fcc1ae32de5b5213afc639ca4a6e035118e2c6 ...
Preparing environment 00:01
Running on runner-vyqm-znw-project-40659015-concurrent-0 via ionburst-runner-public...
Getting source from Git repository 00:01
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/ionburst/ionfs-cicd-example/.git/
Checking out e3acd96e as main...
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:08
Using docker image sha256:cb41b40fca0d36126a9eccf01b1b0e6d6fd4b55380314543b9450b8fd1ba9142 for jishf/golang-runner-1.19 with digest jishf/golang-runner-1.19@sha256:6730a3a0603d91780e15a297a2fcc1ae32de5b5213afc639ca4a6e035118e2c6 ...
$ export PATH=$PATH:/usr/local/go/bin
$ mkdir -p ~/.ionfs
$ cp $IONFS_CONFIG ~/.ionfs/appsettings.json
$ mkdir -p build
$ export RELEASE_VERSION=0.1.0
$ go get -v -d
$ GOOS=linux GOARCH=amd64 go build -o ./build/namegen-linux-amd64-$RELEASE_VERSION -ldflags="-X=main.appVersion=$RELEASE_VERSION"
$ ionfs put build/namegen-linux-amd64-$RELEASE_VERSION ion://builds/
$ ionfs ls ion://builds/
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Directory of ion://builds/
namegen-linux-amd64-0.1.0 10/17/2022 18:35:38
Cleaning up project directory and file based variables 00:01
Job succeeded

Retrieving the artifact​

Now that we've successfully deployed our pipeline, our artifact is safely stored in IBC S6, and ready to be consumed or accessed. For our internal usage at Ionburst, this is typically to be picked up by a container build, or to distribute internally.

We can demonstrate the latter by configuring the IonFS repo and appropriate credentials used in our pipeline on our local machine. Once setup, we can list the repo, download our namegen artifact, and run it locally:

ionfs ls ion://builds
ionfs get get ion://builds/namegen-linux-amd64-0.1.0 namegen
chmod +x namegen
./namegen

Example output:

[hello@ionfs-cicd-example ~]# ionfs ls
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Directory of ion://builds/
namegen-linux-amd64-0.1.0 10/17/2022 18:35:38
[hello@ionfs-cicd-example ~]# ionfs get ion://builds/namegen-linux-amd64-0.1.0 namegen
[hello@ionfs-cicd-example ~]# chmod +x namegen
[hello@ionfs-cicd-example ~]# ./namegen
tender-boyd-orr

Wrapping up​

In this tutorial, we've covered some background on GitLab CI/CD artifacts and how to protect them with IBC S6 and IonFS CLI.

All the steps covered in this tutorial are currently used by Ionburst Cloud to protect our internal build artifacts. To keep up with the latest developments on using IonFS CLI with GitLab CI/CD, please checkout out our example repository on GitHub.