Skip to main content

6 posts tagged with "IonFS"

View All Tags

· 6 min read
Josh Fraser

Overview

As covered previously in our GitLab backup tutorial, we (Ionburst) use IBC S6 to protect the backups from our internal GitLab instance. In this tutorial, we will look at the other way we use IBC S6 with GitLab; as artifact storage for our GitLab CI/CD builds.

For this tutorial, we'll be using a GitLab repository containing the base code for our namegen utility. We'll add a GitLab CI/CD config file to build the application, then use IonFS CLI to store the build artifact in IBC S6. Finally, we'll download the artifact to the local machine using IonFS CLI, and execute it.

One key point to note for this tutorial: we will be using an IonFS metadata repository set up in Amazon S3. As our GitLab CI/CD pipelines will be running in a containerised, ephemeral environment, this ensures the IonFS metadata is persisted elsewhere. It will also allow the build artifact to be downloaded to the local machine at the end of the tutorial.

Shared Responsibility Model Breakdown

Customer Responsibility

  • You, the customer, are responsible for the secure management of the Ionburst Cloud credentials used by ionfs.
  • You, the customer, are responsible for the security of ionfs metadata repositories and the metadata stored in them.
  • You, the customer, are responsible for the security of the GitLab application and underlying instance - if self-hosting GitLab.
  • You, the customer, are responsible for the security of the GitLab projects used in conjunction with this tutorial.

Ionburst Cloud Responsibility

  • We are responsible for the security of GitLab backup data stored in IBC S6 using ionfs.
  • We are responsible for the underlying security and availability of the Ionburst Cloud platform.

GitLab CI/CD

To enable GitLab CI/CD for a project, a file, gitlab-ci.yml, is added to the root of the project. Within this file, the concepts of stages and jobs are used to define the tasks needed to build, test, and deploy an application or piece of software.

For this tutorial, we will be creating a single build stage, with a job that will compile our namegen application, then upload the binary to IBC S6.

Setting up the repository

First, we'll add a .gitlab-ci.yml file to the root of our project.

image: jishf/golang-runner-1.19
stages:
- build
compile_linux_amd64:
stage: build
before_script:
- export PATH=$PATH:/usr/local/go/bin
- mkdir -p ~/.ionfs
- cp $IONFS_CONFIG ~/.ionfs/appsettings.json
- mkdir -p build
- export RELEASE_VERSION=0.1.0
- go get -v -d
script:
- GOOS=linux GOARCH=amd64 go build -o ./build/namegen-linux-amd64-$RELEASE_VERSION -ldflags="-X=main.appVersion=$RELEASE_VERSION"
- ionfs put build/namegen-linux-amd64-$RELEASE_VERSION ion://builds/
- ionfs ls ion://builds/

To break this file down:

  • image is the Docker image we're going to run our pipeline with, the defined image has both Go and ionfs installed.
  • stages defines the different pipeline stages, we're only specifying a build stage.
  • compile_linux_amd64:
    • this is our configured job, running in the build stage
    • the before_script is ensuring the go command is available in the $PATH, creating the ~/.ionfs config directory and file, creating a build directory, then setting an environment variable with semantic version for our build.
    • the script is building our namegen binary, and embedding our version in the build, then uploading the binary to IBC S6, before listing the IonFS metadata repository.

Before we commit the .gitlab-ci.yml file to our repository, we first need to add some environment variables to our repository CI/CD configuration:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • AWS_REGION
  • IONBURST_ID
  • IONBURST_KEY
  • IONBURST_URI
  • IONFS_CONFIG

The IONFS_CONFIG variable should be of type "file" and look like the following:

{
"IonFS": {
"MaxSize": "50000000",
"Verbose": "false",
"DefaultClassification": "Restricted",
"Repositories": [
{
"Name": "builds",
"Usage": "Data",
"Class": "Ionburst.Apps.IonFS.Repo.S3.MetadataS3",
"Assembly": "Ionburst.Apps.IonFS.Repo.S3",
"DataStore": "ibc-example"
}
],
"DefaultRepository": "builds"
}
}

Once these environment variables have been configured, we can commit the gitlab-ci.yml to our repository to kick off the first pipeline, which looks something like:

Running with gitlab-runner 15.5.0 (0d4137b8)
on ionburst-runner-public vyQm-zNw
Preparing the "docker" executor 00:02
Using Docker executor with image jishf/golang-runner-1.19 ...
Pulling docker image jishf/golang-runner-1.19 ...
Using docker image sha256:cb41b40fca0d36126a9eccf01b1b0e6d6fd4b55380314543b9450b8fd1ba9142 for jishf/golang-runner-1.19 with digest jishf/golang-runner-1.19@sha256:6730a3a0603d91780e15a297a2fcc1ae32de5b5213afc639ca4a6e035118e2c6 ...
Preparing environment 00:01
Running on runner-vyqm-znw-project-40659015-concurrent-0 via ionburst-runner-public...
Getting source from Git repository 00:01
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/ionburst/ionfs-cicd-example/.git/
Checking out e3acd96e as main...
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:08
Using docker image sha256:cb41b40fca0d36126a9eccf01b1b0e6d6fd4b55380314543b9450b8fd1ba9142 for jishf/golang-runner-1.19 with digest jishf/golang-runner-1.19@sha256:6730a3a0603d91780e15a297a2fcc1ae32de5b5213afc639ca4a6e035118e2c6 ...
$ export PATH=$PATH:/usr/local/go/bin
$ mkdir -p ~/.ionfs
$ cp $IONFS_CONFIG ~/.ionfs/appsettings.json
$ mkdir -p build
$ export RELEASE_VERSION=0.1.0
$ go get -v -d
$ GOOS=linux GOARCH=amd64 go build -o ./build/namegen-linux-amd64-$RELEASE_VERSION -ldflags="-X=main.appVersion=$RELEASE_VERSION"
$ ionfs put build/namegen-linux-amd64-$RELEASE_VERSION ion://builds/
$ ionfs ls ion://builds/
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Directory of ion://builds/
namegen-linux-amd64-0.1.0 10/17/2022 18:35:38
Cleaning up project directory and file based variables 00:01
Job succeeded

Retrieving the artifact

Now that we've successfully deployed our pipeline, our artifact is safely stored in IBC S6, and ready to be consumed or accessed. For our internal usage at Ionburst, this is typically to be picked up by a container build, or to distribute internally.

We can demonstrate the latter by configuring the IonFS repo and appropriate credentials used in our pipeline on our local machine. Once setup, we can list the repo, download our namegen artifact, and run it locally:

ionfs ls ion://builds
ionfs get get ion://builds/namegen-linux-amd64-0.1.0 namegen
chmod +x namegen
./namegen

Example output:

[hello@ionfs-cicd-example ~]# ionfs ls
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Directory of ion://builds/
namegen-linux-amd64-0.1.0 10/17/2022 18:35:38
[hello@ionfs-cicd-example ~]# ionfs get ion://builds/namegen-linux-amd64-0.1.0 namegen
[hello@ionfs-cicd-example ~]# chmod +x namegen
[hello@ionfs-cicd-example ~]# ./namegen
tender-boyd-orr

Wrapping up

In this tutorial, we've covered some background on GitLab CI/CD artifacts and how to protect them with IBC S6 and IonFS CLI.

All the steps covered in this tutorial are currently used by Ionburst Cloud to protect our internal build artifacts. To keep up with the latest developments on using IonFS CLI with GitLab CI/CD, please checkout out our example repository on GitHub.

· 11 min read
Josh Fraser
info

Prerequisites - before you begin, please ensure:

Please also note:

  • This tutorial is based on a GitLab instance installed using the Omnibus deployment method, on Rocky Linux 8.6 - other deployment types may require additional steps.

Overview

The self-hosted version of GitLab is a popular tool for privacy-conscious developers, open-source projects, and organisations looking to keep full control of their source code (like us!).

As an organisation operating a GitLab instance internally, one of our key considerations is ensuring we store our GitLab backups in a secure manner. While GitLab provides a suite of functionality allowing backups to be stored on Cloud object storage, we were keen to protect backups of the underlying Ionburst Cloud source code with Ionburst Cloud, while also minimising our configuration overhead.

In this tutorial, we will be covering how to use IBC S6 secure object storage and IonFS CLI, to backup self-hosted GitLab instances.

Shared Responsibility Model Breakdown

Customer Responsibility

  • You, the customer, are responsible for the secure management of the Ionburst Cloud credentials used by ionfs.
  • You, the customer, are responsible for the security of ionfs metadata repositories and the metadata stored in them.
  • You, the customer, are responsible for the security of the GitLab application and underlying instance.

Ionburst Cloud Responsibility

  • We are responsible for the security of GitLab backup data stored in IBC S6 using ionfs.
  • We are responsible for the underlying security and availability of the Ionburst Cloud platform.

GitLab backups

When backing up a GitLab instance, there are two main data sources to consider:

  • the application data - database, repositories etc.
  • the configuration and secrets data - stored within /etc/gitlab

GitLab application backups are typically performed with the gitlab-backup tool. Assuming no additional backup options have been added to the GitLab configuration file, this tool creates a tar archive of all GitLab application data and saves it in a well-known directory: /var/opt/gitlab/backups/.

Depending on how the GitLab instance is used, this archive can end up extremely large (10s of GB), typically when CI/CD build artifacts and the container registry are included. The GitLab backup tool allows aspects of the GitLab application to be skipped when backing up, using the SKIP environment variable. To minimise the amount of data stored, this tutorial will skip the artifacts stage.

By default, the GitLab backup process automatically generates a filename for the backup archive using the current timestamp and version of GitLab installed. This generated filename allows GitLab to automatically manage backup archives stored locally.

However, as we will be transferring the backups to IBC S6, and to make the automation process easier, we will override this automatic name using the BACKUP environment variable.

Getting started

To begin, we need a user account setup on the underlying operating system of the GitLab instance, with sudo access - it is not recommended to use the root account.

IonFS CLI will also need to be installed on the GitLab instance, and configured with an IBC S6 data repository and Ionburst credentials file. For the purposes of this tutorial, we will use a local metadata repository stored in our user account's home directory.

Our sample ionfs configuration file:

{
"IonFS": {
"MaxSize": "50000000",
"Verbose": "false",
"DefaultClassification": "Restricted",
"Repositories": [
{
"Name": "gitlab-local",
"Usage": "Data",
"Class": "Ionburst.Apps.IonFS.Repo.LocalFS.MetadataLocalFS",
"Assembly": "Ionburst.Apps.IonFS.Repo.LocalFS",
"DataStore": "/home/josh/gitlab-local"
}
],
"DefaultRepository": "gitlab-local"
},
"Ionburst": {
"Profile": "gitlab",
"IonburstUri": "https://api.eu-west-1.ionburst.cloud/",
"TraceCredentialsFile": "OFF"
}
}

We can verify the IonFS configuration with:

ionfs repos

Which should return something similar to:

[josh@gitlab-example ~]$ ionfs repos
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Available Repositories (*default):
* [d] ion://gitlab-local/ (Ionburst.Apps.IonFS.Repo.LocalFS.MetadataLocalFS)

Creating the GitLab backups

With ionfs successfully configured, we can now create our GitLab backups. For the application backup, we can use the following:

sudo gitlab-backup create BACKUP="ionfs-example" SKIP="artifacts"

This will look something like:

[josh@git ~]$ sudo gitlab-backup create BACKUP="ionfs-example" SKIP="artifacts"
2022-10-14 20:04:19 +0100 -- Dumping main_database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2022-10-14 20:04:24 +0100 -- Dumping main_database ... done
2022-10-14 20:04:24 +0100 -- Dumping ci_database ... [DISABLED]
2022-10-14 20:04:24 +0100 -- Dumping repositories ...
--- truncated ---
2022-10-14 20:04:28 +0100 -- Dumping repositories ... done
2022-10-14 20:04:28 +0100 -- Dumping uploads ...
2022-10-14 20:04:28 +0100 -- Dumping uploads ... done
2022-10-14 20:04:28 +0100 -- Dumping builds ...
2022-10-14 20:04:28 +0100 -- Dumping builds ... done
2022-10-14 20:04:28 +0100 -- Dumping artifacts ... [SKIPPED]
2022-10-14 20:04:28 +0100 -- Dumping pages ...
2022-10-14 20:04:28 +0100 -- Dumping pages ... done
2022-10-14 20:04:28 +0100 -- Dumping lfs objects ...
2022-10-14 20:04:28 +0100 -- Dumping lfs objects ... done
2022-10-14 20:04:28 +0100 -- Dumping terraform states ...
2022-10-14 20:04:28 +0100 -- Dumping terraform states ... done
2022-10-14 20:04:28 +0100 -- Dumping container registry images ... [DISABLED]
2022-10-14 20:04:28 +0100 -- Dumping packages ...
2022-10-14 20:04:28 +0100 -- Dumping packages ... done
2022-10-14 20:04:28 +0100 -- Creating backup archive: ionfs-example_gitlab_backup.tar ...
2022-10-14 20:04:29 +0100 -- Creating backup archive: ionfs-example_gitlab_backup.tar ... done
2022-10-14 20:04:29 +0100 -- Uploading backup archive to remote storage ... [SKIPPED]
2022-10-14 20:04:29 +0100 -- Deleting old backups ...
2022-10-14 20:04:29 +0100 -- Deleting old backups ... done. (0 removed)
2022-10-14 20:04:29 +0100 -- Deleting tar staging files ...
2022-10-14 20:04:29 +0100 -- Cleaning up /var/opt/gitlab/backups/backup_information.yml
2022-10-14 20:04:29 +0100 -- Cleaning up /var/opt/gitlab/backups/db
2022-10-14 20:04:29 +0100 -- Cleaning up /var/opt/gitlab/backups/repositories
2022-10-14 20:04:29 +0100 -- Cleaning up /var/opt/gitlab/backups/uploads.tar.gz
2022-10-14 20:04:29 +0100 -- Cleaning up /var/opt/gitlab/backups/builds.tar.gz
2022-10-14 20:04:29 +0100 -- Cleaning up /var/opt/gitlab/backups/pages.tar.gz
2022-10-14 20:04:29 +0100 -- Cleaning up /var/opt/gitlab/backups/lfs.tar.gz
2022-10-14 20:04:29 +0100 -- Cleaning up /var/opt/gitlab/backups/terraform_state.tar.gz
2022-10-14 20:04:29 +0100 -- Cleaning up /var/opt/gitlab/backups/packages.tar.gz
2022-10-14 20:04:29 +0100 -- Deleting tar staging files ... done
2022-10-14 20:04:29 +0100 -- Deleting backups/tmp ...
2022-10-14 20:04:29 +0100 -- Deleting backups/tmp ... done
2022-10-14 20:04:29 +0100 -- Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
2022-10-14 20:04:29 +0100 -- Backup ionfs-example is done.

The filesystem location used by GitLab is locked down to the git user only, so before we can upload our backup to IBC S6, we need to move the archive to an accessible location and change the ownership to our user account:

sudo mv /var/opt/gitlab/backups/ionfs-example_gitlab_backup.tar /tmp/
sudo chown josh:josh /tmp/ionfs-example_gitlab_backup.tar
ls -lah /tmp/ionfs-example_gitlab_backup.tar

Example output:

[josh@git ~]$ sudo mv /var/opt/gitlab/backups/ionfs-example_gitlab_backup.tar /tmp/
[josh@git ~]$ ls -lah /tmp/ionfs-example_gitlab_backup.tar
-rw-------. 1 josh josh 244M Oct 14 20:04 /tmp/ionfs-example_gitlab_backup.tar

As noted in the application backup output, the GitLab configuration files have not been included in the backup. We can create a configuration backup with the following:

sudo tar -cf /tmp/gitlab-config.tar /etc/gitlab/
sudo chown josh:josh /tmp/gitlab-config.tar
ls -lah /tmp/gitlab-config.tar

Example output:

[josh@git ~]$ sudo tar -cf /tmp/gitlab-config.tar /etc/gitlab/
[josh@git ~]$ sudo chown josh:josh /tmp/gitlab-config.tar
[josh@git ~]$ ls -lah /tmp/gitlab-config.tar
-rw-r--r--. 1 josh josh 320K Oct 14 20:32 /tmp/gitlab-config.tar

Uploading the backups to IBC S6

Now that our GitLab application and configuration backups are ready, we can upload them to IBC S6 with ionfs.

First, we create a directory within the metadata repository:

ionfs mkdir ion://gitlab-backups

We can now upload each of our backups:

ionfs put /tmp/ionfs-example_gitlab_backup.tar ion://gitlab-backups/
ionfs put /tmp/gitlab-config.tar ion://gitlab-backups/

Finally, we can verify that the backups have uploaded successfully, and remove our local copies:

ionfs ls ion://gitlab-backups
rm -f /tmp/ionfs-example_gitlab_backup.tar
rm -f /tmp/gitlab-config.tar
[josh@git ~]$ ionfs mkdir ion://gitlab-backups
[josh@git ~]$ ionfs put /tmp/ionfs-example_gitlab_backup.tar ion://gitlab-backups/
[josh@git ~]$ ionfs put /tmp/gitlab-config.tar ion://gitlab-backups/
[josh@git ~]$ ionfs ls ion://gitlab-backups
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Directory of ion://gitlab-local/gitlab-backups/
gitlab-backups/gitlab-config.tar 14/10/2022 20:40:17
gitlab-backups/ionfs-example_gitlab_backup.tar 14/10/2022 20:40:12
[josh@git ~]$ rm -f /tmp/ionfs-example_gitlab_backup.tar
[josh@git ~]$ rm -f /tmp/gitlab-config.tar

Building the backup script

Now that we've gone through the backup steps manually, we can have a look at wrapping the steps in a simple bash script that can then be used to automatically backup GitLab to IBC S6. We'll also add in some extra logic to add dates and other useful context to the backup filenames.

So let's take a look at the script:

#!/bin/bash
set -eou pipefail
## setup vars
date=$(date "+%Y%m%d-%H%M%S")
name="$date"_gitlab_backup.tar
data_path=/var/opt/gitlab/backups/
config_path=/etc/gitlab/
user=$(whoami)
## create backups
sudo gitlab-backup create BACKUP="$date" SKIP="artifacts"
sudo mv "$data_path/$name" /tmp/
sudo chown "$user:$user" /tmp/"$name"
sudo tar -cf "/tmp/$date"_gitlab_config.tar "$config_path"
sudo chown "$user:$user" "/tmp/$date"_gitlab_config.tar
## upload to IBC S6
ionfs put "/tmp/$name" ion://gitlab-backups/
ionfs put "/tmp/$date"_gitlab_config.tar ion://gitlab-backups/
## verify and delete local copies
ionfs ls ion://gitlab-backups/
rm -f "/tmp/$name"
rm -f "/tmp/$date"_gitlab_config.tar

Running through the script by section:

  • set -eou pipefail - this is used to ensure the script exits immediately in the event of any failures.
  • We're also setting up the following variables
    • $date is used to add a timestamp to our backup filenames, using the following format: 20221014-205835
    • name is the full filename of the application backup
    • $data_path is the filesystem location used by GitLab to store the application backup
    • $config_path is the location of the GitLab configuration files
    • $user is the current user, used to change the backup file ownership
  • We then create the backups, adding the variables to the manual steps above.
  • Once the backups are created, we upload them to IBC S6 using ionfs
  • Finally, we list the contents of the ionfs metadata repository, and remove the local copies of the backup files.

An example execution of the script would look like:

[josh@git ~]$ ./gitlab-backup.sh
[sudo] password for josh:
2022-10-14 20:59:07 +0100 -- Dumping main_database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2022-10-14 20:59:11 +0100 -- Dumping main_database ... done
2022-10-14 20:59:11 +0100 -- Dumping ci_database ... [DISABLED]
2022-10-14 20:59:11 +0100 -- Dumping repositories ...
--- truncated ---
2022-10-14 20:59:16 +0100 -- Dumping repositories ... done
2022-10-14 20:59:16 +0100 -- Dumping uploads ...
2022-10-14 20:59:16 +0100 -- Dumping uploads ... done
2022-10-14 20:59:16 +0100 -- Dumping builds ...
2022-10-14 20:59:16 +0100 -- Dumping builds ... done
2022-10-14 20:59:16 +0100 -- Dumping artifacts ... [SKIPPED]
2022-10-14 20:59:16 +0100 -- Dumping pages ...
2022-10-14 20:59:16 +0100 -- Dumping pages ... done
2022-10-14 20:59:16 +0100 -- Dumping lfs objects ...
2022-10-14 20:59:16 +0100 -- Dumping lfs objects ... done
2022-10-14 20:59:16 +0100 -- Dumping terraform states ...
2022-10-14 20:59:16 +0100 -- Dumping terraform states ... done
2022-10-14 20:59:16 +0100 -- Dumping container registry images ... [DISABLED]
2022-10-14 20:59:16 +0100 -- Dumping packages ...
2022-10-14 20:59:16 +0100 -- Dumping packages ... done
2022-10-14 20:59:16 +0100 -- Creating backup archive: 20221014-205835_gitlab_backup.tar ...
2022-10-14 20:59:16 +0100 -- Creating backup archive: 20221014-205835_gitlab_backup.tar ... done
2022-10-14 20:59:16 +0100 -- Uploading backup archive to remote storage ... [SKIPPED]
2022-10-14 20:59:16 +0100 -- Deleting old backups ...
2022-10-14 20:59:16 +0100 -- Deleting old backups ... done. (0 removed)
2022-10-14 20:59:16 +0100 -- Deleting tar staging files ...
2022-10-14 20:59:16 +0100 -- Cleaning up /var/opt/gitlab/backups/backup_information.yml
2022-10-14 20:59:16 +0100 -- Cleaning up /var/opt/gitlab/backups/db
2022-10-14 20:59:16 +0100 -- Cleaning up /var/opt/gitlab/backups/repositories
2022-10-14 20:59:16 +0100 -- Cleaning up /var/opt/gitlab/backups/uploads.tar.gz
2022-10-14 20:59:16 +0100 -- Cleaning up /var/opt/gitlab/backups/builds.tar.gz
2022-10-14 20:59:16 +0100 -- Cleaning up /var/opt/gitlab/backups/pages.tar.gz
2022-10-14 20:59:16 +0100 -- Cleaning up /var/opt/gitlab/backups/lfs.tar.gz
2022-10-14 20:59:16 +0100 -- Cleaning up /var/opt/gitlab/backups/terraform_state.tar.gz
2022-10-14 20:59:16 +0100 -- Cleaning up /var/opt/gitlab/backups/packages.tar.gz
2022-10-14 20:59:16 +0100 -- Deleting tar staging files ... done
2022-10-14 20:59:16 +0100 -- Deleting backups/tmp ...
2022-10-14 20:59:16 +0100 -- Deleting backups/tmp ... done
2022-10-14 20:59:16 +0100 -- Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
2022-10-14 20:59:16 +0100 -- Backup 20221014-205835 is done.
tar: Removing leading `/' from member names
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Directory of ion://gitlab-local/gitlab-backups/
gitlab-backups/20221014-205835_gitlab_backup.tar 14/10/2022 20:59:30
gitlab-backups/20221014-205835_gitlab_config.tar 14/10/2022 20:59:34
gitlab-backups/gitlab-config.tar 14/10/2022 20:40:17
gitlab-backups/ionfs-example_gitlab_backup.tar 14/10/2022 20:40:12

Wrapping up

In this tutorial we've covered some background on the self-hosted GitLab backups, how to create them, and how to upload them to IBC S6 with the IonFS CLI. Finally we wrapped the steps in a simple bash script to allow the process to be automated.

All the steps covered in this tutorial, and the backup script, are currently used by Ionburst Cloud to backup and protect our internal GitLab instance. To keep up with the latest developments on the backup script, please checkout out our examples repository on GitHub.

· 8 min read
Josh Fraser

Overview

The IonFS CLI provides a set of tools to manage secrets stored in IBC NKV as if it were a remote filesystem. While the secrets are stored within IBC NKV, the metadata is stored in a customer-controlled metadata repository.

Anyone that has been granted access to this repository, and the appropriate Ionburst Cloud Platform credentials, can interact with the stored secrets.

To get up and running quickly, we will be using the newly released IonFS CLI local metadata repository functionality.

Shared Responsibility Model Breakdown

Customer Responsibility

  • You, the customer, are responsible for the secure management of the Ionburst Cloud credentials used by ionfs.
  • You, the customer, are responsible for the security of ionfs metadata repositories and the metadata stored in them.

Ionburst Cloud Responsibility

  • We are responsible for the security of all secrets stored in IBC NKV using ionfs.
  • We are responsible for the underlying security and availability of the Ionburst Cloud platform.

Getting Started

In this tutorial we will cover:

  1. Setting up ionfs.

  2. Working with ionfs metadata repositories.

  3. Listing IBC classifications with ionfs.

  4. Working with ionfs directories.

  5. Managing secrets with ionfs.

    Basic Usage

ionfs allows us to do the following:

  • List configured metadata repositories.
  • List available IBC classifications.
  • Create, list and delete ionfs directories.
  • Upload, download and delete secrets from IBC NKV.

1. Setting up

ionfs makes use of metadata repositories, or repos, to track the secrets that have been secured by IBC NKV. Metadata repos are specified in the configuration file stored under ~/.ionfs/appsettings.json.

For this tutorial, we are going to create a new local directory to use for ionfs metadata, along with the ~/.ionfs directory used to store our configuration file.

mkdir ~/local-ionfs
mkdir ~/.ionfs

We can now set up our ionfs configuration file. First, add a new file to our newly created .ionfs directory.

For MacOS and Linux users:

touch ~/.ionfs/appsettings.json

For Windows users:

New-Item ~/.ionfs/appsettings.json -type file

Open this file in your text editor of choice, and add the following:

{
"IonFS": {
"MaxSize": "50000000",
"Verbose": "false",
"DefaultClassification": "Restricted",
"Repositories": [
{
"Name": "local-ionfs",
"Usage": "Secrets",
"Class": "Ionburst.Apps.IonFS.Repo.LocalFS.MetadataLocalFS",
"Assembly": "Ionburst.Apps.IonFS.Repo.LocalFS",
"DataStore": "/Users/username/local-ionfs"
},
],
"DefaultRepository": "local-ionfs"
},
"Ionburst": {
"Profile": "example",
"TraceCredentialsFile": "OFF"
}
}

Key points to note:

  • setting the Usage entry to secrets is required to configure the repo for IBC NKV.
  • the DataStore entry references the local directory we've created for metadata (remember to change the username), but it cannot use relative paths, i.e:
    • for MacOS: /Users/username/local-ionfs
    • for Linux: /home/username/local-ionfs
    • for Windows: /
  • the Ionburst section relates to the Ionburst SDK credentials file. If you have an existing profile, you can add it here.

If you do not have an existing Ionburst credentials file, one can be created with the following:

For MacOS and Linux users:

mkdir ~/.ionburst
touch ~/.ionburst/credentials

For Windows users:

mkdir ~/.ionburst
New-Item ~/.ionburst/credentials -type file

Open this file in your text editor of choice, and add the following (remember to add your Ionburst Cloud API credentials here):

[example]
ionburst_id=your-ionburst-id
ionburst_key=your-ionburst-key

2. Metadata Repos

Now that we have ionfs setup, we can now start working with our metadata repo. To list the configured repos, the following ionfs command can be used:

ionfs repos

An example output would look like:

[hello@ionfs-example ~]$ ionfs repos
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Available Repositories (*default):
* [s] ion://local-ionfs/ (Ionburst.Apps.IonFS.Repo.LocalFS.MetadataLocalFS)

3. Classifications

Secrets can be secured by Ionburst Cloud according to available security policies. ionfs can be used to view the policies currently available to an Ionburst Cloud party.

To list available policies, the following can be used:

ionfs policy

An example output would look like:

[hello@ionfs-example ~]$ ionfs policy
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Available Classifications:
2:Restricted

4. Directories

Secrets secured by IBC S6 through ionfs can be organised within a repo using a typical directory structure.

List directories

To list available directories within a repo, the following can be used:

ionfs list ion://local-ionfs

As we marked the local-ionfs repo as the default, we can omit the name as it will be treated as the root.

ionfs list ion://

An example output would look like:

[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Directory of ion://local-ionfs/
d example/

By default, this will list the contents of the repo's root directory. To list a specific directory, the following can be used:

ionfs list ion://example

An example output would look like:

[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Directory of ion://local-ionfs/example/
Remote directory is empty

Create a directory

To create a new directory within a repo, the following can be used:

ionfs mkdir ion://new-directory

An example output would look like:

[hello@ionfs-example ~]$ ionfs mkdir ion://new-directory
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Directory of ion://local-ionfs/
d example/
d new-directory/

Delete a directory

To remove a directory within a repo, the following can be used:

ionfs rmdir ion://new-directory

An example output would look like:

[hello@ionfs-example ~]$ ionfs rmdir ion://new-directory
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Directory of ion://local-ionfs/
d example/

5. Secrets

Finally, and most importantly we can now look at uploading (Put), downloading (Get) and deleting secrets from IBC NKV using ionfs. In these examples, we'll use a secret called my-secret, with the value "We may guard your data, but we'll never take its freedom".

First, we need to create my-file.txt:

echo "We may guard your data, but we'll never take its freedom" > my-file.txt

Put

To upload a secret with ionfs, the following can be used:

ionfs secrets put "We may guard your data, but we'll never take its freedom" ion:// my-secret

An example output would look like:

[hello@ionfs-example ~]$ ionfs secrets put "We may guard your data, but we'll never take its freedom" ion:// my-secret
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Secrets Repo of ion://s3-example-ionfs-nkv/
d example/
my-secret 24/08/2022 13:10:14

To upload a secret to a specific directory within your repo, use the following:

ionfs secrets put "We may guard your data, but we'll never take its freedom" ion://example my-secret

An example output would look like:

[hello@ionfs-example ~]$ ionfs secrets put "We may guard your data, but we'll never take its freedom" ion:// my-secret
[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Secrets Repo of ion://s3-example-ionfs-nkv/example/
example/my-secret 24/08/2022 13:11:52

Get

To retrieve a secret with ionfs, use the following:

ionfs secrets get ion://example/my-secret

An example output would look like:

[hello@ionfs-example ~]$ ionfs secrets get ion://example/my-secret
We may guard your data, but we'll never take its freedom

Delete

To delete a secret from the ionfs repo and from Ionburst Cloud NKV, the following can be used:

ionfs secrets del ion://example/my-secret

An example output would look like:

[hello@ionfs-example ~]$ ionfs secrets del ion://example/my-secret
[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Secrets Repo of ion://s3-example-ionfs-nkv/example/
Remote Secrets repository is empty

Conclusion

You should now be able to perform basic secrets operations on IBC NKV with ionfs. If you're interested in learning more about the IonFS CLI, please see the Ionburst Cloud docs.

· 8 min read
Josh Fraser

Overview

The IonFS CLI provides a set of tools to manage objects and files stored in IBC S6 as if it were a remote filesystem. While the data is stored within IBC S6, the metadata is stored in a customer-controlled metadata repository.

Anyone that has been granted access to this repository, and the appropriate Ionburst Cloud Platform credentials, can interact with the stored files or objects.

To get up and running quickly, we will be using the newly released IonFS CLI local metadata repository functionality.

Shared Responsibility Model Breakdown

Customer Responsibility

  • You, the customer, are responsible for the secure management of the Ionburst Cloud credentials used by ionfs.
  • You, the customer, are responsible for the security of ionfs metadata repositories and the metadata stored in them.

Ionburst Cloud Responsibility

  • We are responsible for the security of all data stored in IBC S6 using ionfs.
  • We are responsible for the underlying security and availability of the Ionburst Cloud platform.

Getting Started

In this tutorial we will cover:

  1. Setting up ionfs.
  2. Working with ionfs metadata repositories.
  3. Listing IBC classifications with ionfs.
  4. Working with ionfs directories.
  5. Managing files stored on IBC S6 with ionfs.

Basic Usage

ionfs allows us to do the following:

  • List configured metadata repositories.
  • List available IBC classifications.
  • Create, list and delete ionfs directories.
  • Upload, download and delete data from IBC S6.

1. Setting up

ionfs makes use of metadata repositories, or repos, to track the objects and files that have been secured by IBC S6. Metadata repos are specified in the configuration file stored under ~/.ionfs/appsettings.json.

For this tutorial, we are going to create a new local directory to use for ionfs metadata, along with the ~/.ionfs directory used to store our configuration file.

mkdir ~/local-ionfs
mkdir ~/.ionfs

We can now set up our ionfs configuration file. First, add a new file to our newly created .ionfs directory.

For MacOS and Linux users:

touch ~/.ionfs/appsettings.json

For Windows users:

New-Item ~/.ionfs/appsettings.json -type file

Open this file in your text editor of choice, and add the following:

{
"IonFS": {
"MaxSize": "50000000",
"Verbose": "false",
"DefaultClassification": "Restricted",
"Repositories": [
{
"Name": "local-ionfs",
"Usage": "Data",
"Class": "Ionburst.Apps.IonFS.Repo.LocalFS.MetadataLocalFS",
"Assembly": "Ionburst.Apps.IonFS.Repo.LocalFS",
"DataStore": "/Users/username/local-ionfs"
},
],
"DefaultRepository": "local-ionfs"
},
"Ionburst": {
"Profile": "example",
"TraceCredentialsFile": "OFF"
}
}

Key points to note:

  • the DataStore entry references the local directory we've created for metadata (remember to change the username), but it cannot use relative paths, i.e:
    • for MacOS: /Users/username/local-ionfs
    • for Linux: /home/username/local-ionfs
    • for Windows: /
  • the Ionburst section relates to the Ionburst SDK credentials file. If you have an existing profile, you can add it here.

If you do not have an existing Ionburst credentials file, one can be created with the following:

For MacOS and Linux users:

mkdir ~/.ionburst
touch ~/.ionburst/credentials

For Windows users:

mkdir ~/.ionburst
New-Item ~/.ionburst/credentials -type file

Open this file in your text editor of choice, and add the following (remember to add your Ionburst Cloud API credentials here):

[example]
ionburst_id=your-ionburst-id
ionburst_key=your-ionburst-key

2. Metadata Repos

Now that we have ionfs setup, we can now start working with our metadata repo. To list the configured repos, the following ionfs command can be used:

ionfs repos

An example output would look like:

[hello@ionfs-example ~]$ ionfs repos
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Available Repositories (*default):
* [d] ion://local-ionfs/ (Ionburst.Apps.IonFS.Repo.LocalFS.MetadataLocalFS)

3. Classifications

Data can be secured by Ionburst Cloud according to available security policies. ionfs can be used to view the policies currently available to an Ionburst Cloud party.

To list available policies, the following can be used:

ionfs policy

An example output would look like:

[hello@ionfs-example ~]$ ionfs policy
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Available Classifications:
2:Restricted

4. Directories

Files and objects secured by IBC S6 through ionfs can be organised within its repo using a typical directory structure.

List directories

To list available directories within a repo, the following can be used:

ionfs list ion://local-ionfs

As we marked the local-ionfs repo as the default, we can omit the name as it will be treated as the root.

ionfs list ion://

An example output would look like:

[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://local-ionfs/
d example/

By default, this will list the contents of the repo's root directory. To list a specific directory, the following can be used:

ionfs list ion://example

An example output would look like:

[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://local-ionfs/example/
Remote directory is empty

Create a directory

To create a new directory within a repo, the following can be used:

ionfs mkdir ion://new-directory

An example output would look like:

[hello@ionfs-example ~]$ ionfs mkdir ion://new-directory
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://local-ionfs/
d example/
d new-directory/

Delete a directory

To remove a directory within a repo, the following can be used:

ionfs rmdir ion://new-directory

An example output would look like:

[hello@ionfs-example ~]$ ionfs rmdir ion://new-directory
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://local-ionfs/
d example/

5. Files

Finally, and most importantly we can now look at uploading (Put), downloading (Get) and deleting data from IBC S6 using ionfs. In these examples, we'll use a file called my-file.txt.

First, we need to create my-file.txt:

echo "We may guard your data, but we'll never take its freedom" > my-file.txt

Put

To upload a file to Ionburst Cloud with ionfs, the following can be used:

ionfs put my-file.txt ion://

An example output would look like:

[hello@ionfs-example ~]$ ionfs put my-file.txt ion://
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://local-ionfs/
d example/
my-file.txt 23/08/2022 13:49:51

To upload data to a specific directory within your repo, use the following:

ionfs put my-file.txt ion://example

An example output would look like:

[hello@ionfs-example ~]$ ionfs put my-file.txt ion://example
[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://local-ionfs/example/
example/my-file.txt 23/08/2022 13:50:23

Get

To retrieve a file with ionfs, use the following:

ionfs get ion://example/my-file.txt

An example output would look like:

[hello@ionfs-example ~]$ rm my-file.txt
[hello@ionfs-example ~]$ ionfs get ion://example/my-file.txt
[hello@ionfs-example ~]$ ls
my-file.txt
[hello@ionfs-example ~]$ cat my-file.txt
We may guard your data, but we'll never take its freedom

By default, this will download the file from IBC S6 to the current directory, with the name used in ionfs. To download to a specific local directory, or to download to a different filename, use the following:

ionfs get ion://example/my-file.txt my-file-2.txt

An example output would look like:

[hello@ionfs-example ~]$ ionfs get ion://example/my-file.txt my-file-2.txt
[hello@ionfs-example ~]$ ls
my-file.txt my-file-2.txt
[hello@ionfs-example ~]$ cat my-file-2.txt
We may guard your data, but we'll never take its freedom

Delete

To delete a file from the ionfs repo and from IBC S6, the following can be used:

ionfs del ion://example/my-file.txt

An example output would look like:

[hello@ionfs-example ~]$ ionfs del ion://example/my-file.txt
[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://local-ionfs/example/
Remote directory is empty

Conclusion

You should now be able to perform basic file operations on IBC S6 with ionfs. If you're interested in learning more about the IonFS CLI, please see the Ionburst Cloud docs.

· 7 min read
Josh Fraser

Overview

The IonFS Command Line Interface provides a set of tools to manage data stored by Ionburst Cloud as if it were a remote filesystem. With the release of NKV, IonFS can now store secrets, while storing the metadata in a customer-owned metadata repository.

Anyone that has been granted access to this repository, and the appropriate Ionburst Cloud credentials, can interact with the stored secrets.

For this tutorial, we will be using Amazon S3 as the ionfs metadata repository.

Shared Responsibility Model Breakdown

Customer Responsibility

  • You, the customer, are responsible for the secure management of the Ionburst Cloud credentials used by ionfs.
  • You, the customer, are responsible for the security of ionfs metadata repositories and the metadata stored in them.

Ionburst Cloud Responsibility

  • We are responsible for the security of all secrets stored in Ionburst Cloud NKV using ionfs.
  • We are responsible for the underlying security and availability of the Ionburst Cloud platform.

Getting Started

In this tutorial we will cover:

  1. Working with ionfs metadata repositories.
  2. Listing IBC classifications with ionfs.
  3. Working with ionfs directories.
  4. Managing secrets with ionfs.

Basic Usage

ionfs allows us to do the following:

  • List configured metadata repositories.
  • List available IBC classifications.
  • Create, list and delete ionfs directories.
  • Upload, download and delete secrets from IBC.

1. Metadata Repositories

ionfs makes use of metadata repositories, or repos, to track secrets that have been secured by Ionburst Cloud NKV. Metadata repos are specified in the configuration file stored under ~/.ionfs/appsettings.json.

To list the configured repos, the following ionfs command can be used:

ionfs repos

An example output would look like:

[hello@ionfs-example ~]$ ionfs repos
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Available Repositories (*default):
* [s] ion://s3-example-ionfs-nkv/ (Ionburst.Apps.IonFS.Repo.S3.MetadataS3)

2. Classifications

Secrets can be secured by Ionburst Cloud according to available security policies. ionfs can be used to view the policies currently available to an Ionburst Cloud party.

To list available policies, the following can be used:

ionfs policy

An example output would look like:

[hello@ionfs-example ~]$ ionfs policy
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Available Classifications:
2:Restricted

3. Directories

Secrets secured by Ionburst Cloud NKV through ionfs can partition its repo using a typical directory structure.

List directories

To list available directories within a repo, the following can be used:

ionfs list

An example output would look like:

[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Secrets Repo of ion://s3-example-ionfs-nkv/
d example/

By default, this will list the contents of the repo's root directory. To list a specific directory, the following can be used:

ionfs list ion://example

An example output would look like:

[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Secrets Repo of ion://s3-example-ionfs-nkv/example/
Remote Secrets repository is empty is empty

Create a directory

To create a new directory within a repo, the following can be used:

ionfs mkdir ion://new-directory

An example output would look like:

[hello@ionfs-example ~]$ ionfs mkdir ion://new-directory
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Secrets Repo of ion://s3-example-ionfs-nkv/
d example/
d new-directory/

Delete a directory

To remove a directory within a repo, the following can be used:

ionfs rmdir ion://new-directory

An example output would look like:

[hello@ionfs-example ~]$ ionfs rmdir ion://new-directory
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Secrets Repo of ion://s3-example-ionfs-nkv/
d example/

4. Secrets

Finally, and most importantly we can now look at uploading (Put), downloading (Get) and deleting secrets from IBC NKV using ionfs. In these examples, we'll use a secret called my-secret, with the value "We may guard your data, but we'll never take its freedom".

Put

To upload a secret with ionfs, the following can be used:

ionfs secrets put "We may guard your data, but we'll never take its freedom" ion:// my-secret

An example output would look like:

[hello@ionfs-example ~]$ ionfs secrets put "We may guard your data, but we'll never take its freedom" ion:// my-secret
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Secrets Repo of ion://s3-example-ionfs-nkv/
d example/
my-secret 22/07/2021 13:10:14

To upload a secret to a specific directory within your repo, use the following:

ionfs secrets put "We may guard your data, but we'll never take its freedom" ion://example my-secret

An example output would look like:

[hello@ionfs-example ~]$ ionfs secrets put "We may guard your data, but we'll never take its freedom" ion:// my-secret
[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Secrets Repo of ion://s3-example-ionfs-nkv/example/
example/my-secret 22/07/2021 13:11:52

Get

To retrieve a secret with ionfs, use the following:

ionfs secrets get ion://example/my-secret

An example output would look like:

[hello@ionfs-example ~]$ ionfs secrets get ion://example/my-secret
We may guard your data, but we'll never take its freedom

Delete

To delete a secret from the ionfs repo and from Ionburst Cloud NKV, the following can be used:

ionfs secrets del ion://example/my-secret

An example output would look like:

[hello@ionfs-example ~]$ ionfs secrets del ion://example/my-secret
[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Secrets Repo of ion://s3-example-ionfs-nkv/example/
Remote Secrets repository is empty

Conclusion

You should now be able to perform basic secrets operations on Ionburst Cloud NKV using the ionfs tool. If you're interested in learning more about the IonFS CLI, please see the Ionburst Cloud docs.

· 7 min read
Josh Fraser

Overview

The IonFS Command Line Interface provides a set of tools to manage data stored by Ionburst Cloud S6 as if it were a remote filesystem. While the IonFS CLI stores files within Ionburst Cloud S6, the metadata is stored in a customer-owned metadata repository.

Anyone that has been granted access to this repository, and the appropriate Ionburst Cloud credentials, can interact with the stored data.

For this tutorial, we will be using Amazon S3 as the ionfs metadata repository.

Shared Responsibility Model Breakdown

Customer Responsibility

  • You, the customer, are responsible for the secure management of the Ionburst Cloud credentials used by ionfs.
  • You, the customer, are responsible for the security of ionfs metadata repositories and the metadata stored in them.

Ionburst Cloud Responsibility

  • We are responsible for the security of all data stored in Ionburst Cloud S6 using ionfs.
  • We are responsible for the underlying security and availability of the Ionburst Cloud platform.

Getting Started

In this tutorial we will cover:

  1. Working with ionfs metadata repositories.
  2. Listing IBC classifications with ionfs.
  3. Working with ionfs directories.
  4. Managing files with ionfs.

Basic Usage

ionfs allows us to do the following:

  • List configured metadata repositories.
  • List available IBC classifications.
  • Create, list and delete ionfs directories.
  • Upload, download and delete data from IBC.

1. Metadata Repositories

ionfs makes use of metadata repositories, or repos, to track data that has been secured by Ionburst Cloud S6. Metadata repos are specified in the configuration file stored under ~/.ionfs/appsettings.json.

To list the configured repos, the following ionfs command can be used:

ionfs repos

An example output would look like:

[hello@ionfs-example ~]$ ionfs repos
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Available Repositories (*default):
* [d] ion://s3-example-ionfs/ (Ionburst.Apps.IonFS.Repo.S3.MetadataS3)

2. Classifications

Data can be secured by Ionburst Cloud according to available security policies. ionfs can be used to view the policies currently available to an Ionburst Cloud party.

To list available policies, the following can be used:

ionfs policy

An example output would look like:

[hello@ionfs-example ~]$ ionfs policy
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Available Classifications:
2:Restricted

3. Directories

Data secured by Ionburst Cloud S6 through ionfs can partition its repo using a typical directory structure.

List directories

To list available directories within a repo, the following can be used:

ionfs list

An example output would look like:

[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/
d example/

By default, this will list the contents of the repo's root directory. To list a specific directory, the following can be used:

ionfs list ion://example

An example output would look like:

[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/example/
Remote directory is empty

Create a directory

To create a new directory within a repo, the following can be used:

ionfs mkdir ion://new-directory

An example output would look like:

[hello@ionfs-example ~]$ ionfs mkdir ion://new-directory
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/
d example/
d new-directory/

Delete a directory

To remove a directory within a repo, the following can be used:

ionfs rmdir ion://new-directory

An example output would look like:

[hello@ionfs-example ~]$ ionfs rmdir ion://new-directory
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/
d example/

4. Files

Finally, and most importantly we can now look at uploading (Put), downloading (Get) and deleting data from IBC S6 using ionfs. In these examples, we'll use a file called my-file.txt.

First, we need to create my-file.txt:

echo "We may guard your data, but we'll never take its freedom" > my-file.txt

Put

To upload a file to Ionburst Cloud with ionfs, the following can be used:

ionfs put my-file.txt ion://

An example output would look like:

[hello@ionfs-example ~]$ ionfs put my-file.txt ion://
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/
d example/
my-file.txt 23/4/2021 13:49:51

To upload data to a specific directory within your repo, use the following:

ionfs put my-file.txt ion://example

An example output would look like:

[hello@ionfs-example ~]$ ionfs put my-file.txt ion://example
[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/example/
example/my-file.txt 23/4/2021 13:50:23

Get

To retrieve a file with ionfs, use the following:

ionfs get ion://example/my-file.txt

An example output would look like:

[hello@ionfs-example ~]$ rm my-file.txt
[hello@ionfs-example ~]$ ionfs get ion://example/my-file.txt
[hello@ionfs-example ~]$ ls
my-file.txt
[hello@ionfs-example ~]$ cat my-file.txt
We may guard your data, but we'll never take its freedom

By default, this will download the file from Ionburst Cloud S6 to the current directory, with the name used in ionfs. To download to a specific local directory, or to download to a different name, use the following:

ionfs get -n my-file-2.txt ion://example/my-file.txt

An example output would look like:

[hello@ionfs-example ~]$ ionfs get -n my-file-2.txt ion://example/my-file.txt
[hello@ionfs-example ~]$ ls
my-file.txt my-file-2.txt
[hello@ionfs-example ~]$ cat my-file-2.txt
We may guard your data, but we'll never take its freedom

Delete

To delete a file from the ionfs repo and from Ionburst Cloud S6, the following can be used:

ionfs del ion://example/my-file.txt

An example output would look like:

[hello@ionfs-example ~]$ ionfs del ion://example/my-file.txt
[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/example/
Remote directory is empty

Conclusion

You should now be able to perform basic file operations on Ionburst Cloud S6 using the ionfs tool. If you're interested in learning more about the IonFS CLI, please see the Ionburst Cloud docs.