Skip to main content

8 posts tagged with "Ionburst Cloud"

View All Tags

· 3 min read
Josh Fraser

Overview

An IBC S6 API transaction currently supports a maximum upload size of 50MB. To upload objects larger than this limit, objects have to be split into 50MB chunks and uploaded individually, placing an implementation and management overhead on developers and client tools to track and manage these chunks.

To make it easier for developers and client tools to upload large objects to IBC S6, we've developed what we call IBC S6 SDK manifests. The SDK manifest is comprised of two main parts:

  • An implementation within each Ionburst Cloud SDK to chunk large objects and manage the upload of each chunk to IBC S6
  • A metadata object that tracks important information about each chunk; their IDs, checksum, ordinality etc.

To keep the manifest implementation simple, and to ensure the amount of information held by IBC S6 about a given object is minimised, this manifest metadata object is also stored within IBC S6 using the external reference (object ID) of the request.

This allows the chunks comprising a large object to be tracked, while avoiding any overhead on the client to track and manage them. It also means that to retrieve a large object, the external reference can be passed to the SDK manifest function, which will in turn retrieve and reconstruct the chunks.

The SDK manifests feature is similar to the multipart upload concept used by object stores like Amazon S3.

Getting Started

In this tutorial we will provide examples and code snippets of how to use the new manifest feature to upload large objects to IBC S6:

  1. Uploading large objects with ioncli
  2. Uploading large objects with the Ionburst Cloud Go SDK

ioncli

In this example, we will upload our large object, my-large-file.png to IBC S6 using the ioncli mput command.

Uploading my-large-file.png with ioncli:

ioncli --profile ioncli-example put manifest-example my-large-file.png

Example output:

[hello@ioncli-example ~]$ ls -lah my-large-file.png
-rw-rw-r--. 1 hello hello 125M 13 Sep 09:39 my-large-file.png
[hello@ioncli-example ~]$ ioncli --profile ioncli-example mput manifest-example my-large-file.png
Split to: fed44c96-f457-4e98-829c-b6809ec26e42
Split to: ac45ec73-ac16-4eaf-8251-79b2e1561a56
Split to: a41336ee-de91-474f-ad18-f5c36fe28060

Go SDK

The following example program shows how the Ionburst Cloud Go SDK PutManifest methods can be used:

package main
import (
"fmt"
"gitlab.com/ionburst/ionburst-sdk-go"
"os"
)
func main() {
client, err := ionburst.NewClient()
if err != nil {
fmt.Println(err)
}
ioReader, _ := os.Open("my-large-file.png")
err = client.PutManifest("manifest-example", ioReader, "")
if err != nil {
fmt.Println(err)
}
err = client.Head("manifest-example")
if err != nil {
fmt.Println(err)
} else {
fmt.Printf("Checked: %s\n", "manifest-example")
}
size, err := client.HeadWithLen("manifest-example")
if err != nil {
fmt.Println(err)
} else {
fmt.Printf("Size: %d\n", size)
}
}

Example output:

[hello@example head]$ go run main.go
Split to: 45202923-bb8f-4775-9d06-f7d69c50e883
Split to: de109e03-de07-46d5-a900-76bcaf612d81
Split to: 97301ce9-8645-431b-b9a9-d2f6bab7c2a9
Checked: manifest-example
Size: 498

· 8 min read
Josh Fraser

Overview

The IonFS CLI provides a set of tools to manage secrets stored in IBC NKV as if it were a remote filesystem. While the secrets are stored within IBC NKV, the metadata is stored in a customer-controlled metadata repository.

Anyone that has been granted access to this repository, and the appropriate Ionburst Cloud Platform credentials, can interact with the stored secrets.

To get up and running quickly, we will be using the newly released IonFS CLI local metadata repository functionality.

Shared Responsibility Model Breakdown

Customer Responsibility

  • You, the customer, are responsible for the secure management of the Ionburst Cloud credentials used by ionfs.
  • You, the customer, are responsible for the security of ionfs metadata repositories and the metadata stored in them.

Ionburst Cloud Responsibility

  • We are responsible for the security of all secrets stored in IBC NKV using ionfs.
  • We are responsible for the underlying security and availability of the Ionburst Cloud platform.

Getting Started

In this tutorial we will cover:

  1. Setting up ionfs.

  2. Working with ionfs metadata repositories.

  3. Listing IBC classifications with ionfs.

  4. Working with ionfs directories.

  5. Managing secrets with ionfs.

    Basic Usage

ionfs allows us to do the following:

  • List configured metadata repositories.
  • List available IBC classifications.
  • Create, list and delete ionfs directories.
  • Upload, download and delete secrets from IBC NKV.

1. Setting up

ionfs makes use of metadata repositories, or repos, to track the secrets that have been secured by IBC NKV. Metadata repos are specified in the configuration file stored under ~/.ionfs/appsettings.json.

For this tutorial, we are going to create a new local directory to use for ionfs metadata, along with the ~/.ionfs directory used to store our configuration file.

mkdir ~/local-ionfs
mkdir ~/.ionfs

We can now set up our ionfs configuration file. First, add a new file to our newly created .ionfs directory.

For MacOS and Linux users:

touch ~/.ionfs/appsettings.json

For Windows users:

New-Item ~/.ionfs/appsettings.json -type file

Open this file in your text editor of choice, and add the following:

{
"IonFS": {
"MaxSize": "50000000",
"Verbose": "false",
"DefaultClassification": "Restricted",
"Repositories": [
{
"Name": "local-ionfs",
"Usage": "Secrets",
"Class": "Ionburst.Apps.IonFS.Repo.LocalFS.MetadataLocalFS",
"Assembly": "Ionburst.Apps.IonFS.Repo.LocalFS",
"DataStore": "/Users/username/local-ionfs"
},
],
"DefaultRepository": "local-ionfs"
},
"Ionburst": {
"Profile": "example",
"TraceCredentialsFile": "OFF"
}
}

Key points to note:

  • setting the Usage entry to secrets is required to configure the repo for IBC NKV.
  • the DataStore entry references the local directory we've created for metadata (remember to change the username), but it cannot use relative paths, i.e:
    • for MacOS: /Users/username/local-ionfs
    • for Linux: /home/username/local-ionfs
    • for Windows: /
  • the Ionburst section relates to the Ionburst SDK credentials file. If you have an existing profile, you can add it here.

If you do not have an existing Ionburst credentials file, one can be created with the following:

For MacOS and Linux users:

mkdir ~/.ionburst
touch ~/.ionburst/credentials

For Windows users:

mkdir ~/.ionburst
New-Item ~/.ionburst/credentials -type file

Open this file in your text editor of choice, and add the following (remember to add your Ionburst Cloud API credentials here):

[example]
ionburst_id=your-ionburst-id
ionburst_key=your-ionburst-key

2. Metadata Repos

Now that we have ionfs setup, we can now start working with our metadata repo. To list the configured repos, the following ionfs command can be used:

ionfs repos

An example output would look like:

[hello@ionfs-example ~]$ ionfs repos
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Available Repositories (*default):
* [s] ion://local-ionfs/ (Ionburst.Apps.IonFS.Repo.LocalFS.MetadataLocalFS)

3. Classifications

Secrets can be secured by Ionburst Cloud according to available security policies. ionfs can be used to view the policies currently available to an Ionburst Cloud party.

To list available policies, the following can be used:

ionfs policy

An example output would look like:

[hello@ionfs-example ~]$ ionfs policy
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Available Classifications:
2:Restricted

4. Directories

Secrets secured by IBC S6 through ionfs can be organised within a repo using a typical directory structure.

List directories

To list available directories within a repo, the following can be used:

ionfs list ion://local-ionfs

As we marked the local-ionfs repo as the default, we can omit the name as it will be treated as the root.

ionfs list ion://

An example output would look like:

[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Directory of ion://local-ionfs/
d example/

By default, this will list the contents of the repo's root directory. To list a specific directory, the following can be used:

ionfs list ion://example

An example output would look like:

[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Directory of ion://local-ionfs/example/
Remote directory is empty

Create a directory

To create a new directory within a repo, the following can be used:

ionfs mkdir ion://new-directory

An example output would look like:

[hello@ionfs-example ~]$ ionfs mkdir ion://new-directory
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Directory of ion://local-ionfs/
d example/
d new-directory/

Delete a directory

To remove a directory within a repo, the following can be used:

ionfs rmdir ion://new-directory

An example output would look like:

[hello@ionfs-example ~]$ ionfs rmdir ion://new-directory
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Directory of ion://local-ionfs/
d example/

5. Secrets

Finally, and most importantly we can now look at uploading (Put), downloading (Get) and deleting secrets from IBC NKV using ionfs. In these examples, we'll use a secret called my-secret, with the value "We may guard your data, but we'll never take its freedom".

First, we need to create my-file.txt:

echo "We may guard your data, but we'll never take its freedom" > my-file.txt

Put

To upload a secret with ionfs, the following can be used:

ionfs secrets put "We may guard your data, but we'll never take its freedom" ion:// my-secret

An example output would look like:

[hello@ionfs-example ~]$ ionfs secrets put "We may guard your data, but we'll never take its freedom" ion:// my-secret
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Secrets Repo of ion://s3-example-ionfs-nkv/
d example/
my-secret 24/08/2022 13:10:14

To upload a secret to a specific directory within your repo, use the following:

ionfs secrets put "We may guard your data, but we'll never take its freedom" ion://example my-secret

An example output would look like:

[hello@ionfs-example ~]$ ionfs secrets put "We may guard your data, but we'll never take its freedom" ion:// my-secret
[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Secrets Repo of ion://s3-example-ionfs-nkv/example/
example/my-secret 24/08/2022 13:11:52

Get

To retrieve a secret with ionfs, use the following:

ionfs secrets get ion://example/my-secret

An example output would look like:

[hello@ionfs-example ~]$ ionfs secrets get ion://example/my-secret
We may guard your data, but we'll never take its freedom

Delete

To delete a secret from the ionfs repo and from Ionburst Cloud NKV, the following can be used:

ionfs secrets del ion://example/my-secret

An example output would look like:

[hello@ionfs-example ~]$ ionfs secrets del ion://example/my-secret
[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/ v0.3.0
Secrets Repo of ion://s3-example-ionfs-nkv/example/
Remote Secrets repository is empty

Conclusion

You should now be able to perform basic secrets operations on IBC NKV with ionfs. If you're interested in learning more about the IonFS CLI, please see the Ionburst Cloud docs.

· 4 min read
Josh Fraser

Overview

As a secure object storage service, IBC S6 was designed to be integrated into both new and existing applications. Firefox Send is an open-source, secure file-sharing service that integrates cloud object stores like Amazon S3 and Google Cloud Storage for its backend storage.

We've integrated IBC S6 with Send to provide Ionburst Cloud customers the ability to easily and safely share files, while also providing an integration example for our Node.js SDK.

Shared Responsibility Model Breakdown

Customer Responsibility

  • You, the customer, are responsible for the secure management of the Ionburst Cloud credentials used by Send to connect with IBC S6.
  • You, the customer, are responsible for the security and administration of the infrastructure running the Send service.

Ionburst Cloud Responsibility

  • We are responsible for the security of all files stored in IBC S6 through the Send integration.
  • We are responsible for the underlying security and availability of the Ionburst Cloud platform.

Getting Started

In this tutorial we will cover:

  1. Setting up Send with IBC S6.
  2. Working Send and IBC S6 in development mode.
  3. Preparing Send for production.

1. Setting up Send with IBC S6

First, we need to clone the Ionburst Cloud Send project, available here.

We then need to install the Node.js dependencies:

git clone https://github.com/ionburstcloud/send.git
cd send
npm install

Once the dependencies are installed, open the Send project in your preferred IDE or editor.

2. Working Send and IBC S6 in development mode

To try out Send with IBC S6 in the Node.js development mode, we need to complete two configuration steps:

  • Edit the Ionburst SDK config.json file, adding our credentials profile and Ionburst Cloud API endpoint
  • Configure the Send backend to use IBC S6 as its backend storage
  • Start a Redis container to handle Send metadata (optional)

The config.json file can be found at the root of the Send project:

{
"Ionburst": {
"Profile": "send",
"IonburstUri": "https://api.eu-west-1.ionburst.cloud/",
"TraceCredentialsFile": "OFF"
}
}

If you have an existing Ionburst credentials profile, it can be added to the Profile key. Otherwise, add a new profile to the Ionburst credentials file with the name send.

IBC S6 can now be enabled for the Send project by editing the config.js file, found in the server directory. Update the ionburst property's default value to true to enable IBC S6 as the backend storage:

ionburst: {
format: String,
default: 'true'
},

If looking to use Redis for metadata while in development mode, you can also update the redis_host property found on line 91 from localhost to 127.0.0.1 (or the address of your Redis host). This step is optional, as Send will use its own memory store for metadata if the property is left as localhost.

To start a Redis container, the following can be used:

docker run --name send-redis -p 127.0.0.1:6379:6379/tcp -tid redis:latest

Once configured, the Send project can be launched and will be available on http://localhost:8080

npm start

3. Preparing Send for production

Preparing Send for production is a matter of reviewing and updating the config.js file, found in the server directory. Configuration items to consider are:

  • redis-host on line 91 - it's recommended to use Redis when running Send in production, rather than the built-in memory store. Update this to the address of your Redis node.
  • listen-port on line 136 - Send will run on port 1443 by default in production. It's recommended to run Send behind a load balancer or reverse proxy.
  • base-url on line 167 - change this to the url your Send instance will be available on.

Once happy with the config, a production build of Send can be created:

npm run build

If running Redis on the same host, you can start the container with:

docker run --name send-redis -p 127.0.0.1:6379:6379/tcp -tid redis:latest

You can now start the production build of Send with:

npm run prod

The Send project production build will now be available on http://localhost:1443

Conclusion

We've covered how to get the Send project up and running with IBC S6, both in development mode, and to prepare it for production deployment. If you have further questions on deploying Send with IBC S6, or integrating Ionburst SDKs with new or existing applications, drop us a line in our community Slack.

· 8 min read
Josh Fraser

Overview

The IonFS CLI provides a set of tools to manage objects and files stored in IBC S6 as if it were a remote filesystem. While the data is stored within IBC S6, the metadata is stored in a customer-controlled metadata repository.

Anyone that has been granted access to this repository, and the appropriate Ionburst Cloud Platform credentials, can interact with the stored files or objects.

To get up and running quickly, we will be using the newly released IonFS CLI local metadata repository functionality.

Shared Responsibility Model Breakdown

Customer Responsibility

  • You, the customer, are responsible for the secure management of the Ionburst Cloud credentials used by ionfs.
  • You, the customer, are responsible for the security of ionfs metadata repositories and the metadata stored in them.

Ionburst Cloud Responsibility

  • We are responsible for the security of all data stored in IBC S6 using ionfs.
  • We are responsible for the underlying security and availability of the Ionburst Cloud platform.

Getting Started

In this tutorial we will cover:

  1. Setting up ionfs.
  2. Working with ionfs metadata repositories.
  3. Listing IBC classifications with ionfs.
  4. Working with ionfs directories.
  5. Managing files stored on IBC S6 with ionfs.

Basic Usage

ionfs allows us to do the following:

  • List configured metadata repositories.
  • List available IBC classifications.
  • Create, list and delete ionfs directories.
  • Upload, download and delete data from IBC S6.

1. Setting up

ionfs makes use of metadata repositories, or repos, to track the objects and files that have been secured by IBC S6. Metadata repos are specified in the configuration file stored under ~/.ionfs/appsettings.json.

For this tutorial, we are going to create a new local directory to use for ionfs metadata, along with the ~/.ionfs directory used to store our configuration file.

mkdir ~/local-ionfs
mkdir ~/.ionfs

We can now set up our ionfs configuration file. First, add a new file to our newly created .ionfs directory.

For MacOS and Linux users:

touch ~/.ionfs/appsettings.json

For Windows users:

New-Item ~/.ionfs/appsettings.json -type file

Open this file in your text editor of choice, and add the following:

{
"IonFS": {
"MaxSize": "50000000",
"Verbose": "false",
"DefaultClassification": "Restricted",
"Repositories": [
{
"Name": "local-ionfs",
"Usage": "Data",
"Class": "Ionburst.Apps.IonFS.Repo.LocalFS.MetadataLocalFS",
"Assembly": "Ionburst.Apps.IonFS.Repo.LocalFS",
"DataStore": "/Users/username/local-ionfs"
},
],
"DefaultRepository": "local-ionfs"
},
"Ionburst": {
"Profile": "example",
"TraceCredentialsFile": "OFF"
}
}

Key points to note:

  • the DataStore entry references the local directory we've created for metadata (remember to change the username), but it cannot use relative paths, i.e:
    • for MacOS: /Users/username/local-ionfs
    • for Linux: /home/username/local-ionfs
    • for Windows: /
  • the Ionburst section relates to the Ionburst SDK credentials file. If you have an existing profile, you can add it here.

If you do not have an existing Ionburst credentials file, one can be created with the following:

For MacOS and Linux users:

mkdir ~/.ionburst
touch ~/.ionburst/credentials

For Windows users:

mkdir ~/.ionburst
New-Item ~/.ionburst/credentials -type file

Open this file in your text editor of choice, and add the following (remember to add your Ionburst Cloud API credentials here):

[example]
ionburst_id=your-ionburst-id
ionburst_key=your-ionburst-key

2. Metadata Repos

Now that we have ionfs setup, we can now start working with our metadata repo. To list the configured repos, the following ionfs command can be used:

ionfs repos

An example output would look like:

[hello@ionfs-example ~]$ ionfs repos
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Available Repositories (*default):
* [d] ion://local-ionfs/ (Ionburst.Apps.IonFS.Repo.LocalFS.MetadataLocalFS)

3. Classifications

Data can be secured by Ionburst Cloud according to available security policies. ionfs can be used to view the policies currently available to an Ionburst Cloud party.

To list available policies, the following can be used:

ionfs policy

An example output would look like:

[hello@ionfs-example ~]$ ionfs policy
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Available Classifications:
2:Restricted

4. Directories

Files and objects secured by IBC S6 through ionfs can be organised within its repo using a typical directory structure.

List directories

To list available directories within a repo, the following can be used:

ionfs list ion://local-ionfs

As we marked the local-ionfs repo as the default, we can omit the name as it will be treated as the root.

ionfs list ion://

An example output would look like:

[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://local-ionfs/
d example/

By default, this will list the contents of the repo's root directory. To list a specific directory, the following can be used:

ionfs list ion://example

An example output would look like:

[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://local-ionfs/example/
Remote directory is empty

Create a directory

To create a new directory within a repo, the following can be used:

ionfs mkdir ion://new-directory

An example output would look like:

[hello@ionfs-example ~]$ ionfs mkdir ion://new-directory
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://local-ionfs/
d example/
d new-directory/

Delete a directory

To remove a directory within a repo, the following can be used:

ionfs rmdir ion://new-directory

An example output would look like:

[hello@ionfs-example ~]$ ionfs rmdir ion://new-directory
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://local-ionfs/
d example/

5. Files

Finally, and most importantly we can now look at uploading (Put), downloading (Get) and deleting data from IBC S6 using ionfs. In these examples, we'll use a file called my-file.txt.

First, we need to create my-file.txt:

echo "We may guard your data, but we'll never take its freedom" > my-file.txt

Put

To upload a file to Ionburst Cloud with ionfs, the following can be used:

ionfs put my-file.txt ion://

An example output would look like:

[hello@ionfs-example ~]$ ionfs put my-file.txt ion://
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://local-ionfs/
d example/
my-file.txt 23/08/2022 13:49:51

To upload data to a specific directory within your repo, use the following:

ionfs put my-file.txt ion://example

An example output would look like:

[hello@ionfs-example ~]$ ionfs put my-file.txt ion://example
[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://local-ionfs/example/
example/my-file.txt 23/08/2022 13:50:23

Get

To retrieve a file with ionfs, use the following:

ionfs get ion://example/my-file.txt

An example output would look like:

[hello@ionfs-example ~]$ rm my-file.txt
[hello@ionfs-example ~]$ ionfs get ion://example/my-file.txt
[hello@ionfs-example ~]$ ls
my-file.txt
[hello@ionfs-example ~]$ cat my-file.txt
We may guard your data, but we'll never take its freedom

By default, this will download the file from IBC S6 to the current directory, with the name used in ionfs. To download to a specific local directory, or to download to a different filename, use the following:

ionfs get ion://example/my-file.txt my-file-2.txt

An example output would look like:

[hello@ionfs-example ~]$ ionfs get ion://example/my-file.txt my-file-2.txt
[hello@ionfs-example ~]$ ls
my-file.txt my-file-2.txt
[hello@ionfs-example ~]$ cat my-file-2.txt
We may guard your data, but we'll never take its freedom

Delete

To delete a file from the ionfs repo and from IBC S6, the following can be used:

ionfs del ion://example/my-file.txt

An example output would look like:

[hello@ionfs-example ~]$ ionfs del ion://example/my-file.txt
[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://local-ionfs/example/
Remote directory is empty

Conclusion

You should now be able to perform basic file operations on IBC S6 with ionfs. If you're interested in learning more about the IonFS CLI, please see the Ionburst Cloud docs.

· 3 min read
Josh Fraser

Overview

The Ionburst Cloud API HEAD method has been added to allow IBC S6 objects and IBC NKV secrets to be verified after upload, or queried to return information.

A HEAD request is functionally similar to a GET request; it is authenticated and requires the external reference of the object or secret to be checked. Instead of returning the specified object or secret, the HEAD request returns a status code and a response header with the size of the stored object or secret.

For full details of the HEAD method, please see the API docs for IBC S6, and IBC NKV.

Getting Started

In this tutorial we will provide examples and code snippets of how to use the new HEAD method:

  1. Using the HEAD method with ioncli
  2. Using the HEAD method with the Ionburst Cloud Go SDK

ioncli

In this example, we will upload a file, my-file.txt to IBC S6 using ioncli, then verify its size with the ioncli head command.

Creating my-file.txt:

echo "We may guard your data, but we'll never take its freedom" > my-file.txt

Uploading my-file.txt with ioncli:

ioncli --profile ioncli-example put head-example my-file.txt

Checking my-file.txt with ioncli:

ioncli --profile ioncli-example head head-example

Example output:

[hello@ioncli-example ~]$ echo "We may guard your data, but we'll never take its freedom" > my-file.txt
[hello@ioncli-example ~]$ ls -lah my-file.txt
-rw-rw-r--. 1 hello hello 57B Sep 04 13:37 my-file.txt
[hello@ioncli-example ~]$ ioncli --profile default head head-example
Size: 57

Go SDK

The following example program shows how the Ionburst Cloud Go SDK Head and HeadWithLen methods can be used:

package main
import (
"fmt"
"gitlab.com/ionburst/ionburst-sdk-go"
"os"
)
func main() {
client, err := ionburst.NewClient()
if err != nil {
fmt.Println(err)
}
ioReader, _ := os.Open("my-file.txt")
err = client.Put("head-example", ioReader, "")
if err != nil {
fmt.Println(err)
}
err = client.Head("head-example")
if err != nil {
fmt.Println(err)
} else {
fmt.Printf("Checked: %s\n", "head-example")
}
size, err := client.HeadWithLen("head-example")
if err != nil {
fmt.Println(err)
} else {
fmt.Printf("Size: %d\n", size)
}
}

Example output:

[hello@example head]$ go run main.go
Checked: head-example
Size: 57

· 7 min read
Josh Fraser

Overview

The IonFS Command Line Interface provides a set of tools to manage data stored by Ionburst Cloud as if it were a remote filesystem. With the release of NKV, IonFS can now store secrets, while storing the metadata in a customer-owned metadata repository.

Anyone that has been granted access to this repository, and the appropriate Ionburst Cloud credentials, can interact with the stored secrets.

For this tutorial, we will be using Amazon S3 as the ionfs metadata repository.

Shared Responsibility Model Breakdown

Customer Responsibility

  • You, the customer, are responsible for the secure management of the Ionburst Cloud credentials used by ionfs.
  • You, the customer, are responsible for the security of ionfs metadata repositories and the metadata stored in them.

Ionburst Cloud Responsibility

  • We are responsible for the security of all secrets stored in Ionburst Cloud NKV using ionfs.
  • We are responsible for the underlying security and availability of the Ionburst Cloud platform.

Getting Started

In this tutorial we will cover:

  1. Working with ionfs metadata repositories.
  2. Listing IBC classifications with ionfs.
  3. Working with ionfs directories.
  4. Managing secrets with ionfs.

Basic Usage

ionfs allows us to do the following:

  • List configured metadata repositories.
  • List available IBC classifications.
  • Create, list and delete ionfs directories.
  • Upload, download and delete secrets from IBC.

1. Metadata Repositories

ionfs makes use of metadata repositories, or repos, to track secrets that have been secured by Ionburst Cloud NKV. Metadata repos are specified in the configuration file stored under ~/.ionfs/appsettings.json.

To list the configured repos, the following ionfs command can be used:

ionfs repos

An example output would look like:

[hello@ionfs-example ~]$ ionfs repos
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Available Repositories (*default):
* [s] ion://s3-example-ionfs-nkv/ (Ionburst.Apps.IonFS.Repo.S3.MetadataS3)

2. Classifications

Secrets can be secured by Ionburst Cloud according to available security policies. ionfs can be used to view the policies currently available to an Ionburst Cloud party.

To list available policies, the following can be used:

ionfs policy

An example output would look like:

[hello@ionfs-example ~]$ ionfs policy
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Available Classifications:
2:Restricted

3. Directories

Secrets secured by Ionburst Cloud NKV through ionfs can partition its repo using a typical directory structure.

List directories

To list available directories within a repo, the following can be used:

ionfs list

An example output would look like:

[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Secrets Repo of ion://s3-example-ionfs-nkv/
d example/

By default, this will list the contents of the repo's root directory. To list a specific directory, the following can be used:

ionfs list ion://example

An example output would look like:

[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Secrets Repo of ion://s3-example-ionfs-nkv/example/
Remote Secrets repository is empty is empty

Create a directory

To create a new directory within a repo, the following can be used:

ionfs mkdir ion://new-directory

An example output would look like:

[hello@ionfs-example ~]$ ionfs mkdir ion://new-directory
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Secrets Repo of ion://s3-example-ionfs-nkv/
d example/
d new-directory/

Delete a directory

To remove a directory within a repo, the following can be used:

ionfs rmdir ion://new-directory

An example output would look like:

[hello@ionfs-example ~]$ ionfs rmdir ion://new-directory
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Secrets Repo of ion://s3-example-ionfs-nkv/
d example/

4. Secrets

Finally, and most importantly we can now look at uploading (Put), downloading (Get) and deleting secrets from IBC NKV using ionfs. In these examples, we'll use a secret called my-secret, with the value "We may guard your data, but we'll never take its freedom".

Put

To upload a secret with ionfs, the following can be used:

ionfs secrets put "We may guard your data, but we'll never take its freedom" ion:// my-secret

An example output would look like:

[hello@ionfs-example ~]$ ionfs secrets put "We may guard your data, but we'll never take its freedom" ion:// my-secret
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Secrets Repo of ion://s3-example-ionfs-nkv/
d example/
my-secret 22/07/2021 13:10:14

To upload a secret to a specific directory within your repo, use the following:

ionfs secrets put "We may guard your data, but we'll never take its freedom" ion://example my-secret

An example output would look like:

[hello@ionfs-example ~]$ ionfs secrets put "We may guard your data, but we'll never take its freedom" ion:// my-secret
[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Secrets Repo of ion://s3-example-ionfs-nkv/example/
example/my-secret 22/07/2021 13:11:52

Get

To retrieve a secret with ionfs, use the following:

ionfs secrets get ion://example/my-secret

An example output would look like:

[hello@ionfs-example ~]$ ionfs secrets get ion://example/my-secret
We may guard your data, but we'll never take its freedom

Delete

To delete a secret from the ionfs repo and from Ionburst Cloud NKV, the following can be used:

ionfs secrets del ion://example/my-secret

An example output would look like:

[hello@ionfs-example ~]$ ionfs secrets del ion://example/my-secret
[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Secrets Repo of ion://s3-example-ionfs-nkv/example/
Remote Secrets repository is empty

Conclusion

You should now be able to perform basic secrets operations on Ionburst Cloud NKV using the ionfs tool. If you're interested in learning more about the IonFS CLI, please see the Ionburst Cloud docs.

· 7 min read
Josh Fraser

Overview

The IonFS Command Line Interface provides a set of tools to manage data stored by Ionburst Cloud S6 as if it were a remote filesystem. While the IonFS CLI stores files within Ionburst Cloud S6, the metadata is stored in a customer-owned metadata repository.

Anyone that has been granted access to this repository, and the appropriate Ionburst Cloud credentials, can interact with the stored data.

For this tutorial, we will be using Amazon S3 as the ionfs metadata repository.

Shared Responsibility Model Breakdown

Customer Responsibility

  • You, the customer, are responsible for the secure management of the Ionburst Cloud credentials used by ionfs.
  • You, the customer, are responsible for the security of ionfs metadata repositories and the metadata stored in them.

Ionburst Cloud Responsibility

  • We are responsible for the security of all data stored in Ionburst Cloud S6 using ionfs.
  • We are responsible for the underlying security and availability of the Ionburst Cloud platform.

Getting Started

In this tutorial we will cover:

  1. Working with ionfs metadata repositories.
  2. Listing IBC classifications with ionfs.
  3. Working with ionfs directories.
  4. Managing files with ionfs.

Basic Usage

ionfs allows us to do the following:

  • List configured metadata repositories.
  • List available IBC classifications.
  • Create, list and delete ionfs directories.
  • Upload, download and delete data from IBC.

1. Metadata Repositories

ionfs makes use of metadata repositories, or repos, to track data that has been secured by Ionburst Cloud S6. Metadata repos are specified in the configuration file stored under ~/.ionfs/appsettings.json.

To list the configured repos, the following ionfs command can be used:

ionfs repos

An example output would look like:

[hello@ionfs-example ~]$ ionfs repos
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Available Repositories (*default):
* [d] ion://s3-example-ionfs/ (Ionburst.Apps.IonFS.Repo.S3.MetadataS3)

2. Classifications

Data can be secured by Ionburst Cloud according to available security policies. ionfs can be used to view the policies currently available to an Ionburst Cloud party.

To list available policies, the following can be used:

ionfs policy

An example output would look like:

[hello@ionfs-example ~]$ ionfs policy
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Available Classifications:
2:Restricted

3. Directories

Data secured by Ionburst Cloud S6 through ionfs can partition its repo using a typical directory structure.

List directories

To list available directories within a repo, the following can be used:

ionfs list

An example output would look like:

[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/
d example/

By default, this will list the contents of the repo's root directory. To list a specific directory, the following can be used:

ionfs list ion://example

An example output would look like:

[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/example/
Remote directory is empty

Create a directory

To create a new directory within a repo, the following can be used:

ionfs mkdir ion://new-directory

An example output would look like:

[hello@ionfs-example ~]$ ionfs mkdir ion://new-directory
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/
d example/
d new-directory/

Delete a directory

To remove a directory within a repo, the following can be used:

ionfs rmdir ion://new-directory

An example output would look like:

[hello@ionfs-example ~]$ ionfs rmdir ion://new-directory
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/
d example/

4. Files

Finally, and most importantly we can now look at uploading (Put), downloading (Get) and deleting data from IBC S6 using ionfs. In these examples, we'll use a file called my-file.txt.

First, we need to create my-file.txt:

echo "We may guard your data, but we'll never take its freedom" > my-file.txt

Put

To upload a file to Ionburst Cloud with ionfs, the following can be used:

ionfs put my-file.txt ion://

An example output would look like:

[hello@ionfs-example ~]$ ionfs put my-file.txt ion://
[hello@ionfs-example ~]$ ionfs list
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/
d example/
my-file.txt 23/4/2021 13:49:51

To upload data to a specific directory within your repo, use the following:

ionfs put my-file.txt ion://example

An example output would look like:

[hello@ionfs-example ~]$ ionfs put my-file.txt ion://example
[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/example/
example/my-file.txt 23/4/2021 13:50:23

Get

To retrieve a file with ionfs, use the following:

ionfs get ion://example/my-file.txt

An example output would look like:

[hello@ionfs-example ~]$ rm my-file.txt
[hello@ionfs-example ~]$ ionfs get ion://example/my-file.txt
[hello@ionfs-example ~]$ ls
my-file.txt
[hello@ionfs-example ~]$ cat my-file.txt
We may guard your data, but we'll never take its freedom

By default, this will download the file from Ionburst Cloud S6 to the current directory, with the name used in ionfs. To download to a specific local directory, or to download to a different name, use the following:

ionfs get -n my-file-2.txt ion://example/my-file.txt

An example output would look like:

[hello@ionfs-example ~]$ ionfs get -n my-file-2.txt ion://example/my-file.txt
[hello@ionfs-example ~]$ ls
my-file.txt my-file-2.txt
[hello@ionfs-example ~]$ cat my-file-2.txt
We may guard your data, but we'll never take its freedom

Delete

To delete a file from the ionfs repo and from Ionburst Cloud S6, the following can be used:

ionfs del ion://example/my-file.txt

An example output would look like:

[hello@ionfs-example ~]$ ionfs del ion://example/my-file.txt
[hello@ionfs-example ~]$ ionfs list ion://example
____ ___________
/ _/___ ____ / ____/ ___/
/ // __ \/ __ \/ /_ \__ \
_/ // /_/ / / / / __/ ___/ /
/___/\____/_/ /_/_/ /____/
Directory of ion://s3-example-ionfs/example/
Remote directory is empty

Conclusion

You should now be able to perform basic file operations on Ionburst Cloud S6 using the ionfs tool. If you're interested in learning more about the IonFS CLI, please see the Ionburst Cloud docs.

· 4 min read
Josh Fraser

Overview

ioncli is a simple Command Line Interface tool that allows data to be uploaded, downloaded and deleted from Ionburst Cloud S6. ioncli also allows the listing of available classifications.

The aim of this tutorial is to learn how to setup ioncli, and use it to perform basic operations against the Ionburst Cloud S6 API.

Shared Responsibility Model Breakdown

Customer Responsibility

  • You, the customer, are responsible for the secure management of the Ionburst Cloud credentials used by the ioncli tool.
  • ioncli does not provide any client-side encryption functionality. If this is used in conjunction with the usage of ioncli, it is the reponsibility of the customer to manage.
  • ioncli does not provide any metadata tracking or management for data stored. It is the customer's responsibility to track and record this information.

Ionburst Cloud Responsibility

  • We are responsible for the security of all data stored in Ionburst Cloud S6 using the ioncli tool.
  • We are responsible for the underlying security and availability of the Ionburst Cloud services.

Getting Started

In this tutorial we will cover:

  1. Retrieving the available Ionburst Cloud classifications.
  2. Uploading a file to Ionburst Cloud S6.
  3. Downloading a file from Ionburst Cloud S6.
  4. Deleting a file from Ionburst Cloud S6.

Basic Usage

ioncli provides functionality for the following:

  • Classifications listing.
  • PUT - Uploading data to Ionburst Cloud S6.
  • GET - Downloading data from Ionburst Cloud S6.
  • DELETE - Deleting data from Ionburst Cloud S6.

1. Classifications

Data can be secured by Ionburst Cloud S6 according to available security classifications. ioncli can be used to view the policies currently available to an Ionburst Cloud party.

To list available policies, the following can be used:

ioncli --profile ioncli-example classifications list

The output for this command should look like:

[hello@ioncli-example ~]$ ioncli --profile ioncli-example classifications list
Classifications: 1
Classification
-----------------
Restricted

2. Uploading Data

To upload data to Ionburst Cloud S6 using ioncli, an object ID and file must be supplied. In this example we will upload the file my-file.txt as ID my-ioncli-put.

Creating my-file.txt:

echo "We may guard your data, but we'll never take its freedom" > my-file.txt

Uploading my-file.txt with ioncli:

ioncli --profile ioncli-example put my-ioncli-put my-file.txt

This operation does not return any output on success.

3. Downloading Data

To retrieve data from Ionburst Cloud S6 using ioncli, an object ID and output path must be provided. In this example, we will download the previously uploaded object, my-ioncli-put, to the path/file my-downloaded-file.txt.

Downloading my-ioncli-put:

ioncli --profile ioncli-example get my-ioncli-put my-downloaded-file.txt

This operation does not return any output on success.

We can now view the downloaded file:

cat my-ioncli-put

The output for this command should look like:

[hello@ioncli-example ~]$ cat my-ioncli-put
We may guard your data, but we'll never take its freedom

4. Deleting Data

To delete data from Ionburst Cloud S6 using ioncli, an object ID must be provided. In this example, we will download the previously used object, my-ioncli-put.

Deleting my-ioncli-put:

ioncli --profile ioncli-example delete my-ioncli-put

This operation does not return any output on success.

We can verify the object has been deleted by attempting to download my-ioncli-put:

ioncli --profile ioncli-example get my-ioncli-put my-downloaded-file.txt

The output for this command should look like:

[hello@ioncli-example ~]$ ioncli --profile ioncli-example get my-ioncli-put my-downloaded-file.txt
2021/06/27 18:59:22 Error performing Ionburst API Operation: [GET] https://api.eu-west-1.ionburst.cloud/api/data/my-ioncli-put :: 404 ->

Conclusion

You should now be able to perform basic data operations on Ionburst Cloud S6 using the ioncli tool.

If you're interested in learning more about our more fully-featured command-line tool, IonFS, which features object metadata management and filesystem-like interactions, please see the Ionburst Cloud docs.