Skip to main content

CloudServer

Overview

Scality CloudServer is an open-source S3-compatible object storage service written in Node.js, that allows a collection of local and remote storage locations to be accessed through a single S3 API.

CloudServer operates using a data layer (file, object storage etc.), and metadata layer (file, MongoDB). As this structure uses the base structure as IonFS, it was a strong candidate for integration with IBC S6 as a backend storage option.

The Ionburst Cloud fork of CloudServer is split over two repositories:

Getting Started

To use the Ionburst Cloud fork of CloudServer, you will need the following installed:

  • Node.js 16 or newer
  • Yarn 1 - latest stable version is v1.22.19, it should be noted that Yarn 1 is now in maintenance mode
  • git 2.17 or newer
  • Python 3.7 or newer (for node-gyp)
  • GCC/G++ (for node-gyp)

Installing CloudServer

First, clone or download the CloudServer repository from the Ionburst Cloud GitHub:

git clone https://github.com/ionburstcloud/cloudserver.git

Open the repository directory in your editor of choice, or cd into it and install the dependencies:

yarn install --frozen-lockfile

Configuring CloudServer for IBC S6

To start CloudServer with IBC S6, some minor adjustments have to be made to the config file config.json.

The Ionburst config block in this file is set to a credentials profile of default and the IBC eu-west-1 API endpoint. To use a different profile or endpoint, update as appropriate.

"Ionburst": {
"Profile": "default",
"IonburstUri": "https://api.eu-west-1.ionburst.cloud/",
"TraceCredentialsFile": "OFF"
}

The IBC S6 integration for CloudServer currently repurposes CloudServer's file storage implementation, so no additional config is required to get started.

You should also note the following directories within the repository directory:

  • localData - As CloudServer now uses IBC S6 to store objects, our implementation uses this as a temporary directory.
  • localMetadata - CloudServer stores the metadata for buckets and objects here, unless configured otherwise.

For authentication, CloudServer uses access/secret key pairs specified in the conf/authdata.json file. By default, the following are available:

{
"accounts": [{
"name": "Bart",
"email": "sampleaccount1@sampling.com",
"arn": "arn:aws:iam::123456789012:root",
"canonicalID": "79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be",
"shortid": "123456789012",
"keys": [{
"access": "accessKey1",
"secret": "verySecretKey1"
}]
},
{
"name": "Lisa",
"email": "sampleaccount2@sampling.com",
"arn": "arn:aws:iam::123456789013:root",
"canonicalID": "79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2bf",
"shortid": "123456789013",
"keys": [{
"access": "accessKey2",
"secret": "verySecretKey2"
}]
},
{
"name": "Clueso",
"email": "inspector@clueso.info",
"arn": "arn:aws:iam::123456789014:root",
"canonicalID": "http://acs.zenko.io/accounts/service/clueso",
"shortid": "123456789014",
"keys": [{
"access": "cluesoKey1",
"secret": "cluesoSecretKey1"
}]
},
{
"name": "Replication",
"email": "inspector@replication.info",
"arn": "arn:aws:iam::123456789015:root",
"canonicalID": "http://acs.zenko.io/accounts/service/replication",
"shortid": "123456789015",
"keys": [{
"access": "replicationKey1",
"secret": "replicationSecretKey1"
}]
},
{
"name": "Lifecycle",
"email": "inspector@lifecycle.info",
"arn": "arn:aws:iam::123456789016:root",
"canonicalID": "http://acs.zenko.io/accounts/service/lifecycle",
"shortid": "123456789016",
"keys": [{
"access": "lifecycleKey1",
"secret": "lifecycleSecretKey1"
}]
}]
}

For the purpose of this example, we will use the accessKey1 / verySecretKey1 pair.

Once the appropriate Ionburst SDK configuration has been added to config.json, CloudServer can be started on localhost:8000 with:

REMOTE_MANAGEMENT_DISABLE=1 yarn start

Basic Operations with CloudServer and IBC S6

Now that CloudServer is running, we can now store and interact with objects on IBC S6.

For these examples we will use the AWS CLI with the s3 and s3api subcommands.

For object operations, we will use a file called my-file.txt.

echo "We may guard your data, but we'll never take its freedom" > my-file.txt

Creating a bucket

First we need to create a new bucket, called ionburst, on CloudServer. This can be done with the s3api or the s3 sub-command:

s3api

AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api create-bucket --bucket ionburst --endpoint-url=http://localhost:8000 --region us-east-1

Output:

AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api create-bucket --bucket ionburst --endpoint-url=http://localhost:8000 --region us-east-1
{
"Location": "/ionburst"
}

s3

AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 mb s3://ionburst2 --endpoint-url=http://localhost:8000 --region us-east-1

Output:

[example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 mb s3://ionburst --endpoint-url=http://localhost:8000 --region us-east-1
make_bucket: ionburst2

Listing buckets

Buckets can be listed with either of the following:

s3api

AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api list-buckets --endpoint-url=http://localhost:8000 --region us-east-1

Output:

[example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api get-object --bucket ionburst --key my-file.txt my-file-get.txt --endpoint-url=http://localhost:8000 --region us-east-1
{
"AcceptRanges": "bytes",
"LastModified": "2023-03-01T13:54:56+00:00",
"ContentLength": 57,
"ETag": "\"03e93310b885668bc883663a3ea274d9\"",
"Metadata": {}
}

s3

AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 ls --endpoint-url=http://localhost:8000 --region us-east-1

Output:

[example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api list-objects-v2 --bucket ionburst --endpoint-url=http://localhost:8000 --region us-east-1
{
"Contents": [
{
"Key": "my-file.txt",
"LastModified": "2023-03-01T14:29:49.757000+00:00",
"ETag": "\"03e93310b885668bc883663a3ea274d9\"",
"Size": 57,
"StorageClass": "STANDARD"
}
]
}

Uploading objects

Next, we can upload our object my-file.txt to our CloudServer bucket, to be stored on IBC S6:

s3api

AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api put-object --bucket ionburst --key my-file.txt --body my-file.txt --endpoint-url=http://localhost:8000 --region us-east-1

Output:

[example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api put-object --bucket ionburst --key my-file.txt --body my-file.txt --endpoint-url=http://localhost:8000 --region us-east-1
{
"ETag": "\"03e93310b885668bc883663a3ea274d9\""
}

s3

AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 cp my-file.txt s3://ionburst/my-file.txt --endpoint-url=http://localhost:8000 --region us-east-1

Output:

[example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 cp my-file.txt s3://ionburst/my-file.txt --endpoint-url=http://localhost:8000 --region us-east-1
upload: ./my-file.txt to s3://ionburst/my-file.txt

Listing objects

We can now verify our object is present in our CloudServer bucket:

s3api

AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api list-objects-v2 --bucket ionburst --endpoint-url=http://localhost:8000 --region us-east-1

Output:

[example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api list-objects-v2 --bucket ionburst --endpoint-url=http://localhost:8000 --region us-east-1
{
"Contents": [
{
"Key": "my-file.txt",
"LastModified": "2023-03-01T14:29:49.757000+00:00",
"ETag": "\"03e93310b885668bc883663a3ea274d9\"",
"Size": 57,
"StorageClass": "STANDARD"
}
]
}

s3

AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 ls s3://ionburst --endpoint-url=http://localhost:8000 --region us-east-1

Output:

AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 ls s3://ionburst --endpoint-url=http://localhost:8000 --region us-east-1
2023-03-01 14:32:58 57 my-file.txt

Downloading objects

Next, we can retrieve our object from the CloudServer bucket:

s3api

AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api get-object --bucket ionburst --key my-file.txt my-file-get.txt --endpoint-url=http://localhost:8000 --region us-east-1

Output:

[example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api get-object --bucket ionburst --key my-file.txt my-file-get.txt --endpoint-url=http://localhost:8000 --region us-east-1
{
"AcceptRanges": "bytes",
"LastModified": "2023-03-01T13:54:56+00:00",
"ContentLength": 57,
"ETag": "\"03e93310b885668bc883663a3ea274d9\"",
"Metadata": {}
}
[example@ibc-s6-cloudserver tmp]$ cat my-file-get.txt
We may guard your data, but we'll never take its freedom

s3

AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 cp s3://ionburst/my-file.txt my-file-get.txt --endpoint-url=http://localhost:8000 --region us-east-1

Output:

[example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 cp s3://ionburst/my-file.txt my-file-get.txt --endpoint-url=http://localhost:8000 --region us-east-1
download: s3://ionburst/my-file.txt to ./my-file-get.txt
[example@ibc-s6-cloudserver tmp]$ cat my-file-get.txt
We may guard your data, but we'll never take its freedom

Deleting objects

Finally, we can delete the my-file.txt object from our CloudServer bucket and IBC S6:

s3api

AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api delete-object --bucket ionburst --key my-file.txt --endpoint-url=http://localhost:8000 --region us-east-1

Output:

[example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api delete-object --bucket ionburst --key my-file.txt --endpoint-url=http://localhost:8000 --region us-east-1

s3

AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 rm s3://ionburst/my-file.txt --endpoint-url=http://localhost:8000 --region us-east-1

Output:

[example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 rm s3://ionburst/my-file.txt --endpoint-url=http://localhost:8000 --region us-east-1
delete: s3://ionburst/my-file.txt