CloudServer Overview Scality CloudServer is an open-source S3-compatible object storage service written in Node.js, that allows a collection of local and remote storage locations to be accessed through a single S3 API.
CloudServer operates using a data layer (file, object storage etc.), and metadata layer (file, MongoDB). As this structure uses the base structure as IonFS, it was a strong candidate for integration with IBC S6 as a backend storage option.
The Ionburst Cloud fork of CloudServer is split over two repositories:
Getting Started To use the Ionburst Cloud fork of CloudServer, you will need the following installed:
Node.js 16 or newer Yarn 1 - latest stable version is v1.22.19, it should be noted that Yarn 1 is now in maintenance mode git 2.17 or newer Python 3.7 or newer (for node-gyp) GCC/G++ (for node-gyp) Installing CloudServer First, clone or download the CloudServer repository from the Ionburst Cloud GitHub:
Copy git clone https://github.com/ionburstcloud/cloudserver.git
Open the repository directory in your editor of choice, or cd into it and install the dependencies:
Copy yarn install --frozen-lockfile
Configuring CloudServer for IBC S6 To start CloudServer with IBC S6, some minor adjustments have to be made to the config file config.json.
The Ionburst config block in this file is set to a credentials profile of default and the IBC eu-west-1 API endpoint. To use a different profile or endpoint, update as appropriate.
Copy "Ionburst" : {
"Profile" : "default" ,
"IonburstUri" : "https://api.eu-west-1.ionburst.cloud/" ,
"TraceCredentialsFile" : "OFF"
}
The IBC S6 integration for CloudServer currently repurposes CloudServer's file storage implementation, so no additional config is required to get started.
You should also note the following directories within the repository directory:
localData - As CloudServer now uses IBC S6 to store objects, our implementation uses this as a temporary directory.localMetadata - CloudServer stores the metadata for buckets and objects here, unless configured otherwise.For authentication, CloudServer uses access/secret key pairs specified in the conf/authdata.json file. By default, the following are available:
Copy {
"accounts" : [ {
"name" : "Bart" ,
"email" : "sampleaccount1@sampling.com" ,
"arn" : "arn:aws:iam::123456789012:root" ,
"canonicalID" : "79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be" ,
"shortid" : "123456789012" ,
"keys" : [ {
"access" : "accessKey1" ,
"secret" : "verySecretKey1"
} ]
} ,
{
"name" : "Lisa" ,
"email" : "sampleaccount2@sampling.com" ,
"arn" : "arn:aws:iam::123456789013:root" ,
"canonicalID" : "79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2bf" ,
"shortid" : "123456789013" ,
"keys" : [ {
"access" : "accessKey2" ,
"secret" : "verySecretKey2"
} ]
} ,
{
"name" : "Clueso" ,
"email" : "inspector@clueso.info" ,
"arn" : "arn:aws:iam::123456789014:root" ,
"canonicalID" : "http://acs.zenko.io/accounts/service/clueso" ,
"shortid" : "123456789014" ,
"keys" : [ {
"access" : "cluesoKey1" ,
"secret" : "cluesoSecretKey1"
} ]
} ,
{
"name" : "Replication" ,
"email" : "inspector@replication.info" ,
"arn" : "arn:aws:iam::123456789015:root" ,
"canonicalID" : "http://acs.zenko.io/accounts/service/replication" ,
"shortid" : "123456789015" ,
"keys" : [ {
"access" : "replicationKey1" ,
"secret" : "replicationSecretKey1"
} ]
} ,
{
"name" : "Lifecycle" ,
"email" : "inspector@lifecycle.info" ,
"arn" : "arn:aws:iam::123456789016:root" ,
"canonicalID" : "http://acs.zenko.io/accounts/service/lifecycle" ,
"shortid" : "123456789016" ,
"keys" : [ {
"access" : "lifecycleKey1" ,
"secret" : "lifecycleSecretKey1"
} ]
} ]
}
For the purpose of this example, we will use the accessKey1 / verySecretKey1 pair.
Once the appropriate Ionburst SDK configuration has been added to config.json, CloudServer can be started on localhost:8000 with:
Copy REMOTE_MANAGEMENT_DISABLE=1 yarn start
Basic Operations with CloudServer and IBC S6 Now that CloudServer is running, we can now store and interact with objects on IBC S6.
For these examples we will use the AWS CLI with the s3 and s3api subcommands.
For object operations, we will use a file called my-file.txt.
Copy echo "We may guard your data, but we'll never take its freedom" > my-file.txt
Creating a bucket First we need to create a new bucket, called ionburst, on CloudServer. This can be done with the s3api or the s3 sub-command:
s3api Copy AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api create-bucket --bucket ionburst --endpoint-url=http://localhost:8000 --region us-east-1
Output:
Copy AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api create-bucket --bucket ionburst --endpoint-url=http://localhost:8000 --region us-east-1
{
"Location": "/ionburst"
}
Copy AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 mb s3://ionburst2 --endpoint-url=http://localhost:8000 --region us-east-1
Output:
Copy [example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 mb s3://ionburst --endpoint-url=http://localhost:8000 --region us-east-1
make_bucket: ionburst2
Listing buckets Buckets can be listed with either of the following:
s3api Copy AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api list-buckets --endpoint-url=http://localhost:8000 --region us-east-1
Output:
Copy [example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api get-object --bucket ionburst --key my-file.txt my-file-get.txt --endpoint-url=http://localhost:8000 --region us-east-1
{
"AcceptRanges": "bytes",
"LastModified": "2023-03-01T13:54:56+00:00",
"ContentLength": 57,
"ETag": "\"03e93310b885668bc883663a3ea274d9\"",
"Metadata": {}
}
Copy AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 ls --endpoint-url=http://localhost:8000 --region us-east-1
Output:
Copy [example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api list-objects-v2 --bucket ionburst --endpoint-url=http://localhost:8000 --region us-east-1
{
"Contents": [
{
"Key": "my-file.txt",
"LastModified": "2023-03-01T14:29:49.757000+00:00",
"ETag": "\"03e93310b885668bc883663a3ea274d9\"",
"Size": 57,
"StorageClass": "STANDARD"
}
]
}
Uploading objects Next, we can upload our object my-file.txt to our CloudServer bucket, to be stored on IBC S6:
s3api Copy AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api put-object --bucket ionburst --key my-file.txt --body my-file.txt --endpoint-url=http://localhost:8000 --region us-east-1
Output:
Copy [example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api put-object --bucket ionburst --key my-file.txt --body my-file.txt --endpoint-url=http://localhost:8000 --region us-east-1
{
"ETag": "\"03e93310b885668bc883663a3ea274d9\""
}
Copy AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 cp my-file.txt s3://ionburst/my-file.txt --endpoint-url=http://localhost:8000 --region us-east-1
Output:
Copy [example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 cp my-file.txt s3://ionburst/my-file.txt --endpoint-url=http://localhost:8000 --region us-east-1
upload: ./my-file.txt to s3://ionburst/my-file.txt
Listing objects We can now verify our object is present in our CloudServer bucket:
s3api Copy AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api list-objects-v2 --bucket ionburst --endpoint-url=http://localhost:8000 --region us-east-1
Output:
Copy [example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api list-objects-v2 --bucket ionburst --endpoint-url=http://localhost:8000 --region us-east-1
{
"Contents": [
{
"Key": "my-file.txt",
"LastModified": "2023-03-01T14:29:49.757000+00:00",
"ETag": "\"03e93310b885668bc883663a3ea274d9\"",
"Size": 57,
"StorageClass": "STANDARD"
}
]
}
Copy AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 ls s3://ionburst --endpoint-url=http://localhost:8000 --region us-east-1
Output:
Copy AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 ls s3://ionburst --endpoint-url=http://localhost:8000 --region us-east-1
2023-03-01 14:32:58 57 my-file.txt
Downloading objects Next, we can retrieve our object from the CloudServer bucket:
s3api Copy AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api get-object --bucket ionburst --key my-file.txt my-file-get.txt --endpoint-url=http://localhost:8000 --region us-east-1
Output:
Copy [example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api get-object --bucket ionburst --key my-file.txt my-file-get.txt --endpoint-url=http://localhost:8000 --region us-east-1
{
"AcceptRanges": "bytes",
"LastModified": "2023-03-01T13:54:56+00:00",
"ContentLength": 57,
"ETag": "\"03e93310b885668bc883663a3ea274d9\"",
"Metadata": {}
}
[example@ibc-s6-cloudserver tmp]$ cat my-file-get.txt
We may guard your data, but we'll never take its freedom
Copy AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 cp s3://ionburst/my-file.txt my-file-get.txt --endpoint-url=http://localhost:8000 --region us-east-1
Output:
Copy [example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 cp s3://ionburst/my-file.txt my-file-get.txt --endpoint-url=http://localhost:8000 --region us-east-1
download: s3://ionburst/my-file.txt to ./my-file-get.txt
[example@ibc-s6-cloudserver tmp]$ cat my-file-get.txt
We may guard your data, but we'll never take its freedom
Deleting objects Finally, we can delete the my-file.txt object from our CloudServer bucket and IBC S6:
s3api Copy AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api delete-object --bucket ionburst --key my-file.txt --endpoint-url=http://localhost:8000 --region us-east-1
Output:
Copy [example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3api delete-object --bucket ionburst --key my-file.txt --endpoint-url=http://localhost:8000 --region us-east-1
Copy AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 rm s3://ionburst/my-file.txt --endpoint-url=http://localhost:8000 --region us-east-1
Output:
Copy [example@ibc-s6-cloudserver tmp]$ AWS_ACCESS_KEY_ID=accessKey1 AWS_SECRET_ACCESS_KEY=verySecretKey1 aws s3 rm s3://ionburst/my-file.txt --endpoint-url=http://localhost:8000 --region us-east-1
delete: s3://ionburst/my-file.txt