Skip to main content

One post tagged with "Manifests"

View All Tags

· 3 min read
Josh Fraser
info

Prerequisites - before you begin, please ensure:

Overview​

An IBC S6 API transaction currently supports a maximum upload size of 50MB. To upload objects larger than this limit, objects have to be split into 50MB chunks and uploaded individually, placing an implementation and management overhead on developers and client tools to track and manage these chunks.

To make it easier for developers and client tools to upload large objects to IBC S6, we've developed what we call IBC S6 SDK manifests. The SDK manifest is comprised of two main parts:

  • An implementation within each Ionburst Cloud SDK to chunk large objects and manage the upload of each chunk to IBC S6
  • A metadata object that tracks important information about each chunk; their IDs, checksum, ordinality etc.

To keep the manifest implementation simple, and to ensure the amount of information held by IBC S6 about a given object is minimised, this manifest metadata object is also stored within IBC S6 using the external reference (object ID) of the request.

This allows the chunks comprising a large object to be tracked, while avoiding any overhead on the client to track and manage them. It also means that to retrieve a large object, the external reference can be passed to the SDK manifest function, which will in turn retrieve and reconstruct the chunks.

The SDK manifests feature is similar to the multipart upload concept used by object stores like Amazon S3.

Getting Started​

In this tutorial we will provide examples and code snippets of how to use the new manifest feature to upload large objects to IBC S6:

  1. Uploading large objects with ioncli
  2. Uploading large objects with the Ionburst Cloud Go SDK

ioncli​

In this example, we will upload our large object, my-large-file.png to IBC S6 using the ioncli mput command.

Uploading my-large-file.png with ioncli:

ioncli --profile ioncli-example put manifest-example my-large-file.png

Example output:

[hello@ioncli-example ~]$ ls -lah my-large-file.png
-rw-rw-r--. 1 hello hello 125M 13 Sep 09:39 my-large-file.png
[hello@ioncli-example ~]$ ioncli --profile ioncli-example mput manifest-example my-large-file.png
Split to: fed44c96-f457-4e98-829c-b6809ec26e42
Split to: ac45ec73-ac16-4eaf-8251-79b2e1561a56
Split to: a41336ee-de91-474f-ad18-f5c36fe28060

Go SDK​

The following example program shows how the Ionburst Cloud Go SDK PutManifest methods can be used:

package main
import (
"fmt"
"gitlab.com/ionburst/ionburst-sdk-go"
"os"
)
func main() {
client, err := ionburst.NewClient()
if err != nil {
fmt.Println(err)
}
ioReader, _ := os.Open("my-large-file.png")
err = client.PutManifest("manifest-example", ioReader, "")
if err != nil {
fmt.Println(err)
}
err = client.Head("manifest-example")
if err != nil {
fmt.Println(err)
} else {
fmt.Printf("Checked: %s\n", "manifest-example")
}
size, err := client.HeadWithLen("manifest-example")
if err != nil {
fmt.Println(err)
} else {
fmt.Printf("Size: %d\n", size)
}
}

Example output:

[hello@example head]$ go run main.go
Split to: 45202923-bb8f-4775-9d06-f7d69c50e883
Split to: de109e03-de07-46d5-a900-76bcaf612d81
Split to: 97301ce9-8645-431b-b9a9-d2f6bab7c2a9
Checked: manifest-example
Size: 498