Integration Deep Dive: Firefox Send and IBC S6

info

Note: Firefox Send was archived by Mozilla in September 2020, but as an open-source project, the source code was left available. We've based our integration on the Send fork maintained by Tim Visée.

Overview

In this tutorial, we're looking at how we integrated IBC S6 secure object storage as the back-end storage for the open-source file-sharing application, Firefox Send.

Firefox Send is a free, end-to-end encrypted file-sharing application developed by Mozilla, that allows users to easily and safely share files over the Web. The Send back-end is written in Node.js, allowing us to integrate with the Ionburst Cloud Node.js SDK.

Digging into the Send source code

From a cursory review of the source code and running the application locally, it looked like the focus of our integration would be the server directory, which contains the code for Send's back-end services.

Of particular interest was the storage sub-directory, which contains the functionality for integrating Send with the following:

Local filesystem storage;
Google Cloud Storage;
Amazon S3;

A review of these files outlined common pieces of functionality expected from storage integrations:

length – returns the object size from the configured storage method;
getStream – retrieves the object from the configured storage method;
set – uploads or writes the object to the configured storage method;
del – removes the object from the configured storage method;

At time of integration, IBC S6 provided functionality for three out of four, as it did not expose the ability to query object size. Further digging suggested that the length function was being used to set the Content-Length header on the file download response to the user, so wasn't a functional requirement for the storage integration.

Since we did this original integration work, we've added a HEAD API method, which can be used to query the size of objects stored in IBC S6.

Exploring the storage sub-directory also confirmed how Send handles object metadata. In ‘development', Send uses a local, in-memory store to track each object, but is designed to use Redis in production. To gain a better understanding of the Send application, all of our integration work was carried out using Redis as the Send metadata store.

The final checks required were to see how the Send back-end handled storage configuration. The base back-end configuration is handled in the config.js file found in the server directory, which defines the storage method selected by the index.js file found in the storage sub-directory.

Integrating IBC S6 - Configuration

To begin Integrating IBC S6 with Send, we first had to add new configuration options to the Send project so it could use IBC S6 as the new storage method, along with the initial Ionburst Cloud SDK configuration.

The Ionburst Cloud SDK was added to the project using npm:

npm install ionburst-sdk-javascript 

A local Redis instance was deployed to track Send metadata using Docker:

docker run -ti -p 6379:6379 redis:latest 

A config.json file was added to the root of the Send project for the Ionburst Cloud SDK configuration file.

{
    "Ionburst": {
        "Profile": "example", 
        "IonburstUri": "https://api.example.ionburst.cloud/", 
        "TraceCredentialsFile": "ON" 
    }
}

A new configuration item was then added to the Send config.js file for IBC S6. Note: this configuration entry is only used to select IBC S6 as the chosen back-end storage, and does not perform any other configuration. The redis_host entry was also adjusted to 127.0.0.1 to override the local memorystore:

const conf = convict({ 
  ionburst: { 
    format: String, 
    default: 'true'
  }, 
--- truncated ---
redis_host: { 
    format: String, 
    default: '127.0.0.1', 
    env: 'REDIS_HOST' 
  }, 
--- truncated ---
}); 

A configuration option was added to the storage index.js file, to ensure IBC S6 was selected as the storage method:

class DB { 
  constructor(config) { 
    let Storage = null; 
    if (config.ionburst) { 
      Storage = require('./ionburst'); 
    } else if (config.s3_bucket) { 
      Storage = require('./s3'); 
    } else if (config.gcs_bucket) { 
      Storage = require('./gcs'); 
    } else { 
      Storage = require('./fs'); 
    }
    this.log = mozlog('send.storage');

    this.storage = new Storage(config, this.log); 
 
    this.redis = createRedisClient(config); 
    this.redis.on('error', err => { 
      this.log.error('Redis:', err); 
    }); 
  } 
--- truncated ---
} 

Finally, an ionburst.js file was added to the storage sub-directory, and a constructor created for applicable configuration:

class IonburstStorage { 
    constructor(config, log) { 
      this.log = log;
    } 

Integrating IBC S6 – File Operations

IBC S6 PUT

From the storage index.js file, we can see how Send kicks off a file upload to its configured storage:

async set(id, file, meta, expireSeconds = config.default_expire_seconds) {
    const prefix = getPrefix(expireSeconds);
    const filePath = `${prefix}-${id}`;
    await this.storage.set(filePath, file);
    this.redis.hset(id, 'prefix', prefix);
    if (meta) {
      this.redis.hmset(id, meta);
    }
    this.redis.expire(id, expireSeconds);
  }

In this snippet, we can see that Send generates an identifier for each file stored, before passing it and the file to the configured storage method. As IBC S6 has no preference to how a given object is identified, we can simply pass this identifier to IBC S6 too.

To upload the file to IBC S6, the following function was created in ionburst.js:

set(id, file) {
    return new Promise((resolve, reject) => {
      const putPath = path.join(this.dir, id);
      const fstream = fs.createWriteStream(putPath);
      file.pipe(fstream);
      file.on('error', err => {
        fstream.destroy(err);
      });
      fstream.on('error', err => {
        fs.unlinkSync(putPath);
        reject(err);
      });
      fstream.on('finish', async function() {
        var upload_data = fs.readFileSync(putPath);
        let put = await ionburst.putAsync({
          id: id,
          data: upload_data
        });
        console.log(put);
        fs.unlink(putPath, function(error) {
          if (error) {
            throw error;
          }
        });
        resolve();
      });
    });
  }

We encountered some issues passing the file object directly to the Ionburst Cloud SDK. To overcome this, we instead leveraged the existing filesystem functionality to write the file to a temporary directory, create a read stream for the Ionburst Cloud SDK, then remove the temporary file after successful upload.

This temporary file/directory leveraged functionality used by Send's file-system storage, and it was simply a matter of pulling the temporary directory configuration into the Ionburst storage constructor:

class IonburstStorage { 
  constructor(config, log) { 
    this.log = log;
    this.dir = config.file_dir;
    mkdirp.sync(this.dir);
} 

IBC S6 GET

Similar to the upload function, the main download functionality can be found in the storage index.js file:

async get(id) { 
  const filePath = await this.getPrefixedId(id); 
  console.log(filePath); 
  return this.storage.getStream(filePath); 
}

To keep things simple, we replicated the same temporary file functionality for the file download from IBC S6:

async getStream(id) {
  let data = await ionburst.getAsync(id);
  var getPath = path.join(this.dir, id);

  fs.writeFileSync(getPath, data);
  var returnData = fs.createReadStream(getPath);
  returnData.on('end', function() {
    fs.unlink(getPath, function(error) {
      if (error) {
        throw error;
      }
    });
  });
return returnData;
}

We first grab the file from IBC S6, write it to the temporary directory, then create and return a read stream.

IBC S6 DELETE

Send requires delete functionality from the configured storage method to remove uploaded files once they have reached their download limit, or expiry time.

The IBC S6 delete function was simple to implement:

del(id) {
  return ionburst.delete(id, function(err, data) {
    if (err) {
      throw err;
    }
    console.log(data);
  });
}

Caveats

Content-Length

At the time of integration, IBC S6 had no method of returning a stored object's size, nor did the Send metadata store it. As IBC S6 now has a HEAD API method, this can be added to the implementation to pass the Content-Length header on the file download reponse.

File Size

Depending on the deployment, Send can handle files up to 2.5GB. IBC S6 currently supports a maximum object size of 50MB, with larger objects requiring client-side processing before upload.

As a simple proof-of-concept, we've kept this 50MB limit in place for our fork of Send. However, since the time of integration, we've started to add our new SDK Manifests feature, which allows the Ionburst Cloud SDK to handle objects larger than 50MB. Once manifests have been added to our Node.js SDK, this functionality will be added to our Send fork.

Conclusion

All in all, integrating Firefox Send with IBC S6 was a relatively quick and simple process, providing us an opportunity to try out our Node.js SDK in an exisiting application. To try our Send fork for yourself, please check out our Getting started with Send and IBC S6 and Secure file-sharing with IBC S6, Firefox Send and the AWS free tier tutorials.

The full project source for our Send fork can be found here.

We'd also like to say thanks to the Mozilla team for building the original Send application, and to Tim Visée for maintaining the main Send fork.

info

Overview​

Digging into the Send source code​

Integrating IBC S6 - Configuration​

Integrating IBC S6 – File Operations​

IBC S6 PUT​

IBC S6 GET​

IBC S6 DELETE​

Caveats​

Content-Length​

File Size​

Conclusion​

Overview

Digging into the Send source code

Integrating IBC S6 - Configuration

Integrating IBC S6 – File Operations

IBC S6 PUT

IBC S6 GET

IBC S6 DELETE

Caveats

Content-Length

File Size

Conclusion