Skip to main content

3 posts tagged with "Firefox Send"

View All Tags

· 8 min read
Josh Fraser
info

Note: Firefox Send was archived by Mozilla in September 2020, but as an open-source project, the source code was left available. We've based our integration on the Send fork maintained by Tim Visée.

Overview

In this tutorial, we're looking at how we integrated IBC S6 secure object storage as the back-end storage for the open-source file-sharing application, Firefox Send.

Firefox Send is a free, end-to-end encrypted file-sharing application developed by Mozilla, that allows users to easily and safely share files over the Web. The Send back-end is written in Node.js, allowing us to integrate with the Ionburst Cloud Node.js SDK.

Digging into the Send source code

From a cursory review of the source code and running the application locally, it looked like the focus of our integration would be the server directory, which contains the code for Send's back-end services.

Of particular interest was the storage sub-directory, which contains the functionality for integrating Send with the following:

  • Local filesystem storage;
  • Google Cloud Storage;
  • Amazon S3;

A review of these files outlined common pieces of functionality expected from storage integrations:

  • length – returns the object size from the configured storage method;
  • getStream – retrieves the object from the configured storage method;
  • set – uploads or writes the object to the configured storage method;
  • del – removes the object from the configured storage method;

At time of integration, IBC S6 provided functionality for three out of four, as it did not expose the ability to query object size. Further digging suggested that the length function was being used to set the Content-Length header on the file download response to the user, so wasn't a functional requirement for the storage integration.

Since we did this original integration work, we've added a HEAD API method, which can be used to query the size of objects stored in IBC S6.

Exploring the storage sub-directory also confirmed how Send handles object metadata. In ‘development', Send uses a local, in-memory store to track each object, but is designed to use Redis in production. To gain a better understanding of the Send application, all of our integration work was carried out using Redis as the Send metadata store.

The final checks required were to see how the Send back-end handled storage configuration. The base back-end configuration is handled in the config.js file found in the server directory, which defines the storage method selected by the index.js file found in the storage sub-directory.

Integrating IBC S6 - Configuration

To begin Integrating IBC S6 with Send, we first had to add new configuration options to the Send project so it could use IBC S6 as the new storage method, along with the initial Ionburst Cloud SDK configuration.

The Ionburst Cloud SDK was added to the project using npm:

npm install ionburst-sdk-javascript

A local Redis instance was deployed to track Send metadata using Docker:

docker run -ti -p 6379:6379 redis:latest

A config.json file was added to the root of the Send project for the Ionburst Cloud SDK configuration file.

{
"Ionburst": {
"Profile": "example",
"IonburstUri": "https://api.example.ionburst.cloud/",
"TraceCredentialsFile": "ON"
}
}

A new configuration item was then added to the Send config.js file for IBC S6. Note: this configuration entry is only used to select IBC S6 as the chosen back-end storage, and does not perform any other configuration. The redis_host entry was also adjusted to 127.0.0.1 to override the local memorystore:

const conf = convict({
ionburst: {
format: String,
default: 'true'
},
--- truncated ---
redis_host: {
format: String,
default: '127.0.0.1',
env: 'REDIS_HOST'
},
--- truncated ---
});

A configuration option was added to the storage index.js file, to ensure IBC S6 was selected as the storage method:

class DB {
constructor(config) {
let Storage = null;
if (config.ionburst) {
Storage = require('./ionburst');
} else if (config.s3_bucket) {
Storage = require('./s3');
} else if (config.gcs_bucket) {
Storage = require('./gcs');
} else {
Storage = require('./fs');
}
this.log = mozlog('send.storage');
this.storage = new Storage(config, this.log);
this.redis = createRedisClient(config);
this.redis.on('error', err => {
this.log.error('Redis:', err);
});
}
--- truncated ---
}

Finally, an ionburst.js file was added to the storage sub-directory, and a constructor created for applicable configuration:

class IonburstStorage {
constructor(config, log) {
this.log = log;
}

Integrating IBC S6 – File Operations

IBC S6 PUT

From the storage index.js file, we can see how Send kicks off a file upload to its configured storage:

async set(id, file, meta, expireSeconds = config.default_expire_seconds) {
const prefix = getPrefix(expireSeconds);
const filePath = `${prefix}-${id}`;
await this.storage.set(filePath, file);
this.redis.hset(id, 'prefix', prefix);
if (meta) {
this.redis.hmset(id, meta);
}
this.redis.expire(id, expireSeconds);
}

In this snippet, we can see that Send generates an identifier for each file stored, before passing it and the file to the configured storage method. As IBC S6 has no preference to how a given object is identified, we can simply pass this identifier to IBC S6 too.

To upload the file to IBC S6, the following function was created in ionburst.js:

set(id, file) {
return new Promise((resolve, reject) => {
const putPath = path.join(this.dir, id);
const fstream = fs.createWriteStream(putPath);
file.pipe(fstream);
file.on('error', err => {
fstream.destroy(err);
});
fstream.on('error', err => {
fs.unlinkSync(putPath);
reject(err);
});
fstream.on('finish', async function() {
var upload_data = fs.readFileSync(putPath);
let put = await ionburst.putAsync({
id: id,
data: upload_data
});
console.log(put);
fs.unlink(putPath, function(error) {
if (error) {
throw error;
}
});
resolve();
});
});
}

We encountered some issues passing the file object directly to the Ionburst Cloud SDK. To overcome this, we instead leveraged the existing filesystem functionality to write the file to a temporary directory, create a read stream for the Ionburst Cloud SDK, then remove the temporary file after successful upload.

This temporary file/directory leveraged functionality used by Send's file-system storage, and it was simply a matter of pulling the temporary directory configuration into the Ionburst storage constructor:

class IonburstStorage {
constructor(config, log) {
this.log = log;
this.dir = config.file_dir;
mkdirp.sync(this.dir);
}

IBC S6 GET

Similar to the upload function, the main download functionality can be found in the storage index.js file:

async get(id) {
const filePath = await this.getPrefixedId(id);
console.log(filePath);
return this.storage.getStream(filePath);
}

To keep things simple, we replicated the same temporary file functionality for the file download from IBC S6:

async getStream(id) {
let data = await ionburst.getAsync(id);
var getPath = path.join(this.dir, id);
fs.writeFileSync(getPath, data);
var returnData = fs.createReadStream(getPath);
returnData.on('end', function() {
fs.unlink(getPath, function(error) {
if (error) {
throw error;
}
});
});
return returnData;
}

We first grab the file from IBC S6, write it to the temporary directory, then create and return a read stream.

IBC S6 DELETE

Send requires delete functionality from the configured storage method to remove uploaded files once they have reached their download limit, or expiry time.

The IBC S6 delete function was simple to implement:

del(id) {
return ionburst.delete(id, function(err, data) {
if (err) {
throw err;
}
console.log(data);
});
}

Caveats

Content-Length

At the time of integration, IBC S6 had no method of returning a stored object's size, nor did the Send metadata store it. As IBC S6 now has a HEAD API method, this can be added to the implementation to pass the Content-Length header on the file download reponse.

File Size

Depending on the deployment, Send can handle files up to 2.5GB. IBC S6 currently supports a maximum object size of 50MB, with larger objects requiring client-side processing before upload.

As a simple proof-of-concept, we've kept this 50MB limit in place for our fork of Send. However, since the time of integration, we've started to add our new SDK Manifests feature, which allows the Ionburst Cloud SDK to handle objects larger than 50MB. Once manifests have been added to our Node.js SDK, this functionality will be added to our Send fork.

Conclusion

All in all, integrating Firefox Send with IBC S6 was a relatively quick and simple process, providing us an opportunity to try out our Node.js SDK in an exisiting application. To try our Send fork for yourself, please check out our Getting started with Send and IBC S6 and Secure file-sharing with IBC S6, Firefox Send and the AWS free tier tutorials.

The full project source for our Send fork can be found here.

We'd also like to say thanks to the Mozilla team for building the original Send application, and to Tim Visée for maintaining the main Send fork.

· One min read
Josh Fraser
info

Note: This tutorial is based on the use of a new AWS account with access to 12 month free tier offers for EC2, ElastiCache and ALB.

In previous tutorials, we've looked at how to get started with Firefox Send and IBC S6, and at a deep dive on how we added IBC S6 as a back-end storage option. For this tutorial, we'll be deploying Firefox Send with IBC S6 using the AWS Free Tier, to enable secure file sharing in the cloud.

· 4 min read
Josh Fraser

Overview

As a secure object storage service, IBC S6 was designed to be integrated into both new and existing applications. Firefox Send is an open-source, secure file-sharing service that integrates cloud object stores like Amazon S3 and Google Cloud Storage for its backend storage.

We've integrated IBC S6 with Send to provide Ionburst Cloud customers the ability to easily and safely share files, while also providing an integration example for our Node.js SDK.

Shared Responsibility Model Breakdown

Customer Responsibility

  • You, the customer, are responsible for the secure management of the Ionburst Cloud credentials used by Send to connect with IBC S6.
  • You, the customer, are responsible for the security and administration of the infrastructure running the Send service.

Ionburst Cloud Responsibility

  • We are responsible for the security of all files stored in IBC S6 through the Send integration.
  • We are responsible for the underlying security and availability of the Ionburst Cloud platform.

Getting Started

In this tutorial we will cover:

  1. Setting up Send with IBC S6.
  2. Working Send and IBC S6 in development mode.
  3. Preparing Send for production.

1. Setting up Send with IBC S6

First, we need to clone the Ionburst Cloud Send project, available here.

We then need to install the Node.js dependencies:

git clone https://github.com/ionburstcloud/send.git
cd send
npm install

Once the dependencies are installed, open the Send project in your preferred IDE or editor.

2. Working Send and IBC S6 in development mode

To try out Send with IBC S6 in the Node.js development mode, we need to complete two configuration steps:

  • Edit the Ionburst SDK config.json file, adding our credentials profile and Ionburst Cloud API endpoint
  • Configure the Send backend to use IBC S6 as its backend storage
  • Start a Redis container to handle Send metadata (optional)

The config.json file can be found at the root of the Send project:

{
"Ionburst": {
"Profile": "send",
"IonburstUri": "https://api.eu-west-1.ionburst.cloud/",
"TraceCredentialsFile": "OFF"
}
}

If you have an existing Ionburst credentials profile, it can be added to the Profile key. Otherwise, add a new profile to the Ionburst credentials file with the name send.

IBC S6 can now be enabled for the Send project by editing the config.js file, found in the server directory. Update the ionburst property's default value to true to enable IBC S6 as the backend storage:

ionburst: {
format: String,
default: 'true'
},

If looking to use Redis for metadata while in development mode, you can also update the redis_host property found on line 91 from localhost to 127.0.0.1 (or the address of your Redis host). This step is optional, as Send will use its own memory store for metadata if the property is left as localhost.

To start a Redis container, the following can be used:

docker run --name send-redis -p 127.0.0.1:6379:6379/tcp -tid redis:latest

Once configured, the Send project can be launched and will be available on http://localhost:8080

npm start

3. Preparing Send for production

Preparing Send for production is a matter of reviewing and updating the config.js file, found in the server directory. Configuration items to consider are:

  • redis-host on line 91 - it's recommended to use Redis when running Send in production, rather than the built-in memory store. Update this to the address of your Redis node.
  • listen-port on line 136 - Send will run on port 1443 by default in production. It's recommended to run Send behind a load balancer or reverse proxy.
  • base-url on line 167 - change this to the url your Send instance will be available on.

Once happy with the config, a production build of Send can be created:

npm run build

If running Redis on the same host, you can start the container with:

docker run --name send-redis -p 127.0.0.1:6379:6379/tcp -tid redis:latest

You can now start the production build of Send with:

npm run prod

The Send project production build will now be available on http://localhost:1443

Conclusion

We've covered how to get the Send project up and running with IBC S6, both in development mode, and to prepare it for production deployment. If you have further questions on deploying Send with IBC S6, or integrating Ionburst SDKs with new or existing applications, drop us a line in our community Slack.