Skip to main content
Kinetic Community

Filehub

Overview

Kinetic Filehub is the main web application that is used to store and run various Filestore Adapters that can be used to access files from different systems using a single, standardized interface.  Filehub exposes a REST-like interface for retrieving, creating, updating, and deleting files from local storage, cloud storage, or any other datastore capable of handling files.

Kinetic Filehub works by executing requests that have been signed with pre-shared access credentials.  This allows integrating applications, either Kinetic apps or your own custom/homegrown applications, to use their own access control logic to determine if a user should be able to upload or download a file.  Kinetic Filehub is used heavily by Kinetic Request CE, which requires Filehub to store Attachment field questions, and may also leverage Filehub to expose internal documents, picture, or other files on forms or portal pages.

Getting Started

Installation

  1. Download the kinetic-filehub.war and place it in the tomcat webapps directory to deploy.
  2. After the kinetic-filehub.war successfully deploys, you will be able to access Kinetic Filehub at http://server:port/kinetic-filehub.
  3. Log in as admin/admin.
  4. Optionally configure the applications data directory to store configurations outside of the kinetic webapp (this will make future upgrades much easier).

Configuring a new Filestore

Filestores are instances of Filehub adapters that are configured to point to a specific datastore.  Multiple filestores can be created in Kinetic Filehub to expose different datastores with different access credentials.

  1. From the Filestores tab of the admin console click the Add Filestore button.
  2. Select the Local filestore adapter.
  3. Configure the common filestore properties:
    • Name is a friendly label used to list filestores in the admin console (example: "My Filestore").
    • Slug is a unique identifier used by integrating applications to select which filestore to connect to (example: "my-filestore").
  4. Configure the adapter specific properties (other adapters will have different properties):
    • Directory specifies the fully qualified path to where file should be read and written to (example: "/Users/wally/filestore" or "C:\Users\wally\filestore").
  5. Click the Save button.

Browsing a Filestore

The Filehub admin console provides a browser for navigating a filestore.  This browser is typically only used during integration development or for troubleshooting, and in this case can be helpful to verify that the Filestore has been set up correctly.

  1. From the Filestores tab of the admin console, click on the My Filestore name link.
  2. Use the Upload Files form to upload a test file to this directory.
  3. To open or download the file, click on the file link.
  4. To delete the, click on the Delete link.

Creating a Filestore Access Key

Filestore access keys are needed for integrating applications to sign requests to Kinetic Filehub filestores.  These keys are generated for each filestore from the admin console.

  1. From the Filestores tab of the admin console, click the Edit link for My Filestore.
  2. Click on the Add Access Key button.
  3. Enter a short Description of the access key (example: "Kinetic Request CE" or "My Integrating Application").
  4. If you are using this filestore for a real integration, copy the Key and Secret values, which will be necessary for the application to create signatures.

Setting up a Filestore for use with Kinetic Request CE Attachments

The instructions/example using this newly set up filestore to store Kinetic Request CE attachments are here.

Filestore Adapters

Filestore Adapters are implemented as Java .jar files that can be deployed with the Kinetic Filehub web application .war file.  The following adapters are included with the Kinetic Filehub web application, but additional adapters can be added to interact with other file storage solutions.  Additionally, the source code for all of the adapters included with Kinetic Filehub is available from the Kinetic Community github organization: https://github.com/KineticCommunity

BMC® Remedy® Ars Adapter

The Filehub BMC® Remedy® Ars Adapter provides the functionality necessary for Kinetic Filehub to read and write files into Ars entry file fields.

Cloud Adapter

The Filehub Cloud Adapter provides the functionality necessary for Kinetic Filehub to read and write files into common cloud services such as Amazon S3, Microsoft Azure, and Openstack Swift.

Local Adapter

The Filehub Local Adapter provides the functionality necessary for Kinetic Filehub to read and write files into local directories.  These directories may correspond to storage attached to the webserver or to shared storage mounted by the webserver.

Microsoft SharePoint Adapter

The Filehub Microsoft SharePoint Adapter provides the functionality necessary for Kinetic Filehub to read and write files into a Microsoft SharePoint Document folder.

Authoring New Filestore Adapters

Filestore Adapters are simple Java libraries containing classes that implement the com.kineticdata.filehub.adapter.FilestoreAdapter and com.kineticdata.filehub.adapter.Document interfaces.  The easiest way to get started authoring your own Filestore Adapters it to copy the source code for one of the existing adapters (the Local Adapter Github Repository is a good, simple example that includes the Netbeans project files used to develop it).

In addition to implementing the FilestoreAdapter and Document interfaces, the library must have a META-INF/adapters/com.kineticdata.filehub.adapter.FilestoreAdapter file present that includes one or more lines for what the name of the FilestoreAdapter implementing class is.  Kinetic Filehub scans the class path for these files, reads the contents, and uses them to build up the Filestore Adapter list.  Here is example contents from the LocalFilestoreAdapter:

com.kineticdata.filehub.adapters.local.LocalFilestoreAdapter

Integrating with Kinetic Filehub

In order to leverage Kinetic Filehub, an integrating application will need to generate pre-signed requests.  The Java code snippit below shows an example using the Guava library (google commons) to generate the signed URL.

/**
 * The following method returns a signed URL that is valid for 5 seconds.  The results can be used 
 * to redirect a requester to filehub to download the file.  For example:
 * 
 * 1) End user makes request to: https://acme.com/integrating-application/photos/directory/file.png
 * 2) Integrating application determines end user should have access to the file
 * 3) Integrating application redirects the request to the result of the method below
 * 4) End user displays or downloads the file from: 
 *    https://acme.com/kinetic-filehub/employee-photos/directory/file.png
 *      ?expiration=1436541747284&key=2IQ1ISG&signature=SIGNATURE&filename=wally.png
 */
public String signUrl(
    String baseUrl,          // https://acme.com/kinetic-filehub
    String filestoreSlug,    // employee-photos
    String path,             // parent-directory/subdirectory/file.png
    String filenameOverride, // wally.png
    String accessKeyId,      // 2IQ1ISG
    String accessKeySecret   // BCSPDCO7LPF2INIR87QKIBS4KH
) {
    // Calculate the expiration to be 5 seconds from now
    Long expiration = System.currentTimeMillis()+(5*1000);
    // Build the signature
    String seed = accessKeySecret+" GET "+path+"?expiration="+expiration;
    if (filenameOverride != null) {
        seed += "&filename="+filenameOverride;
    }
    String signature = com.google.common.io.BaseEncoding.base64Url().encode(
        com.google.common.hash.Hashing.sha1()
            .hashString(seed, com.google.common.base.Charsets.UTF_8).asBytes());
    // Return the redirection path
    String result = baseUrl+"/filestores/"+filestoreSlug+"/"+path+
        "?expiration="+expiration+
        "&key="+accessKeyId+
        "&signature="+signature;
    if (filenameOverride != null) {
        result += "&filename="+
            com.google.common.net.UrlEscapers.urlFormParameterEscaper().escape(filenameOverride);
    }
    return result;
}

Configuring Filehub Clustering Support with Cassandra 

Configuring Filehub clustering allows multiple Filehub instances to be running with the same exact configuration. Any time an update happens in one Filehub instance, the other Filehub instances that are running in that cluster will automatically update the configuration details that changed. This cluster support makes it easier to run a load balancer among multiple Filehub instances. This clustering is accomplished by storing the configuration information in a Cassandra database and then having each Filehub instance poll for changes that happen on other instances in the cluster.

Setting up Cassandra 

To setup Cassandra it is recommended to either use cqlsh (a command line program) or DevCenter (a GUI program). To get more information on using Cassandra with our products, view the FAQ on the preferred architecture for Request CE.

Create your keyspace 

There are multiple way to create a keyspace in Cassandra, so no one example will universally work for all environments. The following are a couple examples labeled with what situation they are used in.

Example Development Keyspace

CREATE KEYSPACE IF NOT EXISTS kinetic_filehub_development WITH REPLICATION = { 'class': 'SimpleStrategy', 'replication_factor': 1 };

Example Production Keyspace

CREATE KEYSPACE IF NOT EXISTS kinetic_filehub_production WITH REPLICATION = { 'class': 'SimpleStrategy', 'replication_factor': 3};

Use the keyspace that was created 

USE kinetic_filehub_development;

Create the tables that are needed for clustering and storing created Bridge configurations 

CREATE TABLE IF NOT EXISTS cluster_events (
    action text,
    bucket text,
    content text,
    id timeuuid,
    source_node_key text,
    PRIMARY KEY ((bucket), id)
)
WITH CLUSTERING ORDER BY (id ASC)
 AND COMPACTION={
    'sstable_size_in_mb': '256',
    'tombstone_threshold': '0.05',
    'unchecked_tombstone_compaction': 'true',
    'tombstone_compaction_interval': '3600',
    'class': 'LeveledCompactionStrategy'
};

CREATE TABLE IF NOT EXISTS cluster_configuration_items (
    bucket text,
    content text,
    key text,
    PRIMARY KEY ((bucket), key)
)
WITH CLUSTERING ORDER BY (key ASC)
 AND COMPACTION={
    'sstable_size_in_mb': '256',
    'tombstone_threshold': '0.05',
    'unchecked_tombstone_compaction': 'true',
    'tombstone_compaction_interval': '3600',
    'class': 'LeveledCompactionStrategy'
};

Setting up Tomact 

Cassandra connection details as Java System Properties

These should be specified as Java System Properties for the Tomcat instance running Kinetic Filehub

Property Name (Bold are required) Sample Value Description
com.kineticdata.filehub.clustering.contactpoints 127.0.0.1 Contact points are addresses of Cassandra nodes that the driver uses to discover the cluster topology (Read a more detailed explanation of contact points in the DataStax Documention).
com.kineticdata.filehub.clustering.keyspace kinetic_filehub_development The name of the keyspace that you previously created to store Filehub's data.
com.kineticdata.filehub.clustering.username a_user The username for the Cassandra instance.
com.kineticdata.filehub.clustering.password a_password The password for the Cassandra instance.
com.kineticdata.filehub.clustering.useSsl false true to use SSL, false if you are not.
com.kineticdata.filehub.clustering.ssl.keystore /path/to/client.keystore Fully qualified path to the keystore file.
com.kineticdata.filehub.clustering.ssl.keystore.password secret_password Password for the keystore (default blank)
com.kineticdata.filehub.clustering.ssl.truststore /path/to/client.truststore Fully qualified path to the truststore file.
com.kineticdata.filehub.clustering.ssl.truststore.password secret_password Password for the truststore (default blank)

 

Cassandra connection details in a clustering.properties file

Cassandra clustering connection details can also be defined in place of or alongside the Java System Property configuration values in a clustering.properties property file. This file should be placed in the Filehub data directory and the properties correspond to the Java System Properties with the com.kineticdata.filehub.clustering removed from the front of the property. Below is what a sample property file might look like:

contactpoints=127.0.0.1
keyspace=kinetic_filehub_development
username=a_user
password=a_password
useSsl=false
ssl.keystore=/path/to/client.keystore
ssl.keystore.password=secret_password

Restart Tomcat 

Restart any tomcat instance that java properties were added to - the changes won't take affect until restart.

Structured Logging 

 Kinetic Bridgehub v1.2.0 introduced Structured Logging into the application. Structured logging essentially is ensuring that a log file is a preset, consistent, and machine readable format. Structured logs are useful for enterprise log aggregation tools like Splunk, Graylog, or Elastic. More information about using Structured Logging in Bridgehub and Filehub can be found here.

Changelog

v1.2.1 (2018-04-26) - Download

  • Show better messaging when adapter initialization fails
  • Added Microsoft SharePoint Adapter

v1.2.0 (2017-11-15) - Download

  • Implement structured logging
  • Added support for encryption when clustering with a Cassandra database
  • Cassandra connection properties can be defined in a clustering.properties file
  • Updated Cloud Adapter to v1.0.2

v1.1.1 (2017-07-05) - Download

  • Fix local configuration issue where sensitive properties were being encrypted multiple times leading to incorrect authentication after a Filehub restart

v1.1.0 (2017-05-17) - Download

  • Add Filehub clustering support with Cassandra
  • Removed sensitive properties from API return objects
  • Filestores sorted by name in the console
  • Improved error messaging for a slug that can't be found
  • Updated BMC Remedy ARS Adapter to v1.0.1
  • Updated Cloud Adapter to v1.0.1
  • Updated Local Adapter to v1.0.1

v1.0.1 (2016-05-31) - Download

  • KFH-50 - DocumentController incorrectly applies 'filename' parameter on upload
  • KFH-49 - Document download should only use the filename override parameter for file naming if it was provided
  • KFH-43 - Improved messaging for expired signatures

v1.0.0 (2016-05-06) - Download

  • Initial Version