Skip to main content
Kinetic Community

Preparing SSL Certificates

Overview

Configuring SSL certificates with Request CE and Cassandra

Preparing SSL Certificates

SSL certificates are required when enabling either Cassandra node-node or client-node encryption. This article provides some guidelines about how to do this. Keep in mind there isn't a single correct way to create and maintain certificates, but the procedure outlined in this article provides the easiest maintainability.

Certificate Authority

Typically when using SSL certificates for a web server that is to be used by public users, the certificates must be signed by a third party certificate authority (CA) that guarantees the web server really is the server it says it is.

For the purposes of Cassandra inter-node communication or Cassandra client-server communication, this really isn't necessary. Since end-users are not directly involved with the communication, self-signed certificates are sufficient for this purpose.

A certificate authority is necessary when building a cluster because it provides a simple way to maintain trust while scaling the cluster by adding nodes, or when adding more client services such as Kinetic Request CE. The additional node or client certificates do not need to be added to the trust store, because they will be signed by the CA public key, and the CA public key will be added to the trust store.

Building a Certificate Authority

To build a certificate authority (CA), you will need a text editor and two applications:

  • Java 8 JDK
  • OpenSSL

Most of the documentation in this section is a combination of information that came from the following two articles:

The first step for creating a CA is to define a configuration file for the self-signed certificate. This is not a mandatory step, but it does provide a template if you will be generating certificates for more than one cluster. It also eliminates the manual interaction when generating the self-signed certificate.

Before getting started, please note there is a limitation in Cassandra that requires the same password for the key store as for the private key. So anytime you are asked to provide either the -keypass or the -storepass parameters, ensure you are always using the same value.

  1. Create a test file called ca-cert.conf with the following information:

    [ req ]
    default_bits            = 2048
    distinguished_name      = req_distinguished_name
    output_password         = cassandra
    prompt                  = no
    
    [ req_distinguished_name ]
    C                       = US
    ST                      = MN
    L                       = Saint Paul
    O                       = Kinetic Data
    OU                      = ProdCluster
    CN                      = ProdClusterCA
    emailAddress            = support@kineticdata.com
    

    Replace the values with appropriate data for your cluster.

  2. The next step is to create the CA using the values in the configuration file:

    openssl req -config ca-cert.conf -new -x509 -keyout ca.key.pem -out ca.cer.pem -days 365
    

    According to the OpenSSL documentation, here is what this command is doing:

    • openssl: Use the OpenSSL application
    • req: An OpenSSL sub-command that creates a self-signed certificate
    • -config: The path to the configuration file you created in the step above to avoid having to manually enter the information
    • -new: Indicates this is a new signing request
    • -x509: The certificate should be a X.509 compatible self-signed certificate that will be used as a root CA
    • -keyout: The filename to output the self-signed public certificate
    • -days: The number of days the generated certificate will be valid
  3. The last step is to add the CA public key to the trust store that will be installed on each Cassandra node, and any Cassandra API facing applications if client-server encryption is enabled. The trust store is used to verify incoming connections from the rest of the cluster, and from each API facing application.

    keytool -alias ProdClusterCA -keystore prodcluster-truststore.jks -importcert -file ca.cer.pem -keypass cassandra -storepass cassandra -noprompt
    

    Since all of the node certificates will be signed by the CA, this trust store can now be shared across all the nodes in the cluster. Since the CA is trusted, all connections whose client certificates were signed by the CA will also be trusted.

    According to the JDK 8 documentation, the keytool command used in this step can be broken down as follows:

    • keytool: use the JDK keytool utility
    • -alias: gives the certificate a name by which to reference it in the key store
    • -keystore: the name of the key store file that contains the CA public key. In this case the key store is actually a trust store with the filename prodcluster-truststore.jks
    • -importcert: instructs keytool to import the CA public key into the key store
    • -file: the file name for the CA public key
    • -keypass: the password for the CA private key, this must match the value in the configuration file output_passwordoption
    • -storepass: the password for the key store, which must match the -keypass parameter
    • -noprompt: imports the certificate without prompting for confirmation

Generating Node / Client Certificates

Now each Cassandra server node needs a certificate that is signed by the CA we created in the previous section. Also if client-server encryption is also enabled, then each client needs a certificate as well. The process is the same for server nodes and client applications, and needs to be repeated for each certificate.

  1. Generate a public certificate and private key in a keystore. The CN part of the -dname parameter is typically set as the alias value or the IP address of the node.

    keytool -genkeypair -keyalg RSA -alias node1 -keystore node1.keystore.jks -keypass cassandra -storepass cassandra -validity 365 -keysize 2048 -dname "CN=node1, OU=None, O=None, L=None, C=None"
    

    This command is similar to the one we ran when creating the trust store, except this certificate is not signed.

    • keytool: use the JDK keytool utility
    • -genkeypair: generate a public/private key pair combination
    • -keyalg: the algorithm to use, RSA
    • -alias: gives the certificate a name by which to reference it in the key store, should be the dns name or ip address of the node
    • -keystore: the name of the key store file that contains the certificate for the node. Each node should have its own key store file
    • -keypass: the password for the private key, this must match the value in the configuration file output_passwordoption
    • -storepass: the password for the key store, which must match the -keypass parameter
    • -validity: the number of days for which the key pair will be valid, starting from the time the pair was generated
    • -keysize: the number of bits to use when generating the key pair
    • -dname: information about the node and the cluster - really the only value that is important for the node certificates is the common name (CN)
  2. Export the certificate as a signing request so the CA can add its signature.

    keytool -alias node1 -keystore node1.keystore.jks -certreq -file node1.csr -keypass cassandra -storepass cassandra
    
    • -certreq: the keytool sub-command that exports the certificate in a format that can be signed by a CA
    • -file: the name of the file the signing request will be written to
    • -keypass: the password for the private key, this must match the value in the configuration file output_passwordoption
    • -storepass: the password for the key store, which must match the -keypass parameter
  3. Sign the certificate with the CA public key.

    openssl x509 -req -CA ca.cer.pem -CAkey ca.key.pem -in node1.csr -out node1.cer.signed -days 365 -CAcreateserial -passin pass:cassandra
    

    Use OpenSSL to sign the certificate. This adds the CA public key to the node certificate's public key, which allows the trust store to authenticate this certificate.

    • x509: the openssl sub-command that is used for signing
    • -req: indicates the input is a certificate signing request as opposed to a certificate
    • -CA: the CA public key file, as specified in the -out parameter used when creating the CA
    • -CAkey: the CA private key file, as specified in the -keyout parameter used when creating the CA
    • -in: The node certificate request file that needs to be signed, as specified in the -file parameter when creating the signing request
    • -out: The newly-signed certificate file to create, which contains the CA public key along with the node certificate public key
    • -days: the number of days for which the signed request will be valid
    • -CAcreateserial: creates a serial number for the CSR
    • -passin: the pass: portion of the value indicates the password for the CA private key will be passed in directly, the part after the colon is the CA private key password which was defined in the output_password option in the configuration file
  4. Add the CA to the node key store. This step is necessary for the trust chain to function correctly.

    keytool -alias ProdClusterCA -keystore node1.keystore.jks -import -file ca.cer.pem -noprompt -keypass cassandra -storepass cassandra
    
    • -alias: this is the alias for the CA, use the same value as the -alias parameter when creating the trust store
    • -keystore: the file name of the node key store file
    • -import: instructs keytool to import the public key file
    • -file: the name of the CA public key file
    • -noprompt: import the CA public key without prompting for confirmation
    • -keypass: the password for the CA private key, this must match the value in the configuration file output_passwordoption
    • -storepass: the password for the key store, which must match the -keypass parameter
  5. Add the signed node certificate to the key store.

    Once the node certificate is signed by the CA, it can be imported back into the node key store.

    keytool -alias node1 -keystore node1.keystore.jks -import -file node1.cer.signed -keypass cassandra -storepass cassandra
    
    • -alias: this is the alias for the node, use the same value used when creating the key store so the existing unsigned public key gets replaced with the signed public key
    • -keystore: the file name of the node key store file
    • -import: instructs keytool to import the public key file
    • -file: the name of the signed public key file
    • -keypass: the password for the private key, this must match the value in the configuration file output_passwordoption
    • -storepass: the password for the key store, which must match the -keypass parameter

Configuring the Cluster

In order to configure the cluster to use encryption, the following steps need to be taken on each node:

  • Install the node key store and cluster trust store
  • Configure internode encryption (Gossip API)
  • Configure client-node encryption (Native and Thrift API)
  • Configure JMX encryption

Install Certificates

The first item of business is to install the trust store, and the key store for the node onto the actual node server. It doesn't really matter where these files are installed, but file permissions and ownership are important.

A convenient location to place these files is the Cassandra configuration directory. On package installations, this location will most likely be located at /etc/cassandra/conf, and on binary archive installations, this location will most likely be located at the /conf.

  • mv /path/to/truststore.jks /path/to/truststore.jks
  • mv /path/to/node1.keystore.jks /path/to/keystore.jks

Once the files are copied to a location and named appropriately, make sure to set the ownership and file permissions:

  • Change the ownership to the user that runs the Cassandra service, by default this user is cassandra  chown cassandra:cassandra /path/to/*.jks 
  • Change the file permissions so only the file owner has read access  chmod 400 /path/to/*.jks 

Internode Encryption (Gossip API)

In a multi-node cluster, each Cassandra node communicates with peer nodes using the Gossip protocol. For non-encrypted connections, the Gossip protocol uses a TCP port defined by the following cassandra.yaml option:

storage_port: 7000

When SSL is enabled for the Gossip protocol, the following cassandra.yaml file option defines the port number used:

ssl_storage_port: 7001

All nodes in a cluster should be configured to use the same storage_port and ssl_storage_port. To prevent eavesdropping or unauthorized disruptions, the gossip protocol should be secured in production environments. However, because the protocol is used for high-performance operations such as replicating data between nodes, encryption is not recommended except for communication between remote locations.

For co-located nodes, the easiest way to secure the Gossip API is to deploy all Cassandra nodes on the same subnet and disallow access to the Gossip port from outside the subnet.

In large Cassandra deployments where multiple "racks" or “data centers” are deployed, each having some number of Cassandra nodes, the Gossip protocol can be secured for cross-rack or cross-data center communication. This is done with the following options in the cassandra.yaml file:

encryption_options:
    internode_encryption: rack
    keystore: /path/to/keystore.jks
    keystore_password: cassandra
    truststore: /path/to/truststore.jks
    truststore_password: cassandra

Internode encryption (over the Gossip API) is enabled or disabled by the setting of the internode_encryption option. The following options are recognized:

  • none: This disables all inter-node encryption, meaning Cassandra nodes use unencrypted communication using the defined storage_port.
  • all: This enables encryption for all inter-node communication using the defined ssl_storage_port.
  • rack: This uses non-encrypted communication for nodes defined to be in the same rack (cabinet) and encrypted communication between nodes defined to be in different racks.
  • dc: This uses non-encrypted communication for nodes defined to be in the same data center and encrypted communication between nodes defined to be in different data centers.

When any encryption is enabled for the Gossip protocol, all authentication, key exchange, and data transfer occurs with TLS v1 using either RSA 1024 bit keys or RSA 2048 bit keys. This encryption suite is referred to as either TLS_RSA_WITH_AES_128_CBC_SHA or TLS_RSA_WITH_AES_256_CBC_SHA. This requires that key store and trust store files are defined and initialized. These files are password-protected using the keystore_password and truststore_passwordoptions. Instructions for creating these files can be found publicly, such as in this link:http://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#CreateKeystore.

Client-Node Encryption (Native and Thrift API)

There are two application protocols for Cassandra: Thrift and CQL. Thrift is the original protocol and is not used directly by Kinetic Request. CQL is now considered the native protocol, with is what Kinetic Request uses. Both protocols support TLS for encryption.

By default, both APIs use an unencrypted connection and allow any process to connect and authenticate. To prevent unauthorized applications from directly accessing Cassandra, you can enable TLS.

The general steps for enabling TLS are described below:

  1. In the cassandra.yaml file on each Cassandra node: enable TLS by setting enabled to true underclient_encryption_options. Require client authentication by setting require_client_auth to true. When client authentication is enabled, the truststore and truststore_password options must also be set. Finally, cipher_suitesshould be set to one or more cipher suites that are accessible to the JRE. If your environment is setup to only allow AES 256 bit strong encryption, (TLS_RSA_WITH_AES_256_CBC_SHA), you will need to ensure the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files are installed into the ${java.home}/jre/lib/security/directory.

    An example of the required settings is shown below:

      client_encryption_options:
        enabled: true
        keystore: /path/to/keystore.jks
        keystore_password: cassandra
        require_client_auth: true
        truststore: /path/to/truststore.jks
        truststore_password: cassandra
        cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA]
    
  2. The next step is to install the trust store and the key store you created for the client application server. The process is the same as installing the files on the Cassandra node, except you are now installing the files onto the server that hosts the client application.

    • Install the files to the client application server, the directory you install the files to may be a bit different than on the Cassandra server.
    • Set the file ownership to the user that runs the client application instead of the cassandra user. For Kinetic Request CE running on an Apache web server, this user is typically tomcat.
  3. In the Kinetic Request CE cassandra.properties file, enable TLS by setting the cassandra.ssl property to true, and configure the trust store and key store settings. An example of these settings is shown below:

    cassandra.ssl=true
    cassandra.ssl.truststore=/path/to/truststore.jks
    cassandra.ssl.truststorePassword=cassandra
    cassandra.ssl.keystore=/path/to/keystore.jks
    cassandra.ssl.keystorePassword=cassandra
    

More information about enabling TLS for client to node encryption can be found in the Cassandra documentation:

CQLSH

If you are using the cqlsh python program and have enabled client to node encryption, you aren't done yet. What you need to do is install the public node keys on the workstation where cqlsh is run, and also install both the public and private keys for that workstation.

To accomplish this, there are a few extra steps you need to go through since cqlsh only works with x509 certificates. Foreach of the nodes in your cluster, AND the for the workstation where cqlsh will be run if that is a different workstation than any of the nodes, you need to perform the following steps on the workstation where the CA was created:

  1. Create a temporary PKCS12 keystore to work with OpenSSL

    keytool -importkeystore -srckeystore node1.keystore.jks -destkeystore node1.keystore.p12 -deststoretype PKCS12 -srcstorepass cassandra -deststorepass cassandra
    
  2. Export the public certificate in a format that works with OpenSSL

    openssl pkcs12 -in node1.keystore.p12 -nokeys -out node1.cer.pem -passin pass:cassandra
    

    The public key that needs to be copied to the cqlsh workstation is the value specified by the -out parameter, in this case node1.cer.pem.

  3. Export the private key in a format that works with OpenSSL

    openssl pkcs12 -in node1.keystore.p12 -nodes -nocerts -out node1.key.pem -passin pass:cassandra
    

    The private key that needs to be copied to the cqlsh workstation is the value specified by the -out parameter, in this case node1.key.pem.

Once all the public and private key files have been created, follow the same instructions for installing the key stores to a node. Private key files typically need to be restricted to mode 400, so that would mean if multiple user accounts will be using cqlsh, it may be necessary to install the keys in multiple locations, one for each user account.

Now a cqlshrc configuration file needs to be modified, or created if it doesn't yet exist. The location of this file is usually in the ${user.home}/.cassandra directory.

An example ${user.home}/.cassandra/cqlshrc file would look something like the following: ``` [connection] hostname = 127.0.0.1 port = 9042 factory = cqlshlib.ssl.ssl_transport_factory

[ssl]
userkey = ~/.cassandra/workstation.key.pem
usercert = ~/.cassandra/workstation.cer.pem

[certfiles]
node1.ip.address = ~/.cassandra/node1.cer.pem
node2.ip.address = ~/.cassandra/node2.cer.pem
node3.ip.address = ~/.cassandra/node3.cer.pem
```

Substitute the node IP addresses in place of the node1.ip.address, etc... values in the [certfiles] section of the above example.

To connect to a node with cqlsh, you now need to pass the --ssl argument when running the program, as well as the IP address of the node you are wanting to connect to:  cqlsh --ssl 111.111.111.111 

If your environment is setup to only allow AES 256 bit strong encryption, (TLS_RSA_WITH_AES_256_CBC_SHA), you will need to ensure the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files files are installed into the${java.home}/jre/lib/security/ directory.

Additional Resources

In addition to the articles previously linked in this document, the following articles and pages were also helpful in providing information in this article: