As an enterprise level NoSQL database software, Apache Cassandra provides many out-of-the-box security features that fall into the following categories:
- Cassandra Authentication
- Cassandra Authorization
- At-rest Cassandra Data Encryption
- In-transit Cassandra Data Encryption
- Secure Cassandra JMX Access
Cassandra 3.x has made some improvement in these areas. In this blog post, I will go through the available security features in Cassandra version 3.9, one by one. All these features are tested in a CCM based 3-node cluster deployed in a VMWare-based Ubuntu 16.04 virtual machine.
In the discussion below, unless the version is explicitly stated, when a Cassandra feature is mentioned, it refers to the feature available in Cassandra version 3.9. Along with the discussion, when necessary, I will also explain in deeper details of the underlying mechanisms on which these security features are based.
Please note that within different versions of Cassandra, there might be some difference around available features and configuration details. Please refer to the official documentation that matches your own Cassandra version.
Cassandra has provided simple user and permission management since its early days (e.g. CASSANDRA-547). Since version 1.2.2, Cassandra starts to provide internal authentication and authorization through CQL. From Cassandra 2.2 (CASSANDRA-7653, as part of a broader Cassandra Auth change by CASSANDRA-8394), a more fundamental and flexible improvement regarding Cassandra access control is implemented in Cassandra as a core security functionality, which is role based access control (RBAC).
1.1 Introduction to Cassandra RBAC
Within Cassandra RBAC, the concept of “ROLE” replaces the concept of “USER” as used in earlier Cassandra versions and it becomes the core part of both authentication and authorization. A “ROLE” can either refer to a single user or a group of users that share the same access permission needs.
With proper permission (e.g. as a superuser), a role can be created, altered, dropped, and listed through CQL statements, the syntax of which is as below:
CREATE ROLE [ IF NOT EXISTS ]
WITH [ PASSWORD = [ AND
LOGIN = [ AND
SUPERUSER = [ AND
OPTIONS = Note that when a ROLE is created, it by default does not have “LOGIN” privileges and “SUPERUSER” status.
At the moment, Cassandra only supports internal authentication feature. External authentication mechanism such as through 3rd party authentication/authorization tool (e.g. LDAP) is not supported yet.
When internal authentication is configured (more on this below), a ROLE created with “LOGIN” privilege can be authenticated to access Cassandra using the password as specified in the “CREATE ROLE” statement. For backward compatibility purpose, the concept of “USER” also remains. A USER is equivalent to a ROLE with “LOGIN” privilege and can be created, altered, dropped, and listed using the following CQL statements:
CREATE USER [ IF NOT EXISTS ]
[ WITH PASSWORD ] [ SUPERUSER | NOSUPERUSER ]
ALTER USER
[ WITH PASSWORD ] [ SUPERUSER | NOSUPERUSER ]
DROP USER [ IF EXISTS ]
LIST USERS
2.1. Configure Cassandra Internal Authentication
Cassandra internal authentication is configured in cassandra.yaml file through the following setting. The default value of this setting is “AllowAllAuthenticator” which doesn’t enforce authentication at all. Everyone who has access to the underlying host of a Cassandra node can access Cassandra data.
authenticator: PasswordAuthenticator
Once enabled, a ROLE with “LOGIN” privilege is needed. By default, a ROLE with “LOGIN” privilege and “SUPERUSER” status, called “cassandra” (with password “cassandra”), can be used to connect to the authentication-enabled node to create other ROLEs, as below:
Once the new ROLE is created, you can use it to access Cassandra with the defined password. Please note that for safety purpose, it is suggested to create your own ROLE with “SUPERUSER” privilege and “LOGIN” status and drop the default “cassandra” ROLE (or at least changes its password).
2.2. Things to Pay Attention To
First, the configuration change is made on each node. In my test, I enabled internal authentication on node1 and node2, but NOT on node3. This leads to the situation that I can log in CQLSH on node3 without any constraint, which creates a security hole in the system.
In order to avoid this problem, when enabling internal authentication in a Cassandra cluster, the change has to be made on every single node in the cluster.
Second, with the default, out-of-the-box CassandraRoleManager implementation, the access control related information is stored in Cassandra’s system_auth keyspace. By default, the replication setting of this keyspace is “SimpleStrategy” with replication factor 1. In this case, if we lose the node on which the ROLE was created, other nodes that have internal authentication enabled will have trouble connecting to Cassandra. To test this out, I shutdown node1 and try to connect to CQLSH on node2 using the newly created ROLE, john. The connection is refused:
In order to solve this issue, we should change the replicate factor of system_auth keyspace to multiple nodes (I would recommend to change to ALL nodes in the cluster). After changing the replication factor, it is recommended to run “nodetool repair system_auth” command to bring all nodes in sync right away.
Once these changes are made, node2 can connect to CQLSH successfully using ROLE john, even if node1 is down.
2.3. Caching Roles and Credentials
In order save the cost associated with fetching role authentication, the following configuration items can be set in “cassandra.yaml” file to enable role/credential caching behavior:
roles_validity_in_ms:
* validity period for role cache
* default value 2000 (0 to disable)
* disabled automatically with "AllowAllAuthenticator"
roles_update_interval_in_ms:
* refresh interval for role cache
* default to the value of "roles_validity_in_ms"
credentials_validity_in_ms:
* validity period of a stored credential in cache (in encrypted form)
* default value 2000 (0 to disable)
credentials_update_interval_in_ms:
* refresh interval for credential cache
* default to the value of "credentials_validity_in_ms"
Like Authentication, currently Cassandra only supports internal authorization feature. External authorization mechanism such as through 3rd party tool (e.g. LDAP) is not supported yet.
In Cassandra, permissions on database resources are granted to ROLEs.
3.1. Resource Permission Management
The full list of permissions are:
CREATE
ALTER
DROP
SELECT
MODIFY
AUTHORIZE
DESCRIBE
EXECUTE
The resources are organized in hierarchies of the following categories:
ALL KEYSPACES -> KEYSPACE -> TABLE
ALL FUNCTIONS -> ALL FUNCTIONS IN KEYSPACE -> FUNCTION
ALL ROLES -> ROLE
ALL MBEANS -> MBEAN (Cassandra 3.6 and later)
One ROLE with enough privilege (with “SUPERUSER” status or “GRANT/REVOKE” privilege) can grant/revoke certain permission to/from a specified ROLE on a particular resource. The CQL statements of doing so is as below:
GRANT ALL | ON TO
REVOKE ALL | ON FROM
** and are from the 2 lists above.
Please note that:
- NOT all permissions are applicable to every resource type. For example, “EXECUTE” permission can only be applied to MBEAN resources. Applying a permission to an incompatible resource will lead to the following error.
SyntaxException: Resource type DataResource does not support any of the requested permissions - When “GRANT ALL ON “resource_name” TO “role_name” ” format is used, all permissions that are compatible with the specified resource is automatically determined.
- When a resource (e.g. keyspace, table, function, etc.) is created, the creator ROLE is automatically granted all compatible permissions on this resource.
- Permission inheritance or hierarchical permission structure can be achieved by granting a ROLE to another ROLE. Please be aware that in this hierarchy, only the parent ROLE’s permissions and SUPERUSER status are inherited, but not its LOGIN status.
3.2. Configure Cassandra Internal Authorization
Cassandra internal authentication is configured in cassandra.yaml file through the following setting. The default value of this setting is “AllowAllAuthorizer” which doesn’t enforce authorization at all. Cassandra allows any action on any resource.
authenticator: CassandraAuthorizer
Once enabled, a ROLE with SUPERUSER status or ‘AUTHORIZE’ permission can be used to grant permissions on resources to other ROLEs.
3.3. Caching Permissions
In order save the cost associated with fetching permission, the following configuration items can be set in “cassandra.yaml” file to enable permission caching behavior:
permissions_validity_in_ms:
* validity period for permission cache
* default value 2000 (0 to disable)
* disabled automatically with "AllowAllAuthorizer"
permissions_update_interval_in_ms:
* refresh interval for role cache
* default to the value of "permissions_validity_in_ms"
From version 3.2, Cassandra starts to gradually add support for at-rest data encryption through Transparent Data Encryption (TDE). As of version 3.9, TDE encryption for on-disk commitlog and hints are supported. But the support for on-disk SSTable related files (Data, Index, Summary, Filter, and etc.) is not in place yet.
The following JIRA tickets lists the efforts so far to enable TDE encryption for Cassandra:
- CASSANDRA-9945: Add transparent data encryption core classes (Resolved, v3.2)
- CASSANDRA-6018: Add option to encrypt commitlog (Resolved, v3.4)
- CASSANDRA-11040: Encrypted hints (Resolved, v3.4)
- CASSANDRA-9633: Add ability to encrypt sstables (Unresolved)
4.1. Configure TDE Encryption for Cassandra
CASSANDRA-9945 introduced a section in “cassandra.yaml” file for TDE encryption setup, as below (copied from the default Cassandra 3.9 cassandra.yaml file).
transparent_data_encryption_options:
enabled: false
chunk_length_kb: 64
cipher: AES/CBC/PKCS5Padding
key_alias: testing:1
# CBC IV length for AES needs to be 16 bytes
# (which is also the default size)
# iv_length: 16
key_provider:
- class_name: org.apache.cassandra.security.JKSKeyProvider
parameters:
- keystore: conf/.keystore
keystore_password: cassandra
store_type: JCEKS
key_password: cassandra
By default, TDE encryption is disabled (“enabled” setting is false).
When enabled, Cassandra’s out-of-the-box key provider (JKSKeyProvider) reads the key from a Java Cryptography Extension (JCE)- style keystore that you can specify the following properties in cassandra.yaml:
- store_type: keystore type (e.g. JCEKS/JCE/PK12/etc.)
- keystore: keystore location/name (e.g. full local path)
- keystore_password: the password to access the keystore
- key_password: the password to access the key
The encryption algorithm type is also specified through property: cipher
4.2. JCE Unlimited Strength Policy
JCEKS means Java Cryptography Extension Keystore and it is an alternative key store format on Java platform. A keystore contains individual keys or certificates and can protect them from being exposed.
The strength of encryption is enforced by a JCE strength policy file that meets the requirements from the laws of particular countries. Due to import regulations in some countries, the default JCE strength policy file provided by Oracle Java implementation has an upper limit of maximum key sizes, as below:
Algorithm | Max. Keysize
-------------|-----------------
DES | 64
DESede | 64
RC2 | 128
RC4 | 128
RC5 | 128
RSA | 128
all others | 128
If stronger encryption, such as Advanced Encryption Standard (AES) with maximum key size greater than 128-bit(e.g. 256), JCE Unlimited Strength Policy is needed. This is also highly recommended by Cassandra.
The installation of JCE Unlimited Strength Policy is quite simple:
1). Download the policy file (in zip format) for your java version (e.g. for Java 8, the file can be downloaded from here)
2). Unzip the zipped file
3). Copy the unzipped local_policy.jar and US_export_policy.jar files to location: $JAVA_HOME/jre/lib/security (e.g. /usr/lib/jvm/java-8-oracle/jre/lib/security). Please note that in that location, the 2 files should have already existed as the default policy file. You can make a copy of them before overwriting them with the new policy file.
4.3. Generate Key and Keystore for TDE Encryption
In Java, key stores are often created using the “keytool” utility provided with Java SDK. The description of how to use this tool to generate keys, certificates, and key stores are beyond the scope of this document. Please refer to Oracle Java documentation for more information.
For example, the following command creates a key, aliased as “cass_node1”, using AES-256 encryption and the key is stored in a JCEKS-typed keystore named as “.keystore”. Both the key and the keystore are protected with the same password “john123”.
keytool -genseckey -alias cass_node1 -keyalg AES -keystore .keystore
-keysize 256 -storetype JCEKS -storepass john123 -keypass john123
Using the above information, we can enable Cassandra TDE encryption (AES-256) like this:
transparent_data_encryption_options:
chunk_length_kb: 64
cipher: AES/CBC/PKCS5Padding
enabled: true
key_alias: cass_node1
iv_length: 16
key_provider:
- class_name: org.apache.cassandra.security.JKSKeyProvider
parameters:
- key_password: john123
keystore: /.keystore
keystore_password: john123
store_type: JCEKS
After restarting Cassandra service, the commitlog and hints files on this node are AES-256 encrypted.
In Cassandra, in-transit data encryption is achieved through Security Socket Layer (SSL) protocol and it has been supported since very early release (0.8) of Cassandra. 2-Way SSL certificate Authentication is available in 1.2.3 (more on this in section 5.1).
Security Socket Layer and its successor, Transport Layer Security (TLS), are commonly known under the same term, TLS/SSL, or simply just as SSL, are cryptographic protocols designed to provide confidentiality, data integrity, identification, and authentication using digital certificates. In Cassandra, when SSL encryption is enabled, TLS is the default protocol (more on this in section 5.2)
5.1. 1-Way and 2-Way SSL Authentication
In the context of SSL, two entities are needed for one SSL communication session. The entity that initiates the request can be called as SSL-client and the other entity that provides the resource being requested can be called as SSL-server. Please don’t get this confused with a Cassandra client and a Cassandra server. In the context of Cassandra node-to-node communication, the Cassandra server on any node can be either as an SSL-client or an SSL-server.
In 1-Way SSL Authentication, only SSL-server is verified when SSL-client requests for a resource provided by SSL-server. SSL-server doesn’t verify the identify of SSL-client.
In 2-Way SSL Authentication, also called Manual or Client SSL Authentication, when a SSL-client requests for a protected resource on SSL-server side, SSL-server identify is first verified and then SSL-server verifies the identify of SSL-client as well.
The identity verification in this process can be summarized in high level as below. In 1-Way SSL Authentication, only steps 1~3 and 6 are involved. In 2-way SSL Authentication, the complete 6 steps are involved.
1). A client requests access to a protected resource.
2). The server presents its certificate to the client.
3). The client verifies the server’s certificate.
4). If successful, the client sends its certificate to the server.
5). The server verifies the client’s credentials.
6). If successful, the server grants the client’s access to the protected resource.
Compared with 1-Way SSL Authentication, 2-Way SSL Authentication provides better security and identification flexibility.
5.2. Prepare SSL Certificate, Keystore, and Truststore
Preparing SSL certification, Keystore, and Truststore is a standard security operation procedure. From operation perspective, when executing this procedure for a cluster of nodes, there are two different approaches, as described below in very high level.
Approach 1: Copy the public part of each node's certificate to all other nodes. On each node, import all copied parts (from all nodes, including the node itself) into the Truststore of that node. Approach 2: Use a Certificate Authority (CA)'s root certificate to sign each node's certificate. Import the root certificate and the signed certificate into the Keystore of each node. Use the root certificate to build a common Truststore and copy the common Truststore to each node.
When the number of nodes in the cluster increase, the operation overhead of approach 1 quickly becomes unmanageable. So unless for a very small cluster and/or for development purpose, approach 2 is always preferred.
DataStax’s document for Cassandra 3.x provides a good description of how to prepare SSL Certificates, Keystore, and Truststore for both approaches. You can find the detail instructions for approach 1 here and for approach 2 here.
Once this step is done, it becomes easy to configure Cassandra node-to-node and client-to-node encryption.
5.3. Configure Cassandra Node-to-Node Encryption
Node-to-node encryption is enabled by making the following “server_encryption_options” settings in “cassandra.yaml” file.
server_encryption_options:
internode_encryption: [none|all|dc]
keystore: /to/keystore>
keystore_password:
truststore: /to/truststore>
truststore_password:
# More advanced defaults below:
protocol: TLS
algorithm: SunX509
store_type: JKS
cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
require_client_auth: true
require_endpoint_verification: true
Most of these settings are straightforward to understand except 2 points:
1). If 2-Way SSL Authenticate is needed, “require_client_auth” needs to be set as true.
2). If a CA is used to build the trust chain, A type of Man in the Middle (MITM) attack can happen even with TLS/SSL communication (see here) for more background info. If the trust chain is built through simply importing all node certificates into each Truststore, this problem does not exist.
In order to address this issue, Hostname Verification is needed. Cassandra version 3.6 introduces the support for it through CASSANDRA-9220. The “require_endpoint_verification” setting in node-to-node encryption options is used for this purpose.
5.4. Configure Cassandra Client-to-Node Encryption
Client-to-node encryption is enabled by making the following “client_encryption_options” settings in “cassandra.yaml” file.
# enable client/server encryption.
client_encryption_options:
enabled: true
# If enabled and optional is set to true encrypted and unencrypted connections are handled.
optional: false
keystore: /to/keystore>
keystore_password:
require_client_auth: true
# Set trustore and truststore_password if require_client_auth is true
truststore: /to/truststore>
truststore_password:
# More advanced defaults below:
protocol: TLS
algorithm: SunX509
store_type: JKS
cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
Similarly, there are several points worthy a few more words:
1). When “require_client_auth” is set to true, 2-Way SSL Authentication is enabled and “truststoe” and “trustore_password” settings have to be set with the proper values.
2). When client-to-node encryption is enabled (“enabled” is set to true), “optional” setting determines whether or not unecrypted traffic is allowed along with encrypted traffic. “optional” being set to true allows both uncrypted and encrypted traffic.
3). Starting from Cassandra version 3.0, an additional dedicated port (“native_transport_port_ssl”, default to 9142) is added in “cassandra.yaml” file for encrypted traffic. If it is not set, the original native transport port (“native_transport_port”, default to 9042) is used for encrypted traffic.
When both unecrypted and encrypted traffic are allowed (“optional” setting to true), “native_transport_port_ssl” will be used for encrypted traffic and “native_transport_port” will be used for unencrypted traffic.
4). The MITM problem for client-to-node communication is addressed in client drivers. For example, JAVA-841 is the solution in Java driver.
For Cassandra, JMX access control is achieved through the following code blocks in “cassandra-env.sh” file.
if [ "x$LOCAL_JMX" = "x" ]; then
LOCAL_JMX=yes
fi
JMX_PORT="7100"
if [ "$LOCAL_JMX" = "yes" ]; then
.... ....
else
... ...
fi
By default, “LOCAL_JMX” system variable is “yes” and Cassandra JMX authentication is disabled via the following setting:
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false"
In order to enable JMX Authentication/Authorization for Cassandra, the system variable “LOCAL_JMX” needs to change to “no” (or any value not equal to “yes”). This change makes sure JMX authentication is enabled:
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
After doing this, there are two different ways to achieve secure JMX access depending on the version of Cassandra:
- Before Cassandra version 3.6, file-based Password Authentication is used to authenticate/authorize JMX access.
- Starting from Cassandra version 3.6, JMX authentication/authorization can be delegated to Cassandra internal authentication and authorization as we discussed in sections 2 and 3. This approach is based on Java Authentication and Authorization Service (JAAS) technology.In this section, we’ll go through the configuration of both approaches.
6.1. Configure File-based Password Authentication/Authorization
In order to enable file-based Password Authentication, the following key settings need to be set in “cassandra-env.sh” file:
// Specify password file
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file="
// Specify access control file.
// If this line is not included, the default JRE access control file is then verified
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.access.file="
The password file is used to define different roles and their passwords and the access control file is used to define the permitted access for each role. By default, JRE implementation includes a default password template file and a default access control file under folder “<$JRE_HOME>/lib/management/”: jmxremote.password.template and jmxremote.access.
In order to enable file-based Password Authentication for Cassandra, we can follow the steps below:
- 1. Create your own password file by copying from the JRE template password file (e.g. /etc/cassandra/jmxremote.password)
cp /lib/management/jmxremote.password.template /etc/cassandra/jmxremote.password
- 2. Add (JMX) roles for Cassandra JMX access and their passwords in your own password file, e.g.
cassandra
- 3. Because the file contains plain-text password, please make sure that the ownership of this file is changed to “cassandra” service user and the file permission is read-only (this assumes that the Cassandra is running under the service user named “cassandra”).
chown cassandra:cassandra /etc/cassandra/jmxremote.password chmod 400 /etc/cassandra/jmxremote.password
- 4. You can either edit the default JRE access control file or copy it to a new file in a different location. Grant read/write JMX access privilege for the roles (e.g. “cassandra”) that are added in the password file.
cassandra readwrite
- 5. Restart Cassandra service
6.2. Configure JAAS-based Internal Authentication/Authorization
In order to use JAAS-based JMX authentication/authorization through Cassandra internal authentication/authorization,
- 1. We have to make sure that the file-based Password Authentication is disabled, by commenting out the following two lines in “cassandra-env.sh” file.
#JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=" #JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.access.file="
- 2. Make sure Cassandra internal authentication is enabled, as per the discussion in section 2. Enable the following settings in “cassandra-env.sh” file for JMX authentication.
JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.remote.login.config=CassandraLogin" JVM_OPTS="$JVM_OPTS -Djava.security.auth.login.config=$CASSANDRA_HOME/conf/cassandra-jaas.config"
By default, Cassandra (version 3.6 and later) ships with a JAAS login module, CassandraLoginModule, as specified in the JASS login module configuration file, “cassandra-jass.config”. This module delegates the JMX authentication request to the authenticator as defined in “cassandra.yaml” file (e.g. PasswordAuthenticator)
// Delegates authentication to Cassandra configured IAuthenticator CassandraLogin { org.apache.cassandra.auth.CassandraLoginModule REQUIRED; }; - 3. Make sure Cassandra internal authorization is enabled, as per the discussion in section 3. Enable the following settings in “cassandra-env.sh” file for JMX authorization on Cassandra MBean resources.
JVM_OPTS="$JVM_OPTS -Dcassandra.jmx.authorizer=org.apache.cassandra.auth.jmx.AuthorizationProxy"
6.3. Secure JMX Communication with SSL
In production deployment, when JMX remote authentication is enabled, it is recommended to secure JMX communication with SSL, by enabling the following options in “cassandra-env.sh” file
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=true"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.need.client.auth=true"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.protocols="
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.enabled.cipher.suites="
JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStore="
JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStorePassword="
JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStore="
JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStorePassword="
In this blog post, I went through the security features that are available in the latest Cassandra 3.x release (3.9 as the time of the writing). Compared with earlier releases (2.1 and before), Cassandra 3.x has gradually introduced many security improvements and new features, such as Role-based Access Control, At-rest Data Encryption, JMX Authentication Delegation, and etc. This blog post also aims to provide hands-on guidance on how these security features are configured in Cassandra 3.9, while providing enough underlying background information at the same time.
Want to talk with an expert? Schedule a call with our team to get the conversation started.
