You can implement multiple Content Services instances in a clustered environment.
A cluster represents a collection of nodes. Clustering is implemented to provide high scalability and resilience. Improved performance is enhanced through redundant nodes that provide services when other nodes fail. When integrated with a load balancer, performance is enhanced by distributing, or balancing, server workload across a collection of nodes.
There are a number of prerequisites for upgrading from a version of Content Services prior to 4.2 in a clustered environment.
Before upgrading, ensure that all files and configuration are backed up. Any customization(s) that you’ve made, for example, creation of custom caches, might need to be reapplied using the new Content Services clustering infrastructure.
The following libraries are no longer used in Content Services 4.2 onwards, so any configuration related to these libraries should be removed before upgrading:
Note: You do not need to follow these steps if you’re upgrading from Content Services 4.2. This information is only relevant if you’re upgrading from any version prior to 4.2.
Follow the steps to remove the configuration not supported in version 7.0:
Browse to the <classpathRoot> directory.
For example, for Tomcat 7, browse to the $TOMCAT_HOME/shared/classes/alfresco/extension/ directory.
Delete theehcache-custom.xml file.
Browse to the <classpathRoot> directory.
For example, for Tomcat 7, browse to the $TOMCAT_HOME/shared/classes/ directory.
Open the alfresco-global.properties file.
Remove the following legacy properties from thealfresco-global.properties file:
alfresco.ehcache.rmi.hostnamealfresco.ehcache.rmi.portalfresco.ehcache.rmi.remoteObjectPortalfresco.jgroups.defaultProtocolalfresco.jgroups.bind_addressalfresco.jgroups.bind_interfacealfresco.tcp.start_portalfresco.tcp.initial_hostsalfresco.tcp.port_rangealfresco.udp.mcast_addralfresco.udp.mcast_portalfresco.udp.ip_ttlfilesystem.cluster.enabledfilesystem.cluster.configFileBrowse to the <classpathRoot> directory.
For example, for Tomcat 7, browse to the $TOMCAT_HOME/shared/classes/alfresco/extension directory.
Remove the Hazelcast configuration file,hazelcastConfig.xml, as a centralized configuration is now included within thealfresco.war deployment archive.
Thefilesystem.cluster.configFile property mentioned in Step 5 refers to thehazelcastConfig.xml file.
After you’ve performed all the specified steps, if you want to initiate clustering, seeSetting up repository server cluster.
This information gives an overview of the main components of your Content Services solution.
Within your solution, you have components such as Alfresco Share, Content Services, database, index (Solr), transformations, and the content store. As illustrated in the following diagram, some of these components can be clustered while others are optional.

Clusterable components:
Non-clusterable and replicable components:
It is important to decide how many installations you need and which of the specified components you’ll put on which node.
To explain this further, let’s consider an example. Assume that your application has six nodes. Ideally, it’s recommended that you should have one component on each node. So, for example, node1 has database, node2 is content store, node3 is Content Services, node4 is Alfresco Share, node5 is Solr, and node6 is Transformation Server.
Use the following table as a template to design your solution.
| Clusters/ nodes | Alfresco components | DNS | IP address (optional) | Additional information |
|---|---|---|---|---|
| 1 | Database | |||
| 2 | Content store | |||
| 3 | Content Services | |||
| 4 | Alfresco Share | |||
| 5 | Solr 6 | |||
| 6 | Transformation Server |
However, you can configure your distribution differently. Each distribution and clustering solution has its own advantages and disadvantages. To configure distribution and clustering optimally, contactAlfresco Consulting or your Alfresco certified partner.
For a better understanding of initiating clustering for high availability and high throughput, seeScenario: Clustering for redundancy andScenario: Clustering for high throughput.
There are a number of recommendations for splitting the Content Services architecture in a distributed or clustered environment.
Generally, there are two complementary purposes for distributing or clustering your installation.
The main decision is involved around when to split and how to split.
When to split: There are a number of indicators to help you decide when to split your architecture from a single node environment to a distributed node environment. Some of the indicators to look for include:
How to split: When you’ve decided to upgrade from a single node environment to a distributed or clustered environment, you must find the most appropriate way to cluster architecture.
Consider the following scenarios for setting up a cluster and installing Solr in a cluster.
This is a scenario-based topic describing the clustering architecture for redundancy and high availability of Content Services services.
This scenario shows a single repository database and content store, and two Tomcat nodes/web servers on two separate machines accessing the content simultaneously. The configuration does not guard against the content store or database failure, but allows multiple servers to share the web load, and provides redundancy in case of a server failure. Each server has local indexes (in the local content store).
This is the simplest cluster to set up and is ideal for small-scale installations. A cluster consisting of two or more machines working together provides a higher level of availability, reliability, and scalability than can be obtained from a single node.

A hardware load balancer balances the web requests among multiple servers. The load balancer must support ‘sticky’ sessions so that each client always connects to the same server during the session. The content store and database will reside on separate servers, which allows us to use alternative means for content store and database replication.
Note: All the servers in a cluster should have static IP addresses assigned to them.
This is a scenario-based topic describing the clustering architecture for maximizing throughput of Content Services services.
This setup shows a single repository database and content store. There are four nodes with Content Services/Alfresco Share and two nodes with Solr, all accessing the content simultaneously. This set up provides a higher level of availability, reliability, and scalability, thereby maximizing the throughput of various services. Nodes in a cluster are positioned behind a load balancer that delegates requests to cluster members based on any one member’s ability/availability to handle the load.
Each Share instance is deployed into its own Tomcat servlet container. Content Services services and CPU runtime footprint are optimized for high throughput under heavy concurrency with such a deployment. The load balancer fronts the cluster, and directs traffic to the member of the cluster most able to handle the current request.
Note: All the servers in a cluster should have static IP addresses assigned to them.

In this deployment scenario the following flows are present:
JSESSIONID cookie to stick the client to one of Share nodes.Use this information to configure a cluster for Share.
This information describes the configuration of Hazelcast clustering between instances of Share.
In a load balanced environment, Share now uses Hazelcast to provide multicast messaging between the web-tier nodes. As a result, Share caches no longer need to be disabled for any node, simple cache invalidation message are sent to all nodes when appropriate. Each node functions practically as fast as a single Share instance, enhancing the overall performance of Share.
To enable Hazelcast clustering between Share instances, configure thecustom-slingshot-application-context.xml file found at<TOMCAT-HOME>/shared/classes/alfresco/web-extension. This file is used to override the Spring application context beans for Share.
Note: An example
custom-slingshot-application-context.xml.samplefile is provided in the distribution, which now includes this configuration.
To enable the Hazelcast cluster messaging, edit this section on each Share Tomcat instance:
<?xml version='1.0' encoding='UTF-8'?><beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:hz="http://www.hazelcast.com/schema/spring" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd http://www.hazelcast.com/schema/spring http://www.hazelcast.com/schema/spring/hazelcast-spring-2.4.xsd"> <!-- Hazelcast distributed messaging configuration - Share web-tier cluster config - see http://www.hazelcast.com/docs.jsp - and specifically http://docs.hazelcast.org/docs/2.4/manual/html-single/#SpringIntegration --> <!-- Configure cluster to use either Multicast or direct TCP-IP messaging - multicast is default --> <!-- Optionally specify network interfaces - server machines likely to have more than one interface --> <!-- The messaging topic - the "name" is also used by the persister config below --> <hz:topic instance-ref="webframework.cluster.slingshot" name="slingshot-topic"/> <hz:hazelcast> <hz:config> <hz:group name="slingshot" password="alfresco"/> <hz:network port="5801" port-auto-increment="true"> <hz:join> <hz:multicast enabled="true" multicast-group="224.2.2.5" multicast-port="54327"/> <hz:tcp-ip enabled="false"> <hz:members></hz:members> </hz:tcp-ip> </hz:join> <hz:interfaces enabled="false"> <hz:interface>192.168.1.*</hz:interface> </hz:interfaces> </hz:network> </hz:config> </hz:hazelcast> <bean init-method="init"> <property name="hazelcastInstance" ref="webframework.cluster.slingshot" /> <property name="hazelcastTopicName"><value>slingshot-topic</value></property> </bean></beans>This configuration enables the Hazelcast Spring integration, which in turn, starts the Hazelcast server. The Hazelcast server is easily configurable and can use either multicast (default) or TCP-IP direct, if preferred. For more information, see theHazelcast Documentation{target=”_blank”}.
If this configuration is enabled, the Share instance becomes a cluster node and Hazelcast is started. If this configuration is disabled (such as, for a default install), then Hazelcast is not started. While using Share, only when any of the following actions occur, the cache invalidation messages will be sent from the affected node to other nodes in the cluster:
Note: For activating the default set up, apply identical configuration to each Share node.
The following is a sample output that you get when you start Share:
INFO: /127.0.0.1]:5801 [slingshot] Hazelcast Community Edition 2.4 (20121017) starting at Address[127.0.0.1]:5801Dec 13, 2014 12:09:36 PM com.hazelcast.systemINFO: /127.0.0.1]:5801 [slingshot] Copyright (C) 2008-2012 Hazelcast.comDec 13, 2014 12:09:36 PM com.hazelcast.impl.LifecycleServiceImplINFO: /127.0.0.1]:5801 [slingshot] Address[127.0.0.1]:5801 is STARTINGDec 13, 2014 12:09:37 PM com.hazelcast.impl.TcpIpJoinerINFO: /127.0.0.1]:5801 [slingshot] Connecting to possible member: Address[127.0.0.1]:5802Dec 13, 2014 12:09:37 PM com.hazelcast.impl.TcpIpJoinerINFO: /127.0.0.1]:5801 [slingshot] Connecting to possible member: Address[127.0.0.1]:5803Dec 13, 2014 12:09:37 PM com.hazelcast.nio.SocketConnectorINFO: /127.0.0.1]:5801 [slingshot]Members [1] { Member [127.0.0.1]:5801 this}Dec 13, 2014 12:09:38 PM com.hazelcast.impl.management.ManagementCenterServiceINFO: /127.0.0.1]:5801 [slingshot] Hazelcast Management Center started at port 5901.Dec 13, 2014 12:09:38 com.hazelcast.impl.LifecycleServiceImplINFO: /127.0.0.1]:5801 [slingshot] Address[127.0.0.1]:5801 is STARTEDThe message shows that the configuration has successfully initialized Hazelcast between Share instances.
These steps are required for cluster configuration for Share. If you’re using an HTTP load-balancing mechanism in front of a clustered installation, ‘sticky’ routing must be enabled for the HTTP requests made by the Share tier to the repository tier (the/alfresco application).
This can be achieved in one of two ways:
Hard-wire each /share instance to its own/alfresco instance, bypassing the load balancer.
This can be achieved by populating eachshare-config-custom.xml file with a host name and port number that is not behind your load balancing mechanism.
If Kerberos authentication is enabled with SSO, then Share will use cookie-based sessions and you can configure your load balancer to use sticky routing using theJSESSIONID cookie.
To enable Kerberos with SSO, refer to the instructions inConfiguring authentication to configurealfrescoNtlm or Kerberos authentication, and set (kerberos.authentication.sso.enabled=true).
Note: If you’re configuring a cluster, refer toSetting up clustering.
Use this information to implement a repository server cluster.
The repository server cluster consists of the following components:
Use this information to install and configure nodes in a cluster based on the scenario described in Clustering for high throughput.
Install the Content Services nodes.
SeeInstalling using distribution zip for more information.
Set up multiple Content Services instances in a clustered environment.
SeeSetting up clustering for more information.
This topic describes the instructions for installing and configuring Solr nodes in a cluster.
Configure Solr nodes.
SeeInstalling Alfresco Search Services for more information.
Edit the<SOLR_HOME>/solrhome/templates/rerank/conf/solrcore.properties file.
If you’ve already started Solr and the cores (alfresco andarchive) already exist, open the following files:
<SOLR_HOME>/solrhome/alfresco/conf/solrcore.properties <SOLR_HOME>/solrhome/archive/conf/solrcore.propertiesSet the following Solr properties in thesolrcore.properties file:
alfresco.host=localhost alfresco.port.ssl=8443 (if using HTTPS transport, specify the SSL port) alfresco.port=8080 (if using HTTP transport, specify the HTTP port) alfresco.secureComms=none (if using HTTP transport, or https if HTTPS transport is being used)If you’re using HTTPS transport, then make sure that your load balancer is configured to use the Alfresco certificate.
You’ll need to generate your ownbrowser.p12 certificate, and add it to your load balancer configuration.
If you’re using HTTP transport, then make sure that the following property is set in<classpathRoot>/alfresco-global.properties:
`solr.secureComms=none`Follow these steps to set up a repository cluster.
By default, all Enterprise servers connected to the same database will form a repository cluster.
Follow these steps for each server in the cluster:
Install and configure the repository server. See the install documentation for deploying Content Services (alfresco.war). In addition, ensure that:
alfresco-global.properties.Ensure that port number5701 (the default clustering port) is accessible on each repository server by all the other repository servers in the cluster.
Specify a wildcard (for example,10.50.*.*) or exact (for example,192.168.1.100) IP address of the network interface for clustering to use.
The advantage of using a wildcard IP address is that the configuration can be used on multiple servers without local changes. The java property name to use isalfresco.cluster.interface (optional).
(Optional) Set the following java property to activate Hazelcast’s own JMX reporting:
hazelcast.jmx=trueOpen thealfresco-global.properties file.
Set the following repository properties.
dir.root=<ALFRESCO_HOME>/alf_data dir.remote=/nfs/or/samba/share/common/for/all/nodes dir.contentstore=${dir.remote}/contentstore dir.contentstore.deleted=${dir.remote}/contentstore.deleted dir.cachedcontent=${dir.remote}/cachedcontentSet the following Search properties for mutual TLS (i.e. https).
index.subsystem.name=solr6 dir.keystore=${dir.root}/keystore (Specifies the path to the keystore file) solr.secureComms=https solr.port.ssl=8983 (Specifies the Solr load balancer SSL port) solr.host=172.31.50.59 (Specifies the Solr load balancer IP address or hostname)Set the following Search properties if you’re not using mutual TLS (i.e. http)
index.subsystem.name=solr6 solr.port=8983 (Specifies the Solr load balancer port) solr.secureComms=none (For HTTP transport between Alfresco and Solr) solr.host=172.31.50.59 (Specifies the Solr load balancer IP address or hostname)Use this information to start the repository server cluster.
In most cases, it’s not necessary to apply any clustering-specific configuration - just starting the servers will result in a cluster.
Let’s suppose you have two cluster members on IP addresses, 10.244.50.101 and 10.244.50.102. Upon starting the first member, you should see the log message similar to the one shown:
2013-08-05 17:06:31,794 INFO [cluster.core.ClusteringBootstrap] [Thread-3] Cluster started, name: MainRepository-2c0aa5c6-e38a-4f64-bd29-1a7cf98943502013-08-05 17:06:31,797 INFO [cluster.core.ClusteringBootstrap] [Thread-3] Current cluster members: 10.244.50.101:5701 (hostname: repo1.local)This shows that a cluster name has been automatically generated, based on the repository name (MainRepository) and a UUID (a random/unique identifier). Finally, the cluster has been started and the cluster members are listed. As shown in the log message, only one cluster member is present currently.
Upon starting the second member, you should see the log message similar to the one shown:
2013-08-05 17:06:58,350 INFO [cluster.core.ClusteringBootstrap] [Thread-3] Cluster started, name: MainRepository-2c0aa5c6-e38a-4f64-bd29-1a7cf98943502013-08-05 17:06:58,353 INFO [cluster.core.ClusteringBootstrap] [Thread-3] Current cluster members: 10.244.50.102:5701 (hostname: repo2.local) 10.244.50.101:5701 (hostname: repo1.local)This log message shows that both the servers are now members of the same cluster.
Note: When starting up a clustered environment, the nodes in the cluster should be started in a rolling start, such that each node is fully started before the next is started in the cluster. This prevents any resource/load concurrency conflicts.
Use the Repo Admin Console for repository server clustering.
Servers connected to the same database instance are usually clustered automatically. In most cases no additional configuration is necessary.
Note: Ensure that clustering is enabled for your license.
Open theRepo Admin Console.
In theRepository Services section, clickRepository Server Clustering.
You see theRepository Server Clustering page.
Set the clustering properties:
For Host Server:
| Property | Description |
|---|---|
| Server Name | This specifies the name of the host server that you’re currently connected to, for exampleip-x-x-x-x. |
| Cluster | This shows if clustering is enabled or disabled. You need to have a correct license to enable clustering. |
| IP Address | This specifies the IP address of the server, for examplex.x.x.x. |
| Cluster ID | This specifies the unique id of the server, for examplexxxxxx. |
For Cluster Members: Server Details
| Property | Description |
|---|---|
| Server | This specifies the server name of the cluster member, for exampleip-x-x-x-x. |
| IP | This specifies the IP address of the server, for examplex.x.x.x. |
| Port | This specifies the port number of the server, for example5701. |
| Last Registered | This specifies the date and time when the cluster member was last started, for example02-Oct-2013 12:48:37. |
| Number of Members | This specifies the total number of members in the cluster, for example1. |
For Offline Cluster Members: Server Details
| Property | Description |
|---|---|
| Server | This specifies the server name of a previously clustered server member that is no longer a member of the cluster, for exampleip-x-x-x-x. |
| IP | This specifies the IP address of the offline server, for examplex.x.x.x. |
| Port | This specifies the port number of the offline server, for example5701. |
| Last Registered | This specifies the date and time when the offline cluster server was last started, for example02-Oct-2013 12:48:37. |
ClickRemove from list to decommission a particular cluster member.
The offline cluster member no longer appears in theOffline Cluster Members list.
Set the clustering properties forConnected Non-Clustered Server(s):
In exceptional cases, a server may be connected to the same database as other cluster members, and yet it may not be a member of the repository cluster. In other words, it’ll have clustering disabled. Such a server is called connected non-clustered server.
| Property | Description |
|---|---|
| Server | This specifies the name of the server, for exampleip-x-x-x-x. |
| IP | This specifies the IP address of the server, for examplex.x.x.x. |
To check if clustering is working properly, clickValidate Cluster.
You see theCluster Validation page. This page shows the validation results for a cluster.
Cluster validation performs a check to ensure that communication between the cluster members is working correctly. For a cluster to be considered validated, all cluster members should show success status. If one server fails in a two-server cluster, then both the servers will be marked as failed.
ClickClose.
There are a number of steps required to test repository server clustering.
The quickest and easiest way to test the cluster is by using the Repo Admin Console.
Ensure that the Content Services server is running.
Enter the following URL in a browser window:
http://<your-host-name>:8080/alfresco/service/enterprise/adminWhere<your-host-name> is the host name where you’re running the server.
AnAuthentication Required prompt displays, showing the IP address or name and the port number of the server.
Enter your user name and password.
Your user name and password must be for an account with administrator permissions.
The Admin Console displays in a browser window. The first page you see is theSystem Summary.
In theRepository Services section, clickRepository Server Clustering.
You see theRepository Server Clustering page.
This page displays information regarding the current cluster members under the Cluster Members section.
ClickValidate Cluster to start a quick test to check that communication is available between each pair of cluster members.
You see the Cluster Validation page. This page displays the result in a matrix form showing cluster communication as either Success or Failure.
Configure the repository server cluster by setting these properties in the alfresco-global.properties file.
Note: These properties are optional.
| Property | Description |
|---|---|
| alfresco.cluster.enabled | This enables clustering, for exampletrue. |
| alfresco.cluster.interface | This specifies a particular network interface to use for clustering. It might be a wildcard value, such as10.256.*.*, which means an attempt is made to bind with an interface having an IP address beginning with10.256. |
| alfresco.cluster.nodetype | This specifies the human-friendly description of the cluster member, for exampleRepository Server. It is useful to give a name to the non-clustered servers, such as a transformation server that it attached to the same database as the cluster, but not participating in it (for example,alfresco.cluster.enabled=false). |
| alfresco.hazelcast.port | This specifies the port to use for clustering, for example5701. |
| alfresco.hazelcast.autoinc.port | This enables Hazelcast to make several attempts to find a free port starting at the value ofalfresco.hazelcast.port.Note: It’s recommended that you do not use this property, for example false. |
| alfresco.hazelcast.max.no.heartbeat.seconds | This specifies the maximum timeout of heartbeat in seconds for a node to assume it is dead, for example15. |
Use this information to track clustering issues.
The main clustering debug information can be customized using the following log4j setting (default value isINFO):
log4j.logger.org.alfresco.enterprise.repo.cluster=infoFor a better control and more detailed clustering debug information, the following category can be configured:
org.alfresco.enterprise.repo.cluster.core.ClusteringBootstrapThis controls clustering initialization and shutdown. It providesINFO level startup and shutdown messages. It also providesWARN level messages, if clustering is disabled or an invalid license is installed.
Here is an example output:
12:38:38,769 INFO [org.alfresco.enterprise.repo.cluster.core.ClusteringBootstrap] Cluster started, name: MainRepository-35ee3b27-0276-4224-9613-3fd8089c6e11 12:38:38,776 INFO [org.alfresco.enterprise.repo.cluster.core.ClusteringBootstrap] Current cluster members: 10.248.10.205:5701 (hostname: node1.alf.example.com) 10.208.63.40:5701 (hostname: node2.alf.example.com)When a cluster member leaves or joins, the following class generates an informativeINFO level message:
org.alfresco.enterprise.repo.cluster.core.MembershipChangeLoggerHere is an example output:
12:38:47,560 INFO [org.alfresco.enterprise.repo.cluster.core.MembershipChangeLogger] Member joined: 10.65.41.64:5701 (hostname: node1.alf.example.com) 12:38:47,569 INFO [org.alfresco.enterprise.repo.cluster.core.MembershipChangeLogger] Current cluster members: 10.208.63.40:5701 (hostname: solr.alf.example.com) 10.248.10.205:5701 (hostname: node2.alf.example.com) 10.65.41.64:5701 (hostname: node1.alf.example.com)An important aspect of clustering is caching. To log cache creation (for example, increase the cache related logging to DEBUG level), enable the following log categories:
log4j.logger.org.alfresco.enterprise.repo.cluster.cache=DEBUG log4j.logger.org.alfresco.repo.cache=DEBUGThe underlying clustering technology, Hazelcast, is configured to uselog4j for logging. Therefore, you can configure logging for the whole Hazelcast top-level package, as shown:
log4j.logger.com.hazelcast=infoTo increase logging from Hazelcast’s member joining mechanism, enable the following log category:
log4j.logger.com.hazelcast.impl.TcpIpJoiner=debugAlfresco uses the Hazelcast library internally for clustering synchronization of data. With the addition of Java 11 modules, there is a warning in the application startup log for the repository app, as shown:
WARNING [localhost-startStop-1] com.hazelcast.instance.HazelcastInstanceFactory.null Hazelcast is starting in a Java modular environment (Java 9 and newer) but without proper access to required Java packages. Use additional Java arguments to provide Hazelcast access to Java internal API. The internal API access is used to get the best performance results. Arguments to be used: --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMEDSee theHazelcast documentation for more details.
For more details about the danger of using those parameters, seehttps://openjdk.java.net/jeps/261. Note the warning on that page:
Note: The
--add-exportsand--add-opensoptions must be used with great care. You can use them to gain access to an internal API of a library module, or even of the JDK itself, but you do so at your own risk: If that internal API is changed or removed, then your library or application will fail.
We decided not to hide this warning, since that may hide other problems in other areas of the code. This decision won’t impact the performance or security of the repository app. Any required modifications will be reviewed and addressed in the next available Java releases. It is assumed that Hazelcast will probably do the same.