jWebSocket JMS based Cluster

Introduction to clusters

Application clustering (sometimes called software clustering) is a method of turning multiple computer servers into a cluster (a group of servers that acts like a single system). Clustering software is installed in each of the servers in the group. Each of the servers maintains the same information and collectively they perform administrative tasks such as load balancing, determining node failures, and assigning failover duty. The other clustering method, hardware clustering, requires that specialized hardware be installed in a single server that controls the cluster.

Because servers can easily be added or removed from the cluster as needs dictate, an application cluster is more scalable than its hardware-based counterpart. Furthermore, because it doesn't require specialized hardware, an application cluster tends to be easier and cheaper to configure. For these reasons, application clustering is the more commonly practiced method. http://searchdomino.techtarget.com/definition/application-clustering

A cluster is ALWAYS required when applications critic requirements include:

  • High Performance
  • High Availability
  • Load Balancing
  • Scalability
Why a jWebsocket cluster?

The jWebSocket server is a high performance application server to deploy and run real-time web applications. A single jWebSocket server node, running on a normal computer (non-server specific hardware) can handle thousands of concurrent connections and its incoming/outgoing messages, because the jWebSocket server has been built from the beginning oriented to high performance and real-time scenarios.

However if an application is designed for millions of users or simply it requires being ALWAYS available, a single jWebSocket server node can’t support you.
In the other hand: Who can protect your node against a hardware crash or an electricity power failure? The answer is: a software cluster!

Benefits of using a jWebSocket cluster

  • All your server-side deployed apps become scalable without affecting the processing performance.
  • All your server-side deployed apps will be ALWAYS available, supported by the cluster failover mechanism.
  • Your cluster nodes can be progressively upgraded (software or hardware) without the need to stop the cluster or any running app.
  • A jWebSocket cluster can handle millions of concurrent connections and process millions of concurrent messages.
The jWebSocket JMS based cluster implementation

 

The first support for clustering in jWebSocket is based on the Java Message Service (JMS) API and Apache ActiveMQ server/cluster as message broker. Clients and nodes exchanges messages through the message broker using JMS compliant communication protocols. Multiple jWebSocket clusters can be connected to the same message broker backbone and each of those clusters can have unlimited number of server nodes assigned.
Non-messaging related clusters information like connections and session’s data is stored into a robust MongoDB database server cluster.

ActiveMQ as message broker

“Apache ActiveMQ ™ is the most popular and powerful open source messaging and Integration Patterns server.
Apache ActiveMQ is fast, supports many Cross Language Clients and Protocols, comes with easy to use Enterprise Integration Patterns and many advanced features while fully supporting JMS 1.1 and J2EE 1.4. Apache ActiveMQ is released under the Apache 2.0 License…”
http://activemq.apache.org/

The JMS based jWebSocket cluster uses ActiveMQ as message broker server, in order to get benefits from it clustering support and supported protocols. ActiveMQ also allows web clients to establish WebSocket connections with the message broker and exchange messages using the STOMP protocol.

MongoDB as data persistency layer

The MongoDB database server is used as persistency layer to store non-messaging related jWebSocket cluster information like connections and session’s data.

“MongoDB (from "humongous") is a cross-platform document-oriented database system. Classified as a "NoSQL" database, MongoDB eschews the traditional table-based relational database structure in favor of JSON-like documents with dynamic schemas (MongoDB calls the format BSON), making the integration of data in certain types of applications easier and faster. Released under a combination of the GNU Affero General Public License and the Apache License, MongoDB is free and open source software…” http://en.wikipedia.org/wiki/MongoDB

We choose MongoDB because is very simple to use, offers high performance under heavy load and it’s scalable.

Architecture Design Hints

  • Each cluster uses 3 dedicated JMS topics:
    • cluster_name: The topic represents the communication channel between the clients and the server nodes.
    • cluster_name_nodes: The topic represents the cluster nodes communication channel.
    • cluster_name_messagehub: The topic represents the communication channel used by the cluster high level applications to exchange messages (synchronization/control/application).
  • Only one JMS topic is used for the clients request/reply mechanism, an ActiveMQ security filter blocks possible message intersection by non-authorized consumers.
  • Each jWebSocket server node runs a load balancer instance; however the load balancers are synchronized between each other and only one is able to process a custom client request to finally perform the request redirection to the “optimum” node. The optimum node is selected according to the CPU load and amount of previous processed requests, the less value is the better.
  • The server nodes are automatically disconnected from the cluster when they get shutdown, whatever that be the reason. The correct way to shutdown a server node is first to pause it, wait for 5 minutes (giving some time to finish previous assigned tasks) and finally stop it.
  • Cluster applications require storing their application data into shared databases and keep synchronized each other in the cluster. Cluster applications can communicate each other through the cluster Message Hub.

How to configure a jWebSocket JMS cluster?

To configure and start a jWebSocket JMS cluster is pretty simple, once you solved the two main important pre-requisites:

  1. MongoDB server installed and running. For clustering details please see: https://library.linode.com/databases/mongodb/clusters
  2. ActiveMQ server installed and running. For clustering details please see: http://activemq.apache.org/networks-of-brokers.html

NOTE: For development environments simply running a single instance of MongoDB and ActiveMQ servers with default configurations is enough. Enabling security and clustering for MongoDB and ActiveMQ is required only for production environments depending of the scaling requirements.

The jWebSocket ActiveMQ security filter.

Because the jWebSocket JMS cluster implementation uses one topic per cluster to exchange incoming/outgoing messages between the clients and the server nodes, is required to protect the topic from unauthorized consumers. This means: requests can be only consumed by server nodes and replies by requester clients. Also S2C messages are only consumed by target clients. Only message broadcasts can be read from multiple consumers.
To enable the ActiveMQ filter copy the jWebSocketActiveMQPlugIn-1.0.jar into the ActiveMQ server libs folder and add the following XML configuration to your activemq.xml (each node) configuration file:

<messageAuthorizationPolicy>
...
<bean class="org.jwebsocket.amq.AMQMessageAuthorizationFilter" xmlns="http://www.springframework.org/schema/beans">
  <property name="targetDestinations">
    <list>
      <value>topic://cluster_name</value>
    </list>
  </property>
  <property name="secureMessageHubTopic" value="true" />
  
</bean>
...
</messageAuthorizationPolicy>

The “target destinations” list indicates to the filter the destinations to protect. Regular expressions are allowed on the list entries.

Optionally the filter by default protects the cluster_name_nodes and cluster_name_messagehub topics to avoid unauthorized message consumers on internal topics. This security protection can be disabled and configured using the AMQ security system. 

<bean class="org.jwebsocket.amq.AMQMessageAuthorizationFilter" xmlns="http://www.springframework.org/schema/beans">
  <property name="targetDestinations">
    <list>
      <value>topic://jws_cloud1</value>
    </list>
  </property>
  <property name="secureMessageHubTopic" value="true" />
  <property name="secureNodesTopic" value="true" />
</bean>

NOTE: In order to get granted to consume client messages or server nodes messages, the server nodes require an authenticated connection with the message broker where the username matches cluster topic name, for example: a cluster named: jws_cluster1, requires a topic named: jws_cluster1 and a user named: jws_cluster1

The jWebSocket ActiveMQ plug-in.

The usage of JMS topics over ActiveMQ network of brokers if "conduitSubscriptions=true", causes that topic messages does not contains the real producer connection id, instead the connection id of the AMQ node that forwarded the message on the AMQ network. So in order to get properly identified and associate the "CONNECTION" handshake message to a client and in consequence perform it "connector-stopped" process if it gets an unexpected disconnection to the AMQ network, we created an AMQ plug-in that require to be enabled.

<plugins> 
  ...
  <bean xmlns="http://www.springframework.org/schema/beans" id="jwsClusterPlugIn" class="org.jwebsocket.amq.AMQClusterPlugIn">
    <property name="targetDestinations">
      <list>
        <value>topic://jws_cloud1</value>
        <value>topic://jws_cloud1_nodes</value>					
      </list>
    </property>
    <property name="mongo">
      <bean xmlns="http://www.springframework.org/schema/beans" id="mongodbConnection0" class="com.mongodb.MongoClient" destroy-method="close">
        <constructor-arg value="localhost"/>
      </bean>
    </property>
  </bean>
  ...
</plugins>

NOTE: The MongoDB driver for Java should be previously copied into the AMQ server lib folder. The jWebSocket ActiveMQ plug-in class is located into the jWebSocketJMSPlugIn-1.0.jar.

The jWebSocket server nodes JMS Engine configuration

Enabling the JMSEngine in the jWebSocket server bootstrap configuration file (jWebSocket.xml):

<engines>	
	<!-- JMS Engine Configuration -->
	<engine>
		<name>org.jwebsocket.jms.JMSEngine</name>
		<id>jms0</id>
		<jar>jWebSocketServer-1.0.jar</jar>
		<port>0</port>
		<sslport>0</sslport>
		<keystore>NA</keystore>
		<password>NA</password>
		<maxframesize>1048840</maxframesize>
		<domains>
			<domain>NA</domain>
		</domains>
		<settings>
		<setting key="spring_config">${JWEBSOCKET_HOME}conf/JMSEngine/cluster.xml</setting>
		</settings>		
	</engine>
</engines>

NOTE: If the JMSEngine is enabled, and you have multiple jWebSocket server nodes in the cluster, DO NOT enable other engines to avoid unexpected behaviors.

The JMSEngine configuration file is located by default at: ${JWEBSOCKET_HOME}conf/JMSEngine/cluster.xml please read inline comments:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" " http://www.springframework.org/dtd/spring-beans.dtd"> 
<beans>
	<bean id="destination" class="java.lang.String">
		<!-- the cluster topic name for client2server communication	-->
		<constructor-arg value="jws_cloud1"/>
	</bean>
	<bean id="nodeDescription" class="java.lang.String">
		<!-- the current cluster node description for administrative description -->
		<constructor-arg value="jWebSocket Server Node"/>
	</bean>

	<!-- the JMSEngine ActiveMQ connection factory	-->
	<bean id="amqFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
		<property name="brokerURL" value="tcp://0.0.0.0:61616" />
		<property name="useAsyncSend" value="true" />
		<property name="userName">
			<value>jws_cloud1</value>
		</property>
		<property name="password">
			<value>password</value>
		</property>
	</bean>
	<bean id="connectionFactory"
		  class="org.springframework.jms.connection.SingleConnectionFactory">
		<property name="targetConnectionFactory" ref="amqFactory" />
	</bean>

	<!-- the MongoDB server connection instance -->
	<bean id="mongodbConnection0" class="com.mongodb.MongoClient" destroy-method="close">
		<constructor-arg value="localhost"/>
	</bean>

	<!-- the MongoDB database instance to store the load balancer data -->
	<bean id="jws_lb_db" class="org.jwebsocket.storage.mongodb.MongoDBUtils" factory-method="getDB">
		<constructor-arg ref="mongodbConnection0"/>
		<constructor-arg value="jws_lb"/>
		<!--<constructor-arg value="user"/>
		<constructor-arg value="pass"/>-->
	</bean>

	<!-- the cluster 'connectorsManager' instance for client connections management -->
	<bean id="connectorsManager" class="org.jwebsocket.jms.mongodb.MongoDBConnectorsManager">
		<property name="collection">
			<bean factory-bean="jws_lb_db" factory-method="getCollection">
				<constructor-arg>
					<bean factory-bean="destination" factory-method="concat">
						<constructor-arg value="_connectors"/>
					</bean>
				</constructor-arg>
			</bean>
		</property>
	</bean>
	
	<!-- the cluster 'nodesManager' instance for server nodes management -->
	<bean id="nodesManager" class="org.jwebsocket.jms.mongodb.MongoDBNodesManager">
		<property name="collection">
			<bean factory-bean="jws_lb_db" factory-method="getCollection">
				<constructor-arg>
					<bean factory-bean="destination" factory-method="concat">
						<constructor-arg value="_nodes"/>
					</bean>
				</constructor-arg>
			</bean>
		</property>
		<property name="nodeDescription" ref="nodeDescription"></property>
		<!-- the cluster 'synchronizer' instance for cluster workers synchronization -->
		<property name="synchronizer">
			<bean class="org.jwebsocket.jms.mongodb.MongoBDClusterSynchronizer" init-method="initialize">
				<property name="collection">
					<bean factory-bean="jws_lb_db" factory-method="getCollection">
						<constructor-arg>
							<bean factory-bean="destination" factory-method="concat">
								<constructor-arg value="_synchronizer"/>
							</bean>
						</constructor-arg>
					</bean>
				</property>
			</bean>
		</property>
	</bean>
</beans>

Starting a jWebSocket server cluster node

After to configure the JMS Engine in the jWebSocket server configuration, just start the server normally as you did it before. Just ensure that MongoDB and ApacheMQ servers are running.

To run multiple servers as cluster nodes, just clone the jWebSocket server installation and change the “node_id” configuration entry on the jWebSocket.xml configuration file:

<!-- jWebSocket server configuration -->
<jWebSocket>
	<!--
		define the protocol, default protocol if not specified is 'json'
		possible values are [json|xml|csv]
	-->
	<protocol>json</protocol>

	<!--
		the unique ID of this instance within a jWebSocket cluster
		can be left empty to run just a single stand-alone server
		Preparation for pending cluster-plug-in
	-->
	<node_id>01</node_id>
...

And optionally adjust the node description at the JMSEngine XML configuration file, for administrative management:

<beans>
	<bean id="destination" class="java.lang.String">
		<!-- the cluster topic name for client2server communication	-->
		<constructor-arg value="jws_cloud1"/>
	</bean>
	<bean id="nodeDescription" class="java.lang.String">
		<!-- the current cluster node description for administrative description -->
		<constructor-arg value="jWebSocket Server Node"/>
	</bean>

Connecting to the cluster from Web clients

The Web clients support for ActiveMQ server connection is based on the STOMP.js javascript library that uses the HTML5 WebSocket procotol to connect ActiveMQ servers.

Import the following libraries into your Web app:

<script src="JWS_Web_Client/lib/stomp/stomp.js" type="text/javascript"></script>
<script src="JWS_Web_Client/res/js/jWebSocket.js" type="text/javascript"></script>
<script src="JWS_Web_Client/res/js/jWebSocketStomp.js" type="text/javascript"></script>

Creating a basic connection:

var lClient = new jws.jWebSocketJSONClient();
lClient.open('ws://localhost:61614/stomp?cluster=clusterName', {
	wsClass: STOMPWebSocket,
	OnOpen: function(){
		alert('Connected!');
	}
});


Connecting to the cluster from Java clients

The Java client classes are located in the jWebSocketJavaSEClient module (jWebSocketJavaSEClient-1.0.jar):

JWebSocketTokenClient lClient = new JWebSocketTokenClient(new JWebSocketJMSClient("the cluster name"));
lClient.addListener(new WebSocketClientTokenListener() {
	...
});
lClient.open("tcp://localhost:61616");
...

Publications

Learn more about WebSockets in general, get background information and gain deeper insight!

Join jWebSocket

Wether developer, designer or translator – join the jWebSocket team and grow together with our success!

Copyright © 2013 Innotrade GmbH. All rights reserved.