High Quality Enterprise Messaging.

High Availability Test Suite

Introduction

To test your JMS applications in regard to High Availability can be really painful if you have to stop and start HA instances manually or by your own scripts. The best would be if the HA instances are stopped and started in intervals and to monitor your JMS applications only to see how they behave under HA conditions.

This is exactly what the HA Test Suite will do for you. It consists of launchers for each HA instance that starts and stops the instance in intervals, configured via a simple XML configuration file. The HA instances are launched as a separate process and hard killed (like kill -9) after a configurable runtime. The other HA instance will take over processing. So you will have a continuous failover scenario while your JMS application is running. Monitor tools (text and GUI based) are also included to view System.out, System.err of the HA instance as well as launcher control commands.

Both launchers are connected via a TCP connection (called command connection). One launcher acts as master, the other one as slave. Slave connects to master. The master controls the slave and tells when to stop and start its HA instance. The master ensures that one instance is always running. Start/stop of HA instances goes round-robin, starting with the master's instance. Both launchers offer a monitor port to connect with the HA Monitor. This HA Monitor is available as a pure text monitor and a Swing GUI monitor.

Note #1: Ensure that you don't use a configuration where one instance has the "preferred-active" flag enabled. All control about failover has to be at the HA test suite. So if you run the HA Test Suite with your configuration, disable the preferred-active flag.

Note #2: The HA Test Suite is configured for the Replicated File Store. The launcher scripts will delete the whole store content of the HA instances before the launcher is executed. This ensures that you have always a clean store for your tests. If you use Shared File Store or Shared JDBC Store, you should clean the store yourself before executing the launcher scripts.

Setup and Start of the Master Launcher

It doesn't matter which launcher is the master. Just choose one of the HA instances and go to directory hatest/config/unix or /hatest/config/win32 (depends on your platform). Edit master.xml. Specify the absolute directory of the SwiftMQ distribution of the HA instance and optionally change startdelay, stopinterval (all in milliseconds), masterport and monitorport. Default for stopinterval is 120000 ms (2 minutes) and 30000 ms (30 seconds) for the startdelay. That means, one of the HA instances (round robin) is killed after 2 minutes runtime and restarted after 30 seconds.

Note: Both instances must have enough time to turn into ACTIVE/STANDBY state before one instances is killed from the launcher due to the stopinterval. The default stopinterval of 2 minutes can be used if the store contains up to 50'000 messages. If your store is larger, increase the stopinterval accordingly.

Go to directory hatest/scripts/unix or /hatest/scripts/win32 (depends on your platform). If you are on Unix/Linux and if you use the zip distribution, you have to execute

      chmod +x *

On Unix/Linux perform a

      ./launcher ../../config/unix/master.xml instance1

On Windows perform a

      launcher ../../config/win32/master.xml instance1

That's it. The master launcher is running.

Setup and Start of the Slave Launcher

Go to directory hatest/config/unix or /hatest/config/win32 (depends on your platform) of the 2nd HA instance. Edit slave.xml. Specify the absolute directory of the SwiftMQ distribution of the HA instance and optionally change masterport and monitorport according to the ports you did specify in master.xml.

Go to directory hatest/scripts/unix or /hatest/scripts/win32 (depends on your platform). If you are on Unix/Linux and if you use the zip distribution, you have to execute

      chmod +x *

On Unix/Linux perform a

      ./launcher ../../config/unix/slave.xml instance2

On Windows perform a

      launcher ../../config/win32/slave.xml instance2

That's it. The slave launcher is running and should be connected to the master launcher.

Using the HA Monitor Text Console

Go to directory hatest/scripts/unix or /hatest/scripts/win32 (depends on your platform). If you are on Unix/Linux and if you use the zip distribution, you have to execute

      chmod +x *

Call the text console. The following example has a launcher on host "sun1" with monitor port 7500 and the 2nd launcher on host "sun14" on monitor port 7501:

      ./textconsole sun1 7500 sun14 7501

Output example:

      *** Connected: sun1:7500
      *** Connected: sun14:7501
      02.01.2006-17:10:01 Instance-1 CTL: Starting instance ...
      02.01.2006-17:10:02 Instance-1 OUT: Booting SwiftMQ 6.0.0 Production ...
      02.01.2006-17:10:02 Instance-1 OUT: ... startup: Trace Swiftlet
      02.01.2006-17:10:03 Instance-1 OUT: ... startup: Log Swiftlet
      02.01.2006-17:10:03 Instance-1 OUT: ... startup: Threadpool Swiftlet
      02.01.2006-17:10:03 Instance-1 OUT: ... startup: Timer Swiftlet
      02.01.2006-17:10:03 Instance-1 OUT: ... startup: Network Swiftlet
      02.01.2006-17:10:03 Instance-1 OUT: ... startup: High Availability Controller Swiftlet
      02.01.2006-17:06:42 Instance-2 OUT: +++ High Availability State: INITIALIZE/STANDALONE
      02.01.2006-17:10:03 Instance-1 OUT: SwiftMQ 6.0.0 Production is ready.
      02.01.2006-17:10:03 Instance-1 OUT: +++ Waiting for Negotiation (timeout: 120000 ms) ...
      02.01.2006-17:10:03 Instance-1 OUT: +++ High Availability State: UNKNOWN/ACTIVE
      02.01.2006-17:10:03 Instance-1 OUT: +++ High Availability State: INITIALIZE/ACTIVE
      02.01.2006-17:06:42 Instance-2 OUT: +++ High Availability State: NEGOTIATE/STANDALONE
      02.01.2006-17:06:42 Instance-2 OUT: +++ High Availability State: ACTIVE-SYNC-PREPARE/ACTI...
      02.01.2006-17:10:03 Instance-1 OUT: +++ High Availability State: NEGOTIATE/ACTIVE
      02.01.2006-17:10:03 Instance-1 OUT: ... standby: Authentication Swiftlet
      02.01.2006-17:10:04 Instance-1 OUT: ... standby: Store Swiftlet (HA)
      02.01.2006-17:10:04 Instance-1 OUT: ... standby: Queue Manager Swiftlet
      02.01.2006-17:10:04 Instance-1 OUT: ... standby: Topic Manager Swiftlet
      02.01.2006-17:10:04 Instance-1 OUT: ... standby: Management Swiftlet
      02.01.2006-17:10:04 Instance-1 OUT: ... standby: XA Resource Manager Swiftlet
      02.01.2006-17:10:04 Instance-1 OUT: ... standby: Routing Swiftlet (Unlimited Connections)
      02.01.2006-17:10:04 Instance-1 OUT: ... standby: JNDI Swiftlet
      02.01.2006-17:10:04 Instance-1 OUT: ... standby: JMS Swiftlet (XA/ASF)
      02.01.2006-17:10:04 Instance-1 OUT: ... standby: Deploy Swiftlet
      02.01.2006-17:10:04 Instance-1 OUT: ... standby: JMS Application Container Swiftlet
      02.01.2006-17:10:04 Instance-1 OUT: ... standby: Scheduler Swiftlet
      02.01.2006-17:10:04 Instance-1 OUT: +++ High Availability State: STANDBY-SYNC-PREPARE/STA...
      02.01.2006-17:06:43 Instance-2 OUT: +++ High Availability State: ACTIVE-SYNC/ACTIVE-SYNC
      02.01.2006-17:10:04 Instance-1 OUT: +++ High Availability State: STANDBY-SYNC/STANDBY-SYNC
      02.01.2006-17:10:06 Instance-1 OUT: +++ High Availability State: STANDBY/STANDBY
      02.01.2006-17:06:45 Instance-2 OUT: +++ High Availability State: ACTIVE/ACTIVE

Note: The time stamps shown above are the local time of the resp. hosts (not synchronized in this example).

Using the HA Monitor Swing Console

Note: Only 1 monitor connection is allowed at a time. If you have the text console running, stop it now.

Go to directory hatest/scripts/unix or /hatest/scripts/win32 (depends on your platform). If you are on Unix/Linux and if you use the zip distribution, you have to execute

      chmod +x *

Call the Swing console.

      ./swingconsole

Specify host and ports of the 2 launchers in the connection panels and press the connect buttons. Example:

Note: The time stamps shown above are the local time of the resp. hosts (not synchronized in this example).

Testing your Applications

The HA Test Suite is now running and takes care of start/stop of the HA instances. You can disconnect and reconnect with the HA Monitors at any time (remember: only 1 at a time). The launchers will save the last 100 messages and will send it to the monitors after connect.

You are now ready to test your JMS applications. Use the SMQP-URL supplied from the HA Wizard.