SwiftMQ Documentation

SwiftMQ Documentation

  • Client
  • CE
  • UR
  • HA
  • Javadocs
  • Release Notes

›Swiftlets

SwiftMQ Client

  • Getting Started
  • JNDI Client
  • JMS Client
  • AMQP 1.0 Client
  • Filetransfer Client

SwiftMQ CE

  • Getting Started
  • Installation
  • Upgrade
  • Software Architecture
  • Administration

    • Preconfig
    • CLI Command Line Interface
    • JMX Administration
    • CLI Message Interface
    • routerconfig.xml Watch Dog
    • System Properties

    Swiftlets

    • AMQP Swiftlet
    • Authentication Swiftlet
    • Deploy Swiftlet
    • JMS Swiftlet
    • JNDI Swiftlet
    • Log Swiftlet
    • Management Swiftlet
    • MQTT Swiftlet
    • Network Swiftlet
    • Queue Manager Swiftlet
    • Routing Swiftlet
    • Scheduler Swiftlet
    • Store Swiftlet
    • Streams Swiftlet
    • Threadpool Swiftlet
    • Timer Swiftlet
    • Topic Manager Swiftlet
    • Trace Swiftlet
    • XA Resource Manager Swiftlet

    APIs

    • CLI Admin API

    How To

    • TLS Configuration
    • HTTP Tunneling Configuration
    • Upgrade SwiftMQ Router and Clients

    In Depth

    • Versioning
    • File Store

SwiftMQ UR

  • Getting Started
  • License
  • Installation
  • Upgrade
  • Administration

    • SwiftMQ Explorer

    Swiftlets

    • JDBC Authentication Swiftlet
    • FileCache Swiftlet
    • JMS Application Container Swiftlet
    • JMS XA Swiftlet
    • JDBC Store Swiftlet
    • AMQP Bridge Extension Swiftlet
    • JMS Bridge Extension Swiftlet
    • JavaMail Bridge Extension Swiftlet
    • Replicator Extension Swiftlet

SwiftMQ HA

  • Getting Started
  • License
  • Installation
  • Upgrade
  • HA Introduction
  • HA Deployment
  • Administration
  • JNDI/JMS under HA
  • Routing under HA
  • Advanced Configuration
  • HA Test Suite

Topic Manager Swiftlet

The JMS publish/subscribe message model is based on topics which are managed by the Topic Manager Swiftlet in SwiftMQ.

Publish/Subscribe

Topics are location independent and available on all routers in a Federated Router Network due to dynamic subscription exchange. That is, publishers can publish messages on any router and subscriber can connect and consumer from the same topic on any other router.

SwiftMQ provides topic hierarchies that enables subscribers to filter messages directly out of the topic with the need to declare expensive message selectors.

Publish/Subscribe flow control create constant message streams with maximum throughput between publishers and subscribers. Slow subscriber conditions can be declared to handle slow subscribers, delete malfunctioning durable subscribers and so on.

Topic Hierarchies

SwiftMQ provides the possibility to define topic hierarchies in the form <roottopic>.<subtopic>.<subsubtopic> etc.

Example:

    iit.sales.EU
    iit.sales.US
    iit.projects.swiftmq.users
    news.weather.forecasts
    sports.formula1.MichaelSchumacher

Publishing and subscribing rules to nodes in a hierarchy:

  • Selecting a specific node in a topic hierarchy publishes messages to the addressed node and all concerning sub-nodes. Example: publishing to node iit.sales sends the messages also to iit.sales, iit.sales.US, iit.sales.EU.
  • Subscribing to a specific node in a topic hierarchy means to receive all messages addressed to this node and all parent and sub-nodes. Example: subscribing to iit.sales.US one receives all messages published in iit, iit.sales, and iit.sales.US; subscribing to iit.sales one receives all messages published in iit, iit.sales, iit.sales.US, iit.sales.EU, but you don't receive the messages published in the iit.projects hierarchy.
  • A subscriber receives a message only once, regardless of the number of topics his subscription matches.

SQL-Like Predicate Topic Addressing (Indirect Subscriber Selection)

As an extension to the normal topic node- and subnode-addressing scheme, SwiftMQ provides SQL-Like predicate topic addressing. So every subnode could be a SQL-Like predicate. A SQL-Like predicate could contain wildcards which match a single character (the underscore '_'), or any characters (the percent sign '%'). Escape character is '\'.

Root nodes cannot be addressed as a predicate; i. e. it is not possible to specify i%.%.swiftmq.

Predicate topics are normal topic destinations and are not available via JNDI lookups. To create a predicate topic, one has to use the createTopic() method from a TopicSession.

Example:

    Topic allSwiftMQ = topicSession.createTopic("iit.%.US");
    msg.setJMSDestination(allSwiftMQ);
    publisher.publish(msg);

Access to predicate topics must be granted by the Authentication Swiftlet just like normal topics.

What means "Indirect Subscriber Selection"? The topic name inside the message is a SQL-Like predicate. It is used to select those topic names which are defined at the Topic Manager Swiftlet configuration. This is the first step. The next step is to use those matched topics and match it against subscribers. Subscribers are also using SQL-Like predicates. Therefore, in this mode, a topic hierarchy needs to be declared in the Topic Manager configuration in order to work. This is the price for the flexibility and sometimes a disadvantage. For this reason we introduced "Direct Subscriber Selection" in SwiftMQ 7.3.0 (see next section).

Direct Subscriber Selection

This feature does not interpret the topic name in the message as SQL-Like predicate but as a string. Therefore it is possible to select the subscribers directly by matching the subscriber's SQL-Like predicates against the topic name of the message. You don't need to declare the topics in the Topic Manager Swiftlet configuration.

Because it is a very rare case that someone uses a predicate as a topic name when sending a message, e.g. send to iit.%.US, the direct subscriber selection is the default mode starting with SwiftMQ 7.3.0 but can be disabled:

Durable Subscriptions

Durable subscriptions are implemented as queues. The name of a these queues are formed by this pattern:

      <clientid>$<durablename>

A durable subscription is usually managed by a JMS client. It creates and deletes it by use of the JMS API. However, SwiftMQ provides also ways to create a durable subscription administratively by use of SwiftMQ Explorer or CLI.

Managing Durable Subscriber Queues

Durable subscriber queues are system queues which are automatically created and their configuration is not stored in the router's configuration file. Instead they are controlled by a "Queue Controller" that matches with the name of the durable subscriber queue.

It is possible to declare dedicated queue controllers for specific sets of durable subscribers, e.g. that with a particular client id, and configure these sets differently.

Multiple Consumers on a Durable Supscription

The JMS 1.1 specification defines a durable subscription as exclusive, so it cannot have more than a single consumer. This is enforced by requiring that all active JMS connections have a unique JMS client id. Later versions of the JMS specification have relaxed this requirement and allows multiple consumers per durable subscription to handle higher loads.

To allow multiple consumers on a single durable subscription, one needs to allow connections with the same client id by setting the specific attribute in the JMS Swiftlet configuration:

Multiple consumer then behave as defined in the attribute consumer-mode (introduced in 9.6.0) of the respective queue controller.

For example, let's configure 2 different sets of durable subscriptions, one defined by client id failsave and the other set covers simply all others. The failsave set should allow to have "standby" consumers that jump in when the active consumer fails. The other set should be in shared mode where messages should be distributed to all connected consumers.

The configuration of the queue controllers would be:

      <queue-controllers>
        <queue-controller name="01" persistence-mode="non_persistent" predicate="tmp$%"/>
        <queue-controller name="02" predicate="sys$%"/>
        <queue-controller name="03" predicate="swiftmq%"/>
        <queue-controller name="04" predicate="rt$%"/>
        <queue-controller name="05" predicate="unroutable"/>
        <queue-controller name="06" consumer-mode="activestandby" predicate="failsave$%"/>
        <queue-controller name="07" predicate="%$%"/>
        <queue-controller name="08" predicate="routerdlq"/>
        <queue-controller name="09" predicate="%"/>
      </queue-controllers>

In the above example queue controller 06 matches for the failsave durable subcription and queue controller 07 for all others. The latter uses the default consumer-mode of shared. Note the lexical order of the name attribute which is important because there is a top-down matching sequence and the first match on the queue name is used.

Publish/Subscribe Flow Control

SwiftMQ provides 2 different ways to control the message flow:

  • delay based flow control
  • slow subscriber conditions

Delay Based Flow Control

SwiftMQ provides publish/subscribe flow control. As for point-to-point queues, the flow control is based on flow control delays. During the publish/commit operation, all flow control delays of the selected subscriber queues are determined and the maximum delay is delivered back. So, the flow control delay is that of the slowest subscriber.

Slow Subscriber Conditions

In some use case scenarios it is very important to maintain a constant publishing rate. For example, a trading feed in a financial application requires that feed data (stock prices) are published without delays. If there are slow subscribers on that feed topic, either a backlog of messages will increase for that subscribers or, when using pub/sub flow control, the publisher will be throttled down from SwiftMQ to the rate of the slowest subscriber.

To avoid this situations, slow subscriber conditions can be created. Such a condition identifies a slow subscriber during the publish transaction and, if the condition matches, avoids to send further messages to those matching subscribers until the condition doesn't match with future publish transactions. Thus, a constant publishing rate is created.

SwiftMQ also allows to remove non-durable subscriber connections and to remove durable subscriber connections including deleting the durable subscriptions. See sections below.

A slow subscriber condition is defined per 'root topic', which is the first node of a topic hierarchy. Say, all your feeds are summarized under the hierarchy 'feed', e.g. 'feed.bloomberg', 'feed.reuters', a condition can be defined on root topic 'feed'. The condition is valid for all sub-topics in the hierarchy. Further condition attributes are the persistence-mode (all, persistent, non_persistent), the subscription-type (all, local, remote) and the maximum messages in the subscriber queue during the time of the publish operation.

Example:

      <swiftlet name="sys$topicmanager">
        <slow-subscriber-conditions>
          <topic name="feed" max-messages="100" persistence-mode="non_persistent" subscription-type="local"/>
          <topic name="events" max-messages="300" persistence-mode="all" subscription-type="remote"/>
        </slow-subscriber-conditions>
        <topics>
          <topic name="feed"/>
          <topic name="feed.bloomberg"/>
          <topic name="feed.reuters"/>
          <topic name="events"/>
        </topics>
      </swiftlet>

The example defines 2 conditions. The first is for the topic hierarchy 'feed' and matches for nonpersistent messages and local subscribers if the subscriber queue has reached 100 messages. The second condition is for topic hierarchy 'events', matches for all messages but only for remote subscriptions and permits max. 300 messages in the subscriber queue.

Note that for remote subscriptions the subscriber queue is the routing queue of the resp. router and thus the message count of that queue is taken.

Disconnecting Non-Durable Subscribers

SwiftMQ 5.2.0 onwards adds an attribute disconnect-non-durable-subscriber to a slow subscriber condition. The default value is false. If it is set to true, all non-durable subscriber connections for which the condition matches will be removed. This option is useful if there are malfunctioning JMS clients which have topic subscriptions.

Disconnecting Durable Subscribers and Delete Durable Subscriptions

SwiftMQ 5.2.0 onwards adds another attribute disconnect-delete-durable-subscriber to a slow subscriber condition. The default value is false. If it is set to true, all durable subscriber connections for which the condition matches will be removed and their durable subscriptions will be deleted. That is, all messages in their durable subscriber queue will be lost! So be careful with this option. This option is useful if there are malfunctioning JMS clients which have durable topic subscriptions or if there are "dead" durable subscription on a topic (a JMS client created a durable subscription but doesn't use it anymore so the durable subscriber queue grows and grows).

Publish/Subscribe in a Federated Router Network

Dynamic Exchange of remote Subscriptions

All Topic Manager Swiftlets in a router network are working federated. They exchange subscription messages on a "some/none" base. That is, they send a subscription message to all other Topic Manager Swiftlets if they have at least 1 subscription for a particular topic and they send an unsubscription message around if the count reaches zero. The subscription exchange is made on base of root topics. A root topic is the first node in a topic hierarchie, e.g. the root topic of topic 'iit.sales.eu' is 'iit'. For a local Topic Manager Swiftlet these remote subscriptions are just normal subscriptions. The only difference is that the subscriber queue is a remote queue tpc$<roottopic> at the particular remote router.

Remote subscriptions are listed in the Topic Manager's Usage section below node "Remote Router Subscriptions":

Each router node consists of 1 or more root topics for which the particular router has at least 1 subscription.

Usually this dynamic remote subscriptions are only informational. However, there is one case where an administrator has to delete this remote subscriptions: to put a node of a federated router network out of service.

If a node should be removed from a federated router network, this can be done by shutting down its JMS clients which in turn remove all their subscriptions which forces the router node to send an unsubscription notification to all other routers when the last local JMS client unsubscribes from this particular topic. The last action is to shut down the router.

But if the router node did not had an active routing connection to one or more of its neighbours, they will never receive the unsubscription notification and will continously send pub/sub messages for the router's topic subscriptions. In this case an administrator has to delete the resp. remote router subscription entry:

This must be done on all router nodes of the federated router network.

Static Remote Subscriptions

In a federated router network, subscriptions are exchanged dynamically between routers. If a router starts up, it has initially no knowledge about remote subscriptions. Once another router connects to this router, subscriptions are exchanged and messages are forwarded between the routers on base of these subscriptions. To enable message forwarding for times when a router has not yet remote subscriptions (it has just being started and no other router has connected yet), static remote subscriptions can be defined.

Note that, per default, static remote subscriptions are only valid before the remote router connects the first time. Thereafter the subscriptions are turning dynamically. So in the case the remote router unsubscribes from a topic, no further messages are forwarded, even if there is a static remote subscription for it. Starting with release 6.1.0 this behavior can be changed by enabling attribute keep-on-unsubscribe of the static remote subscription. In that case, dynamic unsubscription requests for that topic of that router are not respected and messages are still forwarded. This is very important in the case the remote router performs an orderly shutdown (e.g. for maintenance) and has durable subscribers, because a router unsubscribes from all topics at remote routers during shutdown. If keep-on-unsubscribe is true, messages are still forwarded into the routing queue and will be delivered once the remote router restarts. So no messages are lost in the meantime. If keep-on-unsubscribe is set to false (default), no messages are delivered while the remote router is down and messages are lost.

Static remote subscriptions are defined on a per-router-base. It requires that a static route is defined to the particular remote router.

Example:

      <swiftlet name="sys$routing">
        <connectors/>
        <filters/>
        <listeners>
          <listener name="plainsocket" port="4100">
            <host-access-list/>
          </listener>
        </listeners>
        <static-routes>
          <static-route name="router2"/>
          <static-route name="router3"/>
          <static-route name="router4"/>
        </static-routes>
      </swiftlet>

Thereafter, static remote subscriptions can be defined under the Topic Manager Swiftlet configuration:

Example:

      <swiftlet name="sys$topicmanager" flowcontrol-enabled="false">
        <static-remote-router-subscriptions>
          <static-remote-router-subscription name="router2">
            <static-topic-subscriptions>
              <static-topic-subscription name="testtopic"/>
            </static-topic-subscriptions>
          </static-remote-router-subscription>
          <static-remote-router-subscription name="router3">
            <static-topic-subscriptions>
              <static-topic-subscription name="testtopic"/>
            </static-topic-subscriptions>
          </static-remote-router-subscription>
          <static-remote-router-subscription name="router4">
            <static-topic-subscriptions>
              <static-topic-subscription name="iit" keep-on-unsubscribe="true"/>
              <static-topic-subscription name="testtopic"/>
            </static-topic-subscriptions>
          </static-remote-router-subscription>
        </static-remote-router-subscriptions>
        <topics>
          <topic name="testtopic"/>
          <topic name="iit"/>
          <topic name="iit.sales"/>
          <topic name="iit.sales.eu"/>
          <topic name="iit.sales.us"/>
          <topic name="iit.sales.asia"/>
        </topics>
      </swiftlet>

Note that the 'name' attribute of an element 'static-remote-router-subscription' must contain the remote router name and the 'name' attribute of an element 'static-topic-subscription' must contain the root topic (the first node of a topic hierarchy; if you don't have a topic hierarchy then it's just the topic name).

The above configuration ensures that messages are forwarded to the topic hierarchy beginning with 'testtopic' to routers 'router2', 'router3', and 'router4'. Messages to the topic hierarchy beginning with 'iit' are also forwarded to 'router4'. Forwarding of messages for this topic continues even if the remote router has no subscriptions for it.

Jobs

These jobs can be scheduled via the Scheduler Swiftlet to run at specific times or in intervals, based on calenders and so on.

Delete Durable

The Delete Durable job deletes durable subscribers (incl. all messages) which are selected by 2 SQL-Like predicates, client id and durable name, specified as job parameters. For example, a client id predicate of sales and a durable name predicate of % will delete all durable subscribers of client id sales whereas a client id predicate of s\_\_es% and a durable name predicate of dur% will delete those from saukesta and dur2003. If you set both parameters to %, it will match all durable subscribers and therefore all durable subscribers are deleted.

Parameter Mandatory Description
Client Id Predicate Yes SQL-Like predicate for client ids.
Durable Name Predicate Yes SQL-Like predicate for durable names.

Configuration

The configuration of the Topic Manager Swiftlet is defined within the element

      <swiftlet name="sys$topicmanager" .../>

of the router's configuration file. One can use the SwiftMQ Exlorer or CLI for configuration as well. They both save into that file.

Attributes of Element "swiftlet"

Definition

Attribute Type Mandatory Description
flowcontrol-enabled java.lang.Boolean No Enable/Disable Publish/Subscribe Flow Control
direct-subscriber-selection java.lang.Boolean No Select Subscribers directly and do NOT interpret the Publisher's Topicname as SQL-Like Predicate

Values

Attribute Values
flowcontrol-enabled Default: true
direct-subscriber-selection Default: true

Element List "topics", Parent Element: "swiftlet"

Topic Definitions. This element list contains zero or more "topic" elements with this template definition:

Definition

Attribute Type Mandatory Description
name java.lang.String Yes Name of this Topic

Element List "slow-subscriber-conditions", Parent Element: "swiftlet"

Slow Subscriber Conditions. This element list contains zero or more "topic" elements with this template definition:

Definition

Attribute Type Mandatory Description
name java.lang.String Yes Name of this Root Topic
max-messages java.lang.Long Yes Maximum Messages in Subscriber Queue
persistence-mode java.lang.String No Persistence Mode
subscription-type java.lang.String No Subscription Type
disconnect-non-durable-subscriber java.lang.Boolean No Disconnect Non-Durable Subscriber
disconnect-delete-durable-subscriber java.lang.Boolean No Disconnect and Delete Durable Subscriber

Values

Attribute Values
max-messages Min: 1
Default: 500
persistence-mode Choice: all non_persistent persistent
Default: non_persistent
subscription-type Choice: all local remote
Default: all
disconnect-non-durable-subscriber Default: false
disconnect-delete-durable-subscriber Default: false

Element List "static-remote-router-subscriptions", Parent Element: "swiftlet"

Static Remote Router Subscriptions. This element list contains zero or more "static-remote-router-subscription" elements with this template definition:

Definition

Attribute Type Mandatory Description
name java.lang.String Yes Name of this Static Remote Router Subscription

Element List "static-topic-subscriptions", Parent Element: "static-remote-router-subscription"

Static Topic Subscriptions. This element list contains zero or more "static-topic-subscription" elements with this template definition:

Definition

Attribute Type Mandatory Description
name java.lang.String Yes Name of this Static Topic Subscription
keep-on-unsubscribe java.lang.Boolean No Keep this Subscription even when the remote Router unsubscribes it

Values

Attribute Values
keep-on-unsubscribe Default: false

Element "usage", Parent Element: "swiftlet"

Active Topics.

Element List "durables", Parent Element: "usage"

Active Durable Topic Subscriber. This element list contains zero or more "durables" elements with this template definition:

Definition

Attribute Type Mandatory Description
name java.lang.String Yes Name of this Active Durable Topic Subscriber
durablename java.lang.String Yes Name of the durable Subscriber
clientid java.lang.String Yes Client Id
topic java.lang.String Yes Topic Name
selector java.lang.String No Message Selector
nolocal java.lang.Boolean No Receives no local published Messages
boundto java.lang.String No Is bound to Queue

Values

Attribute Values
durablename
clientid
topic
selector
nolocal Default: false
boundto Default: Do not specify - will be set from SwiftMQ!

Element List "subscriber", Parent Element: "usage"

Active Topic Subscriber. This element list contains zero or more "subscriber" elements with this template definition:

Definition

Attribute Type Mandatory Description
name java.lang.String Yes Name of this An Active Topic Subscriber
clientid java.lang.String No Client Id
topic java.lang.String No Topic Name
selector java.lang.String No Message Selector
nolocal java.lang.Boolean No Receives no local published Messages
boundto java.lang.String No Is bound to Queue

Values

Attribute Values
clientid
topic
selector
nolocal Default: false
boundto

Element List "subscriber-remote", Parent Element: "usage"

Subscriptions from remote Routers. This element list contains zero or more "router" elements with this template definition:

Definition

Attribute Type Mandatory Description
name java.lang.String Yes Name of this Remote Router

Element List "topics", Parent Element: "router"

Subscriptions from this Router. This element list contains zero or more "topic" elements with this template definition:

Definition

Attribute Type Mandatory Description
name java.lang.String Yes Name of this Topic
← Timer SwiftletTrace Swiftlet →
  • Publish/Subscribe
  • Topic Hierarchies
    • SQL-Like Predicate Topic Addressing (Indirect Subscriber Selection)
    • Direct Subscriber Selection
  • Durable Subscriptions
    • Managing Durable Subscriber Queues
    • Multiple Consumers on a Durable Supscription
  • Publish/Subscribe Flow Control
    • Delay Based Flow Control
    • Slow Subscriber Conditions
  • Publish/Subscribe in a Federated Router Network
    • Dynamic Exchange of remote Subscriptions
    • Static Remote Subscriptions
  • Jobs
    • Delete Durable
  • Configuration
Copyright © 2020 IIT Software GmbH