Topic Manager Swiftlet
The JMS publish/subscribe message model is based on topics which are managed by the Topic Manager Swiftlet in SwiftMQ.
Publish/Subscribe
Topics are location independent and available on all routers in a Federated Router Network due to dynamic subscription exchange. That is, publishers can publish messages on any router and subscriber can connect and consumer from the same topic on any other router.
SwiftMQ provides topic hierarchies that enables subscribers to filter messages directly out of the topic with the need to declare expensive message selectors.
Publish/Subscribe flow control create constant message streams with maximum throughput between publishers and subscribers. Slow subscriber conditions can be declared to handle slow subscribers, delete malfunctioning durable subscribers and so on.
Topic Hierarchies
SwiftMQ provides the possibility to define topic hierarchies in the form <roottopic>.<subtopic>.<subsubtopic> etc.
Example:
iit.sales.EU
iit.sales.US
iit.projects.swiftmq.users
news.weather.forecasts
sports.formula1.MichaelSchumacher
Publishing and subscribing rules to nodes in a hierarchy:
- Selecting a specific node in a topic hierarchy publishes messages to the addressed node and all concerning sub-nodes. Example: publishing to node iit.sales sends the messages also to iit.sales, iit.sales.US, iit.sales.EU.
- Subscribing to a specific node in a topic hierarchy means to receive all messages addressed to this node and all parent and sub-nodes. Example: subscribing to iit.sales.US one receives all messages published in iit, iit.sales, and iit.sales.US; subscribing to iit.sales one receives all messages published in iit, iit.sales, iit.sales.US, iit.sales.EU, but you don't receive the messages published in the iit.projects hierarchy.
- A subscriber receives a message only once, regardless of the number of topics his subscription matches.
SQL-Like Predicate Topic Addressing (Indirect Subscriber Selection)
As an extension to the normal topic node- and subnode-addressing scheme, SwiftMQ provides SQL-Like predicate topic addressing. So every subnode could be a SQL-Like predicate. A SQL-Like predicate could contain wildcards which match a single character (the underscore '_'), or any characters (the percent sign '%'). Escape character is '\'.
Root nodes cannot be addressed as a predicate; i. e. it is not possible to specify i%.%.swiftmq.
Predicate topics are normal topic destinations and are not available via JNDI lookups. To create a predicate topic, one has to use the createTopic() method from a TopicSession.
Example:
Topic allSwiftMQ = topicSession.createTopic("iit.%.US");
msg.setJMSDestination(allSwiftMQ);
publisher.publish(msg);
Access to predicate topics must be granted by the Authentication Swiftlet just like normal topics.
What means "Indirect Subscriber Selection"? The topic name inside the message is a SQL-Like predicate. It is used to select those topic names which are defined at the Topic Manager Swiftlet configuration. This is the first step. The next step is to use those matched topics and match it against subscribers. Subscribers are also using SQL-Like predicates. Therefore, in this mode, a topic hierarchy needs to be declared in the Topic Manager configuration in order to work. This is the price for the flexibility and sometimes a disadvantage. For this reason we introduced "Direct Subscriber Selection" in SwiftMQ 7.3.0 (see next section).
Direct Subscriber Selection
This feature does not interpret the topic name in the message as SQL-Like predicate but as a string. Therefore it is possible to select the subscribers directly by matching the subscriber's SQL-Like predicates against the topic name of the message. You don't need to declare the topics in the Topic Manager Swiftlet configuration.
Because it is a very rare case that someone uses a predicate as a topic
name when sending a message, e.g. send to iit.%.US, the direct
subscriber selection is the default mode starting with SwiftMQ 7.3.0 but
can be disabled:
Durable Subscriptions
Durable subscriptions are implemented as queues. The name of a these queues are formed by this pattern:
<clientid>$<durablename>
A durable subscription is usually managed by a JMS client. It creates and deletes it by use of the JMS API. However, SwiftMQ provides also ways to create a durable subscription administratively by use of SwiftMQ Explorer or CLI.
Managing Durable Subscriber Queues
Durable subscriber queues are system queues which are automatically created and their configuration is not stored in the router's configuration file. Instead they are controlled by a "Queue Controller" that matches with the name of the durable subscriber queue.
It is possible to declare dedicated queue controllers for specific sets of durable subscribers, e.g. that with a particular client id, and configure these sets differently.
Multiple Consumers on a Durable Supscription
The JMS 1.1 specification defines a durable subscription as exclusive, so it cannot have more than a single consumer. This is enforced by requiring that all active JMS connections have a unique JMS client id. Later versions of the JMS specification have relaxed this requirement and allows multiple consumers per durable subscription to handle higher loads.
To allow multiple consumers on a single durable subscription, one needs to allow connections with the same client id by setting the specific attribute in the JMS Swiftlet configuration:
Multiple consumer then behave as defined in the attribute
consumer-mode (introduced in 9.6.0) of the respective queue
controller.
For example, let's configure 2 different sets of durable subscriptions,
one defined by client id failsave and the other set covers simply all
others. The failsave set should allow to have "standby" consumers that
jump in when the active consumer fails. The other set should be in
shared mode where messages should be distributed to all connected
consumers.
The configuration of the queue controllers would be:
<queue-controllers>
<queue-controller name="01" persistence-mode="non_persistent" predicate="tmp$%"/>
<queue-controller name="02" predicate="sys$%"/>
<queue-controller name="03" predicate="swiftmq%"/>
<queue-controller name="04" predicate="rt$%"/>
<queue-controller name="05" predicate="unroutable"/>
<queue-controller name="06" consumer-mode="activestandby" predicate="failsave$%"/>
<queue-controller name="07" predicate="%$%"/>
<queue-controller name="08" predicate="routerdlq"/>
<queue-controller name="09" predicate="%"/>
</queue-controllers>
In the above example queue controller 06 matches for the failsave
durable subcription and queue controller 07 for all others. The latter
uses the default consumer-mode of shared. Note the lexical order of
the name attribute which is important because there is a top-down
matching sequence and the first match on the queue name is used.
Publish/Subscribe Flow Control
SwiftMQ provides 2 different ways to control the message flow:
- delay based flow control
- slow subscriber conditions
Delay Based Flow Control
SwiftMQ provides publish/subscribe flow control. As for point-to-point queues, the flow control is based on flow control delays. During the publish/commit operation, all flow control delays of the selected subscriber queues are determined and the maximum delay is delivered back. So, the flow control delay is that of the slowest subscriber.
Slow Subscriber Conditions
In some use case scenarios it is very important to maintain a constant publishing rate. For example, a trading feed in a financial application requires that feed data (stock prices) are published without delays. If there are slow subscribers on that feed topic, either a backlog of messages will increase for that subscribers or, when using pub/sub flow control, the publisher will be throttled down from SwiftMQ to the rate of the slowest subscriber.
To avoid this situations, slow subscriber conditions can be created. Such a condition identifies a slow subscriber during the publish transaction and, if the condition matches, avoids to send further messages to those matching subscribers until the condition doesn't match with future publish transactions. Thus, a constant publishing rate is created.
SwiftMQ also allows to remove non-durable subscriber connections and to remove durable subscriber connections including deleting the durable subscriptions. See sections below.
A slow subscriber condition is defined per 'root topic', which is the first node of a topic hierarchy. Say, all your feeds are summarized under the hierarchy 'feed', e.g. 'feed.bloomberg', 'feed.reuters', a condition can be defined on root topic 'feed'. The condition is valid for all sub-topics in the hierarchy. Further condition attributes are the persistence-mode (all, persistent, non_persistent), the subscription-type (all, local, remote) and the maximum messages in the subscriber queue during the time of the publish operation.
Example:
<swiftlet name="sys$topicmanager">
<slow-subscriber-conditions>
<topic name="feed" max-messages="100" persistence-mode="non_persistent" subscription-type="local"/>
<topic name="events" max-messages="300" persistence-mode="all" subscription-type="remote"/>
</slow-subscriber-conditions>
<topics>
<topic name="feed"/>
<topic name="feed.bloomberg"/>
<topic name="feed.reuters"/>
<topic name="events"/>
</topics>
</swiftlet>
The example defines 2 conditions. The first is for the topic hierarchy 'feed' and matches for nonpersistent messages and local subscribers if the subscriber queue has reached 100 messages. The second condition is for topic hierarchy 'events', matches for all messages but only for remote subscriptions and permits max. 300 messages in the subscriber queue.
Note that for remote subscriptions the subscriber queue is the routing queue of the resp. router and thus the message count of that queue is taken.
Disconnecting Non-Durable Subscribers
SwiftMQ 5.2.0 onwards adds an attribute
disconnect-non-durable-subscriber to a slow subscriber condition. The
default value is false. If it is set to true, all non-durable subscriber
connections for which the condition matches will be removed. This option
is useful if there are malfunctioning JMS clients which have topic
subscriptions.
Disconnecting Durable Subscribers and Delete Durable Subscriptions
SwiftMQ 5.2.0 onwards adds another attribute
disconnect-delete-durable-subscriber to a slow subscriber condition.
The default value is false. If it is set to true, all durable subscriber
connections for which the condition matches will be removed and their
durable subscriptions will be deleted. That is, all messages in their
durable subscriber queue will be lost! So be careful with this option.
This option is useful if there are malfunctioning JMS clients which have
durable topic subscriptions or if there are "dead" durable subscription
on a topic (a JMS client created a durable subscription but doesn't use
it anymore so the durable subscriber queue grows and grows).
Publish/Subscribe in a Federated Router Network
Dynamic Exchange of remote Subscriptions
All Topic Manager Swiftlets in a router network are working federated.
They exchange subscription messages on a "some/none" base. That is, they
send a subscription message to all other Topic Manager Swiftlets if they
have at least 1 subscription for a particular topic and they send an
unsubscription message around if the count reaches zero. The
subscription exchange is made on base of root topics. A root topic is
the first node in a topic hierarchie, e.g. the root topic of topic
'iit.sales.eu' is 'iit'. For a local Topic Manager Swiftlet these remote
subscriptions are just normal subscriptions. The only difference is that
the subscriber queue is a remote queue tpc$<roottopic> at the
particular remote router.
Remote subscriptions are listed in the Topic Manager's Usage section
below node "Remote Router Subscriptions":
Each router node consists of 1 or more root topics for which the particular router has at least 1 subscription.
Usually this dynamic remote subscriptions are only informational. However, there is one case where an administrator has to delete this remote subscriptions: to put a node of a federated router network out of service.
If a node should be removed from a federated router network, this can be done by shutting down its JMS clients which in turn remove all their subscriptions which forces the router node to send an unsubscription notification to all other routers when the last local JMS client unsubscribes from this particular topic. The last action is to shut down the router.
But if the router node did not had an active routing connection to one or more of its neighbours, they will never receive the unsubscription notification and will continously send pub/sub messages for the router's topic subscriptions. In this case an administrator has to delete the resp. remote router subscription entry:
This must be done on all router nodes of the federated router network.
Static Remote Subscriptions
In a federated router network, subscriptions are exchanged dynamically between routers. If a router starts up, it has initially no knowledge about remote subscriptions. Once another router connects to this router, subscriptions are exchanged and messages are forwarded between the routers on base of these subscriptions. To enable message forwarding for times when a router has not yet remote subscriptions (it has just being started and no other router has connected yet), static remote subscriptions can be defined.
Note that, per default, static remote subscriptions are only valid
before the remote router connects the first time. Thereafter the
subscriptions are turning dynamically. So in the case the remote router
unsubscribes from a topic, no further messages are forwarded, even if
there is a static remote subscription for it. Starting with release
6.1.0 this behavior can be changed by enabling attribute
keep-on-unsubscribe of the static remote subscription. In that case,
dynamic unsubscription requests for that topic of that router are not
respected and messages are still forwarded. This is very important in
the case the remote router performs an orderly shutdown (e.g. for
maintenance) and has durable subscribers, because a router unsubscribes
from all topics at remote routers during shutdown. If
keep-on-unsubscribe is true, messages are still forwarded into the
routing queue and will be delivered once the remote router restarts. So
no messages are lost in the meantime. If keep-on-unsubscribe is set to
false (default), no messages are delivered while the remote router is
down and messages are lost.
Static remote subscriptions are defined on a per-router-base. It requires that a static route is defined to the particular remote router.
Example:
<swiftlet name="sys$routing">
<connectors/>
<filters/>
<listeners>
<listener name="plainsocket" port="4100">
<host-access-list/>
</listener>
</listeners>
<static-routes>
<static-route name="router2"/>
<static-route name="router3"/>
<static-route name="router4"/>
</static-routes>
</swiftlet>
Thereafter, static remote subscriptions can be defined under the Topic Manager Swiftlet configuration:
Example:
<swiftlet name="sys$topicmanager" flowcontrol-enabled="false">
<static-remote-router-subscriptions>
<static-remote-router-subscription name="router2">
<static-topic-subscriptions>
<static-topic-subscription name="testtopic"/>
</static-topic-subscriptions>
</static-remote-router-subscription>
<static-remote-router-subscription name="router3">
<static-topic-subscriptions>
<static-topic-subscription name="testtopic"/>
</static-topic-subscriptions>
</static-remote-router-subscription>
<static-remote-router-subscription name="router4">
<static-topic-subscriptions>
<static-topic-subscription name="iit" keep-on-unsubscribe="true"/>
<static-topic-subscription name="testtopic"/>
</static-topic-subscriptions>
</static-remote-router-subscription>
</static-remote-router-subscriptions>
<topics>
<topic name="testtopic"/>
<topic name="iit"/>
<topic name="iit.sales"/>
<topic name="iit.sales.eu"/>
<topic name="iit.sales.us"/>
<topic name="iit.sales.asia"/>
</topics>
</swiftlet>
Note that the 'name' attribute of an element 'static-remote-router-subscription' must contain the remote router name and the 'name' attribute of an element 'static-topic-subscription' must contain the root topic (the first node of a topic hierarchy; if you don't have a topic hierarchy then it's just the topic name).
The above configuration ensures that messages are forwarded to the topic hierarchy beginning with 'testtopic' to routers 'router2', 'router3', and 'router4'. Messages to the topic hierarchy beginning with 'iit' are also forwarded to 'router4'. Forwarding of messages for this topic continues even if the remote router has no subscriptions for it.
Jobs
These jobs can be scheduled via the Scheduler Swiftlet to run at specific times or in intervals, based on calenders and so on.
Delete Durable
The Delete Durable job deletes durable subscribers (incl. all messages)
which are selected by 2 SQL-Like predicates, client id and durable name,
specified as job parameters. For example, a client id predicate of
sales and a durable name predicate of % will delete all durable
subscribers of client id sales whereas a client id predicate of
s\_\_es% and a durable name predicate of dur% will delete those from
saukesta and dur2003. If you set both parameters to %, it will
match all durable subscribers and therefore all durable subscribers are
deleted.
| Parameter | Mandatory | Description |
|---|---|---|
| Client Id Predicate | Yes | SQL-Like predicate for client ids. |
| Durable Name Predicate | Yes | SQL-Like predicate for durable names. |
Configuration
The configuration of the Topic Manager Swiftlet is defined within the element
<swiftlet name="sys$topicmanager" .../>
of the router's configuration file. One can use the SwiftMQ Exlorer or CLI for configuration as well. They both save into that file.
Attributes of Element "swiftlet"
Definition
| Attribute | Type | Mandatory | Description |
|---|---|---|---|
| flowcontrol-enabled | java.lang.Boolean | No | Enable/Disable Publish/Subscribe Flow Control |
| direct-subscriber-selection | java.lang.Boolean | No | Select Subscribers directly and do NOT interpret the Publisher's Topicname as SQL-Like Predicate |
Values
| Attribute | Values |
|---|---|
| flowcontrol-enabled | Default: true |
| direct-subscriber-selection | Default: true |
Element List "topics", Parent Element: "swiftlet"
Topic Definitions. This element list contains zero or more "topic" elements with this template definition:
Definition
| Attribute | Type | Mandatory | Description |
|---|---|---|---|
| name | java.lang.String | Yes | Name of this Topic |
Element List "slow-subscriber-conditions", Parent Element: "swiftlet"
Slow Subscriber Conditions. This element list contains zero or more "topic" elements with this template definition:
Definition
| Attribute | Type | Mandatory | Description |
|---|---|---|---|
| name | java.lang.String | Yes | Name of this Root Topic |
| max-messages | java.lang.Long | Yes | Maximum Messages in Subscriber Queue |
| persistence-mode | java.lang.String | No | Persistence Mode |
| subscription-type | java.lang.String | No | Subscription Type |
| disconnect-non-durable-subscriber | java.lang.Boolean | No | Disconnect Non-Durable Subscriber |
| disconnect-delete-durable-subscriber | java.lang.Boolean | No | Disconnect and Delete Durable Subscriber |
Values
| Attribute | Values |
|---|---|
| max-messages | Min: 1 Default: 500 |
| persistence-mode | Choice: all non_persistent persistent Default: non_persistent |
| subscription-type | Choice: all local remote Default: all |
| disconnect-non-durable-subscriber | Default: false |
| disconnect-delete-durable-subscriber | Default: false |
Element List "static-remote-router-subscriptions", Parent Element: "swiftlet"
Static Remote Router Subscriptions. This element list contains zero or more "static-remote-router-subscription" elements with this template definition:
Definition
| Attribute | Type | Mandatory | Description |
|---|---|---|---|
| name | java.lang.String | Yes | Name of this Static Remote Router Subscription |
Element List "static-topic-subscriptions", Parent Element: "static-remote-router-subscription"
Static Topic Subscriptions. This element list contains zero or more "static-topic-subscription" elements with this template definition:
Definition
| Attribute | Type | Mandatory | Description |
|---|---|---|---|
| name | java.lang.String | Yes | Name of this Static Topic Subscription |
| keep-on-unsubscribe | java.lang.Boolean | No | Keep this Subscription even when the remote Router unsubscribes it |
Values
| Attribute | Values |
|---|---|
| keep-on-unsubscribe | Default: false |
Element "usage", Parent Element: "swiftlet"
Active Topics.
Element List "durables", Parent Element: "usage"
Active Durable Topic Subscriber. This element list contains zero or more "durables" elements with this template definition:
Definition
| Attribute | Type | Mandatory | Description |
|---|---|---|---|
| name | java.lang.String | Yes | Name of this Active Durable Topic Subscriber |
| durablename | java.lang.String | Yes | Name of the durable Subscriber |
| clientid | java.lang.String | Yes | Client Id |
| topic | java.lang.String | Yes | Topic Name |
| selector | java.lang.String | No | Message Selector |
| nolocal | java.lang.Boolean | No | Receives no local published Messages |
| boundto | java.lang.String | No | Is bound to Queue |
Values
| Attribute | Values |
|---|---|
| durablename | |
| clientid | |
| topic | |
| selector | |
| nolocal | Default: false |
| boundto | Default: Do not specify - will be set from SwiftMQ! |
Element List "subscriber", Parent Element: "usage"
Active Topic Subscriber. This element list contains zero or more "subscriber" elements with this template definition:
Definition
| Attribute | Type | Mandatory | Description |
|---|---|---|---|
| name | java.lang.String | Yes | Name of this An Active Topic Subscriber |
| clientid | java.lang.String | No | Client Id |
| topic | java.lang.String | No | Topic Name |
| selector | java.lang.String | No | Message Selector |
| nolocal | java.lang.Boolean | No | Receives no local published Messages |
| boundto | java.lang.String | No | Is bound to Queue |
Values
| Attribute | Values |
|---|---|
| clientid | |
| topic | |
| selector | |
| nolocal | Default: false |
| boundto |
Element List "subscriber-remote", Parent Element: "usage"
Subscriptions from remote Routers. This element list contains zero or more "router" elements with this template definition:
Definition
| Attribute | Type | Mandatory | Description |
|---|---|---|---|
| name | java.lang.String | Yes | Name of this Remote Router |
Element List "topics", Parent Element: "router"
Subscriptions from this Router. This element list contains zero or more "topic" elements with this template definition:
Definition
| Attribute | Type | Mandatory | Description |
|---|---|---|---|
| name | java.lang.String | Yes | Name of this Topic |