Routing between routers of a network requires information about the routes. These routes are exchanged dynamically between each router of the network during the connect phase. That means, each router is able to send messages to each other since it knows every route there. If a router connects another router, it has the direct route to this remote router and gets all routes from here to other routers. Therefore, a message can be routed to a router which is not directly connected but reachable by other intermediate routers.
The Routing Swiftlet stores a table with all dynamic routes, called "routing table", which is displayed below the "Usage" node of the Routing Swiftlet:
The routing table contains an entry for each known destination router. Each of this entries contains a table of dynamic routes, received by this router. A dynamic route consists of a list of intermediate routers (hops) which a message has to travel until it reaches its final destination.
Per default, always the shortest way is choosen by the Routing Swiftlet during messages transfer, that is the route with the minimum hop count. This is the default routing mode. Therefore the message is routable if at least 1 route to its destination is known, because if a route fails, the next best route is choosen and the message travels another, probably longer way. Another routing mode is "round-robin" which has to be enabled separately. Within this mode, messages are distributed over all known routes to its destination evenly. Round-robin is reasonable to spread a message load over multiple routing connections.
With the dynamic routing architecture, a remote queue can be addressed if a route is available. Otherwise the queue is unknown. To enable clients to address remote queues before a dynamic route is announced, static routes can be created. These are simply router names. Static routes are not announced to other routers. Their only sense is to enable clients to address unknown routers.
The advantage of dynamic routing is its plug-and-play nature. No administrators interaction is required. A message is routed automatically to the destination inclusive failover. However, the disadvantage is the amount of routes which need to be exchanged. Every router knows every route to every other router. So, the more routers the more routes are exchanged. Updated routes are exchanged whenever a routing connection somewhere in a router network connects or disconnects. To reduce this traffic, route filters can be defined.
The following example shows a typical star-like router network topology with 2 interconnected main routers, "hq1" and "hq2". Each main router has 3 interconnected satellite routers. There is an additional connection between "sub3" and "sub4":
If a message should be send from "sub6" to a queue at "sub1", it will by default use the shortest way (minimum hop count): sub6 -> hq2 -> hq1 -> sub1. However, there are many other routes as well, for example: sub6 -> hq2 -> sub5 -> sub4 -> sub3 -> hq1 -> sub2 -> sub1.
Many of this routes don't make sense and will never be used. It may also be required to restrict access between routers in a network. There are 2 way to accomplish this: to define a hop limit at a router and/or to define route filters.
SwiftMQ 5.0.2 introduces a new attribute called "route-announce-hop-limit" to limit the routes which are announced from a router to the value specified via this attribute. The router checks each route if the number of hops exceeds the attribute value and only forwards it if the hop count is less. The default value of "route-announce-hop-limit" is 3. A value of -1 disabled the hop limit.
In the above example, if we define for each satellite router "sub1" to "sub6" the route-announce-hop-limit=1 then we would have direct routes (via the direct connection) between them and their correspondend main router "hq1" resp. "hq2". However, since the main routers don't have a hop limit, there were an alternate route for each satellite via it's main router. For example, "sub6" could reach "sub5" directly but also via "hq2". "sub6" could also reach "sub1" via "hq2" and "hq1" because the main router don't have a hop limit. However, all nonsense routes like sub6 -> hq2 -> hq1 -> sub2 -> sub3 -> sub4 -> sub5 are elimited by the hop limit.
If we define route-announce-hop-limit=1 for the main routers "hq1" and "hq2" as well then each satellite could only communicate with its direct neighbor satellite and its main router. Both main routers could communicate as well but not with the satellite of each other. "sub6" could reach "sub5" but not "sub4" and not "sub1". The only exception is that "sub3" and "sub4" can reach each other because they have a direct connection.
Please check the routing table (see picture on top of this page) of your routers while you define your hop limits. Routing connections have to be re-established to force re-announcement of routes.
Route filters provide a way for semantic filtering of routes. They can be hop or destination based and of the include or exclude type. In this context a hop defines a router node to which other routers are connected and which announces routes to the outside. A destination is a single router. Possible filters are:
In the above example, let's define a filter at "sub4" for router "sub3" of type "include by destination" with the value "sub4". This must be read "to router sub3 announce only routes to destination sub4". Since the only route which has a destination "sub4" is the direct route this will filter all routes except the direct route. If we define a similar route at "sub3" (for "sub4", type "include by destination", value "sub3") then both routers can reach each other via their direct connection but not any other router via this direct connection.
If we would further add a filter at "hq2" of type "exclude by hop" with value "hq1" then "sub4" to "sub6" wouldn't be able to reach "hq1", "sub1" to "sub3", except that "sub3" and "sub4" could communicate directly. So "exclude by hop" excludes a complete subnet, identified by a hop. All other routes are announced. If one would like to exclude all routes except those from a particular hop then "include by hop" must be used.
Please check the routing table (see picture on top of this page) of your routers while you define your route filters. Routing connections have to be re-established to force re-announcement of routes.
The exchange of messages between routers takes place in form of XA transactions. An XA transaction is a transaction under control of SwiftMQ's transaction manager, utilizing the 2-phase-commit protocol to move it from one router to the other.
An XA transaction has a specific size of messages, configurable via the attribute "inbound-transaction-size" on both the listener and the connector. Therefore, it is possible to specify different transaction sizes for both directions. "inbound" means that this is the size a router accepts inbound. It is passed to the sending side during the connect. The Routing Swiftlet tries to load up that size messages from the corresponding routing queue and sends it as a single XA transaction to the other router. The default transaction size is 20 messages. The higher the size, the higher the throughput. Determining the proper size is dependent on the message size and on the connection speed. Small messages and high-speed connections should have a higher transaction size, e.g. 500 messages, whereas large messages on a low-speed connection should have a lower transaction size, e.g. 1. The default size of 20 fits usual environments.
2-phase-commit runs completely asynchronous. That is, the transfer of new transactions takes place while previous transactions are prepared or committed in parallel and in both directions. This guarantees high-speed XA. There is another attribute "inbound-window-size" for both, listeners and connectors, to specify the maximum of open XA transactions, that is, uncommitted XA transactions. The sender side of the routing connection sends new XA transactions until the number of open XA transactions reaches this window size. Thereafter it waits until it has a window again (means, one or more XA transactions are committed) and continues sending. The higher the window size, the more XA transactions can be handled asynchronously and the higher the throughput. However, the more open XA transactions, the longer takes recovery during startup/reboot of the router. Therefore, the default value is 10 which is useful.
If a router is shut down, it might have many open XA transactions, dependent on the number of active routing connections and the configured window sizes. Like JMS XA transactions, these are handled by the XA Resource Manager Swiftlet. It checks the XA log during startup and eventually displays a warning message like this:
Example:
Booting SwiftMQ 4.0.0 Production ... ... startup: Trace Swiftlet ... startup: Log Swiftlet ... startup: Authentication Swiftlet ... startup: Threadpool Swiftlet ... startup: Timer Swiftlet ... startup: Network Swiftlet ... startup: Store Swiftlet ... startup: Queue Manager Swiftlet ... startup: Topic Manager Swiftlet ... startup: Management Swiftlet ... startup: XA Resource Manager Swiftlet +++ WARNING! 10 prepared transactions found! +++ Routing XA transactions are automatically recovered. +++ You may also use Explorer/CLI for commit or rollback. ... startup: Routing Swiftlet (Unlimited Connections) ... startup: JNDI Swiftlet ... startup: JMS Swiftlet (XA/ASF) ... startup: Deploy Swiftlet SwiftMQ 4.0.0 Production is ready.
Don't worry! These open (prepared) XA transactions are automatically recovered once the corresponding router connects again. If that will not be the case, e.g. due to changes in the router network topology, the SwiftMQ Explorer or CLI can be used to commit or rollback these open XA transactions manually. See the XA Resource Manager Swiftlet documentation for details.
Flow control is an important instrument to avoid overloading a router with messages. SwiftMQ generates flow control delays on every queue during commit of a producer. The flow control delay is a computed estimate how long it takes to "eat up" queue messages with the current production and consumption rate. If the flow control is respected, an optimal throughput flow will be established which holds the messages of a queue just below the maximum cache size. Therefore, the throughput is maximum at the current consumption rate. During routing, messages are produced to queues during inbound processing. On each XA commit, a flow control delay is returned. An attribute "inbound-flow-control-enabled" can be set to true or false. Setting to true, the flow control is respected and further XA commits are delayed, depending on the flow control delays. These delays are cascaded through the whole router network back to the producer client if each hop on the route has "inbound-flow-control-enabled" set to true. If it is set to false (default), flow control delays are not respected and XA commits are not delayed. So in the worst case, a router is flooded with messages.