MQTT Swiftlet

MQTT stands for MQ Telemetry Transport. It is a publish/subscribe, extremely simple and lightweight messaging protocol, designed for constrained devices and low-bandwidth, high-latency or unreliable networks. The design principles are to minimise network bandwidth and device resource requirements whilst also attempting to ensure reliability and some degree of assurance of delivery. These principles also turn out to make the protocol ideal of the emerging "machine-to-machine" (M2M) or "Internet of Things" world of connected devices, and for mobile applications where bandwidth and battery power are at a premium.

(Description from mqtt.org)

Features

The MQTT Swiftlet provides a full-featured MQTT 3.1/3.1.1 broker implementation. The following MQTT features are supported:

• QoS 0 (at most once), QoS 1 (at least once), QoS 2 (exactly once).
• Topic wildcard filters.
• Retained messages.
• Last Will and Testament.
• Clean and persistent sessions.
• Session takeover.
• Packet replay on reconnect.
• Keep Alive.

Additional features provided by the MQTT Swiftlet:

• Full interoperability between MQTT, AMQP 1.0/0.9.1 and JMS clients.
• Topic order guarantees for all QoS levels.
• Persistent session timeout and automatic cleanup.
• Administrative deletion of persistent sessions.
• Plain and TLS connections.

Each feature is explained in more detail in the following chapters.

Quality of Service (QoS)

In MQTT the publisher of a message decides about the importance of a message by specifying the QoS in the publish package. The QoS levels are as follows:

QoS	Meaning	Description
0	At Most Once	Messages can be lost.
1	At Least Once	Messages can't be lost but duplicates may occur.
2	Exactly Once	Messages are delivered once and only once.

The interaction between publisher and MQTT Swiftlet takes place according to the specified QoS level in the publish packet. A subscriber has it's own QoS specified per subscription. This QoS level is the maximum level the subscriber supports. The QoS level for each message is compared to the QoS of the subscriber and may be downgraded, if necessary. For example, if the publisher has specified QoS 2 and the subscriber only has QoS 1, the message is handled according to QoS 1. On the contrary, if the publisher specifies QoS 0 and the subscriber has QoS 2, QoS 0 is used.

Topic Wildcard Filters

In MQTT topics are structured as a hierarchy, for example devices/device1/temp. The foward slash / is the delimiter. A subscriber can define a topic filter that is applied to the messages which are delivered to the subscription if the filter matches the topic under which the messages have been published.

MQTT defines 2 wildcard characters: + addresses all messages of a hierarchy node and # addresses all messages of a node and all sub-nodes, thus # can only be specified as the last character of a filter.

MQTT topics are mapped to SwiftMQ topic predicates internally. SwiftMQ topics are a bit different as the topic delimiter is a . such as devices.device1.temp and the whole topic name can be a SQL LIKE predicate where the underscore _ matches any single character and the percent sign % any string. SwiftMQ does not allow to specify a wildcard for the root node of a topic hierarchy.

In consequence there are some limitation for MQTT topic filters:

MQTT Topic Filter	SwiftMQ Topic Predicate	Valid	Description
+	-	No	Filter at the root node is not allowed.
#	-	No	Filter at the root node is not allowed.
+/+/temp	-	No	Filter at the root node is not allowed.
/devices/+/temp	devices.%.temp	Yes
/devices/#	devices.%	Yes

Retained Messages

A publisher can flag a message as "retain". The last retained message per topic is sent to new subscriptions of that topic as the very first message. This is a useful feature to e.g. store and communicate the last state sent by a device without the need that the device is currently connected.

The MQTT Swiftlet stores retained messages in memory. So after a reboot of a SwiftMQ Router or a failover of a SwiftMQ High Availability Router these retained messages are lost.

Last Will and Testament (LWT)

A MQTT client can specify a LWT message in the connect packet. This message will be sent to a topic when the client unexpectedly disconnects without sending a disconnect packet. For example, the network connection is lost or the client is disconnected administratively via SwiftMQ Explorer/CLI.

LWT messages can also be marked as "retain". See the previous chapter.

Sessions

In MQTT a session represents the state of a connection. It contains:

• All topic subscriptions.
• A replay log of unacknowledged packets.

A MQTT client specifies a "clean session" flag in the connect packet. If that is set, a previous persistent session including all active subscriptions is deleted and a new (clean) session is created. This clean session is temporary and has a life time of that of the connection. All subscriptions of a clean session are non-durable (backed by a temporary queue).

If the "clean session" flag is not set, the session is persistent. All subscriptions of a persistent session are durable (backed by a durable subscriber queue) and the messages are stored persistently. If a client disconnects, the session is stored persistently (survives a reboot and failover). If a client reconnects and there is a persistent session stored for this client id, the replay log is first sent to the client to finish unacknowledged packets (the client does the same) and the subscriptions are active without the need to subscribe. Message delivery starts immediately.

In case the persistent session is associated with a connection with the same client id (e.g. the previous TCP connection is still alive due to half open sockets), this active connection is closed by the MQTT Swiftlet and the new connection is associated with the persistent session.

Keep Alive

A MQTT client can send ping request packets. This will be answered by a ping response packet by the MQTT Swiftlet. Each MQTT connection has a keepalive timer. If a connection did not sent a packet within 1.5 times of the value of the keepalive field in the connect packet, the connection will be closed by the MQTT Swiftlet.

Interoperability between MQTT, AMQP, JMS

SwiftMQ provides full interoperability between MQTT, AMQP and JMS clients.

The payload of a MQTT message is a stream of bytes. A message published to a topic is internally converted to a JMS BytesMessage. A subscription receives a JMS BytesMessage and converts it into a MQTT publish packet.

The resulting JMS BytesMessage from a MQTT publish contains the following message properties:

Name	Type	Value
JMSXUserID	String	Username of the Publisher.
JMS_SWIFTMQ_CID	String	Client id of the Publisher.
JMS_SWIFTMQ_MQTT_PUBQOS	Integer	QoS level of the Publisher.

These properties can be used by JMS/AMQP clients in message selectors. The properties have only informational purpose, except JMS_SWIFTMQ_MQTT_PUBQOS which is used by MQTT subscribers to determine the resulting QoS for the subscription. If JMS_SWIFTMQ_MQTT_PUBQOS is not set, the subscriber uses the QoS of the subscription.

Message Order Guarantees

The MQTT Swiftlet uses the pub/sub sub system of the SwiftMQ Router. Subscriptions are backed by durable or non-durable queues which guarantee message order independent of the QoS level.

Persistent Session Timeout and Cleanup

Persistent sessions have a configurable timeout (default: 1 week). If they were not associated with a connection during this time, they are deleted including all subscriptions.

Plain and TLS Connectiions

Per default, the MQTT Swiftlet has 2 configured listeners. One for plain TCP connections and one for TLS connections. TLS connections are served by Java JSSE included in the Java distribution.

Please have a look at this chapter how to configure the SwiftMQ Router. Configuration of the MQTT client is client specific.

Management

Listener Configuration

An MQTT listener listens on a specific port and accepts MQTT connections. Per default the MQTT Swiftlet defines 2 MQTT listeners:

• Port 1883: Listener for plain MQTT connections.
• Port 8883: Listener for TLS MQTT connections.

A listener is configured by a connection template which is referenced by its name. See next section.

A new listener on a different port can be created by selecting the Listeners entity, right click, Create a new Entity:

Connection Templates

Connection templates are pre-configured templates which are referenced from MQTT listeners. They are located under the Declarations entity:

Connection template default does not need to be created because it refers to a connection template with all default values. The above tls connection template is a default template with a different socket factory (JSSESocketFactory for TLS).

A new connection template can be created by selecting the Connection Templates entity, right click, Create a new Entity:

Session Timeout

The session timeout is the time of inactivity of a persistent session where it is not associated with a connection. Sessions and their subscriptions are deleted when they reach the timeout. The timeout is in hours (default: 168 hours = 1 week) and can be configured here:

Administratively delete an active MQTT Connection

MQTT connections can be deleted administratively this way:

Administratively delete a persistent Session

Persistent sessions can be deleted administratively this way:

The session to delete must not be associated with a connection, otherwise an error is shown.

Monitoring

Monitoring MQTT Connections

MQTT connections are located under the Usage section of the MQTT Swiftlet. Select it, right click, Show Entity Table is an alternate view of the same information:

Start some MQTT clients and expand the Usage section of the MQTT Swiftlet. The information shown is down to the subscriptions and message transfer per subscription can be oberserved:

To view it in an Entity Table frame, select the Subscriptions node, right click, Show Entity Table:

Monitoring Queues

Another view can be opened on the Queue Manager Swiftlet's Usage section. It shows the message throughput of the queues. Clean sessions use non-durable subscriber queues (tmp$...) while persistent sessions use durable subscriber queues (<clientid>$<number>) By selecting a queue and pushing the first button in the upper left corner, a message viewer will be opened and the queue messages can be inspected:

Monitoring Threadpools

The MQTT Swiftlet uses a single threadpool:

• mqtt.connection: Performs connection tasks and outbound writes.

The threadpool (and all others) can be oberserved by opening an Entity Table frame on the Threadpool Swiftlet's Usage section:

Tracing and Debugging

A SwiftMQ Router has an integrated tracing facility, the Trace Swiftlet. Kernel Swiftlets such as the MQTT Swiftlet are using the kernel trace space to send trace output (if the trace space is enabled) about the internal processing of the MQTT Swiftlet. The trace output goes to <swiftmqinstall>/data/trace/mqtt.trace by default. Tracing can be enabled/disabled dynamically through SwiftMQ admin tools.

Another trace space protocol is used to send information about incoming and outgoing MQTT packets and the output of the packet decoder. This is the more interesting trace space for debugging.

Trace space kernel contains various predicate entries for different Kernel Swiftlets. Every entry can be enabled (default) or disabled. The MQTT Swiftlet predicate is in entry 22 by default. To get trace output, enable the kernel trace space and make sure, predicate 22 is enabled too. Be aware that kernel tracing produces huge output and slows down the router.

To get MQTT protocol information only, change the single predicate of the protocol space to mqtt. Then enable the protocol space. The trace output goes to <swiftmqinstall>/data/trace/protocol.trace by default.

The following snippet shows how to enable the above tracing directly in the routerconfig.xml to have tracing enabled on startup:

<swiftlet name="sys$trace">
    <spaces>
        <space name="kernel" enabled="true">
            <predicates>
                ...
                <predicate name="22" filename="../data/trace/mqtt.trace" value="sys$mqtt"/>
                ...
            </predicates>
        </space>
        <space name="protocol" enabled="true">
            <predicates>
                <predicate name="1" filename="../data/trace/protocol.trace" value="mqtt"/>
            </predicates>
        </space>
        ...
    </spaces>
</swiftlet>

Configuration

The configuration of the MQTT Swiftlet is defined within the element

      <swiftlet name="sys$mqtt" .../>

of the router's configuration file. One can use the SwiftMQ Exlorer or CLI for configuration as well. They both save into that file.

Attributes of Element "swiftlet"

Definition

Attribute	Type	Mandatory	Description
collect-interval	java.lang.Long	No	Collect Interval Messages/Sec
session-timeout	java.lang.Long	No	Time in hours after which a unused Session is deleted

Values

Attribute	Values
collect-interval	Default: 1000
session-timeout	Default: 168

Element "declarations", Parent Element: "swiftlet"

Declarations Section.

Element List "connection-templates", Parent Element: "declarations"

Templates for Connections. This element list contains zero or more "connection-template" elements with this template definition:

Definition

Attribute	Type	Mandatory	Description
name	java.lang.String	Yes	Name of this Connection Template
socketfactory-class	java.lang.String	No	Socketfactory Class
use-tcp-no-delay	java.lang.Boolean	No	Use Tcp No Delay
idle-timeout	java.lang.Long	No	Inactivity timeout (ms) after which a Connection is disconnected
max-message-size	java.lang.Integer	No	Maximum Message Size
reject-disconnect-delay	java.lang.Long	No	Time (ms) after which a rejected Connection is closed
router-input-buffer-size	java.lang.Integer	No	Router Network Input Buffer Size
router-input-extend-size	java.lang.Integer	No	Router Network Input Extend Size
router-output-buffer-size	java.lang.Integer	No	Router Network Output Buffer Size
router-output-extend-size	java.lang.Integer	No	Router Network Output Extend Size

Values

Attribute	Values
socketfactory-class	Default: com.swiftmq.net.PlainSocketFactory
use-tcp-no-delay	Default: true
idle-timeout	Default: 90000
max-message-size	Min: 0 Max: 2147483647 Default: 10485760
reject-disconnect-delay	Min: 1000 Default: 5000
router-input-buffer-size	Min: 65536 Default: 131072
router-input-extend-size	Min: 65536 Default: 65536
router-output-buffer-size	Min: 65536 Default: 131072
router-output-extend-size	Min: 65536 Default: 65536

Element List "listeners", Parent Element: "swiftlet"

Listener Definitions. This element list contains zero or more "listener" elements with this template definition:

Definition

Attribute	Type	Mandatory	Description
name	java.lang.String	Yes	Name of this Listener
bindaddress	java.lang.String	No	Listener Bind IP Address
port	java.lang.Integer	Yes	Listener Port
max-connections	java.lang.Integer	Yes	Maximum Connections for Listener
connection-template	java.lang.String	Yes	Connection Template to use

Values

Attribute	Values
bindaddress
port	Default: 1883
max-connections	Default: -1
connection-template	Default: default

Element List "host-access-list", Parent Element: "listener"

Host Access List. This element list contains zero or more "host-access-entry" elements with this template definition:

Definition

Attribute	Type	Mandatory	Description
name	java.lang.String	Yes	Name of this Host Access Entry

Element "usage", Parent Element: "swiftlet"

Live Usage.

Element List "connections", Parent Element: "usage"

Active MQTT Connections. This element list contains zero or more "usage" elements with this template definition:

Definition

Attribute	Type	Mandatory	Description
name	java.lang.String	Yes	Name of this Active MQTT Connection
client-id	java.lang.String	No	Client Id
connect-time	java.lang.String	No	Connect Time
msgs-received	java.lang.Integer	No	Messages/Sec received from Connection
msgs-sent	java.lang.Integer	No	Messages/Sec sent to Connection
total-received	java.lang.Integer	No	Total Number of Messages received from Connection
total-sent	java.lang.Integer	No	Total Number of Messages sent to Connection
username	java.lang.String	No	User Name
mqtt-protlevel	java.lang.Integer	No	MQTT Protocol Level [4 = 3.1.1]

Values

Attribute	Values
client-id
connect-time
msgs-received	Default: 0
msgs-sent	Default: 0
total-received	Default: 0
total-sent	Default: 0
username
mqtt-protlevel

Element "MQTT Session", Parent Element: "usage"

MQTT Session.

Definition

Attribute	Type	Mandatory	Description
persistent	java.lang.Boolean	No	Persistent Session

Values

Attribute	Values
persistent	Default: false

Element List "subscriptions", Parent Element: "MQTT Session"

Topic Subscriptions. This element list contains zero or more "Topic Subscription" elements with this template definition:

Definition

Attribute	Type	Mandatory	Description
name	java.lang.String	Yes	Name of this Topic Subscription
swiftmq-topic	java.lang.String	No	Translated SwiftMQ Topic Name
qos	java.lang.Integer	No	Quality of Service
msgs-received	java.lang.Integer	No	Messages/Sec received from Subscription
total-received	java.lang.Integer	No	Total Number of Messages received from Subscription

Values

Attribute	Values
swiftmq-topic
qos	Min: 0 Max: 2 Default: 0
msgs-received	Default: 0
total-received	Default: 0

Element List "sessions", Parent Element: "usage"

Persistent MQTT Sessions. This element list contains zero or more "MQTT Session" elements with this template definition:

Definition

Attribute	Type	Mandatory	Description
name	java.lang.String	Yes	Name of this MQTT Session
associated	java.lang.Boolean	No	Associated with a Connection
persistent	java.lang.Boolean	No	Persistent Session

Values

Attribute	Values
associated	Default: false
persistent	Default: false

Element List "subscriptions", Parent Element: "MQTT Session"

Topic Subscriptions. This element list contains zero or more "Topic Subscription" elements with this template definition:

Definition

Attribute	Type	Mandatory	Description
name	java.lang.String	Yes	Name of this Topic Subscription
swiftmq-topic	java.lang.String	No	Translated SwiftMQ Topic Name
qos	java.lang.Integer	No	Quality of Service

Values

Attribute	Values
swiftmq-topic
qos	Min: 0 Max: 2 Default: 0