MQTT Swiftlet
MQTT stands for MQ Telemetry Transport. It is a publish/subscribe, extremely simple and lightweight messaging protocol, designed for constrained devices and low-bandwidth, high-latency or unreliable networks. The design principles are to minimise network bandwidth and device resource requirements whilst also attempting to ensure reliability and some degree of assurance of delivery. These principles also turn out to make the protocol ideal of the emerging "machine-to-machine" (M2M) or "Internet of Things" world of connected devices, and for mobile applications where bandwidth and battery power are at a premium.
(Description from mqtt.org)
Features
The MQTT Swiftlet provides a full-featured MQTT 3.1/3.1.1 broker implementation. The following MQTT features are supported:
- QoS 0 (at most once), QoS 1 (at least once), QoS 2 (exactly once).
- Topic wildcard filters.
- Retained messages.
- Last Will and Testament.
- Clean and persistent sessions.
- Session takeover.
- Packet replay on reconnect.
- Keep Alive.
Additional features provided by the MQTT Swiftlet:
- Full interoperability between MQTT, AMQP 1.0/0.9.1 and JMS clients.
- Topic order guarantees for all QoS levels.
- Persistent session timeout and automatic cleanup.
- Administrative deletion of persistent sessions.
- Plain and TLS connections.
Each feature is explained in more detail in the following chapters.
Quality of Service (QoS)
In MQTT the publisher of a message decides about the importance of a message by specifying the QoS in the publish package. The QoS levels are as follows:
QoS | Meaning | Description |
---|---|---|
0 | At Most Once | Messages can be lost. |
1 | At Least Once | Messages can't be lost but duplicates may occur. |
2 | Exactly Once | Messages are delivered once and only once. |
The interaction between publisher and MQTT Swiftlet takes place according to the specified QoS level in the publish packet. A subscriber has it's own QoS specified per subscription. This QoS level is the maximum level the subscriber supports. The QoS level for each message is compared to the QoS of the subscriber and may be downgraded, if necessary. For example, if the publisher has specified QoS 2 and the subscriber only has QoS 1, the message is handled according to QoS 1. On the contrary, if the publisher specifies QoS 0 and the subscriber has QoS 2, QoS 0 is used.
Topic Wildcard Filters
In MQTT topics are structured as a hierarchy, for example
devices/device1/temp
. The foward slash /
is the delimiter. A
subscriber can define a topic filter that is applied to the messages
which are delivered to the subscription if the filter matches the topic
under which the messages have been published.
MQTT defines 2 wildcard characters: +
addresses all messages of a
hierarchy node and #
addresses all messages of a node and all
sub-nodes, thus #
can only be specified as the last character of a
filter.
MQTT topics are mapped to SwiftMQ topic predicates internally. SwiftMQ
topics are a bit different as the topic delimiter is a .
such as
devices.device1.temp
and the whole topic name can be a SQL LIKE
predicate where the underscore _
matches any single character and the
percent sign %
any string. SwiftMQ does not allow to specify a
wildcard for the root node of a topic hierarchy.
In consequence there are some limitation for MQTT topic filters:
MQTT Topic Filter | SwiftMQ Topic Predicate | Valid | Description |
---|---|---|---|
+ | - | No | Filter at the root node is not allowed. |
# | - | No | Filter at the root node is not allowed. |
+/+/temp | - | No | Filter at the root node is not allowed. |
/devices/+/temp | devices.%.temp | Yes | |
/devices/# | devices.% | Yes |
Retained Messages
A publisher can flag a message as "retain". The last retained message per topic is sent to new subscriptions of that topic as the very first message. This is a useful feature to e.g. store and communicate the last state sent by a device without the need that the device is currently connected.
The MQTT Swiftlet stores retained messages in memory. So after a reboot of a SwiftMQ Router or a failover of a SwiftMQ High Availability Router these retained messages are lost.
Last Will and Testament (LWT)
A MQTT client can specify a LWT message in the connect packet. This message will be sent to a topic when the client unexpectedly disconnects without sending a disconnect packet. For example, the network connection is lost or the client is disconnected administratively via SwiftMQ Explorer/CLI.
LWT messages can also be marked as "retain". See the previous chapter.
Sessions
In MQTT a session represents the state of a connection. It contains:
- All topic subscriptions.
- A replay log of unacknowledged packets.
A MQTT client specifies a "clean session" flag in the connect packet. If that is set, a previous persistent session including all active subscriptions is deleted and a new (clean) session is created. This clean session is temporary and has a life time of that of the connection. All subscriptions of a clean session are non-durable (backed by a temporary queue).
If the "clean session" flag is not set, the session is persistent. All subscriptions of a persistent session are durable (backed by a durable subscriber queue) and the messages are stored persistently. If a client disconnects, the session is stored persistently (survives a reboot and failover). If a client reconnects and there is a persistent session stored for this client id, the replay log is first sent to the client to finish unacknowledged packets (the client does the same) and the subscriptions are active without the need to subscribe. Message delivery starts immediately.
In case the persistent session is associated with a connection with the same client id (e.g. the previous TCP connection is still alive due to half open sockets), this active connection is closed by the MQTT Swiftlet and the new connection is associated with the persistent session.
Keep Alive
A MQTT client can send ping request packets. This will be answered by a ping response packet by the MQTT Swiftlet. Each MQTT connection has a keepalive timer. If a connection did not sent a packet within 1.5 times of the value of the keepalive field in the connect packet, the connection will be closed by the MQTT Swiftlet.
Interoperability between MQTT, AMQP, JMS
SwiftMQ provides full interoperability between MQTT, AMQP and JMS clients.
The payload of a MQTT message is a stream of bytes. A message published to a topic is internally converted to a JMS BytesMessage. A subscription receives a JMS BytesMessage and converts it into a MQTT publish packet.
The resulting JMS BytesMessage from a MQTT publish contains the following message properties:
Name | Type | Value |
---|---|---|
JMSXUserID | String | Username of the Publisher. |
JMS_SWIFTMQ_CID | String | Client id of the Publisher. |
JMS_SWIFTMQ_MQTT_PUBQOS | Integer | QoS level of the Publisher. |
These properties can be used by JMS/AMQP clients in message selectors. The properties have only informational purpose, except JMS_SWIFTMQ_MQTT_PUBQOS which is used by MQTT subscribers to determine the resulting QoS for the subscription. If JMS_SWIFTMQ_MQTT_PUBQOS is not set, the subscriber uses the QoS of the subscription.
Message Order Guarantees
The MQTT Swiftlet uses the pub/sub sub system of the SwiftMQ Router. Subscriptions are backed by durable or non-durable queues which guarantee message order independent of the QoS level.
Persistent Session Timeout and Cleanup
Persistent sessions have a configurable timeout (default: 1 week). If they were not associated with a connection during this time, they are deleted including all subscriptions.
Plain and TLS Connectiions
Per default, the MQTT Swiftlet has 2 configured listeners. One for plain TCP connections and one for TLS connections. TLS connections are served by Java JSSE included in the Java distribution.
Please have a look at this chapter how to configure the SwiftMQ Router. Configuration of the MQTT client is client specific.
Management
Listener Configuration
An MQTT listener listens on a specific port and accepts MQTT connections. Per default the MQTT Swiftlet defines 2 MQTT listeners:
- Port 1883: Listener for plain MQTT connections.
- Port 8883: Listener for TLS MQTT connections.
A listener is configured by a connection template which is referenced by its name. See next section.
A new listener on a different port can be created by selecting the
Listeners
entity, right click, Create a new Entity
:
Connection Templates
Connection templates are pre-configured templates which are referenced
from MQTT listeners. They are located under the Declarations
entity:
Connection template default
does not need to be created because it
refers to a connection template with all default values. The above tls
connection template is a default template with a different socket
factory (JSSESocketFactory for TLS).
A new connection template can be created by selecting the Connection Templates
entity, right click, Create a new Entity
:
Session Timeout
The session timeout is the time of inactivity of a persistent session where it is not associated with a connection. Sessions and their subscriptions are deleted when they reach the timeout. The timeout is in hours (default: 168 hours = 1 week) and can be configured here:
Administratively delete an active MQTT Connection
MQTT connections can be deleted administratively this way:
Administratively delete a persistent Session
Persistent sessions can be deleted administratively this way:
The session to delete must not be associated with a connection, otherwise an error is shown.
Monitoring
Monitoring MQTT Connections
MQTT connections are located under the Usage
section of the MQTT
Swiftlet. Select it, right click, Show Entity Table
is an alternate
view of the same information:
Start some MQTT clients and expand the Usage
section of the MQTT
Swiftlet. The information shown is down to the subscriptions and message
transfer per subscription can be oberserved:
To view it in an Entity Table frame, select the Subscriptions
node,
right click, Show Entity Table
:
Monitoring Queues
Another view can be opened on the Queue Manager Swiftlet's Usage
section. It shows the message throughput of the queues. Clean sessions
use non-durable subscriber queues (tmp$...) while persistent sessions
use durable subscriber queues (<clientid>$<number>
) By
selecting a queue and pushing the first button in the upper left corner,
a message viewer will be opened and the queue messages can be inspected:
Monitoring Threadpools
The MQTT Swiftlet uses a single threadpool:
mqtt.connection
: Performs connection tasks and outbound writes.
The threadpool (and all others) can be oberserved by opening an Entity
Table frame on the Threadpool Swiftlet's Usage
section:
Tracing and Debugging
A SwiftMQ Router has an integrated tracing facility, the Trace
Swiftlet. Kernel Swiftlets such as the MQTT Swiftlet are using the
kernel
trace space to send trace output (if the trace space is
enabled) about the internal processing of the MQTT Swiftlet. The trace
output goes to <swiftmqinstall>/data/trace/mqtt.trace
by
default. Tracing can be enabled/disabled dynamically through SwiftMQ
admin tools.
Another trace space protocol
is used to send information about
incoming and outgoing MQTT packets and the output of the packet decoder.
This is the more interesting trace space for debugging.
Trace space kernel
contains various predicate entries for different
Kernel Swiftlets. Every entry can be enabled (default) or disabled. The
MQTT Swiftlet predicate is in entry 22
by default. To get trace
output, enable the kernel
trace space and make sure, predicate 22
is
enabled too. Be aware that kernel tracing produces huge output and slows
down the router.
To get MQTT protocol information only, change the single predicate of
the protocol
space to mqtt
. Then enable the protocol
space. The
trace output goes to <swiftmqinstall>/data/trace/protocol.trace
by default.
The following snippet shows how to enable the above tracing directly in
the routerconfig.xml
to have tracing enabled on startup:
<swiftlet name="sys$trace">
<spaces>
<space name="kernel" enabled="true">
<predicates>
...
<predicate name="22" filename="../data/trace/mqtt.trace" value="sys$mqtt"/>
...
</predicates>
</space>
<space name="protocol" enabled="true">
<predicates>
<predicate name="1" filename="../data/trace/protocol.trace" value="mqtt"/>
</predicates>
</space>
...
</spaces>
</swiftlet>
Configuration
The configuration of the MQTT Swiftlet is defined within the element
<swiftlet name="sys$mqtt" .../>
of the router's configuration file. One can use the SwiftMQ Exlorer or CLI for configuration as well. They both save into that file.
Attributes of Element "swiftlet"
Definition
Attribute | Type | Mandatory | Description |
---|---|---|---|
collect-interval | java.lang.Long | No | Collect Interval Messages/Sec |
session-timeout | java.lang.Long | No | Time in hours after which a unused Session is deleted |
Values
Attribute | Values |
---|---|
collect-interval | Default: 1000 |
session-timeout | Default: 168 |
Element "declarations", Parent Element: "swiftlet"
Declarations Section.
Element List "connection-templates", Parent Element: "declarations"
Templates for Connections. This element list contains zero or more "connection-template" elements with this template definition:
Definition
Attribute | Type | Mandatory | Description |
---|---|---|---|
name | java.lang.String | Yes | Name of this Connection Template |
socketfactory-class | java.lang.String | No | Socketfactory Class |
use-tcp-no-delay | java.lang.Boolean | No | Use Tcp No Delay |
idle-timeout | java.lang.Long | No | Inactivity timeout (ms) after which a Connection is disconnected |
max-message-size | java.lang.Integer | No | Maximum Message Size |
reject-disconnect-delay | java.lang.Long | No | Time (ms) after which a rejected Connection is closed |
router-input-buffer-size | java.lang.Integer | No | Router Network Input Buffer Size |
router-input-extend-size | java.lang.Integer | No | Router Network Input Extend Size |
router-output-buffer-size | java.lang.Integer | No | Router Network Output Buffer Size |
router-output-extend-size | java.lang.Integer | No | Router Network Output Extend Size |
Values
Attribute | Values |
---|---|
socketfactory-class | Default: com.swiftmq.net.PlainSocketFactory |
use-tcp-no-delay | Default: true |
idle-timeout | Default: 90000 |
max-message-size | Min: 0 Max: 2147483647 Default: 10485760 |
reject-disconnect-delay | Min: 1000 Default: 5000 |
router-input-buffer-size | Min: 65536 Default: 131072 |
router-input-extend-size | Min: 65536 Default: 65536 |
router-output-buffer-size | Min: 65536 Default: 131072 |
router-output-extend-size | Min: 65536 Default: 65536 |
Element List "listeners", Parent Element: "swiftlet"
Listener Definitions. This element list contains zero or more "listener" elements with this template definition:
Definition
Attribute | Type | Mandatory | Description |
---|---|---|---|
name | java.lang.String | Yes | Name of this Listener |
bindaddress | java.lang.String | No | Listener Bind IP Address |
port | java.lang.Integer | Yes | Listener Port |
max-connections | java.lang.Integer | Yes | Maximum Connections for Listener |
connection-template | java.lang.String | Yes | Connection Template to use |
Values
Attribute | Values |
---|---|
bindaddress | |
port | Default: 1883 |
max-connections | Default: -1 |
connection-template | Default: default |
Element List "host-access-list", Parent Element: "listener"
Host Access List. This element list contains zero or more "host-access-entry" elements with this template definition:
Definition
Attribute | Type | Mandatory | Description |
---|---|---|---|
name | java.lang.String | Yes | Name of this Host Access Entry |
Element "usage", Parent Element: "swiftlet"
Live Usage.
Element List "connections", Parent Element: "usage"
Active MQTT Connections. This element list contains zero or more "usage" elements with this template definition:
Definition
Attribute | Type | Mandatory | Description |
---|---|---|---|
name | java.lang.String | Yes | Name of this Active MQTT Connection |
client-id | java.lang.String | No | Client Id |
connect-time | java.lang.String | No | Connect Time |
msgs-received | java.lang.Integer | No | Messages/Sec received from Connection |
msgs-sent | java.lang.Integer | No | Messages/Sec sent to Connection |
total-received | java.lang.Integer | No | Total Number of Messages received from Connection |
total-sent | java.lang.Integer | No | Total Number of Messages sent to Connection |
username | java.lang.String | No | User Name |
mqtt-protlevel | java.lang.Integer | No | MQTT Protocol Level [4 = 3.1.1] |
Values
Attribute | Values |
---|---|
client-id | |
connect-time | |
msgs-received | Default: 0 |
msgs-sent | Default: 0 |
total-received | Default: 0 |
total-sent | Default: 0 |
username | |
mqtt-protlevel |
Element "MQTT Session", Parent Element: "usage"
MQTT Session.
Definition
Attribute | Type | Mandatory | Description |
---|---|---|---|
persistent | java.lang.Boolean | No | Persistent Session |
Values
Attribute | Values |
---|---|
persistent | Default: false |
Element List "subscriptions", Parent Element: "MQTT Session"
Topic Subscriptions. This element list contains zero or more "Topic Subscription" elements with this template definition:
Definition
Attribute | Type | Mandatory | Description |
---|---|---|---|
name | java.lang.String | Yes | Name of this Topic Subscription |
swiftmq-topic | java.lang.String | No | Translated SwiftMQ Topic Name |
qos | java.lang.Integer | No | Quality of Service |
msgs-received | java.lang.Integer | No | Messages/Sec received from Subscription |
total-received | java.lang.Integer | No | Total Number of Messages received from Subscription |
Values
Attribute | Values |
---|---|
swiftmq-topic | |
qos | Min: 0 Max: 2 Default: 0 |
msgs-received | Default: 0 |
total-received | Default: 0 |
Element List "sessions", Parent Element: "usage"
Persistent MQTT Sessions. This element list contains zero or more "MQTT Session" elements with this template definition:
Definition
Attribute | Type | Mandatory | Description |
---|---|---|---|
name | java.lang.String | Yes | Name of this MQTT Session |
associated | java.lang.Boolean | No | Associated with a Connection |
persistent | java.lang.Boolean | No | Persistent Session |
Values
Attribute | Values |
---|---|
associated | Default: false |
persistent | Default: false |
Element List "subscriptions", Parent Element: "MQTT Session"
Topic Subscriptions. This element list contains zero or more "Topic Subscription" elements with this template definition:
Definition
Attribute | Type | Mandatory | Description |
---|---|---|---|
name | java.lang.String | Yes | Name of this Topic Subscription |
swiftmq-topic | java.lang.String | No | Translated SwiftMQ Topic Name |
qos | java.lang.Integer | No | Quality of Service |
Values
Attribute | Values |
---|---|
swiftmq-topic | |
qos | Min: 0 Max: 2 Default: 0 |