Store Swiftlet
The Store Swiftlet provides a transactional file-based messagestore where the following data is stored:
- Persistent messages within a transactional store
- Non-persistent messages within a swap store
- Durable subscriber definitions within a durable subscriber store
Tuning
Disk access has the most influence on the SwiftMQ performance. Normally, the default settings of the Store Swiftlet are sufficient and are optimized for the average standard operation. Concerning advanced applications, various tuning possibilities are discussed within this chapter.
Database Settings
The initial size of the file page.db
may be configured via the
attribute initial-page-size
. The default setting is 512 pages (1 MB)
for releases up to 8.0.0 and 25600 pages (50 MB) from 8.1.0 onwards. It
is not guaranteed, but most of the operating systems create a continuous
space if the file size is given and if procurable. Such a coherent disk
partition reduces the number of positionings during a disk access. If
your page.db
is permanently greater (thus you store simultaneously
many messages), you should increment the initial-page-size
to allocate
a maximum coherent area. Please notice, this attribute only has an
effect if the file does not exist. Therefore you should first delete
page.db
and than start the router with the modified
initial-page-size
. Further, please take notice that all pages are read
during the router startup to locate free pages. Thus, a larger size
implicates a longer startup time.
Cache Size
The cache may be configured by the two attributes min-size
and
max-size
. Both values are numbers of pages. The cache manager makes
sure that the cache size retains during that interval. When reaching the
max-size
, an automatic flushing of all dirty pages takes place unless
these are in use (pin). The flush is continued as long as the min-size
is reached resp. no dirty unpinned pages exist. The default settings are
1024 pages concerning the min-size
and 2048 pages concerning the
max-size
. This is optimal for a write/read-transaction mix, i. e.
messages are stored but they are reread in best time. In that case no
disk access takes place with the exception of the transaction log. All
transactions are served out of the cache as deallocated pages also
remain within the cache. If you have many write-transactions and an
extended message dwell, you should increase respectively the max-size
to avoid the automatical flush.
Checkpoint Size
The attribute checkpoint-size
specifies the size of the transaction
log (transaction.log
). After reaching, a checkpoint is performed. This
means a pause of all transactions, a complete flush of the Cache, sync
of the disk, and the enabling of all transactions. The checkpoint-size
default setting is 10 MB. This proves optimal. A checkpoint then
requires a few milliseconds. A minor size implicates more checkpoints
and, as a result, more disk syncs. A larger size accordingly requires a
bigger cache so that the duration of a checkpoint increases as more is
written. Please consider that all transactions are stopped during a
checkpoint.
Transaction Log Disk Sync and Group Commit
A disk sync of the transaction log may be forced over the attribute
force-sync
. The default setting is false
. In the case of true
a
disk sync is executed with every interval of the log manager. During one
intervall, the log manager may write several log records into the
transaction log (group commit) as it works asynchron. However, this is
only the case when having a high parallelism of transactions. It doesn't
increase the throughput of single transactions but only the overall
throughput (scalability).
The default mode of false
is less reliable and you may loose data if
the computer crashes. However, the throughput is many times higher than
with disk sync. Forcing disk syncs is reliable, but the performance is
then bound to the speed of the disk, which varies.
Backup and Restore
Backup
Since SwiftMQ release 4.5.0 it is possible to perform an online backup of the complete store during operation. A backup is done by intercepting a checkpoint. That is, a checkpoint is initiated and the backup is performed before transactions are restarted to get a consistent state of the store. During a backup operation, a save set is generated which includes all necessary files of the store. A save set has a timestamp and is generated below the backup directory. The number of generated save sets to keep is configurable and is maintained by the backup processor. A backup can be initiated manually via SwiftMQ Explorer / CLI or automatically through a backup job via the Scheduler Swiftlet.
Backup Configuration
The backup configuration takes place via the element backup
of the
Store Swiftlet. It contains 2 attributes. One is path
and points per
default to the directory backup
below the router's store directory.
This is the root path for backups. All save sets are created below that
directory. The other attribute is keep-generations
and specifies how
many save sets should be keeped in the backup directory. The default is
3 generations. That is, if a new save set is created, the backup
processor checks the number of save sets in the backup directory and
deletes the oldest save set until the number of save sets matches the
number, specified in attribute keep-generations
. If the value is
changed, it will be automatically applied to the number of save sets.
For example, if keep-generations
is 4 and the number of save sets is
also 4, and you change keep-generations
to 2, the backup processor
will automatically delete 2 save sets.
The following example uses an alternative backup path and specifies to keep 2 generations:
Example:
<backup path="/usr/local/backup/swiftmq" keep-generations="2"/>
Performing a Backup via CLI
CLI provides a command backup
, available within context
/sys$store/backup
. So you have to change to this context first.
Thereafter, initiate the backup
command:
Example:
router1> cc /sys$store/backup
router1/sys$store/backup> backup
Backup initiated. Please watch Folder 'Generated Backup Save Sets'.
router1/sys$store/backup>_
The backup runs asynchronously. Once it is finished, you'll find it
below the /sys$store/usage
context:
Example:
router1/sys$store/backup> cc /sys$store/usage
router1/sys$store/usage> lc
Entity List: Generated Backup Save Sets
Description: Generated Backup Save Sets
Entities in this List:
----------------------
saveset_20030217125000008
saveset_20030220124842148
saveset_20030220125000023
router1/sys$store/usage>_
Performing a Backup via SwiftMQ Explorer
Go to the folder Backup
of the Store Swiftlet, select it and do a
right mouse click. A popup menu Perform Backup Now
appears:
Click it and you'll see this message:
The backup operation is now performed asynchronously (during the next
checkpoint). Once it is finished, you'll see a new entry in folder
Generated Backup Save Sets
:
Schedule a Backup Job via the Scheduler Swiftlet
The Store Swiftlet registers a job Backup
in job group Store
at the
Scheduler Swiftlet:
This job can be used to schedule a backup at a specific time of a day.
The following example schedules the Backup
job at 00:00, 10:00 and
18:00 on each business day:
Restore
To restore the content of a save set to the router's store, you have to stop the router. It is not possible to perform a restore while the router is running.
Thereafter you have to delete the following files from the router's store:
- page.db
- transaction.log
- xa.log
- all files ending with *.durable
Now go to the backup root path (specified in the path
directory) and
select the save set you want to restore. Verify whether a file named
.completed
is contained in the save set. This file is created as the
last operation during a backup, after all files are sync'ed with disk.
If it is not contained, then the backup is incomplete, you cannot use it
and you have to switch to the previous generation (just check the name
of the save set which contains a time stamp).
From the save set, copy:
- page.db
- all files ending with *.durable
to the resp. router's store directories. Don't wonder why the file transaction.log isn't contained in the save set. It is just that the backup is performed right after a checkpoint when the transaction.log is empty and thus there is no need to save it.
Start the router.
Shrinking the Database
Monitoring the Data Store
The usage of the data store page.db
can be monitored with SwiftMQ
5.2.0 and higher. To this the Usage
section of the Store Swiftlet
contains a Files
entity list with one entry page.db
. This entry
consists of attributes file-size, free-pages and used-pages:
The entity list can also be display as an Entity Table Frame:
By pushing the chart button a usage history of the page.db
can be
displayed:
The attribute values are collected from the data store. The collect
interval is defined in attribute size-collect-interval
of entity
database
. The default value is 10000 ms (10 secs). A value of -1 turns
it off:
Shrinking the Data Store
When pages are released in the data store page.db
, they are kept in a
free page list and will be reused. The file is not automatically
shrinked for performance reasons. This could lead to a large data store
with many free pages. The only way to reduze the file size was to reboot
the router, because the file is shrinked during the startup of the Store
Swiftlet to the last used page. SwiftMQ 5.2.0 and higher allows to
perform a shrink during operation. The shrink
command is available in
context sys$store/database
. It can be invoked from CLI as well as from
the SwiftMQ Explorer.
To invoke it from CLI, switch to context sys$store/database
and
perform the command shrink
:
Example:
router1> cc sys$store
router1/sys$store> cc database
router1/sys$store/database> shrink
Shrink initiated.
router1/sys$store/database> _
To router initiates a checkpoint and runs the shrink thereafter.
You can also shrink it from SwiftMQ Explorer. The following shows the file before the shrink:
Performing the command:
Results in:
Schedule a Shrink Job via the Scheduler Swiftlet
The Store Swiftlet registers a job Shrink
in job group Store
at the
Scheduler Swiftlet:
This job can be used to schedule a shrink at a specific time of a day.
The following example schedules the Shrink
job at 00:00 on each
business day:
Analyzing the Store
Analyzing the Checkpoint Duration
To analyze how long a checkpoint takes, set this system property to true:
-Dswiftmq.store.checkpoint.verbose=true
The Store Swiftlet writes a line on each checkpoint to System.out
containing the time in milliseconds the checkpoint took.
Analyzing the Store Content
To analyze the content of the transactional store, set this system property to true:
-Dswiftmq.store.analyze=true
On startup, the Store Swiftlet creates one file per queue with the
extension .analyze
and one file called rootindex.analyze
within the
working directory. These files contain a dump of the queue index resp.
the root index. The queue messages aren't dumped.
Configuration
The configuration of the Store Swiftlet is defined within the element
<swiftlet name="sys$store" .../>
of the router's configuration file. One can use the SwiftMQ Exlorer or CLI for configuration as well. They both save into that file.
Element "backup", Parent Element: "swiftlet"
Backup Settings.
Definition
Attribute | Type | Mandatory | Description |
---|---|---|---|
path | java.lang.String | No | Root Path for Backup Save Sets |
keep-generations | java.lang.Integer | No | Number of Generations to keep |
Values
Attribute | Values |
---|---|
path | Default: ./ |
keep-generations | Min: 1 Default: 3 |
Element "database", Parent Element: "swiftlet"
Database Settings.
Definition
Attribute | Type | Mandatory | Description |
---|---|---|---|
path | java.lang.String | No | Path of the Database (page.db) |
initial-page-size | java.lang.Integer | No | Initial Page Size Database |
size-collect-interval | java.lang.Long | No | Interval to collect the Usage |
Values
Attribute | Values |
---|---|
path | Default: ./ |
initial-page-size | Min: 10 Default: 25600 |
size-collect-interval | Default: 1000 |
Element "cache", Parent Element: "swiftlet"
Cache Settings.
Definition
Attribute | Type | Mandatory | Description |
---|---|---|---|
min-size | java.lang.Integer | No | Min. Size (Pages) |
max-size | java.lang.Integer | No | Max. Size (Pages) |
Values
Attribute | Values |
---|---|
min-size | Min: 512 Default: 1024 |
max-size | Min: 512 Default: 2048 |
Element "transaction-log", Parent Element: "swiftlet"
Transaction Log Settings.
Definition
Attribute | Type | Mandatory | Description |
---|---|---|---|
path | java.lang.String | No | Path of the Transaction Log (transaction.log) |
checkpoint-size | java.lang.Long | No | Size at which a Checkpoint is performed |
force-sync | java.lang.Boolean | No | Force a sync with the disk |
Values
Attribute | Values |
---|---|
path | Default: ./ |
checkpoint-size | Min: 1048576 Default: 104857600 |
force-sync | Default: false |
Element "swap", Parent Element: "swiftlet"
Swap Settings.
Definition
Attribute | Type | Mandatory | Description |
---|---|---|---|
path | java.lang.String | No | Path of Swap Files |
roll-over-size | java.lang.Long | No | Roll Over Size |
Values
Attribute | Values |
---|---|
path | Default: ./ |
roll-over-size | Min: 1048576 Default: 10485760 |
Element "durable-subscriber", Parent Element: "swiftlet"
Durable Subscriber Settings.
Definition
Attribute | Type | Mandatory | Description |
---|---|---|---|
path | java.lang.String | No | Path of Durable Subscriber Files |
Values
Attribute | Values |
---|---|
path | Default: ./ |
Element "usage", Parent Element: "swiftlet"
Usage.
Element List "backup", Parent Element: "usage"
Generated Backup Save Sets. This element list contains zero or more "backup" elements with this template definition:
Definition
Attribute | Type | Mandatory | Description |
---|---|---|---|
name | java.lang.String | Yes | Name of this Generated Backup Save Set |
Element List "files", Parent Element: "backup"
Files. This element list contains zero or more "file" elements with this template definition:
Definition
Attribute | Type | Mandatory | Description |
---|---|---|---|
name | java.lang.String | Yes | Name of this File |
filesize | java.lang.Long | No | File Size |
Values
Attribute | Values |
---|---|
filesize |
Element List "files", Parent Element: "usage"
Files. This element list contains zero or more "file" elements with this template definition:
Definition
Attribute | Type | Mandatory | Description |
---|---|---|---|
name | java.lang.String | Yes | Name of this File |
free-pages | java.lang.Integer | No | Number of Free Pages |
used-pages | java.lang.Integer | No | Number of Used Pages |
file-size | java.lang.Integer | No | Size of this File in KB |
Values
Attribute | Values |
---|---|
free-pages | Default: 0 |
used-pages | Default: 0 |
file-size | Default: 0 |