Replicator Extension Swiftlet
The Replicator Swiftlet supports the replication of files and directories via a SwiftMQ router network from arbitrary definable sources to arbitrary definable sinks.
A replication source supervises a source directory and uses SwiftMQ's Filetransfer API to replicate the content and subsequent changes into an associated file cache on the local or a remote router. Any number of replication sinks can be defined anywhere within the SwiftMQ Universal Router Network. They synchronized the file cache with a local directory and get subsequent updates from the replication source via a topic. The result is an exact and up-to-date copy of the replication source, fully automatic.
The Replicator Swiftlet is well-suited for the replication of configuration files or entire websites, the distribution of software, the centralized storage of log-files and much more.
Replication Sources
Create a File Cache
A file cache needs to be created on the local or any remote router. This file cache will be associated with a replication source which replicates its files into it.
Assume we want to replicate log files, so we create a new file cache at
the local router and name it logs:
Because the replication source will not use expiration, it can be disabled at the file cache declaration by setting it to -1.
Create a Replication Source
To create a replication source, go to node Replication Sources and do
a right click Create a New Entity:
A new frame appears where the new replication source can be declared:
Field description:
| Attribute | Meaning |
|---|---|
| Name | Some unique name of your choice. |
| Check Interval | The interval in which the directory is checked for changes. |
| Control File | A registry file where the replication source stores links to transferred files. This should be stored outside of the source directory. |
| Directory | The directory which shall be replicated. |
| Enabled | Activates/deactivates the replication source. |
| File Cache Name | The name of the associated file cache. |
| File Cache Router Name | The router name of the associated file cache. |
| Recursive | Enables/disables the replication of subdirectories. |
| Update Topic | The name of the topic where updates events published to. |
The replication source starts replication into the associated file cache if it is enabled.
Replication Sinks
To create a replication sink, go to node Replication Sinks and do a
right click Create a New Entity:
A new frame appears where the new replication sink can be declared:
Field description:
| Attribute | Meaning |
|---|---|
| Name | Some unique name of your choice. |
| Check Interval | The interval in which the topic is checked for changes or a retry of a cache sync takes place if the previous sync failed or if there was any other error. |
| Control File | A registry file where the replication sink stores links to transferred files. This should be stored outside of the sink directory. |
| Directory | The destination directory. |
| Enabled | Activates/deactivates the replication sink. |
| File Cache Name | The name of the associated file cache. |
| File Cache Router Name | The router name of the associated file cache. |
| Update Topic | The name of the topic where updates events are received from. |
The replication sink starts replication from the associated file cache into the destination directory if it is enabled. It starts with synchronizing with the file cache and subsequently receives update events from the replication source on the update topic.
Event Generation
A replication sink can be configured to generate appropriate events and send it as a JMS TextMessage containing the event as an XML document to configured queues and/or topics. This is useful in situations where a file should be processed once it has arrived, for example. In that case, a simple JMS client would listen on a particular queue (or topic) and would perform particular actions (e.g. process an arrived file) on the events it receives.
Event generation is configured per replication sink. There is an entity
list with name Event Sinks. Select it and perform a right mouse click.
Select Create a new Entity:
A dialog appears where you can specify the destination, persistence mode and where you can select which type of events you like to send to that event sink:
An event is send as a JMS TextMessage. The text is an XML document like the following example:
Example:
<?xml version="1.0" encoding="UTF-8"?>
<replication-event
event-type="deleted"
sink="customer-orders"
sink-path="/usr/local/orders"
path="/"
filename="PO-63446.pdf"
modification-time="1040245494822"/>
Field description:
| Attribute | Meaning |
|---|---|
| event-type | The event type. Possible values are "created" or "deleted". |
| sink | The name of the replication sink. |
| sink-path | The path of the sink as specified in the replication sink configuration. |
| path | The path of the file/directory that caused the event. The path is relative to the "sink-path". A value of "/" is equal to the "sink-path" itself. The path seperator character is always "/". |
| filename | The name of the file/directory that caused the event. |
| modification-time | The modification time of the file at the replication source. This attribute is just informational. |
Note that events are generated after the associated operation is
completed. For example, an event created for a file is generated after
the file has been completely received by the replication sink.
Job Scheduling
The Replicator Swiftlet registers jobs in job group Replicator at the
Scheduler Swiftlet:
These jobs can be scheduled via the Scheduler Swiftlet to run at specific times or in intervals, based on calenders and so on.
Source
The Source job activates replication source on job start and deactivates it on job stop. The name of the replication source must be specified as parameter. The job requires to set a maximum runtime to deactivate the source. Otherwise it would be active forever.
The implementation of the Source job is quite simple. It sets the
enabled attribute to true on job start and to false on job stop.
Therefore, a replication source which should run as a job should be
initially disabled.
| Parameter | Mandatory | Description |
|---|---|---|
| Source Name | Yes | Name of the replication source. |
Sink
The Sink job activates replication source on job start and deactivates it on job stop. The name of the replication sink must be specified as parameter. The job requires to set a maximum runtime to deactivate the source. Otherwise it would be active forever.
The implementation of the Sink job is quite simple. It sets the
enabled attribute to true on job start and to false on job stop.
Therefore, a replication sink which should run as a job should be
initially disabled.
| Parameter | Mandatory | Description |
|---|---|---|
| Sink Name | Yes | Name of the replication sink. |
Installation
The installation takes place via 'hot deployment'. See the documentation of the Deploy Swiftlet for further information.
Configuration
The configuration of the Replicator Extension Swiftlet is defined within the element
<swiftlet name="xt$replicator" .../>
of the router's configuration file. One can use the SwiftMQ Exlorer or CLI for configuration as well. They both save into that file.
Element List "sources", Parent Element: "swiftlet"
Replication Source Definitions. This element list contains zero or more "source" elements with this template definition:
Definition
| Attribute | Type | Mandatory | Description |
|---|---|---|---|
| name | java.lang.String | Yes | Name of this Replication Source |
| filecache-routername | java.lang.String | Yes | File Cache Router Name |
| filecache-name | java.lang.String | Yes | File Cache Name |
| ivm-username | java.lang.String | No | IntraVM Connection User Name |
| ivm-password | java.lang.String | No | IntraVM Connection Password |
| enabled | java.lang.Boolean | No | Enabled |
| controlfile | java.lang.String | Yes | Control File |
| directory | java.lang.String | Yes | Directory |
| recursive | java.lang.Boolean | No | Recursive |
| check-interval | java.lang.Long | Yes | Check Interval (ms) |
| update-topic | java.lang.String | Yes | Update Topic |
Values
| Attribute | Values |
|---|---|
| filecache-routername | |
| filecache-name | |
| ivm-username | |
| ivm-password | |
| enabled | Default: false |
| controlfile | |
| directory | |
| recursive | Default: true |
| check-interval | Min: 1000 Default: 60000 |
| update-topic |
Element List "sinks", Parent Element: "swiftlet"
Replication Sink Definitions. This element list contains zero or more "sink" elements with this template definition:
Definition
| Attribute | Type | Mandatory | Description |
|---|---|---|---|
| name | java.lang.String | Yes | Name of this Replication Sink |
| filecache-routername | java.lang.String | Yes | File Cache Router Name |
| filecache-name | java.lang.String | Yes | File Cache Name |
| ivm-username | java.lang.String | No | IntraVM Connection User Name |
| ivm-password | java.lang.String | No | IntraVM Connection Password |
| enabled | java.lang.Boolean | No | Enabled |
| controlfile | java.lang.String | Yes | Control File |
| directory | java.lang.String | Yes | Directory |
| check-interval | java.lang.Long | Yes | Check Interval (ms) |
| update-topic | java.lang.String | Yes | Update Topic |
Values
| Attribute | Values |
|---|---|
| filecache-routername | |
| filecache-name | |
| ivm-username | |
| ivm-password | |
| enabled | Default: false |
| controlfile | |
| directory | |
| check-interval | Min: 1000 Default: 60000 |
| update-topic |
Element List "event-sinks", Parent Element: "sink"
Event Sinks. This element list contains zero or more "event-sink" elements with this template definition:
Definition
| Attribute | Type | Mandatory | Description |
|---|---|---|---|
| name | java.lang.String | Yes | Name of this Event Sink |
| destination-name | java.lang.String | Yes | Name of the Queue/Topic |
| destination-type | java.lang.String | Yes | Destination Type |
| delivery-mode | java.lang.String | Yes | Delivery Mode |
| event-file-created | java.lang.Boolean | No | Event File Created |
| event-file-deleted | java.lang.Boolean | No | Event File Deleted |
Values
| Attribute | Values |
|---|---|
| destination-name | |
| destination-type | Choice: queue topic Default: queue |
| delivery-mode | Choice: non_persistent persistent Default: non_persistent |
| event-file-created | Default: true |
| event-file-deleted | Default: true |
CLI Commands During Deployment
This Swiftlet performs the following CLI commands during deployment:
cc /sys$threadpool/pools
new replicator
cc /sys$threadpool/pools/replicator/threads
new extension.xt$replicator.%
CLI Commands During Undeployment
This Swiftlet performs the following CLI commands during undeployment:
cc /sys$threadpool/pools
delete replicator