<?xml version="1.0" encoding="UTF-8"?>
<!-- $Id: replication.xml 18426 2013-05-18 20:35:41Z joewiz $ --><book><bookinfo><productname>eXist-db – Open Source Native XML Database</productname><title>Replication</title><date>October 2012</date></bookinfo><chapter><title>Replication</title><section id="replication-introduction"><note><para>Please note that this extension has been added to eXist-db recently: It has not been tested (yet)
                    for an high-volume context. Please contact the exist-open mailing list to share experiences with
                    this feature. </para></note><title>Introduction</title><para>This document provides a short introduction on the document replication function of eXist-db. You can
                configure two or more eXist-db instances to work together to automatically synchronize
                collection-specific data sets. This allows you to scale your eXist-db server capacity. For example, with
                multiple eXist-db servers configured to stay in sync as described below, you could add a load-balancer
                to distribute the load of incoming queries across the pool of servers and still maintain high
                performance.</para><para>See also <ulink url="http://en.wikibooks.org/wiki/XQuery/eXist_Clustering">the eXist-db clustering
                    wikibook</ulink>.</para></section><section id="replication-preparation"><title>Preparation</title><procedure><step><para>You need to download a recent version of ActiveMQ from <ulink url="http://activemq.apache.org/download.html">ActiveMQ download page</ulink> if you cannot
                        install it directly from your package manager.</para><note><para>Note that the <filename>.tgz archive file</filename> has additional unix (GNU/Linux,
                            Mac OS X) support. The <filename>zip archive file</filename> is for Windows. The contents of
                            the archives actually differ.</para></note></step><step><note><para>Please follow the instructions on verifying the integrity of the download before you
                            continue with the next step in the preparations.</para></note></step><step><para>Extract content to disk, refered as <parameter>$ACTIVEMQ_HOME</parameter> – Copy the
                            <filename>activemq-all-X.Y.Z.jar</filename> file to
                            <filename>$EXIST_HOME/lib/user</filename></para></step><step><para>Build the replication extension (modify
                        <filename>extensions/local.build.properties</filename>) or copy the pre-built version of
                            <filename>exist-replication.jar</filename> to <filename>lib/extensions</filename></para></step></procedure></section><section id="replication-get-started"><title>Quick start configuration</title><note><para> The XML fragments in this section are functional but do not describe all configuration items.
                    Please consult the files <code>conf.xml</code> and <code>example.xconf</code> in the
                        <code>extensions/replication/doc</code> directory for more details. </para></note><section id="replication-get-started-master"><title>Master (publisher)</title><procedure><step><para>Create collection <filename>/db/mycollection/</filename> that shall be monitored for
                            document changes</para></step><step><para>Create collection <filename>/db/system/config/db/mycollection/</filename></para></step><step><para>Create in there a document <filename>collection.xconf</filename> and add the following
                            content to the document:</para><synopsis language="xml">
&lt;collection xmlns="http://exist-db.org/collection-config/1.0"&gt;
    &lt;triggers&gt;
        &lt;trigger class="org.exist.replication.jms.publish.ReplicationTrigger"&gt;
            &lt;parameter name="java.naming.factory.initial" 
                value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/&gt;
            
            &lt;!-- Set value --&gt;
            &lt;parameter name="java.naming.provider.url" 
                value="tcp://myserver.local:61616"/&gt;
            
            &lt;parameter name="connectionfactory" value="ConnectionFactory"/&gt;
            &lt;parameter name="topic" value="dynamicTopics/eXistdb"/&gt;
            
            &lt;!-- Set value --&gt;
            &lt;parameter name="client-id" value="SetPublisherId"/&gt;
        &lt;/trigger&gt;
    &lt;/triggers&gt;
&lt;/collection&gt;
</synopsis></step><step><para>Set the correct value for <parameter>java.naming.provider.url</parameter> that matches
                            your message broker (ActiveMQ).</para></step><step><para>Set a unique value for the <parameter>client-id</parameter> parameter.</para></step></procedure></section><section id="replication-get-started-slave"><title>Slave (subscriber)</title><procedure><step><para> Add a startup trigger to conf.xml:</para><synopsis language="xml">
&lt;trigger 
    class="org.exist.replication.jms.subscribe.MessageReceiverStartupTrigger"&gt;
    
    &lt;parameter name="java.naming.factory.initial" 
        value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/&gt;
    
    &lt;!-- Set value --&gt;
    &lt;parameter name="java.naming.provider.url" 
        value="tcp://myserver.local:61616"/&gt;
    
    &lt;parameter name="connectionfactory" value="ConnectionFactory"/&gt;
    &lt;parameter name="topic" value="dynamicTopics/eXistdb"/&gt;
    
    &lt;!-- Set values --&gt; 
    &lt;parameter name="client-id" value="SetSubscriberId"/&gt;
    &lt;parameter name="subscriber-name" value="SetSubscriptionId"/&gt;
&lt;/trigger&gt;
    </synopsis></step><step><para>Set the correct value for <parameter>java.naming.provider.url</parameter> that matches
                            your message broker (ActiveMQ).</para></step><step><para>Set unique values for <parameter>client-id</parameter> and
                                <parameter>subscriber-name</parameter>.</para></step><step><para>Create the collection <filename>/db/mycollection/</filename>, this is the collection that
                            receives the documents that are updated in the same collection on the
                                <emphasis>Master</emphasis> server.</para></step></procedure></section></section><section id="replication-startup"><title>Start software</title><procedure><step><para>Start ActiveMQ server:</para><synopsis language="bash">
cd $ACTIVEMQ_HOME
./bin/activemq start
	  </synopsis><para>or use the java wrapper for running it as a service with Mac OS X or GNU/Linux:</para><para>that is for Mac OS X:</para><synopsis language="bash">
$ACTIVEMQ_HOME/bin/macosx/activemq start
 	  </synopsis><para>or for 32 bit GNU/Linux:</para><synopsis language="bash">
$ACTIVEMQ_HOME/bin/linux-x86-32/activemq start
	  </synopsis><para>or 64 bit GNU/Linux:</para><synopsis language="bash">
$ACTIVEMQ_HOME/bin/linux-x86-64/activemq start
	  </synopsis></step><step><para>Start Slave</para><synopsis language="bash">
cd $EXISTSLAVE_HOME
./bin/startup.sh
	  </synopsis></step><step><para>Start Master</para><synopsis language="bash">
cd $EXISTMASTER_HOME
./bin/startup.sh
	  </synopsis></step></procedure></section><section id="replication-distribute"><title>Distribute documents</title><para>Create a document in the master server in <filename>/db/mycollection/</filename> (e.g. using the java
                client or eXide; login as admin); The document will be automatically replicated to the slave servers
                (using the previous configuration</para><section id="replication-performance-test"><title>XQuery Example</title><procedure><step><para>With eXide upload a +/- 50k XML document store as <filename>/db/mydoc.xml</filename></para></step><step><para>Execute the query, check the timing on the slave (see
                            <filename>exist.log</filename>)</para><synopsis language="xquery">
let $doc := doc('/db/mydoc.xml')
for $i in (1000 to 3000)
return
    xmldb:store('/db/mycollection', concat('mydoc', $i , ".xml"), $doc)
	  </synopsis></step></procedure></section></section><section id="Notes"><title>Notes</title><para><itemizedlist><listitem><para>The replication feature is based on the trigger functions of eXist-db. After a document is
                            created or updated the (complete) document is serialized to a compressed binary blob which
                            is sent to the message broker. </para><para>Please note that this trigger is also fired when a document is partially updated e.g. with
                            XUpdate. The consequence is that when a large document is updated very often in a short
                            period of time, many large documents with just small differences are sent to the message
                            broker. As a result the performance will degrade on the publisher, subscriber and message
                            broker. </para></listitem><listitem><para>The default persistence cache size of ActiveMQ might be too small, consult the <ulink url="http://activemq.apache.org/kahadb.html">ActiveMQ website</ulink> how to increase
                            this cache size. </para></listitem></itemizedlist></para></section><section id="Full configuration"><title>Full configuration</title><para>The following XML fragments show all available configuration options.</para><section id="replication-conf-xml"><title>conf.xml</title><para><synopsis language="xml">
&lt;!--
    Start JMS listener for listener of the clustering feature. 
--&gt;
&lt;trigger 
    class="org.exist.replication.jms.subscribe.MessageReceiverStartupTrigger"&gt;

    &lt;!-- 
        Class name of the initial context provider, default value for ActiveMQ
        see javax.naming.Context#INITIAL_CONTEXT_FACTORY
    --&gt;
    &lt;parameter name="java.naming.factory.initial" 
        value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/&gt;

    &lt;!-- 
        URL of the message broker, default value for ActiveMQ
        see javax.naming.Context#PROVIDER_URL
    --&gt;
    &lt;parameter name="java.naming.provider.url" 
        value="tcp://myserver.local:61616"/&gt;

    &lt;!-- 
        Lookup connection factory
        see javax.naming.InitialContext#lookup(String) 
    --&gt;
    &lt;parameter name="connectionfactory" value="ConnectionFactory"/&gt;

    &lt;!--
        Lookup destination (topic)
        see javax.naming.InitialContext#lookup(String) 
    --&gt;
    &lt;parameter name="topic" value="dynamicTopics/eXistdb"/&gt;

    &lt;!--
        Set client identifier for this connection.
        see javax.jms.Connection#setClientID(string)
    --&gt;
    &lt;parameter name="client-id" value="SubscriberId"/&gt;

    &lt;!-- 
        Set the name used to identify this subscription
        see JMS javax.jms.TopicSession#createDurableSubscriber(Topic,String) 
    --&gt;
    &lt;parameter name="subscriber-name" value="SubscriptionId"/&gt;

    &lt;!-- 
        Set the subscription is durable. default = yes
        see JMS javax.jms.Session#createDurableSubscriber(Topic,String) 
        see JMS javax.jms.Session#createConsumer(Destination, String, boolean)
        
        [Optional]
    --&gt;
    &lt;!--&lt;parameter name="durable" value="yes"/&gt;--&gt;

    &lt;!-- 
        A JMS API message selector allows a message consumer to specify 
        the messages it is interested in. The syntax of the expression is based 
        on a subset of the SQL92 conditional expression syntax.
        
        see javax.jms.Message
        
        [Optional]
    --&gt;
    &lt;!--&lt;parameter name="messageselector" 
            value="property1 = 'a' OR property2 = 'b'"/&gt;--&gt;

    &lt;!-- 
        If set, inhibits delivery of messages published by its own connection
        default = yes
        
        [Optional]
    --&gt;
    &lt;!--&lt;parameter name="nolocal" value="yes"/&gt;--&gt;
&lt;/trigger&gt;

</synopsis></para></section><section id="replication-collection-xconf"><title>collection.xconf</title><para><synopsis language="xml">
&lt;collection xmlns="http://exist-db.org/collection-config/1.0"&gt;
    &lt;triggers&gt;
        &lt;trigger class="org.exist.replication.jms.publish.ReplicationTrigger"&gt;

            &lt;!-- 
                Class name of the initial context provider, default value for 
                ActiveMQ
                see javax.naming.Context#INITIAL_CONTEXT_FACTORY
            --&gt;
            &lt;parameter name="java.naming.factory.initial" 
                value="org.apache.activemq.jndi.ActiveMQInitialContextFactory"/&gt;

            &lt;!-- 
                URL of the message broker, default value for ActiveMQ 
                see javax.naming.Context#PROVIDER_URL
            --&gt;
            &lt;parameter name="java.naming.provider.url" 
                value="tcp://myserver.local:61616"/&gt;

            &lt;!-- 
                Lookup connection factory
                see javax.naming.InitialContext#lookup(String) 
            --&gt;
            &lt;parameter name="connectionfactory" value="ConnectionFactory"/&gt;

            &lt;!--
                Lookup destination (topic)
                see javax.naming.InitialContext#lookup(String) 
            --&gt;
            &lt;parameter name="topic" value="dynamicTopics/eXistdb"/&gt;

            &lt;!--
                Set client identifier for this connection.
                see javax.jms.Connection#setClientID(string)
            --&gt;
            &lt;parameter name="client-id" value="PublisherId"/&gt;

            &lt;!-- 
                Set time-to-live in milliseconds, default value is 0 (unlimited) 
                see javax.jms.MessageProducer#setTimeToLive(long)
                
                [Optional]
            --&gt;
            &lt;!--&lt;parameter name="time-to-live" value="0"/&gt;--&gt;

            &lt;!--
                Sets the producer priority, value 0-9 ; default is 4.
                see javax.jms.MessageProducer#setPriority(int)
                
                [Optional]
            --&gt;
            &lt;!-- &lt;parameter name="priority" value="4"/&gt;--&gt;

        &lt;/trigger&gt;
    &lt;/triggers&gt;
&lt;/collection&gt;        
</synopsis></para></section></section></chapter></book>