<?xml version="1.0" encoding="UTF-8"?>
<!-- $Id: backup.xml 18235 2013-02-06 09:52:41Z wolfgang_m $ --><book><bookinfo><productname>eXist-db – Open Source Native XML Database</productname><title>Database Backup and Restore</title><date>August 2012</date><author><firstname>Wolfgang M.</firstname><surname>Meier</surname><affiliation><address format="linespecific"><email>wolfgang at exist-db.org</email></address></affiliation></author></bookinfo><chapter><title>Database Backup and Restore</title><abstract><para>This section discusses eXist-db's database backup/restore procedures. eXist-db
                provides different methods for creating backups, which will be explained
                below.</para></abstract><section><title>Backup Format</title><para>During backup, eXist-db exports the contents of its database (as standard XML files)
                to a hierarchy of directories on the hard drive. This hierarchy is organized
                according to the organization of collections in the database.</para><para>Other files stored during backup include index configuration files and user
                settings. Resource and collection metadata is exported to a special XML file, called
                <filename>__contents__.xml</filename>, which lists information including the
                resource type, owner, modification date and/or the permissions assigned to a
                resource. You will find one <filename>__contents__.xml</filename> file in each
                directory created by the backup. This descriptor file is required to restore the
                backup.</para><para>Since eXist-db uses an open XML format rather than a proprietary format for its
                database files, users can manually modify files in the backup directories without
                requiring special software. Any changes made to these files are reflected in the
                database with a restore or once the data is imported to another database
                system.</para><para>It is even possible to directly edit user data and permissions stored in the file
                <filename>/db/system/users.xml</filename>. This is particularly useful when
                making global changes to the user database. For example, to reset the passwords for
                all your users, you can simply edit the file <filename>users.xml</filename> by
                removing the <option>password</option> attribute, or set it to a default value and
                restore the document.</para><note><para> When migrating to a new eXist-db version, take care to use a version of the
                    client corresponding to your server version. Usually, the backup process is
                    backwards compatible. However, using a newer client version to create the backup
                    from a server running an older version may sometimes lead to unexpected
                    problems.</para></note><important><para>Due to limitations of the ZIP format, please make sure the size of your zipped backup does
                not exceed 4 gigabytes. All backup methods support backups to the file system as an alternative.</para></important></section><section><title>Backup Methods</title><para>There are two main methods for creating a backup:</para><variablelist><varlistentry><term>Client-side</term><listitem><para>You can use the Java admin client or a small command line utility to
                            create a backup of the data on the server. In this case, the client
                            controls the backup process. The server is not blocked and continues to
                            accept requests from other clients. Other users can modify the db while
                            the backup is running, so logical dependencies between different
                            resources may not be preserved correctly.</para><para>Client-side backups are <emphasis>not safe</emphasis>. The client uses
                            the XML:DB API to access the db. This means that it cannot backup
                            documents or collections if they are damaged in any way.</para></listitem></varlistentry><varlistentry><term>Server-side</term><listitem><para>Server-side backups are usually run through eXist-db's job scheduler,
                            though they can also be triggered via the web interface. This type of
                            backup extracts the data directly from the low-level database files. It
                            is thus much faster then a client-side backup. It also supports
                            incremental backups.</para><para>It is guaranteed that the database is in a consistent state while the
                            backup is running. Possible corruptions in the db will be detected and
                            the backup process will try to work around them. In many cases, damaged
                            resources can at least be partially recovered.</para></listitem></varlistentry></variablelist><para>The format of the generated backup archives will be the same for both backup
                methods. They can all be restored via the standard Java client.</para></section><section><title>Server-side Backup</title><para>This is now the recommended backup method. To guarantee consistency, server-side
                backups are always executed as system tasks, which means that the database will be
                switched to a protected service mode before the backup starts. eXist-db will wait for
                all pending transactions to complete before it enters protected mode. A database
                checkpoint will be performed and the backup task is executed. While the system task
                is running, no new transactions will be allowed. Concurrent requests by other
                clients will be blocked and added to the internal queue. Once the backup is
                complete, the database will switch back to normal service and all locks will be
                released.</para><para>You can trigger the backup via the Dashboard:</para><screenshot><graphic fileref="resources/Backup.png"/></screenshot><para>Clicking the <guibutton>Trigger</guibutton> button will schedule a single backup
                task. The server will wait for all running transactions to return before it executes
                the task. You can click on <guibutton>Refresh</guibutton> to update the view, which
                lists all backup archives currently available within the standard backup directory.
                Click on the name of an archive to download it.</para><para>Beginning with version 1.2.5, eXist-db can also create <emphasis>incremental
                    backups</emphasis>. Only resources which were modified since the last backup are
                archived.</para><para>All backups will be stored in ZIP format into a directory <filename>export/</filename> below the main data directory, which means <filename>webapp/WEB-INF/data/export/</filename> by default (unless you configured a
                different data directory in <filename>conf.xml</filename>).</para></section><section><title>Automatic Consistency Check</title><para>The system task runs a consistency and sanity check on the database before backing
                it up. Reports of this check are written into the same directory as the backup
                archive. The last report can also be viewed via JMX (see below).</para><para>The consistency check will first check the collection hierarchy, then scan through
                the stored node tree of every document in the db, testing node properties like the
                node's id, child count, attribute count and node relationships. Contrary to normal
                database operations, the different dbx files are checked independently. This means
                that even if a collection is no longer readable, the tool will still be able to scan
                the documents in the damaged collection.</para><para>The backup task uses the information provided by the consistency check to work
                around damages in the db. It tries to export as much data as possible, even if parts
                of the collection hierarchy are corrupted or documents are damaged:</para><itemizedlist><listitem><para>Descendant collections will be exported properly even if their ancestor
                        collection is corrupted</para></listitem><listitem><para>Documents which are intact but belong to a destroyed collection will be
                        stored into a special collection <filename>/db/lost_and_found</filename></para></listitem><listitem><para>Damaged documents are detected and are removed from the backup</para></listitem></itemizedlist><section><title>Scheduling Backups and Consistency Checks</title><para>The core class for the server-side backup as well as consistency checks is
                    called <classname>ConsistencyCheckTask</classname>. It can be registered as a
                    system task with eXist-db's <ulink url="configuration.xml#N104CF">scheduler</ulink>. To do this, add the following definition to the scheduler
                    section in <filename>conf.xml</filename>:</para><example><title>Job Definition in conf.xml</title><programlisting lang="xml">&lt;job type="system" class="org.exist.storage.ConsistencyCheckTask"
    cron-trigger="0 0 0/12 * * ?"&gt;
    &lt;!-- the output directory. paths are relative to the data dir --&gt;
    &lt;parameter name="output" value="export"/&gt;
    &lt;parameter name="zip" value="yes"/&gt;
    &lt;parameter name="backup" value="yes"/&gt;
    &lt;parameter name="incremental" value="yes"/&gt;
	&lt;parameter name="incremental-check" value="no"/&gt;
&lt;/job&gt;</programlisting></example><para>This will launch a consistency check and database backup every 12 hours,
                    starting at midnight. The time/frequency of the backup is specified in the
                    <option>cron-trigger</option> attribute. The syntax is borrowed from the
                    Unix cron utility, though there are small differences. Please consult the Quartz
                    documentation about <ulink url="http://www.opensymphony.com/quartz/wikidocs/TutorialLesson6.html">CronTrigger</ulink> configuration.</para><para>The task accepts the following parameters:</para><variablelist><varlistentry><term>output</term><listitem><para>The directory to which the backup is written. Relative paths are
                                interpreted relative to eXist-db's main data directory.</para></listitem></varlistentry><varlistentry><term>backup</term><listitem><para>Create a full database backup in addition to running the system
                                checks. Setting this to "no" will not create a backup - unless
                                errors were detected during the consistency check! If errors are
                                found, the task will always try to generate an emergency
                                backup.</para></listitem></varlistentry><varlistentry><term>zip</term><listitem><para>If set to "yes", the backup will be written into a zip archive. For larger databases, please
                            make sure the generated archive is smaller than 4 gigabytes. Due to limitations of the zip format,
                            archives larger than 4 gigabytes may not be readable. In this case, use a backup to the file system
                            instead.</para></listitem></varlistentry><varlistentry><term>incremental</term><listitem><para>Created backups will be incremental. Only resources which were
                                modified since the last backup will be saved. The first backup will
                                always be a full backup, subsequent backups will be
                                incremental.</para><para>Note: you can schedule more than one backup job. For example, an
                                incremental backup could be done multiple times a day while a full
                                backup is created only once during the night </para></listitem></varlistentry><varlistentry><term>incremental-check</term><listitem><para>By default, no consistency check will be run during an incremental
                                backup. For big databases, the consistency check may take too long,
                                so it should be done for full backups only. Set
                                <option>incremental-check</option> to "yes" to run a consistency
                                check during incremental backups.</para></listitem></varlistentry><varlistentry><term>max</term><listitem><para>If incremental backups are enabled, create a full backup every
                                <option>max</option> backup runs. If you set the parameter to
                                e.g. 2, you will get a full backup after two incremental
                                backups.</para></listitem></varlistentry></variablelist></section><section><title>Triggering Backups from XQuery</title><para>System jobs can also be triggered from an XQuery using the <command>system:trigger-system-task</command> function defined in the "system"
                    module: </para><example><title>Triggering a Backup From XQuery</title><programlisting language="xquery">let $params :=
 &lt;parameters&gt;
   &lt;param name="output" value="export"/&gt;
    &lt;param name="backup" value="yes"/&gt;
    &lt;param name="incremental" value="yes"/&gt;
 &lt;/parameters&gt;
 return
    system:trigger-system-task("org.exist.storage.ConsistencyCheckTask", $params)</programlisting></example><para>The function will schedule a backup to be executed as soon as possible.</para></section><section id="consistency-check"><title>Emergency Export Tool</title><screenshot><graphic align="right" fileref="resources/exportgui.png"/></screenshot><para>eXist provides a graphical interface to the consistency check and backup
                    utilities which can be used in case of an emergency, in particular if the
                    database does not start up properly anymore. The tool needs direct access to the
                    database files, so any running database instance has to be stopped before
                    launching the GUI.</para><para>Use the following command line to start the utility:</para><synopsis>java -jar start.jar org.exist.backup.ExportGUI</synopsis><para>If you installed the eXist distribution using the installer, a shortcut to
                    this should have been placed into the start menu, so you don't need to type
                    above command.</para><para>On a headless system you can use the command-line version instead:</para><synopsis>java -jar start.jar org.exist.backup.ExportMain</synopsis><para>Call it with parameter <parameter>-h</parameter> to get a list of possible
                    options.</para><para>For every check run, an error report will be written into the directory
                    specified in <guimenuitem>Output Directory</guimenuitem>. If you clicked on
                    <guimenuitem>Check Export</guimenuitem>, the utility will also export the
                    database into a zip file in the same directory. This backup can be restored via
                    the standard <ulink url="#restore">backup/restore tools</ulink>.</para></section><section><title>Using JMX to View Check Reports</title><para>If Java Management Extensions (JMX) are enabled in the Java VM that is running
                    eXist, you can use a JMX client to see the latest consistency check reports. The
                    screenshot shows jconsole, which is included with the Java 5 and 6 JDKs.</para><screenshot><graphic fileref="resources/jconsole.png"/></screenshot><para>eXist also includes a command-line JMX client. Call it with parameter
                    <parameter>-s</parameter> to see the latest consistency report:</para><synopsis>java -jar start.jar org.exist.management.client.JMXClient -s</synopsis><para>This may produce output as shown below:</para><example><title>JMXClient Output</title><screen>Sanity report
-----------------------------------------------
                Status: FAIL
      Last check start: Thu May 08 21:40:00 CEST 2008
        Last check end: Thu May 08 21:40:00 CEST 2008
            Check took: 594ms
            Error code: RESOURCE_ACCESS_FAILED
            Description: 32</screen></example><para>You can also subscribe to the notifications made available by the SanityReport
                    MBean to be informed of sanity check results. Please consult the <ulink url="jmx.xml">documentation</ulink> on how to configure JMX.</para></section></section><section><title>Client-side Backup</title><para>You can either use the Java-based Admin Client, or the backup command line
                utility.</para><para>If you are using the Admin Client, do the following:</para><procedure><step><para>Select either the Backup Icon (arrow pointed upward) in the toolbar OR
                        <guimenuitem>Tools » Backup</guimenuitem> from the menu.</para></step><step><para>From the <guimenuitem>Collection</guimenuitem> drop-down menu, select the
                        collection to backup. To backup the entire database, select <filename>/db</filename>. Otherwise, select the topmost collection that should be
                        stored. Note, however, that user data and permissions will only be exported
                        if you backup the entire database.</para><screenshot><graphic fileref="resources/backup1.png"/></screenshot></step><step><para>In the <command>Backup-Directory</command> field, enter the full directory
                        path to the where you want the backup database files to be stored or the
                        path to a zip file into which the backup will be written. In general, if the
                        file name ends with <filename>.zip</filename>, the client will attempt to
                        write to a ZIP. Otherwise, it tries to create the specified
                        directory.</para></step><step><para>Click <command>OK</command>.</para></step></procedure><para>If you are using the command-line utility for the backup/restore, do the
                following:</para><para>To launch the utility, do ONE of the following:</para><itemizedlist><listitem><para>start either the <filename>bin/backup.sh</filename> (Unix), OR the
                        <filename>bin/backup.bat</filename> (Windows/DOS) script file</para></listitem><listitem><para>OR enter on the command-line:</para><synopsis>java -jar start.jar backup 
                        -u <emphasis>[admin_username]</emphasis> 
                        -p <emphasis>[admin_password]</emphasis> 
                        -b <emphasis>[collection_path]</emphasis> 
                        -d <emphasis>[target_path]</emphasis>
                        -ouri=<emphasis>[xml_uri]</emphasis></synopsis><para>To view the all of the available options for this command, use the
                        <parameter>-h</parameter> parameter.</para></listitem></itemizedlist><para>Use the <parameter>-b</parameter> parameter to indicate the <emphasis>collection
                    path</emphasis>, and the <parameter>-d</parameter> parameter to indicate the
                <emphasis>target directory</emphasis> on your system. You can also specify the
                current admin username using the <parameter>-u</parameter> parameter, and the admin
                password using the <parameter>-p</parameter> parameter. For example, to backup the
                entire database on a Unix system to the target directory <filename>/var/backup/hd060501</filename>, you would enter the following:</para><synopsis>bin/backup.sh -u admin -p admin-pass -b /db -d /var/backup/hd060501</synopsis><para>By default, the utility connects to the database at the URI: <filename>xmldb:exist://localhost:8080/exist/xmlrpc</filename>. If you want to backup a
                database at a different location, specify its <command>XML:DB URI</command>
                (excluding any collection path) using the <parameter>-ouri</parameter> parameter.
                For example, the following backup on a Unix Tomcat system running on port 80 specifies the database URI
                <filename>xmldb:exist://192.168.1.2:80/xmlrpc</filename></para><synopsis>bin/backup.sh -u admin -p admin-pass -b /db -d /var/backup/hd060501
                -ouri=xmldb:exist://192.168.1.2:80/xmlrpc</synopsis><note><para>Default settings for the user, password or server URIs can also be set via the
                    <filename>backup.properties</filename> file.</para></note></section><section id="restore"><title>Restoring the Database</title><section><title>Important Note about the Restore Process</title><para>Restoring from a backup (or parts of it) does not mean that the existing data
                    in the current database instance will be deleted entirely. The restore process
                    will upload the collections and documents contained in the backup. Collections
                    and documents which exist in the database but are not part of the backup will
                    not be modified.</para><para>This is a feature, not a bug. It allows us to restore selected parts of the
                    database without touching the rest.</para><para>If you really need to restore into a fresh, completely clean database, proceed
                    as follows:</para><procedure><step><para>Stop the running eXist database instance</para></step><step><para>Change into directory <filename>EXIST_HOME/webapp/WEB-INF/data</filename> or another directory you
                            specified as data directory in the configuration (conf.xml). </para></step><step><para>Remove all <filename>.dbx</filename>, <filename>.lck</filename> and
                            <filename>.log</filename> files. This means removing all your old
                            data! eXist will recreate those files upon the next restart.</para></step><step><para>Start eXist again and launch a restore.</para></step></procedure></section><section><title>Restore Using the Java Client</title><para>To restore the database files from a backup, you can again use either the
                    Admin Client, or the backup command line utility.</para><note><para>For eXist 1.2.x, the restore tool can not directly read from a zipped
                        backup. You have to extract it before restoring. Version 1.4 can handle the
                        zip.</para><para>Also, if you experience any issues with bad characters in collection
                        names, use the standard Java <filename>jar</filename> tool to unpack the
                        zip. Contrary to other zip tools, this utility handles character encodings
                        correctly.</para></note><para>If you are using the Admin Client, do the following:</para><procedure><step><para>Select either the Restore Icon (arrow pointed downward) in the toolbar
                            OR <guimenuitem>Tools » Restore</guimenuitem> from the menu.</para></step><step><para>The dialog box shown below will then prompt you to select the backup
                            descriptor <filename>__contents__.xml</filename> from the topmost
                            directory you want restored. To restore the entire database, select the
                            <filename>__contents__.xml</filename> from the <filename>db/</filename> directory.</para><screenshot><graphic fileref="resources/backup2.png"/></screenshot><para>eXist 1.3 allows to directly select the ZIP archive of a
                            backup.</para></step><step><para>A second dialog box will then prompt you for an admin password to use
                            for the restore process. This password is required ONLY IF the password
                            of the "admin" user set during the backup differs from the log-in
                            password for the current session. (If you provide an incorrect password,
                            the restore will be aborted.) If the passwords are different, note that
                            restoring the user settings from the backup will cause the current user
                            password to become invalid. </para></step><step><para>If the restore was accepted, a progress dialog box will display the
                            restored files:</para><screenshot><graphic fileref="resources/backup3.png"/></screenshot></step></procedure><section><title>Using the Command Line</title><para>To restore from a backup using the command-line utility, follow the
                        instructions above for launching <filename>bin/backup.sh</filename> (Unix),
                        OR the <filename>bin/backup.bat</filename> (Windows/DOS) script files.
                        Include the <parameter>-r</parameter> parameter, and the full path of the
                        <filename>__contents__.xml</filename> file to restore. As with the Admin
                        Client, if the backup uses a different password for the "admin" user than
                        the current session, you must specify the backup password using the
                        <parameter>-P</parameter>. For Example:</para><synopsis>bin/backup.sh -u admin -p admin-pass -P backup-pass 
                        -r /var/backup/hd060501/db/__contents__.xml</synopsis></section></section><section><title>Repairing the Package Repository After Restore</title><para>After a complete restore, your package repository will probably be out of
                    sync. The dashboard will not show all the packages you had previously installed,
                    even though their data collections have hopefully been restored (into
                        <filename>/db/apps</filename> unless you changed the default repository
                    root).</para><para>This happens because the package registry is stored on the file system and
                    is not part of a backup. During a restore, only the contents of your packages are
                    written back into the database.</para><para>A manual "repair" step is required to get the package repository into
                    sync again. The repair procedure is implemented as an XQuery module, which you can
                    run via eXide or the Java admin client. The module provides two
                    functions:</para><variablelist><varlistentry><term>repair:clean-all()</term><listitem><para>Unlinks all currently installed packages from the package manager without 
                                removing deployed data from the db.</para></listitem></varlistentry><varlistentry><term>repair:repair()</term><listitem><para>Scan the app root collection in the db for deployed packages and register
                                each of them with the package repository. This reconstructs the metadata for
                                the package. The data stored in the db will not be modified in any way.</para></listitem></varlistentry><varlistentry><term>repair:repair ($collection)</term><listitem><para>Only try to repair the application whose deployed data is stored in the given
                                collection.</para></listitem></varlistentry></variablelist><para>After a complete restore into a clean database, the clean-all and repair functions would
                    typically be combined in the following XQuery:</para><programlisting language="xquery">xquery version "3.0";

import module namespace repair="http://exist-db.org/xquery/repo/repair" 
at "resource:org/exist/xquery/modules/expathrepo/repair.xql";

repair:clean-all(),
repair:repair()</programlisting></section></section><section><title>Backing Up Single Apps</title><para>Creating a backup of a single expath application or library package is rather
                straightforward within eXide. Either use:</para><itemizedlist><listitem><para>the <ulink url="development-starter.xml#synchronize">synchronize
                            feature</ulink> to write the package contents to a directory on disk,
                        or</para></listitem><listitem><para>call <guimenuitem>Application</guimenuitem> / <guimenuitem>Download
                            App</guimenuitem> to retrieve a <filename>.xar</filename> package which
                        can be deployed into another eXist-db instance</para></listitem></itemizedlist></section><section id="standalone"><title>Standalone Backup/Restore</title><section><title>Standalone Backup/Restore on Server platforms</title><para>The above instructions assume that you have a standard eXist installation directory in place, either from a release package or a full build environment. That is not
                    always the case on remotely deployed server instances that might be running using only an eXist WAR file in an application server (eg. Tomcat).</para><para>To support such deployment scenarios, and make it easier to do restores local to the deployed server, there is an ant build target called:
                    <synopsis>backrest</synopsis></para><para>Building the backrest target will create a fully-self contained zip file in the backrest directory that can be copied to a remote server and used to do both restores and backups local to that server.
                </para><para>Just unzip the backrest zip file, preserviding subdirectories, and execute either the backup.bat or backup.sh scripts per the prior instructions and parameter usage.
                </para></section></section></chapter></book>