<?xml version="1.0" encoding="UTF-8"?>
<!-- $Id: troubleshooting.xml 18722 2013-07-17 09:16:42Z wolfgang_m $ --><book><bookinfo><productname>eXist-db – Open Source Native XML Database</productname><title>Troubleshooting</title><date>October 2012</date></bookinfo><chapter><title>Troubleshooting</title><section id="introduction"><title>Introduction</title><para>This document contains hints and tips about troubleshooting problems. It also
                tells you where to find information (such as version numbers, log information, etc.)
                that is very helpful to know when you need to reach out to <ulink url="getting-help.xml">get help</ulink>.</para></section><section id="normal-start-up"><title>Normal Start Up</title><para>While eXist-db starts up, log output appears in the console. If you started
                eXist-db via the system tray launcher (default), the console output is captured and
                can be viewed by selecting the <guibutton>Show Tool Window</guibutton> menu entry in
                the system tray popup, then select <guibutton>Show console messages</guibutton>. If
                you launched eXist-db via one of the shell scripts, the output should directly
                appear in those.</para><para>If eXist-db launched properly, you will find output similar to the following (this
                example output is taken from Mac OS X):</para><example><title>Console Output</title><screen>
06 Oct 2012 16:56:32,797 [main] INFO  (JettyStart.java [run]:116) - Configuring eXist from /Applications/eXist/conf.xml 
06 Oct 2012 16:56:32,798 [main] INFO  (JettyStart.java [run]:117) -  
06 Oct 2012 16:56:32,798 [main] INFO  (JettyStart.java [run]:118) - Running with Java 1.6.0_35 [Apple Inc. (Java HotSpot(TM) 64-Bit Server VM) in /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home] 
06 Oct 2012 16:56:32,798 [main] INFO  (JettyStart.java [run]:123) -  
06 Oct 2012 16:56:32,824 [main] INFO  (JettyStart.java [run]:127) - [eXist Version : 2.1-dev] 
06 Oct 2012 16:56:32,824 [main] INFO  (JettyStart.java [run]:129) - [eXist Build : 20121006] 
06 Oct 2012 16:56:32,825 [main] INFO  (JettyStart.java [run]:131) - [eXist Home : unknown] 
06 Oct 2012 16:56:32,825 [main] INFO  (JettyStart.java [run]:133) - [SVN Revision : 17031] 
06 Oct 2012 16:56:32,825 [main] INFO  (JettyStart.java [run]:141) - [Operating System : Mac OS X 10.8.2 x86_64] 
06 Oct 2012 16:56:32,825 [main] INFO  (JettyStart.java [run]:144) - [jetty.home : /Applications/eXist/tools/jetty] 
06 Oct 2012 16:56:32,825 [main] INFO  (JettyStart.java [run]:146) - [log4j.configuration : file:/Applications/eXist/log4j.xml] 
06 Oct 2012 16:56:33,698 [main] INFO  (JettyStart.java [lifeCycleStarting]:387) - Jetty server starting... 
06 Oct 2012 16:56:33,699 [main] INFO  (Server.java [doStart]:253) - jetty-8.1.3.v20120416 
Logging already initialized. Skipping...
06 Oct 2012 16:56:34,572 [main] INFO  (NCSARequestLog.java [doStart]:644) - Opened /Applications/eXist/tools/jetty/logs/2012_10_06.request.log 
06 Oct 2012 16:56:34,589 [main] INFO  (AbstractConnector.java [doStart]:333) - Started SelectChannelConnector@0.0.0.0:8080 
06 Oct 2012 16:56:34,690 [main] INFO  (SslContextFactory.java [doStart]:298) - Enabled Protocols [SSLv2Hello, SSLv3, TLSv1] of [SSLv2Hello, SSLv3, TLSv1] 
06 Oct 2012 16:56:34,691 [main] INFO  (AbstractConnector.java [doStart]:333) - Started SslSelectChannelConnector@0.0.0.0:8443 
06 Oct 2012 16:56:34,691 [main] INFO  (JettyStart.java [lifeCycleStarted]:393) - Jetty server started. 
06 Oct 2012 16:56:34,691 [main] INFO  (JettyStart.java [run]:221) - ----------------------------------------------------- 
06 Oct 2012 16:56:34,691 [main] INFO  (JettyStart.java [run]:222) - Server has started on ports 8080 8443. Configured contexts: 
06 Oct 2012 16:56:34,692 [main] INFO  (JettyStart.java [run]:229) - '/exist' 
06 Oct 2012 16:56:34,692 [main] INFO  (JettyStart.java [run]:260) - ----------------------------------------------------- 
</screen></example><para>When you see the "Server has started" message, and no further errors appear, you know that your eXist-db
                installation is working normally.</para><para>However, if you do not even see this message, you should follow these troubleshooting steps</para></section><section id="db-locked"><title>Database Refuses to Start</title><para>If eXist-db was not shut down properly, it may start a recovery process to redo committed transactions and roll
            back uncommitted ones. If an inconsistency is found during this process, eXist-db will automatically abort the startup
            and print out a warning. This emergency stop is done to avoid potential damage and give an administrator a chance to check
            the db and create a backup. It does not necessarily indicate a real problem. In most cases, the db should be ok and restarting it will be save.</para><para>However, we definitely recommend to run a <ulink url="backup.xml#consistency-check">consistency check</ulink> in those cases. If inconsistencies are found, make sure you
                have a backup before continuing. If only one or two resources are affected, it might still be ok to restart, but it's
                good to have a backup just in case.</para></section><section id="clearing-db"><title>Going Back to an Empty Database</title><para>During development and testing you may sometimes wish to go back to a completely empty, fresh database. Here's how to
                really <emphasis>remove everything</emphasis> and reset the db to its initial state:</para><procedure><step><para>Make sure eXist-db is no longer running</para></step><step><para>If you installed the source code (and thus the development tools), call ant as follows:</para><synopsis>./build.sh clean-default-data-dir</synopsis><para>If you do not have build.sh (or build.bat), you may just manually remove the contents of
                        your data directory. By default, the data directory is in <filename>EXIST_HOME/webapp/WEB-INF/data</filename></para></step></procedure></section><section id="environmental-variables"><title>JAVA_HOME and EXIST_HOME Environmental Variables</title><para>When using one of the shell or batch scripts, eXist-db can fail to start up properly if either of the two key environmental
                variables, <envar>JAVA_HOME</envar> and <envar>EXIST_HOME</envar>, are not set
                properly. Both variables are used in the <filename>startup.bat</filename> and
                <filename>startup.sh</filename> scripts and have to be set correctly before the
                scripts are run (you can also insert the lines required in the beginning of the
                scripts themselves).</para><itemizedlist><listitem><para><envar>JAVA_HOME</envar> should point to the directory where Java—the JRE or JDK—is
                        installed. For instructions about how to set JAVA_HOME on
                        Windows, follow the instructions in this <ulink url="http://confluence.atlassian.com/display/DOC/Setting+the+JAVA_HOME+Variable+in+Windows">guide</ulink>; on Linux, follow this <ulink url="http://www.cyberciti.biz/faq/linux-unix-set-java_home-path-variable/">guide</ulink>, and on Mac OS X, follow this <ulink url="http://www.mehtanirav.com/2008/09/02/setting-java_home-on-mac-os-x-105">guide</ulink>. </para></listitem><listitem><para><envar>EXIST_HOME</envar> should point to the directory that contains the
                        configuration file <filename moreinfo="none">conf.xml</filename>, so that
                        the server uses the path <filename>EXIST_HOME/conf.xml</filename>. For
                        example, if the <envar>EXIST_HOME</envar> path is <filename>C:\Program
                            Files\eXist</filename>, the server will look for <filename>C:\Program
                                Files\eXist\conf.xml</filename>. You can set EXIST_HOME in the same way
                        that you set JAVA_HOME; thus, on Mac OS X, you would enter "export
                        EXIST_HOME=/Applications/eXist" in the Terminal. </para></listitem></itemizedlist><para>You should also ensure that you have "write" permissions set for the <filename moreinfo="none">data</filename> directory located in <filename moreinfo="none">webapp/WEB-INF/</filename>.</para></section><section id="port-conflicts"><title>Port Conflicts</title><para>eXist-db can fail to start up if another service on
                your system is using port 8080 or 8443, the default ports that eXist's embedded web server, Jetty, uses. To see whether this is the case, enter <ulink url="http://localhost:8080/">http://localhost:8080/</ulink> in your browser. If
                another service occupies this port, you cannot start up eXist-db unless you shut
                down the service in question or make eXist-db use another port. To make eXist-db use
                another port, open the file <filename moreinfo="none">/tools/jetty/etc/jetty.xml</filename> inside your eXist-db installation in a
                text or XML editor and change the value "8080" in</para><synopsis format="linespecific">&lt;Set name="port"&gt;&lt;SystemProperty name="jetty.port" default="8080"/&gt;&lt;/Set&gt;</synopsis><para>to a port that is not used, e.g. "8899":</para><synopsis format="linespecific">&lt;Set name="port"&gt;&lt;SystemProperty name="jetty.port" default="8899"/&gt;&lt;/Set&gt;</synopsis><para>eXist-db uses port 8443 for confidential communication. Another service may also
                be using this port. To make eXist-db use another port, open the file <filename moreinfo="none">/tools/jetty/etc/jetty.xml</filename> and change the value
                "8443" in</para><synopsis format="linespecific">&lt;Set name="confidentialPort"&gt;8443&lt;/Set&gt;</synopsis><para>and</para><synopsis format="linespecific">&lt;Set name="Port"&gt;8443&lt;/Set&gt;</synopsis><para>to a port that is not used, e.g. "8444".</para><para>If these scripts do not launch eXist-db, you can launch it by changing to the
                directory where you installed eXist-db and entering the following into the
                console:</para><synopsis>java -Xmx1024M -Djava.endorsed.dirs=lib/endorsed -jar start.jar jetty</synopsis><para>If you have problems running the shell/batch scripts, read the section <ulink url="advanced-installation.xml#bin-executables">Running Executable Files</ulink>.</para></section><section id="logs"><title>Using the Logs</title><para>If you experience any problems while using eXist-db, your first step should be
                to check the log files to get additional information about the source of the
                problem. eXist-db uses the <emphasis>log4j-package</emphasis> to write output logs to
                files. By default, this output is written to the directory <filename moreinfo="none">webapp/WEB-INF/logs</filename>.
                Or, if you are running eXist as a service, check the directory <filename>tools/wrapper/logs</filename>. The
                main log files for eXist itself are <filename>exist.log</filename> and
                <filename>xmldb.log</filename>.
                <!-- The other log files show output from Cocoon.  --></para></section><section id="memory"><title>Out of Memory</title><para>Running out of memory typically throws Java into an inconsistent state: some threads may still be alive and continue
            to run while others have died. It is thus important to avoid memory errors up front by checking the memory consumption of
            your queries before they go into production. Should you encounter an out of memory error, please make sure to restart
            eXist and follow the emergency procedure.</para><section><title>Streaming Large Files</title><para>If you have to generate large binaries, e.g. a ZIP or PDF, from within an XQuery, please ensure the content
                    does not need to be kept in memory. There are various XQuery functions which directly stream to the HTTP response.</para><para>There's also a known issue with the betterform XForms filter caching every HTTP response. To work around this,
                your XQuery should be run via an URL which is not processed by the XForms filter: either disable the filter or use
                /rest or /restxq.</para></section></section><section id="emergency"><title>Killing the Database</title><para>If you ever feel you have to kill the database (e.g. because it does not respond - for whatever reason), the recommended 
                procedure is as follows:</para><procedure><step><para>Check if a query is running wild and try to kill it. This can be done either through the
                    "Scheduler" plugin in the dashboard, or the "Running Jobs" section in the "Admin Web Client". Try to kill
                    the query there and wait for a minute if the system returns to normal operations:</para><graphic fileref="resources/schedulerplugin.png"/></step><step><para>Attempt to trigger a proper shutdown either via the system tray icon or the dashboard. Wait for at least
                    3 minutes. Even if eXist-db does not stop completely, it may still be able to complete the shutdown procedure
                    for the core database.</para></step><step><para>It may now be safe to kill the eXist-db process. Check the logs to see if the database has properly 
                    shut down. The last message in the logs would indicate this.</para></step><step><para>If the logs indicate a proper shutdown: before restarting, remove any <filename>.log</filename> and
                    <filename>.lck</filename> files from the data directory. This will prevent a recovery run, which would certainly take
                    time.</para></step><step><para>Otherwise:</para><itemizedlist><listitem><para>You are sure you have no valuable changes in this db instance, e.g. because it's a development system:
                            follow the step above and remove the <filename>.log</filename> files before restart to reduce startup time.</para></listitem><listitem><para>Before restart, archive the contents of the data directory: you may need them if anything goes wrong. Restart 
                                the database but be prepared for a recovery run, which may take considerable time (depending on the size of
                            your db).</para></listitem><listitem><para>If inconsistencies are detected during the recovery, eXist will switch to read-only mode. In this case,
                            stop it again and run a <ulink url="backup.xml#consistency-check">consistency check</ulink>, which can also create a low-level backup.</para><para>If the consistency check reports a number of errors, eXist may still be able to run, but there might
                            be errors in the data structures. So please prepare for a complete restore into a clean data directory
                            as soon as you can take the database offline for maintenance.</para></listitem></itemizedlist></step></procedure><warning><para>Do not repeatedly kill the database. If it does not come up immediately, it may run recovery. This can take some
                time. Killing eXist during recovery will most likely result in additional damages. Always check the logs and console output 
                to see what eXist is doing before you kill it.</para></warning></section></chapter></book>