<?xml version="1.0" encoding="UTF-8"?>
<!-- $Id: indexing.xml 18225 2013-02-05 20:33:39Z wolfgang_m $ --><book><bookinfo><productname>eXist-db – Open Source Native XML Database</productname><title>Configuring Database Indexes</title><date>November 2009</date><orgname>The eXist-db Project</orgname></bookinfo><chapter><title>Configuring Database Indexes</title><section><title>Overview</title><para>In this section, we discuss the types of database indexes used by eXist-db, as well
                as how they are created, configured and maintained. It assumes readers have a basic
                understanding of XML and XQuery.</para><para>Database indexes are used extensively by eXist-db to facilitate efficient querying of
                the database. This is accomplished both by system-generated and user-configured
                database indexes. The current version of eXist-db by default includes the following
                types of indexes:</para><orderedlist><listitem><para><emphasis><ulink url="#structuralidx">Structural Indexes</ulink></emphasis>: This index keeps track of the elements, attributes, and the nodal structure
	  of all XML documents in a collection.</para></listitem><listitem><para><emphasis><ulink url="#rangeidx">Range Indexes</ulink></emphasis>: These map specific text nodes and attributes of the documents
                        in a collection to typed values.</para></listitem><listitem><para><emphasis><ulink url="ftlegacy.xml">Legacy Full Text Indexes</ulink></emphasis>These map specific text nodes and attributes of the documents in
                        a collection to text tokens.</para></listitem><listitem><para><emphasis><ulink url="lucene.xml">Full Text Indexes</ulink></emphasis>This full text indexing module features faster and
                        customizable full text indexing by transparently integrating Lucene into the
                        XQuery engine. Prefer this index over the Legacy Full Text Index.
                    </para></listitem><listitem><para><emphasis><ulink url="ngram.xml">NGram Indexes</ulink></emphasis>: These map specific text nodes and attributes of the documents
                        in a collection to splitted tokens of n-characters (where n = 3 by default).
			This is very efficient for exact substring searches and for queries on scripts
                        (mostly non-European ones) which can not be easily split into whitespace
                  separated tokens and are thus a bad match for the Lucene full text index. </para></listitem><listitem><para><emphasis><ulink url="devguide_indexes.xml#spatial">Spatial Indexes</ulink>
                            (Experimental)</emphasis>: These map elements of the documents in a
                        collection containing georeferenced geometries to dedicated data structures
                        that allow efficient spatial queries. </para></listitem><listitem><para><emphasis><ulink url="devguide_indexes.xml#spatial">xml:id Index</ulink></emphasis>: An index of all xml:id attribute values is automatically created. 
                    These values can be queried by fn:id(). </para></listitem></orderedlist>
      <!--<note>
                <para>Currently, comments and processing instruction nodes are not indexed. Whether
                    they should be or not, and how some processing instructions could "hint" the
                    indexing process is still being considered.</para>
      </note>--><para>Since version 1.2, eXist-db features a new modularized indexing architecture. Most
                types of indexes have been moved out of the database core and are now maintained as
                pluggable extensions. The full text, ngram and spatial indexes fall under this
                category. </para></section><section><title>Built-in Indexes</title><para>This section describes the features of those indexes that are part of the eXist-db
                distribution. Some of those indexes (n-gram, spatial) may need to be enabled first
                (see <ulink url="#moduleconf">below</ulink>). </para><section id="structuralidx"><title>Structural index</title><para>This index keeps track of the elements (tags), attributes, and nodal structure
                    for all XML documents in a collection. It is created and maintained
                    automatically in eXist-db, and can neither be reconfigured nor disabled by the
                    user. The structural index is required for nearly all XPath and XQuery
                    expressions in eXist-db (with the exception of wildcard-only expressions such as
                        "<command>//*</command>"). This index is stored in the database file
                        <filename>elements.dbx</filename>. </para><para>Technically, the structural index maps every element and attribute
                        <emphasis>qname</emphasis> (or <emphasis>qualified name</emphasis>) in a
                    document collection to a list of <sgmltag>documentId, nodeId</sgmltag> pairs.
                    This mapping is used by the query engine to resolve queries for a given XPath
                    expression.</para><para>For example, given the following query:</para><synopsis>//book/section</synopsis><para>eXist-db uses two index lookups: the first for the <sgmltag>book</sgmltag> node,
                    and the second for the <sgmltag>section</sgmltag> node. It then computes the
                        <emphasis>structural join</emphasis> between these node sets to determine
                    which <sgmltag>section</sgmltag> elements are in fact children of
                        <sgmltag>book</sgmltag> elements.</para></section><section id="rangeidx"><title>Range index</title><para>Range indexes provide a shortcut for the database to directly select nodes
                    based on their typed values. They are used when matching or comparing nodes by
                    way of standard XPath operators and functions. Without a range index, comparison
                    operators like =, &gt; or &lt; will default to a "brute-force" inspection of the
                    DOM, which can be extremly slow if eXist-db has to search through maybe millions of
                    nodes: each node has to be loaded and cast to the target type.</para><para>To see how range indexes work, consider the following fragment:</para><example><title>Example: List Entry</title><programlisting><markup>&lt;items&gt;
    &lt;item n="1"&gt;
       &lt;name&gt;Tall Bookcase&lt;/name&gt;
       &lt;price&gt;299.99&lt;/price&gt;
    &lt;/item&gt;
    &lt;item n="2"&gt;
       &lt;name&gt;Low Bookcase&lt;/name&gt;
       &lt;price&gt;199.99&lt;/price&gt;
    &lt;/item&gt;
&lt;/items&gt;</markup></programlisting></example><para>With this short inventory, the text nodes of the <sgmltag>price</sgmltag>
                    elements have dollar values expressed as a floating-point number, (e.g.
                    "299.99"), which has an <ulink url="http://www.w3.org/TR/xmlschema-0/">XML
                        Schema Definition</ulink> (XSD) data type of <command>xs:double</command>.
                    Using this builtin type to define a range index, we can improve the efficiency
                    of searches for <sgmltag>price</sgmltag> values. (Instructions on how to
                    configure range indexes using configuration files are provided under the <ulink url="#idxconf">Configuring Indexes</ulink> section below.) During indexing,
                    eXist-db will apply this data type selection by attempting to cast all
                        <sgmltag>price</sgmltag> values as double floating point numbers, and add
                    appropriate values to the index. Values that cannot be cast as double floating
                    point numbers are therefore ignored. This range index will then be used by any
                    expression that compares <sgmltag>price</sgmltag> to an <command>xs:double</command> value - for instance:</para><synopsis>//item[price &gt; 100.0]</synopsis><para>For non-string data types, the range index provides the query engine with a more
                    efficient method of data conversion. Instead of retrieving the value of each
                    selected element and casting it as a <command>xs:double</command> type, the
                    engine can evaluate the expression by using the range index as a form of lookup
                    index. Without an index, eXist-db has to do a full scan over all price
                        <sgmltag>price</sgmltag> elements, retrieve the string values of their text
                    node and cast them to a double number. This is a time-consuming process which
                    also scales very badly with growing data sets. With a proper index, eXist-db needs
                    just a single index lookup to evaluate <command>price = 100.0</command>. The
                    range expression <command>price &gt; 100.0</command> is processed with an index
                    scan starting at 100. </para><para>For string data, the index will also be used by the standard functions
                        <function>fn:contains()</function>, <function>fn:starts-with()</function>,
                        <function>fn:ends-with()</function> and <function>fn:matches()</function>.</para><para>To illustrate this functionality, let's return to the previous example. If you
                    define a range index of type <command>xs:string</command> for element
                        <sgmltag>name</sgmltag>, a query on this element to select tall bookcases
                    using <function>fn:matches()</function> will be supported by the following
                    index:</para><synopsis>//item[fn:matches(name, '[Tt]all\s[Bb]')]</synopsis><para>Note that <function>fn:matches</function> will by default try to match the
                    regular expression <emphasis>anywhere</emphasis> in the string. We can thus
                    speed up the query dramatically by using "^" to restrict the match to the start
                    of the string: </para><synopsis>//item[fn:matches(name, '^[Tt]all\s[Bb]')]</synopsis><para>Also, if you really need to search for an exact substring in a longer text
                    sequence, it is often better to use the NGram index instead of the range index,
                    i.e. use <function>ngram:contains()</function> instead of <function>fn:contains()</function>. Unfortunately, there's no equivalent NGram
                    function for <function>fn:matches()</function> yet, but we may add one in the
                    future as it could help to increase performance dramatically.</para><para>In general, three conditions must be met in order to optimize a search using a
                    range index:</para><orderedlist><listitem><para><emphasis>The range index must be defined on <emphasis>all</emphasis>
                                items in the input sequence.</emphasis></para><para>For example, suppose you have two collections in the database: C1 and
                            C2. If you have a range index defined for collection C1, but your query
                            happens to operate on both C1 and C2, then the range index would
                                <emphasis>not</emphasis> be used. The query optimizer selects an
                            optimization strategy based on the entire input sequence of the query.
                            Since, in this example, since only nodes in C1 have a range index, no
                            range index optimization would be applied.</para></listitem><listitem><para><emphasis>The index data type (first argument type) must match the test
                                data type (second argument type).</emphasis></para><para>In other words, with range indexes, there is no promotion of data
                            types (i.e. no data type precedes or replaces another data type). For
                            example, if you defined an index of type <command>xs:double</command> on
                                <sgmltag>price</sgmltag>, a query that compares this element's value
                            with a string literal would not use a range index, for instance:</para><synopsis>//item[price = '1000.0']</synopsis><para>In order to apply the range index, you would need to cast the value as
                            a type <command>xs:double</command>, i.e.:</para><synopsis>//item[price = xs:double($price)] (where $price is any test value)</synopsis><para>Similarly, when we compare <command>xs:double</command> values with
                                <command>xs:integer</command> values, as in, for instance:</para><synopsis>//item[price = 1000]</synopsis><para>the range index would again not be used since the
                                <sgmltag>price</sgmltag> data type differs from the test value type,
                            although this conflict might not seem as obvious as it is with string
                            values.</para></listitem><listitem><para><emphasis>The right-hand argument has no dependencies on the current
                                context item.</emphasis></para><para>That is, the test or conditional value must not depend on the value
                            against which it is being tested. For example, range indexes will not be
                            applied given the following expression:</para><synopsis>//item[price = self]</synopsis></listitem></orderedlist><para>Concerning range indexes on strings there's another restriction to be
                    considered: up to version 1.3, range indexes on strings can only be used with
                    the default Unicode collation. Also, string indexes will always be case
                    sensitive (while n-gram and full text indexes are not). It is not yet possible
                    to define a string index on a different collation (e.g. for German or French) or
                    to make it case insensitve. This is a limitation we plan to address in the future.</para></section><section id="spatialidx"><title>Spatial Index</title><para>A working proof-of-concept index, which listens for spatial geometries
                    described through the Geography Markup Language (GML). A detailed description of
                    the implementation can be found in the <ulink url="devguide_indexes.xml#spatial">Developer's Guide to Modularized Indexes</ulink>.</para></section></section><section><title>Optional Index Modules</title><para>eXist-db features a <emphasis>modularized indexing architecture</emphasis>, which
                allows arbitrary indexes to be plugged into an indexing pipeline. Consequently, some
                indexes were moved out of the database core and are now available as plugins. For
                the database core, those indexes are a black box: they handle their own creation,
                configuration, destruction, etc.</para><para>While the structural and the range index are always available, the optional
                indexes can be enabled or disabled on a given database instance. Optional modules
                are:</para><variablelist><varlistentry><term><ulink url="ngram.xml">N-Gram Index</ulink></term><listitem><para>N-gram indexes are optimized for exact substring queries (like
                                <function>contains</function>). Substring searches are nearly as
                            fast as with the full text index. However, the n-gram index also
                            preserves whitespace and punctuation, and is case-insensitive by
                            default.</para></listitem></varlistentry><varlistentry><term><ulink url="lucene.xml">New Full Text Index</ulink></term><listitem><para>Faster, better configurable, more feature rich and reliable than
                            eXist-db's old index.</para></listitem></varlistentry><varlistentry><term><ulink url="ftlegacy.xml">Legacy Full Text Index</ulink></term><listitem><para>Legacy full text index. Deprecated, though still supported for backwards
                            compatibility.</para></listitem></varlistentry><varlistentry><term><ulink url="devguide_indexes.xml">Spatial Index</ulink></term><listitem><para>A detailed description can be found in the <ulink url="devguide_indexes.xml">Developer's Guide to Modularized
                                Indexes</ulink>.</para></listitem></varlistentry></variablelist></section><section id="moduleconf"><title>Enabling Index Modules</title><para>While some indexes (n-gram, full text) are already pre-build in the standard
                eXist-db distribution, other modules may need to be enabled first. For example, the
                spatial index depends on a bunch of external libraries, which do not ship with
                eXist-db. However, enabling the spatial index is a simple process:</para><procedure><step><para>Copy the properties file <filename>extensions/indexes/build.properties</filename> and store it as
                            <filename>local.build.properties</filename> in the same directory if it does not already exist.</para></step><step><para>Edit <filename>extensions/indexes/local.build.properties</filename>:</para><example><title>local.build.properties</title><programlisting>
# N-gram module
include.index.ngram = true

# Spatial module
include.index.spatial = false
                        </programlisting></example><para>To include an index, change the corresponding property to "true".</para></step><step><para>Call the Ant build system once to regenerate the eXist-db libraries:</para><synopsis>build.sh</synopsis><para>or</para><synopsis>build.bat</synopsis></step></procedure><para>The build process should create a jar file for every index implementation in
                directory <filename>lib/extensions</filename>. For example, the spatial index is
                packaged into the jar <filename>exist-spatial-module.jar</filename>.</para><para>Once the index module has been built, it can be announced to eXist-db. To activate an
                index plugin, it needs to be added to the <sgmltag>modules</sgmltag> section within
                the global configuration file <filename>conf.xml</filename>:</para><example><title>Index Plugin Configuration in conf.xml</title><programlisting><markup>&lt;modules&gt;
    &lt;module id="ngram-index" class="org.exist.indexing.ngram.NGramIndex"
        file="ngram.dbx" n="3"/&gt;
    &lt;!-- The full text index is always required and should
         not be disabled. We still have some dependencies on
         this index in the database core. These will be removed
         once the redesign has been completed. --&gt;
    &lt;module id="ft-legacy-index" class="org.exist.fulltext.FTIndex"/&gt;
&lt;/modules&gt;</markup></programlisting></example><para>Every <sgmltag>module</sgmltag> element needs at least an <option>id</option> and
                    <option>class</option> attribute. The class attribute contains the name of the
                plugin class, which has to be an implementation of
                    <classname>org.exist.indexing.Index</classname>.</para><para>All other attributes or nested configuration elements below the
                    <sgmltag>module</sgmltag> element are specific to the implementation and will
                differ between indexes. They should be documented by the index implementor.</para><para>If an index implementation cannot be loaded from the specified class, the entry
                will simply be ignored. A warning will be written to the logs which should provide
                more information on the issue which caused the configuration to fail.</para></section><section id="idxconf"><title>Configuring Indexes</title><para>eXist-db has no "create index" command. Instead, indexes are configured in
                collection-specific configuration files. These files are stored as standard XML
                documents in the system collection: <filename>/db/system/config</filename>, which
                can be accessed like any other document (e.g. using the Admin interface or Java
                Client). In addition to defining settings for indexing collections, the
                configuration document specifies collection-specific other settings such as triggers
                or default permissions.</para><para>The contents of the system collection (<filename>/db/system/config</filename>)
                should mirror the hierarchical structure of the main collection. Configurations are shared
                by descendants in the hierarchy unless they have their own configuration (i.e. the
                configuration settings for the child collection override those set for the parent).
                If no collection-specific configuration file is created for any document, the global
                settings in the main configuration file, <filename>conf.xml</filename>, will apply
                by default. That being said, the <filename>conf.xml</filename> file should only
                define the default global <emphasis>index creation policy</emphasis>.</para><para> To configure a given collection - e.g. <filename>/db/foo</filename> - create a
                file <filename>collection.xconf</filename> and store it as <filename>/db/system/config/db/foo/collection.xconf</filename>. Note the replication of
                the <filename>/db/foo</filename> hierarchy inside <filename>/db/system/config/</filename>. Subcollections which do not have a <filename>collection.xconf</filename> file of their own will be governed by the
                configuration policy specified for the closest ancestor collection which does have
                such a file, so you are not required to specify a configuration for every
                collection. Note, however, that configuration settings do not cascade. If you choose
                to deploy a <filename>collection.xconf</filename> file in a subcollection, you must
                specify in that file <emphasis>all</emphasis> the configuration options you wish to
                have applied to that subcollection (and any lower-level subcollections without
                    <filename>collection.xconf</filename> files of their own). </para><note><para>Due to backward compatibility concerns, the file does not have to be called
                        <filename>collection.xconf</filename>, which is now the preferred file name,
                    but it <emphasis>must</emphasis> have the <filename>.xconf</filename>
                    extension.</para><para>You can only have <emphasis>one</emphasis><filename>.xconf</filename> file at each level.</para></note><section><title>Maintaining Indexes and Re-indexing</title><table><title/><tgroup cols="2"><tbody><tr><td><para>The eXist-db index system automatically maintains and updates
                                        indexes defined by the user. You therefore do not need to
                                        update an index when you update a database document or
                                        collection. eXist-db will even update indexes following partial
                                        document updates via <emphasis>XUpdate</emphasis> or
                                            <emphasis>XQuery Update</emphasis> expressions.</para><para>The only exception to eXist-db's automatic update occurs when
                                        you <emphasis>add a new index definition to an existing
                                            database collection</emphasis>. In this case, the new
                                        index settings will <emphasis>only</emphasis> apply to new
                                        data added to this collection, or any of its
                                        sub-collections, and <emphasis>not</emphasis> to previously
                                        existing data. To apply the new settings to the entire
                                        collection, you need to trigger a "manual reindex" of the
                                        collection being updated. You can re-index collections using
                                        the Java Admin Client (shown on the right). From the Admin
                                        menu, select File»Reindex Collection.
                                    </para><para>You can also index by passing an XQuery to eXist-db:
                                        <example><programlisting language="xquery">
                                            xmldb:reindex('/db/foo')</programlisting></example></para></td><td><screenshot><graphic fileref="resources/client4.png"/></screenshot></td></tr></tbody></tgroup></table></section><section><title>General Configuration Structure and Syntax</title><para>Index configuration files are standard XML documents that have their elements
                    and attributes defined by the eXist-db namespace:</para><synopsis>http://exist-db.org/collection-config/1.0</synopsis><para>The following example shows a configuration example:</para><example><title>Configuration Document</title><programlisting><markup>
&lt;collection xmlns="http://exist-db.org/collection-config/1.0"&gt;
    &lt;index&gt;
        
        &lt;!-- Legacy full text index configuration. Deprecated. --&gt;
        &lt;fulltext default="none" attributes="false"&gt;
            &lt;!-- Full text indexes --&gt;
            &lt;create qname="author"/&gt;
            &lt;create qname="title" content="mixed"/&gt;
            &lt;!-- "old" context-dependant configuration using the path attribute: --&gt;
            &lt;include path="booktitle"/&gt;
        &lt;/fulltext&gt;
        
        &lt;!-- New full text index based on Lucene --&gt;
        &lt;lucene&gt;
            &lt;text qname="SPEECH"&gt;
                &lt;ignore qname="SPEAKER"/&gt;
            &lt;/text&gt;
            &lt;text qname="TITLE"/&gt;
        &lt;/lucene&gt;
        
        &lt;!-- Range indexes --&gt;
        &lt;create qname="title" type="xs:string"/&gt;
        &lt;create qname="author" type="xs:string"/&gt;
        &lt;create qname="year" type="xs:integer"/&gt;
        &lt;!-- "old" context-dependant configuration using the path attribute: --&gt;
        &lt;create path="//booktitle" type="xs:string"/&gt;

        &lt;!-- N-gram indexes --&gt;
        &lt;ngram qname="author"/&gt;
        &lt;ngram qname="title"/&gt;
    &lt;/index&gt;
&lt;/collection&gt;
</markup></programlisting></example><para>All configuration documents documents have an <sgmltag>index</sgmltag>
                    element directly below the root element, which encloses the index configuration.
                    Only <emphasis>one</emphasis><sgmltag>index</sgmltag> element is permitted in a document. Apart from the
                    index configuration, the document may also contain settings not related to indexing,
                    e.g. for triggers; these will not be covered here. </para><para>In the <sgmltag>index</sgmltag> element are elements that define the various
                    index types. Each index type can add its own configuration elements, which are
                    directly forwarded to the corresponding index implementation. The example above
                    configures three different types of indexes: full text, range and ngram.</para><section><title>Namespaces</title><para>If the document to be indexed uses namespaces, you should add an
                            <option>xmlns</option> attribute for each of the required namespaces to
                        the <sgmltag>index</sgmltag> element:</para><example><title>Using Namespaces</title><programlisting><markup>
&lt;collection xmlns="http://exist-db.org/collection-config/1.0"&gt;
    &lt;index xmlns:atom="http://www.w3.org/2005/Atom"&gt;
        &lt;fulltext default="none" attributes="no"&gt;
            &lt;create qname="atom:title"/&gt;
        &lt;/fulltext&gt;
        &lt;create qname="atom:id" type="xs:string"/&gt;
    &lt;/index&gt;
&lt;/collection&gt;
</markup></programlisting></example><para>The example configuration above creates two indexes on a collection of
                        atom documents. The two elements which should be indexed are both in the
                            <literal>atom</literal> namespace and we thus need to declare a mapping
                        for this namespace. Please note that the <option>xmlns</option> namespace
                        attributes have to be specified on the <sgmltag>index</sgmltag> element, not
                        the <sgmltag>create</sgmltag> or <sgmltag>fulltext</sgmltag>
                        elements.</para></section><section><title>Range index configuration</title><example><title>Range Index Configuration</title><programlisting><markup>&lt;!-- Range indexes --&gt;
&lt;create qname="title" type="xs:string"/&gt;
&lt;create qname="author" type="xs:string"/&gt;
&lt;create qname="year" type="xs:integer"/&gt;
&lt;!-- "old" context-dependant configuration using the path attribute: --&gt;
&lt;create path="//booktitle" type="xs:string"/&gt;</markup></programlisting></example><para>A range index is configured by adding a <sgmltag>create</sgmltag> element
                        directly below the root <sgmltag>index</sgmltag> element. As explained above,
                        the node to be indexed is either specified through a <option>path</option>
                        or a <option>qname</option> attribute.</para><para>As range indexes are type specific, the <option>type</option> attribute is
                        always required. The type should be one of the atomic XML schema types,
                        currently including <command>xs:string</command>, <command>xs:integer</command> and its derived types <command>xs:double</command> and <command>xs:float</command>, <command>xs:boolean</command> and <command>xs:dateTime</command>. Further types
                        may be added in the future. If the name of the type is unknown, the index
                        configuration will be ignored and you will get a warning written into the
                        logs.</para><para>Please note that the index configuration will only apply to the node
                        specified via the <option>path</option> or <option>qname</option> attribute,
                        not to descendants of that node. Consider a mixed content element
                        like:</para><example><title>Mixed Content Element</title><programlisting><markup>&lt;mixed&gt;&lt;s&gt;un&lt;/s&gt;&lt;s&gt;even&lt;/s&gt;&lt;/mixed&gt;</markup></programlisting></example><para>If an index is defined on <sgmltag>mixed</sgmltag>, the key for the index
                        is built from the concatenated text nodes of element
                            <sgmltag>mixed</sgmltag> and all its descendants, i.e. "uneven". The
                        created index will only be used to evaluate queries on
                            <sgmltag>mixed</sgmltag>, but not for queries on <sgmltag>s</sgmltag>.
                        However, you can create an additional index on <sgmltag>s</sgmltag> without
                        getting into conflict with the existing index on
                        <sgmltag>mixed</sgmltag>.</para></section><section id="pathvsqname"><title>Configuration by path vs. configuration by qname</title><para>It is important to note the difference between the <option>path</option>
                        and <option>qname</option> attributes used throughout above example. Both
                        attributes are used to define the elements or attributes to which the index
                        should be applied. However, the <option>path</option> attribute creates
                            <emphasis>context-dependant</emphasis> indexes, while the
                            <option>qname</option> attribute does not. The path attribute takes a
                        simple path expression: </para><synopsis>&lt;create path="//book/title" type="xs:string"/&gt;</synopsis><para>The path expression looks like XPath, but it's really not. Index path
                        syntax uses the following components to construct paths:</para><itemizedlist><listitem><para>Elements are specified by their <emphasis>qname</emphasis></para></listitem><listitem><para>Attributes are specified by <option>@attributeName</option>, so if
                                the attribute is called "attrib1", one uses
                                    <option>@attrib1</option> in the index specification.</para></listitem><listitem><para>Child nodes are selected using the forward-slash
                                    (<option>/</option>)</para></listitem><listitem><para>All descendant nodes in a tree are selected using the double
                                forward-slash (<option>//</option>)</para></listitem></itemizedlist><para>The example above creates a range index of type string on all
                            <sgmltag>title</sgmltag> elements which are children of
                            <sgmltag>book</sgmltag> elements, which may occur at an arbitrary
                        position in the document tree. All other <sgmltag>title</sgmltag> elements,
                        e.g. those being children of <sgmltag>section</sgmltag> nodes, are not
                        indexed. The path expression thus defines a <emphasis>selective</emphasis>
                        index, which is also <emphasis>context-dependant</emphasis>: we always need
                        look at the context of each <sgmltag>title</sgmltag> node before we can
                        determine if this particular title is to be indexed or not.</para><para>This kind of context-dependant index definition helps to keep the index
                        small, but unfortunately it makes it hard for the query optimizer to
                        properly rewrite the expression tree without missing some nodes. The
                        optimizer needs to make an optimization decision at compile time, where the
                        context of an expression is unknown or at least not exactly known (read the
                        <ulink url="http://atomic.exist-db.org/blogs/eXist/NewIndexing">blog
                            article</ulink> to get the whole picture). This means that some of the
                        highly efficient optimization techniques can not be applied to
                        context-dependant indexes!</para><para>We thus had to introduce an alternative configuration method which is not
                        context-dependant. To keep things simple, we decided to define the index on
                        the <emphasis>qname</emphasis> of the element or attribute alone and to
                        ignore the context altogether:</para><synopsis>&lt;create qname="title" type="xs:string"/&gt;</synopsis><para>This results in an index being created on every <sgmltag>title</sgmltag>
                        element found in the document node tree. Section titles will be indexed as
                        well as chapter or book titles. Indexes on attributes are defined as above
                        by prepending "@" to the attribute's name, e.g.:</para><synopsis>&lt;create qname="@type" type="xs:string"/&gt;</synopsis><para>defines an index on all attributes named "type", but not on elements with
                        the same name.</para><para>Defining indexes on qnames may result in a considerably larger index, but
                        it also allows the query engine to apply all available optimization
                        techniques, which can improve query times by an order of magnitude. As so
                        often, there's a trade-off between performance and storage space. In many
                        cases, the performance win can be dramatic enough to justify an increase in
                        index size.</para><important><para>To be on the safe side and to benefit from current and future
                            improvements in the query engine, you should prefer
                                <option>qname</option> over <option>path</option> - unless you
                            really need to exclude certain nodes from indexing.</para></important></section><section id="luceneidx"><title>New Full Text Index</title><para>Please refer to the <ulink url="lucene.xml">separate
                        documentation</ulink>.</para></section></section></section><section><title>Check Index Usage</title><para>It is sometimes a bit difficult to see if a range index is correctly defined or
                not. The simplest way to get some information on index usage is to set the priority
                for eXist-db's standard logger to <option>TRACE</option>. For example, change the
                    <sgmltag>root</sgmltag> category in <filename>log4j.xml</filename> as
                follows:</para><example><title>Configure log4j to Display Trace Output</title><programlisting><markup>&lt;root&gt;
    &lt;priority value="trace"/&gt;
    &lt;appender-ref ref="console"/&gt;
&lt;/root&gt;</markup></programlisting></example><para>This enables trace and sends all log output to the console instead of the log
                files. For expressions that can benefit from a range index, you should now see
                messages like "Checking if range index can be used ..." or "Using range index for
                key...".</para><synopsis>//city[population &gt; 100000]</synopsis><para>Without further query optimizations, I receive the following trace output:</para><example><title>TRACE Output</title><screen>
04 Jan 2008 14:17:51,767 [main] TRACE (GeneralComparison.java [quickNodeSetCompare]:543) - found an index of type: xs:integer
04 Jan 2008 14:17:51,768 [main] TRACE (GeneralComparison.java [quickNodeSetCompare]:587) - Checking if range index can be used for key: 10000
04 Jan 2008 14:17:51,768 [main] TRACE (GeneralComparison.java [quickNodeSetCompare]:592) - Using range index for key: 10000
</screen></example><para>If you enabled query rewriting (by setting
                    <option>enable-query-rewriting="yes"</option> in <filename>conf.xml</filename>,
                you will see additional messages from the optimizer:</para><example><title>TRACE Output with Additional Query Optimization</title><screen>
04 Jan 2008 14:26:24,851 [main] TRACE (Optimize.java [visitGeneralComparison]:177) - exist:optimize: found optimizable: org.exist.xquery.GeneralComparison
04 Jan 2008 14:26:24,856 [main] TRACE (Optimize.java [before]:198) - exist:optimize: context step: descendant-or-self::country[child::population &gt; 10000]
04 Jan 2008 14:26:24,856 [main] TRACE (GeneralComparison.java [preSelect]:260) - Using QName index on type xs:integer
04 Jan 2008 14:26:24,856 [main] TRACE (GeneralComparison.java [preSelect]:292) - Using QName range index for key: 10000
04 Jan 2008 14:26:24,865 [main] TRACE (Optimize.java [eval]:89) - exist:optimize: pre-selection: 1483
04 Jan 2008 14:26:24,866 [main] TRACE (Optimize.java [eval]:116) - Ancestor selection took 1
04 Jan 2008 14:26:24,866 [main] TRACE (Optimize.java [eval]:117) - Found: 49
</screen></example><para>Another possibility to see what's in your index is to use the <function>util:index-keys</function> function:</para><example><title>Query to List Range Index Contents</title><programlisting language="xquery">declare function local:term-callback($term as xs:string, $data as xs:integer+) as element() {
   &lt;entry&gt;
     &lt;term&gt;{$term}&lt;/term&gt;
     &lt;frequency&gt;{$data[1]}&lt;/frequency&gt;
     &lt;documents&gt;{$data[2]}&lt;/documents&gt;
     &lt;position&gt;{$data[3]}&lt;/position&gt;
   &lt;/entry&gt;
};

util:index-keys(//name, "A", util:function(xs:QName("local:term-callback"), 2), 1000)</programlisting></example><para>This query will show you the first 1000 keys (starting with the letter 'A')
                indexed for the element selected by the range index on <option>name</option>
                together with some information about this key. The first argument to index-keys
                specifies the node set for which index entries should be listed. The second argument
                contains a start value which also determines the index type. For example, if the
                start value is a string, eXist-db will only search for indexes configured with type
                    <option>xs:string</option>, if it is an integer number, only indexes with type
                    <option>xs:integer</option> will be considered. For a string index, you may also
                pass the empty string to retrieve all keys in the index.</para><para>To view the contents of the Lucene full text index, you need to specify the index type as well:</para><example><title>Query to List Full Text Index Contents</title><programlisting language="xquery">
declare function local:term-callback($term as xs:string, $data as xs:integer+) as element() {
   &lt;entry&gt;
     &lt;term&gt;{$term}&lt;/term&gt;
     &lt;frequency&gt;{$data[1]}&lt;/frequency&gt;
     &lt;documents&gt;{$data[2]}&lt;/documents&gt;
     &lt;position&gt;{$data[3]}&lt;/position&gt;
   &lt;/entry&gt;
};

util:index-keys(//SCENE, "a", util:function(xs:QName("local:term-callback"), 2), 1000, 'lucene-index')
</programlisting></example><para>The Admin panel contains an Indexing page which provides access to the indexes through the above functions. 
                The Query Profiling page provides ways to trace whether the indexes are used optimally.</para></section></chapter></book>