<?xml version="1.0" encoding="UTF-8"?>
<!-- $Id: lucene.xml 18249 2013-02-06 22:53:25Z ellefj $ --><book><bookinfo><productname>eXist-db – Open Source Native XML Database</productname><title>Lucene Index Module</title><date>February 2013</date><orgname>The eXist-db Project</orgname></bookinfo><chapter><title>Lucene-based Full Text Index</title><section><title>Introduction</title><para>The 1.4 version of eXist-db introduced a new full text indexing module which replaced
                eXist-db's former built-in full text index. This new module is faster, more configurable and
                more feature-rich than eXist-db's old index. It will also be the basis for eXist-db's
                implementation of the W3C's full text extensions for XQuery.</para><para>The new full text module is based on <ulink url="http://lucene.apache.org">Apache
                    Lucene</ulink>. It thus benefits from a stable, well-designed and widely-used
                framework. The module is tightly integrated with eXist-db's <emphasis>modularized
                    indexing architecture</emphasis>: the index behaves like a plugin which adds
                itself to the db's index pipelines. Once configured, the index will be notified of
                all relevant events, like adding/removing a document, removing a collection or
                updating single nodes. No manual reindex is required to keep the index up-to-date.
                The module also implements common interfaces which are shared with other indexes,
                e.g. for highlighting matches. It is thus easy to switch between the Lucene index
                and e.g. the ngram index without rewriting too much XQuery code.</para></section><section><title>Enabling the Lucene Module</title><para>The Lucene full text index is enabled by default since version 1.4 of eXist-db. However, in             
                case it is not enabled in your installation, here's how to get it up and
                running:</para><procedure><step><para>Before building eXist-db, enable the Lucene full text index by enabling it according to the instructions in the documentation on <ulink url="indexing.xml#moduleconf">index
                        modules</ulink>.</para></step><step><para>Then <emphasis>(re-)build eXist-db</emphasis> using the provided <filename>build.sh</filename> or <filename>build.bat</filename>. The build
                        process downloads the required Lucene jars automatically. If everything
                        builds OK, you should find a jar <filename>exist-lucene-module.jar</filename> in the <filename>lib/extensions</filename> directory. Next, edit the main configuration
                        file, <filename>conf.xml</filename> and un-comment the Lucene-related
                        section:</para><example><title>conf.xml</title><programlisting><markup>&lt;modules&gt;
    &lt;module id="lucene-index" class="org.exist.indexing.lucene.LuceneIndex" buffer="32"/&gt;
    ...
&lt;/modules&gt;
</markup></programlisting></example></step></procedure><section><title>Global configuration options</title><para>The index has a single configuration parameter which can be specified on the
                        <sgmltag>module</sgmltag> element within the <sgmltag>modules</sgmltag>
                    section:</para><variablelist><varlistentry><term>buffer</term><listitem><para>Defines the amount of memory (in megabytes) Lucene will use for
                                buffering index entries before they are written to disk. See the
                                    <ulink url="http://lucene.apache.org/core/3_6_2/api/all/org/apache/lucene/index/IndexWriterConfig.html#setRAMBufferSizeMB(double)">Lucene javadocs</ulink>.</para></listitem></varlistentry></variablelist></section></section><section><title>Configuring the Index</title><para>Like other indexes, you create a Lucene index by configuring it in a
                    <filename>collection.xconf</filename> document. If you have never done that
                before, read the corresponding <ulink url="indexing.xml#idxconf">documentation</ulink>. An example <filename>collection.xconf</filename> is
                shown below:</para><example><title>collection.xconf</title><programlisting><markup>&lt;collection xmlns="http://exist-db.org/collection-config/1.0"&gt;
    &lt;index xmlns:atom="http://www.w3.org/2005/Atom"
        xmlns:html="http://www.w3.org/1999/xhtml"
        xmlns:wiki="http://exist-db.org/xquery/wiki"&gt;
	    &lt;!-- Disable the old full text index --&gt;
        &lt;fulltext default="none" attributes="no"/&gt;
	    &lt;!-- Lucene index is configured below --&gt;
        &lt;lucene&gt;
	        &lt;analyzer class="org.apache.lucene.analysis.standard.StandardAnalyzer"/&gt;
            &lt;analyzer id="ws" class="org.apache.lucene.analysis.WhitespaceAnalyzer"/&gt;
	        &lt;text qname="TITLE" analyzer="ws"/&gt;
	        &lt;text qname="p"&gt;
	            &lt;inline qname="em"/&gt;
            &lt;/text&gt;
            &lt;text match="//foo/*"/&gt;
            &lt;!-- "inline" and "ignore" can be specified globally or per-index as
                 shown above --&gt;
	        &lt;inline qname="b"/&gt;
	        &lt;ignore qname="note"/&gt;
        &lt;/lucene&gt;
    &lt;/index&gt;
&lt;/collection&gt;
</markup></programlisting></example><para>You can either define a Lucene index on a single element or attribute name
                (qname="...") or a node path with wildcards (match="...", see below). It is
                important make sure to <emphasis>choose the right context</emphasis> for an index,
                which has to be the same as in your query. To better understand this, let's have a
                look at how the index creation is handled by eXist-db and Lucene. The following
                configuration:</para><synopsis language="xml">&lt;text qname="SPEECH"/&gt;</synopsis><para>creates an index ONLY on SPEECH. What is passed to Lucene is the string value of
                SPEECH, which includes the text of all its descendant text nodes (*except* those
                filtered out by an optional <sgmltag>ignore</sgmltag>). For example, consider the
                fragment:</para><programlisting><markup>&lt;SPEECH&gt;
    &lt;SPEAKER&gt;Second Witch&lt;/SPEAKER&gt;
    &lt;LINE&gt;Fillet of a fenny snake,&lt;/LINE&gt;
    &lt;LINE&gt;In the cauldron boil and bake;&lt;/LINE&gt;
&lt;/SPEECH&gt;</markup></programlisting><para>If you have an index on SPEECH, Lucene will create a "document" with the text
                "Second Witch Fillet of a fenny snake, In the cauldron boil and bake;" and index
                it. eXist-db internally links this Lucene document to the SPEECH node, but Lucene has
                no knowledge of that (it doesn't know anything about XML nodes).</para><para>The query:</para><synopsis language="xquery">//SPEECH[ft:query(., 'cauldron')]</synopsis><para>searches the index and finds the "document" containing the SPEECH text, which
                eXist-db can trace back to the SPEECH node in the XML document. However, it is required
                that you use the same context (SPEECH) for creating and querying the index. The
                query:</para><synopsis language="xquery">//SPEECH[ft:query(LINE, 'cauldron')]</synopsis><para>will not return anything, even though LINE is a child of SPEECH and 'cauldron' was
                indexed. This particular 'cauldron' is linked to its ancestor SPEECH node, not its
                parent LINE.</para><para>However, you are free to give the user both options, i.e. use SPEECH and LINE as
                context at the same time. How? Simply define a second index on LINE:</para><programlisting><markup>&lt;text qname="SPEECH"/&gt;
&lt;text qname="LINE"/&gt;</markup></programlisting><para>Let's use a different example to illustrate that. Assume you have a document with
                encoded place names:</para><example><title>Paragraph with place name</title><programlisting><markup>&lt;p&gt;He loves &lt;placeName&gt;Paris&lt;/placeName&gt;.&lt;/p&gt;</markup></programlisting></example><para>For a general query you probably want to search through all paragraphs. However,
                you may also want to provide an advanced search option, which allows the user to
                restrict his query to place names. To make this possible, simply define an index on
                placeName as well:</para><example><title>collection.xconf fragment</title><programlisting><markup>&lt;lucene&gt;
    &lt;text qname="p"/&gt;
    &lt;text qname="placeName"/&gt;
&lt;/lucene&gt;</markup></programlisting></example><para>Based on this setup, you'll be able to query for the word 'Paris' anywhere in a
                paragraph:</para><synopsis language="xquery">//p[ft:query(., 'paris')]</synopsis><para>as well as 'Paris' occurring within a <sgmltag>placeName</sgmltag>:</para><synopsis language="xquery">//p[ft:query(placeName, 'paris')]</synopsis><section><title>Using match="..."</title><para>In addition to defining an index on a given qname, you may also specify a
                    "path" with wildcards. <emphasis>This feature is subject to change</emphasis>,
                    so please be careful when using it.</para><para>Assume you want to define an index on all the possible elements below SPEECH.
                    You can do this by creating one index for every element:</para><programlisting><markup>&lt;text qname="LINE"/&gt;
&lt;text qname="SPEAKER"/&gt;</markup></programlisting><para>As a shortcut, you can use a <option>match</option> attribute with a
                    wildcard:</para><synopsis language="xml">&lt;text match="//SPEECH/*"/&gt;</synopsis><para>which will create a separate index on each child element of SPEECH it
                    encounters. Please note that the argument to match is a simple path pattern, not
                    an XPath expression. It only allows / and // to denote a child or descendant
                    step, plus the wildcard to match an arbitrary element.</para><para>As explained above, you have to figure out which parts of your document will
                    likely be interesting as context for a full text query. The full text index will
                    work best if the context isn't too narrow. For example, if you have a document
                    structure with section divs, headings and paragraphs, you would probably want to
                    create an index on the divs and maybe on the headings, so the user can
                    differentiate between the two. In some cases, you could decide to put the index on
                    the paragraph level, but then you don't need the index on the section since you can
                    always get from the paragraph back to the section.</para><para>If you query a larger context, you can use the <ulink url="kwic.xml">KWIC</ulink> module to show the user only a certain chunk of text
                        <emphasis>surrounding</emphasis> each match. Or you can ask eXist-db to <ulink url="kwic.xml#highlight">highlight each match</ulink> with an
                        <sgmltag>exist:match</sgmltag> tag, which you can later use to locate the
                    matches within the text.</para></section><section><title>Whitespace Treatment and Ignored Content</title><section><title>Inlined elements</title><para>By default, eXist-db's indexer assumes that element boundaries break a word
                        or token. For example, if you have an element:</para><example><title>Not a Mixed Content Element</title><programlisting><markup>&lt;size&gt;&lt;width&gt;12&lt;/width&gt;&lt;height&gt;8&lt;/height&gt;&lt;/size&gt;</markup></programlisting></example><para>You want "12" and "8" to be indexed as separate tokens, even though
                        there's no whitespace between the elements. By default, eXist-db will indeed
                        pass the content of the two elements to Lucene as separate strings and
                        Lucene will thus see two tokens instead of just "128".</para><para>However, you usually don't want this behaviour for mixed content nodes.
                        For example:</para><example><title>Mixed Content Node</title><programlisting><markup>&lt;p&gt;This is &lt;b&gt;un&lt;/b&gt;clear.&lt;/p&gt;</markup></programlisting></example><para>In this case, you want "unclear" to be indexed as one word. This can be
                        done by telling eXist-db which nodes are "inline" nodes. The example
                        configuration above defines:</para><synopsis>&lt;inline qname="b"/&gt;</synopsis></section><para>The <sgmltag>inline</sgmltag> option can be specified globally, which means it
                    will be applied to all <sgmltag>b</sgmltag> elements, or per-index:</para><synopsis language="xml">&lt;text qname="p"&gt;
    &lt;inline qname="em"/&gt;
&lt;/text&gt;</synopsis><section><title>Ignored elements</title><para>Also, it is sometimes necessary to skip the content of an inlined element,
                        which can appear in the middle of a text sequence you want to index. Notes
                        are a good example:</para><example><title>Paragraph With Inline Note</title><programlisting><markup>&lt;p&gt;This is a paragraph
&lt;note&gt;containing an inline note&lt;/note&gt;.&lt;/p&gt;</markup></programlisting></example><para>Use an <sgmltag>ignore</sgmltag> element in the collection configuration
                        to have eXist-db ignore the note:</para><synopsis language="xml">&lt;ignore qname="note"/&gt;</synopsis><para>Basically, <sgmltag>ignore</sgmltag> simply allows you to hide a chunk of
                        text before Lucene sees it.</para><para>Like the <sgmltag>inline</sgmltag> tag, <sgmltag>ignore</sgmltag> may
                        appear globally or within a single index definition.</para><para>The <sgmltag>ignore</sgmltag> only applies to descendants of an indexed
                        element. You can still create another index on the ignored element itself.
                        For example, you can have index definitions for <sgmltag>p</sgmltag> and
                            <sgmltag>note</sgmltag>:</para><example><title>collection.xconf fragment</title><programlisting><markup>&lt;lucene&gt;
    &lt;text qname="p"/&gt;
    &lt;text qname="note"/&gt;
    &lt;ignore qname="note"/&gt;
&lt;/lucene&gt;</markup></programlisting></example><para>If <sgmltag>note</sgmltag> appears within <sgmltag>p</sgmltag>, it will
                        not be added to the index on <sgmltag>p</sgmltag>, but only to the index on
                            <sgmltag>note</sgmltag>. This means that the query</para><synopsis language="xquery">//p[ft:query(., "note")]</synopsis><para>may not return a hit if "note" occurs within a <sgmltag>note</sgmltag>,
                        while</para><synopsis language="xquery">//p[ft:query(note, "note")] </synopsis><para>may still find a match.</para></section></section><section><title>Boost</title><para>A boost value can be assigned to an index to give it a higher score. The score
                    for each match will be multiplied by the boost factor (default is: 1.0). For
                    example, you may want to rank matches in titles higher than other matches.
                    Here's how we configure the documentation search indexes in eXist-db:</para><example><title>collection.xconf using boost</title><programlisting><markup>&lt;lucene&gt;
    &lt;analyzer class="org.apache.lucene.analysis.standard.StandardAnalyzer"/&gt;
    &lt;text qname="section"&gt;
        &lt;ignore qname="title"/&gt;
        &lt;ignore qname="programlisting"/&gt;
        &lt;ignore qname="screen"/&gt;
        &lt;ignore qname="synopsis"/&gt;
    &lt;/text&gt;
    &lt;text qname="para"/&gt;
    &lt;text qname="title" boost="2.0"/&gt;
    &lt;ignore qname="title"/&gt;
&lt;/lucene&gt;</markup></programlisting></example><para>The title index gets a boost of 2.0 to make sure that title matches get a
                    higher score. Since the <sgmltag>title</sgmltag> element does occur within
                        <sgmltag>section</sgmltag>, we add an ignore rule to the index definition on
                    the section and create a separate index on title. We also ignore titles occurring inside paragraphs. Without this, title would be
                    matched two times.</para><para>Because the title is now indexed separately, we also need to query it
                    explicitly. For example, to search the section and the title at the same time,
                    one could issue the following query:</para><synopsis language="xquery">for $sect in /book//section[ft:query(., "ngram")] | /book//section[ft:query(title, "ngram")]
order by ft:score($sect) descending return $sect</synopsis></section><section><title>Analyzers</title><para>One of the strengths of Lucene is that it allows the developer to determine
                    nearly every aspect of the text analysis. This is mostly done through analyzer
                    classes, which combine a tokenizer with a chain of filters to post-process the
                    tokenized text. eXist-db's Lucene module already allows different analyzers to
                    be used for different indexes.</para><example><title/><programlisting><markup>&lt;lucene&gt;
    &lt;analyzer class="org.apache.lucene.analysis.standard.StandardAnalyzer"/&gt;
    &lt;analyzer id="ws" class="org.apache.lucene.analysis.WhitespaceAnalyzer"/&gt;
    &lt;text match="//SPEECH//*"/&gt;
    &lt;text qname="TITLE" analyzer="ws"/&gt;
&lt;/lucene&gt;</markup></programlisting></example><para>In the example above, we define that Lucene's <ulink url="http://lucene.apache.org/core/3_6_2/api/all/org/apache/lucene/analysis/standard/StandardAnalyzer.html">StandardAnalyzer</ulink> should be used by default (the
                        <sgmltag>analyzer</sgmltag> element without <option>id</option> attribute).
                    We provide an additional analyzer and assign it the id <option>ws</option>, by
                    which the analyzer can be referenced in the actual index definitions.</para><para>The <ulink url="http://lucene.apache.org/core/3_6_2/api/all/org/apache/lucene/analysis/WhitespaceAnalyzer.html">whitespace analyzer</ulink> is the most basic one. As the name says, it
                    tokenizes the text at white space characters, but treats all other characters -
                    including punctuation - as part of the token. The tokens are not converted to
                    lower case and there's no stopword filter applied.</para><section><title>Configuring the Analyzer</title><para>We provide the capability to send configuration parameters to the instantiation of the Analyzer. These parameters must match a Constructor signature on the underlying Java class of the Analyzer, so we would first recommend that you review the Javadoc for the Analyzer that you wish to configure.</para><para>We currently support passing: String, File, Integer, Boolean and Set&lt;String&gt; value parameters to the Analyzer's constructor. If no type is specified in the parameter configuration, then String is assumed by default.</para><example><title>Providing a Stop Words file for the Standard Analyzer</title><programlisting><markup>
&lt;analyzer id="stdstops" class="org.apache.lucene.analysis.standard.StandardAnalyzer"&gt;
    &lt;param name="stopwords" type="java.io.File" value="/tmp/stop.txt"/&gt;
&lt;/analyzer&gt;
                            </markup></programlisting></example><example><title>Providing a list of Stop Words for the Standard Analyzer</title><programlisting><markup>
&lt;analyzer id="stdstops" class="org.apache.lucene.analysis.standard.StandardAnalyzer"&gt;
    &lt;param name="stopwords" type="java.util.Set"&gt;
        &lt;value&gt;the&lt;/value&gt;
        &lt;value&gt;this&lt;/value&gt;
        &lt;value&gt;and&lt;/value&gt;
        &lt;value&gt;that&lt;/value&gt;
    &lt;/param&gt;
&lt;/analyzer&gt;
                            </markup></programlisting></example><example><title>Using the Snowball Analyzer</title><para>Note that using the Snowball analyzer requires you to add additional libraries to lib/user.</para><programlisting><markup>
&lt;analyzer id="sbstops" class="org.apache.lucene.analysis.snowball.SnowballAnalyzer"&gt;
    &lt;param name="name" value="English"/&gt;
    &lt;param name="stopwords" type="java.util.Set"&gt;
        &lt;value&gt;the&lt;/value&gt;
        &lt;value&gt;this&lt;/value&gt;
        &lt;value&gt;and&lt;/value&gt;
        &lt;value&gt;that&lt;/value&gt;
    &lt;/param&gt;
&lt;/analyzer&gt;
                            </markup></programlisting></example></section><para>We will certainly add more features in the future, e.g. a possibility to
                    construct a new analyzer from a set of filters. For the time being, you can
                    always provide your own analyzer or use one of those supplied by Lucene or
                    compatible software.</para></section><section><title>Defining Fields</title><para>Sometimes you may want to define different Lucene indexes on the same set of elements, e.g.
                to use a different analyzer. eXist-db allows to name a certain index using the <option>field</option>
                attribute:</para><synopsis language="xquery">&lt;text field="title" qname="title" analyzer="en"/&gt;</synopsis></section></section><section><title>Querying the Index</title><para>Querying lucene from XQuery is straightforward. For example:</para><example><title>A Simple Query</title><programlisting language="xquery">for $m in //SPEECH[ft:query(., "boil bubble")]
order by ft:score($m) descending
return $m</programlisting></example><para>The query function takes a query string in Lucene's default <ulink url="http://lucene.apache.org/core/3_6_0/queryparsersyntax.html">query
                    syntax</ulink>. It returns a set of nodes which are relevant with respect to the
                query. Lucene assigns a relevance score or rank to each match. This score is
                preserved by eXist-db and can be accessed through the score function, which returns a
                decimal value. The higher the score, the more relevant is the text. You can use
                Lucene's features to "boost" a certain term in the query, i.e. give it a higher or
                lower influence on the final rank.</para><para>Please note that the score is computed relative to the root context of the index.
                If you created an index on SPEECH, all scores will be computed on basis of the text
                in the SPEECH nodes, even though your actual query may only return LINE children of
                SPEECH.</para><para>The Lucene module is fully supported by eXist-db's query-rewriting optimizer, which
                means that the query engine can rewrite the XQuery expression to make best use of
                the available indexes. All the rules and hints given in the <ulink url="tuning.xml">tuning</ulink> guide fully apply to the Lucene index.</para><para>To present search results in a <emphasis>Keywords in Context</emphasis> format,
                you may want to have a look at eXist-db's <ulink url="kwic.xml">KWIC</ulink>
                module.</para><section><title>Query a Named Index</title><para>To query a named index, use the <function>ft:query-field($fieldName, $query)</function> instead of
                    <function>ft:query</function>:</para><synopsis language="xquery">ft:query-field("title", "xml")</synopsis><para><function>ft:query-field</function> works exactly like <function>ft:query</function>, except that the set of nodes
                    to search is determined by the nodes in the named index. The function returns the nodes selected by 
                    the query, which would be <sgmltag>title</sgmltag> elements in the example above.</para><para>You can thus use <function>ft:query-field</function> with an XPath filter expression just as you would call
                    <function>ft:query</function>:</para><synopsis language="xquery">//section[ft:query-field("title", "xml")]</synopsis></section><section><title>Describing Queries in XML</title><para>Lucene's default query syntax does not provide access to all available
                    features. However, eXist-db's <function>ft:query</function> function also accepts a
                    description of the query in XML as an alternative to passing a query string. The
                    XML description closely mirrors Lucene's query API. It is transformed into an
                    internal tree of query objects, which is directly passed to Lucene for
                    execution. This has some advantages. For example, you can specify if the order
                    of terms should be relevant for a phrase query:</para><example><title>Using an XML Definition of the Query</title><programlisting language="xquery">let $query :=
    &lt;query&gt;
        &lt;near ordered="no"&gt;miserable nation&lt;/near&gt;
    &lt;/query&gt;
return
    //SPEECH[ft:query(., $query)]</programlisting></example><para>The following elements may occur within a query description:</para><variablelist><varlistentry><term><sgmltag>term</sgmltag></term><listitem><para>Defines a single term to be searched in the index. If the root
                                query element contains a sequence of term elements, they will be
                                combined as in a boolean "or" query. For example: </para><synopsis language="xquery">let $query :=
    &lt;query&gt;
        &lt;term&gt;nation&lt;/term&gt;&lt;term&gt;miserable&lt;/term&gt;
    &lt;/query&gt;
return
//SPEECH[ft:query(., $query)]</synopsis><para>finds all SPEECH elements containing either "nation" or
                                "miserable" or both.</para></listitem></varlistentry><varlistentry><term><sgmltag>wildcard</sgmltag></term><listitem><para>A string with a '*' wildcard in it, which will be matched against
                                the terms of a document. Can be used instead of a
                                    <sgmltag>term</sgmltag> element. For example:</para><synopsis language="xquery">let $query :=
    &lt;query&gt;
        &lt;term&gt;nation&lt;/term&gt;&lt;wildcard&gt;miser*&lt;/wildcard&gt;
    &lt;/query&gt;
return
//SPEECH[ft:query(., $query)]</synopsis></listitem></varlistentry><varlistentry><term><sgmltag>regex</sgmltag></term><listitem><para>A regular expression which will be matched against the terms of a
                                document. Can be used instead of a <sgmltag>term</sgmltag> element.
                                For example:</para><synopsis language="xquery">let $query :=
    &lt;query&gt;
        &lt;term&gt;nation&lt;/term&gt;&lt;regex&gt;miser.*&lt;/regex&gt;
    &lt;/query&gt;
return
//SPEECH[ft:query(., $query)]</synopsis></listitem></varlistentry><varlistentry><term><sgmltag>bool</sgmltag></term><listitem><para>Constructs a boolean query from its children. Each child element
                                may have an occurrence indicator, which could be either
                                    <option>must</option>, <option>should</option> or
                                    <option>not</option>:</para><variablelist><varlistentry><term>must</term><listitem><para>this part of the query <emphasis>must</emphasis> be
                                            matched</para></listitem></varlistentry><varlistentry><term>should</term><listitem><para>this part of the query <emphasis>should</emphasis> be
                                            matched, but doesn't need to</para></listitem></varlistentry><varlistentry><term>not</term><listitem><para>this part of the query <emphasis>must not</emphasis>
                                            be matched</para></listitem></varlistentry></variablelist><synopsis language="xquery">let $query :=
    &lt;query&gt;
        &lt;bool&gt;&lt;term occur="must"&gt;boil&lt;/term&gt;&lt;term occur="should"&gt;bubble&lt;/term&gt;&lt;/bool&gt;
    &lt;/query&gt;
return //SPEECH[ft:query(LINE, $query)]</synopsis></listitem></varlistentry><varlistentry><term><sgmltag>phrase</sgmltag></term><listitem><para>Searches for a group of terms occurring in the correct order. The
                                element may either contain explicit <sgmltag>term</sgmltag> elements
                                or text content. Text will be automatically tokenized into a
                                sequence of terms. For example:</para><synopsis language="xquery">let $query :=
    &lt;query&gt;
        &lt;phrase&gt;cauldron boil&lt;/phrase&gt;
    &lt;/query&gt;
return //SPEECH[ft:query(., $query)]
</synopsis><para>has the same effect as:</para><synopsis language="xquery">let $query :=
    &lt;query&gt;
        &lt;phrase&gt;&lt;term&gt;cauldron&lt;/term&gt;&lt;term&gt;boil&lt;/term&gt;&lt;/phrase&gt;
    &lt;/query&gt;
return //SPEECH[ft:query(., $query)]
</synopsis><para>The attribute <parameter>slop</parameter> can be used for a proximity
                                search: Lucene will try to find terms which are within the specified
                                distance:</para><synopsis language="xquery">let $query :=
    &lt;query&gt;
        &lt;phrase slop="10"&gt;&lt;term&gt;frog&lt;/term&gt;&lt;term&gt;dog&lt;/term&gt;&lt;/phrase&gt;
    &lt;/query&gt;
return //SPEECH[ft:query(., $query)]
</synopsis></listitem></varlistentry><varlistentry><term><sgmltag>near</sgmltag></term><listitem><para><sgmltag>near</sgmltag> is a powerful alternative to
                                    <sgmltag>phrase</sgmltag> and one of the features not available
                                through the standard Lucene query parser.</para><para>If the element has text content only, it will be tokenized into
                                terms and the expression behaves like <sgmltag>phrase</sgmltag>.
                                Otherwise it may contain any combination of <sgmltag>term</sgmltag>,
                                    <sgmltag>first</sgmltag> and nested <sgmltag>near</sgmltag>
                                elements. This makes it possible to search for two sequences of
                                terms which are within a specific distance. For example:</para><synopsis language="xquery">let $query :=
    &lt;query&gt;
        &lt;near slop="20"&gt;&lt;term&gt;snake&lt;/term&gt;&lt;near&gt;tongue dog&lt;/near&gt;&lt;/near&gt;
    &lt;/query&gt;
return //SPEECH[ft:query(., $query)]</synopsis><para>Element <sgmltag>first</sgmltag> matches a span against the start
                                of the text in the context node. It takes an optional attribute
                                    <parameter>end</parameter> to specify the maximum distance from
                                the start of the text. For example:</para><synopsis language="xquery">let $query :=
    &lt;query&gt;
        &lt;near slop="50"&gt;&lt;first end="2"&gt;&lt;near&gt;second witch&lt;/near&gt;&lt;/first&gt;&lt;near&gt;tongue dog&lt;/near&gt;&lt;/near&gt;
    &lt;/query&gt;
    return //SPEECH[ft:query(., $query)]</synopsis><para>As shown above, the content of <sgmltag>first</sgmltag> can again
                                be text, a <sgmltag>term</sgmltag> or
                                <sgmltag>near</sgmltag>.</para><para>Contrary to <sgmltag>phrase</sgmltag>, <sgmltag>near</sgmltag> can
                                be told to ignore the order of its components. Use parameter
                                    <parameter>ordered="yes|no"</parameter> to change near's
                                behaviour. For example:</para><synopsis language="xquery">let $query :=
    &lt;query&gt;
        &lt;near slop="100" ordered="no"&gt;&lt;term&gt;snake&lt;/term&gt;&lt;term&gt;bake&lt;/term&gt;&lt;/near&gt;
    &lt;/query&gt;
return //SPEECH[ft:query(., $query)]</synopsis></listitem></varlistentry></variablelist><para>All elements in a query may have an optional <parameter>boost</parameter>
                    parameter (a float value). The score of the nodes matching the corresponding
                    query part will be multiplied by the <emphasis>boost</emphasis>.</para></section><section><title>Additional parameters</title><para>The ft:query function allows a third parameter, which can be used to pass
                    some additional settings to the query engine. The parameter should contain an XML 
                    fragment which lists the configuration properties to be set as child elements:</para><example><title/><programlisting language="xquery">
let $options :=
    &lt;options&gt;
        &lt;default-operator&gt;and&lt;/default-operator&gt;
        &lt;phrase-slop&gt;1&lt;/phrase-slop&gt;
        &lt;leading-wildcard&gt;no&lt;/leading-wildcard&gt;
        &lt;filter-rewrite&gt;yes&lt;/filter-rewrite&gt;
    &lt;/options&gt;
return
    //SPEECH[ft:query(., $query, $options)]</programlisting></example><para>The meaning of those properties is as follows</para><variablelist><varlistentry><term>filter-rewrite</term><listitem><para>Controls how terms are expanded for wildcard or regular expression
                            searches. If set to "yes", Lucene will use a filter to pre-process
                            matching terms. If set to "no", all matching terms will be added to a single
                            boolean query which is then executed. This may generate a "too many clauses"
                            exception when applied to large data sets. Setting filter-rewrite to "yes"
                            avoids those issues.</para></listitem></varlistentry><varlistentry><term>default-operator</term><listitem><para>The default operator with which multiple terms will be combined.
                            Allowed values: "or", "and".</para></listitem></varlistentry><varlistentry><term>phrase-slop</term><listitem><para>Sets the default slop for phrases. If zero, then exact phrase matches are 
                                required. Default value is zero.</para></listitem></varlistentry><varlistentry><term>leading-wildcard</term><listitem><para>When set to "yes", * or ? are allowed as the first character of a PrefixQuery and 
                                WildcardQuery. Note that this can produce very slow queries on big indexes.</para></listitem></varlistentry></variablelist></section></section><section><title>Adding Constructed Fields to a Document</title><para>This feature allows to add arbitrary fields to a binary or XML document and have them indexed
                with lucene. It was developed as part of the <ulink url="contentextraction.xml">content extraction 
                framework</ulink> to attach metadata extracted from e.g. a PDF to the binary document. It works
                equally well for XML documents though and is an efficient method, e.g. to attach computed fields to
                a document, containing information which does not exist in the XML as such.</para><para>The field indexes are not configured via <filename>collection.xconf</filename>. Instead we add
            fields programmatically from an XQuery (which could be run via a trigger):</para><programlisting language="xquery">ft:index("/db/demo/test.xml", &lt;doc&gt;
    &lt;field name="title" store="yes"&gt;Indexing&lt;/field&gt;
    &lt;field name="author" store="yes"&gt;Me&lt;/field&gt;
    &lt;field name="date" store="yes"&gt;2013&lt;/field&gt;
&lt;/doc&gt;)</programlisting><para>The <option>store</option> attribute indicates that the fields content should be stored as a string. 
                Without this attribute, the content will be indexed for search, but you won't be able to retrieve the contents.
            </para><para>To get the contents of a field, use the <function>ft:get-field</function> function:</para><synopsis language="xquery">ft:get-field("/db/demo/test.xml", "title")</synopsis><para>To query this index, use the <function>ft:search</function> function:</para><synopsis language="xquery">ft:search("/db/demo/test.xml", "title:indexing and author:me")</synopsis><para>Custom field indexes are automatically deleted when their parent document is removed. If you want to update
                fields without removing the document, you need to delete the old fields first though. This can be done
                using the <function>ft:remove-index</function> function:</para><synopsis language="xquery">ft:remove-index("/db/demo/test.xml")</synopsis></section></chapter></book>