<?xml version="1.0" encoding="UTF-8"?>
<!-- $Id: ftlegacy.xml 17299 2012-10-10 23:47:33Z joewiz $ --><book><bookinfo><productname>Open Source Native XML Database</productname><title>Legacy Full Text Index</title><date>September 2009</date><orgname>The eXist Project</orgname></bookinfo><chapter><title>Legacy Full Text Index</title><para>This index is used to query for a sequence of separate "words" or tokens in a longer
            stream of text. While building the index, the text is parsed into single tokens which
            are then stored in the index.</para><important><para>Historically, eXist has been creating a default full text index on all text nodes
                and attribute values. This is no longer the case! Creating default indexes on data
                which you may never use is too expensive, in particular, since eXist is now
                providing a wider range of different indexes which can be much better optimized than
                the default index and will offer superior performance.</para></important><para>Anyway, as for the other index types, you can configure the full text index in the
            collection configuration and we will try to keep the configuration of the new index
            backwards compatible. We thus recommend to create a collection configuration file,
            disable the default index-all behaviour and define some explicit full text indexes on
            your documents. The details of this process will be described below.</para><para>The full text index is only used in combination with eXist's fulltext search
            extensions. In particular, you can use the following eXist-specific operators and
            functions that apply a fulltext index:</para><itemizedlist><listitem><para>Operators: <command>&amp;=</command> and <command>|=</command></para></listitem><listitem><para>Main Functions: <function>text:match-all()</function>,
                        <function>text:match-any()</function> and <function>near()</function></para></listitem></itemizedlist><note><para>It is important to note that, if you have disabled full text indexing for certain
                elements, these operators and functions will also be effectively disabled, and will
                not return matches. As a result, eXist will not return results for queries that
                normally would have results provided fulltext indexing was enabled. Note also that
                this is in direct contrast to the operation of range indexing, which does fallback
                to full searching of the document if no range index applies (see below).</para></note><section id="ftidxconf"><title>Full text index configuration</title><para>The <emphasis>fulltext index</emphasis> is defined by the
                    <sgmltag>fulltext</sgmltag> element - along with <sgmltag>include</sgmltag>,
                    <sgmltag>exclude</sgmltag> and <sgmltag>create</sgmltag> elements. The full text
                index is currently subject to a major redesign and the configuration syntax somewhat
                reflects this because it is a bit inconsistent. eXist's current default behaviour is
                to create a full text index on all text nodes in a document. The
                    <sgmltag>include</sgmltag> and <sgmltag>exclude</sgmltag> tags are used to add
                or hide nodes from the default indexing. The indexes created by the default indexing
                are always context-dependant. The nodes to include/exclude are thus specified via a
                    <option>path</option> attribute, not a <option>qname</option> attribute. </para><para>However, you can create explicit indexes on a qname using the
                    <sgmltag>create</sgmltag> element. This is the recommended approach. In fact, as
                the full text index is currently being redesigned, we are not sure if we will keep
                the current default full text indexing in its current state. A fulltext
                configuration which only uses <sgmltag>create</sgmltag> elements is shown below: </para><example><title>Full text index configuration</title><programlisting><markup>
&lt;fulltext default="none" attributes="false"&gt;
      &lt;!-- Full text indexes --&gt;
      &lt;create qname="author"/&gt;
      &lt;create qname="title" content="mixed"/&gt;
&lt;/fulltext&gt;
</markup></programlisting></example><para>With this example, the full text default attribute is set to "none", which
                disables the default full text indexing for all document elements. Attribute nodes
                are handled separately. Setting <option>attributes="false"</option> disables the
                default indexing for attributes as well. </para><para>The first child element creates a standard full text index on all
                    <sgmltag>author</sgmltag> elements, identified by their
                    <emphasis>qname</emphasis>. The second one puts an index on
                    <sgmltag>title</sgmltag>, but adds an option <option>content="mixed"</option>.
                This parameter causes the indexer to watch out for mixed-content nodes. For example,
                if your source XML contains markup like:</para><example><title>Mixed Content Element</title><programlisting><markup>&lt;p&gt;Some &lt;span&gt;un&lt;/span&gt;&lt;span&gt;even&lt;/span&gt; amount.&lt;/p&gt;</markup></programlisting></example><para>You may want to treat "uneven" as a single word so you can query for p |=
                "uneven". In this case, you need to pass content="mixed" to the indexer. The
                concatenated text nodes of element mixed and all its descendants will be passed to
                the indexer as one single string. The indexer thus sees and indexes "uneven" as a
                single token.</para><para>On the other hand, if you have</para><example><title>Not a Mixed Content Element</title><programlisting><markup>&lt;date&gt;&lt;year&gt;1183&lt;/year&gt;&lt;month&gt;March&lt;/month&gt;&lt;date&gt;</markup></programlisting></example><para>you probably want to be able to query for "March" even though there's no space
                between the year and month elements. In this case the standard settings are ok as
                they will add a virtual break between the elements.</para></section><section id="ftidx"><title>Querying Text (Fulltext Searching)</title><para>The standard XPath/XQuery function library contains most of the common string
                manipulation functions provided by most programming languages. However, these
                functions are insufficient for conducting keyword or phrase searches inside a
                larger portion of text or mixed content. This is a weak point if you have to
                work with <emphasis>document-centric</emphasis> (i.e. mainly free-form text), as
                opposed to <emphasis>data-centric</emphasis> documents. For many types of
                documents, the standard string functions do not yield satisfactory search
                results.</para><para>For example, suppose upon reading a chapter in an electronic text, you
                encountered something about "XML" and "databases", but later you could not
                recall the exact section where you read it. Using standard XPath, you could try
                a query like:</para><synopsis format="linespecific">//chapter[contains(., 'XML') and contains(., 'databases')]</synopsis><para>This query execution will likely be quite slow, since the XPath engine will,
                in this case, scan the entire character content of all chapter nodes and their
                descendants. And yet, there is no certainty that all possible text matches will
                be found - for example, "databases" might have been written with a capital
                letter at the start of the sentence, and so would not be included in the
                results.</para><para>To resolve this issue, eXist-db offers additional operators and extension
                functions for efficient, index-based access to the full text content of nodes.
                With eXist-db, you could alternatively formulate the above query as follows:</para><synopsis format="linespecific">//chapter[near(., 'XML database?', 50)]</synopsis><para>This will return all chapters containing both keywords in the correct order,
                and as well, will find matches that have under 50 words between them.
                Additionally, the wildcard character <command moreinfo="none">?</command> in
                <command moreinfo="none">database?</command> will match the singular as well
                as the plural instances of "database", and the search would NOT be
                case-sensitive. Furthermore, since the query is index-based, it will usually be
                an order of magnitude faster than the standard XPath query above.</para><section><title>Operators</title><para>In this section, we discuss each of eXist-db's text-search extensions. In
                    cases where the order and distance of search terms is not important, eXist-db
                    offers two additional operators for simple keyword queries: <command moreinfo="none">&amp;=</command> and <command moreinfo="none">|=</command>.</para><variablelist><varlistentry><term>node-set &amp;= 'string of keywords'</term><listitem><para>This operator selects context nodes containing
                                <emphasis>ALL</emphasis> of the keywords in the right-hand
                                argument in any order. The default tokenizer is used to split
                                the right-hand argument into single tokens, i.e. any punctuation
                                or white spaces are used to separate the keywords and, after
                                which, are omitted. Note also that wildcards are allowed, and
                                keyword comparison is NOT case-sensitive.</para></listitem></varlistentry><varlistentry><term>node-set |= 'string of keywords'</term><listitem><para>Similar to above, this operator selects context nodes
                                containing <emphasis>ANY</emphasis> of the keywords in the
                                right-hand argument.</para></listitem></varlistentry></variablelist><note><para>With the <command moreinfo="none">&amp;=</command> and <command moreinfo="none">|=</command>operations, keyword search strings are
                        split into tokens using the default tokenizer function. The current
                        implementation of this operation will work well for all European
                        languages. For non-European languages, however, eXist-db uses the
                        predefined Unicode code points (0 to 10FFFF) to determine where the
                        string will be split.</para></note><para>Both of the above operators accept simple wildcards in the keyword string.
                    A <command moreinfo="none">?</command> matches zero or one character,
                    <command moreinfo="none">*</command> matches zero or more characters. A
                    character range <command moreinfo="none">[abc]</command> (as a regular
                    expression) matches any of the characters in that range. You may use a
                    backslash to escape wildcard characters.</para><para>To match more complex patterns, full regular expression syntax is
                    supported through additional functions, which are discussed below.</para><note><para>There is an important semantic difference between the following two
                        expressions:</para><synopsis format="linespecific">//SPEECH[LINE &amp;= "cursed spite"]</synopsis><para>and</para><synopsis format="linespecific">//SPEECH[LINE &amp;= "cursed" and LINE &amp;="spite"]</synopsis><para>The first expression selects all distinct <option>LINE</option> nodes
                        that contain both of the search terms. The second expression selects all
                        context nodes (<option>SPEECH</option> nodes) that have
                        <option>LINE</option> children containing either or both of the
                        terms, and should yield more results than the first one. To make the
                        first expression select the same nodes (at least, nearly the same
                        nodes), you would have to change the first expression to:</para><synopsis format="linespecific">//SPEECH[. &amp;= "cursed spite"]</synopsis><para>Note, however, that this new expression will also include other nodes,
                        for instance <option>SPEAKER</option> or <option>STAGEDIR</option>,
                        which are children of the <option>SPEECH</option> parent node.</para></note></section><section><title>Functions utilising the legacy full text index</title><variablelist><varlistentry><term>near()</term><listitem><para>As shown in a previous example, the <command moreinfo="none">near()</command> function behaves quite similarly to the
                                <command moreinfo="none">&amp;=</command> operator, but also
                                pays attention to the order of search terms and their distance
                                from each other in the source document.</para><para>The syntax for this function is as follows:</para><synopsis format="linespecific">near(node-list, 'string of keywords' [, max-distance])</synopsis><para>The function measures the distance between two search terms by
                                counting the number of words between them. A maximum distance of
                                1 is assumed by default, in which case the search terms occur
                                next to each other. Other values for the maximal and minimal
                                distance may be specified in the optional third argument. As a
                                special case, if the string in the second argument contains only
                                one token, any distance values in the third and fourth argument
                                are ignored, and the function performs identically to the &amp;=
                                operator. For example, with the following search
                                expression:</para><synopsis format="linespecific">//SPEECH[near(., 'love marriage', 25)</synopsis><para>the search engine will return any <option>SPEECH</option>
                                elements containing the words "love" and "marriage" within the
                                range of 25 words between them.</para><para>Similar to the <command moreinfo="none">&amp;=</command>
                                operator, <command moreinfo="none">near()</command> accepts
                                wildcards in the keyword string, and punctuation and whitespace
                                will be skipped according to the default tokenization
                                rules.</para></listitem></varlistentry><varlistentry><term>text:match-all() / text:match-any()</term><listitem><para>These two functions are variations of the <command moreinfo="none">&amp;=</command> and <command moreinfo="none">|=</command> operators, and interpret their
                                arguments as regular expressions. <emphasis>However</emphasis>,
                                contrary to the <command>matches()</command> function in the
                                XQuery core library, <command>text:match-all()</command> and
                                <command>text:match-any()</command> try to match the regular
                                expression argument against the keywords contained in the
                                full text index, but <emphasis>NOT</emphasis> against the entire
                                text.</para><para>For example, assume you have a document that contains the
                                following paragraph:</para><synopsis>&lt;para&gt;Peter lives in Frankfurt&lt;/para&gt;</synopsis><para>Then the following expression:</para><synopsis>text:match-all(para, "li[vf]e.?", "frank.*")</synopsis><para>will match this paragraph because it contains two keywords
                                matching the specified regular expression patterns.</para><para><command moreinfo="none">text:match-all()</command> corresponds
                                to <command moreinfo="none">&amp;=</command> in that it will
                                select context nodes with keywords matching
                                <emphasis>ALL</emphasis> of the specified regular
                                expressions. <command moreinfo="none">text:match-any()</command>
                                will select nodes with keywords matching
                                <emphasis>ANY</emphasis> of the specified regular
                                expression.</para><para>Since tokenization doesn't work correctly with regular
                                expression patterns, each keyword has to be specified as a
                                separate argument, so the syntax looks like:</para><synopsis format="linespecific">text:match-all(node-set, 'regexp' [, 'regexp' ...])</synopsis><note><para>Please note that the <command moreinfo="none">text:match-any()</command> functions will try to match
                                    the regular expression against the entire keyword. For
                                    example, the expression</para><synopsis format="linespecific">//SPEECH[text:match-all(LINE, 'li[vf]e')]</synopsis><para>will match 'live', 'life', but not 'lives'.</para><para>eXist-db uses the <ulink url="http://java.sun.com/j2se/1.4.2/docs/api/java/util/regex/package-summary.html">java.util.regex API</ulink> for regular expressions. A
                                    description of the supported regexp syntax can be found on
                                    the <ulink url="http://java.sun.com/docs/books/tutorial/extra/regex/">Sun Java Tutorial</ulink>.</para></note></listitem></varlistentry></variablelist></section></section></chapter></book>