<?xml version="1.0" encoding="UTF-8"?>
<!-- $Id: kwic.xml 17855 2012-12-15 19:18:29Z jenspetersen $ --><book><bookinfo><productname>eXist-db – Open Source Native XML Database</productname><title>KWIC: Keywords in Context</title><date>September 2009</date><orgname>The eXist Project</orgname></bookinfo><chapter><title>Generating KWIC (Keywords in Context) Output</title><abstract><para>A KWIC display helps users to quickly scan through search results by
                listing hits surrounded by their context. eXist provides a
                KWIC module that is not bound to a specific index or query
                operation, but can be applied to query results from all indexes that support match
                highlighting. This includes the Lucene-based index, the ngram index, as
                well as the old full text index.</para><para>The documentation search function on eXist's home page is a good example. It
                queries documents written in the DocBook format. However, the KWIC module has also
                been successfully used and deployed with different schemas (e.g. TEI) and languages
                (e.g. Chinese).</para></abstract><section><title>Preparing your Query</title><para>The KWIC module is entirely written in XQuery. <!--It requires eXist-db 1.3/1.4 or above and won't
                work with eXist 1.2.x (the old KWIC functions in eXist 1.2.x are now deprecated).--> To
                use the module, simply import its namespace into your query:</para><synopsis language="xquery">import module namespace kwic="http://exist-db.org/xquery/kwic";</synopsis><para>You don't need to specify a location since the module is already registered in
                    <filename>conf.xml</filename>. If you would still like to provide one, change
                the import as follows:</para><synopsis language="xquery">import module namespace kwic="http://exist-db.org/xquery/kwic"
at "resource:org/exist/xquery/lib/kwic.xql";</synopsis><para>The module is part of the main <filename>exist.jar</filename>, so we can use a
                resource link here.</para></section><section><title>Using the Module</title><para>The easiest way to get KWIC output is to call the
                    <function>kwic:summarize</function> function on an element node returned from a
                full text or ngram query:</para><programlisting language="xquery">import module namespace kwic="http://exist-db.org/xquery/kwic";
for $hit in doc("/db/shakespeare/plays/hamlet.xml")//SPEECH[ft:query(., "'nature'")] 
order by ft:score($hit) descending
return
    kwic:summarize($hit, &lt;config width="40"/&gt;)</programlisting><para>Every call to <function>kwic:summarize</function> will return an HTML paragraph
                containing 3 spans with the text before and after each match as well as the match
                text itself:</para><programlisting language="xml">&lt;p&gt;
    &lt;span class="previous"&gt;... s effect, sir; after what flourish your &lt;/span&gt;
    &lt;span class="hi"&gt;nature&lt;/span&gt;
    &lt;span class="following"&gt; will.&lt;/span&gt;
&lt;/p&gt;</programlisting><para>The <sgmltag>config</sgmltag> element, passed to
                    <function>kwic:summarize</function> as second parameter, determines the
                appearance of the generated HTML. There are 3 different attributes you can set
                here:</para><variablelist><varlistentry><term>width</term><listitem><para>The maximum number of characters to be printed before and after the
                            match</para></listitem></varlistentry><varlistentry><term>table</term><listitem><para>if set to "yes", <function>kwic:summarize</function> will return an
                            HTML table row (<sgmltag>tr</sgmltag>). The text chunks will be enclosed
                            in a table column (<sgmltag>td</sgmltag>).</para><para>The default behaviour, <option>table="no"</option>, is to return an
                            HTML paragraph with spans.</para></listitem></varlistentry><varlistentry><term>link</term><listitem><para>If present, each match will be enclosed within a link, using the URI
                            in the link attribute as target. <!--This feature should be documented--></para></listitem></varlistentry></variablelist><para>If you look at the output of above query, you may notice that a space is missing
                between words if the previous or following chunk extends to a different
                    <sgmltag>LINE</sgmltag> element. Also, it would be nicer to only display text
                from <sgmltag>LINE</sgmltag> elements and to ignore <sgmltag>SPEAKER</sgmltag> or
                    <sgmltag>STAGEDIR</sgmltag> tags. This can be achieved with the help of a
                callback function:</para><programlisting language="xquery">import module namespace kwic="http://exist-db.org/xquery/kwic";
                
declare function local:filter($node as node(), $mode as xs:string) as xs:string? {
  if ($node/parent::SPEAKER or $node/parent::STAGEDIR) then 
      ()
  else if ($mode eq 'before') then 
      concat($node, ' ')
  else 
      concat(' ', $node)
};

for $hit in doc("/db/shakespeare/plays/hamlet.xml")//SPEECH[ft:query(., "'nature'")] 
order by ft:score($hit) descending
return
kwic:summarize($hit, &lt;config width="40"/&gt;, util:function(xs:QName("local:filter"), 2))</programlisting><para>The third parameter to <function>kwic:summarize</function> should be a reference
                to a function <!--(passing references to functions is a non-standard feature of eXist)-->
                accepting 2 arguments: 1) a single text node which should be appended or prepended
                to the current text chunk, 2) a string indicating the current direction in which
                text is appended, i.e. "before" or "after". The function may return the empty
                sequence if the current node should be ignored (e.g. if it belongs to a "footnote"
                which should not be displayed). Otherwise it should return a single string.</para><para>The <function>local:filter</function> function above first checks if the passed
                node has a SPEAKER or STAGEDIR parent and if yes, <emphasis>ignores</emphasis> that
                node by returning the empty sequence. If not, the function adds a single whitespace
                before or after the string, so adjacent lines will be properly separated.</para></section><section><title>Advanced Use</title><para>Using <function>kwic:summarize</function>, you will get one KWIC-formatted item
                for every match, even if the matches are in the same paragraph. Also, the context
                from which the text is taken is always the same: the element you queried.</para><para>To get more control over the output, you can directly call
                    <function>kwic:get-summary</function>, which is the module's core function. It
                expects 3 or 4 parameters, where the first two parameters are: a) the current
                context root, b) the match object to process. Parameters 3 and 4 are the same as for
                    <function>kwic:summarize</function>.</para><para>Before passing nodes to <function>kwic:get-summary</function> you have to
                    <emphasis>expand</emphasis> them, which basically means to create an in-memory
                copy in which all matches are properly marked up with <sgmltag>exist:match</sgmltag>
                tags. The main part of the query should look as follows:</para><programlisting language="xquery">for $hit in doc("/db/shakespeare/plays/hamlet.xml")//SPEECH[ft:query(., "'nature'")]
let $expanded := kwic:expand($hit)
order by ft:score($hit) descending
return
    kwic:get-summary($expanded, ($expanded//exist:match)[1], &lt;config width="40"/&gt;,
        util:function(xs:QName("local:filter"), 2))</programlisting><para>In this example, we select the first <sgmltag>exist:match</sgmltag> only, thus
                ignoring all other matches within <option>$expanded</option>.</para><para>Sometimes you may also want to change the context to restrict the KWIC display to
                certain elements within the larger query context, e.g. paragraphs within sections.
                The following example still queries <sgmltag>SPEECH</sgmltag>, but displays a KWIC
                entry for each <sgmltag>LINE</sgmltag> with a match, grouped by speech:</para><programlisting language="xquery">for $hit in doc("/db/shakespeare/plays/hamlet.xml")//SPEECH[ft:query(., "nature")]
let $expanded := kwic:expand($hit)
order by ft:score($hit) descending
return
    &lt;div class="speech"&gt;{
        for $line in $expanded//LINE[.//exist:match]
        return
            kwic:get-summary($line, ($line/exist:match)[1], &lt;config width="40"/&gt;,
                util:function(xs:QName("local:filter"), 2))
    }&lt;/div&gt;</programlisting><para>You may ask why we are not querying <sgmltag>LINE</sgmltag> directly to get a
                different context, e.g. as in:</para><synopsis language="xquery">//SPEECH[ft:query(LINE, "nature")]</synopsis><para>Well, we want Lucene to compute the relevance of each match with respect to the
                SPEECH context, not LINE. If we queried LINE, each single line would get a match
                score and the matches would end up in a completely different order.</para></section><section id="highlight"><title>Marking up Matches without using KWIC</title><para>Sometimes you don't want to use the KWIC module, but you would still like to have indicated
                where matches were found in the text. eXist's XML serializer can automatically highlight
                matches when it writes out a piece of XML. All the matches will be surrounded by an
                <sgmltag>exist:match</sgmltag> tag.</para><para>You can achieve the same within an XQuery by calling the extension function
                <funcsynopsis>util:expand</funcsynopsis>:</para><example><title>Using util:expand</title><programlisting language="xquery">let $expanded := util:expand($hit, "expand-xincludes=no")
return $expanded//exist:match</programlisting></example><para><function>util:expand</function> returns a copy of the XML fragment it received
                in its first parameter, which - unless configured otherwise - has all matches
                wrapped into <sgmltag>exist:match</sgmltag> tags.</para></section></chapter></book>