<?xml version="1.0" encoding="UTF-8"?>
<!-- $Id: devguide_indexes.xml 17299 2012-10-10 23:47:33Z joewiz $ --><book><bookinfo><productname>eXist-db – Open Source Native XML Database</productname><title>Developer's Guide</title><date>June 2007</date><author><firstname>Pierrick</firstname><surname>Brihaye</surname><affiliation><jobtitle>Software engineer</jobtitle><orgname>Service régional de l'inventaire de Bretagne</orgname><address format="linespecific"><email>pierrick dot brihaye at culture dot gouv dot fr</email></address><address format="linespecific"><email>pierrick dot brihaye at region dash bretagne dot fr</email></address></affiliation></author></bookinfo><chapter id="ch01"><title>Developer's Guide to Modularized Indexes</title><para>This document has been reviewed for eXist-db 1.2.</para><section><title>The new modularized indexes</title><section><title>Brief overview</title><para>Since around SVN revision 6000, spring 2007, i.e. <emphasis>after</emphasis> the 1.1 release, eXist-db 
	provides a new mechanism to index XML data. This mechanism is modular and should ease index development as 
	well as the development of related (possibly not so) custom functions. As a proof of concept, eXist currently 
	ships with two index types :</para><itemizedlist><listitem><term>NGram index</term><para>An NGram index will store the N-grams contained in the data's characters, i.e. if the index is configured to
		  index 3-grams, <markup>&lt;data&gt;abcde&lt;/data&gt;</markup> will generate these index entries :
		  <itemizedlist><listitem><para>abc</para></listitem><listitem><para>bcd</para></listitem><listitem><para>cde</para></listitem><listitem><para>de␣</para></listitem><listitem><para>e␣␣</para></listitem></itemizedlist></para></listitem><listitem><term>Spatial index</term><para>A spatial index will store some of the geometric characteristics of 
		  <ulink url="http://www.opengeospatial.org/standards/gml">Geography Markup Language</ulink> 
		  geometries (currently only tested with GML version 2.1.2). 
		  </para><para><programlisting><markup>
		  &lt;gml:Polygon xmlns:gml = 'http://www.opengis.net/gml' srsName='osgb:BNG'&gt;
		    &lt;gml:outerBoundaryIs&gt;
		      &lt;gml:LinearRing&gt;
		        &lt;gml:coordinates&gt;
			  278515.400,187060.450 278515.150,187057.950 278516.350,187057.150
			  278546.700,187054.000 278580.550,187050.900 278609.500,187048.100
			  278609.750,187051.250 278574.750,187054.650 278544.950,187057.450
			  278515.400,187060.450 
		        &lt;/gml:coordinates&gt;
		      &lt;/gml:LinearRing&gt;
		    &lt;/gml:outerBoundaryIs&gt;
		  &lt;/gml:Polygon&gt;
		  </markup></programlisting></para><para>will generate index entries among which most important are :
		  <itemizedlist>
		    <!-- TODO : change to <literal> --><listitem><para>the <ulink url="http://en.wikipedia.org/wiki/Spatial_referencing_systems">spatial referencing system</ulink> 
		    (<envar>osgb:BNG</envar> for this polygon)</para></listitem><listitem><para>the polygon itself, stored in a binary form (<ulink url="http://en.wikipedia.org/wiki/Well-known_text">Well-Known Binary</ulink>)</para></listitem><listitem><para>the coordinates of its <ulink url="http://en.wikipedia.org/wiki/Minimum_bounding_rectangle">bounding box</ulink></para></listitem></itemizedlist></para><para>
		  The spatial index will we discussed in details further below.
		</para></listitem></itemizedlist><para>So, the new architecture introduces a new package, <classname>org.exist.indexing</classname> 
	which contains a class that we will immediately study, <classname>IndexManager</classname>.</para></section><section><title><classname>org.exist.indexing.IndexManager</classname></title><para>As its name suggests, this is the class which is responsible for index management. It is created by 
	<classname>org.exist.storage.BrokerPool</classname> which allocates <classname>org.exist.storage.DBBroker</classname>s
	to each process accessing each DB instance.
	Each time a DB instance is created (most installations generally have only one, most often called <envar>exist</envar>), the 
	<methodname>initialize()</methodname> method contructs an <classname>IndexManager</classname> that will be available through the 
	<methodname>getIndexManager()</methodname> method of <classname>org.exist.storage.BrokerPool</classname>. 
	</para><synopsis format="linespecific">public IndexManager(BrokerPool pool, Configuration config)</synopsis><para>This constructor keeps track of the <classname>BrokerPool</classname> that has created the instance
	and receives the database's configuration object, usually defined in an XML file called <filename>conf.xml</filename>.
	This new entry is expected in the configuration file :
	</para><para><programlisting><markup>
	&lt;modules&gt;
	&lt;module id="ngram-index" class="org.exist.indexing.ngram.NGramIndex" file="ngram.dbx" n="3"/&gt;            
	&lt;module id="spatial-index" class="org.exist.indexing.spatial.GMLHSQLIndex" connectionTimeout="100000" flushAfter="300" /&gt;                     
	&lt;/modules&gt;
	</markup></programlisting></para><para>... which defines 2 indexes, backed-up by their specific classes (<sgmltag class="attribute">class</sgmltag> 
	attribute ; these classes implement the <classname>org.exist.indexing.Index</classname> interface as will
	be seen below), eventually assigns them a human-readable (writable even) identifier and passes them custom 
	parameters which are implementation-dependant. Then, it configures (by calling their 
	<methodname>configure()</methodname> method), opens (by calling their <methodname>open()</methodname> 
	method) and keeps track of each of them.</para><para><classname>org.exist.indexing.IndexManager</classname> also provides these public methods :</para><synopsis format="linespecific">public BrokerPool getBrokerPool()</synopsis><para>... which returns the <classname>org.exist.storage.BrokerPool</classname> for which this <classname>IndexManager</classname> was created.</para><synopsis format="linespecific">public synchronized Index getIndexById(String indexId)</synopsis><para>A method that returns an <classname>Index</classname> given its class identifier (see below). Allows custom functions to access <classname>Index</classname>es whatever their human-defined
	name is. This is probably the only method in this class that will be really needed by a developer.</para><synopsis format="linespecific">public synchronized Index getIndexByName(String indexName)</synopsis><para>The counterpart of the previous method. Pass the human-readable name of the <classname>Index</classname> as defined in the configuration.</para><synopsis format="linespecific">public void shutdown()</synopsis><para>This method is called when eXist shuts down. <methodname>close()</methodname> will be called
	for every registered <classname>Index</classname>. That allows them to free the resources they have allocated.</para><synopsis format="linespecific">public void removeIndexes()</synopsis><para>This method is called when <methodname>repair()</methodname> is called from <classname>org.exist.storage.NativeBroker</classname>.</para><para><note><para><methodname>repair()</methodname> reconstructs every index (including the structural one)
	from what is contained in the persistent DOM (usually <filename>dom.dbx</filename>).</para></note></para><para><methodname>remove()</methodname> will be called for every registered <classname>Index</classname>.
	  That allows each index to destroy its persistent storage if it wants to do so (but it is probably 
	suitable given that <methodname>repair()</methodname> is called when the DB and/or its indexes are corrupted).</para><synopsis format="linespecific">public void reopenIndexes()</synopsis><para>This method is called when <methodname>repair()</methodname> is called from <classname>org.exist.storage.NativeBroker</classname>.</para><para><note><para><methodname>repair()</methodname> reconstructs every index (including the structural one)
	from what is contained in the persistent DOM (usually <filename>dom.dbx</filename>).</para></note></para><para><methodname>open()</methodname> will be called for every registered <classname>Index</classname>.
	That allows each index to (re)allocate the resources it needs for its persistent storage.</para></section><section><title><classname>org.exist.indexing.IndexController</classname></title><para>Another important class is <classname>org.exist.indexing.IndexController</classname> which, as its
	name suggests, controls the way data to be indexed are dispatched to the registered indexes, using 
	<classname>org.exist.indexing.IndexWorker</classname>s that will be described below. Each <classname>org.exist.storage.DBBroker</classname>
	constructs such an <classname>IndexController</classname> when it is itself constructed, using this constructor :</para><synopsis format="linespecific">public IndexController(DBBroker broker)</synopsis><para>... that registers the <parameter>broker</parameter>'s <classname>IndexWorker</classname>s, 
	one for each registered <classname>Index</classname>. These <classname>IndexWorker</classname>s,
	that will be described below, are returned by the <methodname>getWorker()</methodname> method in 
	<classname>org.exist.indexing.Index</classname>, which is usually a good place to create such an 
	<classname>IndexWorker</classname>, at least the first time it is called.</para><para>This <classname>IndexController</classname> will be available through the 
	<methodname>getIndexController()</methodname> method of <classname>org.exist.storage.DBBroker</classname>.</para><para> Here are the other public methods :</para><synopsis format="linespecific">public Map configure(NodeList configNodes, Map namespaces)</synopsis><para>This method receives the database's configuration object, usually defined in an XML file called <filename>conf.xml</filename>.
	Both configuration nodes and namespaces (remember that some configuration settings including e.g. pathes need namespaces to be defined)
	will be passed to the <methodname>configure()</methodname> method of each <classname>IndexWorker</classname>. The returned object is a 
	<classname>java.util.Map</classname> that will be available from <methodname>collection.getIndexConfiguration(broker).getCustomIndexSpec(INDEX_CLASS_IDENTIFIER)</methodname>.
	</para><synopsis format="linespecific">public IndexWorker getWorkerByIndexId(String indexId)</synopsis><para>A method that returns an <classname>IndexWorker</classname> given the class identifier of its associated <classname>Index</classname> identifier. Very useful
	to the developer since it allows custom functions to access <classname>IndexWorker</classname>s whatever the human-defined
	name of their <classname>Index</classname> is. This is probably the only method in this class that will be really needed by a developer.</para><synopsis format="linespecific">public IndexWorker getWorkerByIndexName(String indexName)</synopsis><para>The counterpart of the previous method. For the human-readable name of the <classname>Index</classname> as defined in the configuration.</para><synopsis format="linespecific">public void setDocument(DocumentImpl doc)</synopsis><para>This method sets the <classname>org.exist.dom.DocumentImpl</classname> on which the <classname>IndexWorker</classname>s shall work.
	Calls <methodname>setDocument(doc)</methodname> on each registered <classname>IndexWorker</classname>.</para><synopsis format="linespecific">public void setMode(int mode)</synopsis><para>This method sets the operating mode in which the <classname>IndexWorker</classname>s shall work. 
	See below for further details on operating modes.
	Calls <methodname>setMode(mode)</methodname> on each registered <classname>IndexWorker</classname>.</para><synopsis format="linespecific">public void setDocument(DocumentImpl doc, int mode)</synopsis><para>A convenience method that sets both the <classname>org.exist.dom.DocumentImpl</classname> and the operating mode.
	Calls <methodname>setDocument(doc, mode)</methodname> on each registered <classname>IndexWorker</classname>.</para><synopsis format="linespecific">public DocumentImpl getDocument()</synopsis><para>Returns the <classname>org.exist.dom.DocumentImpl</classname> on which the <classname>IndexWorker</classname>s will have to work.</para><synopsis format="linespecific">public int getMode()</synopsis><para>Returns the operating mode in which the <classname>IndexWorker</classname>s will have to work.</para><synopsis format="linespecific">public void flush()</synopsis><para>Called in various places when pending operations, obviously data insertion, update or removal, have to be completed. Calls <methodname>flush()</methodname>
	on each registered <classname>IndexWorker</classname>.</para><synopsis format="linespecific">public void removeCollection(Collection collection, DBBroker broker)</synopsis><para>Called when a collection is to be removed. That allows to delete index entries for this collection in a single operation.
	Calls <methodname>removeCollection()</methodname> on each registered <classname>IndexWorker</classname>.</para><synopsis format="linespecific">public void reindex(Txn transaction, StoredNode reindexRoot, int mode)</synopsis><para>Called when a document is to be reindexed. Only the <parameter>reindexRoot</parameter> node and its descendants 
	will have their index entries updated or removed depending of the <parameter>mode</parameter> parameter.</para><synopsis format="linespecific">public StoredNode getReindexRoot(StoredNode node, NodePath path)</synopsis><para>Determines the node which should be reindexed together with its descendants. 
	Calls <methodname>getReindexRoot()</methodname> on each registered <classname>IndexWorker</classname>. 
	The top-most node will be the actual node from which the <classname>DBBroker</classname> will start reindexing.</para><synopsis format="linespecific">public StoredNode getReindexRoot(StoredNode node, NodePath path, boolean includeSelf)</synopsis><para>Same as above, with more parameters.</para><synopsis format="linespecific">public StreamListener getStreamListener()</synopsis><para>Returns the first <classname>org.exist.indexing.StreamListener</classname> in the
	<classname>StreamListener</classname>s pipeline. There is at most one <classname>StreamListener</classname>
	per <classname>IndexWorker</classname>  that will intercept the (re)indexed nodes stream. <classname>IndexWorker</classname>s that are not interested
	in the data (depending of e.g. the document and/or the operating mode) may return <parameter>null</parameter> through
	their <methodname>getListener()</methodname> method and thus not participate in the (re)indexing process. In other terms,
	they will not listen to the indexed nodes.</para><synopsis format="linespecific">public void indexNode(Txn transaction, StoredNode node, NodePath path, StreamListener listener)</synopsis><para>Index any kind of indexable node (currently elements, attributes and text nodes ; comments and especially
	processing instructions might be considered in the future).</para><synopsis format="linespecific">public void startElement(Txn transaction, ElementImpl node, NodePath path, StreamListener listener)</synopsis><para>More specific than <methodname>indexNode()</methodname>. For an element. Will call <methodname>startElement()</methodname>
	on <parameter>listener</parameter> if it is not <parameter>null</parameter>. Hence the analogy with 
	<ulink url="http://www.xml.com/pub/a/2003/09/17/stax.html">STAX events</ulink> is obvious.</para><synopsis format="linespecific">public void attribute(Txn transaction, AttrImpl node, NodePath path, StreamListener listener)</synopsis><para>More specific than <methodname>indexNode()</methodname>. For an attribute. Will call <methodname>attribute()</methodname>
	on <parameter>listener</parameter> if it is not <parameter>null</parameter>.</para><synopsis format="linespecific">public void characters(Txn transaction, TextImpl node, NodePath path, StreamListener listener)</synopsis><para>More specific than <methodname>indexNode()</methodname>. For a text node. Will call <methodname>characters()</methodname>
	on <parameter>listener</parameter> if it is not <parameter>null</parameter>.</para><synopsis format="linespecific">public void endElement(Txn transaction, ElementImpl node, NodePath path, StreamListener listener)</synopsis><para>Signals end of indexing for an element node. Will call <methodname>endElement()</methodname>
	on <parameter>listener</parameter> if it is not <parameter>null</parameter></para><synopsis format="linespecific">public MatchListener getMatchListener(NodeProxy proxy)</synopsis><para>Returns a <classname>org.exist.indexing.MatchListener</classname>	for the given node.</para><para>The two classes aim to be essentially used by eXist itself. As a programmer you
	will probably need to use just one or two of the above methods.</para></section><section><title><classname>org.exist.indexing.Index</classname> and <classname>org.exist.indexing.AbstractIndex</classname></title><para>Now let's get into the interfaces and classes that will need to be extended by the index programmer. The
	first of them is the interface <classname>org.exist.indexing.Index</classname> which will maintain the index itself.</para><para>As described above, a new instance of the interface will be created by the constructor of <classname>org.exist.indexing.IndexManager</classname> 
	which calls the interface's <methodname>newInstance()</methodname> method. No need for a constructor then.</para><para>Here are the methods that have to be implemented in an implementation: </para><synopsis format="linespecific">String getIndexId()</synopsis><para>Returns the class identifier of the index.</para><synopsis format="linespecific">String getIndexName()</synopsis><para>Returns the human-defined name of the index, if one was defined in the configuration file.</para><synopsis format="linespecific">BrokerPool getBrokerPool()</synopsis><para>Returns the <classname>org.exist.storage.BrokerPool</classname> that has created the index.</para><synopsis format="linespecific">void configure(BrokerPool pool, String dataDir, Element config)</synopsis><para>Notifies the <classname>Index</classname> a data directory (normally <filename>${EXIST_HOME}/webapp/WEB-INF/data</filename>) and the
	configuration element in which it is declared.</para><synopsis format="linespecific">void open()</synopsis><para>Method that is executed when the <classname>Index</classname> is opened, whatever it means. Consider
	this method as an initialization and allocate the necessary resources here.</para><synopsis format="linespecific">void close()</synopsis><para>Method that is executed when the <classname>Index</classname> is closed, whatever it means. Consider
	this method as a finalization and free the allocated resources here.</para><synopsis format="linespecific">void sync()</synopsis><para>Unused.</para><synopsis format="linespecific">void remove()</synopsis><para>Method that is executed when eXist requires the index content to be entitrely deleted, e.g. before repairing a corrupted database.</para><synopsis format="linespecific">IndexWorker getWorker(DBBroker broker)</synopsis><para>Returns the <classname>IndexWorker</classname> that operates on this
	<classname>Index</classname> on behalf of <parameter>broker</parameter>. One may want to create a new <classname>IndexWorker</classname>
	here or pick one form a pool.</para><synopsis format="linespecific">boolean checkIndex(DBBroker broker)</synopsis><para>To be called by applications that want to implement a consistency check on the <classname>Index</classname>.</para><para>There is also an abstract class that implements <classname>org.exist.indexing.Index</classname>, 
	<classname>org.exist.indexing.AbstractIndex</classname> that can be used a a basis for most <classname>Index</classname> implementations. 
	Most of its methods are abstract and still have to be implemented in the concrete classes. 
	These are the few concrete methods:</para><synopsis format="linespecific">public String getDataDir()</synopsis><para>Returns the directory in which this <classname>Index</classname> operates. Usually defined
	by <methodname>configure()</methodname> which itself receives eXist's configuration settings. NB! There might
	be some <classname>Index</classname>es for which the concept of data directory isn't accurate.</para><synopsis format="linespecific">public void configure(BrokerPool pool, String dataDir, Element config)</synopsis><para>Its minimal implementation retains the <classname>org.exist.storage.BrokerPool</classname>, 
	the data directory and the human-defined name, if defined in the configuration file (in an attribute
	called <sgmltag class="attribute">id</sgmltag>).
	Sub-classes may call <methodname>super.configure()</methodname> to retain this default behaviour.</para><para>This member is protected :</para><synopsis format="linespecific">protected static String ID = "Give me an ID !"</synopsis><para>This is where the class identifier of the <classname>Index</classname> is defined. Override
	this member with, say, <methodname>MyClass.class.getName()</methodname> to provide a reasonably
	unique identifier within your system.</para></section><section><title><classname>org.exist.indexing.IndexWorker</classname></title><para>The next important interface that will need to be implemented is <classname>org.exist.indexing.IndexWorker</classname>
	which is responsible for managing the data in the index. Remember that each <classname>org.exist.storage.DBBroker</classname>
	will have such an <classname>IndexWorker</classname> at its disposal and that their 
	<classname>IndexController</classname> will know what method of <classname>IndexWorker</classname> to call and when to call it.</para><para>Here are the methods that have to be implemented in the concrete implementations : </para><synopsis format="linespecific">public String getIndexId()</synopsis><para>Returns the class identifier of the index.</para><synopsis format="linespecific">public String getIndexName()</synopsis><para>Returns the human-defined name of the index, if one was defined in the configuration file.</para><synopsis format="linespecific">Object configure(IndexController controller, NodeList configNodes, Map namespaces)</synopsis><para>This method receives the database's configuration object, usually defined in an XML file called <filename>conf.xml</filename>.
	Both configuration nodes and namespaces (remember that some configuration settings including e.g. pathes need namespaces to be defined)
	will be passed to the <methodname>configure()</methodname> method of the <classname>IndexWorker</classname>'s 
	<classname>IndexController</classname>. The <classname>IndexWorker</classname> can use this method to retain
	custom configuration options in a custom object that will be available in the <classname>java.util.Map</classname> 
	returned by <methodname>collection.getIndexConfiguration(broker).getCustomIndexSpec(INDEX_CLASS_IDENTIFIER)</methodname>.
	The return type is free but will probably generally be an implementation of <classname>java.util.Collection</classname> in order
	to retain several parameters.
	</para><synopsis format="linespecific">void setDocument(DocumentImpl doc)</synopsis><para>This method sets the <classname>org.exist.dom.DocumentImpl</classname> on which this <classname>IndexWorker</classname> will have to work.</para><synopsis format="linespecific">void setMode(int mode)</synopsis><para>This method sets the operating mode in which this <classname>IndexWorker</classname> will have to work. 
	See below for further details on operating modes.</para><synopsis format="linespecific">void setDocument(DocumentImpl doc, int mode)</synopsis><para>A convenience method that sets both the <classname>org.exist.dom.DocumentImpl</classname> and the operating mode.</para><synopsis format="linespecific">DocumentImpl getDocument()</synopsis><para>Returns the <classname>org.exist.dom.DocumentImpl</classname> on which this <classname>IndexWorker</classname> will have to work.</para><synopsis format="linespecific">int getMode()</synopsis><para>Returns the operating mode in which this <classname>IndexWorker</classname> will have to work.</para><synopsis format="linespecific">void flush()</synopsis><para>Called periodically by the <classname>IndexController</classname> or by any other process. That is 
	where data insertion, update or removal should actually take place.</para><synopsis format="linespecific">void removeCollection(Collection collection, DBBroker broker)</synopsis><para>Called when a collection is to be removed. That allows to delete index entries for this collection 
	in a single operation without a need for a <classname>StreamListener</classname> (see below) or a call to 
	<methodname>setMode()</methodname> nor <methodname>setDocument()</methodname>.</para><synopsis format="linespecific">StoredNode getReindexRoot(StoredNode node, NodePath path, boolean includeSelf)</synopsis><para>Determines the node which should be reindexed together with its descendants. This will give a hint to 
	the <classname>IndexController</classname> to determine from which node reindexing should start.</para><synopsis format="linespecific">StreamListener getListener()</synopsis><para>Returns a <classname>StreamListener</classname> that will intercept the (re)indexed nodes stream.
	<classname>IndexWorker</classname>s that are not interested in the data (depending of e.g. the document and/or the operating mode) may return <parameter>null</parameter> here.</para><synopsis format="linespecific">MatchListener getMatchListener(NodeProxy proxy)</synopsis><para>Returns a <classname>org.exist.indexing.MatchListener</classname>	for the given node.</para><synopsis format="linespecific">boolean checkIndex(DBBroker broker)</synopsis><para>To be called by applications that want to implement a consistency check on the index.</para><synopsis format="linespecific">Occurrences[] scanIndex(DocumentSet docs)</synopsis><para>Returns an array of <classname>org.exist.dom.DocumentImpl.Occurrences</classname> that is an
	<emphasis>ordered</emphasis> list of the index entries, in a textual form, associated with the number of occurences for the 
	entries and a list of the documents containing them. NB! For some indexes, the
	concept of ordered or textual occurrences might not be meaningful.</para></section><section><title><classname>org.exist.indexing.StreamListener</classname> and <classname>org.exist.indexing.AbstractStreamListener</classname></title><para>The interface <classname>org.exist.indexing.StreamListener</classname> has these public members :</para><synopsis format="linespecific">public final static int UNKNOWN = -1;</synopsis><synopsis format="linespecific">public final static int STORE = 0;</synopsis><synopsis format="linespecific">public final static int REMOVE_ALL_NODES = 1;</synopsis><synopsis format="linespecific">public final static int REMOVE_SOME_NODES = 2;</synopsis><para>Obviously, they are used by the <methodname>setMode()</methodname> method in <classname>org.exist.indexing.IndexController</classname>
	which is istself called by the different <classname>org.exist.storage.DBBroker</classname>s when they have to (re)index
	a node and its descendants. As their name suggests, there is a mode for storing nodes and two modes
	for removing them from the indexes. The difference between <classname>StreamListener.REMOVE_ALL_NODES</classname> and
	<classname>StreamListener.REMOVE_SOME_NODES</classname> is that the former removes all the nodes from a document
	whereas the latter removes only some nodes from a document, usually the descendants of the node returned by
	<methodname>getReindexRoot()</methodname>. We thus have the opportunity to trigger a process that will directly
	remove all the nodes from a given document without having to listen to each of them. Such a technique is described below.</para><para>Here are the methods that must be implement by an implemetation:</para><synopsis format="linespecific">IndexWorker getWorker()</synopsis><para>Returns the <classname>IndexWorker</classname> that owns this <classname>StreamListener</classname>.</para><synopsis format="linespecific">void setNextInChain(StreamListener listener);</synopsis><para>Should not be used. Used to specify which is the next <classname>StreamListener</classname> in the
	<classname>IndexController</classname>'s <classname>StreamListener</classname>s pipeline.</para><synopsis format="linespecific">StreamListener getNextInChain();</synopsis><para>Returns the next <classname>StreamListener</classname> in the
	<classname>IndexController</classname>'s <classname>StreamListener</classname>s pipeline. Very important
	because it is the responsability of the <classname>StreamListener</classname> to forward the event stream to
	the next <classname>StreamListener</classname> in the pipeline.</para><synopsis format="linespecific">void startElement(Txn transaction, ElementImpl element, NodePath path)</synopsis><para>Signals the start of an element to the listener.</para><synopsis format="linespecific">void attribute(Txn transaction, AttrImpl attrib, NodePath path)</synopsis><para>Passes an attribute to the listener.</para><synopsis format="linespecific">void characters(Txn transaction, TextImpl text, NodePath path)</synopsis><para>Passes some character data to the listener.</para><synopsis format="linespecific">void endElement(Txn transaction, ElementImpl element, NodePath path)</synopsis><para>Signals the end of an element to the listener. Allow to free any temporary resource created since
	the matching <methodname>startElement()</methodname> has been called.</para><para>Beside the <classname>StreamListener</classname> interface, each custom
	listener should extend <classname>org.exist.indexing.AbstractStreamListener</classname>.</para><para>This abstract class provides concrete implementations for <methodname>setNextInChain()</methodname>
	and <methodname>getNextInChain()</methodname> that should normally never be overridden.</para><para>It also provides dummy <methodname>startElement()</methodname>, <methodname>attribute()</methodname>,
	<methodname>characters()</methodname>, <methodname>endElement()</methodname> methods that do nothing but
	forwarding the node to the next <classname>StreamListener</classname> in the
	<classname>IndexController</classname>'s <classname>StreamListener</classname>s 
	pipeline.</para><synopsis format="linespecific">public abstract IndexWorker getWorker()</synopsis><para>remains abstract though, since we still can not know what <classname>IndexWorker</classname>
	will own the <classname>Listener</classname> until we haven't a concrete implementation.</para></section></section><section id="spatial"><title>A use case : developing an indexing architecture for GML geometries</title><section><title>Introduction</title><para>To demonstrate how modular eXist <classname>Index</classname>es are, we have decided to show how
	a spatial <classname>Index</classname> could be implemented. What makes its design interesting is that this
	kind of <classname>Index</classname> doesn't store character data from the document, nor does it use a
	<classname>org.exist.storage.index.BFile</classname> to store the index entries. Instead, we will store
	WKB index entries in a JDBC database, namely a <ulink url="http://hsqldb.org/">HSQLDB</ulink> to keep the distribution as light as possible and reduce the
	number of external dependencies, but it wouldn't be too difficult to use another one like 
	<ulink url="http://postgis.refractions.net/">PostGIS</ulink> given that the implementation
	has itself been designed in a quite modular way.
	</para><para>In eXist's SVN repository, the modularized <classname>Index</classname>es code is in 
	<filename>extensions/indexes</filename> and the file system's architecture is designed to follow
	eXist's core architecture, i.e. <classname>org.exist.indexing.*</classname> for the <classname>Index</classname>es
	and <classname>org.exist.xquery.*</classname> for their associated <classname>Module</classname>s. There is also a dedicated location for required external
	libraries and for the test cases. The build system should normally be able to download the required
	libraries from the WWW (do no forget to adjust your proxy server's properties in <filename>build.properties</filename>
	if required) build the all the files automatically, in particular the <parameter>extension-modules</parameter> 
	Ant target, and even launch the tests provided that the DB's configuration file declares the
	<classname>Index</classname>es (see above) and their associated <classname>Module</classname>s (see below).</para><para>The described spatial <classname>Index</classname> heavily relies on the excellent open source 
	librairies provided by the <ulink url="http://geotools.codehaus.org/">Geotools</ulink> project. We
	have experienced a few problems that will be mentioned further, but since feedback has been provided,
	the situation will unquestionably improve in the future, making current workarounds redundant.</para><para>The <classname>Index</classname> has been tested with only one file which is available from the
	<ulink url="http://www.ordnancesurvey.co.uk/oswebsite/">Ordnance Survey of Great-Britain</ulink>, a topography layer of Port-Talbot, which
		is available as <ulink url="http://www.ordnancesurvey.co.uk/oswebsite/products/try-now/sample-data.html">sample data</ulink>.
	Shall we mention that obtaining <emphasis>valid</emphasis> and sizeable GML data is still extremely difficult?
	</para></section><section><title>Writing the concrete implementation of <classname>org.exist.indexing.AbstractIndex</classname></title><para>Well, in fact we will start by writing an abstract implementation first. As said above, we have
	planned a modular JDBC spatial <classname>Index</classname>, which will be abstract, and that will be extended by a concrete HSQLDB <classname>Index</classname>.</para><para>Let's start with this :</para><programlisting format="linespecific">
	package org.exist.indexing.spatial;			
	
	public abstract class AbstractGMLJDBCIndex extends AbstractIndex {

	public final static String ID = AbstractGMLJDBCIndex.class.getName();	
	private final static Logger LOG = Logger.getLogger(AbstractGMLJDBCIndex.class);
	protected HashMap workers = new HashMap();
	protected Connection conn = null;
	
	}
	</programlisting><para> ... where we define an abstract class that extends <classname>org.exist.indexing.AbstractIndex</classname> and thus
	implements <classname>org.exist.indexing.Index</classname>. We also
	define a few members like <parameter>ID</parameter> that will be returned by the unoverriden <methodname>getIndexId()</methodname> from
	<classname>org.exist.indexing.AbstractIndex</classname>, a <classname>Logger</classname>, 
	a <classname>java.util.HashMap</classname> that will be a "pool" of	<classname>IndexWorker</classname>s
	(one for each <classname>org.exist.storage.DBBroker</classname>) and a <classname>java.sql.Connection</classname>
	that will handle the database operations at the index level.
	</para><para>Let' now introduce this general purpose interface :</para><programlisting format="linespecific">
	public interface SpatialOperator { 
	  public static int UNKNOWN = -1;
	  public static int EQUALS = 1;
	  public static int DISJOINT = 2;
	  public static int INTERSECTS = 3;
	  public static int TOUCHES = 4;
	  public static int CROSSES = 5;
	  public static int WITHIN = 6;
	  public static int CONTAINS = 7;
	  public static int OVERLAPS = 8;
	}			
	</programlisting><para>... that defines the spatial operators that will be used by spatial queries (what would be
	worth a spatial index that doesn't support spatial queries?). For more information about the
	semantics, see the <ulink url="http://www.vividsolutions.com/jts/bin/JTS%20Technical%20Specs.pdf">JTS documentation</ulink> 
	(chapter 11). We will use this wonderful library everytime a spatial computation is required. So does
	the Geotools project by the way.</para><para>Here are a few concrete methods that should be usable by any JDBC-enabled database:</para><programlisting format="linespecific">			 
	public AbstractGMLJDBCIndex() {    	
	}  
	
	public void configure(BrokerPool pool, String dataDir, Element config) throws DatabaseConfigurationException {        
	  super.configure(pool, dataDir, config);
	  try {
	    checkDatabase();
	  } catch (ClassNotFoundException e) {
	  throw new DatabaseConfigurationException(e.getMessage()); 
	  } catch (SQLException e) {
	  throw new DatabaseConfigurationException(e.getMessage()); 
	  }
	}

	public void open() throws DatabaseConfigurationException {     
	}

	public void close() throws DBException {
	  Iterator i = workers.values().iterator();
	  while (i.hasNext()) {
	    AbstractGMLJDBCIndexWorker worker = (AbstractGMLJDBCIndexWorker)i.next();		
	    worker.flush();		
	    worker.setDocument(null, StreamListener.UNKNOWN);
	  }
	  shutdownDatabase();
	}

	public void sync() throws DBException {
	}
	
	public void remove() throws DBException {
	  Iterator i = workers.values().iterator();
	  while (i.hasNext()) {
	    AbstractGMLJDBCIndexWorker worker = (AbstractGMLJDBCIndexWorker)i.next();		
	    worker.flush();		
	    worker.setDocument(null, StreamListener.UNKNOWN);
	  }
	  removeIndexContent();
	  shutdownDatabase();
	  deleteDatabase();	
	}			

	public boolean checkIndex(DBBroker broker) {
	  return getWorker(broker).checkIndex(broker);
	} 
	</programlisting><para>First, an empty constructor, not even necessary since the <classname>Index</classname> is created through
	the <methodname>newInstance()</methodname> method of its interface (see above).</para><para>Then, a configuration method that calls its ancestor, whose behaviour fullfills our needs.
	This method calls a <methodname>checkDatabase()</methodname> method whose semantics will be dependant of the
	underlying DB. The basic idea is to prevent eXist to continue its initialization if there is a problem with the DB.</para><para>Then we will do nothing during <methodname>open()</methodname>. No need to open a database, which is costly,
	if we dont need it.</para><para>The <methodname>close()</methodname> will flush any pending operation currently queued by the <classname>IndexWorker</classname>s
	and resets their state in order to prevent them to start any further operation, which should never be possible if eXist is their only user.
	Then it will call a <methodname>shutdownDatabase()</methodname> method whose semantics will be dependant
	of the	underlying DB. They can be fairly simple for DBs that shut down automatically when the virtual
	machine shuts down.</para><para>The <methodname>sync()</methodname> is never called by eXist. It's here to make the interface happy.</para><para>The <methodname>remove()</methodname> method is similar to <methodname>close()</methodname>. It then
	calls two database-dependant methods that are pretty redundant. <methodname>deleteDatabase()</methodname> will
	probably not be able to do what its name suggests if eXist doesn't own the admin rights. Conversely,
	<methodname>removeIndexContent()</methodname> wiould probably have nothing to do if eXist owns the admin rights
	since physically destroying the DB would probably be more efficient than deleteing table contents.</para><para><methodname>checkIndex()</methodname> will delegate the task to the <parameter>broker</parameter>'s
	<classname>IndexWorker</classname>.</para><para>The remaining methods are DB-dependant and thus abstract :</para><programlisting format="linespecific">				
	public abstract IndexWorker getWorker(DBBroker broker);
	protected abstract void checkDatabase() throws ClassNotFoundException, SQLException;
	protected abstract void shutdownDatabase() throws DBException;
	protected abstract void deleteDatabase() throws DBException;
	protected abstract void removeIndexContent() throws DBException;
	protected abstract Connection acquireConnection(DBBroker broker) throws SQLException;   
	protected abstract void releaseConnection(DBBroker broker) throws SQLException;			
	</programlisting><para>Let's see now how a HSQL-dependant implementation would be going by describing
	the concrete class :</para><programlisting format="linespecific">				
	package org.exist.indexing.spatial;

	public class GMLHSQLIndex extends AbstractGMLJDBCIndex {
	
	  private final static Logger LOG = Logger.getLogger(GMLHSQLIndex.class);
	  public static String db_file_name_prefix = "spatial_index";
	  public static String TABLE_NAME = "SPATIAL_INDEX_V1";
	  private DBBroker connectionOwner = null;
	  private long connectionTimeout = 100000L;
	
	  public GMLHSQLIndex() {    	
	  } 
	
	}
	</programlisting><para>Of course, we extend <classname>org.exist.indexing.spatial.AbstractGMLJDBCIndex</classname>, then
	a few members are defined : a <classname>Logger</classname>, a file prefix (which will be required by the files 
	required by HSQLDB storage, namely <filename>spatial_index.lck</filename>, <filename>spatial_index.log</filename>,
	<filename>spatial_index.script</filename> and <filename>spatial_index.properties</filename>), then a table name in which
	the spatial index data will be stored, then a variable that will hold the <classname>org.exist.storage.DBBroker</classname>
	that currently holds a connection to the DB (we could have used an <classname>IndexWorker</classname> here,
	given their 1:1 relationship). The problem is that we run HSQLDB in embedded mode and that only one connection is available at a given time.</para><para>A more elaborated	DBMS, or HSQLDB running in server mode would permit the allocation of one connection per 
	<classname>IndexWorker</classname>, but we have chosen to keep things simple for now. Indeed, if 
	<classname>IndexWorker</classname>s are thread-safe (because each <classname>org.exist.storage.DBBroker</classname> 
	operates within its own thread), a single connection will have to be controlled by the <classname>Index</classname>
	which is controlled by the <classname>org.exist.storage.BrokerPool</classname>. See below how we will 
	handle concurrency, given such perequisites.</para><para>The last member is the timeout when a <classname>Connection</classname> to the DB is requested.</para><para>As we can see, we have an empty constructor again.</para><para>The next method calls its ancestor's <methodname>configure()</methodname> method and just retains 
	the content of the <sgmltag class="attribute">connectionTimeout</sgmltag> attribute as defined in the configuration file.</para><programlisting format="linespecific">				
	public void configure(BrokerPool pool, String dataDir, Element config) 
	  throws DatabaseConfigurationException {
	  super.configure(pool, dataDir, config);
	  String param = ((Element)config).getAttribute("connectionTimeout");
	  if (param != null) {
	    try {
	    connectionTimeout = Long.parseLong(param);
	    } catch (NumberFormatException e) {
	    LOG.error("Invalid value for 'connectionTimeout'", e);
	    }
	  }    	
	}
	</programlisting><para>The next method is also quite straightforward :</para><programlisting format="linespecific">				
	public IndexWorker getWorker(DBBroker broker) {
	  GMLHSQLIndexWorker worker = (GMLHSQLIndexWorker)workers.get(broker);
	  if (worker == null) {
	    worker = new GMLHSQLIndexWorker(this, broker);
	    workers.put(broker, worker);
	  }
	  return worker;
	}	
	</programlisting><para>It picks an <classname>IndexWorker</classname> (more precisely a <classname>org.exist.indexing.spatial.GMLHSQLIndexWorker</classname> 
	that will be described below) for the given <parameter>broker</parameter> from the "pool".
	If needed, namely the first time the method is called with with parameter, it creates one. Notice that 
	this <classname>IndexWorker</classname> is DB-dependant. It will be described below.</para><para>Then come a few general-purpose methods:</para><programlisting format="linespecific">				
	protected void checkDatabase() 
	  throws ClassNotFoundException, SQLException {	
	  Class.forName("org.hsqldb.jdbcDriver");		
	}
	
	protected void shutdownDatabase()
	  throws DBException {
	  try {
	    if (conn != null) {
	    Statement stmt = conn.createStatement();				
	    stmt.executeQuery("SHUTDOWN");
	    stmt.close();
	    conn.close();				
	    if (LOG.isDebugEnabled()) 
	      LOG.debug("GML index: " + getDataDir() + "/" + db_file_name_prefix + " closed");
	    }
	  } catch (SQLException e) {
	    throw new DBException(e.getMessage()); 
	  } finally {
	  conn = null;
	  }
	}
	
	protected void deleteDatabase()
	  throws DBException {
	  File directory = new File(getDataDir());
	  File[] files = directory.listFiles( 
	    new FilenameFilter() {
              public boolean accept(File dir, String name) {
	        return name.startsWith(db_file_name_prefix);
	      }
	    }
	  );
	  boolean deleted = true;
	  for (int i = 0; i &lt; files.length ; i++) {
  	    deleted &amp;= files[i].delete();
	  }
	}
	
	protected void removeIndexContent()
          throws DBException {
	  try {
	    //Let's be lazy here : we only delete th index content if we have a connection
	    if (conn != null) {
	    Statement stmt = conn.createStatement(); 
	    int nodeCount = stmt.executeUpdate("DELETE FROM " + GMLHSQLIndex.TABLE_NAME + ";");       
	    stmt.close();
	    if (LOG.isDebugEnabled()) 
 	      LOG.debug("GML index: " + getDataDir() + "/" + db_file_name_prefix + ". " + 
	      nodeCount + " nodes removed");
	    }		
	  } catch (SQLException e) {
	    throw new DBException(e.getMessage()); 
	  }
	}		
	</programlisting><para><methodname>checkDatabase()</methodname> just checks that we have a suitable driver in the CLASSPATH.
	We don't want to open the database right now. It costs too much.</para><para><methodname>shutdownDatabase()</methodname> is just one of the many ways to shutdown a HSQLDB.</para><para><methodname>deleteDatabase()</methodname> is just a file system management problem ; remember that
	the database should be closed at that moment : no file locking issues.</para><para><methodname>removeIndexContent()</methodname> deletes the table that contains spatial data. Less
	efficient than deleteing the whole databse though ;-), as explained above.</para><para>The 2 next methods are totally JDBC-specific and, given the way they are implemented, are
	totally embedded HSQLDB-specific. The <emphasis>current</emphasis> code is directly adapted from <classname>org.exist.storage.lock.ReentrantReadWriteLock</classname>
	to show that connection management should be severely controlled given the concurrency context induced by
	using many <classname>org.exist.storage.DBBroker</classname>. Despite the fact <classname>DBBroker</classname>s are 
	thread-safe, access to <emphasis>shared</emphasis> storage must be concurrential, in particular when <methodname>flush()</methodname> is called.
	</para><para><classname>org.exist.storage.index.BFile</classname> users would call <methodname>getLock()</methodname> to
	acquire and release locks on the index files. Our solution is thus very similar.</para><para>However, since most JDBC databases are able to work in a concurrential context, it 
	would then be better to never call these <classname>Index</classname>-level
	methods from the <classname>IndexWorker</classname>s and let each <classname>IndexWorker</classname>
	handle its connection to the underlying DB.</para><programlisting format="linespecific">				
	protected Connection acquireConnection(DBBroker broker)
	  throws SQLException {
	  synchronized (this) {	
	    if (connectionOwner == null) {
	      connectionOwner = broker;
	      if (conn == null)
	        initializeConnection();
	      return conn;
	    } else {    
	      long waitTime = connectionTimeout;
	      long waitTime = timeOut_;
	      long start = System.currentTimeMillis();
	      try {
	        for (;;) {
		  wait(waitTime);  
		  if (connectionOwner == null) {
		    connectionOwner = broker;	    			
		    if (conn == null)
		      //We should never get there since the connection should have been initialized
		      //by the first request from a worker
		    initializeConnection();
		    return conn; 			
		  } else {
		    waitTime = timeOut_ - (System.currentTimeMillis() - start);
		    if (waitTime &lt;= 0) {
		      LOG.error("Time out while trying to get connection");
		    }
		  }
	      }
	    } catch (InterruptedException ex) {
	      notify();
	      throw new RuntimeException("interrupted while waiting for lock");
	    }
	  }
	  }
	}

	protected synchronized void releaseConnection(DBBroker broker)
	  throws SQLException {   
	  if (connectionOwner == null)
	    throw new SQLException("Attempted to release a connection that wasn't acquired");
	  connectionOwner = null;
	}  			
	</programlisting><para><methodname>acquireConnection()</methodname> acquires an <emphasis>exclusive</emphasis> JDBC <classname>Connection</classname> to
	the storage engine for an <classname>IndexWorker</classname> (or a <classname>org.exist.storage.DBBroker</classname>, which 
	roughly means the same thing). This is where a <classname>Connection</classname> is created if necessary (see below) and makes
	 the first connection's performance cost due only when needed.</para><para><methodname>releaseConnection()</methodname> marks the connection as being unused. It will thus
	become available when requested again.</para><para>The last method concentrates the index-level DB-dependant code in just one place (<methodname>removeIndexContent()</methodname>
	is relatively DB-independant).</para><programlisting format="linespecific">				
	private void initializeConnection()
	  throws SQLException {
	  System.setProperty("hsqldb.cache_scale", "11");
	  System.setProperty("hsqldb.cache_size_scale", "12");
	  System.setProperty("hsqldb.default_table_type", "cached");
	  //Get a connection to the DB... and keep it
	  this.conn = DriverManager.getConnection("jdbc:hsqldb:" + getDataDir() + "/" + db_file_name_prefix, "sa", "");
	  try {	
	    ResultSet rs = this.conn.getMetaData().getTables(null, null, TABLE_NAME, new String[] { "TABLE" });
	    rs.last(); 
	    if (rs.getRow() == 1) {
	      if (LOG.isDebugEnabled()) 
	        LOG.debug("Opened GML index: " + getDataDir() + "/" + db_file_name_prefix); 
	      //Create the data structure if it doesn't exist
	    } else if (rs.getRow() == 0) {
	      Statement stmt = conn.createStatement();
	      stmt.executeUpdate("CREATE TABLE " + TABLE_NAME + "(" +
	      /*1*/ "DOCUMENT_URI VARCHAR, " +        		
	      /*2*/ "NODE_ID_UNITS INTEGER, " + 
	      /*3*/ "NODE_ID BINARY, " +        			
	      ...	        	
	      /*26*/ "IS_VALID BOOLEAN, " +
	      //Enforce uniqueness
	      "UNIQUE (" +
	      "DOCUMENT_URI, NODE_ID_UNITS, NODE_ID" +
	      ")" +
	      ")"
	      );
	      stmt.executeUpdate("CREATE INDEX DOCUMENT_URI ON " + TABLE_NAME + " (DOCUMENT_URI);");
	      ...
	      stmt.close();        	
	      if (LOG.isDebugEnabled()) 
	        LOG.debug("Created GML index: " + getDataDir() + "/" + db_file_name_prefix);  
	    } else {
	      throw new SQLException("2 tables with the same name ?"); 
	    }
	  } finally {
	    if (rs != null)
	    rs.close();    				
	  }        
	} 			
	</programlisting><para>This method opens a <classname>Connection</classname> and, if it is a new one (<emphasis>the</emphasis>
	new one since we only have one), checks that we have a SQL table for the spatial data. If not, i.e. if
	the spatial index doesn't exist yet, a table is created with the following structure :</para><table colsep="5" rowsep="5"><title/><tgroup cols="4"><tbody><row><entry style="background-color: #C5DAFF" align="left">Field name</entry><entry style="background-color: #C5DAFF" align="center">Field type</entry><entry style="background-color: #C5DAFF" align="left">Description</entry><entry style="background-color: #C5DAFF" align="left">Comments</entry></row><row><entry>DOCUMENT_URI</entry><entry align="center">VARCHAR</entry><entry>The document's URI</entry><entry/></row><row><entry>NODE_ID_UNITS</entry><entry align="center">INTEGER</entry><entry>The number of useful <emphasis>bits</emphasis> in NODE_ID</entry><entry>See below</entry></row><row><entry>NODE_ID</entry><entry align="center">BINARY</entry><entry>The node ID, as a byte array</entry><entry>See above. Only <emphasis>some</emphasis> bits might be considered due to obvious data alignment requirements</entry></row><row><entry>GEOMETRY_TYPE</entry><entry align="center">VARCHAR</entry><entry>The geometry type</entry><entry>As returned by the JTS</entry></row><row><entry>SRS_NAME</entry><entry align="center">VARCHAR</entry><entry>The SRS of the geometry</entry><entry><sgmltag class="attribute">srsName</sgmltag> attribute in the GML element</entry></row><row><entry>WKT</entry><entry align="center">VARCHAR</entry><entry>The <ulink url="http://de.wikipedia.org/wiki/Well_Known_Text">Well-Known Text</ulink> representation of the geometry</entry><entry/></row><row><entry>WKB</entry><entry align="center">BINARY</entry><entry>The WKB representation of the geometry</entry><entry/></row><row><entry>MINX</entry><entry align="center">DOUBLE</entry><entry>The minimal X of the geometry</entry><entry/></row><row><entry>MAXX</entry><entry align="center">DOUBLE</entry><entry>The maximal X of the geometry</entry><entry/></row><row><entry>MINY</entry><entry align="center">DOUBLE</entry><entry>The minimal Y of the geometry</entry><entry/></row><row><entry>MAXY</entry><entry align="center">DOUBLE</entry><entry>The maximal Y of the geometry</entry><entry/></row><row><entry>CENTROID_X</entry><entry align="center">DOUBLE</entry><entry>The X of the geometry's centroid</entry><entry/></row><row><entry>CENTROID_Y</entry><entry align="center">DOUBLE</entry><entry>The Y of the geometry's centroid</entry><entry/></row><row><entry>AREA</entry><entry align="center">DOUBLE</entry><entry>The area of the geometry</entry><entry>Expressed in the measure defined in its SRS</entry></row><row><entry>EPSG4326_WKT</entry><entry align="center">VARCHAR</entry><entry>The WKT representation of the geometry</entry><entry>In the <ulink url="http://en.wikipedia.org/wiki/EPSG:4326">epsg:4326</ulink> SRS</entry></row><row><entry>EPSG4326_WKB</entry><entry align="center">BINARY</entry><entry>The WKB representation of the geometry</entry><entry>In the epsg:4326 SRS</entry></row><row><entry>EPSG4326_MINX</entry><entry align="center">DOUBLE</entry><entry>The minimal X of the geometry</entry><entry>In the epsg:4326 SRS</entry></row><row><entry>EPSG4326_MAXX</entry><entry align="center">DOUBLE</entry><entry>The maximal X of the geometry</entry><entry>In the epsg:4326 SRS</entry></row><row><entry>EPSG4326_MINY</entry><entry align="center">DOUBLE</entry><entry>The minimal Y of the geometry</entry><entry>In the epsg:4326 SRS</entry></row><row><entry>EPSG4326_MAXY</entry><entry align="center">DOUBLE</entry><entry>The maximal Y of the geometry</entry><entry>In the epsg:4326 SRS</entry></row><row><entry>EPSG4326_CENTROID_X</entry><entry align="center">DOUBLE</entry><entry>The X of the geometry's centroid</entry><entry>In the epsg:4326 SRS</entry></row><row><entry>EPSG4326_CENTROID_Y</entry><entry align="center">DOUBLE</entry><entry>The Y of the geometry's centroid</entry><entry>In the epsg:4326 SRS</entry></row><row><entry>EPSG4326_AREA</entry><entry align="center">DOUBLE</entry><entry>The area of the geometry</entry><entry>In the epsg:4326 SRS (measure unknown, to be clarified)</entry></row><row><entry>IS_CLOSED</entry><entry align="center">BOOLEAN</entry><entry>Whether or not this geometry is "closed"</entry><entry>See the <ulink url="http://www.vividsolutions.com/jts/bin/JTS%20Technical%20Specs.pdf">JTS documentation</ulink> (chapter 13)</entry></row><row><entry>IS_SIMPLE</entry><entry align="center">BOOLEAN</entry><entry>Whether or not this geometry is "simple"</entry><entry>See the <ulink url="http://www.vividsolutions.com/jts/bin/JTS%20Technical%20Specs.pdf">JTS documentation</ulink> (chapter 13)</entry></row><row><entry>IS_VALID</entry><entry align="center">BOOLEAN</entry><entry>Whether or not this geometry is "valid"</entry><entry>See the <ulink url="http://www.vividsolutions.com/jts/bin/JTS%20Technical%20Specs.pdf">JTS documentation</ulink> (chapter 13). Should always be TRUE</entry></row></tbody></tgroup></table><para>Uniqueness will be enforced on a <parameter>(DOCUMENT_URI, NODE_ID_UNITS, NODE_ID)</parameter> basis. Indeed, we
      can have at most one index entry for a given node in a given document.</para><para>Also, indexes are created on these fields to help queries :</para><itemizedlist><listitem><para>DOCUMENT_URI</para></listitem><listitem><para>NODE_ID</para></listitem><listitem><para>GEOMETRY_TYPE</para></listitem><listitem><para>WKB</para></listitem><listitem><para>EPSG4326_WKB</para></listitem><listitem><para>EPSG4326_MINX</para></listitem><listitem><para>EPSG4326_MAXX</para></listitem><listitem><para>EPSG4326_MINY</para></listitem><listitem><para>EPSG4326_MAXY</para></listitem><listitem><para>EPSG4326_CENTROID_X</para></listitem><listitem><para>EPSG4326_CENTROID_Y</para></listitem></itemizedlist><para>Every geometry will be internally stored in <emphasis>both</emphasis> its original SRS and in the epsg:4326 SRS. 
      Having this kind of common, world-wide applicable, SRS for <emphasis>all</emphasis> geometries in the index allows to make operations on them even if they are originally defined in different SRSes.</para><important><para>By default, eXist's build will download the lightweight <filename>gt2-epsg-wkt-XXX.jar</filename> library
	which lacks some parameters, the <ulink url="http://udig.refractions.net/docs/api-geotools/org/geotools/referencing/datum/BursaWolfParameters.html">
	Bursa-Wolf</ulink> ones. A better accuracy for geographic transformations might be obtained by using
	a heavier library like <ulink url="http://lists.refractions.net/m2/org/geotools/gt2-epsg-hsql/"><filename>gt2-epsg-hsql-XXX.jar</filename></ulink>
	which is documented <ulink url="http://javadoc.geotools.fr/snapshot/org/geotools/referencing/factory/epsg/FactoryOnHSQL.html">here</ulink>.</para></important></section><section><title>Writing the concrete implementation of <classname>org.exist.indexing.IndexWorker</classname></title><para>Just like for <classname>org.exist.indexing.spatial.AbstractGMLJDBCIndex</classname>, we will start
      to design a database-independant abstract class. This class should normally be the basis of every JDBC spatial
      index. It will handle most of the hard work.</para><para>Let's start by a few members and a few general-purpose public methods :</para><programlisting format="linespecific">	
      package org.exist.indexing.spatial;
      
      public abstract class AbstractGMLJDBCIndexWorker implements IndexWorker {
      
        public static String GML_NS = "http://www.opengis.net/gml";    
        protected final static String INDEX_ELEMENT = "gml";   
      
	private static final Logger LOG = Logger.getLogger(AbstractGMLJDBCIndexWorker.class);
      
	protected IndexController controller;
	protected AbstractGMLJDBCIndex index;
	protected DBBroker broker;
	protected int currentMode = StreamListener.UNKNOWN;    
	protected DocumentImpl currentDoc = null;  
	private boolean isDocumentGMLAware = false;
	protected Map geometries = new TreeMap();    
	NodeId currentNodeId = null;    
	Geometry streamedGeometry = null;
	boolean documentDeleted = false;
	int flushAfter = -1;
	protected GMLHandlerJTS geometryHandler = new GeometryHandler(); 
	protected GMLFilterGeometry geometryFilter = new GMLFilterGeometry(geometryHandler); 
	protected GMLFilterDocument geometryDocument = new GMLFilterDocument(geometryFilter);
	protected TreeMap transformations = new TreeMap();
	protected boolean useLenientMode = false;   
	protected GMLStreamListener gmlStreamListener = new GMLStreamListener(); 
	protected GeometryCoordinateSequenceTransformer coordinateTransformer = new GeometryCoordinateSequenceTransformer();
	protected GeometryTransformer gmlTransformer = new GeometryTransformer();		
	protected WKBWriter wkbWriter = new WKBWriter();
	protected WKBReader wkbReader = new WKBReader();
	protected WKTWriter wktWriter = new WKTWriter();
	protected WKTReader wktReader = new WKTReader();
	protected Base64Encoder base64Encoder = new Base64Encoder();
	protected Base64Decoder base64Decoder = new Base64Decoder();  
      
	public AbstractGMLJDBCIndexWorker(AbstractGMLJDBCIndex index, DBBroker broker) {
	  this.index = index;
	  this.broker = broker;
	}
      
	public String getIndexId() {
	  return AbstractGMLJDBCIndex.ID;
	}        
      
	public String getIndexName() {
	  return index.getIndexName();
	}
      
	public Index getIndex() {
	  return index;
        }
      
      }			
      </programlisting><para>Of course, <classname>org.exist.indexing.spatial.AbstractGMLJDBCIndexWorker</classname> implements
      <classname>org.exist.indexing.IndexWorker</classname>.</para><para><parameter>GML_NS</parameter> is the GML namespace for which the spatial index is specially designed. 
      Use this public member to avoid redundancy and, worse, inconsistencies.</para><para><parameter>INDEX_ELEMENT</parameter> is the configuration's element name which is accurate for our <classname>Index</classname>
      configuration. To configure a collection in order to index its GML data, define such a configuration : </para><programlisting format="linespecific">				
      &lt;collection xmlns="http://exist-db.org/collection-config/1.0"&gt;
        &lt;index&gt;
          &lt;gml flushAfter="200"/&gt;
        &lt;/index&gt;
      &lt;/collection&gt;		
      </programlisting><para>Got the <sgmltag>gml</sgmltag> element? We will shortly see how this information is able
      to configure our <classname>IndexWorker</classname>.</para><para><parameter>controller</parameter>, <parameter>index</parameter> and <parameter>broker</parameter>
      should now be quite straightforward.</para><para><parameter>currentMode</parameter> and <parameter>currentDoc</parameter> should also be straightforward.</para><para><parameter>geometries</parameter> is a collection of <classname>com.vividsolutions.jts.geom.Geometry</classname> 
      instances that are currently held in memory, waiting for being "flushed" to the database. Depending of 
      <parameter>currentMode</parameter>, they're pending for insertion or removal.</para><para><parameter>currentNodeId</parameter> is used to share the ID of the node currently being processed between the different inner classes.</para><para><parameter>streamedGeometry</parameter> is the last <classname>com.vividsolutions.jts.geom.Geometry</classname> that has been generated by 
      GML parsing. It is <parameter>null</parameter> if the geometry is topologically
      not well formed. This latter case is maybe a too restrictive feature of Geotools parser which also
      throws <classname>NullPointerException</classname>s (!) if the GML is somehow not well-formed. See
      <ulink url="http://jira.codehaus.org/browse/GEOT-742">GEOT-742</ulink> for more information on this issue.</para><para><parameter>documentDeleted</parameter> is a flag indicating that the current document has been deleted and
      that we don't have to process it any more. Remember that <classname>StreamListener.REMOVE_ALL_NODES</classname> send some
      events for <emphasis>all</emphasis> nodes.</para><para><parameter>flushAfter </parameter> will hold our configuration's setting.</para><para><parameter>geometryHandler</parameter> is our GML geometries SAX handler that will convert GML
      to a <classname>com.vividsolutions.jts.geom.Geometry</classname> instance. It is included in a handler chain 
      composed of <parameter>geometryFilter</parameter> and <parameter>geometryDocument</parameter>.</para><para><parameter>transforms</parameter> will cache a list a transformations between a source and a target SRS.</para><para><parameter>useLenientMode</parameter> will be set to <parameter>true</parameter> is the transformation libraries
      that are in the CLASSPATH don't have the Bursa-Wolf parameters. Transformations will be attempted,
      but with a precision loss (see above).</para><para><parameter>gmlStreamListener</parameter> is our own implementation of <classname>org.exist.indexing.StreamListener</classname>.
      Since there is a 1:1 (or even 1:0) relationship with the <classname>IndexWorker</classname>, it will
      be implemented as an inner class and will be described below.</para><para><parameter>coordinateTransformer</parameter> will be needed during <classname>Geometry</classname> transformations to other SRSes.</para><para><parameter>gmlTransformer</parameter> will be needed during <classname>Geometry</classname> transformations to XML.</para><para><parameter>wkbWriter</parameter> and <parameter>wkbReader</parameter> will be needed during <classname>Geometry</classname> serialization
      and deserialization to and from the database.</para><para><parameter>wktWriter</parameter> and <parameter>wktReader</parameter> will be needed during <classname>Geometry</classname> WKT serialization
      and deserialization to and from the database. WKT could be dynamically generated from <classname>Geometry</classname> but we
      have chosen to store it in the HSQLDB.</para><para><parameter>base64Encoder</parameter> and <parameter>base64Decoder</parameter> will be needed to convert binary date, namely WKB,
      to XML types, namely <parameter>xs:base64Binary</parameter>.</para><para>No need to comment the methods, expect maybe <methodname>getIndexId()</methodname> that will return
      the <emphasis>static</emphasis> ID of the <classname>Index</classname>. No chance to be wrong with such a design.</para><para>The next method is a bit specific :</para><programlisting format="linespecific">				
      public Object configure(IndexController controller, NodeList configNodes, Map namespaces) 
        throws DatabaseConfigurationException {
	this.controller = controller;
	Map map = null;      
	for(int i = 0; i &lt; configNodes.getLength(); i++) {
	  Node node = configNodes.item(i);
	  if (node.getNodeType() == Node.ELEMENT_NODE &amp;&amp; INDEX_ELEMENT.equals(node.getLocalName())) { 
	    map = new TreeMap();
	    GMLIndexConfig config = new GMLIndexConfig(namespaces, (Element)node);
	    map.put(AbstractGMLJDBCIndex.ID, config);
	  }
        }
	return map;
      }	
      </programlisting><para>It is only interested in the <sgmltag>gml</sgmltag> element of the configuration. If it finds one,
      it creates a <classname>org.exist.indexing.spatial.GMLIndexConfig</classname> instance wich is a very
      simple class :</para><programlisting format="linespecific">				
      package org.exist.indexing.spatial;

      public class GMLIndexConfig {

        private static final Logger LOG = Logger.getLogger(GMLIndexConfig.class);
      
	private final static String FLUSH_AFTER = "flushAfter";	
	private int flushAfter = -1;
      
	public GMLIndexConfig(Map namespaces, Element node)
	  throws DatabaseConfigurationException {       
	  String param = ((Element)node).getAttribute(FLUSH_AFTER);
	  if (param != null &amp;&amp; !"".equals(param)) {) {
	    try {
	      flushAfter = Integer.parseInt(param);
	    } catch (NumberFormatException e) {
	      LOG.info("Invalid value for '" + FLUSH_AFTER + "'", e);
	    }
	  }	    	
        }
      
	public int getFlushAfter() {
	  return flushAfter;
        }
      }			
      </programlisting><para>... that retains the configuration attribute and provides a getter for it.</para><para>This configuration object is saved in a Map with the <classname>Index</classname> ID and 
      will be available as shown in the next method :</para><programlisting format="linespecific">				
      public void setDocument(DocumentImpl document) {  
        isDocumentGMLAware = false;
	documentDeleted= false;
	if (document != null) {
	  IndexSpec idxConf = document.getCollection().getIndexConfiguration(document.getBroker());
	  if (idxConf != null) {
	    Map collectionConfig = (Map) idxConf.getCustomIndexSpec(AbstractGMLJDBCIndex.ID);
	    if (collectionConfig != null) {
	      isDocumentGMLAware = true;
	      if (collectionConfig.get(AbstractGMLJDBCIndex.ID) != null)
	        flushAfter = ((GMLIndexConfig)collectionConfig.
	      get(AbstractGMLJDBCIndex.ID)).getFlushAfter();
	    }
          }
        }
	if (isDocumentGMLAware) {
	  currentDoc = document;	        
	} else {
	  currentDoc = null;
	  currentMode = StreamListener.UNKNOWN;    		
        }
      }			
      </programlisting><para>The objective is to determine if <parameter>document</parameter> should be indexed by the spatial <classname>Index</classname>.</para><para>For this, we look up its collection configuration and try to find a "custom" index specification for
      our <classname>Index</classname>. If one is found, our <parameter>document</parameter> will be processed by the <classname>IndexWorker</classname>.
      We also take advantage of this process to set one of our members. If <parameter>document</parameter> doesn't
      interest our <classname>IndexWorker</classname>, we reset some members to avoid having an inconsistent sticky state.</para><para>The next methods don't require any particular comment :</para><programlisting format="linespecific">		
      public void setDocument(DocumentImpl doc, int mode) {
        setDocument(doc);
	setMode(mode);
      }

      public DocumentImpl getDocument() {
        return currentDoc;
      }

      public int getMode() {
        return currentMode;
      }    			
      </programlisting><para>The next method is somehow tricky :</para><programlisting format="linespecific">				
      public StreamListener getListener() {      
        if (currentDoc == null || currentMode == StreamListener.REMOVE_ALL_NODES)
	  return null;
	return gmlStreamListener;
      }			
      </programlisting><para>It doesn't return any <classname>StreamListener</classname> in the 
      <classname>StreamListner.REMOVE_ALL_NODES</classname>. It would be totally unnecessary to listen
      at every node whereas a JDBC database will be able to delete all the document's nodes in 
      one single statement.</para><para>The next method is a place holder that needs more thinking. How to highlight a geometric information
      smartly?</para><programlisting format="linespecific">				
      public MatchListener getMatchListener(NodeProxy proxy) {
        return null;
      }			
      </programlisting><para>The next method computes the reindexing root. We will go bottom-up form the not to be modified
      until the top-most element in the GML namespace. Indeed, GML allows "nested" or "multi" geometries. 
      If a single part of such <classname>Geometry</classname> is modified, the whole geometry has to be 
      recomputed.</para><programlisting format="linespecific">				
      public StoredNode getReindexRoot(StoredNode node, NodePath path, boolean includeSelf) {
        if (!isDocumentGMLAware)
	  //Not concerned
	  return null;
	StoredNode topMost = node;
	StoredNode currentNode = node;
	for (int i = path.length() ; i &gt; 0; i--) {
	  currentNode = (StoredNode)currentNode.getParentNode();
	  if (GML_NS.equals(currentNode.getNamespaceURI()))		
	    topMost = currentNode;
        }
	return topMost;    	
      }   			
      </programlisting><para>The next method delegates the write operations :</para><programlisting format="linespecific">				
      public void flush() {
        if (!isDocumentGMLAware)
	  //Not concerned
	  return;
	//Is the job already done ?
	if (currentMode == StreamListener.REMOVE_ALL_NODES &amp;&amp; documentDeleted)
	  return;
	Connection conn = null;
	try {
	  conn = acquireConnection();
	  conn.setAutoCommit(false);
	  switch (currentMode) {
	    case StreamListener.STORE :
	      saveDocumentNodes(conn);
	      break;
	    case StreamListener.REMOVE_SOME_NODES :
	      dropDocumentNode(conn);
	      break;
	    case StreamListener.REMOVE_ALL_NODES:
	      removeDocument(conn);
	      documentDeleted = true;
	      break;
	  }
	  conn.commit();	  
	} catch (SQLException e) {
	  LOG.error("Document: " + currentDoc + " NodeID: " + currentNodeId, e);
	  try {
	    conn.rollback();
	  } catch (SQLException ee) {
	    LOG.error(ee);
	  }
	} finally {
	  try {
	    if (conn != null) {
	      conn.setAutoCommit(true);
	      releaseConnection(conn);
	    }
	  } catch (SQLException e) {
	    LOG.error(e);
	  }        		
	}
      }	
      </programlisting><para>Even though its code looks thick, it proves to be a good way to acquire (then release) a <classname>Connection</classname>
      whatever the way it is provided by the <classname>IndexWorker</classname> (see above for these aspects,
      concurrency in particular). It then	delegates the write operations to dedicated methods, which do not have to care about the <classname>Connection</classname>.
      Write operations are embedded in a transaction. Should an exception occur, it would be logged and
      swallowed: eXist doesn't like exceptions when it flushes its data.</para><para>The next method delegates node storage:</para><programlisting format="linespecific">				
      private void saveDocumentNodes(Connection conn) throws SQLException {
        if (geometries.size() == 0)
	  return;  
	try {	        
	  for (Iterator iterator = geometries.entrySet().iterator(); iterator.hasNext();) {
	    Map.Entry entry = (Map.Entry) iterator.next();
	    NodeId nodeId = (NodeId)entry.getKey();
	    SRSGeometry srsGeometry = (SRSGeometry)entry.getValue();       	
	    try {
	      saveGeometryNode(srsGeometry.getGeometry(), srsGeometry.getSRSName(), 
	      currentDoc, nodeId, conn);
	    } finally {
	      //Help the garbage collector
	      srsGeometry = null;
	    }
	  }
	} finally {
	  geometries.clear();
	}
      }			
      </programlisting><para>It will call <methodname>saveGeometryNode()</methodname> (see below) passing a container
      inner class that will not be described given its simplicity.</para><para>The next two methods are built with the same design. The first one destroys the index
      entry for the currently processed node and the second one removes the index entries for the whole
      document.</para><programlisting format="linespecific">				
      private void dropDocumentNode(Connection conn)
        throws SQLException {    	
	if (currentNodeId == null)
	  return;        
	try {         
	  boolean removed = removeDocumentNode(currentDoc, currentNodeId, conn);
	  if (!removed)
	    LOG.error("No data dropped for node " + currentNodeId.toString() + " from GML index");
	  else {
	    if (LOG.isDebugEnabled())	            
	      LOG.debug("Dropped data for node " + currentNodeId.toString() + " from GML index");
	  }
	} finally {        
	currentNodeId = null;
	}
      }

      private void removeDocument(Connection conn)
        throws SQLException {
	if (LOG.isDebugEnabled())
	  LOG.debug("Dropping GML index for document " + currentDoc.getURI());
	int nodeCount = removeDocument(currentDoc, conn);
	if (LOG.isDebugEnabled())
	  LOG.debug("Dropped " + nodeCount + " nodes from GML index");
      }	
      </programlisting><para>The next method is a mix of the designs described above. It also previously makes a check:</para><programlisting format="linespecific">	
      public void removeCollection(Collection collection, DBBroker broker) {
        boolean isCollectionGMLAware = false;
	IndexSpec idxConf = collection.getIndexConfiguration(broker);
	if (idxConf != null) {
	  Map collectionConfig = (Map) idxConf.getCustomIndexSpec(AbstractGMLJDBCIndex.ID);
	  if (collectionConfig != null) {
	    isCollectionGMLAware = (collectionConfig != null);
	  }
	}
	if (!isCollectionGMLAware)
	  return;  
      
	Connection conn = null;
	try {
	  conn = acquireConnection();
	  if (LOG.isDebugEnabled())
	    LOG.debug("Dropping GML index for collection " + collection.getURI());
	  int nodeCount = removeCollection(collection, conn);
	  if (LOG.isDebugEnabled())
	    LOG.debug("Dropped " + nodeCount + " nodes from GML index");
	} catch (SQLException e) {
	  LOG.error(e);
	} finally {
	  try {
	    if (conn != null)
	      releaseConnection(conn);
	  } catch (SQLException e) {
	    LOG.error(e);
	  }    		
	}
      }							
      </programlisting><para>Indeed, we have to check if the collection is indexable by the <classname>Index</classname>
      before trying to delete its index entries.</para><para>The next methods are built on the same design (<classname>Collection</classname> and exception
      management) and will thus not be described.</para><programlisting format="linespecific">
      public NodeSet search(DBBroker broker, NodeSet contextSet, Geometry EPSG4326_geometry, int spatialOp)
        ...    	
      }

      public Geometry getGeometryForNode(DBBroker broker, NodeProxy p, boolean getEPSG4326) throws  SpatialIndexException {
        ...
      }

      protected Geometry[] getGeometriesForNodes(DBBroker broker, NodeSet contextSet, boolean getEPSG4326) throws SpatialIndexException {
        ...
      }

      public AtomicValue getGeometricPropertyForNode(DBBroker broker, NodeProxy p, String propertyName) 
      throws  SpatialIndexException {
        ...
      }    

      public ValueSequence getGeometricPropertyForNodes(DBBroker broker, NodeSet contextSet, String propertyName) 
      throws  SpatialIndexException {
        ...
      }			
      </programlisting><para>... because all these methods delegate to the following abstract methods that will have to be 
      implemented by the DB-dependant concrete classes :</para><programlisting format="linespecific">		
      protected abstract boolean saveGeometryNode(Geometry geometry, String srsName, DocumentImpl doc, NodeId nodeId, Connection conn)
        throws SQLException;

      protected abstract boolean removeDocumentNode(DocumentImpl doc, NodeId nodeID, Connection conn)
        throws SQLException;

      protected abstract int removeDocument(DocumentImpl doc, Connection conn)
        throws SQLException;

      protected abstract int removeCollection(Collection collection, Connection conn)
        throws SQLException;
      
      protected abstract Connection acquireConnection() throws SQLException;

      protected abstract void releaseConnection(Connection conn) throws SQLException;	 
      
      protected abstract NodeSet search(DBBroker broker, NodeSet contextSet, Geometry EPSG4326_geometry, int spatialOp, Connection conn)
        throws SQLException;		 

      protected abstract Map getGeometriesForDocument(DocumentImpl doc, Connection conn)
        throws SQLException;

      protected abstract AtomicValue getGeometricPropertyForNode(DBBroker broker, NodeProxy p, Connection conn, 
      String propertyName) throws SQLException, XPathException;

      protected abstract ValueSequence getGeometricPropertyForNodes(DBBroker broker, NodeSet contextSet, Connection conn, String propertyName)
        throws SQLException, XPathException;

      protected abstract Geometry getGeometryForNode(DBBroker broker, NodeProxy p, boolean getEPSG4326, Connection conn)
        throws SQLException;

      protected abstract Geometry[] getGeometriesForNodes(DBBroker broker, NodeSet contextSet, boolean getEPSG4326, Connection conn)
        throws SQLException;

      protected abstract boolean checkIndex(DBBroker broker, Connection conn)
        throws SQLException, SpatialIndexException;	 
      
      </programlisting><para>Let's have a look however at this method that doesn't need a DB-dependant implementation :</para><programlisting format="linespecific">	
      public Occurrences[] scanIndex(DocumentSet docs) {    	
        Map occurences = new TreeMap();
	Connection conn = null;
	try { 
	  conn = acquireConnection();
          //Collect the (normalized) geometries for each document
	  for (Iterator iDoc = docs.iterator(); iDoc.hasNext();) {
	    DocumentImpl doc = (DocumentImpl)iDoc.next();
	    //TODO : check if document is GML-aware ?
	    //Aggregate the occurences between different documents
	    for (Iterator iGeom = getGeometriesForDocument(doc, conn).entrySet().iterator(); iGeom.hasNext();) {
	      Map.Entry entry = (Map.Entry) iGeom.next();
	      Geometry key = (Geometry)entry.getKey();
	      //Do we already have an occurence for this geometry ?
	      Occurrences oc = (Occurrences)occurences.get(key);
	      if (oc != null) {
	        //Yes : increment occurence count
		oc.addOccurrences(oc.getOccurrences() + 1);
		//...and reference the document
		oc.addDocument(doc);
              } else {
	        //No : create a new occurence with EPSG4326_WKT as "term"
		oc = new Occurrences((String)entry.getValue());
		//... with a count set to 1
		oc.addOccurrences(1);
		//... and reference the document
		oc.addDocument(doc);
		occurences.put(key, oc);
	      }
	    }
	  }
	} catch (SQLException e) {
	  LOG.error(e);
	  return null;
	} finally {
	  try {
	    if (conn != null)
	      releaseConnection(conn);
	  } catch (SQLException e) {
	    LOG.error(e);
	    return null;
	  } 
        }
	Occurrences[] result = new Occurrences[occurences.size()];
	occurences.values().toArray(result);
	return result;
      }						
      </programlisting><para>Same design (<classname>Collection</classname> and exception management, delegation mechanism).
      We probably will add more like this in the future.</para><para>The following methods are utility methods to stream <classname>Geometry</classname> instances 
      to XML and vice-versa.</para><programlisting format="linespecific">				
      public Geometry streamNodeToGeometry(XQueryContext context, NodeValue node)
        throws SpatialIndexException {
	try {
	  context.pushDocumentContext();
	  try {			
	    node.toSAX(context.getBroker(), geometryDocument, null);
	  } finally {
	    context.popDocumentContext();
	  }
	} catch (SAXException e) {
	  throw new SpatialIndexException(e);
	}
	return streamedGeometry;
      }	

      public Element streamGeometryToElement(Geometry geometry, String srsName, Receiver receiver)
        throws SpatialIndexException {           
	String gmlString = null;
	try {
	  gmlString = gmlTransformer.transform(geometry);
	} catch (TransformerException e) {
	  throw new SpatialIndexException(e);
	}
	try {
	  SAXParserFactory factory = SAXParserFactory.newInstance();
	  factory.setNamespaceAware(true);
	  InputSource src = new InputSource(new StringReader(gmlString));
	  SAXParser parser = factory.newSAXParser();
	  XMLReader reader = parser.getXMLReader();
	  reader.setContentHandler((ContentHandler)receiver);
	  reader.parse(src);
	  Document doc = receiver.getDocument();
	  return doc.getDocumentElement(); 
	} catch (ParserConfigurationException e) {				
	  throw new SpatialIndexException(e);
	} catch (SAXException e) {
	  throw new SpatialIndexException(e);
	} catch (IOException e) {
	  throw new SpatialIndexException(e);	
	}
      }    		
      </programlisting><para>The first one uses a <classname>org.geotools.gml.GMLFilterDocument</classname> (see below) and the second one uses
      a <classname>org.geotools.gml.producer.GeometryTransformer</classname> which needs some polishing because,
      despite it is called a transformer, it doesn't cope easily with a <classname>Handler</classname> and returns a...
      <classname>String</classname> ! See <ulink url="http://codehaus01a.managed.contegix.com/browse/GEOT-1315">GEOT-1315</ulink>.</para><para>The last method is also a utility method :</para><programlisting format="linespecific">	
      public Geometry transformGeometry(Geometry geometry, String sourceCRS, String targetCRS)
        throws SpatialIndexException {
	if ("osgb:BNG".equalsIgnoreCase(sourceCRS.trim()))
	  sourceCRS = "EPSG:27700";  	    
	if ("osgb:BNG".equalsIgnoreCase(targetCRS.trim()))
	  targetCRS = "EPSG:27700"; 
	MathTransform transform = (MathTransform)transformations.get(sourceCRS + "_" + targetCRS);
	if (transform == null) {
	  try {
	    try {        	
	      transform = CRS.findMathTransform(CRS.decode(sourceCRS), CRS.decode(targetCRS), useLenientMode);
	    } catch (OperationNotFoundException e) {
	      LOG.info(e);
	      LOG.info("Switching to lenient mode... beware of precision loss !");
	      useLenientMode = true;
	      transform = CRS.findMathTransform(CRS.decode(sourceCRS), CRS.decode(targetCRS), useLenientMode);	
	    }
	    transformations.put(sourceCRS + "_" + targetCRS, transform);
	    LOG.debug("Instantiated transformation from '" + sourceCRS + "' to '" + targetCRS + "'");
	  } catch (NoSuchAuthorityCodeException e) {
	    LOG.error(e);
	  } catch (FactoryException e) {
	    LOG.error(e);
	  }
	}
	if (transform == null) {
	  throw new SpatialIndexException("Unable to get a transformation from '" + sourceCRS + "' to '" + targetCRS +"'");        		           	
	}
	coordinateTransformer.setMathTransform(transform);
	try {
	  return coordinateTransformer.transform(geometry);
	} catch (TransformException e) {
	  throw new SpatialIndexException(e);
        }
      }					
      </programlisting><para>It implements a workaround for our test file SRS which isn't yet known by Geotools libraries (see
      <ulink url="http://codehaus01a.managed.contegix.com/browse/GEOT-1307">GEOT-1307</ulink>),
      then it tries to get the transformation from our cache. If it doesn't succeed, it tries to find one
      in the libraries that are in the CLASSPATH. Should those libraries lack the Bursa-Wolf parameters,
      it will make another attempt in lenient mode, which will induce a loss of accuracy. Then, it transforms
      the <classname>Geometry</classname> from its <parameter>sourceCRS</parameter> to the required
      <parameter>targetCRS</parameter>.</para><para>Now, let's study how the abstract methods are implement by the HSQLDB-dependant class :</para><programlisting format="linespecific">	
      package org.exist.indexing.spatial;

      public class GMLHSQLIndexWorker extends AbstractGMLJDBCIndexWorker {

        private static final Logger LOG = Logger.getLogger(GMLHSQLIndexWorker.class);
      
	public GMLHSQLIndexWorker(GMLHSQLIndex index, DBBroker broker) {
	  super(index, broker);
	}
      }						
      </programlisting><para>The only noticeable point is that we indeed extend our <classname>org.exist.indexing.spatial.AbstractGMLJDBCIndexWorker</classname></para><para>Now, this method will do something more interesting, store the <classname>Geometry</classname>
      associated to a node :</para><programlisting format="linespecific">				
      protected boolean saveGeometryNode(Geometry geometry, String srsName, DocumentImpl doc, NodeId nodeId, Connection conn)
        throws SQLException {
	PreparedStatement ps = conn.prepareStatement("INSERT INTO " + GMLHSQLIndex.TABLE_NAME + "(" +
	/*1*/ "DOCUMENT_URI, " +            		
	/*2*/ "NODE_ID_UNITS, " + 
	/*3*/ "NODE_ID, " +        			
	/*4*/ "GEOMETRY_TYPE, " +
	/*5*/ "SRS_NAME, " +
	/*6*/ "WKT, " +
	/*7*/ "WKB, " +
	/*8*/ "MINX, " +
	/*9*/ "MAXX, " +
	/*10*/ "MINY, " +
	/*11*/ "MAXY, " +
	/*12*/ "CENTROID_X, " +
	/*13*/ "CENTROID_Y, " +
	/*14*/ "AREA, " +
	//Boundary ?        		
	/*15*/ "EPSG4326_WKT, " +
	/*16*/ "EPSG4326_WKB, " +
	/*17*/ "EPSG4326_MINX, " +
	/*18*/ "EPSG4326_MAXX, " +
	/*19*/ "EPSG4326_MINY, " +
	/*20*/ "EPSG4326_MAXY, " +
	/*21*/ "EPSG4326_CENTROID_X, " +
	/*22*/ "EPSG4326_CENTROID_Y, " +
	/*23*/ "EPSG4326_AREA," +
	//Boundary ?
	/*24*/ "IS_CLOSED, " +
	/*25*/ "IS_SIMPLE, " +
	/*26*/ "IS_VALID" +    			
	") VALUES (" +
	"?, ?, ?, ?, ?, " +
	"?, ?, ?, ?, ?, " +
	"?, ?, ?, ?, ?, " +
	"?, ?, ?, ?, ?, " +
	"?, ?, ?, ?, ?, " +
	"?"
	+ ")"
	);       
	try {
	  Geometry EPSG4326_geometry = null;
	  try {
	    EPSG4326_geometry = transformGeometry(geometry, srsName, "EPSG:4326");
	  } catch (SpatialIndexException e) {
	    //Transforms the exception into an SQLException.
	    SQLException ee = new SQLException(e.getMessage());
	    ee.initCause(e);
	    throw ee;
	  }
	  /*DOCUMENT_URI*/ ps.setString(1, doc.getURI().toString());	
	  /*NODE_ID_UNITS*/ ps.setInt(2, nodeId.units());
	  byte[] bytes = new byte[nodeId.size()];
	  nodeId.serialize(bytes, 0);
	  /*NODE_ID*/ ps.setBytes(3, bytes);
	  /*GEOMETRY_TYPE*/ ps.setString(4, geometry.getGeometryType());
	  /*SRS_NAME*/ ps.setString(5, srsName);
	  /*WKT*/ ps.setString(6, wktWriter.write(geometry));
	  /*WKB*/ ps.setBytes(7, wkbWriter.write(geometry));
	  /*MINX*/ ps.setDouble(8, geometry.getEnvelopeInternal().getMinX());
	  /*MAXX*/ ps.setDouble(9, geometry.getEnvelopeInternal().getMaxX());
	  /*MINY*/ ps.setDouble(10, geometry.getEnvelopeInternal().getMinY());
	  /*MAXY*/ ps.setDouble(11, geometry.getEnvelopeInternal().getMaxY());
	  /*CENTROID_X*/ ps.setDouble(12, geometry.getCentroid().getCoordinate().x);   
	  /*CENTROID_Y*/ ps.setDouble(13, geometry.getCentroid().getCoordinate().y);  		
	  /*AREA*/ ps.setDouble(14, geometry.getArea());		
	  /*EPSG4326_WKT*/ ps.setString(15, wktWriter.write(EPSG4326_geometry));
	  /*EPSG4326_WKB*/ ps.setBytes(16, wkbWriter.write(EPSG4326_geometry));		
	  /*EPSG4326_MINX*/ ps.setDouble(17, EPSG4326_geometry.getEnvelopeInternal().getMinX());
	  /*EPSG4326_MAXX*/ ps.setDouble(18, EPSG4326_geometry.getEnvelopeInternal().getMaxX());
	  /*EPSG4326_MINY*/ ps.setDouble(19, EPSG4326_geometry.getEnvelopeInternal().getMinY());
	  /*EPSG4326_MAXY*/ ps.setDouble(20, EPSG4326_geometry.getEnvelopeInternal().getMaxY());
	  /*EPSG4326_CENTROID_X*/ ps.setDouble(21, EPSG4326_geometry.getCentroid().getCoordinate().x);   
	  /*EPSG4326_CENTROID_Y*/ ps.setDouble(22, EPSG4326_geometry.getCentroid().getCoordinate().y);
	  //EPSG4326_geometry.getRepresentativePoint()
	  /*EPSG4326_AREA*/ ps.setDouble(23, EPSG4326_geometry.getArea());
	  //For empty Curves, isClosed is defined to have the value false.
	  /*IS_CLOSED*/ ps.setBoolean(24, !geometry.isEmpty());
	  /*IS_SIMPLE*/ ps.setBoolean(25, geometry.isSimple());
	  //Should always be true (the GML SAX parser makes a too severe check)
	  /*IS_VALID*/ ps.setBoolean(26, geometry.isValid());
	  return (ps.executeUpdate() == 1);
	} finally {
	if (ps != null)
	  ps.close();
	//Let's help the garbage collector...
	geometry = null;
	}    	
      }			
      </programlisting><para>The generated SQL statement should be straightforward. We make a heavy use of the methods
      provided by <classname>com.vividsolutions.jts.geom.Geometry</classname>, both on the "native" <classname>Geometry</classname>
      and on its EPSG:4326 transformation. Would could probably store other properties here (like, e.g. the 
      geometry's boundary). Other <classname>IndexWorker</classname>s, especially those accessing a spatially-enabled
      DBMS, might prefer to store fewer properties if they can be computed dynamically at a cheap price.</para><para>The next method is even much easier to understand :</para><programlisting format="linespecific">				
      protected boolean removeDocumentNode(DocumentImpl doc, NodeId nodeId, Connection conn)
        throws SQLException {
	PreparedStatement ps = conn.prepareStatement(
	"DELETE FROM " + GMLHSQLIndex.TABLE_NAME + 
	" WHERE DOCUMENT_URI = ? AND NODE_ID_UNITS = ? AND NODE_ID = ?;"
	); 
	ps.setString(1, doc.getURI().toString());
	ps.setInt(2, nodeId.units());
	byte[] bytes = new byte[nodeId.size()];
	nodeId.serialize(bytes, 0);        
	ps.setBytes(3, bytes);
	try {	 
	  return (ps.executeUpdate() == 1);
	} finally {
	  if (ps != null)
	    ps.close();
        }
      }
      </programlisting><para>... and this one even more :</para><programlisting format="linespecific">				
      protected int removeDocument(DocumentImpl doc, Connection conn)
        throws SQLException {    	
	PreparedStatement ps = conn.prepareStatement(
	"DELETE FROM " + GMLHSQLIndex.TABLE_NAME + " WHERE DOCUMENT_URI = ?;"
	); 
	ps.setString(1, doc.getURI().toString());
	try {
	  return ps.executeUpdate();	 
	} finally {
	  if (ps != null)
	    ps.close();
	}
      }  			
      </programlisting><para>This one however, is a little bit trickier :</para><programlisting format="linespecific">		
      protected int removeCollection(Collection collection, Connection conn)
        throws SQLException {
	PreparedStatement ps = conn.prepareStatement(
	"DELETE FROM " + GMLHSQLIndex.TABLE_NAME + " WHERE SUBSTRING(DOCUMENT_URI, 1, ?) = ?;"
	); 
	ps.setInt(1, collection.getURI().toString().length());
	ps.setString(2, collection.getURI().toString());
	try {
	  return ps.executeUpdate();
	} finally {
	  if (ps != null)
	  ps.close();
	}
      }	
      </programlisting><para>... maybe because it makes use of a SQL function to filter the right documents ?</para><para>The two next methods are straightforward, now that we have explained that <classname>Connection</classname>s
      had to be requested from the <classname>Index</classname> to avoid concurrency problems on an embedded
      HSQLDB instance.</para><programlisting format="linespecific">	
      protected Connection acquireConnection()
        throws SQLException {   
	return index.acquireConnection(this.broker);
      }

      protected void releaseConnection(Connection conn)
        throws SQLException {   
	index.releaseConnection(this.broker);
      }
      </programlisting><para>The next method is much more interesting. This is where is the core of the spatial index is:</para><programlisting format="linespecific">				
      protected NodeSet search(DBBroker broker, NodeSet contextSet, Geometry EPSG4326_geometry, int spatialOp, Connection conn)
        throws SQLException {
	String extraSelection = null;
	String bboxConstraint = null;    	
	switch (spatialOp) {
	  //BBoxes are equal
	  case SpatialOperator.EQUALS:
	    bboxConstraint = "(EPSG4326_MINX = ? AND EPSG4326_MAXX = ?)" +
	    " AND (EPSG4326_MINY = ? AND EPSG4326_MAXY = ?)";
	    break;
	  //Nothing much we can do with the BBox at this stage
	  case SpatialOperator.DISJOINT:
	    //Retrieve the BBox though...
	    extraSelection = ", EPSG4326_MINX, EPSG4326_MAXX, EPSG4326_MINY, EPSG4326_MAXY";
	    break;
	  //BBoxes intersect themselves
	  case SpatialOperator.INTERSECTS:        		
	  case SpatialOperator.TOUCHES:        		   		
	  case SpatialOperator.CROSSES:        		      		
	  case SpatialOperator.OVERLAPS: 
	    bboxConstraint = "(EPSG4326_MAXX &gt;= ? AND EPSG4326_MINX &lt;= ?)" +
	    " AND (EPSG4326_MAXY &gt;= ? AND EPSG4326_MINY &lt;= ?)";
	    break;
	  //BBox is fully within
	  case SpatialOperator.WITHIN:   
	    bboxConstraint = "(EPSG4326_MINX &gt;= ? AND EPSG4326_MAXX &lt;= ?)" +
	    " AND (EPSG4326_MINY &gt;= ? AND EPSG4326_MAXY &lt;= ?)";
	    break;
	  //BBox fully contains
	  case SpatialOperator.CONTAINS: 
	    bboxConstraint = "(EPSG4326_MINX &lt;= ? AND EPSG4326_MAXX &gt;= ?)" +
	    " AND (EPSG4326_MINY &lt;= ? AND EPSG4326_MAXY &gt;= ?)";
	    break;       		
	  default:
	    throw new IllegalArgumentException("Unsupported spatial operator:" + spatialOp);
        }
	PreparedStatement ps = conn.prepareStatement(
	"SELECT EPSG4326_WKB, DOCUMENT_URI, NODE_ID_UNITS, NODE_ID" + (extraSelection == null ? "" : extraSelection) +
	" FROM " + GMLHSQLIndex.TABLE_NAME + 
	(bboxConstraint == null ? "" : " WHERE " + bboxConstraint) + ";"
	);
	if (bboxConstraint != null) {
	  ps.setDouble(1, EPSG4326_geometry.getEnvelopeInternal().getMinX());
	  ps.setDouble(2, EPSG4326_geometry.getEnvelopeInternal().getMaxX());
	  ps.setDouble(3, EPSG4326_geometry.getEnvelopeInternal().getMinY());
	  ps.setDouble(4, EPSG4326_geometry.getEnvelopeInternal().getMaxY());
	}
	ResultSet rs = null;
	NodeSet result = null;
	try { 
	  int disjointPostFiltered = 0;
	  rs = ps.executeQuery();
	  result = new ExtArrayNodeSet(); //new ExtArrayNodeSet(docs.getLength(), 250)
	  while (rs.next()) {
	    DocumentImpl doc = null;
	    try {
	      doc = (DocumentImpl)broker.getXMLResource(XmldbURI.create(rs.getString("DOCUMENT_URI")));        			
	    } catch (PermissionDeniedException e) {
	      LOG.debug(e);
	      //Ignore since the broker has no right on the document
	      continue;
	    }
	    //contextSet == null should be use to scan the whole index
	    if (contextSet == null || contextSet.getDocumentSet().contains(doc.getDocId())) {
	      NodeId nodeId = new DLN(rs.getInt("NODE_ID_UNITS"), rs.getBytes("NODE_ID"), 0); 
	      NodeProxy p = new NodeProxy((DocumentImpl)doc, nodeId);
	      if (contextSet == null || contextSet.get(p) != null) {
	        boolean geometryMatches = false;
		if (spatialOp == SpatialOperator.DISJOINT) {
		  //No BBox intersection : obviously disjoint
		  if (rs.getDouble("EPSG4326_MAXX") &lt; EPSG4326_geometry.getEnvelopeInternal().getMinX() ||	        			
		  rs.getDouble("EPSG4326_MINX") &gt; EPSG4326_geometry.getEnvelopeInternal().getMaxX() ||	        			
		  rs.getDouble("EPSG4326_MAXY") &lt; EPSG4326_geometry.getEnvelopeInternal().getMinY() ||	        			
		  rs.getDouble("EPSG4326_MINY") &gt; EPSG4326_geometry.getEnvelopeInternal().getMaxY()) {
		    geometryMatches = true;
		    disjointPostFiltered++;
		  }
		}
		//Possible match : check the geometry
		if (!geometryMatches) {	
		  try {	        	
		    Geometry geometry = wkbReader.read(rs.getBytes("EPSG4326_WKB"));
		    switch (spatialOp) {
		      case SpatialOperator.EQUALS:
		        geometryMatches = geometry.equals(EPSG4326_geometry);
			break;
		      case SpatialOperator.DISJOINT:        		
		        geometryMatches = geometry.disjoint(EPSG4326_geometry);
		        break;  		
		      case SpatialOperator.INTERSECTS:        		
		        geometryMatches = geometry.intersects(EPSG4326_geometry);
		        break;
		      case SpatialOperator.TOUCHES:
		        geometryMatches = geometry.touches(EPSG4326_geometry);
		        break; 		
		      case SpatialOperator.CROSSES:
		        geometryMatches = geometry.crosses(EPSG4326_geometry);
		        break;
		      case SpatialOperator.WITHIN:        		
		        geometryMatches = geometry.within(EPSG4326_geometry);
		        break; 		
		      case SpatialOperator.CONTAINS:	        		
		        geometryMatches = geometry.contains(EPSG4326_geometry);
		        break;
		      case SpatialOperator.OVERLAPS:	        		
		        geometryMatches = geometry.overlaps(EPSG4326_geometry);
		        break;
		    }
		  } catch (ParseException e) {
		    //Transforms the exception into an SQLException.
		    //Very unlikely to happen though...
		    SQLException ee = new SQLException(e.getMessage());
		    ee.initCause(e);
		    throw ee;
		  }
		}
		if (geometryMatches)        	
		  result.add(p);
	      }
	    }
	  }
	  if (LOG.isDebugEnabled()) {
	    LOG.debug(rs.getRow() + " eligible geometries, " + result.getItemCount() + "selected" +
	    (spatialOp == SpatialOperator.DISJOINT ? "(" + disjointPostFiltered + " post filtered)" : ""));
	    }
	    return result;	    	
        } finally { 
          if (rs != null)
	    rs.close();
	  if (ps != null)
	    ps.close();	    		
        }
      }		
      </programlisting><para>The trick is to filter the geometries on (fast) BBox operations first (intersecting geometries have BBox intersecting as well) which is possible in every case but for the <methodname>Spatial.DISJOINT</methodname>
      operator. For the latter case, we will have to fetch the BBox coordinates in order to apply a further
      filtering. Then, we examine the results and filter out the documents that are not in the <parameter>contextSet</parameter>.
      <methodname>Spatial.DISJOINT</methodname> filtering is then applied to avoid the next step in case the BBoxes are themselves disjoint. Only then, 
      we perform the costly operations, namely <classname>Geometry</classname> deserialization from the DB 
      then performing spatial operations on it. Matching nodes are then returned.</para><para>The next method is quite straightforward:</para><programlisting format="linespecific">		
      protected Map getGeometriesForDocument(DocumentImpl doc, Connection conn)
        throws SQLException {
	PreparedStatement ps = conn.prepareStatement(
	"SELECT EPSG4326_WKB, EPSG4326_WKT FROM " + GMLHSQLIndex.TABLE_NAME + " WHERE DOCUMENT_URI = ?;"
	); 
	ps.setString(1, doc.getURI().toString());
	ResultSet rs = null;
	try {
	  rs = ps.executeQuery();
	  Map map = new TreeMap();
	  while (rs.next()) {
	    Geometry EPSG4326_geometry = wkbReader.read(rs.getBytes("EPSG4326_WKB"));
	    //Returns the EPSG:4326 WKT for every geometry to make occurrence aggregation consistent
	    map.put(EPSG4326_geometry, rs.getString("EPSG4326_WKT"));
	  }
	  return map;
	} catch (ParseException e) {
	  //Transforms the exception into an SQLException.
	  //Very unlikely to happen though...
	  SQLException ee = new SQLException(e.getMessage());
	  ee.initCause(e);
	  throw ee;
	} finally {   
	  if (rs != null)
	    rs.close();
	  if (ps != null)
	    ps.close();
	}
      }
      </programlisting><para>Notice that it will return EPSG:4326 <classname>Geometry</classname>ies and that it will rethrow
      a <classname>com.vividsolutions.jts.io.ParseException</classname> as a <classname>java.sql.SQLException</classname>.</para><para>The next method is a bit more restrictive and modular :</para><programlisting format="linespecific">
      protected Geometry getGeometryForNode(DBBroker broker, NodeProxy p, boolean getEPSG4326, Connection conn)
        throws SQLException {
	PreparedStatement ps = conn.prepareStatement(
	"SELECT " + (getEPSG4326 ? "EPSG4326_WKB" : "WKB") +
	" FROM " + GMLHSQLIndex.TABLE_NAME + 
	" WHERE DOCUMENT_URI = ? AND NODE_ID_UNITS = ? AND NODE_ID = ?;"
	);
	ps.setString(1, p.getDocument().getURI().toString());
	ps.setInt(2, p.getNodeId().units());
	byte[] bytes = new byte[p.getNodeId().size()];
	p.getNodeId().serialize(bytes, 0);
	ps.setBytes(3, bytes);   
	ResultSet rs = null;    	
	try {
	  rs = ps.executeQuery();
	  if (!rs.next())
	    //Nothing returned
	    return null;    		
	  Geometry geometry = wkbReader.read(rs.getBytes(1));        			
	  if (rs.next()) {   	
	    //Should be impossible    		
	    throw new SQLException("More than one geometry for node " + p);
	  }
	  return geometry;    
	} catch (ParseException e) {
	  //Transforms the exception into an SQLException.
	  //Very unlikely to happen though...
	  SQLException ee = new SQLException(e.getMessage());
	  ee.initCause(e);
	  throw ee;
	} finally {
	  if (rs != null)
	    rs.close();
	  if (ps != null)
	    ps.close();
        }
      }
      </programlisting><para>... because if directly selects the right node and allows to return either the original <classname>Geometry</classname>, either
      its EPSG:4326 transformation.</para><para>The next method is a generalization of the previous one:</para><programlisting format="linespecific">
      protected Geometry[] getGeometriesForNodes(DBBroker broker, NodeSet contextSet, boolean getEPSG4326, Connection conn)
        throws SQLException {
	PreparedStatement ps = conn.prepareStatement(
	"SELECT " + (getEPSG4326 ? "EPSG4326_WKB" : "WKB") + ", DOCUMENT_URI, NODE_ID_UNITS, NODE_ID" +
	" FROM " + GMLHSQLIndex.TABLE_NAME 
	);
	ResultSet rs = null;    	
	try {
	  rs = ps.executeQuery();
	  Geometry[] result = new Geometry[contextSet.getLength()];
	  int index= 0;
	  while (rs.next()) {
	    DocumentImpl doc = null;
	    try {
	      doc = (DocumentImpl)broker.getXMLResource(XmldbURI.create(rs.getString("DOCUMENT_URI")));        			
	    } catch (PermissionDeniedException e) {
	      LOG.debug(e);
	      result[index++] = null;
	      //Ignore since the broker has no right on the document
	      continue;
	    }
	    if (contextSet.getDocumentSet().contains(doc.getDocId())) {
	      NodeId nodeId = new DLN(rs.getInt("NODE_ID_UNITS"), rs.getBytes("NODE_ID"), 0); 
	      NodeProxy p = new NodeProxy((DocumentImpl)doc, nodeId);
	      if (contextSet.get(p) != null) {
	        Geometry geometry = wkbReader.read(rs.getBytes(1));
		result[index++] = geometry;
	      }
            }
          }
	  return result;
	} catch (ParseException e) {
        //Transforms the exception into an SQLException.
	//Very unlikely to happen though...
	SQLException ee = new SQLException(e.getMessage());
	ee.initCause(e);
	throw ee;
        } finally {   
          if (rs != null)
	    rs.close();
	  if (ps != null)
	    ps.close();
	}
      }
      </programlisting><para>It queries the whole index for the requested <classname>Geometry</classname>, ignoring the documents that are not in 
      the <parameter>contextSet</parameter>, and it also ignores the nodes that are not in the <parameter>contextSet</parameter>.
      After that the <classname>Geometry</classname> is deserialized.</para><note><para>This method is not yet used by the spatial functions but it is planned to use it in a future
      optimization effort.</para></note><para>This is the next method, designed like <methodname>getGeometryForNode()</methodname>:</para><programlisting format="linespecific">	
      protected AtomicValue getGeometricPropertyForNode(DBBroker broker, NodeProxy p, Connection conn, String propertyName)
        throws SQLException, XPathException {
	PreparedStatement ps = conn.prepareStatement(
	"SELECT " + propertyName + 
	" FROM " + GMLHSQLIndex.TABLE_NAME + 
	" WHERE DOCUMENT_URI = ? AND NODE_ID_UNITS = ? AND NODE_ID = ?"
	);
	ps.setString(1, p.getDocument().getURI().toString());
	ps.setInt(2, p.getNodeId().units());
	byte[] bytes = new byte[p.getNodeId().size()];
	p.getNodeId().serialize(bytes, 0);
	ps.setBytes(3, bytes);    	
	ResultSet rs = null;    	
	try {
	  rs = ps.executeQuery();
	  if (!rs.next())
	    //Nothing returned
	    return AtomicValue.EMPTY_VALUE;
          AtomicValue result = null;
	  if (rs.getMetaData().getColumnClassName(1).equals(Boolean.class.getName())) {
	    result = new BooleanValue(rs.getBoolean(1));
	  } else if (rs.getMetaData().getColumnClassName(1).equals(Double.class.getName())) {
	    result = new DoubleValue(rs.getDouble(1));
	  } else if (rs.getMetaData().getColumnClassName(1).equals(String.class.getName())) {
	    result = new StringValue(rs.getString(1));
	  } else if (rs.getMetaData().getColumnType(1) == java.sql.Types.BINARY) {
	    result = new Base64Binary(rs.getBytes(1));
	  } else 
	    throw new SQLException("Unable to make an atomic value from '" + rs.getMetaData().getColumnClassName(1) + "'");		
	  if (rs.next()) {   	
	    //Should be impossible    		
	    throw new SQLException("More than one geometry for node " + p);
	  }
	  return result;    
	} finally {   
	  if (rs != null)
	    rs.close();
	  if (ps != null)
	    ps.close();
	}
      }
      </programlisting><para>It directly requests the required property from the DB and returns an appropriate XML atomic value.</para><para>The next method is a generalization of the previous one :</para><programlisting format="linespecific">			
      protected ValueSequence getGeometricPropertyForNodes(DBBroker broker, NodeSet contextSet, Connection conn, String propertyName)
        throws SQLException, XPathException {
	PreparedStatement ps = conn.prepareStatement(
	"SELECT " + propertyName + ", DOCUMENT_URI, NODE_ID_UNITS, NODE_ID" + 
	" FROM " + GMLHSQLIndex.TABLE_NAME
	);
	ResultSet rs = null;    	
	try {
	  rs = ps.executeQuery();
	  ValueSequence result = new ValueSequence(contextSet.getLength());    		
	  while (rs.next()) {
	    DocumentImpl doc = null;
	    try {
	      doc = (DocumentImpl)broker.getXMLResource(XmldbURI.create(rs.getString("DOCUMENT_URI")));        			
	    } catch (PermissionDeniedException e) {
	      LOG.debug(e);				
	      if (rs.getMetaData().getColumnClassName(1).equals(Boolean.class.getName())) {
	        result.add(BooleanValue.EMPTY_VALUE);
	      } else if (rs.getMetaData().getColumnClassName(1).equals(Double.class.getName())) {
	        result.add(DoubleValue.EMPTY_VALUE);
	      } else if (rs.getMetaData().getColumnClassName(1).equals(String.class.getName())) {
	        result.add(StringValue.EMPTY_VALUE);
	      } else if (rs.getMetaData().getColumnType(1) == java.sql.Types.BINARY) {
	        result.add(Base64Binary.EMPTY_VALUE);
              } else 
	        throw new SQLException("Unable to make an atomic value from '" + rs.getMetaData().getColumnClassName(1) + "'");
	      //Ignore since the broker has no right on the document
	      continue;
	    }
	    if (contextSet.getDocumentSet().contains(doc.getDocId())) {
	      NodeId nodeId = new DLN(rs.getInt("NODE_ID_UNITS"), rs.getBytes("NODE_ID"), 0); 
	      NodeProxy p = new NodeProxy((DocumentImpl)doc, nodeId);
	      if (contextSet.get(p) != null) {
	        if (rs.getMetaData().getColumnClassName(1).equals(Boolean.class.getName())) {
		  result.add(new BooleanValue(rs.getBoolean(1)));
	        } else if (rs.getMetaData().getColumnClassName(1).equals(Double.class.getName())) {
	          result.add(new DoubleValue(rs.getDouble(1)));
	        } else if (rs.getMetaData().getColumnClassName(1).equals(String.class.getName())) {
	          result.add(new StringValue(rs.getString(1)));
	        } else if (rs.getMetaData().getColumnType(1) == java.sql.Types.BINARY) {
	          result.add(new Base64Binary(rs.getBytes(1)));
                } else 
	          throw new SQLException("Unable to make an atomic value from '" + rs.getMetaData().getColumnClassName(1) + "'");
	    }
	  }
        }
        return result;    
        } finally {   
          if (rs != null)
	    rs.close();
	  if (ps != null)
	    ps.close();
        }
      }
      </programlisting><para>It queries the whole index for the requested property, ignoring the documents that are not in 
      the <parameter>contextSet</parameter>, and it also ignores the nodes that are not in the <parameter>contextSet</parameter>.
      Finally the property mapped to the appropriate XML atomic value is returned.</para><note><para>This method is not yet used by the spatial functions but it is planned to use it in a future
      optimization effort.</para></note><para>The last method is a utility method and we will only show a part of its body:</para><programlisting format="linespecific">		
      protected boolean checkIndex(DBBroker broker, Connection conn)
        throws SQLException {
	PreparedStatement ps = conn.prepareStatement(
	"SELECT * FROM " + GMLHSQLIndex.TABLE_NAME + ";"
	);
	ResultSet rs = null;
	try {
	  rs = ps.executeQuery();
	  while (rs.next()) {	        	
	    Geometry original_geometry = wkbReader.read(rs.getBytes("WKB"));
	    if (!original_geometry.equals(wktReader.read(rs.getString("WKT")))) {
	      LOG.info("Inconsistent WKT : " + rs.getString("WKT"));
	      return false;
            }		            	
	    Geometry EPSG4326_geometry = wkbReader.read(rs.getBytes("EPSG4326_WKB"));		        	
	    if (!EPSG4326_geometry.equals(wktReader.read(rs.getString("EPSG4326_WKT")))) {
	      LOG.info("Inconsistent WKT : " + rs.getString("EPSG4326_WKT"));
	      return false;
	    }
      
	    if (!original_geometry.getGeometryType().equals(rs.getString("GEOMETRY_TYPE"))) {
	      LOG.info("Inconsistent geometry type: " + rs.getDouble("GEOMETRY_TYPE"));
	      return false;
	    }
	    
	    if (original_geometry.getEnvelopeInternal().getMinX() != rs.getDouble("MINX")) {
	      LOG.info("Inconsistent MinX: " + rs.getDouble("MINX"));
	      return false;
	    }

        ...

	DocumentImpl doc = null;
	try {
	  doc = (DocumentImpl)broker.getXMLResource(XmldbURI.create(rs.getString("DOCUMENT_URI")));
	} catch (PermissionDeniedException e) {
	  //The broker has no right on the document
	  LOG.error(e);
	  return false;
        }
	NodeId nodeId = new DLN(rs.getInt("NODE_ID_UNITS"), rs.getBytes("NODE_ID"), 0); 	    		
	StoredNode node = broker.objectWith(new NodeProxy((DocumentImpl)doc, nodeId));
	if (node == null) {
	  LOG.info("Node " + nodeId + "doesn't exist");
	  return false;
	}      
	if (!GMLHSQLIndexWorker.GML_NS.equals(node.getNamespaceURI())) {
	  LOG.info("GML indexed node (" + node.getNodeId()+ ") is in the '" + 
	  node.getNamespaceURI() + "' namespace. '" + 
	  GMLHSQLIndexWorker.GML_NS + "' was expected !");
	  return false;
	}
	if (!original_geometry.getGeometryType().equals(node.getLocalName())) {
	  if ("Box".equals(node.getLocalName()) &amp;&amp; "Polygon".equals(original_geometry.getGeometryType())) {
	    LOG.debug("GML indexed node (" + node.getNodeId() + ") is a gml:Box indexed as a polygon");
	  } else {
	    LOG.info("GML indexed node (" + node.getNodeId() + ") has '" + 
	    node.getLocalName() + "' as its local name. '" + 
	    original_geometry.getGeometryType() + "' was expected !");
	    return false;
	  }
        }
      
	LOG.info(node);	        		
	}
	return true;
      
        } catch (ParseException e) {
          //Transforms the exception into an SQLException.
	  //Very unlikely to happen though...
	  SQL Exception ee = new SQLException(e.getMessage());
	  ee.initCause(e);
	  throw ee;
	} finally {   
	  if (rs != null)
            rs.close();
          if (ps != null)
            ps.close();	
        }
      }
      </programlisting><para>It deserializes each <classname>Geometry</classname> and checks that its data are consistent with what is stored in the DB.</para></section><section><title>Writing a concrete implementation of <classname>org.exist.indexing.StreamListener</classname></title><para>The <classname>StreamListener</classname>'s main purpose is to
      generate <classname>Geometry</classname> instances, if accurate, from the nodes it listens to.</para><para>This will be done using a <classname>org.geotools.gml.GMLFilterDocument</classname> provided by the Geotools
      libraries. The trick is to map our STAX events to the expected SAX events.</para><para>As stated above, our <classname>StreamListener</classname> will be implemented as an inner class of
      <classname>org.exist.indexing.spatial.AbstractGMLJDBCIndexWorker</classname>. Of course, it will extend
      <classname>org.exist.indexing.AbstractStreamListener</classname>:</para><programlisting format="linespecific">	
      private class GMLStreamListener extends AbstractStreamListener {
      
        Stack srsNamesStack = new Stack();
	ElementImpl deferredElement;
      
	public IndexWorker getWorker() {
	  return AbstractGMLJDBCIndexWorker.this;
	}
      }
      </programlisting><para>There are only two members. <parameter>srsNamesStack</parameter> will maintain a (<classname>String</classname>)
      <classname>java.util.Stack</classname> for the <sgmltag class="attribute">srsName</sgmltag> attribute of the
      elements in the GML namespace (http://www.opengis.net/gml). <parameter>null</parameter> will be pushed
      if such an attribute doesn't exist, namely because it isn't accurate.</para><para><parameter>deferredElement</parameter> will
      hold an element whose streaming is deferred, namely because we still haven't received its attributes.</para><para>The <methodname>getWorker()</methodname> method should be straightforward.</para><para>Let's see how the process is performed:</para><programlisting format="linespecific">		
      public void startElement(Txn transaction, ElementImpl element, NodePath path) { 
        if (isDocumentGMLAware) {
	  //Release the deferred element if any
	  if (deferredElement != null)        		
	    processDeferredElement();
	  //Retain this element
	  deferredElement = element;  
	}
	//Forward the event to the next listener 
	super.startElement(transaction, element, path);
      }
      
      public void attribute(Txn transaction, AttrImpl attrib, NodePath path) { 
        //Forward the event to the next listener 
	super.attribute(transaction, attrib, path);
      }        

      public void characters(Txn transaction, TextImpl text, NodePath path) {
        if (isDocumentGMLAware) {
	  //Release the deferred element if any
	  if (deferredElement != null)        		
	    processDeferredElement();	        	
	  try {
	    geometryDocument.characters(text.getData().toCharArray(), 0, text.getLength());
	  } catch (Exception e) {
	    LOG.error(e);
	  } 
	}
	//Forward the event to the next listener 
	super.characters(transaction, text, path);
      }

      public void endElement(Txn transaction, ElementImpl element, NodePath path) {
        if (isDocumentGMLAware) {   
	  //Release the deferred element if any
	  if (deferredElement != null)        		
	    processDeferredElement();	        	
	  //Process the element 
	  processCurrentElement(element);
	}
        //Forward the event to the next listener 
	super.endElement(transaction, element, path);
      }
      </programlisting><para>Element deferring occurs only if <parameter>currentDoc</parameter> is to be indexed of course. If
      so, an incoming element is deferred but we do not forget to forward the event to the next 
      <classname>StreamListener</classname> in the pipeline.</para><para>If we have a deferred element, we will process it (see below) in order to collect its attributes and if relevant, <methodname>endElement()</methodname>, will add an index entry for the current element.  
      The method <methodname>characters()</methodname> also forwards its data to the SAX handler.</para><note><para>We could have used <methodname>attribute()</methodname> to collect the deferred element's attributes.
      The described design is just a matter of choice.</para></note><para>Let's see how the deferred element is processed :</para><programlisting format="linespecific">				
      private void processDeferredElement() {  
        //We need to collect the deferred element's attributes in order to feed the SAX handler
	AttributesImpl attList = new AttributesImpl();
	NamedNodeMap attrs = deferredElement.getAttributes();
      
	String whatToPush = null;
	
	for (int i = 0; i &lt; attrs.getLength() ; i++) {
	  AttrImpl attrib = (AttrImpl)attrs.item(i);
	  
	  //Store the srs
	  if (GML_NS.equals(deferredElement.getNamespaceURI())) {
	    //Maybe we could assume a configurable default value here
	    if (attrib.getName().equals("srsName")) {        		
	      whatToPush = attrib.getValue();
	    }
	    srsNamesStack.push(whatToPush);
	  } 
      
	  attList.addAttribute(attrib.getNamespaceURI(), 
	  attrib.getLocalName(), 
	  attrib.getQName().getStringValue(), 
	  Integer.toString(attrib.getType()), 
	  attrib.getValue());            		
        } 
      
	srsNamesStack.push(whatToPush);

	try {
	  geometryDocument.startElement(deferredElement.getNamespaceURI(), deferredElement.getLocalName(), deferredElement.getQName().getStringValue(), attList);
	} catch (Exception e) {
	  e.printStackTrace();
	  LOG.error(e);
	} finally {
	  deferredElement = null;       	       	
	}
      }
      </programlisting><para>We first need to collect its attributes and that's why it is deferred, because attributes events
      come <emphasis>after</emphasis> the call to <methodname>startElement()</methodname>. Elements in the GML 
      namespace that carry an <sgmltag class="attribute">srsName</sgmltag> attribute will push its value. 
      If the element is not in the GML namespace or if no <sgmltag class="attribute">srsName</sgmltag> attribute
      exists, <parameter>null</parameter> is pushed.</para><note><para>We could have had a smarter mechanism, but we first have to take a decision about the fact that
      we could define a default SRS here, either from the config, or from a higher-level element. This part of
      the code will thus probably be revisited once the decision is taken.</para></note><para>When the attributes are collected, we can send a <classname>startElement()</classname> event to the SAX
      handler, thus marking the end of the deferring process.</para><para>Processing of the current element with <methodname>endElement()</methodname>:</para><programlisting format="linespecific">	
      private void processCurrentElement(ElementImpl element) {
        String currentSrsName = (String)srsNamesStack.pop();        		
	try {
	  geometryDocument.endElement(element.getNamespaceURI(), element.getLocalName(), element.getQName().getStringValue());
	  //Some invalid/(yet) incomplete geometries don't have a SRS
	  if (streamedGeometry != null &amp;&amp; currentSrsName != null) {   
	    currentNodeId = element.getNodeId();
	    geometries.put(currentNodeId, new SRSGeometry(currentSrsName, streamedGeometry));		        		
	    if (flushAfter != -1 &amp;&amp; geometries.size() &gt;= flushAfter) {
	      //Mmmh... doesn't flush since it is currently dependant from the
	      //number of nodes in the DOM file ; would need refactorings
	      //currentDoc.getBroker().checkAvailableMemory();
	      currentDoc.getBroker().flush();
	      ///Aaaaaargl !
	      final double percent = ((double) Runtime.getRuntime().freeMemory() / (double) Runtime.getRuntime().maxMemory()) * 100;
	      if (percent &lt; 30) {	        		
	        System.gc();
	      }
            }
          }        		
	} catch (Exception e) {
	  LOG.error("Unable to collect geometry for node: " + currentNodeId + ". Indexing will be skipped", e);        		
	} finally {  
	  streamedGeometry = null;	        			
	}
      }     			
      </programlisting><para>We first pop a SRS name from the stack. <parameter>null</parameter> will indicate that the element doesn't
      have any and thus that it is an element which doesn't carry enough information to build a complete geometry. That doesn't
      prevent us to forward this element to the SAX handler.</para><para>The SAX handler might have been able to build a <classname>Geometry</classname> then. If so, the current
      index entry (composed of <parameter>currentSrsName</parameter>, <parameter>streamedGeometry</parameter>, 
      <parameter>currentNodeId</parameter> and the "global" <classname>org.exist.dom.DocumentImpl</classname>) is added to <parameter>geometries</parameter> (wrapped in the convenience
      <classname>SRSGeometry</classname> class). We then check if it's time to flush the pending index entries.</para><para>This is how the <classname>GeometryHandler</classname> looks like. It is also implemented as an
      inner class of <classname>org.exist.indexing.spatial.AbstractGMLJDBCIndexWorker</classname></para><programlisting format="linespecific">				
      private class GeometryHandler extends XMLFilterImpl implements GMLHandlerJTS {
        public void geometry(Geometry geometry) {
	  streamedGeometry = geometry;      	
	  //TODO : null geometries can be returned for many reasons, including a (too) strict
	  //topology check done by the Geotools SAX parser.
	  //It would be nice to have static classes extending Geometry to report such geometries
	  if (geometry == null)
	    LOG.error("Collected null geometry for node: " + currentNodeId + ". Indexing will be skipped");
	}
      }
      </programlisting><para>Thanks to Geotools SAX parser, it hasn't to be more complicated than setting the <parameter>streamedGeometry</parameter>
      "global" member.</para><note><para>However, it may throw some <classname>NullPointerException</classname>s as described above.</para></note></section><section><title>Implementing some functions that cooperate with the spatial index</title><para>We currently provide three sets of functions that are able to cooperate with spatial indexes.</para><para>The functions are declared in the <classname>org.exist.xquery.modules.spatial.SpatialModule</classname> module
      which operates in the <parameter>http://exist-db.org/xquery/spatial</parameter> namespace (whose default prefix is
      <parameter>spatial</parameter>).</para><para>The functions signatures are documented together with the functions themselves in 
      <ulink url="http://demo.exist-db.org/xquery/functions.xq">this page</ulink>. Here we will only look at their <methodname>eval()</methodname> methods.</para><para>The first functions set we will describe is <classname>org.exist.xquery.modules.spatial.FunSpatialSearch</classname>,
      which performs searches on the spatial index:</para><programlisting format="linespecific">				
      public Sequence eval(Sequence[] args, Sequence contextSequence) throws XPathException {
        Sequence result = null;
	Sequence nodes = args[0];        
	if (nodes.isEmpty())
	  result = Sequence.EMPTY_SEQUENCE;
	else if (args[1].isEmpty())
	  //TODO : to be discussed. We could also return an empty sequence here        	
	  result = nodes;
	else {
	  try {
	    AbstractGMLJDBCIndexWorker indexWorker = (AbstractGMLJDBCIndexWorker)        	
	    context.getBroker().getIndexController().getWorkerByIndexId(AbstractGMLJDBCIndex.ID);
	    if (indexWorker == null)
	      throw new XPathException("Unable to find a spatial index worker");
	    Geometry EPSG4326_geometry = null;
	    NodeValue geometryNode = (NodeValue) args[1].itemAt(0);   
	    if (geometryNode.getImplementationType() == NodeValue.PERSISTENT_NODE)
	      //Get the geometry from the index if available
	      EPSG4326_geometry = indexWorker.getGeometryForNode(context.getBroker(), (NodeProxy)geometryNode, true);
	    if (EPSG4326_geometry == null) {
	      String sourceCRS = ((Element)geometryNode.getNode()).getAttribute("srsName").trim();
	      Geometry geometry = indexWorker.streamNodeToGeometry(context, geometryNode);	            
	      EPSG4326_geometry = indexWorker.transformGeometry(geometry, sourceCRS, "EPSG:4326");	        		        
	    }
	    if (EPSG4326_geometry == null) 
	      throw new XPathException("Unable to get a geometry from the node");
	    int spatialOp = SpatialOperator.UNKNOWN;
	    if (isCalledAs("equals"))
	      spatialOp = SpatialOperator.EQUALS;
	    else if (isCalledAs("disjoint"))
	      spatialOp = SpatialOperator.DISJOINT;
            else if (isCalledAs("intersects"))
	      spatialOp = SpatialOperator.INTERSECTS;	 
	    else if (isCalledAs("touches"))
	      spatialOp = SpatialOperator.TOUCHES;
	    else if (isCalledAs("crosses"))
	      spatialOp = SpatialOperator.CROSSES;	   
	    else if (isCalledAs("within"))
	      spatialOp = SpatialOperator.WITHIN;		
	    else if (isCalledAs("contains"))
	      spatialOp = SpatialOperator.CONTAINS;		
            else if (isCalledAs("overlaps"))
	      spatialOp = SpatialOperator.OVERLAPS;
	    //Search the EPSG:4326 in the index
	    result = indexWorker.search(context.getBroker(),  nodes.toNodeSet(), EPSG4326_geometry, spatialOp);
	    hasUsedIndex = true;	        	
	  } catch (SpatialIndexException e) {
	    throw new XPathException(e);
	  }
	}
	return result;
      }
      </programlisting><para>We first build an early result if empty sequences are passed to the function.</para><para>Then, we try to access the <classname>XQueryContext</classname>'s <classname>AbstractGMLJDBCIndex</classname>
      (remember that there is a 1:1 relationship between <classname>XQueryContext</classname> and <classname>DBBroker</classname> and
      a 1:1 relationship between <classname>DBBroker</classname> and <classname>IndexWorker</classname>). If we can not find an <classname>AbstractGMLJDBCIndex</classname> we throw an <classname>Exception</classname> since we will need this and its concrete class to delegate spatial operations to (whatever its underlying DB implementation is,
      thanks to our generic design).</para><para>Then, we examine if the geometry node is persistent, in which case it <emphasis>might</emphasis> be indexed. If so, we try
      to get an EPSG:4326 <classname>Geometry</classname> from the index.</para><para>If nothing is returned here, either because the node isn't indexed or because it is an in-memory node,
      we stream it to a <classname>Geometry</classname> and we transform this into 
      an EPSG:4326 <classname>Geometry</classname>. Of course, this process is slower than a direct lookup into the index.</para><para>Then we search for the geometry in the index after having determined the spatial operator from the function's
      name.</para><para>The second functions set is <classname>org.exist.xquery.modules.spatial.FunGeometricProperties</classname>,
      which retrieves a property for a <classname>Geometry</classname>:</para><programlisting format="linespecific">	
      public Sequence eval(Sequence[] args, Sequence contextSequence) throws XPathException {
        Sequence result = null;
	Sequence nodes = args[0];        
	if (nodes.isEmpty())
	  result = Sequence.EMPTY_SEQUENCE;
	else {
	  try {
	    Geometry geometry = null;
	    String sourceCRS = null;
	    AbstractGMLJDBCIndexWorker indexWorker = (AbstractGMLJDBCIndexWorker)
	    context.getBroker().getIndexController().getWorkerByIndexId(AbstractGMLJDBCIndex.ID);
	    if (indexWorker == null)
	      throw new XPathException("Unable to find a spatial index worker");
	    String propertyName = null;
	    if (isCalledAs("getWKT")) {
	      propertyName = "WKT";
	    } else if (isCalledAs("getWKB")) {
	      propertyName = "WKB";						
	    } else if (isCalledAs("getMinX")) {
	      propertyName = "MINX";						
	    } else if (isCalledAs("getMaxX")) {					
	      propertyName = "MAXX";
	    } else if (isCalledAs("getMinY")) {
	      propertyName = "MINY";
	    } else if (isCalledAs("getMaxY")) {
	      propertyName = "MAXY";
	    } else if (isCalledAs("getCentroidX")) {
	      propertyName = "CENTROID_X";
	    } else if (isCalledAs("getCentroidY")) {
	      propertyName = "CENTROID_Y";
	    } else if (isCalledAs("getArea")) {
	      propertyName = "AREA";
	    } else if (isCalledAs("getEPSG4326WKT")) {
	      propertyName = "EPSG4326_WKT";      
	    } else if (isCalledAs("getEPSG4326WKB")) {
	      propertyName = "EPSG4326_WKB";
	    } else if (isCalledAs("getEPSG4326MinX")) {
	      propertyName = "EPSG4326_MINX";
	    } else if (isCalledAs("getEPSG4326MaxX")) {
	      propertyName = "EPSG4326_MAXX";
	    } else if (isCalledAs("getEPSG4326MinY")) {
	      propertyName = "EPSG4326_MINY";
	    } else if (isCalledAs("getEPSG4326MaxY")) {
	      propertyName = "EPSG4326_MAXY";
	    } else if (isCalledAs("getEPSG4326CentroidX")) {
	      propertyName = "EPSG4326_CENTROID_X";
	    } else if (isCalledAs("getEPSG4326CentroidY")) {
	      propertyName = "EPSG4326_CENTROID_Y";
	    } else if (isCalledAs("getEPSG4326Area")) {
	      propertyName = "EPSG4326_AREA";
	    } else if (isCalledAs("getSRS")) {
	      propertyName = "SRS_NAME";
	    } else if (isCalledAs("getGeometryType")) {
	      propertyName = "GEOMETRY_TYPE";
	    } else if (isCalledAs("isClosed")) {
	      propertyName = "IS_CLOSED";
	    } else if (isCalledAs("isSimple")) {
	      propertyName = "IS_SIMPLE";
	    } else if (isCalledAs("isValid")) {
	      propertyName = "IS_VALID";
	    } else {
	      throw new XPathException("Unknown spatial property: " + mySignature.getName().getLocalName());
	    } 
	    NodeValue geometryNode = (NodeValue) nodes.itemAt(0);
	    if (geometryNode.getImplementationType() == NodeValue.PERSISTENT_NODE) {
	      if (propertyName != null) {
	        //The node should be indexed : get its property
		result = indexWorker.getGeometricPropertyForNode(context.getBroker(), (NodeProxy)geometryNode, propertyName);
		hasUsedIndex = true;
	      } else {
	        //Or, at least, its geometry for further processing
		if (propertyName.indexOf("EPSG4326") != Constants.STRING_NOT_FOUND) {
		  geometry = indexWorker.getGeometryForNode(context.getBroker(), (NodeProxy)geometryNode, true);
		  sourceCRS = "EPSG:4326";
		} else {
		  geometry = indexWorker.getGeometryForNode(context.getBroker(), (NodeProxy)geometryNode, false);
		  sourceCRS = indexWorker.getGeometricPropertyForNode(context.getBroker(), (NodeProxy)geometryNode, "SRS_NAME").getStringValue();
		}
	      }
	    }
	    if (result == null) {
	      //builds the geometry
	      if (geometry == null) {
	        sourceCRS = ((Element)geometryNode.getNode()).getAttribute("srsName").trim();
		geometry = indexWorker.streamNodeToGeometry(context, geometryNode);		            	
	      }	
	      if (geometry == null) 
	        throw new XPathException("Unable to get a geometry from the node");
	      //Transform the geometry to EPSG:4326 if relevant
	      if (propertyName.indexOf("EPSG4326") != Constants.STRING_NOT_FOUND) {
	      geometry = indexWorker.transformGeometry(geometry, sourceCRS, "EPSG:4326");
	      if (isCalledAs("getEPSG4326WKT")) {
	        result = new StringValue(wktWriter.write(geometry));
	      } else if (isCalledAs("getEPSG4326WKB")) {
	        result = new Base64Binary(wkbWriter.write(geometry));
	      } else if (isCalledAs("getEPSG4326MinX")) {
	        result = new DoubleValue(geometry.getEnvelopeInternal().getMinX());
	      } else if (isCalledAs("getEPSG4326MaxX")) {
	        result = new DoubleValue(geometry.getEnvelopeInternal().getMaxX());
	      } else if (isCalledAs("getEPSG4326MinY")) {
	        result = new DoubleValue(geometry.getEnvelopeInternal().getMinY());
              } else if (isCalledAs("getEPSG4326MaxY")) {
	        result = new DoubleValue(geometry.getEnvelopeInternal().getMaxY());
	      } else if (isCalledAs("getEPSG4326CentroidX")) {
	        result = new DoubleValue(geometry.getCentroid().getX());
	      } else if (isCalledAs("getEPSG4326CentroidY")) {
	        result = new DoubleValue(geometry.getCentroid().getY());
	      } else if (isCalledAs("getEPSG4326Area")) {
	        result = new DoubleValue(geometry.getArea());
	      }
	    } else if (isCalledAs("getWKT")) {
	      result = new StringValue(wktWriter.write(geometry));
	    } else if (isCalledAs("getWKB")) {
	      result = new Base64Binary(wkbWriter.write(geometry));
	    } else if (isCalledAs("getMinX")) {
	      result = new DoubleValue(geometry.getEnvelopeInternal().getMinX());
	    } else if (isCalledAs("getMaxX")) {
	      result = new DoubleValue(geometry.getEnvelopeInternal().getMaxX());
	    } else if (isCalledAs("getMinY")) {
	      result = new DoubleValue(geometry.getEnvelopeInternal().getMinY());
	    } else if (isCalledAs("getMaxY")) {
	      result = new DoubleValue(geometry.getEnvelopeInternal().getMaxY());
	    } else if (isCalledAs("getCentroidX")) {
	      result = new DoubleValue(geometry.getCentroid().getX());
	    } else if (isCalledAs("getCentroidY")) {
	      result = new DoubleValue(geometry.getCentroid().getY());
	    } else if (isCalledAs("getArea")) {
	      result = new DoubleValue(geometry.getArea());
	    } else if (isCalledAs("getSRS")) {
	      result = new StringValue(((Element)geometryNode).getAttribute("srsName"));
	    } else if (isCalledAs("getGeometryType")) {
	      result = new StringValue(geometry.getGeometryType());
	    } else if (isCalledAs("isClosed")) {
	      result = new BooleanValue(!geometry.isEmpty());
	    } else if (isCalledAs("isSimple")) {
	      result = new BooleanValue(geometry.isSimple());
	    } else if (isCalledAs("isValid")) {
	      result = new BooleanValue(geometry.isValid());
	    } else {
	      throw new XPathException("Unknown spatial property: " + mySignature.getName().getLocalName());
	    }
	  }
	  } catch (SpatialIndexException e) {
	    throw new XPathException(e);
	  }
        }
	return result;
      }
      </programlisting><para>The design is very much the same : we build an early result if empty sequences are involved, we get a
      <classname>AbstractGMLJDBCIndex</classname>, then we set a <parameter>propertyName</parameter>, which is actually
      a SQL field name, depending on the function's name.</para><para>An attempt to retrieve the field content from the DB is made and, if unsuccessful, we try to get the
      node's <classname>Geometry</classname> from the index.</para><para>Then, if we still haven't got this <classname>Geometry</classname>, either because the node isn't indexed or
      because it is an in-memory node, we stream it to a <classname>Geometry</classname> and we transform this 
      into an EPSG:4326 <classname>Geometry</classname> if the function's name requires to do so.</para><para>We then <emphasis>dynamically</emphasis> build the property to be returned.</para><note><para>This mechanism if far from being efficient compared to the index lookup, but it shows how easy it would be to return a property 
      which is not available in a spatial index.</para></note><para>The third functions set, <classname>org.exist.xquery.modules.spatial.FunGMLProducers</classname>, uses the same design :</para><programlisting format="linespecific">				
      public Sequence eval(Sequence[] args, Sequence contextSequence)
        throws XPathException {
	Sequence result = null; 
	try {
	  AbstractGMLJDBCIndexWorker indexWorker = (AbstractGMLJDBCIndexWorker)
	  context.getBroker().getIndexController().getWorkerByIndexId(AbstractGMLJDBCIndex.ID);
	  if (indexWorker == null)
	    throw new XPathException("Unable to find a spatial index worker");
	  Geometry geometry = null;	        
	  String targetSRS = null;
	  if (isCalledAs("transform")) {
	    if (args[0].isEmpty())
	      result = Sequence.EMPTY_SEQUENCE;
	    else {
	      NodeValue geometryNode = (NodeValue) args[0].itemAt(0);
	      //Try to get the geometry from the index
	      String sourceSRS = null;
	      if (geometryNode.getImplementationType() == NodeValue.PERSISTENT_NODE) {
	        sourceSRS = indexWorker.getGeometricPropertyForNode(context.getBroker(), (NodeProxy)geometryNode, "SRS_NAME").getStringValue();
		geometry = indexWorker.getGeometryForNode(context.getBroker(), (NodeProxy)geometryNode, false);		        		
		hasUsedIndex = true;
	      }
	      //Otherwise, build it
	      if (geometry == null) {
	        sourceSRS = ((Element)geometryNode.getNode()).getAttribute("srsName").trim();
		geometry = indexWorker.streamNodeToGeometry(context, geometryNode);
	      }
	      if (geometry == null) 
	        throw new XPathException("Unable to get a geometry from the node");
		targetSRS = args[1].itemAt(0).getStringValue().trim();
      
		geometry = indexWorker.transformGeometry(geometry, sourceSRS, targetSRS);
	      }
	    } else if (isCalledAs("WKTtoGML")) {
	      if (args[0].isEmpty())
	        result = Sequence.EMPTY_SEQUENCE;
	      else {
	        String wkt = args[0].itemAt(0).getStringValue();
		WKTReader wktReader = new WKTReader();
		try {        	
		 geometry = wktReader.read(wkt);
		} catch (ParseException e) {
		throw new XPathException(e);	
	    }
	    if (geometry == null) 
	      throw new XPathException("Unable to get a geometry from the node");
	    targetSRS = args[1].itemAt(0).getStringValue().trim();
	  }
	} else if (isCalledAs("buffer")) {
	  if (args[0].isEmpty())
	    result = Sequence.EMPTY_SEQUENCE;
	  else {
	    NodeValue geometryNode = (NodeValue) args[0].itemAt(0);	        		        	
	    //Try to get the geometry from the index
	    if (geometryNode.getImplementationType() == NodeValue.PERSISTENT_NODE) {
	      targetSRS = indexWorker.getGeometricPropertyForNode(context.getBroker(), (NodeProxy)geometryNode, "SRS_NAME").getStringValue();
	      geometry = indexWorker.getGeometryForNode(context.getBroker(), (NodeProxy)geometryNode, false);		        		
	      hasUsedIndex = true;
	    }
	    //Otherwise, build it
	    if (geometry == null) {
	      targetSRS =  ((Element)geometryNode.getNode()).getAttribute("srsName").trim();
	      geometry = indexWorker.streamNodeToGeometry(context, geometryNode);
	    }
	    if (geometry == null) 
	      throw new XPathException("Unable to get a geometry from the node");
	    double distance = ((DoubleValue)args[1].itemAt(0)).getDouble();
	    int quadrantSegments = 8;	
	    int endCapStyle = BufferOp.CAP_ROUND;
	    if (getArgumentCount() &gt; 2 &amp;&amp; Type.subTypeOf(args[2].itemAt(0).getType(), Type.INTEGER))
	      quadrantSegments = ((IntegerValue)args[2].itemAt(0)).getInt();
	    if (getArgumentCount() &gt; 3 &amp;&amp; Type.subTypeOf(args[3].itemAt(0).getType(), Type.INTEGER))
	      endCapStyle = ((IntegerValue)args[3].itemAt(0)).getInt();
	    switch (endCapStyle) {
	      case BufferOp.CAP_ROUND:
	      case BufferOp.CAP_BUTT:
	      case BufferOp.CAP_SQUARE:
	        //OK
		break;
	      default:
	        throw new XPathException("Invalid line end style");	
	    }        	
	    geometry = geometry.buffer(distance, quadrantSegments, endCapStyle);
	  }
	} else if (isCalledAs("getBbox")) {
	  if (args[0].isEmpty())
	    result = Sequence.EMPTY_SEQUENCE;
	  else {
	    NodeValue geometryNode = (NodeValue) args[0].itemAt(0);
	    //Try to get the geometry from the index
	    if (geometryNode.getImplementationType() == NodeValue.PERSISTENT_NODE) {
	      targetSRS = indexWorker.getGeometricPropertyForNode(context.getBroker(), (NodeProxy)geometryNode, "SRS_NAME").getStringValue();
	      geometry = indexWorker.getGeometryForNode(context.getBroker(), (NodeProxy)geometryNode, false);		        		
	      hasUsedIndex = true;
	    }
	    //Otherwise, build it
	    if (geometry == null) {
	      targetSRS = ((Element)geometryNode.getNode()).getAttribute("srsName").trim();
	      geometry = indexWorker.streamNodeToGeometry(context, geometryNode);		            	
	    }
	    if (geometry == null) 
	      throw new XPathException("Unable to get a geometry from the node");
	      
	    geometry = geometry.getEnvelope();
	  }
	} else if (isCalledAs("convexHull")) {
	  if (args[0].isEmpty())
	    result = Sequence.EMPTY_SEQUENCE;
	  else {
	    NodeValue geometryNode = (NodeValue) args[0].itemAt(0);
	    //Try to get the geometry from the index
	    if (geometryNode.getImplementationType() == NodeValue.PERSISTENT_NODE) {
	      targetSRS = indexWorker.getGeometricPropertyForNode(context.getBroker(), (NodeProxy)geometryNode, "SRS_NAME").getStringValue();
	      geometry = indexWorker.getGeometryForNode(context.getBroker(), (NodeProxy)geometryNode, false);		        		
	      hasUsedIndex = true;
	    }
	    //Otherwise, build it
	    if (geometry == null) {
	      targetSRS = ((Element)geometryNode.getNode()).getAttribute("srsName").trim();
	      geometry = indexWorker.streamNodeToGeometry(context, geometryNode);
	    }
	    if (geometry == null) 
	      throw new XPathException("Unable to get a geometry from the node");
	      
	    geometry = geometry.convexHull();
	  }
	} else if (isCalledAs("boundary")) {
	  if (args[0].isEmpty())
	    result = Sequence.EMPTY_SEQUENCE;
	  else {
	    NodeValue geometryNode = (NodeValue) args[0].itemAt(0);
	    //Try to get the geometry from the index
	    if (geometryNode.getImplementationType() == NodeValue.PERSISTENT_NODE) {
	      targetSRS = indexWorker.getGeometricPropertyForNode(context.getBroker(), (NodeProxy)geometryNode, "SRS_NAME").getStringValue();
	      geometry = indexWorker.getGeometryForNode(context.getBroker(), (NodeProxy)geometryNode, false);		        		
	      hasUsedIndex = true;
	    }
	    //Otherwise, build it
	    if (geometry == null) {
	      targetSRS = ((Element)geometryNode.getNode()).getAttribute("srsName").trim();
	      geometry = indexWorker.streamNodeToGeometry(context, geometryNode);
	    }
	    if (geometry == null) 
	      throw new XPathException("Unable to get a geometry from the node");
	      
	    geometry = geometry.getBoundary();		        	
	  }
	} else {
	  Geometry geometry1 = null;
	  Geometry geometry2 = null;
	  if (args[0].isEmpty() &amp;&amp; args[1].isEmpty())
	    result = Sequence.EMPTY_SEQUENCE;
	  else if (!args[0].isEmpty() &amp;&amp; args[1].isEmpty())
	    result = args[0].itemAt(0).toSequence();
	  else if (args[0].isEmpty() &amp;&amp; !args[1].isEmpty())
	    result = args[1].itemAt(0).toSequence();
	  else {
	    NodeValue geometryNode1 = (NodeValue) args[0].itemAt(0);
	    NodeValue geometryNode2 = (NodeValue) args[1].itemAt(0);
	    String srsName1 = null;
	    String srsName2 = null;
	    //Try to get the geometries from the index
	    if (geometryNode1.getImplementationType() == NodeValue.PERSISTENT_NODE) {
	      srsName1 = indexWorker.getGeometricPropertyForNode(context.getBroker(), (NodeProxy)geometryNode1, "SRS_NAME").getStringValue();
	      geometry1 = indexWorker.getGeometryForNode(context.getBroker(), (NodeProxy)geometryNode1, false);		        		
	      hasUsedIndex = true;
	    }
	    if (geometryNode2.getImplementationType() == NodeValue.PERSISTENT_NODE) {
	      srsName2 = indexWorker.getGeometricPropertyForNode(context.getBroker(), (NodeProxy)geometryNode2, "SRS_NAME").getStringValue();
	      geometry2 = indexWorker.getGeometryForNode(context.getBroker(), (NodeProxy)geometryNode2, false);		        		
	      hasUsedIndex = true;
	    }
	    //Otherwise build them
	    if (geometry1 == null) {
	      srsName1 = ((Element)geometryNode1.getNode()).getAttribute("srsName").trim();
	      geometry1 = indexWorker.streamNodeToGeometry(context, geometryNode1);
	    }
	    if (geometry2 == null) {
	      srsName2 = ((Element)geometryNode2.getNode()).getAttribute("srsName").trim();
	      geometry2 = indexWorker.streamNodeToGeometry(context, geometryNode2);	            	
	    }
      
	    if (geometry1 == null) 
	      throw new XPathException("Unable to get a geometry from the first node");
	    if (geometry2 == null) 
	      throw new XPathException("Unable to get a geometry from the second node");		        	
	      
	    //Transform the second geometry in the SRS of the first one if necessary
	    if (!srsName1.equalsIgnoreCase(srsName2)) {
	      geometry2 = indexWorker.transformGeometry(geometry2, srsName1, srsName2);      	
	    }				
	    
	    if (isCalledAs("intersection")) {
	      geometry = geometry1.intersection(geometry2);
	    } else if (isCalledAs("union")) {
	      geometry = geometry1.union(geometry2);
	    } else if (isCalledAs("difference")) {	
	      geometry = geometry1.difference(geometry2);
	    } else if (isCalledAs("symetricDifference")) {
	      geometry = geometry1.symDifference(geometry2);
	    }	
      
	    targetSRS = srsName1;
	  }
	}
      
	if (result == null) {	        
	  String gmlPrefix = context.getPrefixForURI(AbstractGMLJDBCIndexWorker.GML_NS);
	  if (gmlPrefix == null) 
	    throw new XPathException("'" + AbstractGMLJDBCIndexWorker.GML_NS + "' namespace is not defined");	
	    
	  context.pushDocumentContext();
	  try {
	    MemTreeBuilder builder = context.getDocumentBuilder();
	    DocumentBuilderReceiver receiver = new DocumentBuilderReceiver(builder);
	    result = (NodeValue)indexWorker.streamGeometryToElement(geometry, targetSRS, receiver);
	  } finally {
	    context.popDocumentContext();
	  }
	}
	} catch (SpatialIndexException e) {
	  throw new XPathException(e);
        }        
	return result;
      }
      </programlisting><para><!-- probably this procedure should be listed once (ordered list) and then be referred to /ljo--> It looks more complicated because of the multiple possible argument counts. However, the pinciple
      remains the same: early result when empty sequences are involved, fetching of the <classname>Geometry</classname>ies
      (and of its/their SRS) from the DB, streaming if nothing can be fetched, then geometric computations after a
      transformation of the second <classname>Geometry</classname> if relevant.</para><para>The final process streams the resulting <classname>Geometry</classname> as the result, in the SRS specified
      by the relevant argument or in the SRS of the first <classname>Geometry</classname>, depending on the function's name.</para></section><section><title>Playing with the spatial index</title><para/><para>Now that we have described the spatial index, it is time to play with it. Only a few of its features
      will be demonstrated, but we will explain again what happens under the hood.</para><para>The first step is to make sure to have a recent enough release version of eXist : 1.2 of later.</para><para>Then, you have to prepare eXist to build the spatial index library. To do that, go into the <filename>${EXIST_HOME}/extensions/indexes</filename> directory and, if necessary, copy
      <filename>build.properties</filename> to a new file, <filename>local.properties</filename>.</para><para>Open this file and check that <command>include.index.spatial</command> is set to <command>true</command>.</para><para>Invoke <command>build.bat clean</command> or <command>build.sh clean</command>, depending on your platform,
       from a command line. This will generate <filename>${EXIST_HOME}/extensions/indexes/build.xml</filename>, which is needed by the modularized indexes infrastructure.</para><para>Invoke <command>build.bat extension-indexes</command> or <command>build.sh extension-indexes</command>, depending on your platform,
       from a command line.</para><para>If necessary, the required external (large) libraries will be downloaded from the WWW into the <filename>${EXIST_HOME}/extensions/indexes/spatial/lib</filename> directory. 
       Most of them have a <filename>gt2-*.jar</filename> name. Make sure to make them available in your application's classpath !</para><note><para>If you are behind a proxy, do not forget to set its settings in <filename>${EXIST_HOME}/build.properties</filename>.</para></note><para>A file named <filename>exist-spatial-module.jar</filename> should be generated into the <filename>${EXIST_HOME}/lib/extensions</filename> directory.</para><para>Enable the spatial index and the spatial module in <filename>${EXIST_HOME}/conf.xml</filename> if it is not already done.</para><para>the spatial index:</para><programlisting format="linespecific">				
      &lt;modules&gt;
        &lt;module id="ngram-index" class="org.exist.indexing.ngram.NGramIndex"
         file="ngram.dbx" n="3"/&gt;            
        &lt;module id="spatial-index" class="org.exist.indexing.spatial.GMLHSQLIndex"
         connectionTimeout="10000" flushAfter="300" /&gt;
      &lt;/modules&gt; 
      </programlisting><para>and the spatial module:</para><programlisting format="linespecific">				
      &lt;xquery enable-java-binding="no" enable-query-rewriting="no"  backwardCompatible="no"&gt;
        &lt;builtin-modules&gt;
          ...
      
          &lt;module class="org.exist.xquery.modules.spatial.SpatialModule"
           uri="http://exist-db.org/xquery/spatial" /&gt;
      
          ...
      
          &lt;!-- Optional Modules --&gt;
      
          ...

        &lt;/builtin-modules&gt;
      &lt;/xquery&gt;				
      </programlisting><para>This concludes the prerequisites for running the test.</para><para>Our demonstration file is taken from the <ulink url="http://www.ordnancesurvey.co.uk/oswebsite/">Ordnance Survey of Great-Britain's WWW site</ulink>          	
      	which offers <ulink url=" http://www.ordnancesurvey.co.uk/oswebsite/products/try-now/sample-data.html">sample data</ulink>.</para><para>The chosen topography layer is of Port-Talbot, which is available as <ulink url="http://www.ordnancesurvey.co.uk/products/osmastermap/layers/topography/sampledata/2182-SS7886-2c1.gz">2182-SS7886-2c1.gz</ulink>.
      Download this file, gunzip it, and give to the resulting file a <filename>.gml</filename> extension (<filename>port-talbot.gml</filename>) this will
      allow eXist to reckognize it as an XML file.</para><note><para>If you have previously executed <command>build test</command>, the file should have been downloaded
      and gunzipped for you in <filename>${EXIST_HOME}/extensions/indexes/spatial/test/resources</filename>.</para></note><para>Since this file refers to an OSGB-hosted schema, we will need to bypass validation in <filename>${EXIST_HOME}/conf.xml</filename>.</para><para>Make sure the mode value is set like this:</para><programlisting format="linespecific">		
      &lt;validation mode="no"&gt;
      </programlisting><para>We are now ready to start the demonstration and we will use the interactive client for that. Run either <filename>${EXIST_HOME}/bin/client.bat</filename> or <filename>${EXIST_HOME}/bin/client.sh</filename> from the command line (please read elsewhere if you do not know how to start it).</para><para>Let's start by creating a collection named <filename>spatial</filename> in the <filename>/db</filename> collection. The menus might be localised, but in english it is <command>File/Create a collection...</command>.</para><para>Then, we will configure this collection by creating a configuration collection.</para><para>Let's navigate to <filename>/db/system/config</filename></para><para>If required, let's create a general configuration collection : <command>File/Create a collection...</command> name it <filename>db</filename> and get into it.</para><para>Then let's create a configuration collection for <filename>/db/spatial</filename>: <command>File/Create a collection...</command> name it <filename>spatial</filename> and get into it.</para><para>We are now in <filename>/db/system/config/db/spatial</filename>.</para><para>Let's now create a configuration file for this collection: <command>File/Create an empty document...</command> name it
      <filename>collection.xconf</filename>.</para><para>Double-click on this document and let's replace its auto-generated content :</para><programlisting format="linespecific">
      &lt;template/&gt;
      </programlisting><para>with this one:</para><programlisting format="linespecific">
      &lt;collection xmlns="http://exist-db.org/collection-config/1.0"&gt;
        &lt;index&gt;
          &lt;gml flushAfter="200"/&gt;
        &lt;/index&gt;
      &lt;/collection&gt;
      </programlisting><note><para>Do not forget to save the document before closing the window.</para></note><para>The <filename>/db/system/config/db/spatial</filename> collection is now configured to index GML geometries when they are uploaded. The in-memory index entries will be flushed 
      to the HSQLDB every 200 geometries and will wait at most 100 seconds, the default value, to establish
      a connection to the HSQL db.</para><para>Let's navigate to <filename>/db/spatial</filename>.</para><para>Let's upload <filename>port-talbot.gml</filename>: File/Upload files/directories...</para><para>On my computer, the operation on this 23.6 Mb file is performed in about 100 seconds, including
      some default fulltext indexing. Let's close the upload window and quit the interactive client.</para><para>Let's look our our GML file looks like on GML Viewer, a free viewer provided by <ulink url="http://www.snowflakesoftware.co.uk/products/gmlviewer/index.htm">Snowflake software</ulink> :</para><screenshot><graphic fileref="resources/port-talbot.jpg"/></screenshot><para>If you want to have a look at the spatial index HSQLDB, which, if you are using the default data-dir, is in <filename>${EXIST_HOME}/webapp/WEB-INF/data/spatial_index.*</filename> there is a dedicated script file in <filename>${EXIST_HOME}/extensions/indexes/spatial/</filename>.
      to launch HSQL's GUI client: Use either <filename>hsql.bat</filename> or <filename>hsql.sh [data-dir]</filename> (you only need to supply data-dir if it is not the default one).</para><para>If the SQL command <command>SELECT * FROM SPATIAL_INDEX_V1;</command> is executed, the result window
      shows that 21961 geometries have been indexed.</para><para>Let's get back to the interactive client and open the query window (the one we get when clicking on the binocular button in the toolbar).</para><para>This query:</para><programlisting format="linespecific">				
      declare namespace gml = "http://www.opengis.net/gml";
      spatial:transform(
        &lt;gml:Polygon srsName="osgb:BNG" xmlns:gml='http://www.opengis.net/gml'&gt;
        &lt;gml:outerBoundaryIs&gt;&lt;gml:LinearRing&gt;&lt;gml:coordinates&gt;
          278200,187600 278400,187600 278400,188000 278200,188000 278200,187600
         &lt;/gml:coordinates&gt;&lt;/gml:LinearRing&gt;&lt;/gml:outerBoundaryIs&gt;
        &lt;/gml:Polygon&gt;, 
        "epsg:4326")	
      </programlisting><para>... is processed in a little bit less than 2 seconds. That could seem high, but there is a cost for
      the Geotools transformation factories initialization. Subsequent requests will be much faster, although
      there will always be a small cost for the streaming of the in-memory node to a <classname>Geometry</classname>
      object.</para><para>The result is:</para><programlisting format="linespecific">				
      &lt;gml:Polygon xmlns:gml="http://www.opengis.net/gml"&gt;
        &lt;gml:outerBoundaryIs&gt;
          &lt;gml:LinearRing&gt;
            &lt;gml:coordinates decimal="." cs="," ts=" "&gt;-3.7578,51.5743 -3.7579,51.5779 -3.755,51.5779 -3.7549,51.5743 -3.7578,51.5743
	    &lt;/gml:coordinates&gt;
	  &lt;/gml:LinearRing&gt;
	&lt;/gml:outerBoundaryIs&gt;
      &lt;/gml:Polygon&gt;
      </programlisting><note><para>Due to the current Geotools limitations, there is no <sgmltag class="attribute">srsName</sgmltag> attribute on <sgmltag>gml:Polygon</sgmltag> ! See above.</para></note><para>... but people might find more convenient to perform this query :</para><programlisting format="linespecific">
      declare namespace gml = "http://www.opengis.net/gml";
      spatial:getEPSG4326WKT(
        &lt;gml:Polygon srsName="osgb:BNG" xmlns:gml='http://www.opengis.net/gml'&gt;
          &lt;gml:outerBoundaryIs&gt;&lt;gml:LinearRing&gt;
	    &lt;gml:coordinates&gt;
	      278200,187600 278400,187600 278400,188000 278200,188000 278200,187600
	    &lt;/gml:coordinates&gt;
	  &lt;/gml:LinearRing&gt;
	&lt;/gml:outerBoundaryIs&gt;
      &lt;/gml:Polygon&gt;)	
      </programlisting><para>... which returns:</para><programlisting format="linespecific">				
      POLYGON ((-3.7577853800140995 51.57430250509819, -3.7579241102503356 
      51.57789774692169, -3.7550389942943365 51.57794093512567, -3.754900491457913 51.57434568777196, 
      -3.7577853800140995 51.57430250509819)
      </programlisting><para>So, 3 degrees West, 51 deegrees North... we must be indeed northern of Brittany, i.e. in south-western Great-Britain.</para><para>Let's see what our polygon looks like:</para><screenshot><graphic fileref="resources/rectangle.jpg"/></screenshot><para>Now, we continue doing something more practical:</para><programlisting format="linespecific">				
      declare namespace gml = "http://www.opengis.net/gml";
      spatial:intersects(//gml:Polygon,
        &lt;gml:Polygon srsName="osgb:BNG" xmlns:gml='http://www.opengis.net/gml'&gt;
          &lt;gml:outerBoundaryIs&gt;
	    &lt;gml:LinearRing&gt;
	      &lt;gml:coordinates&gt;
	        278200,187600 278400,187600 278400,188000 278200,188000 278200,187600
	      &lt;/gml:coordinates&gt;
	    &lt;/gml:LinearRing&gt;
	 &lt;/gml:outerBoundaryIs&gt;
      &lt;/gml:Polygon&gt;
      )	
      </programlisting><para>This query returns 756 <sgmltag>gml:Polygon</sgmltag>s in about 15 seconds.
      A subsequent call returns in just about 450 ms, not having the cost for initializations (in particular the first connection to the HSQLDB). A slighly modified query, in order to show the performance without utilising eXist's performant cache:</para><programlisting format="linespecific">
      declare namespace gml = "http://www.opengis.net/gml";
      spatial:intersects(//gml:Polygon,
        &lt;gml:Polygon srsName="osgb:BNG" xmlns:gml='http://www.opengis.net/gml'&gt;
          &lt;gml:outerBoundaryIs&gt;
	    &lt;gml:LinearRing&gt;
	      &lt;gml:coordinates&gt;
	        278201,187600 278401,187600 278401,188000 278201,188000 278201,187600
	      &lt;/gml:coordinates&gt;
	    &lt;/gml:LinearRing&gt;
	  &lt;/gml:outerBoundaryIs&gt;
        &lt;/gml:Polygon&gt;
      )
      </programlisting><para>... retuns 755 <sgmltag>gml:Polygon</sgmltag> (one less) in just about 470 ms.</para><para>The result of our first intersection query looks like this:</para><screenshot><graphic fileref="resources/intersection.jpg"/></screenshot><para>Let's try another type of spatial query:</para><programlisting format="linespecific">
      declare namespace gml = "http://www.opengis.net/gml";
      spatial:within(//gml:Polygon,
        &lt;gml:Polygon srsName="osgb:BNG" xmlns:gml='http://www.opengis.net/gml'&gt;
          &lt;gml:outerBoundaryIs&gt;
	    &lt;gml:LinearRing&gt;
	      &lt;gml:coordinates&gt;
	        278200,187600 278400,187600 278400,188000 278200,188000 278200,187600
	      &lt;/gml:coordinates&gt;
	    &lt;/gml:LinearRing&gt;
	  &lt;/gml:outerBoundaryIs&gt;
        &lt;/gml:Polygon&gt;
      )	
      </programlisting><para>It returns 598 <sgmltag>gml:Polygon</sgmltag>s in just a little bit more than 400 ms. Here is what they look like:</para><screenshot><graphic fileref="resources/within.jpg"/></screenshot><para>The last query of this session is just to demonstrate some interesting capabilities of the spatial functions:</para><programlisting format="linespecific">
      declare namespace gml = "http://www.opengis.net/gml";
      spatial:buffer(
        &lt;gml:Polygon srsName="osgb:BNG" xmlns:gml='http://www.opengis.net/gml'&gt;
          &lt;gml:outerBoundaryIs&gt;
	    &lt;gml:LinearRing&gt;
	      &lt;gml:coordinates&gt;
	        278200,187600 278400,187600 278400,188000 278200,188000 278200,187600
	      &lt;/gml:coordinates&gt;
	    &lt;/gml:LinearRing&gt;
          &lt;/gml:outerBoundaryIs&gt;
       &lt;/gml:Polygon&gt;,
       500
      )
      </programlisting><para>See the (not so) rounded corners of our 500 metres buffer over Port-Talbot :</para><screenshot><graphic fileref="resources/buffer.jpg"/></screenshot></section><section><title>Facts and thoughts</title><para>As of june 2007, the spatial index is in working condition. It provides an interesting set of functionalities 
	and its performance is satisfactory given the lightweight, non spatially-enabled, database engine that stores
	the <classname>Geometry</classname> objects and their properties. The main objective was to return within the second; we're there.</para><para>Here are still some tentative improvements.</para><para>The first improvement is to plug in the <methodname>getGeometriesForNodes()</methodname> and 
	<methodname>getGeometricPropertyForNodes()</methodname> (in <classname>org.exist.indexing.spatial.AbstractGMLJDBCIndexWorker</classname>) to allow sequence/set optimization.</para><para>Indeed, a query like this one on our test file :</para><programlisting format="linespecific">
declare namespace gml = "http://www.opengis.net/gml";
for $pol in //gml:Polygon
return spatial:getArea($pol)
	</programlisting><para>... returns 5339 items through as many calls to the DB in... 51 seconds on an initialized index ! 
	Intercepting the <parameter>SINGLE_STEP_EXECUTION</parameter> flag when the expression is analyzed would allow 
	to call the 2 above methods rather than their "individual" counterparts, namely <methodname>getGeometryForNode()</methodname> 
	and <methodname>getGeometricPropertyForNode()</methodname>. The expected performance improvement would be
	interesting.</para><para>A second improvement could be to refine the queries on the HSQLDB. Currently, <methodname>search()</methodname>
	(in <classname>org.exist.indexing.spatial.GMLHSQLIndexWorker</classname>) filters the records on the BBox of the
	searched <classname>Geometry</classname>.
	It would also be nice to refine the query on the context nodes and, in particular, on their involved collections 
	and/or documents. The like applies for the HSQL implementation of <methodname>getGeometryForNode()</methodname> 
	and <methodname>getGeometricPropertyForNode()</methodname> too.</para><para>However, we have to be aware that writing such a SQL statement and passing it to the DB server might become
	counter-productive. The idea is then to define some (configurable) threshold values that would refine the query
	on the documents if there are fewer than, say, 10 documents in the context nodeset, and if there are more than 
	10 documents in it,	but less than, say, 15 collections, refine the query on the collection.</para><para>It would be quite easy to determine those threshold values above which writing a long SQL statement and
	passing it to the DB server takes more time than filtering the fectched data.</para><para>We might also consider the field in which the document's URI is stored (<parameter>DOCUMENT_URI</parameter>)
	and possibly split it into two fields, one for the collection and the second one for the document. Of course, having
	indexed integer values here would probably be interesting.</para><para>Having some better algorithms to prefilter the <classname>Geometry</classname>ies could probably help as well
	and, more generally, everything a DB server could bring (caching for instance) should be considered.</para><para>An other improvement would be to introduce some error margins for <classname>Geometry</classname>ies computations
	or BBox ones. The <ulink url="http://www.vividsolutions.com/jts/jtshome.htm">Java Topology Suite</ulink>, widely
	used by Geotools, has all the necessary material for this.</para><para>Another interesting improvement would be to compute and return simplified <classname>Geometry</classname>ies 
	depending of a "hint". Applications might want to return simplified polygons and even points at wider scales. Depending on
	the hint's value, passed in the function's parameters, the index could use the right DB fields in order to work on 
	<emphasis>precomputed</emphasis> simpler (or, more generally, different) entries.</para><para>This is how a hint configuration could look like :</para><programlisting format="linespecific">
  &lt;gml flushAfter="200"&gt;
    &lt;!-- divide the number of vertices by 2 --&gt;
    &lt;hint value="simplify1" method="simplify" parameter="2"/&gt;
    &lt;!-- divide the number of vertices by 4 --&gt;
    &lt;hint value="simplify2" method="simplify" parameter="4"/&gt;
    &lt;!-- reduce to a point --&gt;
    &lt;hint value="simplify3" method="point"/&gt;  
  &lt;/gml&gt;
	</programlisting><para>We should also discuss with the other developers about the opportunity to have a <classname>org.exist.indexing.Index</classname>
	interface in the modularized indexes hierarchy. The abstract class <classname>org.exist.indexing.AbstractIndex</classname>
	provides some nice general-purpose methods and allows <emphasis>static</emphasis> members that are nearly 
	mandatory (like <parameter>ID</parameter>). The like for <classname>org.exist.indexing.StreamListener</classname> 
	versus <classname>org.exist.indexing.AbstractStreamListener</classname>.</para><para>More tests should also be driven. The spatial index has only be tested on one file until now although this
	 file is sizeable. It might be interesting to see how it behaves with unusual geometries like rectangles, 
	 multi-geometries and collections. It might also be interesting to know more about the error margin when 
	 geometries in different SRSes are involved. The accuracy of the referencing libraries available in the 
	 CLASSPATH would play an important role here.</para><para>As always, the code could be written in a more efficient way. There are probably too many <programlisting format="linespecific">
	if (...) ... else if(...) ... else ...
	</programlisting> constructs in the code for instance. Also, we will have to follow Geotools progress to get
	rid of some of the more or less elegant workarounds we've had to implement.</para></section></section></chapter></book>