<?xml version="1.0" encoding="UTF-8"?>
<!-- $Id: validation.xml 18703 2013-07-15 20:19:15Z dizzzz $ --><book><bookinfo><productname>eXist-db – Open Source Native XML Database</productname><title>XML Validation</title><date>September 2009</date><author><firstname>Dannes</firstname><surname>Wessels</surname><affiliation><address format="linespecific"><email>dizzzz at exist-db.org</email></address></affiliation></author></bookinfo><chapter><title>XML Validation</title><section id="sect1"><title>Introduction</title><para>eXist-db supports implicit and explicit validation of XML documents. Implicit validation can be executed
                automatically when documents are being inserted into the database, explicit validation can be performed
                through the use of provided XQuery extension functions.</para></section><section id="sect2"><title>Implicit validation</title><para>To enable implicit validation, the eXist-db configuration must be changed by editing the file
                    <filename>conf.xml</filename>. The following items must be configured:<itemizedlist><listitem><para>Validation mode</para></listitem><listitem><para>Catalog Entity Resolver</para></listitem></itemizedlist></para><section><title>Validation mode</title><example><title>Default configuration</title><programlisting format="linespecific">
    &lt;validation mode="auto"&gt;
        &lt;entity-resolver&gt;
            &lt;catalog uri="${WEBAPP_HOME}/WEB-INF/catalog.xml" /&gt;
        &lt;/entity-resolver&gt;
    &lt;/validation&gt;</programlisting></example><para>With the parameter <emphasis>mode</emphasis> it is possible to switch on/off the validation
                    capabilities of the (Xerces) XML parser. The possible values are:</para><section><title>yes</title><para>Switch on validation. All XML documents will be validated. Note - If the grammar (XML schema,
                        DTD) document(s) cannot be resolved, the XML document is rejected.</para></section><section><title>no</title><para>Switch off validation. No grammar validation is performed and all well-formed XML documents
                        will be accepted.</para></section><section><title>auto (default)</title><para>Validation of an XML document will be performed based on the content of the document. When a
                        document contains a reference to a grammar (<emphasis>XML schema</emphasis> or
                            <emphasis>DTD</emphasis>) document, the XML parser tries to resolve this grammar and the XML
                        document will be validated against this grammar, just like <emphasis>mode="yes"</emphasis> is
                        configured. Again, if the grammar cannot be resolved, the XML document will be rejected. When
                        the XML document does not contain a reference to a grammar, it will be parsed like
                            <emphasis>mode="no"</emphasis> is configured.</para></section></section><section><title>Catalog Entity Resolver</title><para>All grammars (XML schema, DTD) that are used for implicit validation must be registered with eXist
                    using <ulink url="http://www.oasis-open.org/committees/download.php/14809/xml-catalogs.html">OASIS
                        catalog</ulink> files. These catalog files can be stored on disk and/or in the database itself.
                    In eXist the actual resolving is performed by the apache <ulink url="http://xml.apache.org/commons/components/resolver/">xml-commons resolver</ulink>
                    library.</para><para>It is possible to configure any number of catalog entries in the entity-resolver section of
                    conf.xml . The relative "uri="s are resolved relative to the location of the catalog
                    document.</para><example><title>Catalog stored in database</title><programlisting format="linespecific">
    &lt;validation mode="auto"&gt;
        &lt;entity-resolver&gt;
            &lt;catalog uri="xmldb:exist:///db/grammar/catalog.xml" /&gt;
            &lt;catalog uri="${WEBAPP_HOME}/WEB-INF/catalog.xml" /&gt;
        &lt;/entity-resolver&gt;
    &lt;/validation&gt;</programlisting></example><para>A catalog stored in the database can be addressed by a URL like '<emphasis role="bold">xmldb:exist:///db/mycollection/catalog.xml</emphasis>' (note the 3 leading slashes which imply
                    localhost) or the shorter equivalent '/db/mycollection/catalog.xml'.</para><para>In the preceeding example <emphasis role="bold">${WEBAPP_HOME}</emphasis> is substituted by a
                        <emphasis>file://</emphasis> URL pointing to the 'webapp'-directory of eXist (e.g. '<emphasis role="bold">$EXIST_HOME/webapp/</emphasis>') or the equivalent directory of a deployed WAR file
                    when eXist is deployed in a servlet container (e.g. '<emphasis role="bold">${CATALINA_HOME}/webapps/exist/</emphasis>')</para><para><example><title>Default OASIS catalog file</title><programlisting format="linespecific">
    &lt;?xml version="1.0" encoding="UTF-8"?&gt;
    &lt;catalog xmlns="urn:oasis:names:tc:entity:xmlns:xml:catalog"&gt;
        &lt;public publicId="-//PLAY//EN" uri="entities/play.dtd"/&gt;
        &lt;system systemId="play.dtd" uri="entities/play.dtd"/&gt;
        &lt;system systemId="mondial.dtd" uri="entities/mondial.dtd"/&gt;    
        
        &lt;uri name="http://exist-db.org/samples/shakespeare" uri="entities/play.xsd"/&gt;
        
        &lt;uri name="http://www.w3.org/XML/1998/namespace" uri="entities/xml.xsd"/&gt;
    	&lt;uri name="http://www.w3.org/2001/XMLSchema" uri="entities/XMLSchema.xsd"/&gt;
    
        &lt;uri name="urn:oasis:names:tc:entity:xmlns:xml:catalog" uri="entities/catalog.xsd" /&gt;
    &lt;/catalog&gt;</programlisting></example></para></section><section><title>Collection configuration</title><para>Within the database the validation mode for each individal collection can be configured using
                        <emphasis>collection.xconf</emphasis> documents, in the same way these are used for configuring
                        <ulink url="indexing.xml">indexes</ulink>. These documents need to be stored in
                    '/db/system/config/db/....'.</para><example><title>collection.xconf</title><programlisting format="linespecific">
&lt;?xml version='1.0'?&gt;
&lt;collection xmlns="http://exist-db.org/collection-config/1.0"&gt;
    &lt;validation mode="no"/&gt;
&lt;/collection&gt;</programlisting></example><para>This example xconf file turns the implicit validation off.</para></section></section><section id="sect3"><title>Explicit validation</title><para>Extension functions for validating XML in an XQuery script are provided. Starting with eXist-db
                release 1.4 the following validation options are provided: <itemizedlist><listitem><para>JAXP</para></listitem><listitem><para>JAXV</para></listitem><listitem><para>Jing</para></listitem></itemizedlist>Each of these options are discussed in the following sections. Consult the <ulink url="{fundocs}/view.html?uri=http://exist-db.org/xquery/validation">XQuery Function Documentation</ulink> for detailed functions
                descriptions.</para><section><title>JAXP</title><para>The JAXP validation functions are based on the validation capabilities of the <ulink url="http://java.sun.com/j2se/1.5.0/docs/api/javax/xml/parsers/package-summary.html">javax.xml.parsers</ulink> API. The actual validation is performed by the <ulink url="http://xerces.apache.org/xerces2-j/">Xerces2</ulink> library.</para><para>When parsing an XML document a reference to a grammar (either DTDs or XSDs) is found, then the
                    parser attempts resolve the grammar reference by following either the XSD xsi:schemaLocation,
                    xsi:noNamespaceSchemaLocation hints, the DTD DOCTYPE SYSTEM information, or by outsourcing the
                    retrieval of the grammars to an Xml Catalog resolver. The resolver identifies XSDs by the
                    (target)namespace, DTDs are identified by the PublicId.</para><para>Validation performance is increased through grammar caching; the cached compiled grammars are
                    shared by the implicit validation feature.</para><para>The jaxp() and jaxp-report() functions accept the following parameters: <variablelist><varlistentry><term>$instance</term><listitem><para>The XML instance document, referenced as document node (returned by fn:doc()),
                                    element node, xs:anyURI or as Java file object.</para></listitem></varlistentry><varlistentry><term>$cache-grammars</term><listitem><para>Set this to true() to enable grammar caching.</para></listitem></varlistentry><varlistentry><term>$catalogs</term><listitem><para>One or more OASIS catalog files referenced as xs:anyURI. Depending on the
                                    xs:anyURI a different resolver will be used: <itemizedlist><listitem><para>When an empty sequence is set, the catalog files defined in conf.xml
                                                are used.</para></listitem><listitem><para>If the URI ends with ".xml" the specified catalog is used.</para></listitem><listitem><para>If the URI points to a collection (when the URL ends with "/") the
                                                grammar files are searched in the database using an xquery. XSDs are
                                                found by their targetNamespace attributes, DTDs are found by their
                                                publicId entries in catalog files.</para></listitem></itemizedlist></para></listitem></varlistentry></variablelist></para></section><section><title>JAXV</title><para>The JAXV validation functions are based on the <ulink url="http://java.sun.com/j2se/1.5.0/docs/api/javax/xml/validation/package-summary.html">java.xml.validation</ulink> API which has been introduced in Java 5 to provide a
                    schema-language-independent interface to validation services. Although officially the specification
                    allows use of additional schema languages, only XML schemas can be really used so far. </para><para/><para>The jaxv() and jaxv-report() functions accept the following parameters: <variablelist><varlistentry><term>$instance</term><listitem><para>The XML instance document, referenced as document node (returned by fn:doc()),
                                    element node, xs:anyURI or as Java file object.</para></listitem></varlistentry><varlistentry><term>$grammars</term><listitem><para>One or more grammar files, referenced as document nodes (returned by fn:doc()),
                                    element nodes, xs:anyURI or as Java file objects.</para></listitem></varlistentry></variablelist></para></section><section><title>Jing</title><para>The Jing validation functions are based on James Clark's <ulink url="http://www.thaiopensource.com/relaxng/jing.html">Jing</ulink> library. eXist uses the maintained version
												that is available via <ulink url="http://code.google.com/p/jing-trang/">Google Code</ulink>. The library relies on the <ulink url="http://www.thaiopensource.com/relaxng/api/jing/com/thaiopensource/validate/ValidationDriver.html">com.thaiopensource.validate.ValidationDriver</ulink> which supports a wide range of grammar types: <itemizedlist><listitem><para>XML schema (.xsd)</para></listitem><listitem><para>Namespace-based Validation Dispatching Language (.nvdl)</para></listitem><listitem><para>RelaxNG (.rng and .rnc)</para></listitem><listitem><para>Schematron 1.5 (.sch)</para></listitem></itemizedlist></para><para>The jing() and jing-report() functions accept the following parameters: <variablelist><varlistentry><term>$instance</term><listitem><para>The XML instance document, referenced as document node (returned by fn:doc()),
                                    element node, xs:anyURI or as Java file object.</para></listitem></varlistentry><varlistentry><term>$grammar</term><listitem><para>The grammar file, referenced as document node (returned by fn:doc()), element
                                    node, as xs:anyURI, binary document (returned by util:binary-doc() for RNC files) or
                                    as Java file object.</para></listitem></varlistentry></variablelist></para></section></section><section id="sect3a"><title>Validation report</title><para> The validation report contains the following information: <example><title>Valid document</title><programlisting format="linespecific">
    &lt;?xml version='1.0'?&gt;
    &lt;report&gt;
        &lt;status&gt;valid&lt;/status&gt;
        &lt;namespace&gt;MyNameSpace&lt;/namespace&gt;
        &lt;duration unit="msec"&gt;106&lt;/duration&gt;
    &lt;/report&gt;</programlisting></example><example><title>Invalid document</title><programlisting format="linespecific">
    &lt;?xml version='1.0'?&gt;
    &lt;report&gt;
        &lt;status&gt;invalid&lt;/status&gt;
        &lt;namespace&gt;MyNameSpace&lt;/namespace&gt;
        &lt;duration unit="msec"&gt;39&lt;/duration&gt;
        &lt;message level="Error" line="3" column="20"&gt;cvc-datatype-valid.1.2.1: 'aaaaaaaa' is not a valid value for 'decimal'.&lt;/message&gt;
        &lt;message level="Error" line="3" column="20"&gt;cvc-type.3.1.3: The value 'aaaaaaaa' of element 'c' is not valid.&lt;/message&gt;
    &lt;/report&gt;</programlisting></example><example><title>Exception</title><programlisting format="linespecific">
    &lt;?xml version='1.0'?&gt;
    &lt;report&gt;
        &lt;status&gt;invalid&lt;/status&gt;
        &lt;duration unit="msec"&gt;2&lt;/duration&gt;
        &lt;exception&gt;
            &lt;class&gt;java.net.MalformedURLException&lt;/class&gt;
            &lt;message&gt;unknown protocol: foo&lt;/message&gt;
            &lt;stacktrace&gt;java.net.MalformedURLException: unknown protocol: foo 
            at java.net.URL.&lt;init&gt;(URL.java:574) 
            at java.net.URL.&lt;init&gt;(URL.java:464) 
            at java.net.URL.&lt;init&gt;(URL.java:413) 
            at org.exist.xquery.functions.validation.Shared.getStreamSource(Shared.java:140) 
            at org.exist.xquery.functions.validation.Shared.getInputSource(Shared.java:190) 
            at org.exist.xquery.functions.validation.Parse.eval(Parse.java:179) 
            at org.exist.xquery.BasicFunction.eval(BasicFunction.java:68) 
            at ......
            &lt;/stacktrace&gt;
        &lt;/exception&gt;
    &lt;/report&gt;</programlisting></example></para></section><section id="sect4"><title>Grammar management</title><para>The XML parser (Xerces) compiles all grammar files (dtd, xsd) upon first use. For efficiency reasons
                these compiled grammars are cached and made available for reuse, resulting in a significant increase of
                validation processing performance. However, under certain circumstances (e.g. grammar development) it
                may be desirable to manually clear this cache, for this purpose two grammar management functions are
                provided: <itemizedlist><listitem><para><emphasis>clear-grammar-cache()</emphasis> : removes all cached grammar and returns the
                            number of removed grammar</para></listitem><listitem><para><emphasis>pre-parse-grammar(xs:anyURI)</emphasis> : parses the referenced 
                            grammar and returns the namespace of the parsed XSD.</para></listitem><listitem><para><emphasis>show-grammar-cache()</emphasis> : returns an XML report about all cached
                            grammar</para></listitem></itemizedlist></para><example><title>Cached grammars Report</title><programlisting format="linespecific">
    &lt;?xml version='1.0'?&gt;
    &lt;report&gt;
    &lt;grammar type="http://www.w3.org/2001/XMLSchema"&gt;
        &lt;Namespace&gt;http://www.w3.org/XML/1998/namespace&lt;/Namespace&gt;
        &lt;BaseSystemId&gt;file:/Users/guest/existdb/trunk/webapp//WEB-INF/entities/XMLSchema.xsd&lt;/BaseSystemId&gt;
        &lt;LiteralSystemId&gt;http://www.w3.org/2001/xml.xsd&lt;/LiteralSystemId&gt;
        &lt;ExpandedSystemId&gt;http://www.w3.org/2001/xml.xsd&lt;/ExpandedSystemId&gt;
    &lt;/grammar&gt;
    &lt;grammar type="http://www.w3.org/2001/XMLSchema"&gt;
        &lt;Namespace&gt;http://www.w3.org/2001/XMLSchema&lt;/Namespace&gt;
        &lt;BaseSystemId&gt;file:/Users/guest/existdb/trunk/schema/collection.xconf.xsd&lt;/BaseSystemId&gt;
    &lt;/grammar&gt;
    &lt;/report&gt;</programlisting></example><para>Note: the element <emphasis>BaseSystemId</emphasis> typically does not provide usefull
                information.</para></section><section id="sect4a"><title>Interactive Client</title><para>The interactive shell mode of the <ulink url="java-admin-client.xml">Java Admin Client</ulink> provides a simple
                    <emphasis>validate</emphasis> command that accepts the similar explicit validation arguments.</para></section>
        <!--
        <section id="sect5">
            <title>XML instance examples</title>
            <para>This section provides a number of XML fragments demonstrating the required format
                of the XML documents. Note that a root element should <emphasis>always</emphasis>
                have a reference to a namespace.</para>
            <para>
                <example>
                    <title>namespace</title>
                    <para>Most simple reference to an XML schema. The xmlns info is used by the
                        parser to resolve the grammar document.</para>
                    <programlisting format="linespecific"><![CDATA[
    <?xml version='1.0'?>
    <addressBook xmlns="http://jmvanel.free.fr/xsd/addressBook">
        .....
    </addressBook>]]></programlisting>
                </example>
                <example>
                    <title>schemaLocation</title>
                    <para>xsi:schemaLocation provides additional information to the parser on how to
                        resolve grammar file. According to the XML schema specifications this
                        information is considered to be a hint and might be ignored. eXist will
                        ignore this informaton, the grammar will be resolved like the previous
                        example.</para>
                    <programlisting format="linespecific"><![CDATA[
    <?xml version='1.0'?>
    <addressBook xsi:schemaLocation="http://jmvanel.free.fr/xsd/addressBook http://myshost/schema.xsd" 
                 xmlns="http://jmvanel.free.fr/xsd/addressBook" 
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
        .....
    </addressBook>]]></programlisting>
                </example>
                <example>
                    <title>noNamespaceSchemaLocation</title>
                    <para>Taken from: conf.xml. The xsi:noNamespaceSchemaLocation is honoured by the
                        parser during implicit validation.</para>
                    <programlisting format="linespecific"><![CDATA[
    <?xml version='1.0'?>
    <exist xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
           xsi:noNamespaceSchemaLocation="schema/conf.xsd">
        .....
    </exist>]]></programlisting>
                </example>
                <example>
                    <title>DTD DOCTYPE</title>
                    <para>Taken from 'samples/validation/dtd'. eXist resolves the grammar by
                        searching catalog files for the PUBLIC identifier.</para>
                    <programlisting format="linespecific"><![CDATA[
    <?xml version='1.0'?>
    <!DOCTYPE PLAY PUBLIC "-//VALIDATION//EN" "hamlet.dtd">
    <PLAY>
        .....
    </PLAY>]]></programlisting>
                </example>
            </para>
        </section>--><section id="sect6"><title>Special notes</title><para><itemizedlist><listitem><para>Tomcat has an long standing bug which makes it impossible to register a custom <ulink url="http://java.sun.com/developer/onlineTraining/protocolhandlers/">protocol
                                handler</ulink> (object <ulink url="http://java.sun.com/j2se/1.5.0/docs/api/java/net/URLStreamHandler.html">URLStreamHandler</ulink>) to the JVM. The alternative is to register the object by setting
                            the system property <code>java.protocol.handler.pkgs</code> but this fails as well.</para><para>As a result the validation features are only partly useable in tomcat. There are two
                            altenatives: (1) switch to a recent version of Jetty, or (2) use absolute URLs pointing the
                            the REST interface, e.g.
                                <code>http://localhost:8080/exist/rest/db/mycollection/schema.xsd</code>.</para></listitem><listitem><para>eXist relies heavily on features provided by the Xerces XML parser. Out of the box the
                            eXist izPack installer provides all required jar files. However, when eXist is installed in
                            e.g. Tomcat the required parser libraries need to be copied manually from the eXist
                            lib/endorsed directory into the server '<ulink url="http://tomcat.apache.org/tomcat-5.5-doc/class-loader-howto.html">endorsed</ulink>'
                            directory.</para><para>Required endorsed files: resolver-*.jar xalan-*.jar serializer-*.jar
                            xercesImpl-*.jar</para></listitem><listitem><para>To avoid potential deadlocking it is considered good practice to store XML instance
                            documents and grammar documents in separate collections.</para></listitem><listitem><para>The explicit validation is performed by <ulink url="http://xerces.apache.org/xerces2-j/javadocs/api/javax/xml/validation/package-summary.html">Xerces</ulink> (XML schema, DTD) and by <ulink url="http://www.oxygenxml.com/onvdl.html">oNVDL</ulink> - oXygen XML NVDL implementation based on Jing (XSD, RelaxNG, Schematron and
                            Namespace-based Validation Dispatching Language)</para></listitem></itemizedlist></para></section><section id="sect7"><title>References</title><para><itemizedlist><listitem><para>Apache <ulink url="http://xml.apache.org/commons/components/resolver/">xml-commons
                                resolver</ulink></para></listitem><listitem><para>OASIS <ulink url="http://www.oasis-open.org/committees/entity/">XML Catalog Specification</ulink>
                            V1.1</para></listitem><listitem><para>Xerces <ulink url="http://xerces.apache.org/xerces2-j/faq-grammars.html">caching
                            grammars</ulink>.</para></listitem><listitem><para><ulink url="http://code.google.com/p/jing-trang/">jing-trang</ulink> Schema validation and 
												conversion based on RELAX NG</para></listitem><listitem><para><ulink url="http://en.wikipedia.org/wiki/Namespace-based_Validation_Dispatching_Language">NVDL</ulink> (Namespace-based Validation Dispatching Language)</para></listitem><listitem><para><ulink url="http://en.wikipedia.org/wiki/Schematron">Schematron</ulink></para></listitem><listitem><para><ulink url="http://en.wikipedia.org/wiki/RELAX_NG">Relax NG</ulink></para></listitem></itemizedlist></para></section></chapter></book>