<?xml version="1.0" encoding="UTF-8"?>
<book><bookinfo><productname>eXist-db – Open Source Native XML Database</productname><title>Package Repository</title><date>October 2012</date></bookinfo><chapter><title>Package Repository</title><para>The eXist-db package repository is a central component in eXist-db 2.0. It makes
                it easy to manage and deploy external packages (.xar archives) which include
                everything they need to run third party XQuery libraries to full applications or
                other XML technology functionality. This document provides technical details on the
                packaging format.</para><para>In previous versions of eXist-db, most applications were split into two parts:</para><itemizedlist><listitem><para>the application code (XQuery modules, HTML pages etc.) residing in the webapp
                    directory on the file system</para></listitem><listitem><para>the data stored inside the database</para></listitem></itemizedlist><para>This split made it difficult to redistribute applications. For larger set ups,
            maintenance easily became tedious. To solve those problems, eXist-db 2.0 introduces the
            concept of self-contained, modular applications which can be deployed into any database
            instance using a standardized packaging format. The eXist-db 2.0 distribution is build around this
            concept: documentation, examples and administration utilities have been moved out of the
            webapp directory and into separate application packages, which can be easily installed
            or removed on demand. The dashboard is now the central hub for managing packages.</para><para>The package repository is based on and extends the <ulink url="http://expath.org/modules/pkg/">EXPath packaging system</ulink>.  The core of
            the EXPath packaging specification has been designed to work across different XQuery
            implementations and is targeted at managing extension libraries (including XQuery, Java
            or XSLT code modules). eXist-db extends this core by adding a facility for the automatic
            deployment of entire applications into the database. </para><para>eXist-db packages may fall into one of the following categories:</para><itemizedlist><listitem><para><emphasis role="bold">Applications</emphasis> containing application code,
                    HTML views, associated services, resources and data. An application always has a
                    web interface which can be displayed - for example - if the user clicks on the
                    application icon in the dashboard.</para></listitem><listitem><para><emphasis role="bold">Resource packages</emphasis> containing only data or
                    resources used by other applications, e.g. javascript libraries shared by
                    several application packages. A resource package has no web view, but needs to
                    be deployed into the database.</para></listitem><listitem><para><emphasis role="bold">Library packages</emphasis> providing a set of XQuery
                    library modules to be registered with eXist-db and used by other packages. A
                    library package may also contain Java jar archives to be loaded into the
                    eXist-db classpath. It has no web view and is not deployed into the
                    database.</para></listitem></itemizedlist><para>Those categories are not exclusive: an application may also include resources and
            XQuery libraries and is not required to move those into separate packages.</para><section><title>Creating Packages</title><para>Creating new packages is fairly easy if you use eXide. This is described in the
                    <ulink url="development-starter.xml">web development starter</ulink> document.
                The following sections will cover some details of the packaging format from a more
                technical perspective.</para></section><section><title>EXPath Packaging Format</title><para>An EXPath package is essentially an archive file in ZIP format. By convention, the
                file name extension of the package is <filename>.xar</filename>. The archive
                    <emphasis>must</emphasis> contain two XML descriptor files in the root
                directory: <filename>expath-pkg.xml</filename> and
                <filename>repo.xml</filename>:</para><variablelist><varlistentry><term><filename>expath-pkg.xml</filename></term><listitem><para>This is the standard EXPath descriptor as defined by the EXPath
                            specification. It specifies the unique name of the package, lists
                            dependencies and any library modules to register globally.</para></listitem></varlistentry><varlistentry><term><filename>repo.xml</filename></term><listitem><para>The eXist-db specific deployment descriptor: it contains additional
                            metadata about the package and controls how it will be deployed into the
                            database.</para></listitem></varlistentry></variablelist><para>Though library packages do not really need <filename>repo.xml</filename>, we
                recommend to always provide both for better tool integration.</para><section><title>Descriptors: expath-pkg.xml</title><para>As an example, the EXPath descriptor for the documentation app is shown
                    below:</para><programlisting language="xml">&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;package xmlns="http://expath.org/ns/pkg" name="http://exist-db.org/apps/doc" abbrev="doc" version="0.1" spec="1.0"&gt;
    &lt;title&gt;eXist-db Documentation App&lt;/title&gt;
    &lt;dependency package="http://exist-db.org/apps/shared"/&gt;
&lt;/package&gt;</programlisting><para>The schema of this file is documented in the specification. In short, the
                    attributes are as follows:</para><variablelist><varlistentry><term>name</term><listitem><para>an URI used as a unique identifier for the package. The URI does
                                not need to point to an existing web site. The package repository
                                will use this URI to identify a package within the system.</para></listitem></varlistentry><varlistentry><term>abbrev</term><listitem><para>a short abbreviation for the package. This will be used as part of
                                the file name for the <filename>.xar</filename>. We thus recommend
                                to choose a short, simple name without spaces or punctuation
                                characters.</para></listitem></varlistentry><varlistentry><term>version</term><listitem><para>the version of the package: allows the package manager to
                                determine if newer versions of the same package are
                                available.</para></listitem></varlistentry><varlistentry><term>spec</term><listitem><para> the version of the packaging specification the package conforms
                                to. Always "1.0" for the current specification.</para></listitem></varlistentry><varlistentry><term>title</term><listitem><para>a descriptive title to display to the user, e.g. in the
                                dashboard</para></listitem></varlistentry></variablelist></section><section><title>Dependency Management</title><para>As shown above, a package may depend on one or more other packages. The
                    package manager in the dashboard will resolve dependencies before deployment.
                    Dependant packages will be installed automatically from the public repository.
                    It is an error if a dependency cannot be resolved.</para><para>A dependency on another package is defined by reference to the unique name of
                    the other package (as given in the name attribute of the expath-pkg.xml
                    descriptor):</para><programlisting language="xml">&lt;dependency package="http://exist-db.org/apps/shared"/&gt;</programlisting><para>It is also possible to create a dependancy on a specific version. This can be
                    done by adding either of the attributes: version, semver, semver-min,
                    semver-max. The attributes are mutually exclusive, except for semver-min and
                    semver-max, which may appear together.</para><variablelist><varlistentry><term>version</term><listitem><para>A simple version string which has to exactly match the version
                                string of the package to install.</para></listitem></varlistentry><varlistentry><term>semver</term><listitem><para>A "semantic" version string: the version number has to follow the
                                scheme "x.x.x".  Selects the highest version in the range of
                                versions starting with semver. For example, if semver is "1.2", a
                                package with version "1.2.3" will be selected because it is in the
                                1.2 release series. Likewise, if semver is "1", any package with a
                                version starting with 1 will be chosen.</para></listitem></varlistentry><varlistentry><term>semver-min</term><listitem><para>Defines a minimal required version according to the semver
                                scheme.</para></listitem></varlistentry><varlistentry><term>semver-max</term><listitem><para>Maximum version allowed.</para></listitem></varlistentry></variablelist><para>We definitely recommend to prefer semver, semver-min and semver-max where
                    possible.</para><para>In addition to dependencies on other packages, it is also possible to require
                    a certain eXist-db version for a package. However, this is currently not
                    enforced because it is difficult to determine versions reliably.  The dashboard
                    will display a hint to the user though, informing him about the
                    requirement.</para><para>To require a specifc eXist-db version, include a processor dependency in your
                    descriptor:</para><synopsis language="xml">&lt;dependency processor="eXist-db" version="trunk &amp;gt; rev 18070"/&gt;</synopsis></section><section><title>Library Modules</title><para>A package may list one or more library modules to register with eXist-db. The
                    registered modules will become globally available within the eXist-db instance
                    and may be used by other packages without knowing where the module code is
                    stored. For example, the following descriptor registers the module functx.xql
                    using the given namespace: </para><programlisting language="xml">&lt;package xmlns="http://expath.org/ns/pkg"
         name="http://www.functx.com"
         abbrev="functx"
         version="1.0"
         spec="1.0"&gt;

   &lt;title&gt;FunctX library&lt;/title&gt;

   &lt;xquery&gt;
      &lt;namespace&gt;http://www.functx.com&lt;/namespace&gt;
      &lt;file&gt;functx.xql&lt;/file&gt;
   &lt;/xquery&gt;

&lt;/package&gt;</programlisting><para>The namespace has to correspond to the namespace defined in the module
                    declaration of the XQuery module. The file should be placed into a subdirectory
                    of the .xar archive, named "content". The structure of the .xar for the functx
                    library would thus look as follows:</para><programlisting>/expath-pkg.xml
/repo.xml
/content/functx.xql</programlisting><para>Only XQuery files which are registered in expath-pkg.xml need to go into the
                    special directory. You are free to keep other XQuery files wherever you want.
                    Also, XQuery resources which are only used by the application should
                        <emphasis>not</emphasis> be registered (to avoid messing up the global
                    context). Registering a module does only make sense for libraries which will
                    likely be used by several applications.</para><para>After installing the package, you should be able to use the registered XQuery
                    modules from anywhere within the database instance without knowing the exact
                    import path. Thus the following import statement will be sufficient to import
                    the functx module:</para><programlisting language="xquery">import module namespace functx="http://www.functx.com";

functx:capitalize-first('hello')</programlisting></section><section><title>Java Libraries</title><para>eXist-db also supports XQuery extension modules written in Java. They require
                    a slightly different mechanism for integration into a <filename>.xar</filename>
                    package. This is an extension to the standard EXPath format and should thus go
                    into a separate file, named <filename>exist.xml</filename>. As an example, the
                    exist.xml descriptor of the cryptographic extension module is shown
                    below:</para><programlisting language="xml">&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;package xmlns="http://exist-db.org/ns/expath-pkg"&gt;
    &lt;jar&gt;expath-crypto.jar&lt;/jar&gt;
    &lt;java&gt;
        &lt;namespace&gt;http://expath.org/ns/crypto&lt;/namespace&gt;
        &lt;class&gt;org.expath.exist.crypto.ExistExpathCryptoModule&lt;/class&gt;
    &lt;/java&gt;
&lt;/package&gt;</programlisting><para>The descriptor may contain one or more jar elements, each pointing to a Java
                        <filename>.jar</filename> archive to be installed. Arbitrary jars can be
                    listed here: they do not need to be XQuery extension modules.  Again, the jar
                    files should be placed into a subdirectory of the .xar with the same name as the
                    abbreviation of the package.</para><para>All jars will be dynamically added to eXist-db's class loader and become
                    immediately available after deploying a package. A restart of eXist-db is not
                    required!</para><para>The java element registers an XQuery extension module written in Java. This is
                    similar to the xquery element discussed above, except that the namespace is
                    mapped to a Java class instead of an XQuery file. The Java class should point to
                    the <classname>Module</classname> class which defines the module.</para></section></section><section><title>The repo.xml Deployment Descriptor</title><para>The deployment descriptor contains additional metadata and defines how the package
                will be installed into an eXist-db database instance. An example is given
                below:</para><programlisting language="xml">&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;meta xmlns="http://exist-db.org/xquery/repo"&gt;
    &lt;description&gt;eXist 2.0 documentation application.&lt;/description&gt;
    &lt;author&gt;Joe Wicentowski&lt;/author&gt;
    &lt;author&gt;Wolfgang Meier&lt;/author&gt;
    &lt;website&gt;http://exist-db.org&lt;/website&gt;
    &lt;status&gt;alpha&lt;/status&gt;
    &lt;license&gt;GNU-LGPL&lt;/license&gt;
    &lt;copyright&gt;true&lt;/copyright&gt;
    &lt;type&gt;application&lt;/type&gt;
    &lt;target&gt;doc&lt;/target&gt;
    &lt;prepare&gt;pre-install.xql&lt;/prepare&gt;
    &lt;finish/&gt;
    &lt;permissions user="admin" password="" group="dba" mode="rw-rw-r--"/&gt;
    &lt;deployed&gt;2012-10-28T23:15:39.646+01:00&lt;/deployed&gt;
&lt;/meta&gt;</programlisting><para>The two settings: <code>type</code> and <code>target</code> determine how a
                package is handled by the installer:</para><informaltable frame="void" role="table"><col width="33%"/><col width="33%"/><col width="33%"/><thead><tr><th>Type of package</th><th>type</th><th>target</th><th/></tr></thead><tbody><tr><td class="">Application package</td><td>application</td><td>specified</td></tr><tr><td>Resource package</td><td>library</td><td>specified</td></tr><tr><td>Library package</td><td>library</td><td>not specified</td></tr></tbody></informaltable><para>So an application package has <code>type</code> set to "application" and specifies
                a <code>target</code> because it needs to be deployed into the database. Contrary to
                this, a library package only registers XQuery or other modules, but no resources
                need to be stored into the db, so <code>target</code> is empty.</para><para>The general metadata fields should not need to be explained. The relevant elements
                are:</para><variablelist><varlistentry><term>type</term><listitem><para>Should be set to either "application" or "library". We assume a
                            library has no GUI (i.e. no HTML view). A library will thus not be shown
                            on the main dashboard page, which only lists applications.</para></listitem></varlistentry><varlistentry><term>target</term><listitem><para>Specifies the collection where the contents of the package will be
                            stored to. Top-level files in the package will end up in this
                            collection, resources in sub directories will go into sub collections.
                            Please note that the target collection can be changed by the package
                            manager during install. It is just a recommendation, not a
                            requirement.</para><para>The collection path should always be relative to the repository root
                            collection defined in the configuration.</para></listitem></varlistentry><varlistentry><term>permissions</term><listitem><para>The permissions to use when uploading package contents. All resources and collections will be owned by the specified
                            user and permissions will be changed to those given in <option>mode</option>. If the user does not exist, the deploy function 
                            will try to create it, using the password specified in attribute <option>password</option>.</para><para>Concerning permissions, the execute ("x") flag will be set
                            automatically on all XQuery files in addition to the default permissions
                            defined in the descriptor. For more control over permissions, use a
                            post-install XQuery script (see element "finish" below).</para></listitem></varlistentry><varlistentry><term>prepare</term><listitem><para>Points to an XQuery script inside the root of the package archive,
                            which will be executed before any package data is uploaded to the
                            database. By convention the XQuery script should be called
                                <filename>pre-install.xql</filename>, though this is not a
                            requirement.</para><para>If you create a package via eXide, it will generate a default
                                <filename>pre-install.xql</filename> which uploads the default
                            collection configuration to the system collection. This needs to be done
                            before deployment to guarantee that index definitions are applied when
                            data is uploaded to the db.</para><para>The target collection, the file system path to the current package
                            directory and eXist's home directory are passed to the script as
                            external variables:</para><synopsis language="xquery">(: file path pointing to the exist installation directory :)
declare variable $home external;
(: path to the directory containing the unpacked .xar package :)
declare variable $dir external;
(: the target collection into which the app is deployed :)
declare variable $target external;</synopsis><para>The script may use those variables to read files contained in the
                            package.</para></listitem></varlistentry><varlistentry><term>finish</term><listitem><para>Like <sgmltag>prepare</sgmltag>, this element should point to an
                            XQuery script, which will be executed <emphasis>after</emphasis> all
                            data has been uploaded to the database. It receives the same external
                            variables as the prepare script. The convention is to name the script
                                <filename>post-install.xql</filename>.</para><para>Use the finish trigger to run additional tasks or move data into
                            different collections. For example, the XQuery function documentation
                            app runs an indexing task from the finish trigger to extract
                            documentation from all XQuery modules known to the db at the
                            time.</para></listitem></varlistentry><varlistentry><term>deployed</term><listitem><para>This element will be set automatically when the package is deployed
                            into a database instance. It is used by eXide to track changes and
                            should not need to be specified in the original repo.xml
                            descriptor.</para></listitem></varlistentry></variablelist></section><section><title>Configuring the repository root</title><para>The root collection for deployed packages can be configured in
                    <filename>conf.xml</filename>:</para><programlisting language="xml">&lt;repository root="/db/apps"/&gt;</programlisting><para>The install location specified in the <option>target</option> element of
                    <filename>repo.xml</filename> will always be relative to this root
                collection.</para><para>eXist-db's URL rewriting is by default configured to map any path starting with
                    <code>/apps</code> to the repository root collection. Check
                    <filename>webapp/WEB-INF/controller-config.xml</filename> and the <ulink url="http://localhost:8080/exist/apps/doc/urlrewrite.xml">URL rewriting
                    documentation</ulink>.</para></section><section><title>Programmatically installing packages</title><para>The <option>repo</option> XQuery module provides a number of functions to
                programmatically install, remove or inspect packages. The dashboard package manager
                relies on the same functions.</para><para>The module distinguishes between <emphasis>installation</emphasis> and
                    <emphasis>deployment</emphasis> steps. The reason for this distinction is: while
                the installation process is standardized by the expath packaging specification, the
                deployment step is implementation defined and specific to eXist-db.
                    <emphasis>installation</emphasis> will register a package with the expath
                packaging system, but not copy anything into the database. <emphasis>Deployment
                </emphasis> will deploy the application into the database as specified by the
                    <filename>repo.xml</filename> descriptor.</para><para>The most convenient way to install a package are the
                    <code>repo:install-and-deploy</code> and
                    <code>repo:install-and-deploy-from-db</code> functions. repo:install-and-deploy
                downloads the specified package from a public repository. For example, one can
                install the eXist-db demo apps using the following call:</para><synopsis language="xquery">repo:install-and-deploy("http://exist-db.org/apps/demo", "0.2.2", "http://demo.exist-db.org/exist/apps/public-repo/modules/find.xql")</synopsis><para>The first parameter denotes the unique name of the package to install. The second
                may contain a specific version or the empty sequence. The third parameter is the URI
                for the public repository API. The function call will download, install and deploy
                the package as well as any dependencies it defines. If the installation succeeded,
                an element will be returned to indicate the target collection into which the package
                was deployed.</para><para>The <code>repo:install-and-deploy-from-db</code> function works in a similar way,
                but reads the package data to install from a resource stored in the database.</para><para>To uninstall a package, you should first call <code>repo:undeploy</code>, followed
                by <code>repo:remove</code>, e.g.:</para><programlisting language="xquery">repo:undeploy("http://exist-db.org/apps/demo"), repo:remove("http://exist-db.org/apps/demo")</programlisting><para>To list all installed packages, call <code>repo:list</code>, which will return a
                the unique name of every installed package. </para></section><section><title>Running your own repository</title><para>You can run your private repository and install packages from it, e.g. to
                distribute applications to your customers. The eXist-db repository is implemented by
                the application package <code>http://exist-db.org/apps/public-repo</code>. The code
                can be downloaded from the <ulink url="https://sourceforge.net/p/exist/code/18144/tree/apps/public-repo/">eXist-db
                    SVN</ulink>.</para><para>Once you have build and installed the app, you can upload the package xars to
                distribute into the collection <filename>public-repo/public</filename>. To make the
                uploaded xars available, run the query
                    <filename>public-repo/modules/update.xql</filename> once as an admin user. This
                will create a document <filename>apps.xml</filename> in
                    <filename>public-repo/public</filename>.</para></section><section><title>General considerations when writing a package</title><para>Packages should be portable and should thus not make any assumptions about the
                collection path in which they will be installed. In general it is best to use
                relative paths throughout XQuery modules, in particular for import statements. Just
                as a reminder: a relative path in an "import module namespace..." expression is
                always relative to the XQuery which contains the import.</para><para>If an XQuery needs to access data provided by another package, it should locate
                the other package by its package name and not by using an absolute collection path
                which may change in the future. For example, if an application requires access to
                data stored in another package called "data-pkg", it could define a variable to
                point to the correct collection as follows:</para><programlisting language="xquery">declare variable $local:data-pkg-collection :=
    let $descriptor := 
        collection(repo:get-root())//expath:package[@name = "http://foo.com/data-pkg"]
    return
        util:collection-name($descriptor)
;

(: Query data provided by data-pkg :)
collection($local:data-pkg-collection)//foo</programlisting></section></chapter></book>