Importing and Exporting Audit Trail Information (Technology Preview)

Archiving the Audit Trail for production environments is a mandatory task that needs to be done on a regular basis to guarantee a long term well performing IPP runtime. Infinity provides an enhanced archiving functionality including export, import, unload and load of Audit Trail information.

Note
This feature is customer technology preview only. It may not be functionally complete and is not intended for production use.

Configuration of the ZIP Archiver

To enable the functionality, you have to configure the ZIP archiver by adding the following the files at engine-core/meta-inf/services:

Advantages of the IPP Audit Trail Archive Functionality

The Infinity archive functionality has the following advantages:

Exporting Process Instances from an Audit Trail

The export functionality transfers process instances into a specific location and format. It can be used to archive or dump Audit Trail information.

An export/unload of process instances can be done:

Variants of Exporting Audit Trail Information

The following two variants of exporting process instances from an Audit Trail can be used:

Note that active process instances can only be dumped. They are not considered for a backup or archive runs.

Archiving Audit Trail Information

When exporting via the archive variant, Audit Trail information is extracted from the database and stored in an archive structure, which could be a file system or database. Hereby Audit Trail information is deleted from the database. A UUID is generated which uniquely identifies the exported process instance (see Process Instance UUID) and stored in the index.

Archives are structured by timestamps (completion / last modification timestamps folders). Each backup or archive run creates at least one ZIP file. For backups or archive on process completion a different approach is taken. All dependent entities of a root process instance (complete process closure) are archived. Transient process data (data that is available as long as process instances are active) will not be archived.

An IPP archive can be exported completely (dump also includes active process instances) or partially

Dumping Audit Trail Information

By exporting via dump, Process instances are extracted from the database and stored in an archive structure (file system) to be specified via parameter. This is the location where to dump the extracted data. Hereby Audit Trail information is not deleted from the database. A UUID is generated which uniquely identifies the exported process instance (see Process Instance UUID) and stored in the index. Dumping Audit Trail information is only supported for file-based Archive Manager. The dump process is not incremental. A dump will include previously exported process instances. If a dump operation is run more than once, previously exported and dumped processes will be included again.

Dumping process instances having links

If you dump process instances with links, the linked process instances are always added to the current batch.

Provided Tools for Exporting Audit Trail Information

The export can be triggered via one of the following tools:

Export Filtering

On export data can be filtered by the following criteria:

Exporting Documents

Optionally documents attached to process instances can be exported together with the process instances they belong to. Only documents with process context are considered for an export, e.g. DocumentType and Process Attachments.

The document lookup is done after the process instance has been stored in the archive. Meta information are separately stored with the document.

It is possible to export the latest document version only or all revisions. Note that only those revisions that existed when the process instance was started, activated and finally terminated are exported!

The documents are deleted from the Jackrabbit repository for the archive use case and kept there for the dump use case.

Exporting Process Instances on Completion

The export of process instances on completion is only supported for the archive use case. That means on process instance completion the process instance is moved into the archive and deleted from the active Audit Trail.

The automatic archive on complete is only happening if the root process instance completes. It does not have any effect on sub-process instance completion.

Configuration

The Eclipse modeler as well as the Browser modeler can be configured via a checkbox for archiving process instances on completion.

If this checkbox is selected the process instance will automatically be archived in the following cases:

When archiving processes with links, the linked process instances are either added to the export batch or removed if linked instances are not terminated yet, no matter what the filters for targeted instances are.

Note
This behavior can lead to the situation where same process instances can be exported multiple times in different batches if a link causes the process instances to be linked across filter criteria (e.g. the same day filter). Thus custom archive writer implementations need to handle writing same process instances correctly.
Also note that for dump operations the process instance may have been modified in the mean time.

Archiving operation with linked process instances

The following applies if process instances which have linked process instances are archived:

Dumping operation with linked process instances

If process instances which have linked process instances are dumped, the linked process instances are always added to the current batch.

Importing Process Instances from an Audit Trail

The Import functionality can be used to import processes from a dump or an archive into any compatible environment. A compatible environment has the same IPP version installed and the same set of models deployed.

If a process instance is imported that already exists, e.g. its process instance UUID (ExportProcessID) exists in the database, it will be ignored from the import.

An import of process instances can be done:

Process instances loaded from an IPP archive back into an existing Audit Trail are marked as archived by assigning a unique ID. This mechanism is the same when exporting process instances without deleting them from the Audit Trail database, as described in the previous section.

Marked process instances are ignored on recurring backup or archive runs. An archive would delete the instance from the Audit Trail database though. If a dump is imported, the process instances are handled as if they would have never been exported before. Common use cases are migration or de-provisioning.

Documents can be optionally restored together with the process instances they are attached to.

Import Filtering

By default all exported processes of an archive will be imported. Imports can further be filtered by the following parameter:

If filter criteria are provided and no process instances are found matching this criteria nothing will be imported.

In case no filter criteria is provided then all process instances for the current partition will be imported.

Importing Documents

If documents have been exported into the archive it is possible to import documents/revisions together with the import of process instances. Hereby, after each import of a process instance, its documents are moved into the JCR under the same folder structure it was stored before. It will be checked first, if the document already exists. If it exists, the document will not be imported again.

UUIDs Generation

Runtime UUID

Each runtime (Audit Trail of a Infinity installation) gets a UUID assigned. This UUID is generated on the first archive run and stored in the property table.

Process Instance UUID

Each exported process instance gets a UUID assigned. The UUID consists of a runtime UUID, the process instance OID and the model UUID. The UUID is generated on archive and stored as process instance property (ExportProcessID) in the index and when imported again.

Model UUID

The model UUID correlates to the MD5 String of the XPDL. It will be calculated when the model is bootstrapped and hold in memory (ModelManager) at runtime. The XPDL representation of a model gets extracted once the first process instance of a model is exported. The XPDL is extracted from the Audit Trail (CLOB_DATA) and stored in the archive folder under its MD5 value as name.

Each model variant that has been deployed into the Audit Trail can be exported that way, even in scenarios where a model has been overridden. Since the MD5 value is calculated on model bootstrap and stored in the ModelManager at runtime, the lookup if a model has been exported or not should be fast.

Model UUID Example

The following example demonstrates a Model UUID scenario.

  1. Model A is deployed in the Audit Trail
  2. PI1 based on Model A is exported
  3. Model A is extracted from CLOB_TABLE and stored in the root folder of the archive (e.g. 7e716d0e702df0505fc72e2b89467910.xpdl)
  4. Model A gets overridden by Model A'
  5. PI2 based on Model A' is exported
  6. Model A' is extracted from CLOB_TABLE and stored in the root folder of the archive (e.g. 7e716d0e702df0505fc72e2b89467910_NEW.xpdl)

The MD5 Spring is generated on basis of the exported XPDL. This MD5 String should be part of the UUID assigned to each exported process instance (see Process Instance UUID). The MD5 String is also the UUID of the model itself.

This way each process instance can be exactly associated to a model and be evaluated against during import. An import will fail if the destination Audit Trail does not contain the model with the model UUID of the process instance to be imported. Storing the XPDLs in the archive would also allow to automatically deploy models during process instance import.

Use Cases

Exporting Root Process Instances by OID into a File

If Audit Trail information is exported the original information stays in the Audit Trail. In case the process instance is exported in order to backup it, the instance will be marked with a unique ID that indicates into which archive the instance has been backed up to.

The unique ID will be stored in the PROC_INST_PROPERTY table. It contains the archive ID as well as an UUID that would identify the instance globally. The ID will also be stored in the archive.

Process instances with an ID are not considered for backup jobs again if the archive IDs are matching.

Unloading Process Instances by OID into a File

An unload of process instances deletes the corresponding Audit Trail information from the database.

On unload the process instance gets a unique ID which contains the archive ID as well as a UUID that would identify the instance globally. Since the process instance is deleted from the Audit Trail, the ID is only stored in the archive.

Exporting Process Instance(s) by Business Identifier to the File System

You have the option to export/unload Audit Trail information by business identifier such as Contract ID or Customer ID, etc.

Example use cases are the following:

Key descriptors can be set in order to find exported Audit Trail information by business identifier. The business identifier can be set as export filter.

For example, you can use the filter in the command line tool:

-user motu -password motu export -keyDescriptor customerID=XYZ

The filter by key descriptor can be used alone or in combination of other filters such as model or data range filter.

Importing Process Instance(s) by Business Identifier from the File System

You can import/load Audit Trail information by business identifier such as the Contract ID or Customer ID, etc.

Example use cases are:

On export key descriptors are added to the index in order to find the exported Audit Trail information by business identifier. This index allows to find all instances with a certain business identifier value on import.

The export API / command line allows to use business identifier as export filter, for example:

-user motu -password motu import -keyDescriptor customerID=XYZ

The filter by key descriptor can be used alone or in combination of other filters such as model or data range filter.