Content Import Resolution Tool

We can encounter several failures while importing content/data in Asta7. So far, our approach for handling failures has been to throw an exception and abort the import. However, this approach is not user-friendly, as most of the failures are recoverable.

We want a tool to handle imports better to ease the mobility of archives between different Asta7 installations and within the same Asta7. The tool should be based on current functionality in Asta5 and adapted to the challenges we have in Asta7.

This tool will be used only for the Single Archive import from Asta7

Conflicts

The conflicts can be divided into several categories. They are as follows

UNIQUE

Most of the conflicts can be put into this category. This conflict can happen for fields in entities that are marked as UNIQUE. In most cases, this conflict will happen if you try to import an already existing object. An entity can have the following types of unique fields

AMID

Every entity implicitly has a system field called AMID (_amid). This field uniquely identifies an object of a particular entity and project. This field is of type UUID.

Primary Keys

Every entity should have one or more user-defined unique identifiers as well. These fields are marked as primary keys in the entity. These fields can be of any type.

We are considering a single primary key for now

Unique

Entities can have other fields as well which are marked as unique by the user. These can be of any type as well.

FVP Unique

Root entity fields that have a Field Value Pattern associated with them are considered UNIQUE as well.

Example

There is an object of entity A with AMID 821df1d4-3ed8-463e-a9a3-ea841be60f81 in the system. If you try to import this object again or any other object of entity A with the same AMID, this conflict will arise.

There is an entity A which has a field identifier with a field value pattern. The system already has an object of entity A with identifier A-0001. If you try to import any object of entity A with the identifier A-0001 again, this conflict will arise.

Solutions

GENERATE
SET_BY_USER
SET_NULL
FAIL

FK

Entities can have fields that are used to store links to their parent objects. These fields are called foreign key fields. If any foreign key field contains a link to a parent object that does not exist this conflict will trigger.

Not yet supported

Example

Entity A has a sub-entity Entity B. Entity B then will have a field like ref_a which will store the id of its parent object of Entity A. In case you are trying to import an object of Entity B which has a value in the ref_a field, but there is no object of Entity A with that id, then this conflict will arise.

Solutions

SET_NULL
FAIL

REQUIRED

Entities can have fields that are marked as required, meaning there must be a value present for that field. In case an import does not contain any value for any required field this conflict will happen. AMID and primary key fields are implicitly required.

Example

Entity A has a primary key field id. If the import file for entity A contains an object which has no value set for the field id then this conflict will arise.

Solutions

GENERATE
SET_BY_USER
FAIL

TYPE

Each field in an entity has a type (UUID, TEXT, etc.). If the provided value type of a field in the import does not match the field type then this conflict will happen.

Example

Entity A has a field id of type integer. if the import file for entity A contains any value for the field id which is not an integer then this conflict will arise.

Solutions

GENERATE
SET_BY_USER
SET_NULL
FAIL

LENGTH

VARCHAR fields can have an optional length property set. This property dictates how long the text value can be. If any value exceeds the length then this conflict will happen.

Example

Entity A has a varchar field name with length 10. If the import file for entity A contains any value for the field name with a length greater than 10 this conflict will arise.

Solutions

TRIM_EXTRA
SET_BY_USER
SET_NULL
FAIL

Root System Entity UNIQUE

For the UNIQUE conflict of root system entities (AKTOR, TAG, RESTRICTION) a couple of additional solutions are available.

Example

The system already has an AKTOR with identifikator AKT-0001. If you try to import any data that has an AKTOR with identifikator AKT-0001 this conflict will arise.

Solutions

IGNORE
REPLACE_WITH_EXISTING

Solutions

Type	Description	Challenges
GENERATE	Generates a new value based on the field type or using the field value pattern Not available for PK and UNIQUE if field type is not `UUID`	For AMID, replace all `objectReference` For PK, replace all FK How to handle composite PK? Should we generate PK for other types? If so then how?
SET_BY_USER	Input from the user Not applicable for conflicts that are gathered during import	Requires user interaction
SET_NULL	Applicable for nullable or not required fields
SET_NULL_OR_REGENERATE	Set `null` if nullable or generate a value (dummy) based on the field type Applicable for conflicts that are gathered during import
TRIM_EXTRA	Trim the extra part Applicable for `varchar` fields with a length set
IGNORE	Ignore this object and it’s descendants Applicable only for root system entities Highest precedence	Ignore the whole subtree for this object as well
REPLACE_WITH_EXISTING	Ignore this object and it’s descendants but keep the relation objects (AKTOR_RELASJON, TAG_MAPPING, OBJECT_RESTRICTION) Import relation objects but change the relation to connect to the conflicted row 2nd highest precedence	Ignore the whole subtree for this object as well Keep the relation objects but change their FK Should we consider the possible new sub-entity objects? Should we consider the possible changes for this subtree?
FAIL	Throw an error and abort Applicable for conflicts that are gathered during import

Implementation

We can divide the implementation into four key steps.

Execute Normal Import

As before we will try to import the data without any changes. In case of an exception, we will analyze the exception to check if it is a resolvable exception. If resolvable then the import job dialog will show a button to start the resolving process.

Clicking the Resolve button will start the 2nd step.

Conflict Resolution Options

After clicking the Resolve button a new tab will be opened. In the tab, there will be options to choose how do you want to proceed. There will be options present per conflict type. Either you can gather the conflicts beforehand, or gather and solve them while importing, and the third option is to mix both, meaning gather some types beforehand but resolve the rest while importing.

If all the types are set to resolve at import time then the Resolve and Import button will be shown. In this case, the 3rd step is not needed anymore. Otherwise, you can proceed to the 3rd step by clicking the Gather Conflicts button.

Gather and Show Conflicts with Possible Solutions

After clicking the Gather Conflicts button a background job will be started, which will go through the import file and gather all the resolvable conflicts for the chosen types. After gathering all the conflict they will be shown in a table with possible solutions to choose from. The conflicts will be grouped by the entity name and the conflict type. It is also possible to pick a solution for the whole group.

The current limit for gathering conflicts is 10,000.

Resolve and Import

Clicking the Resolve and Import button will begin the 4th and final step. In this step, the import will run again. All the conflicts gathered beforehand and conflicts gathered while running will be resolved based on the chosen solution.

Things To Do

Support Excel import
Support Composite PK
Support FK conflict
Generate other types than UUID
Optimize using temporary table and/or files rather than the memory
Add more types of solutions