Content Import Resolution Tool

We can encounter several failures while importing content/data in Asta7. So far our approach for handling failures has been to throw an exception and abort the import. But this approach is not user-friendly as most of the failures are recoverable.

We want a tool to handle imports better to ease the mobility of archives between different Asta7 installations and within the same Asta7. The tool should be based on current functionality in Asta5 and adapted to the challenges we have in Asta7.

This tool will be used only for the Single Archive import from Asta7

Failures and Possible Solutions

Normal Entities

Conflict	Solutions	Challenges	Solutions(NO)
AMID An object with this AMID already exists	Genarate	Replace all `objectReference`	Erstatt alle objektreferanser
PK An object with this primary key already exists Single PK only for now	Genarate if UUID	Replace all `FK` How to handle composite PK? Should we generate other types? If so then how?	Erstatt alle fremmednøkler
	Input from the user	Requires user interaction
Unique An object with this value for this field already exists	Genarate if UUID	Should we generate other types? If so then how?
	Input from the user	Requires user interaction
	Set `null` if nullable
FVP Unique An object with this value for this field already exists Root entities only for now	Genarate
	Input from the user	Requires user interaction
	Set `null` if nullable
FK The parent object not found Not yet supported	Set `null` if nullable
Required No value present for this required field	Genarate	Generate dummy values if not UUID or FVP
Required No value present for this required field	Input from the user	Requires user interaction
Wrong Type Value type not compatible with the field type	Genarate	Generate dummy values if not UUID or FVP
	Input from the user	Requires user interaction
	Set `null` if nullable
Wrong Length Text length is longer than the limit	Trim extra
	Input from the user	Requires user interaction
	Set `null` if nullable

System Entities

For this section, we need to consider the system root entities like AKTOR, TAG, etc. and ignore AMID/PK conflicts for all sub-entities. We have added options for handling AMID/PK/Unique/FVP Unique conflicts for the system root entities.

Failure

Solutions

Challenges

AMID/PK/Unique/FVP Unique conflict

Ignore

Ignore the whole subtree for this row as well
Highest precedence

Replace with the conflicted row

Replace all objectReference and FK as well
Keep the relation objects but change their FK
2nd highest precedence
Should we consider the possible new sub-entity objects?
Should we consider the possible changes for this subtree?

Replace with a chosen row (Maybe later)

Implementation

We can divide the implementation into four key steps.

Execute Normal Import

As before we will try to import the data without any changes. In case of an exception, we will analyze the exception to check if it is a resolvable exception. If resolvable then the import job dialog will show a button to start the resolving process.

Clicking the Resolve button will start the 2nd step.

Conflict Resolution Options

After clicking the Resolve button a new tab will be opened. In the tab, there will be options to choose how do you want to proceed. There will be options present per conflict type. Either you can gather the conflicts beforehand, or gather and solve them while importing, and the third option is to mix both, meaning gather some types beforehand but resolve the rest while importing.

Conflict Type	Default Solutions (Used while importing)	Description
AMID	Generate	Generate a new UUID
AMID	Fail	Throw error and abort
PK	Generate	Generate a new UUID Available if all are UUID
PK	Fail	Throw error and abort
Unique	Generate	Generate a new UUID Available if all are UUID
Unique	Fail	Throw error and abort
FVP Unique	Generate	Generate a new FVP
FVP Unique	Fail	Throw error and abort
FK (Not supported yet)	Set Null	Set `null` if nullable
FK (Not supported yet)	Fail	Throw error and abort
Required	Generate	Generate a dummy value
Required	Fail	Throw error and abort
Wrong Type	Set Null or Generate	Set `null` if nullable or generate a dummy value
Wrong Type	Fail	Throw error and abort
Wrong Length	Trim Extra	Trim the extra part
Wrong Length	Fail	Throw error and abort
System Entity	Generate	Generate a new UUID or FVP
	Ignore	Ignore this row and it’s subtree
	Replace with the conflicted row	Ignore this row and it’s subtree (Without the relation objects) Import relation objectss but change the relation to connect to the conflicted row
	Fail	Throw error and abort

If all the types are set to resolve at import time then the Resolve and Import button will be shown. In this case, the 3rd step is not needed anymore. Otherwise, you can proceed to the 3rd step by clicking the Gather Conflicts button.

Gather and Show Conflicts with Possible Solutions

After clicking the Gather Conflicts button a background job will be started, which will go through the import file and gather all the resolvable conflicts for the chosen types. After gathering all the conflict they will be shown in a table with possible solutions to choose from. The conflicts will be grouped by the entity name and the conflict type. It is also possible to pick a solution for the whole group.

The current limit for gathering conflicts is 10,000.

Resolve and Import

Clicking the Resolve and Import button will begin the 4th and final step. In this step, the import will run again. All the conflicts gathered beforehand and conflicts gathered while running will be resolved based on the chosen solution.

Things To Do

Support Excel import
Support Composite PK
Support FK conflict
Generate other types than UUID
Optimize using temporary table and/or files rather than the memory
Add more types of solutions