Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 17 Next »

We can encounter several failures while importing content/data in Asta7. So far our approach for handling failures has been to throw an exception and abort the import. But this approach is not user-friendly as most of the failures are recoverable.

We want a tool to handle imports better to ease the mobility of archives between different Asta7 installations and within the same Asta7. The tool should be based on current functionality in Asta5 and adapted to the challenges we have in Asta7.

This tool will be used only for the Single Archive import from Asta7

Conflicts

Type

Description

Solutions

AMID

  • An object with this AMID already exists

  • GENERATE

  • FAIL

PK

  • An object with this primary key already exists

  • Single PK only for now

  • GENERATE

  • SET_BY_USER

  • FAIL

UNIQUE

  • An object with this value for this field already exists

  • GENERATE

  • SET_BY_USER

  • SET_NULL

  • FAIL

FVP_UNIQUE

  • Applicable for fields with field value pattern

  • An object with this value for this field already exists

  • Root entities only for now

  • GENERATE

  • SET_BY_USER

  • SET_NULL

  • FAIL

FK

  • The parent object referred by this field not found

  • Not yet supported

  • SET_NULL

  • FAIL

REQUIRED

  • No value present for this required field

  • GENERATE

  • SET_BY_USER

  • FAIL

TYPE

  • Value type not compatible with the field type

  • GENERATE

  • SET_BY_USER

  • SET_NULL

  • FAIL

LENGTH

  • Applicable for varchar fields with a length set

  • Text length is longer than the limit

  • TRIM_EXTRA

  • SET_BY_USER

  • SET_NULL

  • FAIL

Root System Entity

  • AMID

  • PK

  • UNIQUE

  • FVP_UNIQUE

  • Applicable for root system entities (AKTOR, TAG, RESTRICTION)

  • IGNORE

  • REPLACE_WITH_CONFLICTED

Solutions

Type

Description

Challenges

GENERATE

  • Generates a new value based on the field type or using the field value pattern

  • Not available for PK and UNIQUE if field type is not UUID

  • For AMID, replace all objectReference

  • For PK, replace all FK

  • How to handle composite PK?

  • Should we generate PK for other types? If so then how?

SET_BY_USER

  • Input from the user

  • Not applicable for conflicts that are gathered during import

  • Requires user interaction

SET_NULL

  • Applicable for nullable or not required fields

SET_NULL_OR_REGENERATE

  • Set null if nullable or generate a value (dummy) based on the field type

  • Applicable for conflicts that are gathered during import

TRIM_EXTRA

  • Trim the extra part

  • Applicable for varchar fields with a length set

IGNORE

  • Ignore this object and it’s descendants

  • Applicable only for root system entities

  • Highest precedence

  • Ignore the whole subtree for this object as well

REPLACE_WITH_CONFLICTED

  • Ignore this object and it’s descendants but keep the relation objects (AKTOR_RELASJON, TAG_MAPPING, OBJECT_RESTRICTION)

  • Import relation objects but change the relation to connect to the conflicted row

  • 2nd highest precedence

  • Ignore the whole subtree for this object as well

  • Keep the relation objects but change their FK

  • Should we consider the possible new sub-entity objects?

  • Should we consider the possible changes for this subtree?

FAIL

  • Throw an error and abort

  • Applicable for conflicts that are gathered during import

Implementation

We can divide the implementation into four key steps.

Execute Normal Import

As before we will try to import the data without any changes. In case of an exception, we will analyze the exception to check if it is a resolvable exception. If resolvable then the import job dialog will show a button to start the resolving process.

image-20240822-051919.png

Clicking the Resolve button will start the 2nd step.

Conflict Resolution Options

After clicking the Resolve button a new tab will be opened. In the tab, there will be options to choose how do you want to proceed. There will be options present per conflict type. Either you can gather the conflicts beforehand, or gather and solve them while importing, and the third option is to mix both, meaning gather some types beforehand but resolve the rest while importing.

image-20240829-111207.png

If all the types are set to resolve at import time then the Resolve and Import button will be shown. In this case, the 3rd step is not needed anymore. Otherwise, you can proceed to the 3rd step by clicking the Gather Conflicts button.

Gather and Show Conflicts with Possible Solutions

After clicking the Gather Conflicts button a background job will be started, which will go through the import file and gather all the resolvable conflicts for the chosen types. After gathering all the conflict they will be shown in a table with possible solutions to choose from. The conflicts will be grouped by the entity name and the conflict type. It is also possible to pick a solution for the whole group.

The current limit for gathering conflicts is 10,000.

image-20240829-111502.png

Resolve and Import

Clicking the Resolve and Import button will begin the 4th and final step. In this step, the import will run again. All the conflicts gathered beforehand and conflicts gathered while running will be resolved based on the chosen solution.

Things To Do

  • Support Excel import

  • Support Composite PK

  • Support FK conflict

  • Generate other types than UUID

  • Optimize using temporary table and/or files rather than the memory

  • Add more types of solutions

  • No labels