Search and Indexing

Use Cases

There are lots of use cases, some of which are noted below

• Finding archives and/or participants based on alternative names.

• Finding Aksesjons/Tilveksts using the Mottat-fra values.

• Finding archives that have a Serie/Mappe with a particular name.

• Finding archives and/or their descendants based on their creator values.

Indexing entities

As Asta7 is a generic system all the entities in the system are treated equally and all of them get indexed in Elasticsearch separately. But this does not work well for most of the cases. Some entities (like Alternativtnavn, Geografy, etc. in ISADG) should not be indexed separately but rather should be a part of the parent entity.

To mitigate this shortcoming a new option has been added in the entity to make it searchable or not.

Indexing related/inherited system entities

Often it is needed to search for something based on some related system entity (participant, restriction, and tag). But as each entity resides on its own index in Elasticsearch and join/subquery is not possible it has not been possible so far.

To overcome this shortcoming, some options have been added in the entity to make it possible to index the related and even inherited system entities with each object.

Indexing member/descendant entities

As Elasticsearch is a flat document-based database that does not support joining or subqueries the only way to do a search based on member/descendant entities is to index them with the parent. So, every document will have the necessary members/descendants and related system entities indexed with them.

But indexing all the member/descendant entities is not a good option, as it will increase the document size quite substantially, also all of them might not be necessary for searching. So an option has been added to the entity structures for this.

It is also possible to index not only member entities but also descendant entities this way. For example in ISADG, if Arkivdel is selected to be indexed with Arkiv and Serie is selected to be indexed with Arkivdel then the Series under Arkivdels also gets indexed with Arkivs.

Limits

As it is possible to self-link entities in Asta7 (Serie → Serie) in theory it is possible to nest entities infinitely. Moreover the way ES mapping works (Every level of nesting adds a whole bunch of fields to the mapping). So we had to add some limitations to member/descendant indexing.

1. Total nesting limit: 10 (Arkiv → Arkivdel → Serie → Serie → Serie → Stykke → Mappe → Mappe → Mappe → Geografi)

2. Self nesting limit: 2 (A → Aa → Aaa)

3. File nesting limit: 1 (Arkiv → Fil)

4. Nested member limit: 1000

5. ES Total field limit: 5000 (Default was 1000)

6. ES Nested field limit: 500 (Default was 50)

7. ES Nested object limit: 50_000 (Default was 10_000)

Field Limits

1. Level 1: All fields

2. Level 2: All fields for leaf entities (Alternativtnavn, Geografy, etc.), otherwise only required fields (Serie, Stykke)

3. Level below 2: Required fields

Searching

Tree Search

The tree search has been updated with the following changes

• The search field now works with ES simple query string syntax.

• The search will work on all the fields in the entity not just the title fields as before. But the title fields will have a 2x boost.

• This search will do a text search on all fields (Even date, number, and other types). For this, every type of field will also have a text version of it indexed.

• For the ISADG Arkiv and Aktor entities, their Alternativtnavn members' navn field will also be added to the search.

• For the ISADG Aksesjon entity, its Mottar-fra participant's navn field will also be added to the search.

Basic and Advanced Search

We now have a combined basic and advanced search with support for a seamless transition between them. It is possible to work on any mode switch to another mode then come back to the previous mode. Both modes support the same set of fields so the basic search is also quite powerful and can be used to make AND queries. For, more control like OR queries, grouping queries, sorting, and various other kinds of queries the advanced search can be used. For details about the advanced search see Advance/Query Builder Search.

Free Text Search

Just like the tree search, there is a text field here as well for doing a cross-project free text search. This free text search supports the same functionalities as the tree search. The difference is that this will be searching on all the entities not just the root entities.

Project and Entities Select

There is a dropdown to select a project and a multi-select to select entities within that project to narrow down the scope of the search. All the initialized projects that the user has access to will be shown in the project dropdown. Then upon selecting a project all the searchable entities (based on preference) of that project will appear on the entities multi-select.

Search Fields

After selecting a project and one or more entities the search fields will be populated with all the fields (including nested entity fields) of the selected entities based on the preference. Clicking the Search Fields button will toggle a tree-like structure with all the fields. It is possible to select/unselect multiple fields from this tree to further filter the search. After selecting the needed fields hide the tree-select to get a filter like below with the selected fields.

Based on the selected field type a particular type of query/operator would be selected, although there can be other operators available for that field type as well which can be found in the advanced mode. The basic mode will do an AND query. There are many more options available in the advanced mode.

Take a look at Search Examples for some example searches.

Preference

It is also possible to further control which entities and fields to use for search using preference. Only the selected entities and their fields in preference will appear in the search filters. The order of entities and their fields can also be configured in the preference by using drag-drop.

Things To Do

• Need to handle orphan members and system relations during sync.

• More control over which member to include, like Serie should be included with Arkiv but not with Arkivdel.

• Control which field is searchable and/or searchable as a member. This should make the field limitations unnecessary. If not then need to make the field limitations configurable instead of hard-coding.

• Include member’s/descendant's related system entities?

• Although there are limitations on the number of nested members, they are not applied during syncing at the moment. Need to fix this.

• Need to use Asta7 models and properties in Essync instead of duplicating them.

• Multi-level nesting makes the search quite complex and might not be needed for all the projects. Should we consider adding support for flat nesting as well?

• Add support for doing a level-agnostic search using descendant entities.