Use Cases
There are lots of use cases, some of which are noted below
Finding archives and/or participants based on alternative names.
Finding Aksesjons/Tilveksts using the mottat fra values.
Finding archives that have a Serie/Mappe with a particular name.
Finding archives and/or their descendants based on their creator values.
Indexing entities
As Asta7 is a generic system all the entities in the system are treated equally and all of them get indexed in Elasticsearch separately. But this does not work well for most of the cases. Some entities (like Alternativtnavn, Geografy, etc. in ISADG) should not be indexed separately but rather should be a part of the parent entity.
To mitigate this shortcoming a new option has been added in the entity to make it searchable or not.
Indexing related/inherited system entities
Often it is needed to search for something based on some related system entity (participant, restriction, and tag). But as each entity resides on its own index in Elasticsearch and join/subquery is not possible it has not been possible so far.
To overcome this shortcoming, some options have been added in the entity to make it possible to index the related and even inherited system entities with each object.
Indexing member/descendant entities
As Elasticsearch is a flat document-based database that does not support joining or subqueries the only way to do a search based on member/descendant entities is to index them with the parent. So, every document will have the necessary members/descendants and related system entities indexed with them.
But indexing all the member/descendant entities is not a good option, as it will increase the document size quite substantially, also all of them might not be necessary for searching. So an option has been added to the entity structures for this.
It is also possible to index not only member entities but also descendant entities this way. For example in ISADG, if Arkivdel is selected to be indexed with Arkiv and Serie is selected to be indexed with Arkivdel then the Series under Arkivdels also gets indexed with Arkivs.
Limits
As it is possible to self-link entities in Asta7 (Serie → Serie) in theory it is possible to nest entities infinitely. Moreover the way ES mapping works (Every level of nesting adds a whole bunch of fields to the mapping). So we had to add some limitations to member/descendant indexing.
Total nesting limit: 10 (Arkiv → Arkivdel → Serie → Serie → Serie → Stykke → Mappe → Mappe → Mappe → Geografi)
Self nesting limit: 2 (A → Aa → Aaa)
File nesting limit: 1 (Arkiv → Fil)
Nested member limit: 1000
ES Total field limit: 5000 (Default was 1000)
ES Nested field limit: 500 (Default was 50)
ES Nested object limit: 50_000 (Default was 10_000)
Field Limits
Level 1: All fields
Level 2: All fields for leaf entities (Alternativtnavn, Geografy, etc.), otherwise only required fields (Serie, Stykke)
Level below 2: Required fields
Althogh it is possible to change the search settings any time, even after project inilization, it will no be in effect immediately. Project search data has to be re-indexed after any settings change. Otherwise the sarch might be broken or not work as expected.
Searching
Tree Search
The tree search has been updated with the following changes
The search field now works with ES simple query string syntax.
The search will work on all the fields in the entity not just the title fields as before. But the title fields will have a 2x boost.
This search will do a text search on all fields (Even date, number, and other types). For this, every type of field will also have a text version of it indexed.
For the ISADG Arkiv and Aktor entities, their Alternativtnavn members' navn field will also be added to the search.
For the ISADG Aksesjon entity, its mottar-fra participant’s navn field will also be added to the search.
Basic Search
Coming soon…
Advance Search
Coming soon…
Preference
The search preference has been updated to show only the searchable entities.
Things To Do
Need to handle orphan members and system relations during sync.
More control over which member to include, like Serie should be included with Arkiv but not with Arkivdel.
Control which field is searchable and/or searchable as a member. This should make the field limitations unnecessary. If not then need to make the field limitations configurable instead of hard-coding.
Include member’s/descendant's related system entities?
Although there are limitations on the number of nested members, they are not applied during syncing at the moment. Need to fix this.
Need to use Asta7 models and properties in Essync instead of duplicating them.
Multi-level nesting makes the search quite complex and might not be needed for all the projects. Should we consider adding support for flat nesting as well?