The ODPi Egeria Open Metadata Conformance Suite – Repository Workbench

By December 4, 2019 ODPi Egeria, Tech Deep Dive

The Open Metadata Conformance Suite is used to verify the behaviour of the Egeria platform and repository services. The Conformance Suite is the basis of the Egeria Conformance Program.

This document describes the tests run by the Repository Workbench. This workbench defines a set of tests that verify the behaviour of the repository services and is used to verify that a repository connector fulfils the requirements of the metadata collection REST API and the OMRS event exchange.

The Conformance Suite is augmented with each Egeria release, with more tests being added and existing tests being extended or refined. A repository is tested and certified as conformant to a specific version of the Conformance Suite. At the time of writing, Egeria release 1.2 is about to be made available. The built-in Egeria in-memory repository and graph repository are both fully conformant with Conformance Suite release 1.2. Details of the results of testing of the built-in connectors is included below.

What the Conformance Suite does

The Conformance Suite uses a pair of servers – one is the server under test and the other is the server that drives the test suite. The server under test should be configured to use the repository you need to test. For read/write repository connectors you can start with an empty repository. For a read-only connector (one that does not support creation of instances) connect to a repository that already contains instances of the types supported. The conformance tests are quite thorough and will drive a connector into many states (both valid and invalid), so you should not use a repository containing valuable metadata.

The servers need to be in the same cohort. There could be other servers in the cohort, but for simplicity you may want to run with just the two servers: the server under test and the test driver. The servers can be run on the same OMAG Server Platform, or they could be run on separate platforms.

The Conformance Suite repository workbench consists of a number of testcases that exercise the repository under test. The testcases invoke the OMRS REST interface and perform operations that generate OMRS events that the server under test reacts to. The Conformance Suite therefore tests more than just the repository under test. It is also testing the function of the OMRS cohort and the local connectors.

The functions performed by the metadata collection API are a combination of mandatory functions that a conformant repository must support, plus optional functions that a repository should support if possible. A repository is declared as “Conformant” if it supports all the mandatory requirements and behaves properly for any optional requirements that it does not support. When asked if it can perform an optional requirement that it does not support, the repository should respond indicating that the function is not supported.

The Conformance Suite specifies sets of functional requirements and divides them into profiles. A profile can be either mandatory or optional. Profiles and requirements are described in more detail a little further on. Before that, the next section describes how to get started with running the Conformance Suite.

To run the Conformance Suite Repository Workbench

Before you run the workbench, you need to decide a few things:

  • One is to know the repository that you want to test – shown in purple in the diagram;
  • Another is to think up a name for the server under test.
  • The other is to think up a name for a cohort to which you can register both the server under test and the CTS server.

The following steps can be used to configure and run the workbench.

Configure the Server Under Test

Configure the server that is to be the subject of the test. There is nothing special about the configuration of this server – except that it needs to join the cohort and should be configured to use the repository you want to test. For example, the server may be configured to use a local graph repository or an in-memory repository.

In the following examples, the user name is ‘user1’ and the server is called ‘test1’, but you should replace these with whatever names you choose.

You should set its server type name is set to ‘Metadata Repository Server’:

POST http://localhost:8080/open-metadata/admin-services/users/user1/servers/test1/server-type?typeName=Metadata Repository Server

Set the serverURLRoot:

POST http://localhost:8080/open-metadata/admin-services/users/user1/servers/test1/server-url-root?url=https://localhost:8080

Set the EventBus:

POST http://localhost:8080/open-metadata/admin-services/users/user1/servers/test1/event-bus

with request body:

{
“producer”:    {   “bootstrap.servers”:”localhost:9092″  },
“consumer”:   {   “bootstrap.servers”:”localhost:9092″  }
}

The above assumes you are running Kafka on its default ports. If this is not the case insert your preferred port numbers.

In addition, set the repository mode, to whichever repository you need to test. For example:`

http://localhost:8080/open-metadata/admin-services/users/user1/servers/test1/local-repository/mode/in-memory-repository

The server under test must be configured to join the cohort you chose above:

POST http://localhost:8080/open-metadata/admin-services/users/user1/servers/test1/cohorts/myCohort

Configure the CTS Server

Then configure a second server – referred to as the ‘CTS server’. This is the server that actually initiates the workbenches and testcases. The CTS server’s configuration is largely the default configuration.

The CTS server will by default use a local in-memory repository, so you don’t need to set a repository mode.

In the following examples, the user name is ‘user1’ and the server is called ‘CTS1’, but you should replace these with whatever names you choose.

You should set its server type name is set to ‘Conformance Test Server’:

POST http://localhost:8080/open-metadata/admin-services/users/user1/servers/CTS1/server-type?typeName=Conformance Test Server

Set its serverURLRoot:

POST http://localhost:8080/open-metadata/admin-services/users/user1/servers/CTS1/server-url-root?url=https://localhost:8080

Set its EventBus:

POST http://localhost:8080/open-metadata/admin-services/users/user1/servers/CTS1/event-bus

with request body:

{
“producer”:    {   “bootstrap.servers”:”localhost:9092″  },
“consumer”:   {   “bootstrap.servers”:”localhost:9092″  }
}

The above assumes you are running Kafka on its default ports. If this is not the case insert your preferred port numbers.

The CTS server must be configured to join the same cohort as the server under test:

POST http://localhost:8080/open-metadata/admin-services/users/user1/servers/CTS1/cohorts/myCohort

The key difference to the configuration of the CTS server is that the repository test workbench is enabled:

POST http://localhost:8080/open-metadata/admin-services/users/user1/servers/CTS1/conformance-suite-workbenches/repository-workbench/repositories/test1

Starting the Servers

Start the CTS server:

http://localhost:8080/open-metadata/admin-services/users/user1/servers/CTS1/instance

The CTS server starts and will load the Egeria types then wait for cocvoMDS1 to register with the cohort.

Start the server under test:

http://localhost:8080/open-metadata/admin-services/users/user1/servers/test1/instance

When the server under test starts, it registers with the cohort. When the CTS server sees the registration of the server under test, it will start the workbench.

The workbench runs the repository conformance testcases which will query the set of types supported by the server under test. For each discovered type, the workbench will run a set of lifecycle tests and other tests that exercise the OMRS interface and repository in the server under test, to determine which of the conformance requirements it supports.

A full run of the workbench includes a lot of tests (several thousand) so it can take quite a long time to run – for example between 30 minutes and an hour.

The results of the testing are collected in the memory of the CTS server. On completion of the tests, the results can be harvested and inspected by issuing a REST request to the CTS server:

localhost:8080/servers/CTS1/open-metadata/conformance-suite/users/user1/report

The results will be returned as JSON. If the conformance test run was fairly small (only ran over a subset of profiles or types), or you use some of the reporting options to request a subset of the results, the results can be formatted and displayed in a tool like Postman. However, a full run of the test suite will generate a very large report and you may find that Postman cannot handle it. For a full run of the workbench it is advisable to use a different means of retrieving the results. For example, you may want to use the httpie tool (available in homebrew), e.g. from a bash command line:

http --json --pretty format  GET localhost:8080/servers/CTS1/open-metadata/conformance-suite/users/user1/report

You may want to additionally ‘tee’ this output to a file for subsequent browsing.

The last part of the test report provides a summary of the number of tests that were run and the number that passed or failed:

The key things to check are there are no failed or skipped tests. If you browse the earlier sections of the report, you can find per-profile and per-requirement results as well as details of which assertions succeeded and which failed.

 

Using the CTS Notebook

There is a Jupyter notebook in the Egeria release, the latest version of which is in the master branch and is located under open-metadata-resources/open-metadata-labs/conformance-testing-labs and is called run-conformance-test-suite.ipynb . The notebook has cells for configuring and running the conformance suite as described above. At the end of the notebook there are cells that retrieve the workbench results and summarise them.

With release 1.2 of the conformance suite, the profile summary for a full-function, conformant repository should look like the following:

The two profiles reported as having “unknown status” are not explicitly tested in the 1.2 release of the conformance suite. It is expected that tests for test profiles will be added soon.

How the Conformance Suite Repository Workbench works.

Profiles

The repository conformance workbench contains a set of test profiles, listed below. The METADATA_SHARING profile is mandatory – so a repository must support it in order for that repository to be certified as conformant. The other profiles are optional. The repository can either support the optional profiles or respond appropriately to indicate to a caller that the function they want to perform is not supported by the repository.

Profile What is tested… Mandatory Optional
METADATA_SHARING The ability to share metadata with other members of the cohort

X

REFERENCE_COPIES The ability to save, lock and purge reference copies of metadata from other members of the cohort

X

METADATA_MAINTENANCE The ability to support requests to create, update and purge metadata instances

X

DYNAMIC_TYPES         The ability to support changes to the list of its supported types while it is running

X

GRAPH_QUERIES         The ability to support graph-like queries that return collections of metadata instances

X

HISTORICAL_SEARCH     The ability to support search for the state of the metadata instances at a specific time in the past

X

ENTITY_PROXIES        The ability to store stubs for entities to use on relationships when the full entity is not available

X

SOFT_DELETE_RESTORE The ability for an instance to be soft-deleted and restored.

X

UNDO_UPDATE           The ability to restore an instance to its previous version (although the version number is updated).”,

X

REIDENTIFY_INSTANCE   The ability to change the unique identifier (guid) of a metadata instance

X

RETYPE_INSTANCE       The ability to change the type of a metadata instance to either its super type or a subtype

X

REHOME_INSTANCE       The ability to update the metadata collection id for a metadata instance

X

ADVANCED_SEARCH       The ability to support the use regular expressions to search for metadata instances

X

 

There are some notes on each of these profiles in the following sections.

METADATA SHARING

This mandatory profile contains a broad set of function that every repository connector must support. The functions include fundamental OMRS capabilities such as joining a cohort, being able to connect to and identify a metadata collection, supporting the open metadata type system and related type queries and notification events. It also includes the ability to retrieve and search for metadata instances, to manage version numbers appropriately and to generate and respond to OMRS events associated with changes to instances.

Requirements in this profile:

  • COHORT_REGISTRATION
  • REPOSITORY_CONNECTOR
  • METADATA_COLLECTION_ID
  • SUPPORTED_TYPE_QUERIES
  • SUPPORTED_TYPE_NOTIFICATIONS
  • CONSISTENT_TYPES
  • METADATA_INSTANCE_ACCESS
  • CURRENT_PROPERTY_SEARCH
  • CURRENT_VALUE_SEARCH
  • INSTANCE_NOTIFICATIONS
  • INSTANCE_VERSIONING
  • TYPE_ENFORCEMENT
  • UNSUPPORTED_TYPE_ERRORS
  • TYPEDEF_CONFLICT_MANAGEMENT

REFERENCE COPIES

This optional profile tests that the repository can save and retrieve reference copies of instances homed elsewhere. It tests that the repository will only allow valid operations to be performed on a saved reference copy. For example, it must not allow a caller to modify the status, properties, type or identity of a reference copy – all these operations can only be performed on the master copy. This profile also tests that reference copies can be deleted.

Requirements in this profile:

  • REFERENCE_COPY_STORAGE
  • REFERENCE_COPY_LOCKING
  • REFERENCE_COPY_DELETE

METADATA MAINTENANCE

This optional profile tests that the repository can manage the lifecycle of metadata instances, including create, update, and purge of instances.

Requirements in this profile:

  • ENTITY_LIFECYCLE
  • CLASSIFICATION_LIFECYCLE
  • RELATIONSHIP_LIFECYCLE

DYNAMIC_TYPES

This profile defines operations that enable the addition and update of type definitions.

  • TYPEDEF_ADD
  • TYPEDEF_MAINTENANCE

This profile is not yet tested in the Conformance Suite v1.2 release of the repository workbench.

GRAPH_QUERIES

This profile defines a set of operations that enable the caller to navigate parts of the metadata instance graph. These operations can return the entities and relationships attached to and surrounding a specified entity (to a range of depths) or return the entities that are connected (directly or indirectly) to the specified entity. Another operation enables the caller to ask for all entities and relationships on all paths between a start entity and end entity, allowing the caller to ascertain how the two entities are related.

Requirements in this profile:

  • ENTITY_NEIGHBORHOOD
  • CONNECTED_ENTITIES
  • LINKED_ENTITIES

Support for this profile is tested by ascertaining what entity and relationship types the repository supports. The testcase then constructs an in-memory graph using a variety of relationship types and directions and a variety of entity types, all of which the repository should be able to support. In parallel with the construction of this in-memory instance graph, operations are dispatched to the repository so that it constructs and stores a matching instance graph. The three operations listed above are then tested, and for each operation the expected result is computed from the in-memory graph and compared to the result returned by the repository connector.

HISTORICAL_SEARCH

This optional profile describes the ability to store historic state of the repository and to search for and retrieve instances as they existed at an earlier time.

Requirements in this profile:

  • HISTORICAL_PROPERTY_SEARCH
  • HISTORICAL_VALUE_SEARCH

Support for this profile is tested as part of the entity and relationship lifecycle testcases. During the running of these testcases, a note is made of the time just prior to an instance being deleted from the repository. On completion of the delete tests, the testcase performs a getEntityDetail() or getRelationship() operation specifying the earlier time. Because the time specified was prior to the delete, the operation should return the version of the instance that existed prior to being deleted.

ENTITY_PROXIES

This optional profile describes the ability to store a proxy instance of an entity, in lieu of a full copy of the entity.

Requirements in this profile:

  • STORE_ENTITY_PROXIES
  • RETRIEVE_ENTITY_PROXIES

Support for this profile will be tested as part of the relationship lifecycle testcase. During the running of this testcase, instances of relationships are created, mostly alongside local instances of the entities at the ends of the relationship. It is also possible to create a relationship instance when there is no local entity instance (neither a locally homed instance nor a reference copy).

SOFT_DELETE_RESTORE

This optional profile includes the ability to perform a ‘soft’ delete or an instance, such that the instance can subsequently be restored. Following a soft delete the instance remains in the repository but is marked as being in deleted state and is not returned in response to searches or read requests.

Requirements in this profile:

  • SOFT_DELETE_INSTANCE
  • UNDELETE_INSTANCE
  • NEW_VERSION_NUMBER_ON_RESTORE

To test for support of this profile, the testcases attempt to delete entity and relationship instances. If the delete operation is supported (does not respond with FunctionNotSupported) further tests are performed to ensure that the deleted instances are not returned in response to requests to ‘get’ or ‘find’ the instances. Further tests are also run to restore instances and to purge deleted instances.

UNDO_UPDATE

This optional profile includes the ability to revert changes previously made to an instance. The result of an undo operation should be that the instance is returned to its previous state, with the exception that its version number will haev moved forward.

Requirements in this profile:

  • RETURN_PREVIOUS_VERSION
  • NEW_VERSION_NUMBER_ON_UNDO

The ability to undo an update is tested as part of the entity and relationship lifecycle testcases.

REIDENTIFY_INSTANCE

This optional profile includes the ability to change the identity (i.e. the GUID) of an instance. This operation can only be performed by the metadata repository that is the home of the instance (i.e. holds the master copy of the instance).

Requirements in this profile:

  • UPDATE_INSTANCE_IDENTIFIER
  • SEND_REIDENTIFIED_EVENT
  • PROCESS_REIDENTIFIED_EVENT

The TestSupportedEntityReidentify and TestSupportedRelationshipReidentify testcases test support for this profile. These testcases create entity and relationship instances and then change their GUIDs. This is only a valid operation on the home repository for the instance. If the repository reports that the reidentify operation is supported further tests are performed to ensure that the reidentified instances can be retrieved using the new GUID and that they are not returned in response to requests using the original GUID.

RETYPE_INSTANCE

This optional profile includes the ability to change the type of an instance. This operation can only be performed by the metadata repository that is the home of the instance (i.e. holds the master copy of the instance).

Requirements in this profile:

  • UPDATE_INSTANCE_TYPE
  • SEND_RETYPED_EVENT
  • PROCESS_RETYPED_EVENT

The TestSupportedEntityRetype testcase tests support for this profile. This testcase creates entity instances and then retypes them to each of their subtypes and back again to their original type. The reason this is limited to entity instances is that subtype and reverting to original type means that all properties of the original instance should remain valid throughout the testcase, which is a convenient condition to verify. It would be possible to add further tests for alternative type changes to entity and to test retype of relationship instances.

REHOME_INSTANCE 

This optional profile includes the ability to change the home (i.e the metadataCollectionId) of an instance. This is a ‘pull’ operation – i.e. the operation can only be performed by the metadata repository that is to become the new home of the instance. Such a repository must currently be in possession of a reference copy of the instance. If the operation is successful, that reference copy will be promoted to become the new master copy.

Requirements in this profile:

  • UPDATE_INSTANCE_HOME
  • SEND_REHOMED_EVENT
  • PROCESS_REHOMED_EVENT

The TestSupportedEntityReferenceCopyLifecycle and TestSupportedRelationshipReferenceCopyLifecycle testcases test support for this profile. These testcases create entity and relationship instances at the CTS server, which triggers the creation of reference copy instances in the repository under test. The testcases then instruct the server under test to become the new home repository for each instance. This has to be performed prior to the delete of the instance in the CTS repository, otherwise the reference copy would also be purged in the repository under test – making the test impossible. Because this is not a natural sequence of events (the instance in the CTS still exists) no further tests are performed after this in each of the testcases.

ADVANCED_SEARCH

This optional profile includes the ability to search for entity and relationship instances using property values or search criteria that contain regular expressions. This is an optional capability of the findEntitiesByProperty and findEntitiesByPropertyValue methods of the metadata collection interface, and the corresponding methods for relationships. It is also an optional capability of the findEntitiesByClassification method. The mandatory search capabilities of the METADATA_SHARING profile do not need to include support for general regular expressions. Instead, it is sufficient for that profile to support literal values and the limited range of regular expressions formulated by the Repository Helper methods. In contrast, the ADVANCED_SEARCH profile includes support for generic Java regular expression syntax.

Requirements in this profile:

  • ADVANCED_PROPERTY_SEARCH
  • ADVANCED_VALUE_SEARCH

The TestSupportedEntityPropertyAdvancedSearch and TestSupportedRelationshipPropertyAdvancedSearch testcases test support for this profile. These testcases create entity and relationship and then issue find requests to search for them, using regular expressions that will match different subsets of the known property values of the instances. The result of each search operation is compared to a computed expected result.

Testcases

There are currently 18 testcases in the repository conformance workbench. Each testcase focusses on a set of requirements that are generally associated with one profile, but some testcases that test function across multiple profiles.

Most tests are associated with a type – such as a particular entity type, relationship type or classification type. In these cases, a test case (such as TestSupportEntityLifecycle) will be run for each entity type supported by the repository being tested. A test case such as the TestGraphQueries test case is not concerned with a particular type – instead it discovers the set of types supported by the repository under test, and constructs a test graph of instances of types within the set of supported types.

Most testcases create instances by invoking the metadata collection interface of the repository; then they update, search for or delete those instances. Each testcase is self-contained so that it can be run in isolation if needed. So testcases that create instances also provide a cleanup method that removes the entities on completion of the test.

Some repositories do not support creation of instances – they are effectively used in a read-only capacity – so to test them, there are testcases that search for existing instances in the repository. Two testcases of this kind test the findXXXByProperty and findXXXByPropertyValue methods that search for instances. This capability is part of the mandatory profile for metadata sharing, which does not require that a repository supports creation or maintenance of instances. These testcases perform a broad search to retrieve an initial set of results, then use that known set to issue further, finer-grain searches and check that the results are consistent with the initial set and the query performed.

Some testcases are ‘single-phase’ – meaning they are created and invoked just once for a given type – within that invocation the testcase will create, test and destroy whatever instances it uses. Other testcases are ‘multi-phase’ – meaning they are created and invoked multiple times. The search test cases are multi-phase testcases. They are called with a phase indicator which is set to CREATE, EXECUTE and finally CLEAN. During the CREATE phase the testcase creates the instances it will need to test its given entity, relationship or classification type. The EXECUTE phase is when the tests are run, such as performing ‘find’ operations against the repository. The CLEAN phase is used to clean up the instances created by this testcase. The benefit of multi-phase testcases is that all the testcases’ CREATE invocations can be performed first, so during the EXECUTE phase the ‘finds’ are run against a repository that is populated with instances of all supported types. This allows testing of features like sub-type retrieval and type filtering by specifying the typeGUID.

Conformance of ODPi Egeria 1.2 Repository Connectors

As we get ready to finalise release 1.2 of Egeria, we’ve been testing the built-in repository connectors against the 1.2 release of the Conformance Suite. The results are shown below.

  • FULL means the repository fully supports this profile
  • PARTIAL means the repository supports some of this profile
  • NONE means the repository does not support this profile
  • NA means this profile is not tested in the current release
Profile What is tested… In-memory repository Local graph repository
METADATA_SHARING The ability to share metadata with other members of the cohort

FULL

FULL
REFERENCE_COPIES The ability to save, lock and purge reference copies of metadata from other members of the cohort FULL

FULL

METADATA_MAINTENANCE The ability to support requests to create, update and purge metadata instances FULL

FULL

DYNAMIC_TYPES         The ability to support changes to the list of its supported types while it is running NA

NA

GRAPH_QUERIES         The ability to support graph-like queries that return collections of metadata instances PARTIAL

FULL

HISTORICAL_SEARCH     The ability to support search for the state of the metadata instances at a specific time in the past FULL

NONE

ENTITY_PROXIES        The ability to store stubs for entities to use on relationships when the full entity is not available NA

NA

SOFT_DELETE_RESTORE The ability for an instance to be soft-deleted and restored. FULL

FULL

UNDO_UPDATE           The ability to restore an instance to its previous version (although the version number is updated).”, FULL

NONE

REIDENTIFY_INSTANCE   The ability to change the unique identifier (guid) of a metadata instance FULL

FULL

RETYPE_INSTANCE       The ability to change the type of a metadata instance to either its super type or a subtype FULL

FULL

REHOME_INSTANCE       The ability to update the metadata collection id for a metadata instance FULL

FULL

ADVANCED_SEARCH       The ability to support the use regular expressions to search for metadata instances FULL

FULL

Going Forward

As new Egeria releases are developed and new function is added, additional test cases will be added to the Conformance Suite and we expect to add refinements to existing test cases to improve the scope and granularity. Tests for dynamic types and entity proxies will be added.

Repository connectors are certified as conformant at a particular version of the Conformance Suite. If you are developing or testing a repository connector, ensure that you test with the relevant release of the conformance suite.

For more information:

Please refer to the Open Metadata Conformance Suite page in the ODPi Egeria git repository:

https://github.com/odpi/egeria/blob/master/open-metadata-conformance-suite/docs/README.md

 

Social Media Auto Publish Powered By : XYZScripts.com