Evaluation

ASIMOV has been evaluated using a real scenario in which we can apply all evolution instructions with several assistance blocks for the instructions that require it. This scenario is a commercial scenario named Alps Furniture (AF) built in the Enterprise Architecture Laboratory at Universidad de los Andes. This scenario has a complex metamodel, which includes 81 EClasses and 89 EReferences. The model conforms to the AF metamodel has up to 207 instances. However, for this evaluation we used one specific domain from AF named BusinessPartners (BP).

Metamodel of Business Partners domain from Alps Furniture Scenario

The objective of this evaluation is to demonstrate that 1) ASIMOV is useful for metamodelers and modelers that are working in the Enterprise Architecture context, and 2) ASIMOV has a great level of expressiveness and usability. To achive this, we made two experiments.

The first experiment consists in the coevolution process using two models conform to the AF metamodel in the BP domain. The coevolution has been performed by several modelers using EMF tools and ASIMOV. In this evaluation, we made changes to evolve the AF BP metamodel. Also, we measured to modelers the next variables: time, number of actions, errors, and attempts to coevolve the models. Based on the results, the time used by modelers to coevolve a model using ASIMOV is around 19% of the time required to coevolve the model using EMF tools. Also, due ASIMOV can apply a specic assistance to several instances, the number of actions applied to a model using EMF is almost three times than the number of actions using ASIMOV. In addition, ASIMOV is intuitive; as a result, the number of error trends to zero; however, it is easy to place several errors changing the model using EMF. Finally, due the errors, the number of attempts is higher using EMF.

Time Results

Time Results with experience

Actions

The second experiment consists in the coevolution process using a third model conforms to the AF metamodel in the BP domain. This third model has much greater number of instances. Based on the results of the previous experiment and this new experiment, we measured average time and average number of actions. We compare these variables considering the number of instances of each model. Based on this second experiment, the average increment in time having three models using EMF is 85% and using ASIMOV y 16%; and the average increment in ctions using EMF is 126% and using ASIMOV is 23%.

Errors & Attempts

As a result, we could prove that ASIMOV 1) performs the process eciently and eectively; 2) oers to modelers knowledge about metamodel evolution through the information written by metamodelers; 3) can solve the coevolution in a complex problem; 4) is usable for metamodelers and modelers; and 5) ensures model conformity in all steps of the process.