WP08 Evaluation

In this project, evaluation is an integral part of the development work throughout the project. Evaluation is a continuous task running alongside development work. This enables the iterative design approach.

The focus of evaluation is in the applicability of the AMIRE outcome in real settings (both authoring tools and applications). In the beginning of the project, evaluation is mainly formative aiming at providing requirements and boundaries for the AMIRE development work. In the latter stages of the project, summative evaluation will be applied when appropriate.

Evaluation tasks

The evaluation of the AMIRE authoring tools and applications consists of four tasks. In the first task (starting in M3), the evaluation criteria are defined. This will be done according to the results of a context of use analysis (based on initial specification) that describes the relevant aspects of the use situation and environment.

Evaluation will be conducted mainly from the standpoint of usability. ISO 9241 (part 11) that defines usability with attributes effectiveness, efficiency, and user satisfaction, will used as the underlying framework. This will be supported with other models describing additional attributes of usability (e.g. in Usability Engineering by Nielsen 1993: learnability, memorability, little/no errors, effectiveness, satisfaction).

Task 1 results in Evaluation Criteria Specification (deliverable D8.1) describing criteria both for applications and authoring tools. This deliverable will be delivered to, discussed and processed further with the other AMIRE partners. Mixed Reality framework-component and gems experts will have these criteria as underlying basic information when creating specifications for the AMIRE authoring tools and applications.

Initial evaluation

Initial evaluation (task 2; M6-M8) will be done based on revised specifications. The specifications will be analysed against use scenarios that have been formed in task 1 (definition of evaluation criteria). This task is mainly analytical and based on expert/heuristic evaluation. The results of this work are reported in Initial Evaluation Report (D8.2).

Ongoing evaluation

Ongoing evaluation (task 3; M9-M23) provides active support for the development work. The aim is to enable active evaluation probes that can be arranged as design alternatives or problems arise in the design and implementation work. These evaluation probes form part of the iterative specification-implementation-evaluation cycle.

Ongoing evaluation will be conducted with a variety of methods that may include following:

• Usability studies in the usability laboratory. The aim of these studies is to test the usability and ease of use in laboratory conditions with pre-defined tasks. Laboratory settings provide good means for detailed interaction analysis. Test arrangements may vary from traditional thinking-aloud tests to pluralistic usability walkthroughs and to more detailed level interaction studies.
• Field studies. As not all aspects of the context of use can be set up in a natural way in laboratory settings, field usability studies are required. Field studies can have a wider evaluation scope consisting of parts that address additional issues like social acceptability.

The results from the different formative studies will be reported to the specification and implementation parties. The joint results form the deliverable "Ongoing Evaluation Report".

Final evaluation

Because of intensive evaluation during the development work, the role of the Final Evaluation (task 4; M24-M26) is expected to be mainly affirmative. In these final evaluations and tests, the initial evaluation criteria (with indication about evaluation methodology) will be evoked and the final complete outcome of the work will be tested against these criteria. Should this test raise additional concerns or problems, these will be recorded for future user training and product documentation purposes. The Final Evaluation Report (deliverable D8.4) will describe these findings.