WP03 AMIRE Gems

The development of MR applications requires solutions in a number of different areas. The real scene has to be analysed (e.g. using computer vision techniques) and tracked according to user interactions. The analysed data will be used to generate content (using 2D, 3D and audio) and combine different media assets.

A mixed reality application requires solutions for the following tasks

• Object recognition and tracking
• Media generation (2D, 3D, audio, text)
• Composition (Generated media assets with real video images)
• Networking / Streaming functionality (protocols, codec, etc)
• High-Level animation and interaction access

Solutions to these general problems often exist already and many are available from third party developers and can be easily tailored to the special needs of an individual mixed reality application. We call such solutions (unshaped) "mixed reality gems".

The main problem of reusing MR gems is to identify and find gems, to adapt gems according to the special needs of the current application and to integrate gems technically in a new framework different from the gem's original environment. At the beginning of the AMIRE projects a classification of possible gems will be defined. During the project specification phase all necessary gems will be identified and classified with regard to the classification. Additionally, the status of the gem's availability (has already been adapted, exists but not adapted, new development) has to be identified.

Challenges

The biggest challenge is the efficient integration for all types of gems and to ensure that gems can be easily reused by the component expert to design the AMIRE components. Complex heavy-weight components should be built using combination of gems and allow the development of application specific building blocks. Based on the programming environment and component model that will be employed in AMIRE (Java, C++ and CORBA, COM/ActiveX, dot.net, JavaBeans) gem wrappers for different gem types will be provided that allow to encapsulate arbitrary solutions from third parties and provides access through structured programming interfaces. Specification of gems will also be supported using a declarative XML compliant description format.

The MR gems expert will ensure that useful existing MR technology software units will be available in a well defined way, so that they can be used several times in MR applications as well as in the MR authoring tools. There will be MR gems for using augmented reality technologies (e.g. MR gems for identifying markers in a video image) as well as for using rich media (e.g. MR gems for handling and mapping video in a computer generated 3D scene).