Movatterモバイル変換

[0]ホーム
URL:

W3C

Best practices for creating MMIModality Components

W3C Working GroupNote1March2011

This version:
http://www.w3.org/TR/2011/NOTE-mmi-mcbp-20110301/
Latest version:
http://www.w3.org/TR/mmi-mcbp/
Previous version:
This is the first version.
Editor:
Ingmar Kliche , Deutsche Telekom AG
Authors:
Deborah Dahl, Invited Expert
James A. Larson, Invited Expert
B. Helena Rodriguez, Telecom ParisTech
Muthuselvam Selvaraj, until 2009 while at HP

Copyright © 2011 W3C® (MIT,ERCIM,Keio),All Rights Reserved. W3Cliability,trademark anddocumentuse rules apply.

Abstract

This document describes Modality Components in the MMIArchitecture which are responsible for controlling the variousinput and output modalities on various devices by providingguidelines and suggestions for designing Modality Components. Alsothis document shows several possible examples of ModalityComponents, (1) face identification, (2) form-filling usinghandwriting recognition and (3) video display.

Status of this Document

This section describes the status of this document at thetime of its publication. Other documents may supersede thisdocument. A list of current W3C publications and the latestrevision of this technical report can be found in theW3C technical reports index athttp://www.w3.org/TR/.

This is the 1 March 2011W3CWorking Group Note of "Best practices for creating MMI ModalityComponents". The Multimodal Interaction Working Group oncepublished a Working Draft of the "MultimodalArchitecture and Interfaces (MMI Architecture)" on 16 October2008 with this content. However, the Working Group concluded thatthe description on how to create Modality Components and examplesof possible Modality Components should be published as a WorkingGroup Note rather than part of the MMI Architecture specification.The goal of this Working Group Note is to provide guidelines andsuggestions for designing Modality Components in the MMIArchitecture and make it easier to author concrete ModalityComponents for multimodal Web applications. Also this documentshows several possible examples of Modality Components, (1) faceidentification, (2) form-filling using handwriting recognition and(3) video display.

This W3C Working Group Note has been developed by theMultimodal Interaction WorkingGroup of the W3CMultimodalInteraction Activity.

Comments for this note are welcomed and should have a subjectstarting with the prefix '[ARCH]'. Please send them towww-multimodal@w3.org, thepublic email list for issues related to Multimodal. This list isarchivedand acceptance of this archiving policy is requested automaticallyupon first post. To subscribe to this list send an email towww-multimodal-request@w3.orgwith the word "subscribe" in the subject line.

For more information about the Multimodal Interaction Activity,please see theMultimodal InteractionActivity statement.

This document was produced by a group operating under the5February 2004 W3C Patent Policy. W3C maintains apublic listof any patent disclosures made in connection with thedeliverables of the group; that page also includes instructions fordisclosing a patent. An individual who has actual knowledge of apatent which the individual believes containsEssential Claim(s) must disclose the information in accordancewithsection 6 of the W3C Patent Policy.

Publication as a Working Group Note does not imply endorsementby the W3C Membership. This is a draft document and may be updated,replaced or obsoleted by other documents at any time. It isinappropriate to cite this document as other than work inprogress.

Table of Contents

1Introduction
2Modality componentguidelines
    2.1Guideline 1: Eachmodality component must implement all of the MMI life-cycleevents.
    2.2Guideline 2:Identify other functions of the modality component that arerelevant to the interaction manager.
    2.3Guideline 3: Ifthe component uses media, specify the media format. For example,audio formats for speech recognition, or InkML for handwritingrecognition.
    2.4Guideline 4:Specify protocols for use between the component and the InteractionManager (IM) (e.g., SIP or HTTP).
    2.5Guideline 5:Specify supported human languages, e.g., English, German, Chineseand locale, if relevant.
    2.6Guideline 6:Specify supporting languages required by the component, ifany.
    2.7Guideline 7:Modality components sending data to the interaction manager mustuse the format where appropriate.
    2.8Guideline 8:Specify error codes and their meanings to be returned to theIM.
3Modalitycomponent design suggestions
    3.1Designsuggestion 1: Consider constructing a complex modality componentwith multiple functions if one function handles the errorsgenerated by another function.
    3.2Designsuggestion 2: Consider constructing a complex modality componentwith multiple functions rather than several simple modalitycomponents if the functions need to be synchronized.
    3.3Designsuggestion 3: Consider constructing a nested modality componentwith multiple child modality components if the children modalitycomponents are frequently used together but do not handle thererrors generated by the other children components and the childrencomponents do not need to be extensively synchronized.
4Examplesimple modality: Face Identification
    4.1Functionsof a Possible Face Identification Component
    4.2EventSyntax
        4.2.1Examples ofevents for starting the component
        4.2.2Example output event
5Example simple modality: Form-filling using HandwritingRecognition
    5.1Functionsof a Possible Handwriting Recognition Component
    5.2Event Syntax
        5.2.1Examples of events for preparing the component
        5.2.2Examples of events for starting the component
        5.2.3Example outputevent
6Example simplemodality: Video Display
    6.1Functionsof a Possible Video Display Component
    6.2Event Syntax
        6.2.1Examplesof events for starting the component

Appendix

AReferences

1Introduction

The W3C Multimodal Interaction (MMI) Working Group develops anarchitecture[MMI-ARCH] for the MultimodalInteraction framework[MMIF]. The MultimodalArchitecture describes a general and flexible framework forinteroperability of the various components of the multimodalframework (e.g. modality components (MC) and the interactionmanager (IM)) in an abstract way. Among others it definesinterfaces and messages between the constituents of the framework,but it is up to the implementation to decide how these messages aretransferred in case of a distributed implementation.

This Note is an informative supplement to the MultimodalArchitecture and Interfaces specification[MMI-ARCH]. In contrast to the MultimodalArchitecture specification, which defines normative conformance formultimodal constituents, the intention of this document is toprovide additional informative guidelines for authors of MMImodality components. Its purpose is to assist authors in maximizingthe usefulness of their Multimodal Architecture conformantconstituents by describing additional information which will enableconstituents to be more easily incorporated into a multimodalsystem. This additional suggested information includes, forexample, descriptions of how the constituent behaves with respectto the optional aspects of the Architecture. The specific goals ofthe guidelines in this document are to:

  1. promote interoperability when constituents from differentvendors are used in the same system by suggesting importantinformation that can be provided along with a constituent that willenable others to use the constituent effectively (2 Modality componentguidelines )
  2. provide suggestions for authoring Modality Components in orderto maximize their effectiveness (3 Modalitycomponent design suggestions )
  3. provide illustrations of these suggestions using samplemodality components for face identification and handwritingrecognition (4 Examplesimple modality: Face Identification and5 Example simple modality: Form-filling using HandwritingRecognition )


2 Modality componentguidelines

The following guidelines guarantee that modalities are portablefrom interaction manager to interaction manager.

2.1 Guideline 1: Eachmodality component must implement all of the MMI life-cycleevents.

The MMI life-cycle events are the mechanism through which amodality component communicates with the interaction manager. TheModality Component (MC) author must define how the modalitycomponent will respond to each life-cycle event. A modalitycomponent must respond to every life-cycle event it receives fromthe interaction manager in the cases where a response is required,as defined by the MMI Architecture. For example, if a modalitycomponent presents a static display, it must respond to a<pause> event with a <pauseResponse> event even if thestatic display modality component does nothing else in response tothe <pause> event.

For each life-cycle event, define the parameters and syntax ofthe "data" element of the corresponding the life-cycle event thatwill be used in performing that function. For example, the<startRequest> event for a speech recognition modalitycomponent might include parameters like timeout, confidencethreshold, max n-best, and grammar.

2.2 Guideline 2:Identify other functions of the modality component that arerelevant to the interaction manager.

Define an <extensionNotification> event to communicatethese functions to and from the interaction manager.

2.3 Guideline 3: Ifthe component uses media, specify the media format. For example,audio formats for speech recognition, or InkML for handwritingrecognition.

2.4 Guideline 4:Specify protocols for use between the component and the InteractionManager (IM) (e.g., SIP or HTTP).

2.5 Guideline 5:Specify supported human languages, e.g., English, German, Chineseand locale, if relevant.

2.6 Guideline 6:Specify supporting languages required by the component, ifany.

For example:

2.7 Guideline 7:Modality components sending data to the interaction manager mustuse the[EMMA] format where appropriate.

If a modality component captures or generates information, thenit should format the information using the EMMA format and use anextension event to send that information to the interactionmanager.

2.8 Guideline 8:Specify error codes and their meanings to be returned to theIM.

The MC developer must specify all error codes that are specificto the component. If the MC is based on another technology, thedeveloper can provide a reference to that technology specification.For instance, if the MC is based on VoiceXML, a reference to theVoiceXML spec for VoiceXML errors can be included instead oflisting each VoiceXML error.

Errors such as XML errors and MMI protocol errors must behandled in accordance with the guidelines laid out in the MMIarchitecture. These errors do not need to be documented.

3 Modality componentdesign suggestions

The following design suggestions should be helpful for modalitycomponent authors to make modalities portable from interactionmanager to interaction manager.

3.1Design suggestion 1: Consider constructing a complex modalitycomponent with multiple functions if one function handles theerrors generated by another function.

For example, if the ASR fails to recognize a user's utterance, aprompt may be presented to the user asking the user to try again bythe TTS function. As another example, if the ASR fails to recognizea user's utterance, a GUI function might display the n-best list ona screen so the user can select the desired word. Efficiencyconcerns may indicate that two modality components be combined intoa single complex modality component.

3.2Design suggestion 2: Consider constructing a complex modalitycomponent with multiple functions rather than several simplemodality components if the functions need to be synchronized.

For example, a TTS function must be synchronized with a visualtalking head so that the lip movements are synchronized with thewords. As another example, a TTS functions presents informationabout the each graphical item that the user places "in focus."Again, efficiency concerns may indicate that the TTS and talkinghead be two modality components be combined into a single complexmodality component.

3.3Design suggestion 3: Consider constructing a nested modalitycomponent with multiple child modality components if the childrenmodality components are frequently used together but do not handlether errors generated by the other children components and thechildren components do not need to be extensivelysynchronized.

Writing an application using a nested modality component may beeasier than writing the same application using multiple modalitycomponents if the nested modality component hides much of thecomplexity of managing the children modality components.

4 Examplesimple modality: Face Identification

4.1Functions of a Possible Face Identification Component

Consider a theoretical face identification modality componentthat takes an image or images of a face and returns the set ofpossible matches and the confidence of the face identificationsoftware in each match. An API to that modality component wouldinclude events for starting the component, providing data, and forreceiving results back from the component.

This particular example includes the information needed to runthis component in the "startRequest" and "doneNotification" events;that is, in this example no "extensionNotification" events areused, although extensionNotification events could be part ofanother modality component's API. This example assumes that animage has already been acquired from some source; however, anotherpossibility would be to also include image acquisition in theoperation of the component.

Depending on the capabilities of the modality component, otherpossible information that might be included would be some usefulnon-functional information as the capturing context of the stillpicture (e.g. indoor picture or outdoor picture) or the type ofimage (e.g. a portrait photography or a street photography) orwould be some technical information as the algorithm to be used orthe image format to expect. We emphasize that this is just anexample to indicate the kinds of information that might be used bya multimodal application that includes face recognition. The actualinterface used in real applications should be defined by experts inthe field.

The use case is a face identification component that identifiesone of a set of employees on the basis of face images.

The MMI Runtime Framework could use the following events tocommunicate with such a component.

Table 1: Component behavior of Face Identification withrespect to modality component guidelines.
GuidelineComponent Information
Guideline 1: Each modality component must implement all of theMMI life cycle events.See Table 2 for the details of the implementation of thelife-cycle events.
Guideline 2: Identify other functions of the modality componentthat are relevant to the interaction manager.All the functions of the component are covered in thelife-cycle events, no other functions are needed.
Guideline 3: If the component uses media, specify the mediaformat.The component uses the jpeg format for images to be identifiedand for its image database.
Guideline 4: Specify protocols supported by the component fortransmitting media (e.g. SIP).The component uses HTTP for transmitting media.
Guideline 5: Specify supported human languages.This component does not support any human languages.
Guideline 6: Specify supporting languages required by thecomponent.This component does not require any markup languages.
Guideline 7: Modality components sending data to theinteraction manager must use the EMMA format.This component uses EMMA.
Table 2: Component behavior of face identification foreach life-cycle event.
Life Cycle EventComponent Implementation
newContextRequest(Standard) The component requests a new context from theIM.
newContextResponse(Standard) The component starts a new context and assigns thenew context id to it.
prepareRequestThe component prepares resources to be used in identification,specifically, the image database.
prepareResponse(Standard) If the database of known users is not found, theerror message "known users not found" is returned in the<statusInfo> element.
startRequestThe component starts processing if possible, using a specifiedimage, image database, threshold, and limit on the size of nbestresults to be returned.
startResponse(Standard) If the database of known users is not found, theerror message "known users not found" is returned in the<statusInfo> element.
doneNotificationIdentification results in EMMA format are reported in the"data" field.The mode is "photograph", the medium is "visual", thefunction is "identification", and verbal is "false".
cancelRequestThis component stops processing when it receives a"cancelRequest". It always performs a hard stop whether or not theIM requests a hard stop.
cancelResponse(Standard)
pauseRequestThis component cannot pause.
pauseResponse<statusInfo> field is "cannot pause".
resumeRequestThis component cannot resume.
resumeResponse<statusInfo> field is "cannot resume".
extensionNotificationThis component does not use "extensionNotification". It ignoresany "extensionNotification" events that are sent to it by theIM.
clearContextRequest(Standard)
clearContextResponse(Standard)
statusRequest(Standard)
statusResponseThe component returns a standard life cycle response. The"automaticUpdate" attribute is "false", because this component doesnot supply automatic updates.

Note: "(Standard)" means that the component does not doanything over and above the actions specified by the MMIArchitecture.

4.2 EventSyntax

4.2.1Examples of events for starting the component

To start the component, a startRequest event from the IM to theface identification component is sent, asking it to start anidentification. It assumes that images found at a certain URI areto be identified by comparing them against a known set of employeesfound at another URI. The confidence threshold of the component isset to .5 and the IM requests a maximum of five possiblematches.

<mmi xmlns="http://www.w3.org/2008/04/mmi-arch" version="1.0">  <mmi:startRequest source="uri:RTFURI" context="URI-1" requestID="request-1">    <mmi:data>      <face-identification-parameters threshold=".5" unknown="someURI" known="uri:employees" max-nbest="5"/>    </mmi:data>  </mmi:startRequest></mmi:mmi>

As part of support for the life-cycle events, a modalitycomponent is required to respond to a startRequest event with astartResponse event. Here's an example of a startResponse from theface identification component to the IM informing the IM that theface identification component has successfully started.

<mmi xmlns="http://www.w3.org/2008/04/mmi-arch" version="1.0">  <mmi:startResponse source="uri:faceURI" context="URI-1" requestID="request-1" status="success"/> </mmi:mmi>

Here's an example of a startResponse event from the faceidentification component to the IM in the case of failure, with anexample failure message. In this case the failure message indicatesthat the known images cannot be found. 

<mmi xmlns="http://www.w3.org/2008/04/mmi-arch" version="1.0">  <mmi:startResponse source="uri:faceURI" context="URI-1" requestID="request-1" status="failure">    <mmi:statusInfo>      known users not found    </mmi:statusInfo>  </mmi:startResponse></mmi:mmi>

4.2.2 Example output event

Here's an example of an output event, sent from the faceidentification component to the IM, using EMMA to represent theidentification results. Two results with different confidences arereturned.

<mmi xmlns="http://www.w3.org/2008/04/mmi-arch" version="1.0">  <mmi:doneNotification source="uri:faceURI" context="URI-1" status="success" requestID="request-1">    <mmi:data>      <emma:emma version="1.0" xmlns:emma="http://www.w3.org/2003/04/emma">        <emma:one-of emma:medium="visual" emma:verbal="false" emma:mode="photograph" emma:function="identification">          <emma:interpretation emma:confidence=".75">            <person>12345</person>            <name>Mary Smith</name>          </emma:interpretation>          <emma:interpretation emma:confidence=".6">            <person>67890</person>            <name>Jim Jones</name>          </emma:interpretation>        </emma:one-of>      </emma:emma>    </mmi:data>  </mmi:doneNotification></mmi:mmi>

This is an example of EMMA output in the case where the faceimage doesn't match any of the employees.

<mmi xmlns="http://www.w3.org/2008/04/mmi-arch" version="1.0">  <mmi:doneNotification source="uri:faceURI" context="URI-1" status="success" requestID="request-1" >    <mmi:data>      <emma:emma version="1.0" xmlns:emma="http://www.w3.org/2003/04/emma">        <emma:interpretation emma:confidence="0.0"         uninterpreted="true" emma:medium="visual" emma:mode="photograph"         emma:function="identification"/>      </emma:emma>    </mmi:data>  </mmi:doneNotification></mmi:mmi>

5 Example simple modality: Form-filling using HandwritingRecognition

5.1Functions of a Possible Handwriting Recognition Component

Consider an ink recognition modality component for HandwritingRecognition (HWR) that takes digital ink written using anelectronic pen or stylus, performs recognition and returns therecognized text. An API to such a modality component would includeevents for initializing the component, requesting for recognitionby providing digital ink data, and for receiving recognized textresult (possibly an n-best list) back from the component as shownin the below figure.

Example of Japanese handwriting recognition

Figure 1: Example of Japanese handwriting recognition

This example assumes that handwriting ink is captured,represented in W3C InkML format and sent to the IM requesting forrecognition to text. The following sequences of events explain theink recognition request.

  1. The IM requests the ink recognition modality by sending the"prepareRequest" event along with the parameters for configuringthe HWR system.
  2. Ink recognition modality responds with the "prepareResponse"event with the status of the configuration of the HWR system.
  3. IM sends the "startRequest" event to the ink recognitionmodality where the event's data field contains the InkML data to berecognized.
  4. Once the recognition is completed, the ink recognition modalitynotifies the results to the IM using the "doneNotification" eventalong with the recognition choices (N-best list).

The use case is a form-filling application which acceptshandwriting input provided by the user on the form fields. Theinputs are recognized and displayed back as text in thecorresponding fields. An ink capture modality may be used tocapture the ink and send it to IM for recognition. Thecommunication between the ink capture modality and the IM is notcovered here for the sake of brevity. The below section explainsthe details of the communication between the MMI Runtime Framework(RTF) of the IM and the ink recognition modality.

Table 3: Component behavior of Ink modality with respectto modality component guidelines.
GuidelineComponent Information
Guideline 1: Each modality component must implement all of theMMI life cycle events.See Table 4 for the details of the implementation of thelife-cycle events.
Guideline 2: Identify other functions of the modality componentthat are relevant to the interaction manager.All the functions of the component are covered in thelife-cycle events, no other functions are needed.
Guideline 3: If the component uses media, specify the mediaformat.The component uses W3C InkML format to represent handwritingdata (digital ink).
Guideline 4: Specify protocols supported by the component fortransmitting media (e.g. SIP).The component uses HTTP for transmitting media. Other standardprotocols such as TCP may also be explored.
Guideline 5: Specify supported human languages.Virtually any human language script can be supported based onthe HWR component capability.
Guideline 6: Specify supporting languages required by thecomponent.W3C InkML for representing the handwriting data.
Guideline 7: Modality components sending data to theinteraction manager must use the EMMA format.This component uses EMMA.
Table 4: Component behavior of handwriting recognition foreach life-cycle event.
Life Cycle EventComponent Implementation
newContextRequest(Standard) The component requests a new context from theIM.
newContextResponse(Standard) The component starts a new context and assigns thenew context id to it.
prepareRequestThe component prepares resources to be used in recognition.Based on the 'script' parameter, it first selects an appropriaterecognizer. It also configures the recognizer with other parameterssuch as recognition confidence threshold, limit on the size ofn-best results to be returned etc., when available.
prepareResponse(Standard) If the recognizer failed to find a matchingrecognizer for the request language script, a relevant errormessage is returned in the <statusInfo> element.
startRequestThe component performs recognition of the handwritinginput.
startResponse(Standard)The status of recognition as "success" or "failure"is returned in the <statusInfo> element.
doneNotificationIdentification results in EMMA format are reported in the"data" field. The mode is "ink", the medium is "tactile", thefunction is "transcription", and verbal is "true".
cancelRequestThis component stops processing when it receives a"cancelRequest". It always performs a hard stop irrespective of theIM request.
cancelResponse(Standard)
pauseRequestThis component cannot pause.
pauseResponse<statusInfo> field is "cannot pause".
resumeRequestThis component cannot resume.
resumeResponse<statusInfo> field is "cannot resume".
extensionNotificationThis component does not use "extensionNotification". It ignoresany "extensionNotification" events that are sent to it by theIM.
clearContextRequest(Standard)
clearContextResponse(Standard)
statusRequest(Standard)
statusResponseThe component returns a standard life cycle response. The"automaticUpdate" attribute is "false", because this component doesnot supply automatic updates.

Note: "(Standard)" means that the component does not doanything over and above the actions specified by the MMIArchitecture.

5.2 Event Syntax

5.2.1 Examples of events for preparing the component

IM send a prepareRequest event to the ink recognition component.The ink recognition component selects an appropriate recognizerthat matches the given language script, in this example it is setto "English_Lowercase". The "RecoGrammar.xml" grammar file containsconstraints that aid the recognizer. The confidence threshold ofthe component is set to .7 and the IM requests a maximum of fivepossible matches. Based on the capability of the recognizer, otherpossible parameters such as a 'user profile' that containsuser-specific information can be provided.

<mmi:mmi version="1.0" xmlns:mmi="http://www.w3.org/2008/04/mmi-arch">  <mmi:prepareRequest source="uri:RTFURI" context="URI-1" requestID="request-1">    <mmi:data>      <ink-recognition-parameters grammar="RecoGrammar.xml" threshold=".7" script="English_Lowercase" max-nbest="5"/>    </mmi:data>  </mmi:prepareRequest></mmi:mmi>

As part of support for the life cycle events, a modalitycomponent is required to respond to a prepareRequest event with aprepareResponse event. Here's an example of a prepareResponse fromthe ink recognition component to the IM informing the IM that theink recognition component has successfully initialized.

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">  <mmi:prepareResponse source="uri:inkRecognizerURI" context="URI-1" requestID="request-1" status="success"/></mmi:mmi>

Here's an example of a prepareResponse event from the inkrecognition component to the IM in the case of failure, with anexample failure message. In this case the failure message indicatesthat the language script is not supported.

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">  <mmi:prepareResponse source="uri:inkRecognizerURI" context="URI-1" requestID="request-1" status="failure">    <mmi:statusInfo>      Given language script not supported    </mmi:statusInfo>  </mmi:prepareResponse></mmi:mmi>

5.2.2 Examples of events for starting the component

To start the component and recognize the handwriting data, astartRequest event from the IM to the ink recognition component issent. The data field of the event contains InkML representation ofthe ink data.

Along with the ink, additional information such as the referenceco-ordinate system and capture device's resolution may also beprovided in the InkML data. The below example shows that the inkstrokes have X and Y channels and the ink has been captured at aresolution of 1000 DPI. The example ink data contains strokes ofthe Japanese character "手" (te) which means "hand".

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">  <mmi:startRequest source="uri:inkRecognizerURI" context="URI-1" requestID="request-1">    <mmi:data>      <ink:ink version="1.0" xmlns:ink="http://www.w3.org/2003/InkML">        <ink:definitions>          <ink:context>            <ink:traceFormatX" type="decimal">                <ink:channelProperty name="resolution" value="1000" units="1/in"/>              </ink:channel>              <ink:channel name="Y" type="decimal">                <ink:channelProperty name="resolution" value="1000" units="1/in"/>              </ink:channel>            </ink:traceFormat>          </ink:context>        </ink:definitions>       <ink:traceGroup contextRef="#device1Context">        <ink:trace>          106 81, 105 82, 104 84, 103 85, 101 88, 100 90, 99 91, 97 97,          89 105, 88 107, 87 109, 86 110, 84 111, 84 112, 82 113, 78 117,          74 121, 72 122, 70 123, 68 125, 67 125, 66 126, 65 126, 63 127,          57 129, 53 133, 47 135, 46 136, 45 136, 44 137, 43 137, 43 137        </ink:trace>        <ink:trace>          28 165, 29 165, 31 165, 33 165, 35 164, 37 164, 38 164, 40 163,          42 163, 45 163, 49 162, 51 162, 53 162, 56 162, 58 162, 64 160,          69 160, 71 159, 74 159, 76 159, 78 159, 86 157, 91 157, 95 157,          96 157, 99 157, 101 157, 103 157, 109 155, 111 155, 114 155,          116 155, 119 155, 121 154, 124 154, 126 154, 127 154, 129 154,          131 154, 134 153, 135 153, 136 153, 137 153, 138 153, 139 153,          140 153, 141 153, 142 153, 143 153, 144 153, 145 153, 145 153          </ink:trace>        <ink:trace>          10 218, 12 218, 14 218, 20 216, 25 216, 28 216, 31 216, 34 216,          37 216, 45 216, 53 216, 58 215, 60 215, 63 215, 68 215, 72 215,          74 215, 77 215, 85 212, 88 212, 94 210, 100 208, 105 208, 107 208,          109 208, 110 208, 111 207, 114 207, 115 207, 119 207, 121 207,          123 207, 124 207, 128 206, 130 205, 131 205, 134 205, 136 205,          137 205, 138 205, 139 204, 140 204, 141 204, 142 204, 143 204,          144 204, 145 204, 146 204, 147 204, 148 204, 149 204, 150 204,          151 203, 152 203, 153 203, 154 203, 155 203, 156 203, 158 203,          159 202, 160 202, 161 202, 162 202, 163 202, 164 202, 165 202,          166 202, 167 202, 168 202, 169 202, 170 202, 171 202, 172 202,          173 202, 173 201, 173 201        </ink:trace>        <ink:trace>          78 128, 78 127, 79 127, 79 128, 80 129, 80 130, 81 132, 82 133,          82 134, 83 135, 84 137, 85 139, 86 141, 87 142, 88 144, 89 146,          94 152, 95 153, 96 155, 98 160, 99 162, 100 165, 101 167, 101 169,          102 173, 102 176, 102 181, 102 183, 102 185, 102 186, 104 192,          104 195, 104 197, 104 199, 104 201, 104 203, 104 205, 104 206,          104 207, 104 208, 104 209, 104 210, 104 211, 104 213, 104 214,          104 215, 104 216, 104 217, 104 218, 104 220, 103 222, 102 223,          102 224, 102 223, 102 224, 103 225, 103 228, 103 229, 103 230,          103 231, 103 232, 103 233, 103 236, 103 239, 103 242, 103 243,          103 247, 103 248, 102 249, 102 250, 102 251, 101 251, 100 253,          99 255, 99 256, 98 257, 97 258, 97 259, 96 260, 96 261, 95 262,          95 263, 94 264, 94 265, 93 266, 93 267, 92 268, 91 269, 91 270,          90 271, 90 272, 89 273, 89 274, 88 275, 88 276, 87 276, 87 277,          86 277, 86 278, 85 279, 85 280, 84 281, 83 282, 82 284, 82 285,          81 285, 80 286, 79 287, 78 288, 77 288, 77 289, 76 290, 75 290,          75 291, 74 291, 74 290, 74 289, 74 288, 74 287, 73 287, 73 286,          73 285, 72 284, 72 281, 71 280, 70 279, 70 278, 69 277, 68 276,          67 275, 65 274, 62 272, 60 271, 59 271, 58 270, 57 270, 56 269,          55 268, 54 268, 53 267, 52 267, 51 267, 49 267, 48 267, 48 266,          48 266          </ink:trace>       </ink:traceGroup>      </ink:ink>    </mmi:data>   </mmi:startRequest></mmi:mmi>

As part of support for the life cycle events, a modalitycomponent is required to respond to a startRequest event with astartResponse event. Here's an example of a startResponse from theink recognition component to the IM informing the IM that the inkrecognition component has successfully started.

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">  <mmi:startResponse source="uri:inkRecognizerURI" context="URI-1" requestID="request-1" status="success"/></mmi:mmi>

Here's an example of a startResponse event from the inkrecognition component to the IM in the case of failure, with anexample failure message. In this case the failure message indicatesthat the recognition failed due to invalid data format of thehandwriting data.

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">  <mmi:startResponse source="uri:inkRecognizerURI" context="URI-1" requestID="request-1" status="failure">    <mmi:statusInfo>      Invalid data format    </mmi:statusInfo>  </mmi:startResponse></mmi:mmi>

5.2.3 Exampleoutput event

Here's an example of an output event, sent from the inkrecognition component to the IM, using EMMA to represent theidentification results. Two results with different confidences arereturned.

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">  <mmi:doneNotification source="uri:inkRecognizerURI" context="URI-1" status="success" requestID="request-1">    <mmi:data>      <emma:emma version="1.0" xmlns:emma="http://www.w3.org/2003/04/emma">        <emma:one-of emma:medium="tactile" emma:verbal="true"         emma:mode="ink" emma:function="transcription">          <emma:interpretation emma:confidence=".8">            <text> 手 </text>          </emma:interpretation>          <emma:interpretation emma:confidence=".7">           <text> 于 </text>          </emma:interpretation>        </emma:one-of>      </emma:emma>    </mmi:data>  </mmi:doneNotification></mmi:mmi>

This is an example of EMMA output in the case where therecognizer is unable to find a suitable match to the inputhandwriting. The EMMA output contains an empty interpretationresult.

<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">  <mmi:doneNotification source="uri:inkRecognizerURI" context="URI-1" status="success" requestID="request-1" >    <mmi:data>      <emma:emma version="1.0" xmlns:emma="http://www.w3.org/2003/04/emma">        <emma:interpretation emma:confidence="0.0"         emma:medium="tactile" emma:verbal="true" emma:mode="ink"         emma:function="transcription" emma:uninterpreted="true"/>      </emma:emma>    </mmi:data>  </mmi:doneNotification></mmi:mmi>

6 Example simplemodality: Video Display

6.1Functions of a Possible Video Display Component

Consider a theoretical video display modality component thatreceives a video file and displays it in a screen. An API to thatmodality component would include events for starting the component,providing the video file and for receiving player statusinformation back from the component. This example includes theinformation needed to run this component in the "startRequest"event and shows a display codec problem. In order to focus in thebehavior of the output modality component, this example assumesthat the video file is given from some source; however, anotherpossibility would be to also include video acquisition in acomposite (input/output) and more complex real-time displaycomponent. Depending on the capabilities of the modality component,other possible information that might be included would be thevideo formats supported. The MMI Runtime Framework could use thefollowing events to communicate with such a component.

Table 5: Component behavior of Video Display with respectto modality component guidelines.
GuidelineComponent Information
Guideline 1: Each modality component must implement all of theMMI life cycle events.See Table 6 for the details of the implementation of thelife-cycle events.
Guideline 2: Identify other functions of the modality componentthat are relevant to the interaction manager.All the functions of the component are covered in thelife-cycle events, no other functions are needed.
Guideline 3: If the component uses media, specify the mediaformat.The component uses for the moment only the h.264 codecformat.
Guideline 4: Specify protocols supported by the component fortransmitting media (e.g. SIP).The component uses HTTP for transmitting media.
Guideline 5: Specify supported human languages.This component does not support any human languages.
Guideline 6: Specify supporting languages required by thecomponent.This component does not require any markup languages.
Guideline 7: Modality components sending data to theinteraction manager must use the EMMA format.This component uses EMMA.
Table 6: Component behavior of Video Display for eachlife-cycle event.
Life Cycle EventComponent Implementation
newContextRequest(Standard) The component requests a new context from theIM.
newContextResponse(Standard) The component starts a new context and assigns thenew context id to it.
prepareRequestThe component prepares resources to be used in displayconfiguration, specifically, the supported formats table.
prepareResponse(Standard) If the recognizer failed to find a matchingrecognizer for the request language script, a relevant errormessage is returned in the <statusInfo> element.
startRequestThe component starts displaying video if possible. The<mmi:data> element might hold a<video-display-parameters> element containing a "videoFile"attribute. The "videoFile" attribute contains the URI referencingthe video content.
startResponse(Standard)If the current video format (WVM) is not found in thesupported codec formats table, the error message "codec notsupported" is returned in the <statusInfo> element.
doneNotificationDisplay state in EMMA format are reported in the "data" field.The mode is "video", the medium is "visual", the function is"playing", and verbal is "false".
cancelRequestThis component stops processing when it receives a"cancelRequest". It always performs a hard stop irrespective of theIM request.
cancelResponse(Standard)
pauseRequest(Standard)
pauseResponse(Standard)
resumeRequest(Standard)
resumeResponse(Standard)
extensionNotificationThis component does not use "extensionNotification". It ignoresany "extensionNotification" events that are sent to it by theIM.
clearContextRequest(Standard)
clearContextResponse(Standard)
statusRequest(Standard)
statusResponseThe component returns a standard life cycle response. The"automaticUpdate" attribute is "false", because this component doesnot supply automatic updates.

Note: "(Standard)" means that the component does not doanything over and above the actions specified by the MMIArchitecture.

6.2 Event Syntax

6.2.1Examples of events for starting the component

To start the component, a startRequest event from the IM to thedisplay component is sent, asking it to start a video display. Itgives information about a video file in a certain URI.

<mmi xmlns="http://www.w3.org/2008/04/mmi-arch" version="1.0">  <mmi:startRequest source="uri:RTFURI" context="URI-1" requestID="request-1">    <mmi:data>      <video-display-parameters videoFile="someURI"/>    </mmi:data>  </mmi:startRequest></mmi:mmi>

As part of support for the life-cycle events, a modalitycomponent is required to respond to a startRequest event with astartResponse event. Here's an example of a startResponse from thedisplay component to the IM informing the IM that the component issuccessfully started and video is displaying.

 <mmi xmlns="http://www.w3.org/2008/04/mmi-arch" version="1.0">  <mmi:startResponse source="uri:displayURI" context="URI-1" requestID="request-1" status="success"/> </mmi:mmi>

Here's an example of a startResponse event from the displaycomponent to the IM in the case of failure, with an example failuremessage. In this case the failure message indicates that the videocodec is not supported.

<mmi xmlns="http://www.w3.org/2008/04/mmi-arch" version="1.0">  <mmi:startResponse source="uri:displayURI" context="URI-1" requestID="request-1" status="failure">    <mmi:statusInfo>      WVM codec not supported    </mmi:statusInfo>  </mmi:startResponse></mmi:mmi>

A References

MMI-ARCH
"MultimodalArchitecture and Interfaces (Working Draft)" , JimBarnett et al. editors. This specification describes a looselycoupled architecture for multimodal user interfaces, which allowsfor co-resident and distributed implementations, and focuses on therole of markup and scripting, and the use of well definedinterfaces between its constituents. World Wide Web Consortium,2011.
MMIF
"W3CMultimodal Interaction Framework" , James A. Larson,T.V. Raman and Dave Raggett, editors, World Wide Web Consortium,2003.
EMMA
"Extensiblemultimodal Annotation markup language (EMMA)", MichaelJohnson et al. editors. EMMA is an XML format for annotatingapplication specific interpretations of user input with informationsuch as confidence scores, time stamps, input modality andalternative recognition hypotheses, World Wide Web Consortium,2009.
VoiceXML
"VoiceExtensible Markup Language (VoiceXML) Version 2.1" ,Matt Oshry et al. editors. World Wide Web Consortium, 2007.
SSML
"SpeechSynthesis Markup Language (SSML) Version 1.1" , DanielC. Burnett et al. editors. World Wide Web Consortium, 2010.
SRGS
"SpeechRecognition Grammar Specification Version 1.0" , AndrewHunt et al. editors. World Wide Web Consortium, 2004.
SISR
"SemanticInterpretation for Speech Recognition (SISR) Version1.0" , Luc Van Tichelen al. editors. World Wide WebConsortium, 2004.
[8]ページ先頭
©2009-2025 Movatter.jp