Movatterモバイル変換

EMMA: Extensible MultiModal Annotationmarkup language

W3C Recommendation10 February 2009

This version:

http://www.w3.org/TR/2009/REC-emma-20090210/

Latest version:

http://www.w3.org/TR/emma/

Previous version:

http://www.w3.org/TR/2008/PR-emma-20081215/

Editor:

Michael Johnston, AT&T

Authors:

Paolo Baggia, Loquendo

Daniel C. Burnett, Voxeo (formerly of Vocalocity and Nuance)

Jerry Carter, Nuance

Deborah A. Dahl, Invited Expert

Gerry McCobb, Openstream

Dave Raggett, (until 2007, while at W3C/Volantis and W3C/Canon)

Please refer to theerrata for this document, which may include some normative corrections.

See alsotranslations.

Copyright © 2009W3C^® (MIT,ERCIM,Keio), All Rights Reserved. W3Cliability,trademark anddocument use rules apply.

---

Abstract

The W3C Multimodal Interaction Working Group aims to developspecifications to enable access to the Web using multimodalinteraction. This document is part of a set of specifications formultimodal systems, and provides details of an XML markup languagefor containing and annotating the interpretation of user input.Examples of interpretation of user input are a transcription intowords of a raw signal, for instance derived from speech, pen orkeystroke input, a set of attribute/value pairs describing theirmeaning, or a set of attribute/value pairs describing a gesture.The interpretation of the user's input is expected to be generatedby signal interpretation processes, such as speech and inkrecognition, semantic interpreters, and other types of processorsfor use by components that act on the user's inputs such asinteraction managers.

Status of this Document

This section describes the status of this document at thetime of its publication. Other documents may supersede thisdocument. A list of current W3C publications and the latestrevision of this technical report can be found in theW3C technical reports index athttp://www.w3.org/TR/.

This is theRecommendationof "EMMA: Extensible MultiModal Annotation markup language".It has been produced by theMultimodal Interaction Working Group,which is part of theMultimodal Interaction Activity.

Comments are welcome onwww-multimodal@w3.org(archive).SeeW3C mailing list and archiveusage guidelines.

The design of EMMA has been widely reviewed(see thedisposition of comments)and satisfies the Working Group's technical requirements.A list of implementations is included in theEMMA Implementation Report.The Working Group made a few editorial changes to the15 December 2008 Proposed Recommendation.Changes from the Proposed Recommendation can be found inAppendix F.

This document has been reviewed by W3C Members, by software developers, and by other W3C groups and interested parties, and is endorsed by the Director as a W3C Recommendation. It is a stable document and may be used as reference material or cited from another document. W3C's role in making the Recommendation is to draw attention to the specification and to promote its widespread deployment. This enhances the functionality and interoperability of the Web.

This specification describes markup for representinginterpretations of user input (speech, keystrokes, pen input etc.)together with annotations for confidence scores, timestamps, inputmedium etc., and forms part of the proposals for theW3C Multimodal InteractionFramework.

This document was produced by a group operating under the5February 2004 W3C Patent Policy. W3C maintains apublic list of anypatent disclosures made in connection with the deliverables ofthe group; that page also includes instructions for disclosing apatent. An individual who has actual knowledge of a patent whichthe individual believes containsEssential Claim(s) must disclose the information in accordancewithsection 6 of the W3C Patent Policy.

The sections in the main body of this document are normative unlessotherwise specified. The appendices in this document are informativeunless otherwise indicated explicitly.

Conventions of this Document

All sections in this specification are normative, unlessotherwise indicated. The informative parts of this specificationare identified by "Informative" labels within sections.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALLNOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL"in this document are to be interpreted as described in [RFC2119].

Table of Contents

• 1.Introduction
  • 1.1Uses of EMMA
  • 1.2Terminology
• 2.Structure of EMMA documents
  • 2.1Data model
  • 2.2EMMA namespaceprefixes
• 3.EMMA structural elements
  • 3.1Root element:emma:emma
  • 3.2Interpretation element:emma:interpretation
  • 3.3Container elements
    • 3.3.1emma:one-ofelement
    • 3.3.2emma:group element
      • 3.3.2.1Indirect grouping criteria:emma:group-info element
    • 3.3.3emma:sequenceelement
  • 3.4Lattice element
    • 3.4.1Lattice markup:emma:lattice,emma:arc,emma:node elements
    • 3.4.2Annotations on lattices
    • 3.4.3Relative timestamps onlattices
  • 3.5Literal semantics:emma:literal element
• 4EMMA annotations
  • 4.1EMMA annotation elements
    • 4.1.1Data model:emma:modelelement
    • 4.1.2Interpretation derivation:emma:derived-from element andemma:derivation element
    • 4.1.3Reference to grammar used:emma:grammar element
    • 4.1.4Extensibility to application/vendorspecific annotations:emma:info element
    • 4.1.5Endpoint reference:emma:endpoint-info element andemma:endpoint element
  • 4.2EMMA annotation attributes
    • 4.2.1Tokens of input:emma:tokens attribute
    • 4.2.2Reference to processing:emma:process attribute
    • 4.2.3Lack of input:emma:no-input attribute
    • 4.2.4Uninterpreted input:emma:uninterpreted attribute
    • 4.2.5Human language of input:emma:lang attribute
    • 4.2.6Reference to signal:emma:signalandemma:signal-size attributes
    • 4.2.7Media type:emma:media-type attribute
    • 4.2.8Confidence scores:emma:confidence attribute
    • 4.2.9Input source:emma:sourceattribute
    • 4.2.10Timestamps
      • 4.2.10.1Absolute timestamps:emma:start,emma:end attributes
      • 4.2.10.2Relative timestamps:emma:time-ref-uri,emma:time-ref-anchor-point,emma:offset-to-start attributes
      • 4.2.10.3Duration of input:emma:duration attribute
      • 4.2.10.4Composite Input andRelative Timestamps
    • 4.2.11Medium, mode, and function of userinputs:emma:medium,emma:mode,emma:function,emma:verbalattributes
    • 4.2.12Composite multimodality:emma:hook attribute
    • 4.2.13Cost:emma:costattribute
    • 4.2.14Endpoint properties:emma:endpoint-role,emma:endpoint-address,emma:port-type,emma:port-num,emma:message-id,emma:service-name,emma:endpoint-pair-ref,emma:endpoint-info-refattributes
    • 4.2.15Reference toemma:grammar element:emma:grammar-refattribute
    • 4.2.16Reference toemma:modelelement:emma:model-ref attribute
    • 4.2.17Dialog turns:emma:dialog-turn attribute
  • 4.3Scope of EMMA annotations
• 5.Conformance
  • 5.1Conforming EMMA Documents
  • 5.2Using EMMA with other Namespaces
  • 5.3Conforming EMMA Processors
• Appendices
  • Appendix A.XML andRELAX NGschemata(Normative)
  • Appendix B.MIME type(Normative)
    • B.1Registration ofMIME media type application/emma+xml
  • Appendix C.emma:hook and SRGS(Informative)
  • Appendix D.EMMA event interface(Informative)
  • Appendix E.References
    • E.1Normative references
    • E.2Informativereferences
  • Appendix F.Changes since last draft(Informative)
  • Appendix G.Acknowledgements(Informative)

1. Introduction

This section isInformative.

This document presents an XML specification for EMMA, anExtensible MultiModal Annotation markup language, responding to therequirements documented inRequirements for EMMA[EMMARequirements]. Thismarkup language is intended for use by systems that providesemantic interpretations for a variety of inputs, including but notnecessarily limited to, speech, natural language text, GUI and inkinput.

It is expected that this markup will be used primarily as astandard data interchange format between the components of amultimodal system; in particular, it will normally be automaticallygenerated by interpretation components to represent the semanticsof users' inputs, not directly authored by developers.

The language is focused on annotating single inputs from users,which may be either from a single mode or a composite inputcombining information from multiple modes, as opposed toinformation that might have been collected over multiple turns of adialog. The language provides a set of elements and attributes thatare focused on enabling annotations on user inputs andinterpretations of those inputs.

An EMMA document can be considered to hold three types ofdata:

• instance data

  Application-specific markup corresponding to input informationwhich is meaningful to the consumer of an EMMA document. Instancesare application-specific and built by input processors at runtime.Given that utterances may be ambiguous with respect to inputvalues, an EMMA document may hold more than one instance.

• data model

  Constraints on structure and content of an instance. The datamodel is typically pre-established by an application, and may beimplicit, that is, unspecified.

• metadata

  Annotations associated with the data contained in the instance.Annotation values are added by input processors at runtime.

Given the assumptions above about the nature of data representedin an EMMA document, the following general principles apply to thedesign of EMMA:

• The main prescriptive content of the EMMA specification willconsist of metadata: EMMA will provide a means to express themetadata annotations which require standardization. (Notice,however, that such annotations may express the relationship amongall the types of data within an EMMA document.)
• The instance and its data model are assumed to be specified inXML, but EMMA will remain agnostic to the XML format used toexpress these. (The instance XML is assumed to be sufficientlystructured to enable the association of annotative data.)
• The extensibility of EMMA lies in the ability for additionalkinds of metadata to be included in application specificvocabularies. EMMA itself can be extended with application andvendor specific annotations contained within theemma:info element(Section4.1.4).

The annotations of EMMA should be considered 'normative' in thesense that if an EMMA component produces annotations as describedinSection 3andSection4, these annotations must be represented using the EMMAsyntax. The Multimodal Interaction Working Group may address inlater drafts the issues of modularization and profiling; that is,which sets of annotations are to be supported by which classes ofEMMA component.

1.1 Uses of EMMA

The general purpose of EMMA is to represent informationautomatically extracted from a user's input by an interpretationcomponent, where input is to be taken in the general sense of ameaningful user input in any modality supported by the platform.The reader should refer to the sample architecture inW3CMultimodal Interaction Framework[MMIFramework], which shows EMMA conveying content betweenuser input modality components and an interaction manager.

Components that generate EMMA markup:

1. Speech recognizers
2. Handwriting recognizers
3. Natural language understanding engines
4. Other input media interpreters (e.g. DTMF, pointing,keyboard)
5. Multimodal integration component

Components that use EMMA include:

1. Interaction manager
2. Multimodal integration component

Although not a primary goal of EMMA, a platform may also chooseto use this general format as the basis of a general semanticresult that is carried along and filled out during each stage ofprocessing. In addition, future systems may also potentially makeuse of this markup to convey abstract semantic content to berendered into natural language by a natural language generationcomponent.

1.2 Terminology

anchor point

When referencing an input interval withemma:time-ref-uri,emma:time-ref-anchor-point allows you to specifywhether the referenced anchor is the start or end of theinterval.

annotation

Information about the interpreted input, for example,timestamps, confidence scores, links to raw input, etc.

composite input

An input formed from several pieces, often in different modes,for example, a combination of speech and pen gesture, such assaying "zoom in here" and circling a region on a map.

confidence

A numerical score describing the degree of certainty in aparticular interpretation of user input.

data model

For EMMA, a data model defines a set of constraints on possibleinterpretations of user input.

derivation

Interpretations of user input are said to be derived from thatinput, and higher level interpretations may be derived from lowerlevel ones. EMMA allows you to reference the user input orinterpretation a given interpretation was derived from, seesemanticinterpretation.

dialog

For EMMA, dialog can be considered as a sequence ofinteractions between a user and the application.

endpoint

In EMMA, this refers to a network location which is the sourceor recipient of an EMMA document. It should be noted that the usageof the term "endpoint" in this context is different from the waythat the term is used in speech processing, where it refers to theend of a speech input.

gestures

In multimodal applications gestures are communicative acts madeby the user or application. An example is circling an area on a mapto indicate a region of interest. Users may be able to gesture witha pen, keystrokes, hand movements, headmovements, or sound. Gestures often form part ofcomposite input. Applicationgestures are typically animations and/or sound effects.

grammar

A set of rules that describe a sequence of tokens expected in agiven input. These can be used by speech and handwritingrecognizers to increase recognition accuracy.

handwriting recognition

The process of converting pen strokes into text.

ink recognition

This includes the recognition of handwriting and pengestures.

input cost

In EMMA, this refers to a numerical measure indicating theweight or processing cost associated with a user's input or part oftheir input.

input device

The device proving a particular input, for example, amicrophone, a pen, a mouse, a camera, or a keyboard.

input function

In EMMA, this refers tothe use a particular inputis serving, for example, as part of a recording or transcription,as part of a dialog, or as a means to verify the user'sidentity.

input medium

Whether the input is acoustic, visual, or tactile, forinstance, a spoken utterance is an example of an aural input, ahand gesture as seen by a camera is an example of a visual input,pointing with a mouse or pen is an example of a tactile input.

input mode

This distinguishes a particular means of providing an inputwithin a general input medium, for example, speech, DTMF, ink, keystrokes, video, photograph, etc.

input source

This is the device that provided the input, for example aparticular microphone or camera. EMMA allows you to identify thesewith a URI.

input tokens

In EMMA, this refers to a sequence of characters, words orother discrete units of input.

instance data

A representation in XML of an interpretation of userinput.

interaction manager

A processor that determines how an application interacts with auser. This can be at multiple levels of abstraction, for example,at a detailed level, determining what prompts to present to theuser and what actions to take in response to user input, versus ahigher level treatment in terms of goals and tasks for achievingthose goals. Interaction managers are frequently event driven.

interpretation

In EMMA, an interpretation of user input refers to informationderived from the user input that is meaningful to theapplication.

keystroke input

Input provided by the user pressing on a sequence of keys(buttons), such as a computer keyboard or keypad.

lattice

A set of nodes interconnected with directed arcs such that byfollowing an arc, you can never find yourself back at a node youhave already visited (i.e. a directed acyclic graph). Latticesprovide a flexible means to represent the results of speech andhandwriting recognition, in terms of arcs representing words orcharacter sequences. Different arcs from the same node representdifferent local hypotheses as to what the user said or wrote.

metadata

Information describing another set of data, for instance, alibrary catalog card with information on the author, title andlocation of a book. EMMA is designed to support input processors inproviding metadata for interpretations of user input.

multimodal integration

The process of combining inputs from different modes to createan interpretation of composite input. This is also sometimesreferred to asmultimodal fusion.

multimodal interaction

The means for a user to interact with an application using morethan one mode of interaction, for instance, offering the user thechoice of speaking or typing, or in some cases, allowing the userto provide a composite input involving multiple modes.

natural languageunderstanding

The process of interpreting text in terms that are useful foran application.

N-best list

An N-best list is a list of the most likely hypotheses for whatthe user actually said or wrote, where N stands for an integralnumber such as 5 for the 5 most likely hypotheses.

raw signal

An uninterpreted input, such as an audio waveform captured froma microphone.

semantic interpretation

A normalized representation of the meaning of a user input, forinstance, mapping the speech for "San Francisco" into the airportcode "SFO".

semantic processor

In EMMA, this refers to systems that can derive interpretationsof user input, for instance, mapping the speech for "San Francisco"into the airport code "SFO".

signal interpretation

The process of mapping a discrete or continuous signal into asymbolic representation that can be used by an application, forinstance, transforming the audio waveform corresponding to someonesaying "2005" into the number 2005.

speech recognition

The process of determining the textual transcription of a pieceof speech.

speech synthesis

The process of rendering a piece of text into the correspondingspeech, i.e. synthesizing speech from text.

text to speech

The process of rendering a piece of text into the correspondingspeech.

time stamp

The time that a particular input or part of an input began orended.

URI: Uniform Resource Identifier

A URI is a unifying syntax for the expression of names andaddresses of objects on the network as used in the World Wide Web.Within this specification, the term URI refers to a UniversalResource Identifier as defined in [RFC3986]and extended in [RFC3987] with the new nameIRI. The term URI has been retained in preference to IRI to avoidintroducing new names for concepts such as "Base URI" that aredefined or referenced across the whole family of XMLspecifications. A URI is defined as any legalanyURI primitive as defined in XML Schema Part 2:Datatypes Second Edition Section 3.2.17 [SCHEMA2].

user input

An input provided by a user as opposed to something generatedautomatically.

2. Structure of EMMA documents

This section isInformative.

As noted above, the main components of an interpreted user inputin EMMA are the instance data, an optional data model, and themetadata annotations that may be applied to that input. Therealization of these components in EMMA is as follows:

• instance data is contained within an EMMAinterpretation
• thedata model is optionally specified as an annotationof that instance
• EMMAannotations may be applied at different levels ofan EMMA document.

An EMMAinterpretation is the primary unit for holdinguser input as interpreted by an EMMA processor. As will be seenbelow, multiple interpretations of a single input are possible.

EMMA provides a simple structural syntax for the organization ofinterpretations and instances, and an annotative syntax to applythe annotation to the input data at different levels.

An outline of the structural syntax and annotations found inEMMA documents is as follows. A fuller definition may be found inthe description of individual elements and attributes inSection 3 andSection 4.

• EMMAstructuralelements (Section 3)
  • Root element: The root node of anEMMA document, theemma:emma element, holds EMMAversion and namespace information, and provides a container for oneor more of the following interpretation and container elements(Section 3.1)
  • Interpretation element: Theemma:interpretation element contains a giveninterpretation of the input and holds application specific markup(Section 3.2)
  • Container elements:
    • emma:one-of is a container for one or moreinterpretation elements or container elements and denotes thatthese are mutually exclusive interpretations (Section 3.3.1)
    • emma:group is a general container for one or moreinterpretation elements or container elements. It can be associatedwith arbitrary grouping criteria (Section3.3.2).
    • emma:sequence is a container for one or moreinterpretation elements or container elements and denotes thatthese are sequential in time (Section3.3.3).
  • Lattice element: Theemma:lattice element is used to contain a series ofemma:arc andemma:node elements thatdefine a lattice of words, gestures, meanings or other symbols. Theemma:lattice element appears within theemma:interpretation element (Section3.4)
  • Literal element: Theemma:literal element is used as a wrapper when theapplication semantics is a string literal. (Section3.5)
• EMMA annotations (Section 4)
  • EMMA annotation elements: These areEMMA annotations such asemma:derived-from,emma:endpoint-info, andemma:info whichare represented as elements so that they can occur more than oncewithin an element and can contain internal structure. (Section 4.1)
  • EMMA annotation attributes: Theseare EMMA annotations such asemma:start,emma:end ,emma:confidence, andemma:tokens which are represented as attributes. Theycan appear onemma:interpretation elements.Some can appear on container elements, lattice elements, andelements in the application-specific markup. (Section 4.2)

From the defined root nodeemma:emma the structureof an EMMA document consists of a tree of EMMA container elements(emma:one-of,emma:sequence,emma:group) terminating in a number of interpretationelements (emma:interpretation). Theemma:interpretation elements serve as wrappers foreither application namespace markup describing the interpretationof the users input or anemma:lattice element oremma:literal element . A singleemma:interpretation may also appear directly under theroot node.

The EMMA elementsemma:emma,emma:interpretation,emma:one-of,andemma:literaland the EMMA attributesemma:no-input,emma:uninterpreted,emma:medium,andemma:modeare required of allimplementations. The remaining elements and attributes are optionaland may be used in some implementations and not other depending on thespecific modalities and processing being represented.

To illustrate this, here is an exampleofan EMMA documentrepresenting inputto a flight reservation application. In this example there are twospeech recognition results and associated semantic representationsof the input. The system is uncertain whether the user meant"flights from Boston to Denver" or "flights from Austin to Denver".The annotations to be captured are timestamps and confidence scoresfor the two inputs.

Example:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:one-of emma:start="1087995961542" emma:end="1087995963542"     emma:medium="acoustic" emma:mode="voice">    <emma:interpretation emma:confidence="0.75"    emma:tokens="flights from boston to denver">      <origin>Boston</origin>      <destination>Denver</destination>    </emma:interpretation>    <emma:interpretation emma:confidence="0.68"    emma:tokens="flights from austin to denver">      <origin>Austin</origin>      <destination>Denver</destination>    </emma:interpretation>  </emma:one-of></emma:emma>

Attributes on the rootemma:emma element indicatethe version and namespace. Theemma:emma elementcontains anemma:one-of element which contains adisjunctive list of possible interpretations of the input. Theactual semantic representation of each interpretation is within theapplication namespace. In the example here the application specificsemantics involves elementsorigin anddestination indicating the origin and destinationcities for looking up a flight. The timestamp is the same for bothinterpretations and it is annotated using values in milliseconds intheemma:start andemma:end attributes ontheemma:one-of. The confidence scores and tokensassociated with each of the inputs are annotated using the EMMAannotation attributesemma:confidence andemma:tokens on each of theemma:interpretation elements.

2.1 Data model

An EMMA data model expresses the constraints on the structureand content of instance data, for the purposes of validation. Assuch, the data model may be considered as a particular kind ofannotation (although, unlike other EMMA annotations, it is not afeature pertainingto a specific user input at aspecific moment in time, it is rather a static and, by its verydefinition, application-specific structure).Thespecification ofa data model in EMMA is optional.

Since Web applications today use different formats to specifydata models, e.g.XML Schema Part 1: Structures SecondEdition [XML SchemaStructures], XForms1.0 (SecondEdition) [XFORMS],RELAX NGSpecification [RELAX-NG], etc., EMMAitself is agnostic to the format of data model used.

Data model definition and reference is defined inSection 4.1.1.

2.2 EMMA namespace prefixes

An EMMA attribute is qualified with the EMMA namespace prefix ifthe attribute can also be used as an in-line annotation on elementsin the application's namespace. Most of the EMMA annotationattributes inSection 4.2 are in this category.An EMMA attribute is not qualified with the EMMA namespace prefixif the attribute only appears on an EMMA element. This rule ensuresconsistent usage of the attributes across all examples.

Attributes from other namespaces are permissible on all EMMAelements. As an examplexml:lang may be used toannotate the human language of character data content.

3. EMMA structural elements

This section defines elements in the EMMA namespace whichprovide the structural syntax of EMMA documents.

3.1 Root element:emma:emma

The root element of an EMMA document is namedemma:emma. It holds a singleemma:interpretation or EMMA container element(emma:one-of,emma:sequence,emma:group). It MAY also contain a singleemma:derivation element containing earlier stages ofthe processing of the input (SeeSection4.1.2). It MAY also contain an optional single annotationelement:emma:info and multiple optionalemma:grammar,emma:model, andemma:endpoint-info elements.

It MAY hold attributes for information pertaining to EMMAitself, along with any namespaces which are declared for the entiredocument, and any other EMMA annotative data. Theemma:emma element and other elements and attributesdefined in this specification belong to the XML namespaceidentified by the URI "http://www.w3.org/2003/04/emma". In theexamples, the EMMA namespace is generally declared using theattributexmlns:emma on the rootemma:emma element. EMMA processors MUST support thefull range of ways of declaring XML namespaces as defined by theNamespaces in XML 1.1 (Second Edition) [XMLNS]. Application markup MAY be declared in anexplicit application namespace, or an undefined namespace(equivalent to setting xmlns="").

For example:

<emma:emma version="1.0" xmlns:emma="http://www.w3.org/2003/04/emma">    ....</emma:emma>

or

<emma version="1.0" xmlns="http://www.w3.org/2003/04/emma">    ....</emma>

3.2 Interpretation element:emma:interpretation

Theemma:interpretation element holds a singleinterpretation represented in application specific markup, or asingleemma:lattice element, or a singleemma:literal element.

Theemma:interpretation element MUST be empty if itis marked withemma:no-input="true"(Section 4.2.3). Theemma:interpretation elementMUST be emptyif it has been annotated withemma:uninterpreted="true"(Section 4.2.4) oremma:function="recording"(Section 4.2.11).

Attributes:

1. id a REQUIREDxsd:ID value that uniquelyidentifies the interpretation within the EMMA document.

<emma:emma version="1.0" xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation emma:medium="acoustic" emma:mode="voice">    ...  </emma:interpretation></emma:emma>

Whileemma:medium andemma:mode areoptional onemma:interpretation, note that all EMMAinterpretations must be annotated foremma:medium andemma:mode, so either these attributes must appeardirectly onemma:interpretation or they must appear onan ancestoremma:one-of node or they must appear on anearlier stage of the derivation listed inemma:derivation.

3.3 Container elements

3.3.1emma:one-of element

Theemma:one-of element acts as a container for acollection of one or more interpretation(emma:interpretation) or container elements(emma:one-of,emma:group,emma:sequence), and denotes that these are mutuallyexclusive interpretations.

An N-best list of choices in EMMA MUST be represented as a setofemma:interpretation elements contained within anemma:one-of element. For instance, a series ofdifferent recognition results in speech recognition might berepresented in this way.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:one-ofemma:medium="acoustic" emma:mode="voice">    <emma:interpretation>      <origin>Boston</origin>      <destination>Denver</destination>      <date>03112003</date>    </emma:interpretation>    <emma:interpretation>      <origin>Austin</origin>      <destination>Denver</destination>      <date>03112003</date>    </emma:interpretation>  </emma:one-of></emma:emma>

The function of theemma:one-of element is torepresent a disjunctive list of possible interpretations of a userinput. A disjunction of possible interpretations of an input can bethe result of different kinds of processing or ambiguity. Onesource is multiple results from a recognition technology such asspeech or handwriting recognition. Multiple results can also occurfrom parsing or understanding natural language. Another possiblesource of ambiguity is from the application of multiple differentkinds of recognition or understanding components to the same inputsignal. For example, an single ink input signal might be processedby both handwriting recognition and gesture recognition. Another isthe use of more than one recording device for the same input(multiple microphones).

In order to make explicit these different kinds of multipleinterpretations and allow for concise statement of the annotationsassociated with each, theemma:one-of element MAYappear within anotheremma:one-of element. Ifemma:one-of elements are nested then they MUSTindicate the kind of disjunction using the attributedisjunction-type. The values ofdisjunction-type are{recognition,understanding, multi-device, and multi-process}. For themost common use case, where there are multiple recognition resultsand some of them have multiple interpretations, the top-levelemma:one-of isdisjunction-type="recognition" and the embeddedemma:one-of has the attributedisjunction-type="understanding".

As an example, in an interactive flight reservation application,recognition yielded 'Boston' or 'Austin' and each had a semanticinterpretation as either the assertion of city name or thespecification of a flight query with the city as the destination,this would be represented as follows in EMMA:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:one-of disjunction-type="recognition"      start="12457990" end="12457995"emma:medium="acoustic" emma:mode="voice">     <emma:one-of disjunction-type="understanding"         emma:tokens="boston">       <emma:interpretation>          <assert><city>boston</city></assert>       </emma:interpretation>       <emma:interpretation>          <flight><dest><city>boston</city></dest></flight>       </emma:interpretation>     </emma:one-of>     <emma:one-of disjunction-type="understanding"         emma:tokens="austin">       <emma:interpretation>          <assert><city>austin</city></assert>       </emma:interpretation>       <emma:interpretation>          <flight><dest><city>austin</city></dest></flight>       </emma:interpretation>     </emma:one-of>  </emma:one-of></emma:emma>

EMMA MAY explicitly represent ambiguity resulting from differentprocesses, devices, or sources using embeddedemma:one-of and thedisjunction-typeattribute. Multiple different interpretations resulting fromdifferent factors MAY also be listed within a single unstructuredemma:one-of though in this case it is more complex orimpossible to uncover the sources of the ambiguity if required bylater stages of processing. If there is no embedding inemma:one-of, then thedisjunction-typeattribute is not required. If thedisjunction-typeattribute is missing then by default the source of disjunction isunspecified.

The example case above could also be represented as:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:one-of  start="12457990" end="12457995"         emma:medium="acoustic" emma:mode="voice">     <emma:interpretation emma:tokens="boston">        <assert><city>boston</city></assert>     </emma:interpretation>     <emma:interpretation >        <flight><dest><city>boston</city></dest></flight>     </emma:interpretation>     <emma:interpretation emma:tokens="austin">        <assert><city>austin</city></assert>     </emma:interpretation>     <emma:interpretation emma:tokens="austin">        <flight><dest><city>austin</city></dest></flight>     </emma:interpretation>  </emma:one-of></emma:emma>

But in this case information about which interpretationsresulted from speech recognition and which resulted from languageunderstanding is lost.

A list ofemma:interpretation elements within anemma:one-of MUST be sorted best-first by some measureof quality. The quality measure isemma:confidence ifpresent, otherwise, the quality metric is platform-specific.

With embeddedemma:one-of structures there is norequirement for the confidence scores within differentemma:one-of to be on the same scale. For example, thescores assigned by handwriting recognition might not be comparableto those assigned by gesture recognition. Similarly, if multiplerecognizers are used there is no guarantee that their confidencescores will be comparable. For this reason the ordering requirementonemma:interpretation withinemma:one-ofonly applies locally to sisteremma:interpretationelements within eachemma:one-of. There is norequirement on the ordering of embeddedemma:one-ofelements within a higheremma:one-of element.

Whileemma:medium andemma:mode areoptional onemma:one-of, note that all EMMAinterpretations must be annotated foremma:medium andemma:mode, so either these annotations must appeardirectly on all of the containedemma:interpretationelements within theemma:one-of, or they must appearon theemma:one-of element itself, or they must appearon an ancestoremma:one-of element, or they mustappear on an earlier stage of the derivation listed inemma:derivation.

3.3.2emma:group element

Theemma:group element is used to indicate that thecontained interpretations are from distinct user inputs that arerelated in some manner.emma:group MUST NOT be usedfor containing the multiple stages of processing of a single userinput. Those MUST be contained in theemma:derivationelement instead(Section 4.1.2).For groups of inputs in temporal order the more specializedcontaineremma:sequence MUST be used(Section 3.3.3). The following example showsthree interpretations derived from the speech input "Move thisambulance here" and the tactile input related to two consecutivepoints on a map.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:group      emma:start="1087995961542"      emma:end="1087995964542">    <emma:interpretationemma:medium="acoustic" emma:mode="voice">      <action>move</action>      <object>ambulance</object>      <destination>here</destination>    </emma:interpretation>    <emma:interpretationemma:medium="tactile" emma:mode="ink">      <x>0.253</x>      <y>0.124</y>    </emma:interpretation>    <emma:interpretationemma:medium="tactile" emma:mode="ink">      <x>0.866</x>      <y>0.724</y>    </emma:interpretation>  </emma:group></emma:emma>

Theemma:one-of andemma:groupcontainers MAY be nested arbitrarily.

3.3.2.1 Indirect grouping criteria:emma:group-info element

Sometimes it may be convenient to indirectly associate a givengroup with information, such as grouping criteria. Theemma:group-info element might be used to make explicitthe criteria by which members of a group are associated. In thefollowing example, a group of two points is associated with adescription of grouping criteria based upon a sliding temporalwindow of two seconds duration.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example"    xmlns:ex="http://www.example.com/ns/group">  <emma:group>    <emma:group-info>      <ex:mode>temporal</ex:mode>      <ex:duration>2s</ex:duration>    </emma:group-info>    <emma:interpretation      emma:medium="tactile" emma:mode="ink">      <x>0.253</x>      <y>0.124</y>    </emma:interpretation>    <emma:interpretationemma:medium="tactile" emma:mode="ink">      <x>0.866</x>      <y>0.724</y>    </emma:interpretation>  </emma:group></emma:emma>

You might also useemma:group-info to refer to anamed grouping criterion using external reference, forinstance:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example"    xmlns:ex="http://www.example.com/ns/group">  <emma:group>    <emma:group-info ref="http://www.example.com/criterion42"/>    <emma:interpretationemma:medium="tactile" emma:mode="ink">      <x>0.253</x>      <y>0.124</y>    </emma:interpretation>    <emma:interpretationemma:medium="tactile" emma:mode="ink">      <x>0.866</x>      <y>0.724</y>    </emma:interpretation>  </emma:group></emma:emma>

3.3.3emma:sequence element

Theemma:sequence element is used to indicate thatthe contained interpretations are sequential in time, as in thefollowing example, which indicates that two points made with a penare in temporal order.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:sequence>    <emma:interpretationemma:medium="tactile" emma:mode="ink">      <x>0.253</x>      <y>0.124</y>    </emma:interpretation>    <emma:interpretationemma:medium="tactile" emma:mode="ink">      <x>0.866</x>      <y>0.724</y>    </emma:interpretation>  </emma:sequence></emma:emma>

Theemma:sequence container MAY be combined withemma:one-of andemma:group in arbitrarynesting structures. The order of children in the content of theemma:sequence element corresponds to a sequence ofinterpretations. This ordering does not imply any particulardefinition of sequentiality. EMMA processors are expected thereforeto use theemma:sequence element to holdinterpretations which are either strictly sequential in nature(e.g. the end-time of an interpretation precedes the start-time ofits follower), or which overlap in some manner (e.g. the start-timeof a follower interpretation precedes the end-time of itsprecedent). It is possible to use timestamps to provide finegrained annotation for the sequence of interpretations that aresequential in time(seeSection4.2.10).

In the following more complex example, a sequence of two pengestures inemma:sequence and a speech input inemma:interpretationis contained in anemma:group.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:group>     <emma:interpretation emma:medium="acoustic"         emma:mode="voice">       <action>move</action>       <object>this-battleship</object>       <destination>here</destination>     </emma:interpretation>     <emma:sequence>       <emma:interpretation emma:medium="tactile"           emma:mode="ink">         <x>0.253</x>         <y>0.124</y>       </emma:interpretation>     <emma:interpretation emma:medium="tactile"         emma:mode="ink">       <x>0.866</x>       <y>0.724</y>     </emma:interpretation>   </emma:sequence> </emma:group></emma:emma>

3.4 Lattice element

In addition to providing the ability to represent N-best listsof interpretations usingemma:one-of, EMMA alsoprovides the capability to represent lattices of words or othersymbols using theemma:lattice element. Latticesprovide a compact representation of large lists of possiblerecognition results or interpretations for speech, pen, ormultimodal inputs.

In addition to providing a representation for lattice outputfrom speech recognition, another important use case for lattices isfor representation of the results of gesture and handwritingrecognition from a pen modality component. Lattices can also beused to compactly represent multiple possible meaningrepresentations. Another use case for the lattice representation isfor associating confidence scores and other annotations withindividual words within a speech recognition result string.

Lattices are compactly described by a list of transitionsbetween nodes. For each transition the start and end nodes MUST bedefined, along with the label for the transition. Initial and finalnodes MUST also be indicated. The following figure provides agraphical representation of a speech recognition lattice whichcompactly represents eight different sequences of words.

which expands to:

a. flights to boston from portland today pleaseb. flights to austin from portland today pleasec. flights to boston from oakland today pleased. flights to austin from oakland today pleasee. flights to boston from portland tomorrowf. flights to austin from portland tomorrowg. flights to boston from oakland tomorrowh. flights to austin from oakland tomorrow

3.4.1 Lattice markup:emma:lattice,emma:arc,emma:node elements

In EMMA, a lattice is represented using an elementemma:lattice, which has attributesinitial andfinal for indicating theinitial and final nodes of the lattice. For the latticebelow, this will be:<emma:latticeinitial="1" final="8"/>. The nodes are numbered withintegers. If there is more than one distinct final node in thelattice the nodes MUST be represented as a space separated list inthe value of thefinal attribute e.g.<emma:lattice initial="1" final="9 10 23"/>.There MUST only be one initial node in an EMMA lattice. Eachtransition in the lattice is represented as an elementemma:arc with attributesfrom andto which indicate the nodes where the transitionstarts and ends. The arc's label is represented as the content oftheemma:arc element and MUST be any well-formedcharacter or XML content. In the example here the contents arewords. Empty (epsilon) transitions in a lattice MUST be representedin theemma:lattice representation asemma:arcempty elements, e.g.<emma:arc from="1" to="8"/>.

The example speech lattice above would be represented in EMMAmarkup as follows:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretationemma:medium="acoustic" emma:mode="voice">    <emma:lattice initial="1" final="8">      <emma:arc from="1" to="2">flights</emma:arc>      <emma:arc from="2" to="3">to</emma:arc>      <emma:arc from="3" to="4">boston</emma:arc>      <emma:arc from="3" to="4">austin</emma:arc>      <emma:arc from="4" to="5">from</emma:arc>      <emma:arc from="5" to="6">portland</emma:arc>      <emma:arc from="5" to="6">oakland</emma:arc>      <emma:arc from="6" to="7">today</emma:arc>      <emma:arc from="7" to="8">please</emma:arc>      <emma:arc from="6" to="8">tomorrow</emma:arc>    </emma:lattice>  </emma:interpretation></emma:emma>

Alternatively, if we wish to represent the same information asan N-best list usingemma:one-of, we would have themore verbose representation:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:one-ofemma:medium="acoustic" emma:mode="voice">    <emma:interpretation>      <text>flights to boston from portland today please</text>    </emma:interpretation>    <emma:interpretationid="interp2">      <text>flights to boston from portland tomorrow</text>    </emma:interpretation>    <emma:interpretation>      <text>flights to austin from portland today please</text>    </emma:interpretation>    <emma:interpretation>      <text>flights to austin from portland tomorrow</text>    </emma:interpretation>    <emma:interpretation>      <text>flights to boston from oakland today please</text>    </emma:interpretation>    <emma:interpretation>      <text>flights to boston from oakland tomorrow</text>    </emma:interpretation>    <emma:interpretation>      <text>flights to austin from oakland today please</text>    </emma:interpretation>    <emma:interpretation>      <text>flights to austin from oakland tomorrow</text>    </emma:interpretation>  </emma:one-of></emma:emma>

The lattice representation avoids the need to enumerate all ofthe possible word sequences. Also, as detailed below, theemma:lattice representation enables placement ofannotations on individual words in the input.

For use cases involving the representation of gesture/inklattices and use cases involving lattices of semanticinterpretations, EMMA allows for application namespace elements toappear withinemma:arc.

For example a sequence of two gestures, each of which isrecognized as either a line or a circle, might berepresented as follows:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretationemma:medium="acoustic" emma:mode="voice">    <emma:lattice initial="1" final="3">      <emma:arc from="1" to="2">        <circle radius="100"/>      </emma:arc>      <emma:arc from="2" to="3">        <line length="628"/>      </emma:arc>      <emma:arc from="1" to="2">        <circle radius="200"/>      </emma:arc>      <emma:arc from="2" to="3">        <line length="1256"/>      </emma:arc>    </emma:lattice>  </emma:interpretation></emma:emma>

As an example of a lattice of semantic interpretations, in atravel application where the source is either "Boston" or"Austin"and the destination is either "Newark" or "New York", thepossibilities might be represented in a lattice as follows:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretationemma:medium="acoustic" emma:mode="voice">    <emma:lattice initial="1" final="3">      <emma:arc from="1" to="2">        <source city="boston"/>      </emma:arc>      <emma:arc from="2" to="3">        <destination city="newark"/>      </emma:arc>      <emma:arc from="1" to="2">        <source city="austin"/>      </emma:arc>      <emma:arc from="2" to="3">        <destination city="new york"/>      </emma:arc>    </emma:lattice>  </emma:interpretation></emma:emma>

Theemma:arc element MAY contain either anapplication namespace element or character data. It MUST NOTcontain combinations of application namespace elements andcharacter data. However, anemma:info element MAYappear within anemma:arc element alongside characterdata, in order to allow for the association of vendor orapplication specific annotations on a single word or symbol in alattice.

So, in summary, there are four groupings of content that canappear withinemma:arc:

• Character Data e.g. a recognized word in a speech lattice.
• Character Data and a singleemma:info elementproviding vendor or application specific annotations that apply tothe character data.
• An application namespace element e.g. the gesture andsemantic interpretation lattice examples above.
• An application namespace element and a singleemma:info element providing vendor or applicationspecific annotations that apply to the character data.

3.4.2 Annotations on lattices

The encoding of lattice arcs as XML elements(emma:arc) enables arcs to be annotated with metadatasuch as timestamps, costs, or confidence scores:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretationemma:medium="acoustic" emma:mode="voice">    <emma:lattice initial="1" final="8">      <emma:arc       from="1"       to="2"       emma:start="1087995961542"       emma:end="1087995962042"       emma:cost="30">         flights      </emma:arc>      <emma:arc       from="2"       to="3"       emma:start="1087995962042"       emma:end="1087995962542"       emma:cost="20">         to      </emma:arc>      <emma:arc       from="3"       to="4"       emma:start="1087995962542"       emma:end="1087995963042"       emma:cost="50">         boston      </emma:arc>      <emma:arc       from="3"       to="4"       emma:start="1087995963042"       emma:end="1087995963742"       emma:cost="60">         austin      </emma:arc>      ...    </emma:lattice>  </emma:interpretation></emma:emma>

The following EMMA attributes MAY be placed onemma:arc elements: absolute timestamps(emma:start,emma:end), relativetimestamps (emma:offset-to-start,emma:duration),emma:confidence,emma:cost, the human language of the input(emma:lang),emma:medium,emma:mode, andemma:source. The use caseforemma:medium,emma:mode, andemma:source is for lattices which contains contentfrom different input modes. Theemma:arc element MAYalso contain anemma:info element for specification ofvendor and application specific annotations on the arc.

The timestamps that appear onemma:arc elements donot necessarily indicate the start and end of the arc itself. TheyMAY indicate the start and end of the signal corresponding to thelabel on the arc. As a result there is no requirement that theemma:end timestamp on an arc going into a node shouldbe equivalent to theemma:start of all arcs going outof that node. Furthermore there is no guarantee that the left toright order of arcs in a lattice will correspond to the temporalorder of the input signal. The lattice representation is anabstraction that represents a range of possible interpretations ofa user's input and is not intended to necessarily be arepresentation of temporal order.

Costs are typically application and device dependent. There area variety of ways that individual arc costs might be combined toproduce costs for specific paths through the lattice. Thisspecification does not standardize the way for these costs to becombined; it is up to the applications and devices to determine howsuch derived costs would be computed and used.

For some lattice formats, it is also desirable to annotate thenodes in the lattice themselves with information such as costs. Forexample in speech recognition, costs might be placed on nodes as aresult of word penalties or redistribution of costs. For thispurpose EMMA also provides anemma:node element whichcan host annotations such asemma:cost. Theemma:node element MUST have an attributenode-number which indicates the number of the node.There MUST be at most oneemma:node specification fora given numbered node in the lattice. In our example, if there wasa cost of100 on the final state this could be representedas follows:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretationemma:medium="acoustic" emma:mode="voice">    <emma:lattice initial="1" final="8">      <emma:arc       from="1"       to="2"       emma:start="1087995961542"       emma:end="1087995962042"       emma:cost="30">         flights      </emma:arc>      <emma:arc       from="2"       to="3"       emma:start="1087995962042"       emma:end="1087995962542"       emma:cost="20">         to      </emma:arc>      <emma:arc       from="3"       to="4"       emma:start="1087995962542"       emma:end="1087995963042"       emma:cost="50">         boston      </emma:arc>      <emma:arc       from="3"       to="4"       emma:start="1087995963042"       emma:end="1087995963742"       emma:cost="60">         austin      </emma:arc>        ...      <emma:node node-number="8" emma:cost="100"/>    </emma:lattice>  </emma:interpretation></emma:emma>

3.4.3 Relative timestamps on lattices

The relative timestamp mechanism in EMMA is intended to providetemporal information about arcs in a lattice in relative termsusing offsets in milliseconds. In order to do this the absolutetime MAY be specified onemma:interpretation; bothemma:time-ref-uri andemma:time-ref-anchor-point apply toemma:lattice and MAY be used there to set the anchorpoint for offsets to the start of the absolute time specified onemma:interpretation. The offset in milliseconds to thebeginning of each arc MAY then be indicated on eachemma:arc in theemma:offset-to-startattribute.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation          emma:start="1087995961542" emma:end="1087995963042"emma:medium="acoustic" emma:mode="voice">    <emma:lattice emma:time-ref-uri="#interp1"        emma:time-ref-anchor-point="start"        initial="1" final="4">      <emma:arc       from="1"       to="2"       emma:offset-to-start="0">         flights      </emma:arc>      <emma:arc       from="2"       to="3"       emma:offset-to-start="500">         to      </emma:arc>      <emma:arc       from="3"       to="4"       emma:offset-to-start="1000">         boston      </emma:arc>    </emma:lattice>  </emma:interpretation></emma:emma>

Note that the offset for the firstemma:arc MUSTalways be zero since the EMMA attributeemma:offset-to-start indicates the number ofmilliseconds from the anchor point to thestart of the pieceof input associated with theemma:arc, in this casethe word "flights".

3.5 Literal semantics:emma:literalelement

Certain EMMA processing components produce semantic results inthe form of string literals without any surrounding applicationnamespace markup. These MUST be placed with the EMMA elementemma:literal withinemma:interpretation.For example, if a semantic interpreter simply returned "boston"this could be represented in EMMA as:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretationid="r1"
     emma:medium="acoustic" emma:mode="voice">    <emma:literal>boston</emma:literal>  </emma:interpretation></emma:emma>

Note that a raw recognition result of a sequence of words fromspeech recognition is also a kind of string literal and can becontained withinemma:literal. For example,recognition of the string "flights to san francisco" can berepresented in EMMA as follows:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretationid="r1"
     emma:medium="acoustic" emma:mode="voice">    <emma:literal>flights to san francisco</emma:literal>  </emma:interpretation></emma:emma>

4. EMMA annotations

This section defines annotations in the EMMA namespace includingboth attributes and elements. The values are specified in terms ofthe data types defined by XML Schema Part 2: DatatypesSecondEdition [XML SchemaDatatypes].

4.1 EMMA annotation elements

4.1.1 Data model:emma:modelelement

The data model that may be used to express constraints on thestructure and content of instance data is specified as one of theannotations of the instance. Specifying the data model is OPTIONAL,in which case the data model can be said to be implicit. Typicallythe data model is pre-established by the application.

The data model is specified with theemma:modelannotation defined as an element in the EMMA namespace. If the datamodel for the contents of aemma:interpretation,container elements, or application namespace element is to bespecified in EMMA, the attributeemma:model-ref MUSTbe specified on theemma:interpretation, containerelement, or application namespace element. Note that since multipleemma:model elements might be specified under theemma:emma it is possible to refer to multiple datamodels within a single EMMA document. For example, differentalternative interpretations under anemma:one-of mighthave different data models. In this case, anemma:model-ref attribute would appear on eachemma:interpretation element in the N-best list withits value being theid of theemma:modelelement for that particular interpretation.

The data model is closely related to the interpretation data,and is typically specified as the annotation related to theemma:interpretation oremma:one-ofelements.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:model ref="http://example.com/models/city.xml"/>  <emma:interpretation emma:model-ref="model1"emma:medium="acoustic" emma:mode="voice">    <city> London </city>    <country> UK </country>  </emma:interpretation></emma:emma>

Theemma:model annotation MAY reference any elementor attribute in the application instance data, as well as any EMMAcontainer element (emma:one-of,emma:group, oremma:sequence).

The data model annotation MAY be used to either reference anexternal data model with theref attribute or providea data model as in-line content. Either arefattribute or in-line data model (but not both) MUST bespecified.

4.1.2 Interpretation derivation:emma:derived-from element andemma:derivation element

Instances of interpretations are in general derived from otherinstances of interpretation in a process that goes from raw data toincreasingly refined representations of the input. The derivationannotation is used to link any two interpretations that are relatedby representing the source and the outcome of an interpretationprocess. For instance, a speech recognition process can return thefollowing result in the form of raw text:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation
emma:medium="acoustic" emma:mode="voice">    <answer>From Boston to Denver tomorrow</answer>  </emma:interpretation></emma:emma>

A first interpretation process will produce:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation
emma:medium="acoustic" emma:mode="voice">    <origin>Boston</origin>    <destination>Denver</destination>    <date>tomorrow</date>  </emma:interpretation></emma:emma>

A second interpretation process, aware of the current date, willbe able to produce a more refined instance, such as:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretationemma:medium="acoustic" emma:mode="voice">    <origin>Boston</origin>    <destination>Denver</destination>    <date>20030315</date>  </emma:interpretation></emma:emma>

The interaction manager might need to have access to the threelevels of interpretation. Theemma:derived-fromannotation element can be used to establish a chain of derivationrelationships as in the following example:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:derivation>    <emma:interpretation
      emma:medium="acoustic" emma:mode="voice">      <answer>From Boston to Denver tomorrow</answer>    </emma:interpretation>    <emma:interpretation>      <emma:derived-from resource="#raw" composite="false"/>      <origin>Boston</origin>      <destination>Denver</destination>      <date>tomorrow</date>    </emma:interpretation>  </emma:derivation>  <emma:interpretation>    <emma:derived-from resource="#better" composite="false"/>    <origin>Boston</origin>    <destination>Denver</destination>    <date>20030315</date>  </emma:interpretation></emma:emma>

Theemma:derivation element MAY be used as acontainer for representations of the earlier stages in theinterpretation of the input. The latest stage of processing MUST bea direct child ofemma:emma.

The resource attribute onemma:derived-from is aURI which can reference IDs in the current or other EMMAdocuments.

In addition to representing sequential derivations, the EMMAemma:derived-from element can also be used to capturecomposite derivations. Composite derivations involve combination ofinputs from different modes.

In order to indicate whether anemma:derived-fromelement describes a sequential derivation step or a compositederivation step, theemma:derived-from element has anattributecomposite which has a boolean value. Acompositeemma:derived-from MUST be marked ascomposite="true" while a sequentialemma:derived-from element is marked ascomposite="false". If this attribute is not specifiedthe value isfalse by default.

In the following composite derivation example the user said"destination" using the voice mode and circled Boston on a mapusing the ink mode:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:derivation>    <emma:interpretation        emma:start="1087995961500"        emma:end="1087995962542"        emma:process="http://example.com/myasr.xml"        emma:source="http://example.com/microphone/NC-61"        emma:signal="http://example.com/signals/sg23.wav"        emma:confidence="0.6"        emma:medium="acoustic"        emma:mode="voice"        emma:function="dialog"        emma:verbal="true"        emma:lang="en-US"        emma:tokens="destination">      <rawinput>destination</rawinput>    </emma:interpretation>    <emma:interpretation        emma:start="1087995961600"        emma:end="1087995964000"        emma:process="http://example.com/mygesturereco.xml"        emma:source="http://example.com/pen/wacom123"        emma:signal="http://example.com/signals/ink5.inkml"        emma:confidence="0.5"        emma:medium="tactile"        emma:mode="ink"        emma:function="dialog"        emma:verbal="false">      <rawinput>Boston</rawinput>    </emma:interpretation>  </emma:derivation>  <emma:interpretation                  emma:confidence="0.3"emma:start="1087995961500"emma:end="1087995964000"      emma:medium="acoustic tactile"      emma:mode="voice ink"      emma:function="dialog"      emma:verbal="true"      emma:lang="en-US"      emma:tokens="destination">    <emma:derived-from resource="#voice1" composite="true"    <emma:derived-from resource="#ink1" composite="true"    <destination>Boston</destination>  </emma:interpretation></emma:emma>

In this example, annotations on the multimodal interpretationindicate the process used for the integration and there are twoemma:derived-from elements, one pointing to the speechand one pointing to the pen gesture.

The only constraints the EMMA specification places on theannotations that appear on a composite input are that theemma:medium attribute MUST contain the union of theemma:medium attributes on the combining inputs,represented as a space delimited set ofnmtokens asdefined inSection 4.2.11, and that theemma:mode attribute MUST contain the union of theemma:mode attributes on the combining inputs,represented as a space delimited set ofnmtokensas defined inSection 4.2.11. In theexample above this meanings that theemma:medium valueis"acoustic tactile" and theemma:modeattribute is"voice ink". How all other annotationsare handled is author defined. In the following paragraph,informative examples on how specific annotations might be handledare given.

With reference to the illustrative example above, this paragraphprovides informative guidance regarding the determination ofannotations (beyondemma:medium andemma:mode on a composite multimodal interpretation).Generally the timestamp on a combined input should contain theintervals indicated by the combining inputs. For the absolutetimestampsemma:start andemma:end thiscan be achieved by taking the earlier of theemma:start values(emma:start="1087995961500" in our example) and thelater of theemma:end values(emma:end="1087995964000" in the example). Thedetermination of relative timestamps for composite is more complex,informative guidance is given inSection4.2.10.4. Generally speaking theemma:confidencevalue will be some numerical combination of the confidence scoresassigned to the combining inputs. In our example, it is the resultof multiplying the voice and ink confidence scores(0.3). In other cases there may not be a confidencescore for one of the combining inputs and the author may choose tocopy the confidence score from the input which does have one.Generally, foremma:verbal, if either of the inputshas the valuetrue then the multimodal interpretationwill also beemma:verbal="true" as in the example. Inother words the annotation for the composite input is the result ofan inclusive OR of the boolean values of the annotations on theinputs. If an annotation is only specified on one of the combininginputs then it may in some cases be assumed to apply to themultimodal interpretation of the composite input. In the example,emma:lang="en-US" is only specified for the speechinput, and this annotation appears on the composite result also.Similarly in our example, only the voice hasemma:tokens and the author has chosen to annotate thecombined input with the sameemma:tokens value. Inthis example, theemma:function is the same on bothcombining input and the author has chosen to use the sameannotation on the composite interpretation.

In annotating derivations of the processing of the input, EMMAprovides the flexibility of both course-grained or fine-grainedannotation of relations among interpretations. For example, whenrelating two N-best lists, withinemma:one-of elementseither there can be a singleemma:derived-from elementunderemma:one-of referring to the ID of theemma:one-of for the earlier processing stage:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:derivation>    <emma:one-ofemma:medium="acoustic" emma:mode="voice">      <emma:interpretation>       <res>from boston to denver on march eleven two thousand three</res>      </emma:interpretation>      <emma:interpretation>       <res>from austin to denver on march eleven two thousand three</res>      </emma:interpretation>  </emma:one-of></emma:derivation><emma:one-of>  <emma:derived-from resource="#nbest1" composite="false"/>  <emma:interpretation>    <origin>Boston</origin>    <destination>Denver</destination>    <date>03112003</date>  </emma:interpretation>  <emma:interpretation>    <origin>Austin</origin>    <destination>Denver</destination>    <date>03112003</date>  </emma:interpretation></emma:one-of>  </emma:emma>

Or there can be a separateemma:derived-fromelement on eachemma:interpretation element referringto the specificemma:interpretation element it wasderived from.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:one-of>    <emma:interpretation>     <emma:derived-from resource="#int1" composite="false"/>      <origin>Boston</origin>      <destination>Denver</destination>      <date>03112003</date>    </emma:interpretation>    <emma:interpretation>     <emma:derived-from resource="#int2" composite="false"/>      <origin>Austin</origin>      <destination>Denver</destination>      <date>03112003</date>    </emma:interpretation>  </emma:one-of>  <emma:derivation>    <emma:one-of
emma:medium="acoustic" emma:mode="voice">      <emma:interpretation>       <res>from boston to denver on march eleven two thousand three</res>      </emma:interpretation>      <emma:interpretation>       <res>from austin to denver on march eleven two thousand three</res>      </emma:interpretation>    </emma:one-of>  </emma:derivation></emma:emma>

Section 4.3 provides further examples of theuse ofemma:derived-from to represent sequentialderivations and addresses the issue of the scope of EMMAannotations across derivations of user input.

4.1.3 Reference to grammar used:emma:grammar element

The grammar that was used to derive the EMMA result MAY bespecified with theemma:grammar annotation defined asan element in the EMMA namespace.

Example:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:grammarref="someURI"/>  <emma:grammarref="anotherURI"/>  <emma:one-of
emma:medium="acoustic" emma:mode="voice">    <emma:interpretation emma:grammar-ref="gram1">      <origin>Boston</origin>    </emma:interpretation>    <emma:interpretation emma:grammar-ref="gram1">        <origin>Austin</origin>    </emma:interpretation>    <emma:interpretation emma:grammar-ref="gram2">        <command>help</command>    </emma:interpretation>  </emma:one-of></emma:emma>

Theemma:grammar annotation is a child ofemma:emma.

4.1.4 Extensibility to application/vendor specificannotations:emma:info element

InSection 4.2, a series of attributes aredefined for representation of metadata about user inputs in astandardized form. EMMA also provides an extensibility mechanismfor annotation of user inputs with vendor or application specificmetadata not covered by the standard set of EMMA annotations. Theelementemma:info MUST be used as a container forthese annotations, UNLESS they are explicitly covered byemma:endpoint-info. For example, if an input to adialog system needed to be annotated with the number that the calloriginated from, their state, some indication of the type ofcustomer, and the name of the service, these pieces of informationcould be represented withinemma:info as in thefollowing example:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:info>    <caller_id>      <phone_number>2121234567</phone_number>      <state>NY</state>    </caller_id>    <customer_type>residential</customer_type>    <service_name>acme_travel_service</service_name>  </emma:info>  <emma:one-of emma:start="1087995961542"      emma:end="1087995963542"emma:medium="acoustic" emma:mode="voice">    <emma:interpretation emma:confidence="0.75">      <origin>Boston</origin>      <destination>Denver</destination>      <date>03112003</date>    </emma:interpretation>    <emma:interpretation emma:confidence="0.68">      <origin>Austin</origin>      <destination>Denver</destination>      <date>03112003</date>    </emma:interpretation>  </emma:one-of></emma:emma>

It is important to have an EMMA container element forapplication/vendor specific annotations since EMMA elements providea structure for representation of multiple possible interpretationsof the input. As a result it is cumbersome to stateapplication/vendor specific metadata as part of the applicationdata within eachemma:interpretation. An element isused rather than an attribute so that internal structure can begiven to the annotations withinemma:info.

In addition toemma:emma,emma:infoMAY also appear as a child of other structural elements such asemma:interpretation,emma:info and so on.Whenemma:info appears as a child of one of theseelements the application/vendor specific annotations containedwithinemma:info are assumed to apply to all of theemma:interpretation elements within the containingelement. The semantics of conflicting annotations inemma:info, for example when different values are foundwithinemma:emma andemma:interpretation,are left to the developer of the vendor/application specificannotations.

4.1.5 Endpoint reference:emma:endpoint-info element andemma:endpoint element

In order to conduct multimodal interaction, there is a need inEMMA to specify the properties of the endpoint that receives theinput which leads to the EMMA annotation. This allows subsequentcomponents to utilize the endpoint properties as well as theannotated inputs to conduct meaningful multimodal interaction. EMMAelementemma:endpoint can be used for this purpose. Itcan specify the endpoint properties based on a set of commonendpoint property attributes in EMMA, such asemma:endpoint-address,emma:port-num,emma:port-type, etc. (Section4.2.14). Moreover, it provides an extensible annotationstructure that allows the inclusion of application and vendorspecific endpoint properties.

Note that the usage of the term "endpoint" in this context isdifferent from the way that the term is used in speech processing,where it refers to the end of a speech input. As used here,"endpoint" refers to a network location which is the source orrecipient of an EMMA document.

In multimodal interaction, multiple devices can be used and eachdevice can open multiple communication endpoints at the same time.These endpoints are used to transmit and receive data, such as rawinput, EMMA documents, etc. The EMMA elementemma:endpoint provides a generic representation ofendpoint information which is relevant to multimodal interaction.It allows the annotation to be interoperable, and it eliminates theneed for EMMA processors to create their own specializedannotations for existing protocols, potential protocols or yetundefined private protocols that they may use.

Moreover,emma:endpoint-info provides a containerto hold all annotations regarding the endpoint information,includingemma:endpoint and other application andvendor specific annotations that are related to the communication,allowing the same communication environment to be referenced andused in multiple interpretations.

Note that EMMA provides two locations (i.e.emma:info andemma:endpoint-info) forspecifying vendor/application specific annotations. If theannotation is specifically related to the description of theendpoint, then the vendor/application specific annotation SHOULD beplaced withinemma:endpoint-info, otherwise it SHOULDbe placed withinemma:info.

The following example illustrates the annotation of endpointreference properties in EMMA.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example"    xmlns:ex="http://www.example.com/emma/port">  <emma:endpoint-info>    <emma:endpoint        emma:endpoint-role="sink"        emma:endpoint-address="135.61.71.103"        emma:port-num="50204"        emma:port-type="rtp"        emma:endpoint-pair-ref="endpoint2"        emma:media-type="audio/dsr-202212; rate:8000; maxptime:40"        emma:service-name="travel"        emma:mode="voice">      <ex:app-protocol>SIP</ex:app-protocol>    </emma:endpoint>    <emma:endpoint        emma:endpoint-role="source"        emma:endpoint-address="136.62.72.104"        emma:port-num="50204"        emma:port-type="rtp"        emma:endpoint-pair-ref="endpoint1"        emma:media-type="audio/dsr-202212; rate:8000; maxptime:40"        emma:service-name="travel"        emma:mode="voice">      <ex:app-protocol>SIP</ex:app-protocol>    </emma:endpoint>  </emma:endpoint-info>  <emma:interpretation      emma:start="1087995961542" emma:end="1087995963542"      emma:endpoint-info-ref="audio-channel-1"
emma:medium="acoustic" emma:mode="voice">    <destination>Chicago</destination>  </emma:interpretation></emma:emma>

Theex:app-protocol is provided by the applicationor the vendor specification. It specifies that the applicationlayer protocol used to establish the speech transmission from the"source" port to the "sink" port is Session Initiation Protocol(SIP). This is specific to SIP based VoIP communication, in whichthe actual media transmission and the call signaling that controlsthe communication sessions, are separated and typically based ondifferent protocols. In the above example, the Real-timeTransmission Protocol (RTP) is used in the media transmissionbetween the source port and the sink port.

4.2 EMMA annotation attributes

4.2.1 Tokens of input:emma:tokensattribute

Theemma:tokens annotation holds a list of inputtokens. In the following description, the termtokens isused in the computational and syntactic sense ofunits ofinput, and not in the sense ofXML tokens. The valueheld inemma:tokens is the list of the tokens of inputas produced by the processor which generated the EMMA document;there is no language associated with this value.

In the case where a grammar is used to constrain input, thevalue will correspond to tokens as defined by the grammar. So foran EMMA document produced by input to a SRGS grammar [SRGS], the value ofemma:tokens will bethe list of words and/or phrases that are defined as tokens in SRGS(see Section 2.1of [SRGS]). Items in theemma:tokenslist are delimited by white space and/or quotation marks forphrases containing white space. For example:

emma:tokens="arriving at 'Liverpool Street'"

where the three tokens of input arearriving,atandLiverpool Street.

Theemma:tokens annotation MAY be applied not justto the lexical words and phrases of language but to any level ofinput processing. Other examples of tokenization include phonemes,ink strokes, gestures and any other discrete units of input at anylevel.

Examples:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation      emma:tokens="From Cambridge to London tomorrow"
emma:medium="acoustic" emma:mode="voice">    <origin emma:tokens="From Cambridge">Cambridge</origin>    <destination emma:tokens="to London">London</destination>    <date emma:tokens="tomorrow">20030315</date>  </emma:interpretation></emma:emma>

4.2.2 Reference to processing:emma:process attribute

A reference to the information concerning the processing thatwas used for generating an interpretation MAY be made using theemma:process attribute. For example:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:derivation>    <emma:interpretation
emma:medium="acoustic" emma:mode="voice">      <answer>From Boston to Denver tomorrow</answer>    </emma:interpretation>    <emma:interpretation        emma:process="http://example.com/mysemproc1.xml">      <origin>Boston</origin>      <destination>Denver</destination>      <date>tomorrow</date>      <emma:derived-from resource="#raw"/>    </emma:interpretation>  </emma:derivation>  <emma:interpretation      emma:process="http://example.com/mysemproc2.xml">    <origin>Boston</origin>    <destination>Denver</destination>    <date>03152003</date>    <emma:derived-from resource="#better"/>  </emma:interpretation></emma:emma>

The process description document, referenced by theemma:process annotation MAY include information on theprocess itself, such as grammar, type of parser, etc. EMMA is notnormative about the format of the process description document.

4.2.3 Lack of input:emma:no-inputattribute

The case of lack of input MUST be annotated as follows:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation emma:no-input="true"
emma:medium="acoustic" emma:mode="voice"/></emma:emma>

If theemma:interpretation is annotated withemma:no-input="true" then theemma:interpretation MUST be empty.

4.2.4 Uninterpreted input:emma:uninterpreted attribute

Anemma:interpretation element representing inputfor which no interpretation was produced MUST beannotated withemma:uninterpreted="true". Forexample:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma    http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation emma:uninterpreted="true"
emma:medium="acoustic" emma:mode="voice"/></emma:emma>

The notation for uninterpreted input MAY refer to any possiblestage of interpretation processing, including raw transcriptions.For instance, no interpretation would be produced for stagesperforming pure signal capture such as audio recordings. Likewise,if a spoken input was recognized but cannot be parsed by a languageunderstanding component, it can be tagged asemma:uninterpreted as in the following example:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation      emma:process="http://example.com/mynlu.xml"      emma:uninterpreted="true"      emma:tokens="From Cambridge to London tomorrow"
emma:medium="acoustic" emma:mode="voice"/></emma:emma>

Theemma:interpretation MUST be emptyif theemma:interpretation element isannotated withemma:uninterpreted="true".

4.2.5 Human language of input:emma:lang attribute

Theemma:lang annotation is used to indicate thehuman language for the input that it annotates. The values of theemma:lang attribute are language identifiers asdefined byIETF Best Current Practice 47 [BCP47]. For example,emma:lang="fr" denotes French, andemma:lang="en-US" denotes US English.emma:lang MAY be applied to anyemma:interpretation element. Its annotative scopefollows the annotative scope of these elements. Unlike thexml:lang attribute in XML,emma:lang doesnot specify the language used by element contents or attributevalues.

The following example shows the use ofemma:langfor annotating an input interpretation.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation emma:lang="fr"
emma:medium="acoustic" emma:mode="voice">    <answer>arretez</answer>  </emma:interpretation></emma:emma>

Many kinds of input including some inputs made through pen,computer vision, and other kinds of sensors are inherentlynon-linguistic. Examples include drawing areas, arrows etc. using apen and music input for tune recognition. If these non-linguisticinputs are annotated withemma:lang then they MUST beannotated asemma:lang="zxx". For example, pen inputwhere a user circles an area on map display could be represented asfollows whereemma:lang="zxx" indicates that the inkinput is not in any human language.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation      emma:medium="tactile"      emma:mode="ink"      emma:lang="zxx">    <location>      <type>area</type>      <points>42.1345 -37.128 42.1346 -37.120 ... </points>    </location>  </emma:interpretation></emma:emma>

If inputs for which there is no information about whether thesource input is in a particular human language, and if so whichlanguage, are annotated withemma:lang, then they MUSTbe annotated asemma:lang="". Furthermore, in caseswhere there is not explicitemma:lang annotation, andnone is inherited from a higher element in the document, thedefault value foremma:lang is"" meaningthat there is no information about whether the source input is in alanguage and if so which language.

Thexml:lang andemma:lang attributesserve uniquely different and equally important purposes. The roleof thexml:lang attribute in XML 1.0 is to indicatethe language used for character data content in an XML element ordocument. In contrast, theemma:lang attribute is usedto indicate the language employed by a user when entering an input.Critically,emma:lang annotates the language of thesignal originating from the user rather than the specific tokensused at a particular stage of processing. This is most clearlyillustrated through consideration of an example involving multiplestages of processing of a user input. Consider the followingscenario: EMMA is being used to represent three stages in theprocessing of a spoken input to an system for ordering products.The user input is in Italian, after speech recognition, the userinput is first translated into English, then a natural languageunderstanding system converts the English translation into aproduct ID (which is not in any particular language). Since theinput signal is a user speaking Italian, theemma:langwill beemma:lang="it" on all of these three stages ofprocessing. Thexml:lang attribute, in contrast, willinitially be"it", after translation thexml:lang will be"en-US", and afterlanguage understanding it will be"zxx" since theproduct ID is non-linguistic content. The following are examples ofEMMA documents corresponding to these three processing stages,abbreviated to show the critical attributes for discussion here.Note that<transcription>,<translation>, and<understanding> are application namespaceattributes, not part of the EMMA markup.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">   <emma:interpretation emma:lang="it" emma:mode="voice" emma:medium="acoustic">
     <transcription xml:lang="it">condizionatore</transcription>
   </emma:interpretation></emma:emma>

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma    http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">    <emma:interpretation emma:lang="it" emma:mode="voice" emma:medium="acoustic">        <translation xml:lang="en-US">air conditioner</translation>
    </emma:interpretation></emma:emma>

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma    http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">    <emma:interpretation emma:lang="it" emma:mode="voice" emma:medium="acoustic">
       <understanding xml:lang="zxx">id1456</understanding>
    </emma:interpretation></emma:emma>

In orderto handle inputs involving multiplelanguages, such as through code switching, theemma:lang tag MAY contain several language identifiersseparated by spaces.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation      emma:tokens="please stop arretez s'il vous plait"      emma:lang="en fr"emma:medium="acoustic" emma:mode="voice">    <command> CANCEL </command>  </emma:interpretation></emma:emma>

4.2.6 Reference to signal:emma:signalandemma:signal-size attributes

A URI reference to the signal that originated the inputrecognition process MAY be represented in EMMA using theemma:signal annotation.

Here is an example where the reference to a speech signal isrepresented using theemma:signal annotation on theemma:interpretation element:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation      emma:signal="http://example.com/signals/sg23.bin"
emma:medium="acoustic" emma:mode="voice">    <origin>Boston</origin>    <destination>Denver</destination>    <date>03152003</date>  </emma:interpretation></emma:emma>

Theemma:signal-size annotation can be used todeclare the exact size of the associated signal in 8-bit octets. Anexample of the use of an EMMA document to represent a recording,withemma:signal-size indicating the size is asfollows:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation      emma:medium="acoustic"      emma:mode="voice"      emma:function="recording"      emma:uninterpreted="true"      emma:signal="http://example.com/signals/recording.mpg"      emma:signal-size="82102"       emma:duration="10000">  </emma:interpretation></emma:emma>

4.2.7 Media type:emma:media-typeattribute

The data format of the signal that originated the input MAY berepresented in EMMA using theemma:media-typeannotation. An initial set of MIME media types is defined by[RFC2046].

Here is an example where the media type for the ETSI ES 202 212audio codec for Distributed Speech Recognition (DSR) is applied totheemma:interpretation element. The example alsospecifies an optional sampling rate of 8 kHz and maxptime of 40milliseconds.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation        emma:signal="http://example.com/signals/signal.dsr"        emma:media-type="audio/dsr-es202212; rate:8000; maxptime:40"
emma:medium="acoustic" emma:mode="voice">    <origin>Boston</origin>    <destination>Denver</destination>    <date>03152003</date>  </emma:interpretation></emma:emma>

4.2.8 Confidence scores:emma:confidence attribute

The confidence score in EMMA is used to indicate the quality ofthe input, and if confidence is annotated on an input it MUST begiven as the value ofemma:confidence. The confidencescore MUST be a number in the range from 0.0 to 1.0 inclusive. Avalue of 0.0 indicates minimum confidence, and a value of 1.0indicates maximum confidence. Note thatemma:confidence represents not only the confidence ofthe speech recognizer, but rather the confidence of the whateverprocessor was responsible for creating the EMMA result, based onwhatever evidence it has. For a natural language interpretation,for example, this might include semantic heuristics in addition tospeech recognition scores. Moreover, the confidence score values donot have to be interpreted as probabilities. In fact confidencescore values are platform-dependent, since their computation islikely to differ between platforms and different EMMA processors.Confidence scores are annotated explicitly in EMMA in order toprovide this information to the subsequent processes for multimodalinteraction. The example below illustrates how confidence scoresare annotated in EMMA.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:one-of
emma:medium="acoustic" emma:mode="voice">    <emma:interpretation emma:confidence="0.6">      <location>Boston</location>    </emma:interpretation>    <emma:interpretation emma:confidence="0.4">      <location> Austin </location>    </emma:interpretation>  </emma:one-of></emma:emma>

In addition to its use as an attribute on the EMMAinterpretation and container elements, theemma:confidence attribute MAY also be used to assignconfidences to elements in instance data in the applicationnamespace. This can be seen in the following example, where the<destination> and<origin>elements have confidences.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation emma:confidence="0.6"emma:medium="acoustic" emma:mode="voice">     <destination emma:confidence="0.8"> Boston</destination>     <origin emma:confidence="0.6"> Austin </origin>  </emma:interpretation></emma:emma>

Although in general instance data can be represented in XMLusing a combination of elements and attributes in the applicationnamespace, EMMA does not provide a standard way to annotateprocessors' confidences in attributes. Consequently, instance datathat is expected to be assigned confidences SHOULD be representedusing elements, as in the above example.

4.2.9 Input source:emma:sourceattribute

The source of an interpreted input MAY be represented in EMMA asa URI resource using theemma:source annotation.

Here is an example that shows different input sources fordifferent input interpretations.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example"    xmlns:myapp="http://www.example.com/myapp">  <emma:one-of
emma:medium="acoustic" emma:mode="voice">    <emma:interpretation        emma:source="http://example.com/microphone/NC-61">      <myapp:destination>Boston</myapp:destination>    </emma:interpretation>    <emma:interpretation        emma:source="http://example.com/microphone/NC-4024">      <myapp:destination>Austin</myapp:destination>    </emma:interpretation>  </emma:one-of></emma:emma>

4.2.10 Timestamps

The start and end times for input MAY be indicated using eitherabsolute timestamps or relative timestamps. Both are inmilliseconds for ease in processing timestamps. Note that theECMAScript Date object'sgetTime() function is aconvenient way to determine the absolute time.

4.2.10.1 Absolute timestamps:emma:start,emma:end attributes

Here is an example of a timestamp for an absolute time.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation       emma:start="1087995961542"       emma:end="1087995963542"
emma:medium="acoustic" emma:mode="voice">    <destination>Chicago</destination>  </emma:interpretation></emma:emma>

Theemma:start andemma:endannotations on an input MAY be identical, however theemma:end value MUST NOT be less than theemma:start value.

4.2.10.2 Relative timestamps:emma:time-ref-uri,emma:time-ref-anchor-point,emma:offset-to-start attributes

Relative timestamps define the start of an input relative to thestart or end of a reference interval such as another input.

The reference interval is designated withemma:time-ref-uri attribute. This MAY be combined withemma:time-ref-anchor-point attribute to specifywhether the anchor point is the start or end of this interval. Thestart of an input relative to this anchor point is then specifiedwithemma:offset-to-start attribute.

Here is an example where the referenced input is in the samedocument:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:sequence>    <emma:interpretation
emma:medium="acoustic" emma:mode="voice">    <origin>Denver</origin>    </emma:interpretation>    <emma:interpretation
emma:medium="acoustic" emma:mode="voice"        emma:time-ref-uri="#int1"        emma:time-ref-anchor-point="start"        emma:offset-to-start="5000">    <destination>Chicago</destination>    </emma:interpretation>  </emma:sequence></emma:emma>

Note that the reference point refers to an input, but notnecessarily to a complete input. For example, if a speechrecognizer timestamps each word in an utterance, the anchor pointmight refer to the timestamp for just one word.

The absolute and relative timestamps are not mutually exclusive;that is, it is possible to have both relative and absolutetimestamp attributes on the same EMMA container element.

Timestamps of inputs collected by different devices will besubject to variation if the times maintained by the devices are notsynchronized. This concern is outside of the scope of the EMMAspecification.

4.2.10.3 Duration of input:emma:duration attribute

The duration of an input in milliseconds MAY be specified withtheemma:duration attribute. Theemma:duration attribute MAY be used either incombination with timestamps or independently, for example in theannotation of speech corpora.

In the following example, the duration of the signal that gaverise to the interpretation is indicated usingemma:duration.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">    <emma:interpretation emma:duration="2300"
emma:medium="acoustic" emma:mode="voice">    <origin>Denver</origin>    </emma:interpretation></emma:emma>

4.2.10.4 Composite Input and RelativeTimestamps

This section is informative.

The following table provides guidance on how to determine thevalues of relative timestamps on a composite input.

4.2.11 Medium, mode, and function of user inputs:emma:medium,emma:mode,emma:function,emma:verbalattributes

EMMA provides two properties for the annotation of inputmodality. One indicating the broader medium or channel(emma:medium) and another indicating the specific modeof communication used on that channel (emma:mode). Theinput medium is defined from the users perspective and indicateswhether they use their voice (acoustic), touch(tactile), or visual appearance/motion(visual) as input. Tactile includes mosthand-on input device types such as pen, mouse, keyboard, andtouch screen. Visual is used for camera input.

emma:medium =space delimited sequence of values from the set:            [acoustic|tactile|visual]

The mode property provides the ability to distinguish betweendifferent modes of communication that may be within a particularmedium. For example, in the tactile medium, modes includeelectronic ink (ink), and pointing and clicking on agraphical user interface (gui).

emma:mode =space delimited sequence of values from the set:             [voice|dtmf|ink|gui|keys|video|photograph| ... ]

Theemma:medium classification is based on theboundary between the user and the device that they use. Foremma:medium="tactile" the user physically touches thedevice in order to provide input. Foremma:medium="visual" the user's movement is capturedby sensors (cameras, infrared) resulting in an input to the system.In the case whereemma:medium="acoustic" the userprovides input to the system by producing an acoustic signal. Notethen that DTMF input will be classified asemma:medium="tactile" since in order to provide DTMFinput the user physically presses keys on a keypad.

Whileemma:medium andemma:mode areoptional on specific elements such asemma:interpretation andemma:one-of, notethat all EMMA interpretations must be annotated foremma:medium andemma:mode, so eitherthese attributes must appear directly onemma:interpretation or they must appear on an ancestoremma:one-of node or they must appear on an earlierstage of the derivation listed inemma:derivation.

Orthogonal to the mode, user inputs can also be classified withrespect to their communicative function. This enables a simplermode classification.

emma:function = [recording|transcription|dialog|verification| ... ]

For example, speech can be used for recording (e.g. voicemail),transcription (e.g. dictation), dialog (e.g. interactive spokendialog systems), and verification (e.g. identifying users throughtheir voiceprints).

EMMA also supports an additional propertyemma:verbal which distinguishes verbal use of an inputmode from non-verbal. This MAY be used to distinguish the use ofelectronic ink to convey handwritten commands from the user ofelectronic ink for symbolic gestures such as circles and arrows.Handwritten commands, such as writingdowntown in order tochange a map display to show the downtown are classified as verbal(emma:function="dialog" emma:verbal="true"). Pengestures (arrows, lines, circles, etc), such as circling abuilding, are classified as non-verbal dialog(emma:function="dialog" emma:verbal="false"). The useof handwritten words to transcribe an email message is classifiedas transcription (emma:function="transcription"emma:verbal="true").

emma:verbal = [true|false]

Handwritten words and ink gestures are typically recognizedusing different kinds of recognition components (handwritingrecognizer vs. gesture recognizer) and the verbal annotation willbe added by the recognition component which classifies the input.The original input source, a pen in this case, will not be aware ofthis difference. The input source identifier will tell you that theinput was from a pen of some kind but will not tell you if the modeof input was handwriting (show downtown) or gesture (e.g.circling an object or area).

Here is an example of the EMMA annotation for a pen input wherethe user's ink is recognized as either a word ("Boston") or as anarrow:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:one-of>    <emma:interpretation     emma:confidence="0.6"     emma:medium="tactile"     emma:mode="ink"     emma:function="dialog"     emma:verbal="true">       <location>Boston</location>    </emma:interpretation>    <emma:interpretation     emma:confidence="0.4"     emma:medium="tactile"     emma:mode="ink"     emma:function="dialog"     emma:verbal="false">       <direction>45</direction>    </emma:interpretation>  </emma:one-of></emma:emma>

Here is an example of the EMMA annotation for a spoken commandwhich is recognized as either "Boston" or "Austin":

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:one-of>    <emma:interpretation     emma:confidence="0.6"     emma:medium="acoustic"     emma:mode="voice"     emma:function="dialog"     emma:verbal="true">       <location>Boston</location>    </emma:interpretation>    <emma:interpretation     emma:confidence="0.4"     emma:medium="acoustic"     emma:mode="voice"     emma:function="dialog"     emma:verbal="true">       <location>Austin</location>    </emma:interpretation>  </emma:one-of></emma:emma>

The following table shows the relationship between the medium,mode, and function properties and serves as an aid for classifyinginputs. For the dialog function it also shows some examples of theclassification of inputs as verbal vs. non-verbal.

4.2.12 Composite multimodality:emma:hook attribute

The attributeemma:hook MAY be used to mark theelements in the application semantics within anemma:interpretation which are expected to beintegrated with content from input in another mode to yield acomplete interpretation. Theemma:mode to beintegrated at that point in the application semantics is indicatedas the value of theemma:hook attribute. The possiblevalues ofemma:hook are the list of input modes thatcan be values ofemma:mode(seeSection 4.2.11). In addition to these, thevalue ofemma:hook can also be the wildcardany indicating that the other content can come fromany source. The annotationemma:hook differs insemantics fromemma:mode as follows. Annotating anelement in the application semantics withemma:mode="ink" indicates that that part of thesemantics came from theink mode. Annotating anelement in the application semantics withemma:hook="ink" indicates that part of the semanticsneeds to be integrated with content from theinkmode.

To illustrate the use ofemma:hook consider anexample composite input in which the user says "zoom in here" inthe speech input mode while drawing an area on a graphical displayin the ink input mode.The fact that thelocation element needs to come from theink mode is indicated by annotating this applicationnamespace element usingemma:hook

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretationemma:medium="acoustic" emma:mode="voice">    <command>      <action>zoom</action>      <location emma:hook="ink">        <type>area</type>      </location>    </command>  </emma:interpretation></emma:emma>

For more detailed explanation of this example seeAppendix C.

4.2.13 Cost:emma:cost attribute

The cost annotation in EMMA indicates the weight or costassociated with an user's input or part of their input. The mostcommon use ofemma:cost is for representing the costsencoded on a lattice output from speech recognition or otherrecognition or understanding processes.emma:cost MAYalso be used to indicate the total cost associated with particularrecognition results or semantic interpretations.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:one-ofemma:medium="acoustic" emma:mode="voice">    <emma:interpretation emma:cost="1600">      <location>Boston</location>    </emma:interpretation>    <emma:interpretation emma:cost="400">      <location> Austin </location>    </emma:interpretation>  </emma:one-of></emma:emma>

4.2.14 Endpoint properties:emma:endpoint-role,emma:endpoint-address,emma:port-type,emma:port-num,emma:message-id,emma:service-name,emma:endpoint-pair-ref,emma:endpoint-info-refattributes

Theemma:endpoint-role attribute specifies the rolethat the particularemma:endpoint performs inmultimodal interaction. The role valuesink indicatesthat the particular endpoint is the receiver of the input data. Therole valuesource indicates that the particularendpoint is the sender of the input data. The role valuereply-to indicates that the particularemma:endpoint is the intended endpoint for the reply.The sameemma:endpoint-address MAY appear in multipleemma:endpoint elements, provided that the sameendpoint address is used to serve multiple roles, e.g. sink,source, reply-to, router, etc., or associated with multipleinterpretations.

Theemma:endpoint-address specifies the networkaddress of theemma:endpoint, andemma:port-type specifies the port type of theemma:endpoint. Theemma:port-numannotates the port number of the endpoint (e.g. the typical portnumber for an http endpoint is 80). Theemma:message-id annotates the message ID informationassociated with the annotated input. This meta information is usedto establish and maintain the communication context for bothinbound processing and outbound operation. The servicespecification of theemma:endpoint is annotated byemma:service-name which contains the definition of theservice that theemma:endpoint performs. The matchingof thesink endpoint and its pairingsource endpoint is annotated by theemma:endpoint-pair-ref attribute. One sink endpointMAY link to multiple source endpoints throughemma:endpoint-pair-ref. Further bounding of theemma:endpoint is possible by using the annotation ofemma:group (seeSection3.3.2).

Theemma:endpoint-info-ref attribute associates theEMMA result in the container element with anemma:endpoint-info element.

The following example illustrates the use of these attributes inmultimodal interactions where multiple modalities are used.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example"    xmlns:ex="http://www.example.com/emma/port">  <emma:endpoint-info >    <emma:endpoint        emma:endpoint-role="sink"        emma:endpoint-address="135.61.71.103"        emma:port-num="50204"        emma:port-type="rtp"        emma:endpoint-pair-ref="endpoint2"        emma:media-type="audio/dsr-202212; rate:8000; maxptime:40"        emma:service-name="travel"        emma:mode="voice">      <ex:app-protocol>SIP</ex:app-protocol>    </emma:endpoint>    <emma:endpoint emma:endpoint-role="source"        emma:endpoint-address="136.62.72.104"        emma:port-num="50204"        emma:port-type="rtp"        emma:endpoint-pair-ref="endpoint1"        emma:media-type="audio/dsr-202212; rate:8000; maxptime:40"        emma:service-name="travel"        emma:mode="voice">      <ex:app-protocol>SIP</ex:app-protocol>    </emma:endpoint>  </emma:endpoint-info>  <emma:endpoint-info>     <emma:endpoint emma:endpoint-role="sink"         emma:endpoint-address="http://emma.example/sink"         emma:endpoint-pair-ref="endpoint4"         emma:port-num="80" emma:port-type="http"         emma:message-id="uuid:2e5678"         emma:service-name="travel"         emma:mode="ink"/>     <emma:endpoint         emma:endpoint-role="source"         emma:port-address="http://emma.example/source"         emma:endpoint-pair-ref="endpoint3"         emma:port-num="80"         emma:port-type="http"         emma:message-id="uuid:2e5678"         emma:service-name="travel"         emma:mode="ink"/>  </emma:endpoint-info>  <emma:group>    <emma:interpretation emma:start="1087995961542"        emma:end="1087995963542"        emma:endpoint-info-ref="audio-channel-1"
        emma:medium="acoustic" emma:mode="voice">      <destination>Chicago</destination>    </emma:interpretation>    <emma:interpretation emma:start="1087995961542"        emma:end="1087995963542"        emma:endpoint-info-ref="ink-channel-1"
        emma:medium="acoustic" emma:mode="voice">      <location>         <type>area</type>         <points>34.13 -37.12 42.13 -37.12 ... </points>      </location>    </emma:interpretation>  </emma:group></emma:emma>

4.2.15 Reference toemma:grammarelement:emma:grammar-ref attribute

Theemma:grammar-ref annotation associates the EMMAresult in the container element with anemma:grammarelement.

Example:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:grammarref="someURI"/>  <emma:grammarref="anotherURI"/>  <emma:one-of
emma:medium="acoustic" emma:mode="voice">    <emma:interpretation emma:grammar-ref="gram1">      <origin>Boston</origin>    </emma:interpretation>    <emma:interpretation emma:grammar-ref="gram1">      <origin>Austin</origin>    </emma:interpretation>    <emma:interpretation emma:grammar-ref="gram2">      <command>help</command>    </emma:interpretation>  </emma:one-of></emma:emma>

4.2.16 Reference toemma:modelelement:emma:model-ref attribute

Theemma:model-ref annotation associates the EMMAresult in the container element with anemma:modelelement.

Example:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:model ref="someURI"/>  <emma:model ref="anotherURI"/>  <emma:one-of
emma:medium="acoustic" emma:mode="voice">    <emma:interpretation emma:model-ref="model1">      <origin>Boston</origin>    </emma:interpretation>    <emma:interpretation emma:model-ref="model1">      <origin>Austin</origin>    </emma:interpretation>    <emma:interpretation emma:model-ref="model2">      <command>help</command>    </emma:interpretation>  </emma:one-of></emma:emma>

4.2.17 Dialog turns:emma:dialog-turnattribute

Theemma:dialog-turn annotation associates the EMMAresult in the container element with a dialog turn. The syntax andsemantics of dialog turns is left open to suit the needs ofindividual applications. For example, some applications might usean integer value, where successive turns are represented bysuccessive integers. Other applications might combine a name of adialog participant with an integer value representing the turnnumber for that participant. Ordering semantics for comparison ofemma:dialog-turn is deliberately unspecified and leftfor applications to define.

Example:

<emma:emma version="1.0"    emma="http://www.w3.org/2003/04/emma"    xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation emma:dialog-turn="u8"
emma:medium="acoustic" emma:mode="voice">    <quantity>3</quantity>  </emma:interpretation></emma:emma>

4.3 Scope of EMMA annotations

Theemma:derived-from element (Section 4.1.2) can be used to capture both sequentialand composite derivations. This section concerns the scope of EMMAannotations acrosssequential derivations of userinput connected using theemma:derived-from element(Section 4.1.2). Sequential derivationsinvolve processing steps that do not involve multimodalintegration, such as applying natural language understanding andthen reference resolution to a speech transcription. EMMAderivations describe only single turns of user input and are notintended to describe a sequence of dialog turns.

For example, an EMMA document could containemma:interpretation elements for the transcription,interpretation, and reference resolution of a speech input,utilizing theid values:raw,better, andbest respectively:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example"> <emma:derivation>  <emma:interpretation      emma:process="http://example.com/myasr1.xml"emma:medium="acoustic" emma:mode="voice">    <answer>From Boston to Denver tomorrow</answer>  </emma:interpretation>  <emma:interpretation      emma:process="http://example.com/mynlu1.xml">    <emma:derived-from resource="#raw" composite="false"/>    <origin>Boston</origin>    <destination>Denver</destination>    <date>tomorrow</date>  </emma:interpretation> </emma:derivation>  <emma:interpretation      emma:process="http://example.com/myrefresolution1.xml">    <emma:derived-from resource="#better" composite="false"/>    <origin>Boston</origin>    <destination>Denver</destination>    <date>03152003</date>  </emma:interpretation></emma:emma>

Each member of the derivation chain is linked to the previousone by aderived-from element (Section 4.1.2), which has an attributeresource that provides a pointer to theemma:interpretation from which it is derived. Theemma:process annotation (Section4.2.2) provides a pointer to the process used for each stage ofthe derivation.

The following EMMA example represents the same derivation asabove but with a more fully specified set of annotations:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:derivation>    <emma:interpretation        emma:process="http://example.com/myasr1.xml"        emma:source="http://example.com/microphone/NC-61"        emma:signal="http://example.com/signals/sg23.wav"        emma:confidence="0.6"        emma:medium="acoustic"        emma:mode="voice"        emma:function="dialog"        emma:verbal="true"        emma:tokens="from boston to denver tomorrow"        emma:lang="en-US">      <answer>From Boston to Denver tomorrow</answer>    </emma:interpretation>    <emma:interpretation        emma:process="http://example.com/mynlu1.xml"        emma:source="http://example.com/microphone/NC-61"        emma:signal="http://example.com/signals/sg23.wav"        emma:confidence="0.8"        emma:medium="acoustic"        emma:mode="voice"        emma:function="dialog"        emma:verbal="true"        emma:tokens="from boston to denver tomorrow"        emma:lang="en-US">      <emma:derived-from resource="#raw" composite="false"/>      <origin>Boston</origin>      <destination>Denver</destination>      <date>tomorrow</date>    </emma:interpretation>  </emma:derivation>  <emma:interpretation      emma:process="http://example.com/myrefresolution1.xml"      emma:source="http://example.com/microphone/NC-61"      emma:signal="http://example.com/signals/sg23.wav"      emma:confidence="0.8"      emma:medium="acoustic"      emma:mode="voice"      emma:function="dialog"      emma:verbal="true"      emma:tokens="from boston to denver tomorrow"      emma:lang="en-US">    <emma:derived-from resource="#better" composite="false"/>    <origin>Boston</origin>    <destination>Denver</destination>    <date>03152003</date>  </emma:interpretation></emma:emma>

EMMA annotations on earlier stages of the derivation oftenremain accurate at later stages of the derivation. Although thiscan be captured in EMMA by repeating the annotations on eachemma:interpretation within the derivation, as in theexample above, there are two disadvantages of this approach toannotation. First, the repetition of annotations makes theresulting EMMA documents significantly more verbose. Second, EMMAprocessors used for intermediate tasks such as natural languageunderstanding and reference resolution will need to read in all ofthe annotations and write them all out again.

EMMA overcomes these problems by assuming that annotations onearlier stages of a derivation automatically apply to later stagesof the derivation unless a new value is specified. Later stages ofthe derivation essentially inherit annotations from earlier stagesin the derivation. For example, if there was anemma:source annotation on the transcription(raw) it would also apply to the later stages of thederivation such as the result of natural language understanding(better) or reference resolution(best).

Because of the assumption in EMMA that annotations have scopeover later stages of a sequential derivation, the example EMMAdocument above can be equivalently represented as follows:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:derivation>    <emma:interpretation        emma:process="http://example.com/myasr1.xml"        emma:source="http://example.com/microphone/NC-61"        emma:signal="http://example.com/signals/sg23.wav"        emma:confidence="0.6"        emma:medium="acoustic"        emma:mode="voice"        emma:function="dialog"        emma:verbal="true"        emma:tokens="from boston to denver tomorrow"        emma:lang="en-US">      <answer>From Boston to Denver tomorrow</answer>    </emma:interpretation>    <emma:interpretation        emma:process="http://example.com/mynlu1.xml"        emma:confidence="0.8">      <emma:derived-from resource="#raw" composite="false"/>      <origin>Boston</origin>      <destination>Denver</destination>      <date>tomorrow</date>    </emma:interpretation>  </emma:derivation>  <emma:interpretation      emma:process="http://example.com/myrefresolution1.xml">    <emma:derived-from resource="#better" composite="false"/>    <origin>Boston</origin>    <destination>Denver</destination>    <date>03152003</date>  </emma:interpretation></emma:emma>

The fully specified derivation illustrated above is equivalentto the reduced form derivation following it where only annotationswith new values are specified at each stage. These two EMMAdocuments MUST yield the same result when processed by an EMMAprocessor.

Theemma:confidence annotation is respecified onthebetter interpretation. This indicates theconfidence score for natural language understanding, whereasemma:confidence on theraw interpretationindicates the speech recognition confidence score.

In order to determine the full set of annotations that apply toanemma:interpretation element an EMMA processor orscript needs to access the annotations directly on that element andfor any that are not specified follow the reference in theresource attribute of theemma:derived-from element to add in annotations fromearlier stages of the derivation.

The EMMA annotations break down into three groups with respectto their scope in sequential derivations. One group of annotationsalways holds true for all members of a sequentialderivation. A second groupis always respecified oneach stage of the derivation. A third group may or may not berespecified.

One potential problem with this annotation scoping mechanism isthat earlier annotations could be lost if earlier stages of aderivation were dropped in order to reduce message size. Thisproblem can be overcome by considering annotation scope at thepoint where earlier derivation stages are discarded and populatingthe final interpretation in the derivation with all of theannotations which it could inherit. For example, if theraw andbetter stages were dropped theresulting EMMA document would be:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation      emma:start="1087995961542"      emma:end="1087995963542"      emma:process="http://example.com/myrefresolution1.xml"      emma:source="http://example.com/microphone/NC-61"      emma:signal="http://example.com/signals/sg23.wav"      emma:confidence="0.8"      emma:medium="acoustic"      emma:mode="voice"      emma:function="dialog"      emma:verbal="true"      emma:tokens="from boston to denver tomorrow"      emma:lang="en-US">    <emma:derived-from resource="#better" composite="false"/>    <origin>Boston</origin>    <destination>Denver</destination>    <date>03152003</date>  </emma:interpretation></emma:emma>

Annotations on anemma:one-of element are assumedto apply to all of the container elements within theemma:one-of.

Ifemma:one-of appears with anotheremma:one-of then annotations on the parentemma:one-of are assumed to apply to the children ofthe childemma:one-of.

Annotations onemma:group oremma:sequence do not apply to their childelements.

5. Conformance

The contents of this section are normative.

5.1 Conforming EMMA Documents

A document is a Conforming EMMA Document if it meets both thefollowing conditions:

• It is a well-formed XML document [XML]conforming to Namespaces in XML [XMLNS].
• It adheres to the specification described in this document(EMMA Specification) including the constraints expressed in theSchema (seeAppendix A) and having an XMLProlog and root element as specified inSection3.1.

The EMMA specification and these conformance criteria provide nodesignated size limits on any aspect of EMMA documents. There areno maximum values on the number of elements, the amount ofcharacter data, or the number of characters in attributevalues.

Within this specification, the term URI refers to aUniversal Resource Identifier as defined in [RFC3986] and extended in [RFC3987] with the new name IRI. The term URI hasbeen retained in preference to IRI to avoid introducing new namesfor concepts such as "Base URI" that are defined or referencedacross the whole family of XML specifications.

5.2 Using EMMA with other Namespaces

The EMMA namespace is intended to be used with other XMLnamespaces as per the Namespaces in XML Recommendation [XMLNS]. Future work by W3C is expected to address waysto specify conformance for documents involving multiplenamespaces.

5.3 Conforming EMMA Processors

A EMMA processor is a program that can process and/or generateConforming EMMA documents.

In a Conforming EMMA Processor, the XML parser MUST be able toparse and process all XML constructs defined by XML 1.1 [XML] and Namespaces in XML [XMLNS].It is not required that a Conforming EMMA Processor uses avalidating XML parser.

A Conforming EMMA Processor MUST correctly understand and applythe semantics of each markup element or attribute as described bythis document.

There is, however, no conformance requirement with respect toperformance characteristics of the EMMA Processor. For instance, nostatement is required regarding the accuracy, speed or othercharacteristics of output produced by the processor. No statementis made regarding the size of input that a EMMA Processor isrequired to support.

Appendices

Appendix A. XML andRELAX NGschemata

This section is Normative.

This section defines the formal syntax for EMMA documents interms of a normative XML Schema.

There are both an XML Schema andRELAX NG Schemafor the EMMA markup. The latest version of the XML Schema for EMMAis available athttp://www.w3.org/TR/emma/emma.xsdand the RELAX NG Schema can be found athttp://www.w3.org/TR/emma/emma.rng.

For stability it is RECOMMENDED that you use the dated URIavailable athttp://www.w3.org/TR/2009/REC-emma-20090210/emma.xsdandhttp://www.w3.org/TR/2009/REC-emma-20090210/emma.rng.

Appendix B. MIME type

This section isNormative.

This appendix registers a new MIME media type,"application/emma+xml".

The "application/emma+xml" media type isregistered with IANA athttp://www.iana.org/assignments/media-types/application/.

Appendix C.emma:hook and SRGS

This section isInformative.

One way to build an EMMA representation of a spoken input suchas "zoom in here" is to use grammar rules in the W3C SpeechRecognition Grammar Specification [SRGS] usingthe Semantic Interpretation[SISR]tags to build the application semantics with theemma:hook attribute. In this approach[ECMAScript] is specified in order to buildup an object representing the semantics. The resulting ECMAScriptobject is then translated to XML.

For our example case of "zoom in here". The following SRGS rulecould be used. TheSemantic Interpretation for SpeechRecognition specification[SISR] provides a reserved property_nsprefix for indicating the namespace to be used with anattribute.

<rule>  zoom in here  <tag>    $.command = new Object();    $.command.action = "zoom";    $.command.location = new Object();    $.command.location._attributes = new Object();    $.command.location._attributes.hook = new Object();    $.command.location._attributes.hook._nsprefix = "emma";    $.command.location._attributes.hook._value = "ink";    $.command.location.type = "area";  </tag></rule>

Application of this rule will result in the following ECMAScriptobject being built.

command: {      action: "zoom"      location: {        _attributes: {           hook: {             _nsprefix: "emma"             _value: "ink"             }           }        type: "area"      }}

SI processing in an XML environment wouldgenerate the following document:

<command>  <action>zoom</action>  <location emma:hook="ink">     <type>area</type>  </location></command>

This XML fragment might then appear within an EMMA document asfollows:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation      emma:medium="acoustic"      emma:mode="voice">    <command>      <action>zoom</action>      <location emma:hook="ink">         <type>area</type>      </location>    </command>  </emma:interpretation></emma:emma>

Theemma:hook annotation indicates that this speechinput needs to be combined with ink input such as thefollowing:

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation      emma:medium="tactile"      emma:mode="ink">    <location>      <type>area</type>      <points>42.1345 -37.128 42.1346 -37.120 ... </points>    </location>  </emma:interpretation></emma:emma>

This representation could be generated by a pen modalitycomponent performing gesture recognition and interpretation. Theinput to the component would be anInk Markup Languagespecification[INKML] of the inktrace and the output would be the EMMA document above.

The combination will result in the following EMMA document forthe combined speech and pen multimodal input.

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation      emma:medium="acoustic tactile"       emma:mode="voice ink"      emma:process="http://example.com/myintegrator.xml">    <emma:derived-from resource="http://example.com/voice1.emma/#voice1" composite="true"/>    <emma:derived-from resource="http://example.com/pen1.emma/#pen1" composite="true"/>    <command>       <action>zoom</action>       <location>         <type>area</type>         <points>42.1345 -37.128 42.1346 -37.120 ... </points>        </location>     </command>  </emma:interpretation></emma:emma>

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:interpretation      emma:medium="acoustic"      emma:mode="voice"      emma:tokens="send this file to this"      emma:start="1087995961500"      emma:end="1087995963542">    <command>      <action>send</action>        <arg1>          <object emma:hook="ink">            <type>file</type>            <number>1</number>          </object>        </arg1>       <arg2>         <object emma:hook="ink">           <number>1</number>         </object>       </arg2>    </command>  </emma:interpretation></emma:emma>

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example">  <emma:sequence>    <emma:interpretationemma:start="1087995960500"      emma:end="1087995960900"
      emma:medium="tactile"      emma:mode="ink">      <object>       <type>file</type>       <number>1</number>       <id>test.pdf</id>      <object>    </emma:interpretation>    <emma:interpretationemma:start="1087995961000"      emma:end="1087995961100"
      emma:medium="tactile"      emma:mode="ink">      <object>        <type>printer</type>        <number>1</number>        <id>lpt1</id>      <object>    </emma:interpretation>  </emma:sequence></emma:emma>

<emma:emma version="1.0"    xmlns:emma="http://www.w3.org/2003/04/emma"    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"    xsi:schemaLocation="http://www.w3.org/2003/04/emma     http://www.w3.org/TR/2009/REC-emma-20090210/emma.xsd"    xmlns="http://www.example.com/example"><emma:interpretation      emma:medium="acoustic tactile"      emma:mode="voice ink"      emma:tokens="send this file to this"      emma:process="http://example.com/myintegration.xml"      emma:start="1087995960500"      emma:end="1087995963542">  <emma:derived-from resource="http://example.com/voice2.emma/#voice2" composite="true"/>  <emma:derived-from resource="http://example.com/ink2.emma/#ink2" composite="true"/>  <command>   <action>send</action>    <arg1>     <object>       <type>file</type>       <number>1</number>        <id>test.pdf</id>     </object>    </arg1>    <arg2>     <object>       <type>printer</type>        <number>1</number>       <id>lpt1</id>     </object>    </arg2>  </command></emma:interpretation></emma:emma>

Appendix D. EMMA event interface

This section isInformative.

The W3C Document Object Model [DOM] definesplatform and language neutral interfaces that gives programs andscripts the means to dynamically access and update the content,structure and style of documents. DOM Events define a generic eventsystem which allows registration of event handlers, describes eventflow through a tree structure, and provides basic contextualinformation for each event.

This section of the EMMA specification extends the DOM Eventinterface for use with events that describe interpreted user inputin terms of a DOM Node for an EMMA document.

// File: emma.idl#ifndef _EMMA_IDL_#define _EMMA_IDL_#include "dom.idl"#include "views.idl"#include "events.idl"#pragma prefix "dom.w3c.org"module emma{  typedef dom::DOMString DOMString;  typedef dom::Node Node;  interface EMMAEvent : events::UIEvent {    readonly attribute dom::Node  node;    void               initEMMAEvent(in DOMString typeArg,                                   in boolean canBubbleArg,                                   in boolean cancelableArg,                                   in Node node);  };};#endif // _EMMA_IDL_

Appendix E. References

E.1 Normative references

BCP47

A. Phillips and M. Davis, editors.Tags for theIdentification of Languages, IETF, September 2006.

RFC3023

M. Murata et al., editors.XML Media Types. IETF RFC3023, January 2001.

RFC2046

N. Freed and N. Borenstein, editors.Multipurpose Internet MailExtensions (MIME) Part Two: Media Types. IETF RFC 2046,November 1996.

RFC2119

S. Bradner,editor.Key words for use in RFCs toIndicate Requirement Levels, IETFRFC 2119, March1997.

RFC3986

T. Berners-Lee et al., editors.Uniform Resource Identifier(URI): Generic Syntax. IETF RFC 3986, January2005.

RFC3987

M. Duerst and M. Suignard, editors.Internationalized ResourceIdentifiers (IRIs). IETF RFC 3987, January2005.

XML

Tim Brayet al., editors.Extensible MarkupLanguage (XML) 1.1. World Wide Web Consortium,W3CRecommendation, 2004.

XMLNS

Tim Brayet al., editors.Namespaces in XML 1.1,World Wide Web Consortium,W3C Recommendation,2006.

XML Schema Structures

Henry S. Thompsonet al., editors.XML Schema Part 1: StructuresSecond Edition, World Wide Web Consortium, W3CRecommendation, 2004.

XML Schema Datatypes

Paul V. Bironand Ashok Malhotra, editors.XML Schema Part 2:Datatypes Second Edition, World Wide Web Consortium,W3CRecommendation, 2004.

E.2 Informative references

DOM

Document Object Model,World Wide Web Consortium, 2005.

ECMAScript

ECMAScript

INKML

Yi-Min Chee, Max Froumentin, Stephen M. Watt, editors.Ink Markup Language (InkML),World Wide Web Consortium, W3C Working Draft, 2006.

SISR

Luc Van Tichelenand Dave Burke,editors.SemanticInterpretation for Speech Recognition, World Wide WebConsortium,W3C Proposed Recommendation, 2007.

SRGS

Andrew Hunt, Scott McGlashan, editors.Speech Recognition GrammarSpecification Version 1.0, World Wide Web Consortium, W3CRecommendation, 2004.

XFORMS

John M. Boyer et al., editors.XForms1.0(Second Edition), World Wide Web Consortium,W3CRecommendation, 2006.

RELAX-NG

James Clark and Makoto Murata, editors.RELAX NG Specification, OASIS, CommitteeSpecification, 2001.

EMMA Requirements

Stephane H. Maes and Stephen Potter, editors.Requirements for EMMA, WorldWide Web Consortium,W3C Note, 2003.

Graph Unification

Bob Carpenter.The Logic of Typed FeatureStructures, Cambridge Tracts in Theoretical Computer Science32, Cambridge University Press, 1992.

Kevin Knight.Unification: A MultidisciplinarySurvey, ACM Computing Surveys, 21(1), 1989.

Michael Johnston.Unification-based MultimodalParsing, Proceedings of Association for ComputationalLinguistics, pp. 624-630, 1998.

MMI Framework

James A. Larson, T.V. Raman and Dave Raggett, editors.W3C Multimodal InteractionFramework, World Wide Web Consortium, W3C Note,2003.

MMI Requirements

Stephane H. Maes and Vijay Saraswat, editors.Multimodal InteractionRequirements, World Wide Web Consortium, W3C Note,2003.

Appendix F. Changes since last draft

This section isInformative.

Since the publication of the Proposed Recommendation of the EMMAspecification, the following minor editorial changes have beenadded to the draft.

• Fixed wrong style of text.(1.2 Terminology)
• Changed schemaLocation URI in example codes from "http://www.w3.org/TR/2008/PR-emma-20081215/" to "http://www.w3.org/TR/2009/REC-emma-20090210/".(2. Structure of EMMA documents,3. EMMA structural elementsand4 EMMA annotations)
• Changed the note on the status of MIME type registration from "being submitted to the IESG for review, approval, and registration with IANA" to "registered with IANA at http://www.iana.org/assignments/media-types/application/" because the EMMA MIME type is registered with IANA.(Appendix B)

Appendix G. Acknowledgements

This section isInformative.

The editors would like to recognize the contributions of thecurrent and former members of the W3C Multimodal Interaction Group(listed in alphabetical order):

Kazuyuki Ashimura, W3C

Patrizio Bergallo, (until 2008, while at Loquendo)

Wu Chou, Avaya

Max Froumentin, (until 2006, while at W3C)

Katriina Halonen, Nokia

Jin Liu, T-Systems

Roberto Pieraccini, Speechcycle

Stephen Potter, Microsoft

Massimo Romanelli, DFKI

Yuan Shao, Canon