You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
1562 lines
64 KiB
1562 lines
64 KiB
<?xml version="1.0" encoding="UTF-8"?>
|
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
|
|
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
|
|
|
|
<html lang="EN" xmlns="http://www.w3.org/1999/xhtml" xml:lang="EN">
|
|
<head>
|
|
<meta name="generator" content=
|
|
"HTML Tidy for Linux/x86 (vers 1 September 2005), see www.w3.org" />
|
|
<meta http-equiv="Content-Type" content=
|
|
"text/html; charset=utf-8" />
|
|
|
|
<title>Authoring Applications for the Multimodal
|
|
Architecture</title>
|
|
<style type="text/css">
|
|
/*<![CDATA[*/
|
|
code { font-family: monospace; }
|
|
|
|
div.constraint,
|
|
div.issue,
|
|
div.note,
|
|
div.notice { margin-left: 2em; }
|
|
|
|
ol.enumar { list-style-type: decimal; }
|
|
ol.enumla { list-style-type: lower-alpha; }
|
|
ol.enumlr { list-style-type: lower-roman; }
|
|
ol.enumua { list-style-type: upper-alpha; }
|
|
ol.enumur { list-style-type: upper-roman; }
|
|
|
|
|
|
div.exampleInner pre { margin-left: 1em;
|
|
margin-top: 0em; margin-bottom: 0em}
|
|
div.exampleOuter {border: 4px double gray;
|
|
margin: 0em; padding: 0em}
|
|
div.exampleInner { background-color: #d5dee3;
|
|
border-top-width: 4px;
|
|
border-top-style: double;
|
|
border-top-color: #d3d3d3;
|
|
border-bottom-width: 4px;
|
|
border-bottom-style: double;
|
|
border-bottom-color: #d3d3d3;
|
|
padding: 4px; margin: 0em }
|
|
div.exampleWrapper { margin: 4px }
|
|
div.exampleHeader { font-weight: bold;
|
|
margin: 4px}
|
|
/*]]>*/
|
|
</style>
|
|
<link type="text/css" rel="stylesheet" href=
|
|
"http://www.w3.org/StyleSheets/TR/W3C-WG-NOTE" />
|
|
</head>
|
|
|
|
<body>
|
|
<div class="head">
|
|
<p><a href="http://www.w3.org/"><img width="72" height="48"
|
|
alt="W3C" src="http://www.w3.org/Icons/w3c_home" /></a></p>
|
|
|
|
<h1><a id="title" name="title"></a>Authoring Applications for
|
|
the Multimodal Architecture</h1>
|
|
|
|
<h2><a id="w3c-doctype" name="w3c-doctype"></a>W3C Working Group Note 2 July 2008</h2>
|
|
|
|
<dl>
|
|
<dt>This version:</dt>
|
|
<dd>
|
|
<a href="http://www.w3.org/TR/2008/NOTE-mmi-auth-20080702/">
|
|
http://www.w3.org/TR/2008/NOTE-mmi-auth-20080702/
|
|
</a>
|
|
</dd>
|
|
|
|
<dt>Latest version:</dt>
|
|
<dd>
|
|
<a href="http://www.w3.org/TR/mmi-auth/">http://www.w3.org/TR/mmi-auth/</a>
|
|
</dd>
|
|
|
|
<dt>Previous version:</dt>
|
|
<dd>This is the first version.</dd>
|
|
|
|
<dt>Editor:</dt>
|
|
<dd>Ingmar Kliche, Deutsche Telekom AG</dd>
|
|
</dl>
|
|
|
|
<p class="copyright"><a href="http://www.w3.org/Consortium/Legal/ipr-notice#Copyright">Copyright</a> © 2008 <a href="http://www.w3.org/"><acronym title="World Wide Web Consortium">W3C</acronym></a><sup>®</sup> (<a href="http://www.csail.mit.edu/"><acronym title="Massachusetts Institute of Technology">MIT</acronym></a>, <a href="http://www.ercim.org/"><acronym title="European Research Consortium for Informatics and Mathematics">ERCIM</acronym></a>, <a href="http://www.keio.ac.jp/">Keio</a>), All Rights Reserved. W3C <a href="http://www.w3.org/Consortium/Legal/ipr-notice#Legal_Disclaimer">liability</a>, <a href="http://www.w3.org/Consortium/Legal/ipr-notice#W3C_Trademarks">trademark</a> and <a href="http://www.w3.org/Consortium/Legal/copyright-documents">document use</a> rules apply.</p>
|
|
|
|
</div>
|
|
<hr />
|
|
|
|
<div>
|
|
<h2><a id="abstract" name="abstract"></a>Abstract</h2>
|
|
|
|
<p>This document describes a multimodal system which implements
|
|
the W3C Multimodal Architecture and gives an example of a
|
|
simple multimodal application authored using various W3C markup
|
|
languages, including SCXML, CCXML, VoiceXML 2.1 and HTML.</p>
|
|
</div>
|
|
|
|
<div>
|
|
<h2><a id="status" name="status"></a>Status of this
|
|
Document</h2>
|
|
|
|
<p><em>This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the <a href="http://www.w3.org/TR/">W3C technical reports index</a> at http://www.w3.org/TR/.</em></p>
|
|
|
|
<p>
|
|
This is the 2 July 2008
|
|
<a href="http://www.w3.org/2005/10/Process-20051014/tr.html#WGNote">
|
|
W3C Working Group Note
|
|
</a>
|
|
of "Authoring Applications for the Multimodal Architecture".
|
|
|
|
The Multimodal Interaction Working Group published an updated
|
|
Working Draft of the
|
|
"Multimodal Architecture and Interfaces" (MMI Architecture)
|
|
<a href="#MMI-ARCH">[MMI-ARCH]</a>
|
|
on 14 April 2008.
|
|
|
|
However, the Working Draft currently does not take a position
|
|
on several aspects of multimodal applications.
|
|
|
|
These include the startup phase, how components find each other, and
|
|
message transport, all of which must be addressed in actual
|
|
applications.
|
|
|
|
In order to provide a concrete illustration of a multimodal
|
|
application based on the Multimodal Architecture using current W3C
|
|
technologies, the Working Group has prepared this Note.
|
|
|
|
The goal of this Note is to make it easier to author concrete
|
|
multimodal Web applications.
|
|
|
|
This document represents the views of the group at the time of
|
|
publication.
|
|
|
|
The group is planning to enhance the MMI Architecture specification
|
|
based on this Working Group Note if the feedback is positive.
|
|
</p>
|
|
|
|
<p>
|
|
This document is one of a series produced by the
|
|
<a href="http://www.w3.org/2002/mmi/">
|
|
Multimodal Interaction Working Group</a>
|
|
of the W3C
|
|
<a href="http://www.w3.org/2002/mmi/Activity.html">
|
|
Multimodal Interaction Activity</a>.
|
|
</p>
|
|
|
|
<p>For more information about the Multimodal Interaction
|
|
Activity, please see the <a href=
|
|
"http://www.w3.org/2002/mmi/Activity">Multimodal Interaction
|
|
Activity statement</a>.</p>
|
|
|
|
<p>Comments for this specification are welcomed and should have
|
|
a subject starting with the prefix '[AUTH]'. Please send them
|
|
to <a href=
|
|
"mailto:www-multimodal@w3.org">www-multimodal@w3.org</a>, the
|
|
public email list for issues related to Multimodal. This list
|
|
is <a href=
|
|
"http://lists.w3.org/Archives/Public/www-multimodal/">archived</a>
|
|
and acceptance of this archiving policy is requested
|
|
automatically upon first post. To subscribe to this list send
|
|
an email to <a href= "mailto:www-multimodal-request@w3.org">www-multimodal-request@w3.org</a>
|
|
with the word subscribe in the subject line.</p>
|
|
|
|
<p>Publication as a Working Group Note does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.</p>
|
|
|
|
<p> This document was produced by a group operating under the <a href="http://www.w3.org/Consortium/Patent-Policy-20040205/">5 February 2004 W3C Patent Policy</a>. W3C maintains a <a rel="disclosure" href="http://www.w3.org/2004/01/pp-impl/34607/status">public list of any patent disclosures</a> made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains <a href="http://www.w3.org/Consortium/Patent-Policy-20040205/#def-essential">Essential Claim(s)</a> must disclose the information in accordance with <a href="http://www.w3.org/Consortium/Patent-Policy-20040205/#sec-Disclosure">section 6 of the W3C Patent Policy</a>. </p>
|
|
|
|
</div>
|
|
|
|
<div class="toc">
|
|
<h2><a id="contents" name="contents"></a>Table of Contents</h2>
|
|
|
|
<p class="toc">1 <a href="#introduction">Introduction</a><br />
|
|
2 <a href="#overview">Overview</a><br />
|
|
3 <a href="#compontents">Implementation of the
|
|
components</a><br />
|
|
3.1 <a href="#RF">The Runtime
|
|
Framework</a><br />
|
|
3.2 <a href="#GUI-MC">GUI Modality
|
|
Component</a><br />
|
|
3.3 <a href="#VUI-MC">Voice Modality
|
|
Component</a><br />
|
|
4 <a href="#N1018A">Initiating multimodal sessions</a><br />
|
|
5 <a href="#N10196">Authoring example</a><br />
|
|
5.1 <a href=
|
|
"#T-shirt.scxml">T-Shirt.scxml</a><br />
|
|
5.2 <a href=
|
|
"#captureColorSize.html">captureColorSize.html</a><br />
|
|
5.3 <a href=
|
|
"#dispatcher.ccxml">dispatcher.ccxml</a><br />
|
|
5.4 <a href=
|
|
"#captureColor.vxml">captureColor.vxml</a><br /></p>
|
|
|
|
<h3><a id="appendices" name="appendices"></a>Appendices</h3>
|
|
|
|
<p class="toc">A <a href=
|
|
"#acknowledgments">Acknowledgments</a><br />
|
|
B <a href="#references">References</a><br /></p>
|
|
</div>
|
|
<hr />
|
|
|
|
<div class="body">
|
|
<div class="div1">
|
|
<h2><a id="introduction" name="introduction"></a>1
|
|
Introduction</h2>
|
|
|
|
<p>The W3C Multimodal Interaction (MMI) Working Group
|
|
develops an architecture <a href="#MMI-ARCH">[MMI-ARCH]</a>
|
|
for the Multimodal Interaction framework <a href=
|
|
"#MMIF">[MMIF]</a>. The Multimodal Architecture describes a
|
|
general and flexible framework for interoperability of the
|
|
various components of the multimodal framework (e.g. modality
|
|
components and the interaction manager) in an abstract way.
|
|
Among others it defines interfaces and messages between the
|
|
constituents of the framework, but it is up to the
|
|
implementation to decide how these messages are transferred
|
|
in case of a distributed implementation.</p>
|
|
|
|
<p>The intention of this document is to provide a proposal of
|
|
<em>how to implement</em> a multimodal runtime environment as
|
|
well as an application based on the W3C Multimodal
|
|
Architecture using <em>existing</em> W3C technologies. This
|
|
proposal uses CCXML and VoiceXML to implement a voice
|
|
modality component. Note that this is just one possibility
|
|
for implementing it.</p>
|
|
|
|
<p>Note: The W3C Voice Browser Working Group is currently
|
|
developing VoiceXML 3.0 which is the next major release of
|
|
VoiceXML and will enable voice browsers to fit into the W3C
|
|
Multimodal Architecture as a modality component. As VoiceXML
|
|
3.0 implementations are not yet available, this document
|
|
relies on the existing VoiceXML 2.1 specification <a href=
|
|
"#VoiceXML">[VoiceXML]</a>.</p>
|
|
|
|
<p>The Multimodal Interaction Working Group itself wants to
|
|
learn from this authoring example where improvements are
|
|
possible and necessary. We also intend to present how we
|
|
think that multimodal applications will be authored in the
|
|
future.</p>
|
|
</div>
|
|
|
|
<div class="div1">
|
|
<h2><a id="overview" name="overview"></a>2 Overview</h2>
|
|
|
|
<p>The W3C Multimodal Architecture consists of the following
|
|
<em>main</em> constituents (see also: <a href=
|
|
"http://www.w3.org/TR/mmi-arch/#ArchDiagram">MMI Runtime
|
|
Architecture Diagram</a>):</p>
|
|
|
|
<ul>
|
|
<li>Runtime Framework containing the Interaction
|
|
Manager.</li>
|
|
|
|
<li>Modality components.</li>
|
|
</ul>
|
|
|
|
<p>In this document we discuss a distributed implementation
|
|
of the multimodal framework using the following components
|
|
and technologies:</p>
|
|
|
|
<ul>
|
|
<li>Interaction Manager (IM) based on a state machine,
|
|
described using <a href="#SCXML">[SCXML]</a>.</li>
|
|
|
|
<li>GUI Modality component based on <a href=
|
|
"#XHTML">[XHTML]</a> and ECMAScript.</li>
|
|
|
|
<li>Voice Modality component based on <a href=
|
|
"#VoiceXML">[VoiceXML]</a> and <a href=
|
|
"#CCXML">[CCXML]</a>.</li>
|
|
|
|
<li>Modality component API based on HTTP (event transport)
|
|
and XML (event representation).</li>
|
|
</ul>
|
|
|
|
<p>The following figure shows all these components mapped to
|
|
the <a href="http://www.w3.org/TR/mmi-arch/#ArchDiagram">MMI
|
|
Runtime Architecture Diagram</a>:</p><img src=
|
|
"images/MMI-Authoring-Environment.png" alt=
|
|
"Runtime Architecture Diagram" />
|
|
|
|
<p>The dashed boxes correspond to (logical) components within
|
|
the MMI architecture whereas solid lines correspond to actual
|
|
software or hardware components used to implement the
|
|
system.</p>
|
|
|
|
<p>The voice input/output device shown in the figure above
|
|
may be a regular (mobile) phone or a Voice-over-IP (soft)
|
|
phone. In any case a phone connection to a standard voice
|
|
browser is used.</p>
|
|
</div>
|
|
|
|
<div class="div1">
|
|
<h2><a id="compontents" name="compontents"></a>3
|
|
Implementation of the components</h2>
|
|
|
|
<p>This section discusses one possible implementation of the
|
|
Multimodal Architecture.</p>
|
|
|
|
<div class="div2">
|
|
<h3><a id="RF" name="RF"></a>3.1 The Runtime Framework</h3>
|
|
|
|
<p>The Runtime Framework provides the environment which
|
|
hosts the SCXML interpreter. It has to provide an interface
|
|
to receive events from external components (modality
|
|
components) and must be able to inject these events into an
|
|
existing SCXML session or to start SCXML interpreter
|
|
sessions. The Runtime Framework also needs to provide the
|
|
possibility to send events to external components (i.e.
|
|
some implementation of the SCXML <send> tag). In the
|
|
future this feature might be a covered by the "external
|
|
communications module" of the SCXML specification (<a href=
|
|
"#SCXML">[SCXML]</a>).</p>
|
|
|
|
<p>An implementation of an SCXML interpreter written in
|
|
Java is available open source from the Apache Software
|
|
Foundation <a href="#commonsSCXML">[Apache Commons
|
|
SCXML]</a>. One possibility for implementing a simple
|
|
runtime framework could be to combine the Apache commons
|
|
SCXML library <a href="#commonsSCXML">[Apache Commons
|
|
SCXML]</a> with a J2EE servlet engine (e.g. <a href=
|
|
"#apacheTomcat">[Apache Tomcat]</a>). The servlet engine
|
|
would be used to implement the HTTP I/O
|
|
processor.</p><img src=
|
|
"images/MMI-InteractionManager_and_I-O-processor.png" alt=
|
|
"SCXML based Interaction Manager" />
|
|
|
|
<p>Even though HTTP might not be the most efficient
|
|
solution as a transport protocol, it still has some
|
|
advantages. It is a widely used protocol and available in
|
|
nearly every programming language. In a distributed
|
|
scenario, where the Interaction Manger (i.e. Runtime
|
|
Framework) and the modality components are spread across
|
|
the network, proxy and firewall problems are easy to solve.
|
|
Also, our intended modality components (HTML browsers for
|
|
graphical modality and VoiceXML browsers for voice
|
|
modality) inherently support HTTP. Therefore we use HTTP
|
|
for this proof-of-concept implementation proposal. Other,
|
|
more scalable solutions might make use of other
|
|
protocols.</p>
|
|
|
|
<p>The Runtime Framework provides the I/O processor which
|
|
receives HTTP requests from modality components (containing
|
|
XML based life-cycle event representation). Based on the
|
|
event semantics the Runtime Framework logic has either to
|
|
start a new SCXML interpreter instance (when receiving a
|
|
<em>mmi:newContextRequest</em> message) or to inject an
|
|
event into a running SCXML interpreter instance.</p>
|
|
|
|
<p>In this scenario in terms of transport the Runtime
|
|
Framework acts as an HTTP server which receives HTTP
|
|
requests and modality components are HTTP clients sending
|
|
HTTP requests to the Runtime Framework. Therefore sending
|
|
events from modality components to the Interaction Manager
|
|
is relatively easy to implement using existing
|
|
technologies.</p>
|
|
|
|
<p>The multimodal runtime architecture also requires to
|
|
send events from the Interaction Manger to the modality
|
|
components asynchronously. To be able to leverage standard
|
|
components like HTML and VoiceXML browsers as modality
|
|
components (or modality component containers) events should
|
|
still be transferred using HTTP (as HTML and VoiceXML
|
|
browsers supporting the HTTP protocol natively). But the
|
|
browsers act as HTTP clients only. Therefore the
|
|
Interaction Manager has still the role of the HTTP server.
|
|
According to the HTTP model the client has to initiate
|
|
requests. To enable the Interaction Manager to send events
|
|
to the modality component, the modality component therefore
|
|
has to send HTTP requests to the Interaction Manager to ask
|
|
for events. This technique is usually known as polling.
|
|
Simple implementations have obvious drawbacks (e.g.
|
|
increased network traffic, additional delay) but it is
|
|
possible to optimize it to some extend (e.g. by blocking
|
|
the HTTP request server side and using timeouts). This
|
|
technique certainly has limitations for large scale
|
|
implementations, but it is relatively easy to implement
|
|
based on existing technologies and therefore a good choice
|
|
for a proof-of-concept.</p>
|
|
|
|
<p>Another promising approach could be <a href=
|
|
"#COMET">[COMET]</a> which uses long living HTTP
|
|
connections to stream data to the client. Again the client
|
|
has to open the HTTP connection. The server will stream an
|
|
HTTP response to the client and leaves the HTTP connection
|
|
open until the next event has to be sent to the client.
|
|
Meanwhile there are a lot of applications out there using
|
|
this server-push technology. Unfortunately this technology
|
|
is not well standardized yet and therefore requires browser
|
|
dependent implementations. But it is a potential solution
|
|
for the required server-push channel.</p>
|
|
</div>
|
|
|
|
<div class="div2">
|
|
<h3><a id="GUI-MC" name="GUI-MC"></a>3.2 GUI Modality
|
|
Component</h3>
|
|
|
|
<p>The GUI modality component may be implemented using HTML
|
|
and JavaScript.</p>
|
|
|
|
<p>According to the rules defined for the Multimodal
|
|
Architecture, the application logic resides within the
|
|
Interaction Manager. Therefore the modality component has
|
|
to send events (e.g. user initiated events like
|
|
<em>click</em> or <em>change</em>) to the Interaction
|
|
Manager. The Interaction Manager decides on possible
|
|
reactions to this events and sends events to the modality
|
|
component to instruct it to execute some action (e.g.
|
|
displaying something).</p>
|
|
|
|
<p>The modality component API may be implemented using
|
|
<a href="#XMLHttpRequest">[XMLHttpRequest]</a> (also know
|
|
as AJAX). Event handlers for user initiated events like
|
|
<em>change</em> for text input elements and <em>click</em>
|
|
events for button elements may easily convert these into
|
|
XML representations (MMI life-cycle event representation,
|
|
e.g. containing values of input fields) and sent them to
|
|
the Interaction Manager using XMLHttpRequests.</p>
|
|
|
|
<p>The following code snippet demonstrates the principle of
|
|
how to send events to a server side Interaction Manager
|
|
(assuming a servlet at someURL) using ECMAScript and
|
|
XMLHttpRequests:</p>
|
|
|
|
<div class="exampleInner">
|
|
<pre>
|
|
/* The sendMmiLifecycleEvent() function sends the MMI lifecycle
|
|
event, potentially containing data values like color. The implementation of
|
|
this function is vendor specific. The function is called to send a life cycle
|
|
event to the Runtime Framework using AJAX. The parameter "payload" contains a life
|
|
cycle event object.
|
|
*/
|
|
function sendMmiLifecycleEvent(source, context, payload)
|
|
{
|
|
var xmlHttpRequest = new XMLHttpRequest();
|
|
|
|
// relative url, assuming that AJAX requests go to a url
|
|
// relative to the documents url
|
|
var url ="./someURL";
|
|
|
|
var XMLpayload = payload.toXML(source, context);
|
|
|
|
xmlHttpRequest.open("POST", url, true);
|
|
xmlHttpRequest.onreadystatechange = readystatehandler;
|
|
xmlHttpRequest.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
|
|
xmlHttpRequest.send(XMLpayload);
|
|
|
|
function readystatehandler()
|
|
{
|
|
if (xmlHttpRequest.status == 200 || xmlHttpRequest.status==304) {
|
|
// be quiet in case of success
|
|
// alert("success");
|
|
} else {
|
|
// alert error
|
|
alert("send failure");
|
|
}
|
|
}
|
|
}
|
|
|
|
// JavaScript Event (pseudo) object
|
|
function LifeCycleEvent(mmiEvType, eventType, fieldName, fieldValue)
|
|
{
|
|
this.mmiEventType = mmiEvType; // e.g. extension
|
|
this.eventType = eventType; // user initiated event, e.g. change
|
|
this.fieldName = fieldName; // e.g. HTML id of the field
|
|
this.fieldValue = fieldValue; // e.g. value of the field
|
|
}
|
|
|
|
// method of LifeCycleEvent object to generate XML string from its properties
|
|
LifeCycleEvent.prototype.toXML = function(source, context)
|
|
{
|
|
var mmiLifeCycleEvent;
|
|
|
|
mmiLifeCycleEvent = '&lt;mmi version="1.0" xmlns:mmi="http://www.w3.org/2008/04/mmi-arch"&gt;';
|
|
mmiLifeCycleEvent += ' &lt;mmi:" + this.mmiEventType + "\"";
|
|
mmiLifeCycleEvent += ' mmi:source="' + source + '" mmi:context="' + context + '"&gt;';
|
|
mmiLifeCycleEvent += ' &lt;mmi:data&gt;';
|
|
mmiLifeCycleEvent += ' &lt;eventType&gt;' + this.eventType + '&lt;/eventType&gt;';
|
|
mmiLifeCycleEvent += ' &lt;fieldName&gt;' + this.fieldName + '&lt;/fieldName&gt;';
|
|
mmiLifeCycleEvent += ' &lt;fieldValue&gt;' + this.fieldValue + '&lt;/fieldValue&gt;';
|
|
mmiLifeCycleEvent += ' &lt;/mmi:data&gt;';
|
|
mmiLifeCycleEvent += ' &lt;/mmi:' + this.mmiEventType + '&gt;';
|
|
mmiLifeCycleEvent += '&lt;/mmi&gt;";
|
|
|
|
return mmiLifeCycleEvent;
|
|
}
|
|
</pre>
|
|
</div>
|
|
|
|
<p>As described above, receiving events from the
|
|
Interaction Manager requires to send an HTTP request to the
|
|
server (i.e. Runtime Framework). The response contains an
|
|
XML coded event which represents an MMI life-cycle event.
|
|
An event, indicating the change of the value of
|
|
<em>color</em>, would be represented as a MMI life-cycle
|
|
event "mmi:extension" (see <a href=
|
|
"#MMI-ARCH">[MMI-ARCH]</a>) and could look like this:</p>
|
|
|
|
<div class="exampleInner">
|
|
<pre>
|
|
<?xml version="1.0" encoding="UTF-8"?>
|
|
<mmi version="1.0" xmlns:mmi="http://www.w3.org/2008/04/mmi-arch">
|
|
<mmi:startRequest mmi:source="" mmi:target="" mmi:context="">
|
|
<mmi:contentURL href="someContentURI" max-age="" fetchtimeout="1s">
|
|
<mmi:data>
|
|
</mmi:data>
|
|
</mmi:extension>
|
|
</mmi>
|
|
</pre>
|
|
</div>
|
|
|
|
<p>It should be mentioned, that the content of the
|
|
<mmi:data> element is application specific.</p>
|
|
|
|
<p>This approach requires to send an asynchronous
|
|
XMLHttpRequest to the Runtime Framework (expecting the
|
|
request to be blocked at the server) and to interpret the
|
|
response accordingly: either taking any local action based
|
|
on the event semantics and/or re-sending another request to
|
|
the Runtime Framework.</p>
|
|
|
|
<div class="exampleInner">
|
|
<pre>
|
|
/* This function handles all incoming MMI lifecycle events. They may be fetched
|
|
from the server side interaction manager using AJAX. The returned XML document
|
|
is the MMI lifecycle event.
|
|
*/
|
|
function handleIncomingMmiEvents(xml)
|
|
{
|
|
// check if incoming message is MMI lifecycle event
|
|
// perform a very simple check:
|
|
if(xml.match("<mmi:"))
|
|
{
|
|
// parse incoming xml string to DOM
|
|
parser=new DOMParser();
|
|
doc=parser.parseFromString(xml,"text/xml");
|
|
|
|
var element = doc.documentElement;
|
|
if(element.childNodes[0].nodeName=="mmi:newContextResponse")
|
|
{
|
|
_CONTEXT = element.childNodes[0].getAttribute("mmi:context");
|
|
}
|
|
else if(element.childNodes[0].nodeName=="mmi:extension")
|
|
{
|
|
if(element.childNodes[0].childNodes[0].nodeName=="mmi:data")
|
|
{
|
|
// Application specific extension
|
|
// In this example we receive the name of a function and the params.
|
|
// This has to be evaluated locally using eval().
|
|
var functionname = element.childNodes[0].childNodes[0].childNodes[0].childNodes[0].nodeValue;
|
|
var elementname = element.childNodes[0].childNodes[0].childNodes[1].childNodes[0].nodeValue;
|
|
var elementvalue = element.childNodes[0].childNodes[0].childNodes[2].childNodes[0].nodeValue;
|
|
eval(functionname + "(elementname ,elementvalue)");
|
|
}
|
|
}
|
|
else if(element.childNodes[0].nodeName=="mmi:clearContextRequest")
|
|
{
|
|
// create new mmiLifeCycleEvent Object that signals the removal of the Context
|
|
event = new LifeCycleEvent("clearContextResponse", "", "", "");
|
|
|
|
// send clearContextResponse lifecycle event
|
|
sendMmiLifecycleEvent(_SOURCE, _CONTEXT, event);
|
|
}
|
|
else
|
|
{
|
|
// unknown lifecycle event
|
|
alert("MMI lifecycle event not handled.");
|
|
}
|
|
|
|
// send HTTP request to server to receive lifecycle event.
|
|
readMmiLifecycleEvent();
|
|
}
|
|
else // check if contains "<mmi:"
|
|
{
|
|
// --> it is not a valid lifecycle event!
|
|
alert("Error: wrong message!");
|
|
}
|
|
}
|
|
</pre>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="div2">
|
|
<h3><a id="VUI-MC" name="VUI-MC"></a>3.3 Voice Modality
|
|
Component</h3>
|
|
|
|
<p>The Voice Modality Component may be implemented using
|
|
CCXML and VoiceXML 2.1.</p>
|
|
|
|
<p>VoiceXML 2.1 does not provide an external eventing
|
|
functionality. As CCXML 1.0 defines an external event
|
|
interface (<a href=
|
|
"http://www.w3.org/TR/ccxml/#basichttpio">Basic HTTP Event
|
|
I/O Processor</a>), which allows to inject external events
|
|
into a running CCXML session or to start new CCXML
|
|
sessions, CCXML will therefore be used as a event bridge
|
|
between VoiceXML and the Interaction Manager. CCXML will
|
|
receive events from the Interaction Manager and - depending
|
|
on the event semantics - start a VoiceXML dialog.</p>
|
|
|
|
<p>VoiceXML will be used to implement the actual voice user
|
|
interface (play prompt and control ASR). User input
|
|
collected by VoiceXML will be returned to CCXML. CCXML has
|
|
the ability to send HTTP requests to external components.
|
|
This feature will be used to send events back to the
|
|
multimodal runtime framework to inject events into the
|
|
SCXML based Interaction Manager.</p>
|
|
|
|
<p>Due to the fact that VoiceXML must return to CCXML (and
|
|
hence exit) to return results (e.g. recognition results)
|
|
the VoiceXML user interface has to be implemented as small
|
|
independent scripts. Each script corresponds to a single
|
|
action, like <em>play a prompt</em> or <em>start grammar
|
|
and listen to user input</em>.</p>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="div1">
|
|
<h2><a id="N1018A" name="N1018A"></a>4 Initiating multimodal
|
|
sessions</h2>
|
|
|
|
<p>Now, as we have described the basics of all constituents,
|
|
we need to define the setup of a multimodal session.</p>
|
|
|
|
<p>A multimodal session may be initiated using a GUI modality
|
|
component. The user starts a web browser and loads a HTML
|
|
document from a given URL. Upon load, the HTML document
|
|
registers corresponding event handlers (e.g. for change
|
|
events) and is able to send messages to the Interaction
|
|
Manager using AJAX (i.e. XMLHttpRequests).</p>
|
|
|
|
<p>The HTML document may contain a special text input field
|
|
which is used to collect the users phone number or SIP URL.
|
|
Once the user has entered this information it is sent (e.g.
|
|
by pressing a corresponding button) to the Interaction
|
|
Manager. The Interaction Manager generates a message towards
|
|
the CCXML event processor to create a new CCXML session and
|
|
to initiate a phone call to the given telephone number (or
|
|
SIP URL).</p>
|
|
|
|
<p>As soon as the telephone connection has been established
|
|
successfully the multimodal session is initiated. Now the
|
|
Interaction Manager is capable of controlling the two
|
|
modalities by sending life-cycle events.</p>
|
|
</div>
|
|
|
|
<div class="div1">
|
|
<h2><a id="N10196" name="N10196"></a>5 Authoring example</h2>
|
|
|
|
<p>This section ties together the previously described
|
|
components to implement a sample application. The multimodal
|
|
<em>T-Shirt example</em> contains a combined graphical and
|
|
voice user interface and allows to fill in a form containing
|
|
two fields (color and size) either by voice or by
|
|
pen/keyboard.</p>
|
|
|
|
<p>The following figure shows the corresponding state machine
|
|
logic for this example together with the MMI life-cycle
|
|
events.</p><img src="images/T-Shirt_StateMachine.png" alt=
|
|
"State machine logic and MMI life-cycle events" />
|
|
|
|
<div class="div2">
|
|
<h3><a id="T-shirt.scxml" name="T-shirt.scxml"></a>5.1
|
|
T-Shirt.scxml</h3>
|
|
|
|
<p>The state machine could be represented in SCXML source
|
|
code (T-Shirt.scxml) as follows:</p>
|
|
|
|
<div class="exampleInner">
|
|
<pre>
|
|
<?xml version="1.0" encoding="UTF-8"?>
|
|
<scxml version="1.0" profile="ecmascript" initial="getColor" >
|
|
<!-- we assume there is a script library which constructs MMI lifecycle events etc. -->
|
|
<script src="mmi.js"/>
|
|
|
|
<!-- datamodel definition -->
|
|
<datamodel>
|
|
<data id="color" expr=""/>
|
|
<data id="size" expr=""/>
|
|
<data id="received" expr="0"/>
|
|
</datamodel>
|
|
|
|
<!-- state getColor -->
|
|
<state id="getColor">
|
|
<onentry>
|
|
<script>
|
|
mmiEvent = new mmiStartRequest();
|
|
mmiEvent.setURL('captureColorSize.html');
|
|
</script>
|
|
<!-- issue startRequest to GUI -->
|
|
<send event="mmi:startRequest" target="GUI" targetType="x-ajax" namelist="mmiEvent"/>
|
|
<script>
|
|
mmiEvent = new mmiStartRequest();
|
|
mmiEvent.setURL('getColor.vxml');
|
|
</script>
|
|
<!-- issue startRequest to VUI -->
|
|
<send event="mmi:startRequest" target="VUI" targetType="basichttp" namelist="mmiEvent"/>
|
|
</onentry>
|
|
|
|
<!-- handle voice input -->
|
|
<transition event="mmi:done" cond="_event.data..@source.toString() == 'VUI' &&
|
|
_event.data..@status.toString() == 'success'" target="echoColor"/>
|
|
<!-- save color to data model -->
|
|
<assign location="_data.color" expr="_event.data..color.toString()"/>
|
|
<!-- send event to GUI to display information -->
|
|
<script>
|
|
mmiEvent = new mmiExtension();
|
|
// construct content of data element of extension event as XML string
|
|
dataFieldValue = "&lt;eventType&gt;_check&lt;/eventType&gt;";
|
|
dataFieldValue += "&lt;fieldName&gt;color&lt;/fieldName&gt;";
|
|
dataFieldValue += "&lt;fieldValue&gt;" + color + "&lt;/fieldValue&gt;";
|
|
mmiEvent.setDataField(dataFieldValue);
|
|
</script>
|
|
<send event="mmi:extension" target="GUI" targetType="x-ajax" namelist="mmiEvent"/>
|
|
</transition>
|
|
|
|
<!-- handle GUI input -->
|
|
<transition event="mmi:extension" cond="_event.data..@source.toString() == 'GUI' &&
|
|
_event.data..@status.toString() == 'success'" target="echoColor"/>
|
|
<!-- save color to data model -->
|
|
<assign location="_data.color" expr="_event.data..color.toString()"/>
|
|
</transition>
|
|
|
|
<!-- error handling -->
|
|
<transition event="mmi:startResponse" cond="_event.data..@status.toString() == 'error'" target="failure"/>
|
|
<transition event="mmi:done" cond="_event.data..@status.toString() == 'error'" target="failure"/>
|
|
</state>
|
|
|
|
<!-- state echoColor -->
|
|
<state id="echoColor">
|
|
<onentry>
|
|
<!-- play back color to user -->
|
|
<script>
|
|
mmiEvent = new mmiStartRequest();
|
|
mmiEvent.setURL('echoColor.vxml');
|
|
// construct content of data element of extension event as XML string
|
|
dataFieldValue = "&lt;color&gt;" + color + "&lt;/color&gt;";
|
|
mmiEvent.setDataField(dataFieldValue);
|
|
</script>
|
|
<send event="mmi:startRequest" target="VUI" targetType="basichttp" namelist="mmiEvent"/>
|
|
</onentry>
|
|
|
|
<!-- play prompt done -->
|
|
<transition event="mmi:done" cond="_event.data..@source.toString() == 'VUI' &&
|
|
_event.data..@status.toString() == 'success'" target="getSize"/>
|
|
|
|
<!-- error handling -->
|
|
<transition event="mmi:startResponse" cond="_event.data..@status.toString() == 'error'" target="failure"/>
|
|
<transition event="mmi:done" cond="_event.data..@status.toString() == 'error'" target="failure"/>
|
|
</state>
|
|
|
|
<!-- state getSize -->
|
|
<state id="getSize">
|
|
<onentry>
|
|
<script>
|
|
mmiEvent = new mmiStartRequest();
|
|
mmiEvent.setURL('getSize.vxml');
|
|
</script>
|
|
<!-- issue startRequest to VUI -->
|
|
<send event="mmi:startRequest" target="VUI" targetType="basichttp" namelist="mmiEvent"/>
|
|
</onentry>
|
|
|
|
<!-- handle voice input -->
|
|
<transition event="mmi:done" cond="_event.data..@source.toString() == 'VUI' &&
|
|
_event.data..@status.toString() == 'success'" target="echoSize"/>
|
|
<!-- save color to data model -->
|
|
<assign location="_data.size" expr="_event.data..size.toString()"/>
|
|
<!-- send event to GUI to display information -->
|
|
<script>
|
|
mmiEvent = new mmiExtension();
|
|
// construct content of data element of extension event as XML string
|
|
dataFieldValue = "&lt;eventType&gt;_check&lt;/eventType&gt;";
|
|
dataFieldValue += "&lt;fieldName&gt;size&lt;/fieldName&gt;";
|
|
dataFieldValue += "&lt;fieldValue&gt;" + size + "&lt;/fieldValue&gt;";
|
|
mmiEvent.setDataField(dataFieldValue);
|
|
</script>
|
|
<send event="mmi:extension" target="GUI" targetType="x-ajax" namelist="mmiEvent"/>
|
|
</transition>
|
|
|
|
<!-- handle GUI input -->
|
|
<transition event="mmi:extension" cond="_event.data..@source.toString() == 'GUI' &&
|
|
_event.data..@status.toString() == 'success'" target="echoSize"/>
|
|
<!-- save size to data model -->
|
|
<assign location="_data.size" expr="_event.data..size.toString()"/>
|
|
</transition>
|
|
|
|
<!-- error handling -->
|
|
<transition event="mmi:startResponse" cond="_event.data..@status.toString() == 'error'" target="failure"/>
|
|
<transition event="mmi:done" cond="_event.data..@status.toString() == 'error'" target="failure"/>
|
|
</state>
|
|
|
|
<!-- state echoSize -->
|
|
<state id="echoSize">
|
|
<onentry>
|
|
<!-- play back color to user -->
|
|
<script>
|
|
mmiEvent = new mmiStartRequest();
|
|
mmiEvent.setURL('echoSize.vxml');
|
|
// construct content of data element of extension event as XML string
|
|
dataFieldValue = "&lt;size&gt;" + size + "&lt;/size&gt;";
|
|
mmiEvent.setDataField(dataFieldValue);
|
|
</script>
|
|
<send event="mmi:startRequest" target="VUI" targetType="basichttp" namelist="mmiEvent"/>
|
|
</onentry>
|
|
|
|
<!-- play prompt done -->
|
|
<transition event="mmi:done" cond="_event.data..@source.toString() == 'VUI' &&
|
|
_event.data..@status.toString() == 'success'" target="endOfInteraction"/>
|
|
|
|
<!-- error handling -->
|
|
<transition event="mmi:startResponse" cond="_event.data..@status.toString() == 'error'" target="failure"/>
|
|
<transition event="mmi:done" cond="_event.data..@status.toString() == 'error'" target="failure"/>
|
|
</state>
|
|
|
|
<!-- state endOfInteraction-->
|
|
<state id="endOfInteraction">
|
|
<onentry>
|
|
<!-- number of received clearContextResponse messages, we are waiting for two -->
|
|
<assign location="received" expr="0"/>
|
|
|
|
<!-- issue clearContextRequest messages -->
|
|
<script>
|
|
mmiEvent = new mmiClearContextRequest();
|
|
</script>
|
|
<!-- issue clearContextRequest to GUI -->
|
|
<send event="mmi:clearContextRequest" target="GUI" targetType="x-ajax" namelist="mmiEvent"/>
|
|
<script>
|
|
mmiEvent = new mmiClearContextRequest();
|
|
</script>
|
|
<!-- issue clearContextRequest to VUI -->
|
|
<send event="mmi:clearContextRequest" target="VUI" targetType="basichttp" namelist="mmiEvent"/>
|
|
</onentry>
|
|
<transition event="mmi:clearContextResponse" cond="received = 0">
|
|
<!-- increase counter -->
|
|
<assign location="received" expr="1"/>
|
|
</transition>
|
|
<transition event="mmi:clearContextResponse" cond="received > 0" target="end"/>
|
|
</state>
|
|
|
|
<!-- state failure -->
|
|
<state id ="failure">
|
|
<!-- simply stop interaction -->
|
|
<transition target="endOfInteraction"/>
|
|
</state>
|
|
|
|
<!-- final state -->
|
|
<state id="end" final="true"/>
|
|
</scxml>
|
|
</pre>
|
|
</div>
|
|
</div>
|
|
|
|
<p>In this example we assume that the Runtime Framework
|
|
supports the <em>x-ajax</em> and <em>basichttp</em>
|
|
targettypes for the <send> tag. The GUI modality
|
|
component uses AJAX to communicate to the Runtime Framework.
|
|
Therefore we use <em>x-ajax</em> as the targettype, whereas
|
|
the Voice modality component is implemented using
|
|
CCXML/VoiceXML. As the external event interface of CCXML is
|
|
used to inject events into the CCXML session we have to make
|
|
use of the <em>basichttp</em> targettype.</p>
|
|
|
|
<div class="div2">
|
|
<h3><a id="captureColorSize.html" name=
|
|
"captureColorSize.html"></a>5.2 captureColorSize.html</h3>
|
|
|
|
<p>The following code fragment provides the basics of the
|
|
HTML source code for the GUI modality component (i.e.
|
|
captureColorSize.html):</p>
|
|
|
|
<div class="exampleInner">
|
|
<pre>
|
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
|
|
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
|
|
<html xmlns="http://www.w3.org/1999/xhtml">
|
|
|
|
<head>
|
|
|
|
<meta name="application" content="captureColorSize" />
|
|
<meta name="description" content="solicits value for size and color" />
|
|
|
|
<title>CaptureColorSize</title>
|
|
|
|
<script src="sendEvent.js" language="javascript"/>
|
|
<script src="evaluateResponseXML.js" language="javascript"/>
|
|
|
|
<!-- Event handlers that create and send external events -->
|
|
|
|
<script language="JavaScript" type="text/javascript">
|
|
|
|
var _SOURCE="GUI";
|
|
var _CONTEXT="";
|
|
|
|
function onloadHandler()
|
|
{
|
|
// create new mmiLifeCycleEvent Object that requests a new Context
|
|
event = new LifeCycleEvent("newContextRequest", "", "", "");
|
|
|
|
// send newContextRequest lifecycle event
|
|
sendMmiLifecycleEvent(_SOURCE, _CONTEXT, event);
|
|
|
|
// send HTTP request to server to receive lifecycle event.
|
|
readMmiLifecycleEvent();
|
|
}
|
|
|
|
/* HTML event handler. The functions makes use of the browser event object
|
|
which holds the id and the value (in this case the color value) of the HTML object.
|
|
*/
|
|
function eventHandler(event)
|
|
{
|
|
target = new Object();
|
|
if(event.target)
|
|
{
|
|
target = event.target;
|
|
}
|
|
// Internet Explorer has no attribute target
|
|
else if(event.srcElement)
|
|
{
|
|
target = event.srcElement;
|
|
}
|
|
|
|
event = new LifeCycleEvent("extension", event.type, target.id, target.value);
|
|
|
|
sendMmiLifecycleEvent(_SOURCE, _CONTEXT, event);
|
|
}
|
|
|
|
/* initiate AJAX request to the interaction manager to read the next MMI lifecycle
|
|
event. The returned event is handled asynchronously within handleIncomingMmiEvents().
|
|
Finally the next MMI lifecycle event is fetched.
|
|
*/
|
|
function readMmiLifecycleEvent()
|
|
{
|
|
// Start asynchronous XMLHttpRequest to receive MMI lifecycle event.
|
|
// We assume that the the IM always returns a lifecycle event, i.e.
|
|
// we do not handle special timeout events. Once the request returns,
|
|
// handleIncomingMmiEvents() will be called to evaluate the xml encoded event.
|
|
|
|
var xmlHttpRequest = new XMLHttpRequest();
|
|
|
|
// relative url, assuming that AJAX requests go to a url
|
|
// relative to the documents url
|
|
var url ="./getMMILifeCycleEvent";
|
|
|
|
xmlHttpRequest.open("GET", url, true);
|
|
xmlHttpRequest.onreadystatechange = readyhandler;
|
|
xmlHttpRequest.send(null);
|
|
|
|
function readyhandler()
|
|
{
|
|
if (xmlHttpRequest.readyState == 4) {
|
|
if (xmlHttpRequest.status == 200) {
|
|
//handle lifecycle event
|
|
handleIncomingMmiEvents(xmlHttpRequest.responseText);
|
|
} else {
|
|
// alert error
|
|
alert("readMmiLifecycleEvent failure");
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
// function to check elements with the given html id, e.g. radio buttons
|
|
function _check(elementname, elementvalue)
|
|
{
|
|
document.getElementById(elementvalue).checked=true;
|
|
}
|
|
</script>
|
|
</head>
|
|
|
|
<body id="bodyId" onload="onloadHandler();">
|
|
|
|
<form action="" name="Color" id="Color">T-shirt color:
|
|
<table width="200">
|
|
<tr>
|
|
<td>
|
|
<label> <input type="radio" id="red"
|
|
name="radioGroup1" value="Red" onclick="eventHandler(event);" />
|
|
Red</label>
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
<label> <input type="radio" id="green"
|
|
name="radioGroup1" value="Green" onclick="eventHandler(event);" />
|
|
Green</label>
|
|
</td>
|
|
</tr>
|
|
<tr>
|
|
<td>
|
|
<label> <input type="radio" id="blue"
|
|
name="radioGroup1" value="Blue" onclick="eventHandler(event);" />
|
|
Blue</label>
|
|
</td>
|
|
</tr>
|
|
</table>
|
|
</form>
|
|
|
|
<form action="" name="Size" id="Size">T-shirt size:
|
|
<table width="200">
|
|
<tr>
|
|
<td><label> <input type="radio" id="small"
|
|
name="radioGroup2" value="Small" onclick="eventHandler(event);" />
|
|
Small</label></td>
|
|
</tr>
|
|
<tr>
|
|
<td><label> <input type="radio" id="medium"
|
|
name="radioGroup2" value="Medium" onclick="eventHandler(event);" />
|
|
Medium</label></td>
|
|
</tr>
|
|
<tr>
|
|
<td><label> <input type="radio" id="large"
|
|
name="radioGroup2" value="Large" onclick="eventHandler(event);" />
|
|
Large</label></td>
|
|
</tr>
|
|
</table>
|
|
</form>
|
|
|
|
</body>
|
|
</html>
|
|
</pre>
|
|
</div>
|
|
|
|
<p>sendEvent.js:</p>
|
|
|
|
<div class="exampleInner">
|
|
<pre>
|
|
/* The sendMmiLifecycleEvent() function sends the MMI lifecycle
|
|
event, potentially containing data values like color. The implementation of
|
|
this function is vendor specific. The function is called to send a life cycle
|
|
event to the Runtime Framework using AJAX. The parameter "payload" contains a life
|
|
cycle event object.
|
|
*/
|
|
function sendMmiLifecycleEvent(source, context, payload)
|
|
{
|
|
var xmlHttpRequest = new XMLHttpRequest();
|
|
|
|
// relative url, assuming that AJAX requests go to a url
|
|
// relative to the documents url
|
|
var url ="./someURL";
|
|
|
|
var XMLpayload = payload.toXML(source, context);
|
|
|
|
xmlHttpRequest.open("POST", url, true);
|
|
xmlHttpRequest.onreadystatechange = readystatehandler;
|
|
xmlHttpRequest.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
|
|
xmlHttpRequest.send(XMLpayload);
|
|
|
|
function readystatehandler()
|
|
{
|
|
if (xmlHttpRequest.status == 200 || xmlHttpRequest.status==304) {
|
|
// be quiet in case of success
|
|
// alert("success");
|
|
} else {
|
|
// alert error
|
|
alert("send failure");
|
|
}
|
|
}
|
|
}
|
|
|
|
// JavaScript Event (pseudo) object
|
|
function LifeCycleEvent(mmiEvType, eventType, fieldName, fieldValue)
|
|
{
|
|
this.mmiEventType = mmiEvType; // e.g. extension
|
|
this.eventType = eventType; // user initiated event, e.g. change
|
|
this.fieldName = fieldName; // e.g. HTML id of the field
|
|
this.fieldValue = fieldValue; // e.g. value of the field
|
|
}
|
|
|
|
// method of LifeCycleEvent object to generate XML string from its properties
|
|
LifeCycleEvent.prototype.toXML = function(source, context)
|
|
{
|
|
var mmiLifeCycleEvent;
|
|
|
|
mmiLifeCycleEvent = '&lt;mmi version="1.0" xmlns:mmi="http://www.w3.org/2008/04/mmi-arch"&gt;';
|
|
mmiLifeCycleEvent += ' &lt;mmi:' + this.mmiEventType + '"';
|
|
mmiLifeCycleEvent += ' mmi:source="' + source + '" mmi:context="' + context + '"&gt;';
|
|
mmiLifeCycleEvent += ' &lt;mmi:data&gt;";
|
|
mmiLifeCycleEvent += ' &lt;eventType&gt;' + this.eventType + '&lt;/eventType&gt;';
|
|
mmiLifeCycleEvent += ' &lt;fieldName&gt;' + this.fieldName + '&lt;/fieldName&gt;';
|
|
mmiLifeCycleEvent += ' &lt;fieldValue&gt;' + this.fieldValue + '&lt;/fieldValue&gt;';
|
|
mmiLifeCycleEvent += ' &lt;/mmi:data&gt;';
|
|
mmiLifeCycleEvent += ' &lt;/mmi:' + this.mmiEventType + '&gt;';
|
|
mmiLifeCycleEvent += '&lt;/mmi&gt;';
|
|
|
|
return mmiLifeCycleEvent;
|
|
}
|
|
</pre>
|
|
</div>
|
|
|
|
<p>evaluateResponseXML.js:</p>
|
|
|
|
<div class="exampleInner">
|
|
<pre>
|
|
/* This function handles all incoming MMI lifecycle events. They may be fetched
|
|
from the server side interaction manager using AJAX. The returned XML document
|
|
is the MMI lifecycle event.
|
|
*/
|
|
function handleIncomingMmiEvents(xml)
|
|
{
|
|
// check if incoming message is MMI lifecycle event
|
|
// perform a very simple check:
|
|
if(xml.match("<mmi:"))
|
|
{
|
|
// parse incoming xml string to DOM
|
|
parser=new DOMParser();
|
|
doc=parser.parseFromString(xml,"text/xml");
|
|
|
|
var element = doc.documentElement;
|
|
if(element.childNodes[0].nodeName=="mmi:newContextResponse")
|
|
{
|
|
_CONTEXT = element.childNodes[0].getAttribute("mmi:context");
|
|
}
|
|
else if(element.childNodes[0].nodeName=="mmi:extension")
|
|
{
|
|
if(element.childNodes[0].childNodes[0].nodeName=="mmi:data")
|
|
{
|
|
// Application specific extension
|
|
// In this example we receive the name of a function and the params.
|
|
// This has to be evaluated locally using eval().
|
|
var functionname = element.childNodes[0].childNodes[0].childNodes[0].childNodes[0].nodeValue;
|
|
var elementname = element.childNodes[0].childNodes[0].childNodes[1].childNodes[0].nodeValue;
|
|
var elementvalue = element.childNodes[0].childNodes[0].childNodes[2].childNodes[0].nodeValue;
|
|
eval(functionname + "(elementname ,elementvalue)");
|
|
}
|
|
}
|
|
else if(element.childNodes[0].nodeName=="mmi:clearContextRequest")
|
|
{
|
|
// create new mmiLifeCycleEvent Object that signals the removal of the Context
|
|
event = new LifeCycleEvent("clearContextResponse", "", "", "");
|
|
|
|
// send clearContextResponse lifecycle event
|
|
sendMmiLifecycleEvent(_SOURCE, _CONTEXT, event);
|
|
}
|
|
else
|
|
{
|
|
// unknown lifecycle event
|
|
alert("MMI lifecycle event not handled.");
|
|
}
|
|
|
|
// send HTTP request to server to receive lifecycle event.
|
|
readMmiLifecycleEvent();
|
|
}
|
|
else // check if contains "<mmi:"
|
|
{
|
|
// --> it is not a valid lifecycle event!
|
|
alert("Error: wrong message!");
|
|
}
|
|
}
|
|
</pre>
|
|
</div>
|
|
|
|
<p>The ECMAScript function <em>_check(elementname,
|
|
elementvalue)</em> within captureColorSize.html is provided
|
|
to check a radio button. To achieve this, the Interaction
|
|
Manager sends a mmi:extension life-cycle event where the
|
|
(application specific) <em>eventType</em> element within
|
|
the <mmi:data> element is set to <em>_check</em>. The
|
|
<em>fieldValue</em> element contains the HTML id of the
|
|
corresponding object. The <em>_check(...)</em> function
|
|
therefore simply uses the DOM API to activate the radio
|
|
button. The following example shows the MMI life-cycle
|
|
event to activate the green color radio button.</p>
|
|
|
|
<div class="exampleInner">
|
|
<pre>
|
|
<?xml version="1.0" encoding="UTF-8"?>
|
|
<mmi version="1.0" xmlns:mmi="http://www.w3.org/2008/04/mmi-arch">
|
|
<mmi:extension mmi:source="captureColorSize.html" mmi:context="">
|
|
<mmi:data>
|
|
<eventType>_check</eventType>
|
|
<fieldName>color</fieldName>
|
|
<fieldValue>green</fieldValue>
|
|
</mmi:data>
|
|
</mmi:extension>
|
|
</mmi>
|
|
</pre>
|
|
</div>
|
|
|
|
<p>This event is created within the SCXML script. See the
|
|
<em>getColor</em> state of the SCXML sample code (<a href=
|
|
"#T-shirt.scxml"><b>5.1 T-Shirt.scxml</b></a>).</p>
|
|
</div>
|
|
|
|
<div class="div2">
|
|
<h3><a id="dispatcher.ccxml" name=
|
|
"dispatcher.ccxml"></a>5.3 dispatcher.ccxml</h3>
|
|
|
|
<p>CCXML is used as a dispatcher of events between SCXML
|
|
and VoiceXML. The script <em>ccxml_events.js</em> contains
|
|
a collection of support functions to create MMI life-cycle
|
|
events or to start VoiceXML dialogs.</p>
|
|
|
|
<p>Note that this script is written to be application
|
|
independent.</p>
|
|
|
|
<div class="exampleInner">
|
|
<pre>
|
|
<?xml version="1.0" encoding="UTF-8" ?>
|
|
<ccxml version="1.0" xmlns="http://www.w3.org/2002/09/ccxml">
|
|
|
|
<!-- we assume there is a library of functions to send data to
|
|
the Interaction Manager -->
|
|
<script src="ccxml_events.js" />
|
|
|
|
<!-- CCXML session ID -->
|
|
<var name="connectionId" expr="''" />
|
|
|
|
<!-- SCXML session ID -->
|
|
<var name="interactionId" expr="''" />
|
|
<!-- request ID of lifecycle event -->
|
|
<var name="requestID" expr="'123456'" />
|
|
<!-- target type -->
|
|
<var name="SCXML" expr="'basichttp'" />
|
|
|
|
<!-- VXML dialog ID for termination -->
|
|
<var name="vxml_dialogid" expr="0" />
|
|
<!-- whether a VXML dialog is running or not -->
|
|
<var name="vxml_running" expr="false" />
|
|
<!-- whether a VXML dialog is in terminating or not -->
|
|
<var name="vxml_terminating" expr="false" />
|
|
|
|
<var name="prompt" expr="''" />
|
|
<var name="audio" expr="''" />
|
|
<var name="grammarUri" expr="''" />
|
|
<var name="fields" expr="''" />
|
|
|
|
|
|
<!-- Note: All events which are tagged with "(INTERNAL)" are standard events! -->
|
|
<eventprocessor>
|
|
|
|
<!-- ===================================================== -->
|
|
<!-- SCXML events -->
|
|
<!-- ===================================================== -->
|
|
|
|
<!-- CCXML (INTERNAL): when CCXML is started, it throws this internal event -->
|
|
<transition event="ccxml.loaded">
|
|
<script>
|
|
_ccxml.setSCXML_URI(session.values.scxml_serverip,
|
|
session.values.scxml_serverport,
|
|
session.values.scxml_serverpage);
|
|
</script>
|
|
<assign name="connectionId" expr="event$.connectionid" />
|
|
<assign name="interactionId" expr="session.values.interactionid" />
|
|
|
|
<!-- call SIP phone -->
|
|
<var name="sipip" expr="session.values.sip_phoneprefix + '@' + session.values.sip_phoneip +
|
|
':' + session.values.sip_phoneport" />
|
|
<createcall dest="sipip" connectionid="connectionId" />
|
|
</transition>
|
|
|
|
<!-- CCXML: terminate -->
|
|
<transition event="ccxml.terminate">
|
|
<send target="_ccxml.clearContextResponse(interactionId, requestID)"
|
|
targettype="SCXML" name="'ccxml.external'" />
|
|
<send target="session.id" targettype="'ccxml'" name="'this.exit'" />
|
|
</transition>
|
|
<transition event="this.exit">
|
|
<log expr="'CCXML.exit'" />
|
|
<exit />
|
|
</transition>
|
|
|
|
<!-- SIP: disconnect SIP phone -->
|
|
<transition event="sip.disconnect">
|
|
<disconnect connectionid="connectionId" />
|
|
</transition>
|
|
|
|
<!-- VXML: start -->
|
|
<transition event="vxml.start">
|
|
<assign name="prompt" expr="event$.prompt" />
|
|
<assign name="audio" expr="event$.audio" />
|
|
<assign name="grammarUri" expr="event$.grammarUri" />
|
|
<assign name="fields" expr="event$.fields" />
|
|
|
|
<!-- If a VXML dialog is running, terminate. otherwise start -->
|
|
<if cond="vxml_running == false">
|
|
<assign name="vxml_running" expr="true" />
|
|
<var name="sessionid" expr="event$.sessionid" />
|
|
<dialogstart src="_vxml.start(grammarUri, prompt, audio, fields)"
|
|
dialogid="vxml_dialogid" connectionid="connectionId" namelist="sessionid" />
|
|
<else />
|
|
<assign name="vxml_terminating" expr="true" />
|
|
<dialogterminate dialogid="vxml_dialogid" immediate="true" />
|
|
</if>
|
|
</transition>
|
|
|
|
<!-- VXML: terminate -->
|
|
<transition event="vxml.terminate">
|
|
<var name="immediate" expr="event$.immediate" />
|
|
<dialogterminate dialogid="vxml_dialogid" immediate="immediate" />
|
|
</transition>
|
|
|
|
|
|
<!-- ===================================================== -->
|
|
<!-- SIP events -->
|
|
<!-- ===================================================== -->
|
|
|
|
<!-- SIP (INTERNAL): connection to phone completed -->
|
|
<transition event="connection.connected">
|
|
<send target="_ccxml.createResponse(interactionId, requestID, session.id)"
|
|
targettype="SCXML" name="'ccxml.external'" />
|
|
</transition>
|
|
|
|
<!-- SIP (INTERNAL): disconnected -->
|
|
<transition event="connection.disconnected">
|
|
<send target="_ccxml.clearContextRequest(interactionId)"
|
|
targettype="SCXML" name="'ccxml.external'" />
|
|
<send target="session.id" targettype="'ccxml'" name="'this.exit'" />
|
|
</transition>
|
|
|
|
<!-- SIP (INTERNAL): reject call from SIP phone -->
|
|
<transition event="connection.alerting">
|
|
<reject />
|
|
</transition>
|
|
|
|
|
|
<!-- ===================================================== -->
|
|
<!-- VXML events -->
|
|
<!-- ===================================================== -->
|
|
|
|
<!-- VXML (INTERNAL): when VXML is started, it throws this internal event -->
|
|
<transition event="dialog.started">
|
|
<send target="_vxml.startResponse(interactionId, requestID, vxml_dialogid)"
|
|
targettype="SCXML" name="'ccxml.external'" />
|
|
</transition>
|
|
|
|
<!-- VXML (INTERNAL): an exit in VXML throws this internal event
|
|
if it was not just a prompt, get EMMA from VXML and send response to SCXML -->
|
|
<transition event="dialog.exit">
|
|
<assign name="vxml_running" expr="false" />
|
|
<!-- if a VXML dialog was terminated as a cause of dialogstart until a dialog was running,
|
|
start new dialog now (for that case we have the global _vxml.start()-parameters) -->
|
|
<if cond="vxml_terminating == false">
|
|
<send target="_vxml.doneNotification(interactionId, event$.values.emma, vxml_dialogid)"
|
|
targettype="SCXML" name="'ccxml.external'" />
|
|
<else />
|
|
<send target="_vxml.doneNotification(interactionId, '', vxml_dialogid)"
|
|
targettype="SCXML" name="'ccxml.external'" />
|
|
<assign name="vxml_running" expr="true" />
|
|
<var name="sessionid" expr="event$.sessionid" />
|
|
<dialogstart src="_vxml.start(grammarUri, prompt, audio, fields)"
|
|
dialogid="vxml_dialogid" connectionid="connectionId" namelist="sessionid" />
|
|
<assign name="vxml_terminating" expr="false" />
|
|
</if>
|
|
</transition>
|
|
|
|
|
|
<!-- ===================================================== -->
|
|
<!-- error events -->
|
|
<!-- ===================================================== -->
|
|
|
|
<!-- all errors (INTERNAL) -->
|
|
<transition event="error.*">
|
|
<log expr="'CCXML error'" />
|
|
<!--
|
|
<send target="_vxml.sendEvent(interactionId, _vxml.ERROR)"
|
|
targettype="SCXML" name="'ccxml.external'" />
|
|
-->
|
|
</transition>
|
|
|
|
</eventprocessor>
|
|
|
|
</ccxml>
|
|
</pre>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="div2">
|
|
<h3><a id="captureColor.vxml" name=
|
|
"captureColor.vxml"></a>5.4 captureColor.vxml</h3>
|
|
|
|
<p>As mentioned in <a href="#VUI-MC"><b>3.3 Voice Modality
|
|
Component</b></a> VoiceXML must return to CCXML (and hence
|
|
exit the VoiceXML dialog) to return results (e.g.
|
|
recognition results). Therefore the VoiceXML user
|
|
interfaces has to be implemented as small independent
|
|
scripts. Each script corresponds to a single action, like
|
|
<em>play a prompt</em> or <em>start grammar and listen to
|
|
user input</em>.</p>
|
|
|
|
<p>The following code sample shows how the
|
|
captureColor.vxml document could look like. The script
|
|
<em>vxml_emma.js</em>, which is referenced in the VoiceXML
|
|
document, contains a collection of auxiliary ECMAScript
|
|
functions to create an <a href="#EMMA">[EMMA]</a>
|
|
representation of the user input. See Appendix C of
|
|
<a href="#EMMA">[EMMA]</a> for more information of how to
|
|
map a recognition result into an EMMA representation.</p>
|
|
|
|
<p>Note that the other VoiceXML documents are very similar
|
|
and therefore not shown here.</p>
|
|
|
|
<div class="exampleInner">
|
|
<pre>
|
|
<?xml version="1.0" encoding="UTF-8"?>
|
|
<vxml xmlns:vxml="http://www.w3.org/2001/vxml" version="2.1">
|
|
<!-- We assume that there is a script library to convert the
|
|
recognition result into an EMMA string
|
|
(see http://www.w3.org/TR/emma) -->
|
|
<script src="vxml_emma.js" />
|
|
<form>
|
|
<field name="color">
|
|
<prompt>Which color?</prompt>
|
|
<option>red</option>
|
|
<option>blue</option>
|
|
<option>green</option>
|
|
<filled>
|
|
<!-- generate EMMA string from recognition result -->
|
|
<var name="emma" expr="createEmma(application.lastresult$)"/>
|
|
<!-- exit back to CCXML and return the recognized result -->
|
|
<exit namelist="emma"/>
|
|
</filled>
|
|
</field>
|
|
<catch event="help nomatch noinput">
|
|
Your options are <enumerate/>
|
|
</catch>
|
|
</form>
|
|
</vxml>
|
|
</pre>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
|
|
<div class="back">
|
|
<div class="div1">
|
|
<h2><a id="acknowledgments" name="acknowledgments"></a>A
|
|
Acknowledgments</h2>
|
|
|
|
<p>The editor wishes to thank Jim Larson (Intervoice) from
|
|
the Voice Browser Working Group for his contributions to the
|
|
writing of this document and for many helpful comments.</p>
|
|
</div>
|
|
|
|
<div class="div1">
|
|
<h2><a id="references" name="references"></a>B
|
|
References</h2>
|
|
|
|
<dl>
|
|
<dt class="label"><a id="apacheTomcat" name=
|
|
"apacheTomcat"></a>Apache Tomcat</dt>
|
|
|
|
<dd><a href="http://tomcat.apache.org"><cite>Apache Tomcat
|
|
servlet container</cite></a>, The Apache Software
|
|
Foundation, 2008.</dd>
|
|
|
|
<dt class="label"><a id="commonsSCXML" name=
|
|
"commonsSCXML"></a>Apache Commons SCXML</dt>
|
|
|
|
<dd><a href="http://commons.apache.org/scxml"><cite>A Java
|
|
SCXML engine</cite></a>, The Apache Software Foundation,
|
|
2008.</dd>
|
|
|
|
<dt class="label"><a id="CCXML" name="CCXML"></a>CCXML</dt>
|
|
|
|
<dd><a href="http://www.w3.org/TR/ccxml/"><cite>"Voice
|
|
Browser Call Control: CCXML Version 1.0 (Working
|
|
Draft)"</cite></a> , R.J. Auburn, editor. CCXML, or the
|
|
Call Control eXtensible Markup Language, is designed to
|
|
provide telephony call control support for dialog systems,
|
|
such as VoiceXML. World Wide Web Consortium, 2007.</dd>
|
|
|
|
<dt class="label"><a id="COMET" name="COMET"></a>COMET</dt>
|
|
|
|
<dd><a href=
|
|
"http://en.wikipedia.org/wiki/Comet_(programming)"><cite>Comet
|
|
Ajax server-push</cite></a>, Wikipedia.</dd>
|
|
|
|
<dt class="label"><a id="EMMA" name="EMMA"></a>EMMA</dt>
|
|
|
|
<dd><a href="http://www.w3.org/TR/emma/"><cite>"Extensible
|
|
multimodal Annotation markup language (EMMA)"</cite></a>,
|
|
Michael Johnson et al. editors. EMMA is an XML format for
|
|
annotating application specific interpretations of user
|
|
input with information such as confidence scores, time
|
|
stamps, input modality and alternative recognition
|
|
hypotheses. World Wide Web Consortium, 2007.</dd>
|
|
|
|
<dt class="label"><a id="Communicator" name=
|
|
"Communicator"></a>Galaxy</dt>
|
|
|
|
<dd><a href=
|
|
"http://communicator.sourceforge.net/"><cite>"Galaxy
|
|
Communicator"</cite></a> , Galaxy Communicator is an open
|
|
source hub and spoke architecture for constructing dialogue
|
|
systems that was developed with funding from Defense
|
|
Advanced Research Projects Agency (DARPA) of the United
|
|
States Government.</dd>
|
|
|
|
<dt class="label"><a id="MMI-ARCH" name=
|
|
"MMI-ARCH"></a>MMI-ARCH</dt>
|
|
|
|
<dd><a href=
|
|
"http://www.w3.org/TR/2008/WD-mmi-arch-20080414/"><cite>"Multimodal
|
|
Architecture and Interfaces (Working Draft)"</cite></a> ,
|
|
Jim Barnett et al. editors. This specification describes a
|
|
loosely coupled architecture for multimodal user
|
|
interfaces, which allows for co-resident and distributed
|
|
implementations, and focuses on the role of markup and
|
|
scripting, and the use of well defined interfaces between
|
|
its constituents. World Wide Web Consortium, 2008.</dd>
|
|
|
|
<dt class="label"><a id="MMIF" name="MMIF"></a>MMIF</dt>
|
|
|
|
<dd><a href=
|
|
"http://www.w3.org/TR/mmi-framework/"><cite>"W3C Multimodal
|
|
Interaction Framework"</cite></a> , James A. Larson, T.V.
|
|
Raman and Dave Raggett, editors, World Wide Web Consortium,
|
|
2003.</dd>
|
|
|
|
<dt class="label"><a id="MMIUse" name=
|
|
"MMIUse"></a>MMIUse</dt>
|
|
|
|
<dd><a href=
|
|
"http://www.w3.org/TR/mmi-use-cases/#driving-dir"><cite>"W3C
|
|
Multimodal Interaction Use Cases"</cite></a>, Emily Candell
|
|
and Dave Raggett, editors, World Wide Web Consortium,
|
|
2002.</dd>
|
|
|
|
<dt class="label"><a id="SCXML" name="SCXML"></a>SCXML</dt>
|
|
|
|
<dd><a href=
|
|
"http://www.w3.org/TR/2008/WD-scxml-20080516/"><cite>"State
|
|
Chart XML (SCXML): State Machine Notation for Control
|
|
Abstraction (Working Draft)"</cite></a> , Jim Barnett et
|
|
al. editors. SCXML, or the "State Chart extensible Markup
|
|
Language", provides a generic state-machine based execution
|
|
environment based on CCXML and Harel State Tables. World
|
|
Wide Web Consortium, 2008.</dd>
|
|
|
|
<dt class="label"><a id="VoiceXML" name=
|
|
"VoiceXML"></a>VoiceXML</dt>
|
|
|
|
<dd><a href="http://www.w3.org/TR/voicexml21/"><cite>"Voice
|
|
Extensible Markup Language (VoiceXML) Version
|
|
2.1"</cite></a> , Matt Oshry et al. editors. VoiceXML, the
|
|
Voice Extensible Markup Language, is designed for creating
|
|
audio dialogs that feature synthesized speech, digitized
|
|
audio, recognition of spoken and DTMF key input, recording
|
|
of spoken input, telephony, and mixed initiative
|
|
conversations. Its major goal is to bring the advantages of
|
|
Web-based development and content delivery to interactive
|
|
voice response applications. World Wide Web Consortium,
|
|
2007.</dd>
|
|
|
|
<dt class="label"><a id="XHTML" name="XHTML"></a>XHTML</dt>
|
|
|
|
<dd><a href="http://www.w3.org/TR/xhtml1/"><cite>" XHTML
|
|
1.0 The Extensible HyperText Markup Language (Second
|
|
Edition)"</cite></a> , Steven Pemberton et al. editors.
|
|
World Wide Web Consortium, 2004.</dd>
|
|
</dl>
|
|
|
|
<dl>
|
|
<dt class="label"><a id="XMLHttpRequest" name=
|
|
"XMLHttpRequest"></a>XMLHttpRequest</dt>
|
|
|
|
<dd><a href=
|
|
"http://www.w3.org/TR/XMLHttpRequest/"><cite>The
|
|
XMLHttpRequest Object (Working Draft)</cite></a> , Anne van
|
|
Kesteren. World Wide Web Consortium, 2008.</dd>
|
|
</dl>
|
|
</div>
|
|
</div>
|
|
</body>
|
|
</html>
|