Summary of the invention
The purpose of the present invention is to provide a kind of new multi-modal exchange method, ustomer premises access equipment, server and systems.
The purpose of the present invention is realized using following technical scheme.Multi-modal exchange method proposed according to the present invention, packetInclude following steps: one or more equipment ends receive the interaction request of one or more mode of user's input, by the interactionRequest is sent to server-side；The server-side receives the interaction request issued by the equipment end, is asked according to the interactionOne or more interactive commands are sought survival into, the interactive command is sent to the equipment end；In the interactive command at leastSome includes the information of interaction packet or the interaction packet；The equipment end receives and executes the interactive command, including dynamic addsThe interaction packet is carried to interact to realize with user.
The purpose of the present invention can also be further achieved by the following technical measures.
Multi-modal exchange method above-mentioned, the interaction packet include it is one or more based on the object of first language as theAn object and instruction for first object；Interaction packet is described in the dynamically load to realize that interacting with user includes: to adoptWith dynamically load mode and use bridge joint mode, according to first object and the instruction for first object comeSecond object of control and first object bridge joint；Wherein, second object is the equipment end local based on secondThe object of language.
Multi-modal exchange method above-mentioned receives one or more moulds of user's input in one or more of equipment endsBefore the step of interaction request of state, further includes: developed to obtain the interaction packet in advance at exploitation end, the interaction is wrappedIt is sent to the server-side；The server-side is received is wrapped by the interaction that the exploitation end uploads, and stores the interaction packet.
Multi-modal exchange method above-mentioned, second object include the one or more second based on the second languageAttribute and one or more second methods；It is described to be developed to obtain the interaction to include: to pass through bridge at exploitation end in advanceMode is connect, maps to obtain the attribute based on the first language as the first attribute, according to described according to second attributeTwo methods map to obtain the method based on the first language as first method, to obtain comprising first attribute and describedFirst method based on the bridge joint object of the first language as first object；By one or more described first pairsAs, script and resource data are packaged to obtain the interaction packet.
Multi-modal exchange method above-mentioned, the interaction packet include object data and script, and the object data includes usingIn adding the data of extension object for the equipment end, and/or for being updated for original object in the equipment endData, the script include the instruction for the instruction of the extension object, and/or for original object；The dynamicThe interaction is loaded to wrap to realize that interacting with user includes: the object for controlling one or more mode according to the interaction packet,It specifically includes according to the interaction packet and is updated original object, the addition extension object, to original objectThe one or more for being operated, updated original object is operated or the extension object being operated.
Multi-modal exchange method above-mentioned, the information of the interaction packet include the address of the interaction packet；The equipment endIt receives and executes the interactive command further include: before carrying out interaction packet described in the dynamically load, wrapped according to the interactionAddress obtain it is described interaction packet.
Multi-modal exchange method above-mentioned, at least some interactive commands include version number；Described according to the friendshipThe address mutually wrapped obtained before the step of interaction packet, further includes: is searched among the equipment end according to the version numberWhether the interaction packet of identical version number is had existed, if it does not exist, then carrying out the address according to the interaction packetObtain the interaction packet.
Multi-modal exchange method above-mentioned, one or more of equipment ends receive one or more mode of user's inputInteraction request, by the interaction request be sent to server-side include: using the equipment end dynamic engine receive user it is defeatedThe interaction request of the first mode entered, and the interaction request of the first mode is sent to by the service by interactive engineEnd, or using the equipment end interaction SDK receive user's input second mode interaction request, and by second mouldThe interaction request of state is sent to the server-side；It is described to include: according to the one or more interactive commands of interaction request generationThe server-side is judged according to the interaction request received, if the interaction request of the first mode, then generateThe interactive command for including the information of interaction packet or the interaction packet is as the first interactive command, if the second modeInteraction request then generates the interaction number that one or more mode of output are interacted comprising the interaction SDK for the equipment endAccording to and/or the interaction data address and one or more mode instruction the second interactive command.
Multi-modal exchange method above-mentioned, the equipment end receive and execute the interactive command further include: to receivingThe interactive command judged；If the interactive command is that have among first interactive command or the interactive commandAbout the part of the first mode, then interaction packet described in the dynamically load is carried out, is specifically included: utilizing the interactionEngine obtains and decompresses the interaction packet, using the dynamic engine dynamically load and parses the interaction and wraps and pass through instituteOne of equipment end described in dynamic engine calling or multiple modalities engine are stated to operate the object of corresponding mode；If the interactionOrder is the part having among second interactive command or the interactive command about the second mode, then will be describedSecond interactive command or part about the second mode are distributed to the interactive SDK, called by the interactive SDK described inOne of equipment end or multiple modalities engine interact to execute second interactive command to realize with user.
Multi-modal exchange method above-mentioned, it is described the described the step of interactive command is sent to the equipment end itBefore, further includes: one or more interactive commands are integrated into JavaScript object numbered musical notation format；It is described by the interactionIt includes: that the interactive command of the JavaScript object numbered musical notation format is sent to the equipment that order, which is sent to the equipment end,End.
The interactive command of multi-modal exchange method above-mentioned, the JavaScript object numbered musical notation format includes: for rememberingRecord the key field of the modality type of the interactive command and the value field of the content for recording the interactive command；Wherein,The value field include in the form of key-value pair record the version number, it is described interaction packet address, the interaction data and/orThe one or more of the address of the interaction data.
Also the following technical solution is employed for the purpose of the present invention to realize.Ustomer premises access equipment proposed according to the present invention, packetInclude: user inputs receiving module, the interaction request of one or more mode for receiving user's input, by the interaction requestIt is sent to server；Interactive command execution module, for receiving and executing the interactive command issued by the server；Wherein,At least some interactive commands include the information of interaction packet or the interaction packet；Wherein, the interactive command execution module packetDynamically load unit is included, is interacted with realizing with user for interaction packet described in dynamically load.
The purpose of the present invention can also be further achieved by the following technical measures.
Ustomer premises access equipment above-mentioned, the interaction packet is including one or more based on the object of first language as first pairAs and for first object instruction；The dynamically load unit is specifically used for: using dynamically load mode and usingBridge joint mode is controlled according to first object and the instruction for first object and is bridged with first objectThe second object；Wherein, second object is the object based on second language of the ustomer premises access equipment local.
Ustomer premises access equipment above-mentioned, it is described interaction packet include object data and script, the object data include for forThe ustomer premises access equipment addition extends the data of object, and/or for carrying out more for original object in the ustomer premises access equipmentNew data, the script include the instruction for the instruction of the extension object, and/or for original object；It is describedDynamically load unit is specifically used for: controlling the object of one or more mode according to the interaction packet, specifically includes according to instituteInteraction packet is stated to be updated original object, the addition extension object, be operated, original object to updateThe one or more that original object afterwards is operated or operated to the extension object.
Ustomer premises access equipment above-mentioned, the information of the interaction packet include the address of the interaction packet；The interactive command is heldRow module further includes interactive packet acquiring unit, is wrapped for obtaining the interaction according to the address of the interaction packet.
Ustomer premises access equipment above-mentioned, at least some interactive commands include version number；The ustomer premises access equipment further includesWhether version number's judging unit has existed identical version among the ustomer premises access equipment for searching according to the version numberNumber it is described interaction packet, if it does not exist, then notify the interactive packet acquiring unit carry out it is described according to it is described interaction packet groundLocation obtains the interaction packet.
Ustomer premises access equipment above-mentioned, it includes first mode input receiving unit and the second mould that the user, which inputs receiving module,State input receiving unit；The first mode input receiving unit is used to receive the first mode of user's input using dynamic engineInteraction request, and the interaction request of the first mode is sent to by the server by interactive engine, for the clothesDevice be engaged according to the interactive command for described in the generation of the interaction request of the first mode including interaction packet or the interactive information wrappedAs the first interactive command；The second mode input receiving unit is used to receive the second mould of user's input using interaction SDKThe interaction request of state, and the interaction request of the second mode is sent to the server, so that the server is according to instituteThe interaction request for stating second mode, which is generated, interacts the one or more of output comprising the interaction SDK for the ustomer premises access equipmentSecond interactive command of the instruction of the address and one or more mode of the interaction data of mode and/or the interaction data；The interactive command execution module further includes interactive command receiving unit, for receiving the interaction issued by the serverOrder, including receive first interactive command or second interactive command.
Ustomer premises access equipment above-mentioned, the interactive command execution module further include: judging unit, for the institute receivedIt states interactive command to be judged, if the interactive command is that have to close among first interactive command or the interactive commandIn the part of the first mode, then notify that the dynamically load unit is handled, if the interactive command is described secondThere is part about the second mode among interactive command or the interactive command, then notify interactive SDK execution unit intoRow processing；The dynamically load unit is specifically used for: the interaction packet is obtained and decompressed using the interactive engine, using describedDynamic engine dynamically load and parse it is described interaction packet and by one in ustomer premises access equipment described in the dynamic engine callingKind or multiple modalities engine operate the object of corresponding mode；The interactive SDK execution unit is used for: second interaction is orderedOrder or the part about the second mode are distributed to the interaction SDK of the ustomer premises access equipment, pass through the interactive SDK and call instituteOne of ustomer premises access equipment or multiple modalities engine are stated to execute second interactive command to realize and interact with user.
Ustomer premises access equipment above-mentioned, the interactive command execution module are specifically used for: receiving and execute JavaScript pairsAs the interactive command of numbered musical notation format；Wherein, the interactive command of the JavaScript object numbered musical notation format includes for recordingState the key field of the modality type of interactive command and the value field of the content for recording the interactive command；The value fieldAddress, the interaction data and/or the interactive number including the version number, the interaction packet that are recorded in the form of key-value pairAccording to address one or more.
Also the following technical solution is employed for the purpose of the present invention to realize.Server proposed according to the present invention, comprising: hand overMutual request receiving module, for receiving the interaction request of the one or more mode issued by one or more ustomer premises access equipments；Interactive command generation module, for generating one or more interactive commands according to the interaction request；Interactive command sending module,For the interactive command to be sent to the ustomer premises access equipment；Wherein, at least some interactive commands include interaction packet orThe information of the interaction packet.
The purpose of the present invention can also be further achieved by the following technical measures.
Server above-mentioned, the interaction packet include it is one or more based on the object of first language as the first object,With the instruction for being directed to first object；Wherein, first object is the bridge joint object of the second object, and second object isThe object based on second language of the ustomer premises access equipment local.
Server above-mentioned further includes interactive packet memory module, for receiving the interaction packet uploaded by exploitation end, is depositedStore up the interaction packet.
Server above-mentioned, the interaction packet include object data and script, and the object data includes for being describedData of ustomer premises access equipment addition extension object, and/or for being updated for original object in the ustomer premises access equipmentData, the script include the instruction for the instruction of the extension object, and/or for original object.
Server above-mentioned, the interaction request receiving module are specifically used for: what reception was issued by the ustomer premises access equipmentThe interaction request of first mode or the interaction request of second mode；Wherein, the interaction request of the first mode is by describedThe dynamic engine of ustomer premises access equipment and receive, second mode request is the interaction SDK by the ustomer premises access equipmentAnd it receives；The interactive command generation module includes the first interactive command generation unit, the second interactive command generation unitOne or more；The first interactive command generation unit is used to be requested according to the first mode, and generating described includes handing overMutually the interactive command of the information of packet or the interaction packet is as the first interactive command；The second interactive command generation unit is used forIt is requested according to the second mode, generates the one kind or more for interacting output comprising the interaction SDK for the ustomer premises access equipmentSecond interaction life of the instruction of the address and one or more mode of the interaction data and/or interaction data of kind modeIt enables.
Server above-mentioned further includes unified format module, for being integrated into one or more interactive commandsJavaScript object numbered musical notation format；The interactive command sending module is specifically used for the JavaScript object numbered musical notation latticeThe interactive command of formula is sent to the ustomer premises access equipment；Wherein, the interactive command packet of the JavaScript object numbered musical notation formatInclude the value word of the key field of the modality type for recording the interactive command and the content for recording the interactive commandSection；The value field includes the version number recorded in the form of key-value pair, the address of the interaction packet, the interaction dataAnd/or the one or more of the address of the interaction data.
Also the following technical solution is employed for the purpose of the present invention to realize.Multi-modal interaction system proposed according to the present inventionSystem, the system comprises at least one equipment ends and at least one server-side, realize the multi-modal interaction of any one above-mentionedMethod.
The present invention has obvious advantages and beneficial effects compared with the existing technology.By above-mentioned technical proposal, the present inventionIt is proposed multi-modal exchange method, ustomer premises access equipment, server and system at least have following advantages and the utility model has the advantages that
(1) present invention is by the Dynamic loading technique using dynamics engines such as JavaScript, the multimode that will locally developState interaction, which is wrapped, passes to server-side, and the instruction for being issued to equipment is the download address of this packet, and equipment can be from the download addressMulti-modal interaction packet is downloaded, the application range of dynamically load is extended to equipment end and server-side knot from the interactive mode of equipment endThe interactive mode of conjunction increases local exploitation cloud on the basis of multi-modal interaction has the two ways of pure local and pure cloudThe mode of end administration, is provided simultaneously with the advantage in local and cloud；
(2) interaction packet of the invention is to develop environment exploitation in local, is docked by way of bridge joint with local system, thusVersion updating of the extension of ability independent of SDK, convenient for extension, upgrade cost is low；
(3) present invention can get through different language, get through the running environment of different language by way of bridge joint, so that byThe interaction content that server-side issues is docked with the local system of equipment end；By taking GUI as an example, the present invention is enabled to by under server-sideThe GUI control of hair can be docked with a variety of local control systems at multiple equipment end, unified with local control system style；
(4) present invention is that equipment end extends object or the original object of update by utilizing interaction packet, can extend interactive energyPower.
The above description is only an overview of the technical scheme of the present invention, in order to better understand technological means of the invention, andIt can be implemented in accordance with the contents of the specification, and to allow above and other objects, features and advantages of the invention can be brighterShow understandable, it is special below to lift preferred embodiment, and cooperate attached drawing, detailed description are as follows.
Specific embodiment
It is of the invention to reach the technical means and efficacy that predetermined goal of the invention is taken further to illustrate, below in conjunction withAttached drawing and preferred embodiment, to the tool of multi-modal exchange method proposed according to the present invention, ustomer premises access equipment, server and systemBody embodiment, structure, feature and its effect, detailed description is as follows.
Mode of this paper, also referred to as interaction modalities, including speech modality, video modality, display mode etc..
Fig. 1 is the schematic flow chart of the existing multi-modal exchange method based on server-side.Referring to Fig. 1, equipment endInteraction SDK obtain user input, be uploaded to the craft services of server-side；The craft services of server-side be based on business and scene intoThe unified decision-making treatment of row, requested including being inputted according to the user received the service of the multiple modalities such as voice, video, display intoThe respective handlings such as row speech recognition, video processing, to respectively obtain the data such as voice flow, video flowing, image, display layout, thenThese data returned to the service of multiple modalities are uniformly processed, and then issue to the interactive engine of equipment end for equipmentHold the order and data exported to user；The interactive engine of equipment end explains order, and is distributed to interactive SDK；Interaction SDK is calledThe engines such as voice, video, display in equipment end system, interact with end user.
Wherein, the ability interacted with the external world that interactive system, interactive application provide is known as interaction capabilities or is skillEnergy.By taking sound-box device as an example, traditional speaker only has the input for receiving this mode of button operation and control audio broadcasting etc. and hands overMutual ability；Compared with traditional speaker, intelligent sound box has more abilities, such as is generally also equipped with the input for receiving speech modalityAbility, the control ability of other equipment such as desk lamp etc..
It may be noted that in the example depicted in fig. 1, what server-side issued is fixed protocol.Fixed protocol is also referred to as static associationView refers to that the content of the programs such as executable program, the library that exploitation is completed, format are fixed.Ability as defined in fixed protocol byEquipment end interaction SDK ability determines.Fixed protocol is difficult to play the local special of every kind of mode of every kind of equipment to consider versatilityOther ability, the extension of ability depend on the version updating of SDK.
Fig. 2 is the schematic flow block diagram of multi-modal exchange method one embodiment of the invention.Fig. 3 is of the invention moreThe schematic flow chart of another embodiment of mode exchange method.Please refer to Fig. 2 and Fig. 3, the exemplary multi-modal interaction side of the present inventionMethod mainly comprises the steps that
Step S11, one or more equipment ends receive the interaction request of one or more mode of user's input, by the friendshipMutually request is sent to server-side.It may be noted that the interaction request is during the user of human-computer interaction inputs to systemThe signal of transmitting, therefore the interaction request also referred to as interactively enters, equipment end is also referred to as client.
Step S12, server-side receive the interaction request issued by equipment end, are generated according to the interaction request one or moreThe interactive command is sent to equipment end by interactive command, so that equipment end interacts output according to the interactive command.It needs to infuseMeaning, which is in the system of human-computer interaction to the signal transmitted during user feedback, therefore the interactive commandReferred to as interaction output.Wherein, at least some of interactive command includes the information of interaction packet or interaction packet.Optionally, the interactionThe information of packet includes the download address of interaction packet.It optionally, further include right for recording interactive command institute among interactive commandThe field for the modality type answered.
Step S13, equipment end receive and execute the interactive command issued by server-side, interacted with realizing with user.Wherein,Equipment end executes interactive command and specifically includes: dynamically load interaction packet is interacted with realizing with user.
Wherein, the definition of dynamically load is that when program operation, computer system loading procedure itself is not deposited originallyExecutable file and run the code logic in these files.Optionally, it is wrapped using dynamic engine come dynamically load interactionIt is interacted with realizing with user.Optionally, the dynamic engine is by calling such as speech engine, video in equipment end system to drawIt holds up, one or more mode engines such as GUI engine (graphic user interface engine, alternatively referred to as display engine) are realized and userInteraction.
In some embodiments of the invention, interaction packet includes script, which includes the instruction for being issued to equipment end.It needsNote that the present invention does not limit the programming language of interaction packet, but usually scripting language, such as it can be JavaScript language(abbreviation JS), or it is also possible to Lua language.It is furthermore noted that the present invention does not limit the class of specifically used dynamic engineType, but be usually engine corresponding with the programming language of interaction packet.
It should be noted that " the interaction packet " shown in the present invention is different with " interacting SDK ".Interaction SDK is in equipmentThe module for executing interactive instruction is static is installed in equipment.And interaction packet of the invention is interaction content, and is under dynamicIt is dealt into equipment.A part that interaction packet can be considered as to SDK extends, and the advantage than SDK is that dynamically load dynamic updates.ExampleIt can such as be realized by dynamic engine and the dynamically load and dynamic of interaction packet are updated.For this purpose, interaction packet can also be known asSDK kinematic insert.
In an optional example, the programming language of interaction packet is JavaScript language, and dynamic engine isJavaScript engine, then dynamic load capability is the ability of JavaScript engine dynamic operation JavaScript script.ThingIn reality, existing mainstream JavaScript engine all has this ability.
Optionally, the input of client is that each equipment, every kind of mode are individually handled.All inputs upload to serviceEnd, the output that server-side carries out unified decision-making treatment and is packaged discharge device end.
The exemplary multi-modal exchange method of the present invention, by dynamically issuing technical ability, application using Dynamic loading technique, underHair be no longer fixed protocol but dynamic agreement, that is, the format of the interactive command issued, content be not it is fixed, in turnInteraction capabilities are extended and updated by dynamically issuing interaction packet.
In some embodiments of the invention, interaction packet includes object data and script.Further, interaction packet further includesResource data.The object data includes for adding the data of extension object for equipment end, and/or for for the original in equipment endThere are the data that object is updated.The script includes the instruction for the instruction of extension object, and/or for original object.It canSelection of land, the script further include the generic logic for subject instructions and pure mathematics calculating etc. built in dynamic engine.BeforeStating the dynamically load interaction in step S13 includes: the object that one or more mode are controlled according to interaction packet, is specifically includedIt is updated original object according to interaction packet, addition extension object, original object is operated, to updated original rightAs the one or more for being operated or being operated to extension object.Optionally, by call equipment end in wait operateObject corresponding mode engine operate the object.
The exemplary multi-modal exchange method of the present invention is that equipment end extends object or update is original right by utilizing interaction packetAs interaction capabilities can be extended.
It may be noted that although Fig. 3 illustrates only " speech engine, video engine, display engine ", it is also possible to include other classesThe mode engine of type.In fact, in the embodiment using interaction packet addition extension object of the invention, JavaScript engineIt may call upon engine corresponding with extension object, to operate to the extension object.
In some embodiments of the invention, the address comprising interaction packet in interactive command.Aforementioned step S13 is also wrappedIt includes: before carrying out dynamically load interaction packet, interaction packet being obtained according to the address of the interaction packet in interactive command.It is issued to equipmentInteractive command can be the download address of interaction packet, be also possible to completely interaction packet, but be usually the downloading of interaction packetLocation, in order to save flow.
Optionally, interactive command and downloading interaction packet are received and explained using the interactive engine of equipment end.
In some embodiments of the invention, at least some interactive commands include version number, which hands over for indicatingThe mutually version of packet, interactive command or protocol format.The step of interaction packet is downloaded in the address according to interaction packet in step s 13Before, the exemplary multi-modal exchange method of the present invention further include: equipment end local is searched according to the version number in interactive commandAmong whether have existed the interaction packet of identical version number, if there is no or version number it is different, then carry out above-mentioned according to friendshipThe address mutually wrapped obtains downloading interaction packet.It is alternatively possible to execute this step using the interactive engine of equipment end.The present invention showsThe multi-modal exchange method of example is convenient for subsequent modification or Extended Protocol, is improved by adding version number field in interactive commandCompatibility.
In some embodiments of the invention, interaction packet is based on first language.And aforementioned step S13 is specifically wrappedIt includes: using dynamically load mode and using bridge joint mode, to execute the interaction packet based on first language, to according to the friendshipIt mutually wraps to control the object based on second language of equipment end local, is interacted with realizing with user.Specifically, it can use supportThe engine of the first language of dynamically load and bridge joint realizes this step.In general, the ability of first language is by its engineIt provides, provides the ability got through with local second language by the engine of first language, first language can have secondThe ability that language has.In practice, the JavaScript engine of mainstream all has the bridging capability.
Wherein, bridge joint mode be one kind two objects can be linked together and the two between close couplingMethod gets up communication with one another by " bridge ", while allowing their respectively independent variations again.And the exemplary multi-modal friendship of the present inventionMutual method, realizes object extension in the way of bridge joint, can be object extension attribute, call directly nation method.
Further, in some embodiments of the invention, interaction packet includes one or more pair based on first languageAs the first object and for the instruction of the first object.Optionally, the first language should be also based on for the instruction of the first objectSpeech.Optionally, interaction packet further includes resource data.Dynamically load interaction packet in step S13 specifically includes: being added using dynamicLoad mode simultaneously uses bridge joint mode, according to the first object and for the instruction of the first object, to control and the first object bridge jointSecond object.Wherein, which is the object based on second language of equipment end local.It may be noted that the first object andTwo objects can be original object above-mentioned, or be also possible to extension object above-mentioned.
In some instances, by JavaScript engine come in fact now according to JavaScript script and bridge jointJavaScript object controls the object based on native language.Specifically JavaScript engine provides registration function,The user of JavaScript engine registers the binding relationship of JavaScript object and native object.JavaScript engine is logicalIt crosses mapping table and safeguards all binding relationships.Wherein, which can be equipment end or multi-modal interactive system.
It may be noted that the present invention is not limited in its kind for the second language of the development language as equipment end native object,Such as be the situation of JavaScript in first language, second language can be C++ or Java etc..As an optional example,For showing mode, the object above-mentioned based on first language is JavaScript object, the base of equipment end local above-mentionedIt can be the GUI control based on C++ or Java language of equipment end local in the object of second language.
The exemplary multi-modal exchange method of the present invention, different language can be got through in the way of bridge joint, gets through different languageRunning environment so that the interaction content issued by server-side is docked with the local system of equipment end.By taking GUI as an example, the present inventionEnable to the GUI control issued by server-side that can dock with a variety of local control systems at multiple equipment end, with local controlPart system style is unified.
In some embodiments of the invention, before step S11, the exemplary multi-modal exchange method of the present invention is also wrappedIt includes:
Step S21 is developed at exploitation end in advance, is obtained interactive packet, which is sent to server-side.It is optionalGround, the exploitation end can be personal computer (personal computer, the letter of such as desktop computer, notebook computerReferred to as PC) etc. development platforms.
Step S22, server-side is received is wrapped by the interaction that exploitation end uploads, and stores the interaction packet.It is alternatively possible to pass through byInteraction packet is sent to the storage service of server-side to be stored.Optionally, in the situation that first language is JavaScript,The storage service is specifically JavaScript script storage service.
In some embodiments of the invention, the second object include the second attribute of one or more based on second language withAnd one or more second methods.And aforementioned step S21 is specifically included:
Step S31 maps to obtain the corresponding attribute conduct based on first language according to the second attribute by way of bridge jointFirst attribute, according to second method map to obtain it is corresponding based on the method for first language as first method, to be includedThe bridge joint object of first attribute and the second object based on first language of the first method is as the first object；
The first object, script and the resource data of one or more bridge joint are packaged to be interacted by step S32Packet.
In some embodiments of the invention, such as in the situation that the second object is newly added extension object, first existExploitation end is developed to obtain second object, then carries out aforementioned step S31 and S32.
Optionally, the interaction packet of JavaScript script is to be generated completely by developing end.Server-side be merely responsible for storage andForwarding does not need to generate using the service of the multiple modalities such as the voice service of server-side, Video service, display serviceThe interaction packet of JavaScript script.
As a specific embodiment, it is directed to display mode and first language is the situation of JavaScript, it is above-mentionedSecond object is the GUI control of equipment end local, and the first object above-mentioned is JavaScript pair with GUI control bridge jointAs.Exploitation end develop GUI control, the GUI control include the resource datas such as layout attributes, picture resource and literal resource, withAnd exchange method.At exploitation end, by way of bridge joint, layout attributes is mapped as JavaScript object attribute of the same name, are incited somebody to actionExchange method is mapped as JavaScript object method of the same name.Again by the JavaScript object attribute and the JavaScriptObject method is encapsulated as JavaScript object.Then, to one or more bridge joint control objects, resource data andJavaScript script is packaged and is compressed to obtain JavaScript interaction packet.Finally by JavaScript interaction packet hairIt send to server-side, and is stored in the server-side.
The exemplary multi-modal exchange method of the present invention, by developing the object of bridge joint in advance, and in aforementioned step S13The object of the bridge joint is loaded and held using the dynamic engine with dynamically load dynamic updating ability and bridging capabilityRow, so that the first language also has the ability that second language has.
In some embodiments of the invention, as shown in figure 3, interactive engine is used for: obtaining interaction packet, decompression obtains objectData, JavaScript script and resource data.JavaScript engine is used for: parsing and execution JavaScript script.If there is operating the object of some mode in JavaScript script, then the method for the mode engine is called.Mode engine isThe actuator of second object, the second object are the contents being executed.
In some embodiments of the invention, can simultaneously a variety of different mode be carried out with the interaction of different modes.
Specifically, aforementioned step S11 specifically includes one or more below:
The interaction request of the first mode of user's input is received using the dynamic engine of the first language of equipment end, and is passed throughThe interaction request of first mode is sent to server-side by interactive engine；
The interaction request of the second mode of user's input is received using the interaction SDK of equipment end, and by the friendship of second modeMutually request is sent to server-side.
Optionally, dynamic engine provides registration monitoring mechanism for developer, determines which input will be by dynamic by developerEngine processing.For example, based on scene or interaction modalities type can be based on, to select the user handled by dynamic engine to input.
The step of server-side in aforementioned step S12 generates one or more interactive commands according to interaction request is specifically wrappedIt includes:
Server-side is judged according to the interaction request received；
If what is received is the interaction request of first mode, generates and output is interacted by dynamic engine for equipment endIt is above-mentioned include interaction packet or the interaction packet information interactive command, as the first interactive command；Optionally, according to firstThe interaction request of mode simultaneously generates first interactive command by requesting storage service above-mentioned；
If what is received is the interaction request of second mode, the second interactive command is generated, wherein second interactive commandComprising for equipment end by interaction SDK interact one or more interaction datas of output and/or the address of interaction data, withAnd the instruction of one or more mode；Optionally, requested according to the interaction request of second mode and by craft services voice,One or more mode services such as video, display, to obtain one or more friendships such as voice flow, video flowing, image, display layoutMutual data, to obtain second interactive command.Optionally, which is the interactive command based on fixed protocol.
Further, aforementioned step S13 further include:
Equipment end judges the interactive command received, including judging that the interactive command received is the first interaction lifeEnable or the second interactive command, or the interactive command that receives of judgement among whether about first mode part orThe no part about second mode；It is alternatively possible to carry out the judgement using the interactive engine of equipment end；
If the interactive command received is that have among the first interactive command or the interactive command received about the first mouldThe part of state then carries out above-mentioned wrap according to the interactive address downloading interaction wrapped and arrives locally, using dynamic engine come dynamically loadInteraction packet is interacted with realizing with user, is specifically included: being obtained using the interactive engine of equipment end and decompress interaction packet, utilizes equipmentThe dynamic engine dynamically load at end simultaneously parses interactive wrap and by one of dynamic engine calling equipment end or multiple modalitiesEngine operates the object of corresponding mode；
If the interactive command received is that have among the second interactive command or the interactive command received about the second mouldThe part of state then should be distributed to second interactive command or about the part of second mode the interaction SDK of equipment end, by thisInteraction SDK call equipment end system in such as speech engine, video engine, (graphic user interface engine, can also for GUI engineReferred to as display engine) etc. one or more mode engines interacted to execute the second interactive command to realize with user.
In some embodiments of the invention, the step that interactive command is sent to equipment end in aforementioned step S12Before rapid, further includes: one or more interactive commands are integrated into the interactive command of unified format by server-side.Optionally, it will closeIn one or more interactive commands of one or more mode or by the first interactive command and the second interactive command, it is integrated into systemThe interactive command of one format.And interactive command be sent to equipment end specifically include in step S12: the unification that integration is obtainedThe interactive command of format is sent to equipment end.As an alternative embodiment, it is JavaScript object numbered musical notation that this, which unifies format,(JavaScript Object Notation, referred to as JSON) format.Specifically, the friendship of JavaScript object numbered musical notation formatMutually order is key-value pair form (or being key-value form), the key including the modality type for recording interactive orderThe value field of field and the content for recording interactive order.Further, which includes being recorded in the form of key-value pair: version number, interaction packet address, interaction data and/or interaction data address one or more.
It may be noted that JSON is a kind of the recognized standard data format, the overwhelming majority such as C++, java, Lua programming language is allSupport JSON, and the parsing library of JSON is also very much, C++, java have the library JSON of many prevalences, real convenient for interactive engineIt is existing.
As a specific example, a kind of interactive command of JSON reference format proposed by the present invention is as follows:
{
"type":data,
"type":data,
……
}。
Wherein, " type ": data is a key-value pair, for indicating an interactive command.Type therein is key field,For modality type (titles of interaction modalities) of recording interactive order, such as speech modality, video modality, display mode etc..Optionally, type is string format.Data is value field, the data needed for recording interactive.Optionally, data is alsoJSON format, including one or more key-value pairs, a key-value pair in data are used to record the type and specifically of one dataValue.
For example, a kind of format of the interactive command about display mode are as follows:
Wherein, display field is the key field of this order, and expression is handling display command.Square brackets " [] " itMiddle content be all this order value field.The value field entirely ordered includes multiple key-value pairs again: this key assignments of versionTo expression version number；This key-value pair of width is display width, this key-value pair of height is display height, this is for sideJust equipment end display system does the adaptation of screen layout；This key-value pair of url is the download address of GUI interaction packet.
For another example, a kind of format of the interactive command about speech modality are as follows:
Wherein, voice field expression is handling voice command；Version key-value pair indicates version number；Text key-value pairIt is speech text；Url key-value pair is the download address of voice document.It may be noted that the voice document is different from the speech text, speech text is the text for recording the content of voice flow, and voice document is the voice flow of speech text.
It may be noted that the interactive command about display mode is the first interactive command above-mentioned, wherein remembering in aforementioned exemplaryThe download address of interactive packet is recorded；Interactive command about speech modality is the second interactive command above-mentioned, wherein what is recorded isThe particular content of interaction data or the address of interaction data.
The exemplary multi-modal exchange method of the present invention, by the way that interactive command is integrated into JSON format, convenient for extension mode,The data that extension interaction needs.
It may be noted that it is a kind of preferred embodiment of the invention that interactive command, which is integrated into JSON format only, the present invention is notIt must include the steps that integrating interactive command, and the format integrated also is not necessarily JSON format.Such as it can also be withIt is realized with fixed bid value: for example indicating that equipment carries out different function using fixed integer, for example 1 representative is broadcastIt puts the music on, 2 represent bright light etc..
It may be noted that the type at the present invention not limiting device end and server-side.Optionally, server-side above-mentioned is server.Equipment end above-mentioned is ustomer premises access equipment, including smart phone, intelligent sound box, set-top box, smart glasses, smart watches, movementComputer etc. has the product of multi-modal interaction.
Optionally, multi-modal exchange method proposed by the present invention can use application program (Application, referred to as) etc. APP softwares are implemented.Equipment end and server-side are equipped with for realizing the soft of multi-modal exchange method proposed by the present inventionPart, so as to realize each step shown in previous embodiment using the equipment end and server-side.
In some embodiments of the invention, it can use the interaction that web mode realizes GUI mode.Specifically, of the inventionInteraction is realized by being embedded in webview (network view) in interaction SDK.Wherein, which it is clear to can be regarded as the pageLook at device.
It may be noted that web page (also referred to as webpage) is generally made of html+css language, page browser is to show webThe engine of the page.Html language is supported to be embedded in JavaScript language script with<script>label.JavaScript script fortuneIt goes in browser sandbox, the energy that can be provided by the JavaScript object built in operation browser come access browserPower, such as action pane.
Fig. 4 is the schematic diagram of ustomer premises access equipment 100 provided by one embodiment of the present invention.The ustomer premises access equipment100 can be smart phone, intelligent sound box, set-top box, smart glasses, smart watches, removable computer etc. with multi-modal interactionProduct.Fig. 5 is the schematic diagram of server 200 provided by one embodiment of the present invention.
Fig. 4 and Fig. 5 are please referred to, in some embodiments, the exemplary ustomer premises access equipment 100 of the present invention mainly includes that user is defeatedEnter receiving module 110 and interactive command execution module 120.
Wherein, user inputs the interaction request that receiving module 110 is used to receive one or more mode of user's input, willThe interaction request is sent to server 200.
Interactive command execution module 120 is for receiving and executing the interactive command issued by server 200.Wherein, at leastSome interactive commands include the information of interaction packet or interaction packet.
Further, which includes dynamically load unit (not shown).The dynamically loadUnit is interacted with realizing with user for dynamically load interaction packet.
In some embodiments of the invention, interaction packet includes object data and script.Further, interaction packet further includesResource data.The object data includes for adding the data of extension object for ustomer premises access equipment 100, and/or for for userThe data that original object in end equipment 100 is updated.The script includes for the instruction of extension object, and/or for originalThere is the instruction of object.Optionally, which further includes for the subject instructions built in dynamic engine and such as pure mathematics calculatingDeng generic logic.Dynamically load unit above-mentioned is specifically used for: the object of one or more mode is controlled according to interaction packet,Specifically include according to interaction packet come be updated original object, addition extension object, original object is operated, to update afterOriginal object operated or to the one or more that is operated of extension object.Optionally, by calling ustomer premises access equipmentMode engine corresponding with object to be operated in 100 operates the object.
In some embodiments of the invention, the information of interaction packet above-mentioned includes the address of interaction packet.Interactive command is heldRow module 120 further includes interactive packet acquiring unit, for obtaining interaction packet according to the address of interaction packet.
In some embodiments of the invention, at least some interactive commands include version number.Ustomer premises access equipment 100 further includesVersion number's judging unit, is used for: searching whether have existed identical version number among ustomer premises access equipment 100 according to version numberInteraction packet, if there is no or version number it is different, then notify interactive packet acquiring unit above-mentioned to carry out the address according to interaction packetObtain interaction packet.
In some embodiments of the invention, interaction packet is based on first language.Dynamically load unit above-mentioned is specificFor: dynamic engine is utilized, using dynamically load mode and bridge joint mode is used, to execute interactive packet, to according to the interactionIt wraps to control the local object based on second language of ustomer premises access equipment 100.Wherein, the dynamic engine be support dynamically load andThe engine of the first language of bridge joint.
Further, in some embodiments of the invention, interaction packet includes one or more pair based on first languageAs the first object and for the instruction of the first object.Optionally, the first language should be also based on for the instruction of the first objectSpeech.Optionally, interaction packet further includes resource data.Dynamically load unit above-mentioned is specifically used for: using dynamically load mode,And bridge joint mode is used, according to the first object and for the instruction of the first object, to control second with first object bridge jointObject.Wherein, the second object is the local object based on second language of ustomer premises access equipment 100.
In some embodiments of the invention, user input receiving module 110 include first mode input receiving unit andSecond mode input receiving unit.The first mode input receiving unit is used for: receiving the of user's input using dynamic engineThe interaction request of one mode, and the interaction request of the first mode is sent to by server 200 by interactive engine, for serviceDevice 200 generates the interactive command of the information above-mentioned for including interaction packet or interaction packet according to the interaction request of the first mode, makeesFor the first interactive command.The second mode input receiving unit is used for: receiving the second mode of user's input using interaction SDKInteraction request, and the interaction request of the second mode is sent to server 200, so that server 200 is according to the second modeInteraction request generate the second interactive command.Wherein, second interactive command include for ustomer premises access equipment 100 interaction SDK intoThe interaction data of one or more mode of row interaction output and/or the address of interaction data and one or more modeInstruction.Interactive command execution module 120 further includes interactive command receiving unit, is used for: receiving the interaction issued by server 200Order, including receive the first interactive command or the second interactive command above-mentioned.
In some embodiments of the invention, interactive command execution module 120 further includes that judging unit and interaction SDK are executedUnit.
The judging unit is used for: being judged the interactive command received, is specifically included the interaction life that judgement receivesWhether order is the first interactive command or the second interactive command or judges among the interactive command received about first modePart or whether about second mode part；If the interactive command received is the first interactive command or interactive commandAmong there is part about first mode, then notify that interactive packet acquiring unit and/or dynamically load unit are handled；If connecingThe interactive command received is the part having among the second interactive command or interactive command about second mode, then notifies to interactSDK execution unit is handled.
In some embodiments of the invention, interaction packet acquiring unit is specifically used for: utilizing the interaction of ustomer premises access equipment 100Engine obtains and decompresses interactive packet.Dynamically load unit is specifically used for: utilizing the dynamic engine dynamically load of ustomer premises access equipment 100And interactive packet is parsed, and operate by one of dynamic engine calling ustomer premises access equipment 100 or multiple modalities engine pairAnswer the object of mode.
Interaction SDK execution unit is used for: the second interactive command or the part about second mode are distributed to user terminalThe interaction SDK of equipment 100 calls one of ustomer premises access equipment 100 or multiple modalities engine to execute second by interaction SDKInteractive command is interacted with realizing with user.
In some embodiments of the invention, interactive command execution module 120 is specifically used for: receiving, explains and executesThe interactive command of JavaScript object numbered musical notation format.
As an alternative embodiment, the interactive command of JavaScript object numbered musical notation format includes: for recording interactiveThe value field of the key field of the modality type of order and the content for recording interactive order.Wherein, the value field include withKey-value pair form record version number, interaction packet address, interaction data and/or interaction data address one or more.
Fig. 4 and Fig. 5 are please referred to, in some embodiments, the exemplary server 200 of the present invention mainly connects including interaction requestReceive module 210, interactive command generation module 220 and interactive command sending module 230.
Wherein, interaction request receiving module 210 be used to receive the one kind issued by one or more ustomer premises access equipments 100 orThe interaction request of multiple modalities.
Interactive command generation module 220 is used to generate one or more interactive commands according to interaction request.Wherein, at least oneA little interactive commands include the information of interaction packet or interaction packet.Optionally, the information of the interaction packet includes the download address of interaction packet.
Interactive command sending module 230 is used to interactive command being sent to ustomer premises access equipment 100.
In some embodiments of the invention, interaction packet includes object data and script.Further, interaction packet further includesResource data.The object data includes for adding the data of extension object for ustomer premises access equipment 100, and/or for for userThe data that original object in end equipment 100 is updated.The script includes for the instruction of extension object, and/or for originalThere is the instruction of object.Optionally, which further includes for the subject instructions built in dynamic engine and such as pure mathematics calculatingDeng generic logic.
In some embodiments of the invention, at least some interactive commands include version number.
In some embodiments of the invention, server 200 further includes interactive packet memory module.The interaction packet memory moduleAlso referred to as storage service stores interaction packet for receiving the interaction packet uploaded by exploitation end.
In some embodiments of the invention, interaction packet is to be based on first based on first language, including one or moreThe object of language is as the first object and for the instruction of the first object.It optionally, should be also base for the instruction of the first objectIn first language.Wherein, which is the bridge joint object of the second object.Second object is ustomer premises access equipment 100The object based on second language on ground.
In some embodiments of the invention, interaction request receiving module 210 is specifically used for: receiving by ustomer premises access equipmentThe interaction request of 100 first modes issued or the interaction request of second mode.Wherein, the interaction request of first mode is to pass throughThe dynamic engine of ustomer premises access equipment 100 and receive user input interaction request, second mode request be to pass through user terminalThe interaction SDK of equipment 100 and receive user input interaction request.
Interactive command generation module 220 include the first interactive command generation unit, the second interactive command generation unit oneIt is a or multiple.It includes interaction packet or interaction that the first interactive command generation unit, which is used to be generated according to first mode request above-mentioned,The interactive command of the information of packet is as the first interactive command.The second interactive command generation unit according to second mode for requestingGenerate the second interactive command.Wherein, which interacts output comprising the interaction SDK for ustomer premises access equipment 100The interaction data of one or more mode and/or the address of interaction data and one or more mode instruction.
In some embodiments of the invention, server 200 further includes unified format module.This unifies format module useIn: one or more interactive commands are integrated into the interactive command of unified format.It optionally, will be about one or more modeOne or more interactive commands or by the first interactive command and the second interactive command, are integrated into the interactive command of unified format.MakeFor an alternative embodiment, it is JavaScript object numbered musical notation format that this, which unifies format,.Interactive command sending module 230 is specifically usedIn the interactive command for unifying format is sent to ustomer premises access equipment 100.
As an alternative embodiment, the interactive command of JavaScript object numbered musical notation format includes: for recording interactiveThe value field of the key field of the modality type of order and the content for recording interactive order.Wherein, the value field include withKey-value pair form record version number, interaction packet address, interaction data and/or interaction data address one or more.
Fig. 6 is the schematic diagram of 300 one embodiment of multi-modal interactive system of the invention.Referring to Fig. 6, this hairBright exemplary multi-modal interactive system 300 includes mainly at least one equipment end and at least one server-side, realizes above-mentionedMulti-modal exchange method shown in any one embodiment.
The above described is only a preferred embodiment of the present invention, limitation in any form not is done to the present invention, thoughSo the present invention has been disclosed as a preferred embodiment, and however, it is not intended to limit the invention, any technology people for being familiar with this professionMember, without departing from the scope of the present invention, when the technology contents using the disclosure above are modified or are modifiedFor the equivalent embodiment of equivalent variations, but anything that does not depart from the technical scheme of the invention content, according to the technical essence of the inventionAny simple modification, equivalent change and modification made to the above embodiment, all of which are still within the scope of the technical scheme of the invention.