CN112230779A

Movatterモバイル変換

Info

Publication number: CN112230779A
Application number: CN202011208255.9A
Authority: CN
Inventors: 顾正明; 庄光庭
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2020-11-03
Filing date: 2020-11-03
Publication date: 2021-01-15
Anticipated expiration: 2040-11-03
Also published as: CN112230779B

Abstract

Translated fromChinese

本申请实施例公开了一种操作响应方法、装置、设备及存储介质，属于人机交互技术领域。该方法包括：响应于上报的硬件中断，获取目标传感器数据；将目标传感器数据输入操作识别模型，得到操作识别模型输出的操作识别结果；响应于操作识别结果指示目标操作不是误操作，对所述目标操作进行响应。本申请实施例中，在利用阈值划分的基础上进一步利用深度学习神经网络对传感器数据进行识别，提高了操作响应的准确率，避免对误操作进行响应，并且无需采用相关技术中提高阈值的方式提高响应操作的准确率，在保证准确率的基础上扩大了目标操作的识别范围，避免将正常操作识别为误操作。

The embodiments of the present application disclose an operation response method, apparatus, device, and storage medium, which belong to the technical field of human-computer interaction. The method includes: acquiring target sensor data in response to a reported hardware interrupt; inputting the target sensor data into an operation recognition model to obtain an operation recognition result output by the operation recognition model; The target action responds. In the embodiment of the present application, the deep learning neural network is further used to identify sensor data on the basis of threshold division, which improves the accuracy of the operation response, avoids responding to misoperations, and does not need to adopt the method of increasing the threshold in the related art. The accuracy of the response operation is improved, the recognition range of the target operation is expanded on the basis of ensuring the accuracy, and the normal operation is avoided to be identified as a misoperation.

Description

Operation response method, device, equipment and storage medium

Technical Field

The embodiment of the application relates to the technical field of human-computer interaction, in particular to an operation response method, device, equipment and storage medium.

Background

In order to improve convenience and attractiveness of wearable devices such as bluetooth headsets, Augmented Reality (AR) glasses, smartwatches and the like, the size of the wearable devices is smaller and smaller, and an independent control device or a function key is not usually arranged, so that developers need to set corresponding function operation of the wearable devices to simple trigger operation, and users can directly control the devices by separating from the control device or the key.

In the correlation technique, the operation that wearable equipment can receive mainly includes touch operation and knocking operation, to knocking operation, be provided with acceleration sensor in the wearable equipment usually for detect the shock of the wearable equipment that knocking operation leads to, simultaneously in order to improve the rate of recognition of knocking operation, prevent to remove, touch etc. maloperation respond, still be provided with the threshold of knocking in the wearable equipment, when equipment vibrates, if the slope that the displacement of equipment corresponds exceeds the threshold value, then the wearable equipment confirms that receives the knocking operation, otherwise do not respond.

However, in the related art, if the tap threshold is high, the operation recognition rate is decreased, that is, the normal tap operation may be shielded, and if the threshold is set to be low, the response is easily generated to the false touch operation of the user, and the accuracy and the recognition rate cannot be both considered.

Disclosure of Invention

The embodiment of the application provides an operation response method, an operation response device, operation response equipment and a storage medium. The technical scheme is as follows:

in one aspect, an embodiment of the present application provides an operation response method, where the method includes:

responding to reported hardware interruption, and acquiring target sensor data, wherein the target sensor data is sensor data acquired by a sensor in the wearable device before the hardware interruption, and the hardware interruption is reported when the sensor identifies target operation based on a threshold condition;

inputting the target sensor data into an operation recognition model to obtain an operation recognition result output by the operation recognition model, wherein the operation recognition model is a neural network obtained based on deep learning training, and the operation recognition result is used for indicating whether the target operation is misoperation or not;

responding to the target operation in response to the operation identification result indicating that the target operation is not a misoperation.

In one aspect, an embodiment of the present application provides an operation response apparatus, where the apparatus includes:

an obtaining module, configured to obtain target sensor data in response to a reported hardware interrupt, where the target sensor data is sensor data collected by a sensor in the wearable device before the hardware interrupt, and the target sensor data is reported when the sensor recognizes a target operation based on a threshold condition in the hardware interrupt;

the identification module is used for inputting the target sensor data into an operation identification model to obtain an operation identification result output by the operation identification model, the operation identification model is a neural network obtained based on deep learning training, and the operation identification result is used for indicating whether the target operation is misoperation or not;

and the first response module is used for responding to the target operation in response to the operation identification result indicating that the target operation is not misoperation.

In another aspect, an embodiment of the present application provides a wearable device, which includes a processor and a memory, where the memory stores at least one instruction, at least one program, a set of codes, or a set of instructions, and the at least one instruction, the at least one program, the set of codes, or the set of instructions is loaded and executed by the processor to implement the operation response method according to the above aspect.

In another aspect, embodiments of the present application provide a computer-readable storage medium, in which at least one instruction, at least one program, a code set, or a set of instructions is stored, and the at least one instruction, the at least one program, the code set, or the set of instructions is loaded and executed by a processor to implement the operation response method according to the above aspect.

According to an aspect of the application, a computer program product or computer program is provided, comprising computer instructions, the computer instructions being stored in a computer readable storage medium. The processor of the wearable device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the wearable device executes the operation response method provided in the various alternative implementations of the above aspect.

The technical scheme provided by the embodiment of the application has the beneficial effects that at least:

in the embodiment of the application, target sensor data before hardware interruption is identified through the operation identification model, and whether the operation causing the hardware interruption is misoperation or not is judged, so that the target operation is responded when the target operation is determined not to belong to the misoperation, the sensor data is further identified by utilizing the deep learning neural network on the basis of utilizing threshold division, the accuracy of operation response is improved, the misoperation is avoided being responded, the accuracy of response operation is improved without adopting a mode of improving the threshold in the related technology, the identification range of the target operation is expanded on the basis of ensuring the accuracy, and the normal operation is prevented from being identified as the misoperation.

Drawings

Fig. 1 is a schematic diagram of a headset provided by an exemplary embodiment of the present application;

FIG. 2 is a flow chart of an operation response method provided by an exemplary embodiment of the present application;

FIG. 3 is a flow chart of an operational response method provided by another exemplary embodiment of the present application;

FIG. 4 is a graph illustrating a slope change corresponding to a tapping operation provided by an exemplary embodiment of the present application;

FIG. 5 is a network architecture diagram of an operational recognition model provided by an exemplary embodiment of the present application;

FIG. 6 is a flow chart of an operational response method provided by another exemplary embodiment of the present application;

fig. 7 is a block diagram of an operation response device according to an exemplary embodiment of the present application;

fig. 8 is a block diagram of a wearable device according to an exemplary embodiment of the present application.

Detailed Description

To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

Reference herein to "a plurality" means two or more. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

In the related art, hardware is generally used to recognize a user operation, for example, for a tapping operation, an acceleration sensor is provided in a wearable device and is used to acquire a motion condition of the wearable device, and when sensor data corresponding to a target operation is detected, hardware interrupt is reported, so that a processor executes an instruction corresponding to the target operation. In order to improve the accuracy of the operation of identifying the target and avoid the response to the misoperation, a threshold value is set in the wearable device, and when the sensor data reaches the threshold value, the wearable device can identify the sensor data.

However, in the related art, if the threshold is set too high, the target operation recognition rate may be decreased, that is, normal operation may be blocked, and if the threshold is set too low, a response may be generated to a false touch operation of a user, and the accuracy and the report rate cannot be both considered.

In order to solve the problem that the accuracy rate of operation identification and the operation reporting rate cannot be improved simultaneously in the related art, the embodiment of the application provides an operation identification method, after target sensor data meeting a threshold condition are obtained, an operation identification model is used for identifying the target sensor data, whether the target operation is misoperation is further judged, and the accuracy rate of operation response can be improved without adjusting the threshold; and because the accuracy rate of the operation identification model for identifying the target operation is higher, the threshold value can be properly adjusted downwards, the report rate of the target operation is improved, and the shielding of part of normal operations is avoided.

The following embodiments are described taking as an example that the operation response method is used for wearable devices such as earphones, smart watches, AR glasses, and the like. As shown in fig. 1, a schematic diagram of awireless headset 100 is shown. Thewireless earphone 100 has sensors disposed in the

bodies

101 and 102 for collecting sensor data, and the operation performed by the user on the earphone may cause the sensor data to change regularly, for example, a tap operation causes thewireless earphone 100 to vibrate, and the acceleration collected by the acceleration sensor also changes correspondingly, and theearphone 100 reports hardware interruption when the sensor data meets a threshold condition, further identifies the target sensor data by using an operation identification model, and determines whether the target operation is a misoperation, so as to respond to the target operation when the target operation does not belong to the misoperation.

Fig. 2 shows a flowchart of an operation response method provided in an exemplary embodiment of the present application. In this embodiment, the method is described by taking the method as an example for a wearable device, and the method includes the following steps:

step 201, in response to the reported hardware interrupt, obtaining target sensor data, where the target sensor data is sensor data collected by a sensor in the wearable device before the hardware interrupt, and reporting the hardware interrupt when the sensor recognizes a target operation based on a threshold condition.

The interruption refers to that when some unexpected situations occur in the running process of the computer equipment and the host needs to intervene, the equipment can automatically stop the running program and transfer the program into a program for processing new situations, including software interruption and hardware interruption. The hardware interrupt is an asynchronous signal indicating that attention is needed or a currently executed program needs to be changed, and is generated by an external device (such as a network card, a hard disk, a keyboard and the like) connected with the system. In the field of human-computer interaction, hardware interrupt is generally used for Processing a trigger operation, and when a device receives the trigger operation, the device reports the hardware interrupt to a processor, and a Central Processing Unit (CPU) responds to the hardware interrupt, and controls the device to execute an instruction corresponding to the trigger operation.

In a possible implementation manner, a wearable device (e.g., an earphone, a smart watch, etc.) determines whether a touch operation is received or not according to sensor data, a threshold condition of hardware interrupt is preset in the wearable device, when the sensor data meets the threshold condition, the wearable device triggers the hardware interrupt and reports the hardware interrupt to a processor, and after the processor receives the hardware interrupt, the processor acquires target sensor data and further identifies the target sensor data.

The target sensor data is sensor data collected by a sensor in the wearable device before hardware interruption. For example, the wearable device detects a touch operation, a press operation, or the like by the pressure sensor; the shaking operation, the knocking operation, and the like are detected by a gravity sensor. After the target sensor data is acquired, the wearable device further identifies whether the target sensor data is sensor data generated by normal operation.

Step 202, inputting target sensor data into an operation recognition model to obtain an operation recognition result output by the operation recognition model, wherein the operation recognition model is a neural network obtained based on deep learning training, and the operation recognition result is used for indicating whether target operation is misoperation.

When normal operation and misoperation are divided only depending on threshold conditions, if the threshold is too high for shielding the misoperation, the normal operation is possibly shielded, and the operation identification rate is low; if the threshold is too low in order to increase the operation recognition rate, the wearable device may respond to a part of the erroneous operations, and the accuracy of the operation response is not high. Therefore, in the embodiment of the application, after the target sensor data is acquired, the wearable device inputs the target sensor data into the operation recognition model, and recognizes the target sensor data by using machine learning, so that the operation recognition rate and the accuracy rate of operation response are improved.

Optionally, an operation identification model is loaded in the wearable device, and the wearable device directly inputs the acquired target sensor data into its own operation identification model; or, an operation recognition model is loaded in a terminal (for example, a terminal such as a smart phone, a tablet computer, a notebook computer, etc.) connected to the wearable device, the wearable device sends the target sensor data to the connected terminal, and the terminal recognizes the target sensor data by using the operation recognition model and then feeds back the operation recognition result to the wearable device. The embodiments of the present application do not limit this.

Illustratively, the operation recognition model is a Neural Network obtained based on deep learning training, such as a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Support Vector Machine (SVM), and the like.

Andstep 203, responding to the target operation in response to the operation identification result indicating that the target operation is not the misoperation.

And the operation recognition result output by the operation recognition model indicates whether the target operation corresponding to the target sensor data is misoperation or not, and if not, the target operation is responded.

In one possible embodiment, the operation recognition model is used to recognize whether the data of at least one sensor belongs to sensor data resulting from a malfunction. For example, the operation recognition model recognizes target sensor data corresponding to the gravity sensor, and judges whether the target operation is a tapping operation; the operation recognition model recognizes target sensor data corresponding to the pressure sensor, and determines whether the target operation is a pressing operation or not.

To sum up, in the embodiment of the application, the target sensor data before hardware interruption is identified through the operation identification model, and whether the operation causing hardware interruption is misoperation is judged, so that the target operation is responded when the target operation is determined not to belong to the misoperation, the sensor data is further identified by using the deep learning neural network on the basis of threshold division, the accuracy of operation response is improved, the misoperation is avoided being responded, the accuracy of response operation is improved without adopting a mode of improving the threshold in the related technology, the identification range of the target operation is expanded on the basis of ensuring the accuracy, and the normal operation is prevented from being identified as the misoperation.

Fig. 3 is a flowchart illustrating an operation response method according to another exemplary embodiment of the present application. In this embodiment, the method is described by taking the method as an example for a wearable device, and the method includes the following steps:

step 301, in response to the reported hardware interrupt, determining n groups of sensor data continuously collected before the hardware interrupt is reported as target sensor data, where n is a positive integer greater than or equal to 2.

The sensor is used for collecting sensor data according to a preset sampling frequency.

The acquired sensor data meets the preset conditions and then is triggered and reported to the hardware interrupt, so that the acquired sensor data before the hardware interrupt is reported is the sensor data generated by the target operation, and the wearable device needs to identify the acquired sensor data before the hardware interrupt is reported. In addition, the sensor data is changed due to the target operation, and the sensor data is also changed due to the misoperation, but the change processes caused by the target operation and the sensor data are different, so that the wearable device needs to identify the change process of the sensor data and judge whether the change corresponding to the sensor data is caused by the normal operation, and therefore at least 2 groups of sensor data need to be identified, namely the wearable device determines n groups of sensor data continuously collected before the hardware interruption report as the target sensor data.

In one possible embodiment, the sensor is an acceleration sensor, and step 301 includes the steps of:

step 301a, determining an acceleration slope of an x axis, an acceleration slope of a y axis and an acceleration slope of a z axis at the same acquisition time as a set of sensor data, wherein the acceleration slopes are used for representing the change condition of the acceleration.

In one possible embodiment, the target operation is an operation that is in direct contact with the wearable device and can cause the wearable device to oscillate, and the wearable device acquires acceleration changes, i.e., acceleration slopes, in the x-axis direction, the y-axis direction, and the z-axis direction by the acceleration sensor, so as to determine whether the wearable device generates an oscillation caused by normal operation. Therefore, the wearable device determines the acceleration slope of the x-axis, the acceleration slope of the y-axis and the acceleration slope of the z-axis at the same acquisition time as a set of sensor data.

And step 301b, determining n groups of sensor data corresponding to n acquisition moments before hardware interruption reporting as target sensor data.

In order to judge whether a target operation causing hardware interruption is misoperation, the wearable device inputs an acceleration slope before the hardware interruption into an operation identification model for identification, namely the acceleration slope of an x axis, the acceleration slope of a y axis and the acceleration slope of a z axis at the same acquisition time are used as a set of sensor data, and n sets of sensor data corresponding to n acquisition times before the hardware interruption is reported are determined as the target sensor data.

In one possible implementation, the target operation is a tapping operation, the hardware interrupt is performed when the acceleration slope of any one of the x-axis, the y-axis and the z-axis is greater than a slope threshold, the acceleration slope is oscillated and changed in an oscillation period, and the acceleration slope of each axis is reported when the acceleration slope of each axis is less than the slope threshold in a quiescent period.

Schematically, as shown in fig. 4, the change rule of the acceleration in a certain coordinate axis corresponding to a tapping operation is shown. The value of acceleration is 0 when static, and the wearable equipment receives to shake immediately after the operation of knocking, and acceleration is followed the direction of knocking and is crescent, and can exceed the acceleration threshold value, and acceleration value begins to reduce at a certain moment to prolong reverse increase, then reduce gradually to 0 and resume statically.

The method comprises the steps that a vibration period and a static period are preset in the wearable device, when the acceleration collected by an acceleration sensor is larger than an acceleration threshold, continuous n groups of accelerations collected in the subsequent vibration period and the static period are calculated to obtain the slope of the acceleration, and if the acceleration slope in the vibration period is changed in a vibration mode and the acceleration slope of each axis in the static period is smaller than the slope threshold, hardware interruption is reported.

For example, the sensor data when the wearable device receives a tapping operation is shown in table 1, and the sensor data when the wearable device receives an incorrect operation (for example, a touch operation, a moving operation, etc.) is shown in table 2

TABLE 1

X axis	Y-axis	Z axis
			0	1	1
0	-1	0
			-1	1	0
0	-2	1
			-1	-1	-1
-1	-1	0
			-2	2	0
0	0	0
			0	-1	-1
0	2	-1
			2	1	0
1	0	1
			1	0	0

TABLE 2

It can be seen that there is a large difference between the acceleration data generated by the knocking operation and the misoperation, the oscillation amplitude generated by the knocking operation is large, the acceleration value is large, and the oscillation amplitude generated by the misoperation is small, and the acceleration value is also small. According to the embodiment of the application, the acceleration change condition of the wearable device oscillation is analyzed by the operation recognition model, and whether the target operation is misoperation or not is judged.

Because the target operation is identified by the neural network in the embodiment of the application, the identification accuracy is high, and therefore the knocking operation in a large acceleration range can be identified, that is, the wearable device can still identify the knocking operation in which the oscillation amplitude is close to the misoperation oscillation amplitude due to small knocking strength, in a possible implementation manner, before thestep 301, the embodiment of the application further includes the following steps:

and adjusting the slope threshold downwards, wherein the reporting rate of the hardware interrupt after the slope threshold is adjusted downwards is higher than the reporting rate of the hardware interrupt before the slope threshold is adjusted downwards.

A plurality of slope thresholds are set in a wearable device (e.g., an earphone) when the wearable device leaves a factory, and a developer needs to select one of the slope thresholds as a threshold condition according to actual operation corresponding to the wearable device. In the related art, in order to improve the accuracy of operation response and shield misoperation as much as possible, a developer generally needs to select a higher slope threshold as a judgment condition of a tapping operation, which may cause that part of tapping operations with smaller tapping force are shielded, and for a user who is used to a user with smaller tapping force, the situation that the tapping operation is performed and the wearable device does not respond may occur frequently in the process of using the wearable device.

According to the method and the device, the target operation is identified by utilizing the neural network, the accuracy rate is high, so that the slope threshold can be adjusted downwards, the knocking operation with low knocking strength can still be identified, and the reporting rate of the knocking operation is improved, namely the reporting rate of the hardware interruption after the slope threshold is adjusted downwards is higher than that before the slope threshold is adjusted downwards.

Optionally, the developer selects a minimum slope threshold in factory settings of the wearable device as a threshold condition; or the developer sets a lower slope threshold according to the actual requirement, so that the identification range of the knocking operation is larger.

And step 302, inputting the data of the n groups of sensors into the operation recognition model according to the sequence of the acquisition moments to obtain an operation recognition result output by the operation recognition model.

The operation recognition model needs to recognize the change situation of the sensor data, so that whether knocking operation is received or not is judged, the wearable device inputs n groups of sensor data into the operation recognition model according to the sequence of the acquisition moments, the operation recognition model can recognize the vibrating process of the wearable device based on the incidence relation of the acceleration between the adjacent acquisition moments, and whether target operation is misoperation or not is judged.

In one possible implementation, as shown in fig. 5, the operation recognition model is a CNN model, and the operation recognition model includes an input layer, a convolutional layer, a separable convolutional layer, and a fully-connected layer. Step 302 comprises the steps of:

step 302a, acquiring input target sensing data through an input layer.

The input layer of the CNN is used for inputting data to be recognized, and performs preprocessing operations on the input data, such as normalization operations. Schematically, as shown in fig. 5, the operation recognition model receives 13 sets of sensor data at a time, the input data is a feature vector of 3 × 13, and one set of sensor data is a feature vector.

Step 302b, feature extraction is performed on the target sensor data by the convolutional layer and the separable convolutional layer.

The convolutional layer and the separable convolutional layer perform local perception on data input, and perform feature extraction and feature mapping on the input data by using a certain number of convolutional cores.

Illustratively, as shown in fig. 5, the convolutional layer of CNN includes one layer of 8 2 × 2 convolutional kernels, and the separable convolutional layer includes 1 layer of 16 layers of 2 × 2 convolutional kernels.

And step 302c, performing feature classification on the features output by the separable convolutional layer through the full connection layer, and outputting an operation identification result.

The full connection layer is usually arranged at the tail part of the CNN and is used for re-fitting the local features extracted from the previous layers, reducing the loss of feature information, and classifying the input data according to the features obtained by fitting to obtain an identification result.

Schematically, as shown in fig. 5, the fully connected layers of CNN contain 2 layers of 1 × 16 convolution kernels.

Optionally, the operation recognition model in this embodiment of the present application may further include other neural network structures, such as a hidden layer, a pooling layer, an excitation layer, and the like, or may also adopt other neural network models, which is not limited in this embodiment of the present application.

And step 303, responding to the target operation in response to the operation identification result indicating that the target operation is not the misoperation.

For a specific implementation ofstep 303, reference may be made to step 203 described above, and details of this embodiment are not described herein again.

And step 304, responding to the operation identification result to indicate that the target operation is misoperation, and not responding to hardware interruption.

And if the operation identification result indicates that the target operation is misoperation, determining that the user does not perform preset operation on the wearable device, and the wearable device does not respond to the hardware interrupt and continues to execute the instruction before the hardware interrupt.

In a possible implementation manner, multiple operations are set in the wearable device, and are respectively used for triggering different instructions, and the wearable device needs to respond according to an instruction corresponding to a target operation identified within a preset time length. For example, in the tapping operation of the earphone, a single tapping operation is used for triggering the start and pause of music, a double tapping operation is used for starting a voice call function, the time interval of two taps in the double tapping operation is not more than 0.5s, after the earphone recognizes one tapping operation, whether the tapping operation is received again within 0.5s is recognized, and an instruction required to be executed is determined according to the recognition result.

In the embodiment of the application, multiple groups of sensor data continuously collected by the sensor before hardware interruption reporting are identified, and whether the target operation is misoperation is judged by utilizing the change rule of the sensor data identified by the operation identification model, so that the accuracy of responding to the target operation is improved; and moreover, the slope threshold is properly adjusted downwards, the identification range of the target operation is expanded, the report rate of the target operation is improved, and the shielding of the target operation with the sensor data similar to the sensor data of the misoperation is avoided.

The above embodiment shows a process of identifying a target operation by using an operation identification model, where the operation identification model loaded in the wearable device is a neural network model trained in advance, and before the operation identification model identifies the target operation, the operation identification model needs to be trained, so that the accuracy of an operation identification result reaches an expected value. In a possible implementation manner, the operation recognition model is obtained by training according to positive sample data and negative sample data, the positive sample data comprises sensor data acquired by a sensor when the target operation is received, and the negative sample data comprises sensor data acquired by the sensor when the target operation corresponds to the misoperation.

The method comprises the steps that a developer collects sensor data under multiple target operations and sensor data under different kinds of misoperation in advance, for example, for knocking operations, sensor data of knocking operations corresponding to different knocking forces and knocking drop points are collected in advance to serve as positive sample data, and sensor data collected by a sensor when a user wears a wearable device to move and touch operations and wearable device moving operations are collected to serve as negative sample data. The developer adds a target operation or misoperation label to each group of positive sample data and negative sample data, and trains the operation recognition model for multiple times by using the computer equipment until the model converges, for example, the training times reach the preset times, or the accuracy of the operation recognition result output by the operation recognition model reaches the preset value, which is not limited in the embodiments of the present application.

In a possible implementation manner, on the basis of fig. 2, as shown in fig. 6, after thestep 203, the operation response method provided by the embodiment of the present application further includes a step 204:

step 204, in response to receiving the error response indication, determining the target sensor data as negative sample data, where the negative sample data is used to update the training operation recognition model.

The user can trigger the error response instruction if the wearable device executes a response to the target operation after the user performs other operations or the wearable device automatically executes a response to the target operation when the user does not perform the target operation, and after receiving the error response instruction, the wearable device determines the target sensor data as negative sample data and returns the negative sample data to a task before hardware interrupt reporting, or prompts the user to perform correct operation again.

Optionally, when the wearable device receives the operation corresponding to the instruction executed before the hardware interrupt report again within the preset time length after the target operation is responded, or receives the instruction of returning to the previous operation, it determines that the error response instruction is received; alternatively, the wearable device determines that the error response indication is received when receiving a trigger operation of the error response indication by the user, for example, a trigger operation of the error response control.

In a possible implementation manner, after the wearable device obtains a certain amount of negative sample data, the wearable device updates and trains the operation recognition model by using the target sensor data when the error response indication is not received as the positive sample data. Optionally, the wearable device directly updates and trains the loaded operation recognition model, or uploads the collected positive sample data and negative sample data to the cloud server, the cloud server updates and trains the current operation recognition model based on the received sample data, and feeds back various parameters of the operation recognition model after the update training to the corresponding wearable device, and the wearable device updates the operation recognition model after receiving the model parameters sent by the cloud server, so that the accuracy of the wearable device in responding to the target operation is improved.

In the embodiment of the application, when the error response instruction is received, the target sensor data is determined as the negative sample data, the obtained negative sample data is used for updating and training the operation recognition model, the classification result of the operation recognition model on the target operation and the misoperation is closer to the actual operation of the user, the wearable device carries out personalized updating on the operation recognition model according to the use condition of the user, and the applicability of the operation recognition model to different users is improved.

Fig. 7 is a block diagram of an operation response apparatus according to an exemplary embodiment of the present application, where the apparatus includes:

an obtainingmodule 701, configured to obtain target sensor data in response to a reported hardware interrupt, where the target sensor data is sensor data acquired by a sensor in the wearable device before the hardware interrupt, and the target sensor data is reported when the sensor recognizes a target operation based on a threshold condition in the hardware interrupt;

theidentification module 702 is configured to input the target sensor data into an operation identification model, to obtain an operation identification result output by the operation identification model, where the operation identification model is a neural network obtained based on deep learning training, and the operation identification result is used to indicate whether the target operation is an incorrect operation;

a first respondingmodule 703, configured to respond to the target operation in response to the operation identification result indicating that the target operation is not an incorrect operation.

Optionally, the sensor is configured to acquire sensor data according to a preset sampling frequency;

the obtainingmodule 701 includes:

a determining unit, configured to determine n groups of sensor data that are continuously acquired before the hardware interrupt report as the target sensor data, where n is a positive integer greater than or equal to 2;

theidentification module 702 includes:

and the input unit is used for inputting n groups of sensor data into the operation recognition model according to the sequence of the acquisition moments to obtain the operation recognition result output by the operation recognition model.

Optionally, the sensor is an acceleration sensor;

the determining unit is further configured to:

determining an acceleration slope of an x axis, an acceleration slope of a y axis and an acceleration slope of a z axis at the same acquisition time as a group of sensor data, wherein the acceleration slopes are used for representing the change condition of acceleration;

and determining n groups of sensor data corresponding to n acquisition moments before the hardware interruption report as the target sensor data.

Optionally, the target operation is a tapping operation, the acceleration slope of the hardware interrupt in any one of an x axis, a y axis and a z axis is greater than a slope threshold, the acceleration slope oscillates and changes in an oscillation period, and the acceleration slope of each axis in a static period is reported when the acceleration slope is less than the slope threshold.

Optionally, the apparatus further comprises:

and the adjusting module is used for adjusting the slope threshold value downwards, wherein the reporting rate of the hardware interrupt after the slope threshold value is adjusted downwards is higher than the reporting rate of the hardware interrupt before the slope threshold value is adjusted downwards.

Optionally, the operation identification model is a convolutional neural network model, and the operation identification model includes an input layer, a convolutional layer, a separable convolutional layer, and a fully-connected layer;

theidentification module 702 includes:

the acquisition unit is used for acquiring the input target sensing data through the input layer;

a feature extraction unit for performing feature extraction on the target sensor data by the convolutional layer and the separable convolutional layer;

and the classification unit is used for performing characteristic classification on the characteristics output by the separable convolution layer through the full-connection layer and outputting the operation identification result.

Optionally, the operation recognition model is obtained by training according to positive sample data and negative sample data, where the positive sample data includes sensor data acquired by the sensor when the target operation is received, and the negative sample data includes sensor data acquired by the sensor when the target operation corresponds to a misoperation.

Optionally, the apparatus further comprises:

a determination module, configured to determine, in response to receiving an error response indication, the target sensor data as the negative sample data, where the negative sample data is used to update and train the operation recognition model.

Optionally, the apparatus further comprises:

and the second response module is used for responding to the operation identification result to indicate that the target operation is misoperation and not responding to the hardware interrupt.

As shown in fig. 8, an embodiment of the present application provides awearable device 800, where thewearable device 800 may include one or more of the following components: aprocessor 801, amemory 802, apower component 803, anaudio component 804, an Input/Output (I/O)interface 805, asensor component 806, and acommunication component 807.

Theprocessor 801 generally controls the overall operation of the wearable device, such as operations associated with phone calls, data communications, audio playback, data recording, and operation identification.Processor 801 may include one or more processing cores. Theprocessor 801 connects various parts within the entirewearable device 800 using various interfaces and lines, performs various functions of thewearable device 800 and processes data by executing or executing instructions, programs, code sets, or instruction sets stored in thememory 802, and calling data stored in thememory 802. Alternatively, theprocessor 801 may be implemented in hardware using at least one of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Theprocessor 801 may integrate one or a combination of several of a CPU and a modem, etc. Wherein, the CPU mainly processes an operating system, a user interface, an application program and the like; the modem is used to handle wireless communications. It is to be understood that the modem may not be integrated into theprocessor 801, but may be implemented by a communication chip.

Thememory 802 is configured to store various types of data to support operations on the wearable device. Examples of such data include instructions, models, contact data, phonebook data, messages, audio, etc. for any application or method operating on the wearable device. TheMemory 802 may include a Random Access Memory (RAM) or a Read-Only Memory (ROM). Optionally, thememory 802 includes a non-transitory computer-readable medium. Thememory 802 may be used to store instructions, programs, code, sets of codes, or sets of instructions. Thememory 802 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, and the like), instructions for implementing the above-described method embodiments, and the like, and the operating system may be an Android (Android) system (including a system based on Android system depth development), an IOS system developed by apple, including a system based on IOS system depth development), or other systems. The data storage area may also store data (e.g., phone book, audio data, sensor data) collected by thewearable device 800 during use, and the like.

Thepower supply component 803 provides power to the various components of thewearable device 800. Thepower components 803 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for thewearable device 800.

Theaudio component 804 is configured to output and/or input audio signals. For example, theaudio component 804 includes a Microphone (MIC) configured to receive external audio signals when thewearable device 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may further be stored in thememory 802 or transmitted via thecommunication component 807. In some embodiments, theaudio component 804 also includes a speaker for outputting audio signals.

The I/O interface 805 provides an interface between theprocessor 801 and peripheral interface modules, which may be click wheels, buttons, touch pads, etc. These buttons may include, but are not limited to: a volume button, a start button, and a lock button.

Thesensor assembly 806 includes one or more sensors to provide various aspects of state assessment for thewearable device 800. For example, thesensor component 806 can detect the open/closed state of thewearable device 800, the relative positioning of the components, thesensor component 806 can also detect a change in the position of thewearable device 800 or thewearable device 800, the presence or absence of user contact with thewearable device 800, the orientation or acceleration/deceleration of thewearable device 800, and a change in the temperature of thewearable device 800. Thesensor assembly 806 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. In some embodiments, thesensor assembly 806 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor. For example, thewearable device 800 determines the operation type of the control operation by a pressure sensor, and determines whether or not a tapping operation is received by an acceleration sensor.

Thecommunication component 807 is configured to facilitate communications between thewearable device 800 and other devices (e.g., control devices) in a wired or wireless manner. Thewearable device 800 can access a wireless network based on a communication standard. In an exemplary embodiment, thecommunication component 807 receives a broadcast signal or broadcast associated information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, theCommunication component 807 further comprises a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology, and other technologies. Thewearable device 800 synchronously receives information sent by the control device, such as audio files played by the control device, through thecommunication component 807.

In addition, those skilled in the art will appreciate that the configuration of thewearable device 800 shown in the above figures does not constitute a limitation of thewearable device 800, and that the device may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.

The embodiment of the present application further provides a computer-readable storage medium, which stores at least one instruction, where the at least one instruction is loaded and executed by a processor to implement the operation response method according to the above embodiments.

Those skilled in the art will recognize that, in one or more of the examples described above, the functions described in the embodiments of the present application may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable storage medium. Computer-readable storage media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.

The above description is only exemplary of the present application and should not be taken as limiting, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.