Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to specific embodiments of the present invention and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The following describes in detail the technical solutions provided by the embodiments of the present invention with reference to the accompanying drawings.
Example 1
At present, as the manipulator grabbing flexibility of the robot is low, the force used by the manipulator cannot be flexibly adjusted according to the size and shape of the object to be grabbed and the grabbing feeling, so that the manipulator is easy to cause the object grabbing failure in various grabbing processes. To solve this problem, it is considered that if a reliable prediction can be made of the grasping result when the robot starts grasping the object, it is possible to effectively avoid the failed grasping and provide an opportunity to re-grasp the target object, thereby improving the success rate of grasping.
Based on this, in the related art, it is proposed to predict the gripping stability of the manipulator based on the machine learning, the deep learning, and the visual information or the tactile information of the robot, but since the visual information is easily affected by factors such as shielding, light intensity, etc., when the gripping stability of the manipulator is predicted based on the visual information, the situation that the visual image information is omitted easily occurs, so that the prediction result is inaccurate.
In addition, when the grasping stability of the manipulator is predicted based on the haptic information, the machine learning and the deep learning technologies, the haptic signals are generally regarded as regular matrix signals, so that the haptic signals are inconsistent with the actual distribution of the haptic sensing points on the haptic sensor, and further, the predicted result is deviated from the actual grasping condition.
In order to solve the problem that the prediction result and the actual grabbing condition have deviation when the grabbing stability of the manipulator is predicted in the prior art, the embodiment of the invention provides a manipulator grabbing stability recognition method.
The execution subject of the method may be various types of computing devices, or may be an Application program or Application (APP) installed on the computing device. The computing device can be a user terminal such as a mobile phone, a tablet computer, an intelligent wearable device and the like, and also can be a server and the like.
For convenience of description, the embodiment of the present invention uses an execution body of the method as a server as an example, and introduces the method. It will be appreciated by those skilled in the art that the embodiment of the present invention is described by taking the server as an example, and is only an exemplary illustration, and does not limit the scope of protection of the claims corresponding to the present scheme.
Specifically, the implementation flow of the method provided by the embodiment of the invention is shown in fig. 1a, and the implementation flow comprises the following steps:
and 11, collecting target touch data of the object to be identified.
The object to be identified may include objects of various shapes, for example, a cylindrical box, a square box, a sphere, etc., which is not limited in any way by the embodiment of the present invention.
The target touch data can be used for representing the friction force of a grabbing point, the elasticity of the object, the roughness of the object and the contact condition (contact temperature, contact strength and pressure) of the finger and the object when the manipulator grabs the object.
For example, in an alternative embodiment, the target tactile data may include pressure signals, temperature signals, and electrode profile signals (e.g., electrode impedance) generated by a tactile sensor located on a finger of the robot arm during interaction with the object to be identified when the robot arm is gripping the object.
For convenience of description, embodiments of the present invention will be described below taking the target haptic data as an electrode distribution signal as an example. It should be noted that the above-mentioned electrode distribution signal is only an exemplary illustration of the implementation of the present invention, and is not meant to limit the embodiments of the present invention.
In the embodiment of the invention, the touch sensor is arranged on the finger of the manipulator, and then the touch sensor is utilized to collect the target touch data of the object to be identified. For example, a BioTac multimode bionic touch sensor can be mounted on a finger of a manipulator in advance, then the manipulator provided with the BioTac multimode bionic touch sensor is controlled to gradually approach an object to be identified, the grabbing operation is performed on a target object with random grabbing points and grabbing forces, and finally electrode distribution signals generated by the touch sensor in each interaction process are collected to serve as target touch data of the object to be identified.
In the embodiment of the invention, in order to ensure the grabbing stability, after the manipulator grabs the object to be identified, a certain range of shaking motion can be performed in each direction, and then signals generated by the touch sensor in the interaction process are collected as target touch data of the object to be identified.
Optionally, in order to ensure accuracy of the recognition result, in the embodiment of the present invention, after the target tactile data of the object to be recognized is collected, the target tactile data may be further preprocessed. The preprocessing process comprises data truncation, downsampling and Z-score standardization processing.
For example, acquired data before the manipulator touches the preset article may be truncated for discarding to avoid noise interference.
In addition, noise is also generated in consideration of the excessively high sampling frequency, so that the target touch data can be properly downsampled to improve the training speed of the network; finally, Z-score standardization can be performed on the target touch data, so that the target touch data accords with standard normal distribution, and accuracy of the identification result is improved.
And step 12, inputting the target touch data into a manipulator grabbing stability recognition model which is obtained through pre-training, and obtaining an output result.
The manipulator grabbing stability recognition model can be understood as a network model for recognizing grabbing stability of the manipulator, and an output result for representing grabbing stability of the manipulator can be obtained by inputting target tactile data into the manipulator grabbing stability recognition model.
Optionally, the manipulator grabbing stability recognition model may include a graph convolution layer, an activation layer, a full connection layer, and/or a feature fusion layer; the convolution kernel of the graph roll layer may have a size of 1*1, and the number of outputs of the full connection layer may be 2.
According to the embodiment of the invention, the manipulator grabbing stability recognition model can be obtained based on the adjacency matrix training of the touch data. The adjacent matrix of the haptic data can be understood as a matrix constructed based on a graph structure of the haptic data, and the adjacent matrix is attached to the actual distribution situation of the haptic sensing points on the haptic sensor, so that the correlation between the haptic data can be fully reflected.
The graph structure of the haptic data, i.e., the graphical form of the haptic data, may also be referred to as a haptic graph of the haptic data.
In an alternative embodiment, as shown in FIG. 1b, the following steps 120-124 may be employed to determine the adjacency matrix of haptic data.
Step 120, determining an electrode array corresponding to the haptic data.
The electrode array corresponding to the touch data can be understood as an array formed by electrode distribution signals collected by a contact array formed by a sensor on a manipulator finger. In the embodiment of the invention, a preset number of touch sensors can be arranged on a preset finger of the manipulator in advance to form a contact array; then, an electrode array corresponding to the tactile data is determined based on the tactile data collected by each of the tactile sensors.
And 122, determining a graph structure matched with the tactile data through a k-nearest neighbor algorithm according to the distribution information of the electrode array in the tactile data acquisition device.
Step 124, determining an adjacency matrix of the haptic data based on the graph structure and the position coordinates of the contact points in the haptic data.
In the embodiment of the invention, the adjacency matrix of the haptic data can be determined according to the following formula (1) based on the graph structure and the position coordinates of the contact points in the haptic data.
Wherein aij represents an adjacency matrix of haptic data, (vi,vj) represents an undirected edge connecting node i and node j; e represents the undirected edges of a set of connected nodes. Connecting node i and node j is a contact in the grabbing process.
In an alternative embodiment, the graph structure matched with the tactile data can be determined through a k-nearest neighbor algorithm according to the distribution information of the electrode array corresponding to the tactile data in the tactile data acquisition device.
As shown in FIG. 1c, in an embodiment of the present invention, the following steps 126-128 may be used to determine the graph structure of the haptic data:
step 126, calculate Euclidean distance between contacts in the haptic data.
In the embodiment of the invention, the euclidean distance between the contacts in the haptic data can be calculated according to the following formula (2).
Wherein dij represents the Euclidean distance between the contacts in the haptic data; (xi,yi,zi) and (xj,yj,zj) represent the actual three-dimensional coordinates of the contact in space.
And step 128, connecting each contact with a preset number of adjacent contacts closest to each contact according to the Euclidean distance to obtain a graph structure matched with the tactile data.
In the embodiment of the invention, after the Euclidean distance between the contacts in the touch data is calculated, the contacts can be connected with the adjacent contacts closest to the preset number of the contacts according to the Euclidean distance to obtain the graph structure matched with the touch data.
For example, assuming that there are A, B, C, D, E, F contacts and that the contact a is 10cm, 12cm, 7cm, 9cm, and 15cm from the contact B, C, D, E, F, respectively, and the preset number of adjacent contacts is 3, three contacts (D, E, B) closest to the contact a may be selected, and the contact a may be connected to the contact D, E, B, respectively, to form three undirected edges, thereby obtaining a graph structure matching with the tactile data.
In order to improve the recognition efficiency of the grabbing stability of the manipulator, in the embodiment of the invention, the grabbing stability recognition model of the manipulator can be obtained through pre-training, so that after the target touch data of the object to be recognized is acquired, the target touch data can be input into the grabbing stability recognition model of the manipulator obtained through pre-training, and an output result is obtained.
In an alternative embodiment, the manipulator gripping stability recognition model may be trained in the following manner.
First, a haptic data training sample and a haptic data test sample are obtained.
According to the embodiment of the invention, the manipulator can be controlled to execute the grabbing action for the preset article for the preset times by using random grabbing points and grabbing forces, and then the interactive data in the grabbing process is collected to serve as a touch data training sample and a touch data testing sample. Wherein the fingers of the manipulator comprise at least one tactile sensor.
In order to ensure the grabbing stability and improve the accuracy of the identification result, in the embodiment of the invention, when the manipulator grabs the object to be preset, the manipulator can be controlled to shake the object to be preset grabbed each time and then collect the touch data in the shaking process when the touch data training sample and the touch data testing sample are acquired. And finally, dividing the tactile data in the shaking motion process into a tactile data training sample and a tactile data testing sample according to a preset proportion based on the tactile data in the shaking motion process and the result of the shaking motion.
For example, assuming that the preset article is one of a cylindrical object, a square box and a spherical object, respectively, and 1000 grabbing operations are performed on the cylindrical object, 500 grabbing operations are performed on the square box, and 500 grabbing operations are performed on the spherical object, in the embodiment of the present invention, the data of the 2000 grabs may be randomly disordered and divided into a tactile data training sample and a tactile data test sample according to a ratio of 8:2.
The foregoing examples are merely illustrative of embodiments of the present invention and are not intended to limit the embodiments of the present invention in any way.
It should be noted that, in order to facilitate the subsequent verification/test of the recognition result of the manipulator grabbing stability recognition model, in the embodiment of the present invention, each tactile data sample may be labeled based on the result in the shaking motion process.
For example, during a shaking motion, the data may be labeled "successful" assuming that the cylindrical object is successful after a certain shaking. Or if the result after a certain shaking is failure, the data may be labeled "failure".
Alternatively, for ease of recording, the label of the tactile data sample may be recorded as 0 or 1, where 1 indicates successful capture and 0 indicates failed capture.
Optionally, in order to avoid the influence of noise interference of data on the accuracy of the identification result, in the embodiment of the present invention, after the interactive data in the capturing process is collected, the interactive data may be further preprocessed. The preprocessing process comprises data interception, downsampling and Z-score standardization processing.
For example, signals before the manipulator contacts the preset article can be cut off and discarded to avoid noise interference; in addition, noise is also generated in consideration of the excessively high sampling frequency, so that the signal can be properly downsampled to improve the training speed of the network; finally, Z-score standardization can be carried out on the data, so that the data accords with standard normal distribution, and the accuracy of the identification result is improved.
Second, an adjacency matrix for the haptic data training samples is determined.
The method for determining the adjacency matrix of the training sample of the haptic data is the same as the method for determining the adjacency matrix of the haptic data, and the related content of the steps 120 to 124 may be referred to, and will not be described herein.
Thirdly, training a preset network model according to the adjacency matrix to obtain a manipulator grabbing stability recognition model; the preset network model is used for identifying grabbing stability of the manipulator.
It should be noted that, in order to learn the deeper features of the haptic data, the preset network model may include a graph convolution layer, where the convolution kernel of the graph convolution layer may have a size of 1*1. In the embodiment of the invention, after the graph convolution layer is adopted, graph convolution operation processing can be carried out on the input data, so that the deeper features of the tactile data can be learned.
The formula of the graph convolution operation is as follows (3):
Wherein,A represents an adjacency matrix of the training sample of haptic data, and I is an identity matrix, with the aim of introducing self-connections.W is a preset trainable weight matrix, and fin、fout represents the input and output of the convolutional layer respectively.
Optionally, the preset network model may further include an active layer and a full connection layer in addition to the graph roll layer. Wherein the activation function used by the activation layer is a ReLU; the number of outputs of the fully connected layer may be 2, which may correspond to the stability and instability of the haptic data training samples.
In addition, considering the problem that in the related art, a single sensor model is generally adopted, so that the accuracy of the identification result is low easily, in the embodiment of the invention, the preset network model can also comprise a feature fusion layer, and after the feature fusion layer is used, on one hand, the identification result can be optimized through a fusion technology; on the other hand, the multiple touch sensors can be subjected to feature fusion, and compared with a single sensor model, the accuracy of the recognition result can be improved.
In the embodiment of the invention, if the preset network model comprises a graph convolution layer, an activation layer, a full connection layer and a feature fusion layer, the manipulator grabbing stability recognition model can comprise the graph convolution layer, the activation layer, the full connection layer and/or the feature fusion layer; the convolution kernel of the graph roll layer may have a size of 1*1, and the number of outputs of the full connection layer may be 2.
As shown in fig. 1d, a schematic structural diagram of a manipulator grabbing stability recognition model according to an embodiment of the present invention is provided, where the manipulator grabbing stability recognition model includes a graph convolution layer, an activation layer (an activation function ReLU in fig. 1 d), a full connection layer, and a feature fusion layer. Wherein Graph-L, graph-M, graph-R respectively represents electrode data of tactile sensors on the left finger, the middle finger and the right finger of the manipulator.
After electrode data of tactile sensors on left fingers, middle fingers and right fingers of the manipulator are input into a manipulator grabbing stability recognition model, the electrode data firstly reach a graph convolution layer to carry out graph convolution layer operation processing, and a graph convolution layer operation result is obtained; the operation result of the graph convolution layer is input to an activation layer (an activation function ReLU in FIG. 1 d) to obtain an output result; and then, the feature fusion layer performs feature fusion processing based on the output result of the activation layer, and finally, the feature fusion result is processed through the full-connection layer and the activation function Sigmoid function to obtain the output result.
Fourth, the recognition result of the manipulator grabbing stability recognition model is tested based on the touch data test sample.
In order to ensure accuracy of the recognition result, in an alternative embodiment, after the manipulator grabbing stability recognition model is obtained, the validity of the manipulator grabbing stability recognition model may also be tested based on the tactile data test sample.
The indexes for evaluating the effectiveness of the manipulator grabbing stability recognition model can comprise classification accuracy and F1 score.
And step 13, identifying the grabbing stability of the manipulator based on the output result.
The output result can be used for describing the stability of the target touch data and can also be used for representing the grabbing stability of the manipulator. Moreover, when the target touch data is stable, the grabbing stability of the manipulator is relatively high; when the target tactile data is unstable, the gripping stability of the manipulator is relatively low.
In the embodiment of the invention, after the output result is obtained, the grabbing stability of the manipulator can be identified according to the output result.
According to the method provided by the embodiment of the invention, as the manipulator grabbing stability recognition model is obtained by training the adjacent matrix based on the tactile data, and the adjacent matrix of the tactile data is determined based on the graph structure of the tactile data, when the manipulator grabbing stability is recognized based on the tactile data and the manipulator grabbing stability recognition model, the manipulator grabbing stability recognition model can determine the adjacent matrix of the tactile data based on the tactile data, and then the manipulator grabbing stability is recognized based on the adjacent matrix of the tactile data, compared with the related art, the adjacent matrix can fully reflect the correlation between the tactile data and is more attached to the actual distribution situation of the tactile sensing points on the tactile sensor, so that the recognition result is more attached to the actual grabbing stability of the manipulator.
Example 2
The embodiment of the invention also provides a training method of the manipulator grabbing stability recognition model, which is used for training and obtaining the manipulator grabbing stability recognition model in the embodiment 1. Referring to fig. 2, a schematic implementation flow chart of a training method of a manipulator grabbing stability recognition model provided by an embodiment of the present invention is shown, where the flow chart specifically includes the following steps:
step 21, obtaining a tactile data training sample and a tactile data testing sample.
According to the embodiment of the invention, the manipulator can be controlled to execute the grabbing action for the preset article for the preset times by using random grabbing points and grabbing forces, and then the interactive data in the grabbing process is collected to serve as a touch data training sample and a touch data testing sample. Wherein the fingers of the manipulator comprise at least one tactile sensor.
In order to ensure the grabbing stability and improve the accuracy of the identification result, in the embodiment of the invention, when the manipulator grabs the object to be preset, the manipulator can be controlled to shake the object to be preset grabbed each time and then collect the touch data in the shaking process when the touch data training sample and the touch data testing sample are acquired. And finally, dividing the tactile data in the shaking motion process into a tactile data training sample and a tactile data testing sample according to a preset proportion based on the tactile data in the shaking motion process and the result of the shaking motion.
For example, assuming that the preset article is one of a cylindrical object, a square box and a spherical object, respectively, and 1000 grabbing operations are performed on the cylindrical object, 500 grabbing operations are performed on the square box, and 500 grabbing operations are performed on the spherical object, in the embodiment of the present invention, the data of the 2000 grabs may be randomly disordered and divided into a tactile data training sample and a tactile data test sample according to a ratio of 8:2.
The foregoing examples are merely illustrative of embodiments of the present invention and are not intended to limit the embodiments of the present invention in any way.
It should be noted that, in order to facilitate the subsequent verification/test of the recognition result of the manipulator grabbing stability recognition model, in the embodiment of the present invention, each tactile data sample may be labeled based on the result in the shaking motion process.
For example, during a shaking motion, the data may be labeled "successful" assuming that the cylindrical object is successful after a certain shaking. Or if the result after a certain shaking is failure, the data may be labeled "failure".
Alternatively, for ease of recording, the label of the tactile data sample may be recorded as 0 or 1, where 1 indicates successful capture and 0 indicates failed capture.
At step 22, an adjacency matrix for the haptic data training samples is determined.
The method for determining the adjacency matrix of the training sample of the haptic data is the same as that of the method for determining the adjacency matrix of the haptic data in the above-mentioned embodiment 1, and the related contents of steps 120 to 124 in the above-mentioned embodiment 1 may be referred to, and will not be repeated here.
Step 23, training a preset network model according to the adjacency matrix to obtain the manipulator grabbing stability recognition model; the preset network model comprises a graph convolution layer, an activation layer, a feature fusion layer and a full connection layer, wherein the convolution kernel size of the graph convolution layer is 1*1, and the number of outputs of the full connection layer is 2.
It should be noted that, in order to learn the deeper features of the haptic data, the preset network model may include a graph convolution layer, where the convolution kernel of the graph convolution layer may have a size of 1*1. In the embodiment of the invention, after the graph convolution layer is adopted, graph convolution operation processing can be carried out on the input data, so that the deeper features of the tactile data can be learned.
Optionally, the preset network model may further include an active layer and a full connection layer in addition to the graph roll layer. Wherein the activation function used by the activation layer is a ReLU; the number of outputs of the fully connected layer may be 2, which may correspond to the stability and instability of the haptic data training samples.
In addition, considering the problem that in the related art, a single sensor model is generally adopted, so that the accuracy of the identification result is low easily, in the embodiment of the invention, the preset network model can also comprise a feature fusion layer, and after the feature fusion layer is used, on one hand, the identification result can be optimized through a fusion technology; on the other hand, the multiple touch sensors can be subjected to feature fusion, and compared with a single sensor model, the accuracy of the recognition result can be improved.
In the embodiment of the invention, if the preset network model comprises a graph convolution layer, an activation layer, a full connection layer and a feature fusion layer, the manipulator grabbing stability recognition model can comprise the graph convolution layer, the activation layer, the full connection layer and/or the feature fusion layer; the convolution kernel of the graph roll layer may have a size of 1*1, and the number of outputs of the full connection layer may be 2.
By adopting the method provided by the embodiment of the invention, the adjacency matrix of the tactile data training sample can be determined, and then the preset network model is trained according to the adjacency matrix to obtain the manipulator grabbing stability recognition model, so that the correlation between the tactile data can be fully extracted when the model provided by the embodiment of the invention is used for recognition, the characteristics extracted based on the tactile data are more attached to the actual distribution condition of the tactile sensing points on the tactile sensor, and the accuracy of the recognition result is further ensured.
Example 3
In order to solve the problem that in the prior art, when the grasping stability of the manipulator is predicted, the deviation exists between the prediction result and the actual grasping condition, the embodiment of the invention provides a grasping stability recognition device for the manipulator, and the specific structural schematic diagram of the device is shown in fig. 3 and comprises an acquisition module 31, an input module 32 and a recognition module 33. The functions of each unit are as follows:
The acquisition module 31 is used for acquiring target touch data of the object to be identified.
The input module 32 is configured to input the target tactile data into a manipulator grabbing stability recognition model obtained by training in advance, so as to obtain an output result; the manipulator grabbing stability recognition model is obtained through training based on an adjacency matrix of the tactile data, and the adjacency matrix of the tactile data is determined based on a graph structure of the tactile data.
And an identification module 33, configured to identify grasping stability of the manipulator based on the output result.
In an alternative embodiment, the apparatus further comprises:
And the acquisition module is used for acquiring the tactile data training sample and the tactile data testing sample.
And the determining module is used for determining the adjacency matrix of the tactile data training sample.
The training module is used for training a preset network model according to the adjacency matrix to obtain a manipulator grabbing stability recognition model; the preset network model is used for identifying grabbing stability of the manipulator.
And the test module is used for testing the recognition result of the manipulator grabbing stability recognition model based on the touch data test sample.
In an alternative embodiment, the obtaining module includes:
And the control unit is used for controlling the manipulator to execute grabbing actions on the preset article for a preset number of times by using random grabbing points and grabbing forces, wherein the fingers of the manipulator comprise at least one touch sensor.
The acquisition unit is used for controlling the manipulator to shake the preset article grabbed each time and acquiring touch data in the shaking process.
The division unit is used for dividing the tactile data in the shaking motion process into a tactile data training sample and a tactile data test sample according to a preset proportion based on the tactile data in the shaking motion process and the result of the shaking motion.
In an alternative embodiment, the determining module includes:
and the electrode array determining unit is used for determining an electrode array corresponding to the touch data.
And the graph structure determining unit is used for determining the graph structure matched with the tactile data through a k-nearest neighbor algorithm according to the distribution information of the electrode array in the tactile data acquisition device.
And an adjacency matrix determination unit for determining an adjacency matrix of the haptic data based on the graph structure and the position coordinates of the contact points in the haptic data.
In an alternative embodiment, the graph structure determining unit includes:
and the calculating subunit is used for calculating Euclidean distance between the contacts in the touch data.
And the connection subunit is used for connecting each contact with the preset number of adjacent contacts closest to each contact according to the Euclidean distance to obtain a graph structure matched with the tactile data.
In an alternative embodiment, the acquisition module 31 is configured to: the method comprises the steps of controlling a manipulator to grab a target object at random grabbing points and grabbing forces, wherein fingers of the manipulator comprise at least one touch sensor; and collecting interaction data in the grabbing process as tactile data of the target object to be identified.
In an alternative embodiment, the manipulator grabbing stability recognition model comprises a graph convolution layer, an activation layer, a feature fusion layer and a full connection layer; the convolution kernel size of the graph roll lamination is 1*1, and the number of outputs of the full connection layer is 2.
According to the device provided by the embodiment of the invention, the manipulator grabbing stability recognition model is obtained by training the adjacent matrix based on the tactile data, and the adjacent matrix of the tactile data is determined based on the graph structure of the tactile data, so that when the manipulator grabbing stability is recognized based on the tactile data and the manipulator grabbing stability recognition model, the manipulator grabbing stability recognition model can determine the adjacent matrix of the tactile data based on the tactile data, and then the manipulator grabbing stability is recognized based on the tactile data adjacent matrix.
Example 4
The embodiment of the invention also provides a training device of the manipulator grabbing stability recognition model, which is used for training and obtaining the manipulator grabbing stability recognition model in the embodiment 1. The specific structure of the device is shown in fig. 4, and includes a sample acquisition module 41, a matrix determination module 42 and a model training module 43. The functions of each unit are as follows:
the sample acquisition module 41 is used for acquiring a tactile data training sample and a tactile data test sample.
In the embodiment of the present invention, the sample acquiring module 41 may perform a preset number of grabbing actions on a preset article by controlling the manipulator with random grabbing points and grabbing forces, and then collect the interactive data of the grabbing process, as a tactile data training sample and a tactile data testing sample. Wherein the fingers of the manipulator comprise at least one tactile sensor.
It should be noted that, in order to ensure the stability of grabbing and improve the accuracy of the identification result, in the embodiment of the present invention, when the sample acquisition module 41 acquires the tactile data training sample and the tactile data testing sample, after the manipulator grabs the object to be preset, the manipulator may be controlled to shake the preset object grabbed each time, and then the tactile data in the shaking process is acquired. And finally, dividing the tactile data in the shaking motion process into a tactile data training sample and a tactile data testing sample according to a preset proportion based on the tactile data in the shaking motion process and the result of the shaking motion.
For example, assuming that the preset article is one of a cylindrical object, a square box and a spherical object, respectively, and 1000 grabbing operations are performed on the cylindrical object, 500 grabbing operations are performed on the square box, and 500 grabbing operations are performed on the spherical object, in the embodiment of the present invention, the data of the 2000 grabs may be randomly disordered and divided into a tactile data training sample and a tactile data test sample according to a ratio of 8:2.
The foregoing examples are merely illustrative of embodiments of the present invention and are not intended to limit the embodiments of the present invention in any way.
It should be noted that, in order to facilitate the subsequent verification/test of the recognition result of the manipulator grabbing stability recognition model, in the embodiment of the present invention, each tactile data sample may be labeled based on the result in the shaking motion process.
For example, during a shaking motion, the data may be labeled "successful" assuming that the cylindrical object is successful after a certain shaking. Or if the result after a certain shaking is failure, the data may be labeled "failure".
Alternatively, for ease of recording, the label of the tactile data sample may be recorded as 0 or 1, where 1 indicates successful capture and 0 indicates failed capture.
Matrix determination module 42 is configured to determine an adjacency matrix for the haptic data training samples.
The method for determining the adjacency matrix of the training sample of the haptic data by the matrix determining module 42 is the same as that of the above-mentioned embodiment 1, and the relevant content of steps 120 to 124 in the above-mentioned embodiment 1 may be referred to, and will not be repeated here.
The model training module 43 is configured to train a preset network model according to the adjacency matrix to obtain the manipulator grabbing stability recognition model, where the preset network model includes a graph convolution layer, an activation layer, a feature fusion layer and a full connection layer, a convolution kernel of the graph convolution layer is 1*1, and the number of outputs of the full connection layer is 2.
It should be noted that, in order to learn the deeper features of the haptic data, the preset network model may include a graph convolution layer, where the convolution kernel of the graph convolution layer may have a size of 1*1. In the embodiment of the invention, after the graph convolution layer is adopted, graph convolution operation processing can be carried out on the input data, so that the deeper features of the tactile data can be learned.
Optionally, the preset network model may further include an active layer and a full connection layer in addition to the graph roll layer. Wherein the activation function used by the activation layer is a ReLU; the number of outputs of the fully connected layer may be 2, which may correspond to the stability and instability of the haptic data training samples.
In addition, considering the problem that in the related art, a single sensor model is generally adopted, so that the accuracy of the identification result is low easily, in the embodiment of the invention, the preset network model can also comprise a feature fusion layer, and after the feature fusion layer is used, on one hand, the identification result can be optimized through a fusion technology; on the other hand, the multiple touch sensors can be subjected to feature fusion, and compared with a single sensor model, the accuracy of the recognition result can be improved.
In the embodiment of the invention, if the preset network model comprises a graph convolution layer, an activation layer, a full connection layer and a feature fusion layer, the manipulator grabbing stability recognition model can comprise the graph convolution layer, the activation layer, the full connection layer and/or the feature fusion layer; the convolution kernel of the graph roll layer may have a size of 1*1, and the number of outputs of the full connection layer may be 2.
By adopting the device provided by the embodiment of the invention, the adjacency matrix of the tactile data training sample can be determined, and then the preset network model is trained according to the adjacency matrix to obtain the manipulator grabbing stability recognition model, so that the correlation between the tactile data can be fully extracted when the model provided by the embodiment of the invention is used for recognition, the characteristics extracted based on the tactile data are more attached to the actual distribution condition of the tactile sensing points on the tactile sensor, and the accuracy of the recognition result is further ensured.
Example 5
A fifth embodiment of the present specification relates to an electronic apparatus as shown in fig. 5. At the hardware level, the electronic device comprises a processor, optionally an internal bus, a network interface, a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory (non-volatilememory), such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.
The processor, network interface, and memory may be interconnected by an internal bus, which may be an ISA (Industry Standard Architecture ) bus, a PCI (PeripheralComponent Interconnect, peripheral component interconnect standard) bus, or EISA (Extended Industry StandardArchitecture ) bus, among others. The buses may be classified as address buses, data buses, control buses, etc. For ease of illustration, fig. 5 is represented by only one double-headed arrow, but does not represent only one bus or one type of bus.
And the memory is used for storing programs. In particular, the program may include program code including computer-operating instructions. The memory may include memory and non-volatile storage and provide instructions and data to the processor.
The processor reads the corresponding computer program from the nonvolatile memory to the memory and then runs the computer program to form the manipulator grabbing stability recognition device on a logic level. The processor is used for executing the programs stored in the memory and is specifically used for executing the following operations:
collecting target touch data of an object to be identified;
Inputting target touch data into a manipulator grabbing stability recognition model obtained through pre-training to obtain an output result; the manipulator grabbing stability recognition model is obtained through training based on an adjacency matrix of the tactile data, and the adjacency matrix of the tactile data is determined based on a graph structure of the tactile data;
based on the output result, the gripping stability of the manipulator is identified.
In an alternative embodiment, before the haptic data is input into the manipulator grabbing stability recognition model obtained by training in advance, the method further comprises:
Obtaining a tactile data training sample and a tactile data testing sample;
determining an adjacency matrix for the haptic data training samples;
training a preset network model according to the adjacency matrix to obtain a manipulator grabbing stability recognition model; the method comprises the steps that a preset network model is used for identifying grabbing stability of a manipulator;
and testing the recognition result of the manipulator grabbing stability recognition model based on the touch data test sample.
In an alternative embodiment, obtaining a haptic data training sample and a haptic data test sample includes:
The method comprises the steps of controlling a manipulator to execute grabbing actions for preset times on preset articles with random grabbing points and grabbing forces, wherein fingers of the manipulator comprise at least one touch sensor;
Controlling a mechanical arm to shake the preset article grabbed each time, and collecting touch data in the shaking process;
and dividing the tactile data in the shaking motion process into a tactile data training sample and a tactile data testing sample according to a preset proportion based on the tactile data in the shaking motion process and the result of the shaking motion.
In an alternative embodiment, determining an adjacency matrix of haptic data includes:
Determining an electrode array corresponding to the tactile data;
determining a graph structure matched with the tactile data through a k nearest neighbor algorithm according to the distribution information of the electrode array in the tactile data acquisition equipment;
based on the graph structure and the position coordinates of the contacts in the haptic data, an adjacency matrix of the haptic data is determined.
In an alternative embodiment, determining a graph structure matching the haptic data by a k-nearest neighbor algorithm based on distribution information of the electrode array at the haptic data acquisition device includes:
calculating Euclidean distance between contacts in the touch data;
And connecting each contact with a preset number of adjacent contacts closest to each contact according to the Euclidean distance to obtain a graph structure matched with the tactile data.
In an alternative embodiment, collecting haptic data of a target object to be identified includes:
the method comprises the steps of controlling a manipulator to grab a target object at random grabbing points and grabbing forces, wherein fingers of the manipulator comprise at least one touch sensor;
and collecting interaction data in the grabbing process as tactile data of the target object to be identified.
In an alternative embodiment, the manipulator grabbing stability recognition model comprises a graph convolution layer, an activation layer, a feature fusion layer and a full connection layer; the convolution kernel size of the graph roll lamination is 1*1, and the number of outputs of the full connection layer is 2.
Or the processor reads the corresponding computer program from the nonvolatile memory to the memory and then runs the computer program, and a manipulator grabbing stability recognition device is formed on a logic level. The processor is used for executing the programs stored in the memory and is specifically used for executing the following operations:
Obtaining a tactile data training sample and a tactile data testing sample;
determining an adjacency matrix for the haptic data training samples;
training a preset network model according to the adjacency matrix to obtain a manipulator grabbing stability recognition model; the preset network model comprises a graph convolution layer, an activation layer, a feature fusion layer and a full connection layer, wherein the convolution kernel of the graph convolution layer is 1*1, and the number of outputs of the full connection layer is 2.
The manipulator grabbing stability recognition method or the training method of the manipulator grabbing stability recognition model provided in the specification can be applied to a processor or realized by the processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or by instructions in the form of software. The processor may be a general-purpose processor, including a central processing unit (CentralProcessing Unit, CPU), a network processor (Network Processor, NP), etc.; but may also be a digital signal Processor (DIGITAL SIGNAL Processor, DSP), application specific integrated circuit (Application Specific IntegratedCircuit, ASIC), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components. The various methods, steps and logic blocks disclosed in the embodiments of this specification may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The steps of a method disclosed in connection with the embodiments of the present specification may be embodied directly in hardware, in a decoded processor, or in a combination of hardware and software modules in a decoded processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads the information in the memory and, in combination with its hardware, performs the steps of the above method.
The embodiments of the present specification also propose a computer-readable storage medium storing one or more programs, the one or more programs including instructions, which when executed by an electronic device comprising a plurality of application programs, enable the electronic device to perform a method of robot gripping stability recognition, and in particular to perform:
collecting target touch data of an object to be identified;
Inputting target touch data into a manipulator grabbing stability recognition model obtained through pre-training to obtain an output result; the manipulator grabbing stability recognition model is obtained through training based on an adjacency matrix of the tactile data, and the adjacency matrix of the tactile data is determined based on a graph structure of the tactile data;
based on the output result, the gripping stability of the manipulator is identified.
In an alternative embodiment, before the haptic data is input into the manipulator grabbing stability recognition model obtained by training in advance, the method further comprises:
Obtaining a tactile data training sample and a tactile data testing sample;
determining an adjacency matrix for the haptic data training samples;
training a preset network model according to the adjacency matrix to obtain a manipulator grabbing stability recognition model; the method comprises the steps that a preset network model is used for identifying grabbing stability of a manipulator;
and testing the recognition result of the manipulator grabbing stability recognition model based on the touch data test sample.
In an alternative embodiment, obtaining a haptic data training sample and a haptic data test sample includes:
The method comprises the steps of controlling a manipulator to execute grabbing actions for preset times on preset articles with random grabbing points and grabbing forces, wherein fingers of the manipulator comprise at least one touch sensor;
Controlling a mechanical arm to shake the preset article grabbed each time, and collecting touch data in the shaking process;
and dividing the tactile data in the shaking motion process into a tactile data training sample and a tactile data testing sample according to a preset proportion based on the tactile data in the shaking motion process and the result of the shaking motion.
In an alternative embodiment, determining an adjacency matrix of haptic data includes:
Determining an electrode array corresponding to the tactile data;
determining a graph structure matched with the tactile data through a k nearest neighbor algorithm according to the distribution information of the electrode array in the tactile data acquisition equipment;
based on the graph structure and the position coordinates of the contacts in the haptic data, an adjacency matrix of the haptic data is determined.
In an alternative embodiment, determining a graph structure matching the haptic data by a k-nearest neighbor algorithm based on distribution information of the electrode array at the haptic data acquisition device includes:
calculating Euclidean distance between contacts in the touch data;
And connecting each contact with a preset number of adjacent contacts closest to each contact according to the Euclidean distance to obtain a graph structure matched with the tactile data.
In an alternative embodiment, collecting haptic data of a target object to be identified includes:
the method comprises the steps of controlling a manipulator to grab a target object at random grabbing points and grabbing forces, wherein fingers of the manipulator comprise at least one touch sensor;
and collecting interaction data in the grabbing process as tactile data of the target object to be identified.
In an alternative embodiment, the manipulator grabbing stability recognition model comprises a graph convolution layer, an activation layer, a feature fusion layer and a full connection layer; the convolution kernel size of the graph roll lamination is 1*1, and the number of outputs of the full connection layer is 2.
Or the computer-readable storage medium stores one or more programs, the one or more programs comprising instructions, which when executed by an electronic device comprising a plurality of application programs, enable the electronic device to perform a method of training a manipulator grasp stability recognition model, and in particular to perform:
Obtaining a tactile data training sample and a tactile data testing sample;
determining an adjacency matrix for the haptic data training samples;
training a preset network model according to the adjacency matrix to obtain a manipulator grabbing stability recognition model; the preset network model comprises a graph convolution layer, an activation layer, a feature fusion layer and a full connection layer, wherein the convolution kernel of the graph convolution layer is 1*1, and the number of outputs of the full connection layer is 2.
The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer.
For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functions of each element may be implemented in one or more software and/or hardware elements when implemented in the present specification.
It will be apparent to one of ordinary skill in the art that embodiments of the present description may be provided as a method, apparatus, or computer program product. Accordingly, the present specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present description can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
The present description is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the specification. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.