Disclosure of Invention
In view of the above-mentioned shortcomings of the prior art, the present invention provides a target trajectory matching method, device, machine-readable medium and apparatus, which are used to solve the problems of the prior art.
To achieve the above and other related objects, the present invention provides a target trajectory matching method, including:
carrying out target detection and tracking on video data acquired by monitoring equipment to obtain a plurality of track segments;
carrying out frame-to-frame feature matching on any two track segments in the multiple track segments to obtain a feature similarity matrix;
converting the characteristic similarity matrix to obtain similarity scores of the two track segments;
and judging whether the two track segments belong to the track of the same target or not based on the similarity of the two track segments.
Optionally, the method further comprises:
acquiring the similarity score of each track segment and other track segments in the plurality of track segments;
and judging whether any two track segments belong to the track of the same target or not based on the similarity scores of each track segment and other track segments in the multiple track segments.
Optionally, the matching any two track segments of the multiple track segments to obtain a feature similarity matrix includes:
extracting features of any two sections of track segments to obtain a plurality of track feature vectors;
obtaining the similarity of each frame image of a first track segment in any two track segments and all frame images of a second track segment, and determining the characteristic similarity matrix; each row of the characteristic similarity matrix represents the similarity between each frame of the first track segment and all frames of the second track segment, and each column represents the similarity between each frame of the second track segment and all frames of the first track segment.
Optionally, the converting the feature similarity matrix to obtain a similarity score of two track segments includes:
performing maximum pooling operation on each row and each column of the feature similarity matrix respectively to obtain a first similarity score vector and a second similarity score vector;
respectively averaging the first similarity score vector and the second similarity score vector to obtain a first average similarity score and a second average similarity score;
and averaging the first average similarity score and the second average similarity score to obtain the similarity score of the two track segments.
Optionally, the converting the feature similarity matrix to obtain a similarity score of two track segments further includes:
and respectively calculating abnormal values in the first similarity score vector and the second similarity score vector, and eliminating the abnormal values.
Optionally, the outlier in the first similarity score vector and the second similarity score is calculated by a 3 σ criterion.
To achieve the above and other related objects, the present invention also provides a target trajectory matching apparatus, including:
the detection and tracking module is used for carrying out target detection and tracking on the video data acquired by the monitoring equipment to obtain a plurality of track segments;
the first matching module is used for carrying out frame-to-frame feature matching on any two track segments in the multi-segment track segments to obtain a feature similarity matrix;
the conversion module is used for converting the characteristic similarity matrix to obtain similarity scores of the two track segments;
and the first judging module is used for judging whether the two track segments belong to the track of the same target or not based on the similarity of the two track segments.
Optionally, the apparatus further comprises:
the similarity obtaining module is used for obtaining the similarity score of each track segment and other track segments in the plurality of track segments;
and the second judging module is used for judging whether any two track segments belong to the track of the same target or not based on the similarity scores of each track segment and other track segments in the multiple track segments.
Optionally, the first matching module comprises:
the characteristic extraction submodule is used for extracting the characteristics of any two sections of track fragments to obtain a plurality of track characteristic vectors;
the similarity operator module is used for acquiring the similarity of each frame image of the first track segment and all frame images of the second track segment in any two track segments and determining the characteristic similarity matrix; each row of the characteristic similarity matrix represents the similarity between each frame of the first track segment and all frames of the second track segment, and each column represents the similarity between each frame of the second track segment and all frames of the first track segment.
Optionally, the conversion module comprises:
the maximum pooling sub-module is used for performing maximum pooling operation on each row and each column of the feature similarity matrix respectively to obtain a first similarity score vector and a second similarity score vector;
the first averaging submodule is used for respectively averaging the first similarity score vector and the second similarity score vector to obtain a first average similarity score and a second average similarity score;
and the second averaging submodule is used for averaging the first average similarity score and the second average similarity score to obtain the similarity score of the two track segments.
Optionally, the conversion module further comprises:
and the abnormal value removing sub-module is used for respectively calculating the abnormal values in the first similarity score vector and the second similarity score vector and removing the abnormal values.
Optionally, the outlier in the first similarity score vector and the second similarity score is calculated by a 3 σ criterion.
To achieve the above and other related objects, the present invention also provides a target trajectory matching apparatus, including:
one or more processors; and
one or more machine-readable media having instructions stored thereon that, when executed by the one or more processors, cause the apparatus to perform one or more of the methods described previously.
To achieve the above objects and other related objects, the present invention also provides one or more machine-readable media having instructions stored thereon, which when executed by one or more processors, cause an apparatus to perform one or more of the methods described above.
As described above, the target trajectory matching method, device, machine-readable medium and apparatus provided by the present invention have the following beneficial effects:
the invention discloses a target track matching method, which comprises the following steps: carrying out target detection and tracking on video data acquired by monitoring equipment to obtain a plurality of track segments; carrying out frame-to-frame feature matching on any two track segments in the multiple track segments to obtain a feature similarity matrix; converting the characteristic similarity matrix to obtain similarity scores of the two track segments; and judging whether the two track segments belong to the track of the same target or not based on the similarity of the two track segments. The method considers the detail characteristics of the track and the overall characteristics of the track, fully utilizes the unique detail information of a single frame in the track and the associated overall information among multiple frames, and greatly improves the track matching precision; and after feature extraction is carried out based on the single-frame image, feature matching between the inter-track frames and the frames is carried out, so that the method is more efficient and easy to realize.
Detailed Description
The embodiments of the present invention are described below with reference to specific embodiments, and other advantages and effects of the present invention will be easily understood by those skilled in the art from the disclosure of the present specification. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention. It is to be noted that the features in the following embodiments and examples may be combined with each other without conflict.
It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present invention, and the components related to the present invention are only shown in the drawings rather than drawn according to the number, shape and size of the components in actual implementation, and the type, quantity and proportion of the components in actual implementation may be changed freely, and the layout of the components may be more complicated.
As shown in fig. 1, a target trajectory matching method includes:
s11, carrying out target detection and tracking on the video data acquired by the monitoring equipment to obtain a plurality of track segments;
s12, carrying out frame-to-frame feature matching on any two track segments in the multi-segment track segments to obtain a feature similarity matrix;
s13, converting the characteristic similarity matrix to obtain similarity scores of the two track segments;
s14, judging whether the two track segments belong to the same target track or not based on the similarity of the two track segments.
The invention considers the detail characteristic of the track and the integral characteristic of the track, and fully utilizes the unique detail information of a single frame and the mutually related integral information among multiple frames in the track, so that the track matching precision is greatly improved.
In step S11, the video data may be obtained by one monitoring device or different monitoring devices. Taking different monitoring devices as examples, the monitoring sequence devices are respectively a first camera and a second camera, the first camera and the second camera respectively shoot a first video and a second video, and after a target moves from a shooting area of the first camera to a shooting area of the second camera, the target is included in both the first video and the second video. By comparing the moving tracks of the target, whether the target in the first video and the target in the second video are the moving tracks of the same target can be determined.
The first video shot by the first camera or the second video shot by the second camera comprises a plurality of frames of images, and one target in each frame of image in the plurality of frames of images can be regarded as one point. This point can be considered as the location of the object. The point may be a center point of the object or a location of the object. By the method, the track of the target in the first video shot by the first camera and the track of the target in the second video shot by the second camera can be determined.
Before feature matching is performed on the two track segments, feature extraction needs to be performed on the two track segments. The extraction of the track segments can be realized by carrying out feature extraction on the two track segments through a trained target re-recognition convolutional neural network model. In the process of re-identifying the convolutional neural network model for training the target, a single-frame pedestrian image is taken as input, and a feature with the length of D is taken as output. The process of extracting the features of the track segment by the target re-identification convolutional neural network model is shown in fig. 2.
And performing feature extraction on the track segment by using the trained target re-recognition convolutional neural network model to obtain feature representation of each frame of image. And then, performing frame-to-frame feature matching on the two track segments by using a feature matching algorithm to obtain a feature similarity matrix between the two tracks, wherein the feature similarity matrix is shown in fig. 3. And after the characteristic similarity matrix is obtained, converting the characteristic similarity matrix to obtain a similarity score, and judging whether the two track segments belong to the track of the same target or not according to the similarity score. For example, if the similarity score is greater than the set similarity score threshold, the two track segments are considered to belong to the same target track, and otherwise, the two track segments belong to different target tracks.
In the foregoing, whether two track segments belong to the same target track segment is determined, and if more than two track segments are included, assuming that the total track segment number is T, for any two track segments Ti、TjAssuming that the number of images contained in each is M, N, the track segment T is judgedi、TjIn the process of judging whether the tracks belong to the same target or not, a characteristic similarity matrix with the shape of M multiplied by N can be obtained, a score is obtained after the characteristic similarity matrix is converted, and finally, pairwise matching is carried out on all the tracks to obtain a characteristic similarity matrix S, S with the shape of T multiplied by Ti,jRepresenting a track TiAnd TjThe similarity score of (a).
In an embodiment, as shown in fig. 4, the matching any two track segments of the multiple track segments to obtain a feature similarity matrix includes:
s41, extracting the features of any two track segments to obtain a plurality of track feature vectors;
s42, obtaining the similarity of each frame image of the first track segment and all frame images of the second track segment in any two track segments, and determining the characteristic similarity matrix; each row of the characteristic similarity matrix represents the similarity between each frame of the first track segment and all frames of the second track segment, and each column represents the similarity between each frame of the second track segment and all frames of the first track segment.
In an embodiment, as shown in fig. 5, the converting the feature similarity matrix to obtain the similarity score of two track segments includes:
s51, performing maximum pooling operation on each line and each column of the feature similarity matrix respectively to obtain a first similarity score vector and a second similarity score vector;
s52, averaging the first similarity score vector and the second similarity score vector respectively to obtain a first average similarity score and a second average similarity score;
s53, averaging the first average similarity score and the second average similarity score to obtain the similarity score of the two track segments.
Fig. 6 shows a process of converting the feature similarity matrix. As shown in fig. 6, each row of the feature similarity matrix represents the similarity between each frame in the track 1 and all frames in the track 2, and each column represents the similarity between each frame in the track 2 and all frames in the track 1. In order to comprehensively compare the two tracks, the feature similarity matrix is converted, that is, the maximum pooling operation is performed on each row and each column of the feature similarity matrix in sequence, that is, each row and each column take a maximum value, so as to obtain an M-dimensional and an N-dimensional similarity score vector, which indicates that for each frame of the track 1, a frame most similar to the frame is found in the track 2. And finally, averaging the two average similarity scores to obtain the final similarity score. Whether the two track segments belong to the track of the same target or not can be judged through the similarity score.
In an embodiment, the abnormal values in the first similarity score vector and the second similarity score vector may be calculated respectively to determine whether there is an abnormal value far away from the overall distribution (for example, a score significantly lower than other scores in the vector indicates that the frame picture is not similar to all frames in another track, or indicates that the frame picture has a low similarity to other frames in the track where the frame picture is located, and is an outlier), and if there is an abnormal value, the abnormal value is removed, and the remaining score average after removing the noise is taken as the final similarity of the two tracks.
In an embodiment, the outlier in the first similarity score vector and the second similarity score is calculated by a 3 σ criterion.
The 3 σ criterion is as follows:
1. calculating a standard deviation;
2. comparing the absolute value of the difference between each sampling point and the mean value with 3 times of standard deviation, and rejecting the samples if the absolute value is more than 3 times of standard deviation;
3. and (4) repeating the step 1 until the circulation has no rejection.
When the number of pictures in a section of track is too small (N <10), a shielding effect caused by a single-side abnormal value is generated (namely, the average value is greatly influenced by the abnormal value, so that the standard deviation cannot reflect the actual distribution), and the abnormal value cannot be accurately removed, so that a median is selected to replace the average value in the implementation process.
For a section of track of a target, firstly, extracting features for each frame of image by using a target re-identification network model based on the image, not performing merging operation of averaging or maximum value acquisition, but retaining the features of all track images and performing retrieval comparison one by one; and then, respectively carrying out feature matching on the images of the two tracks to obtain a similarity matrix, carrying out row maximum pooling, column maximum pooling and noise filtering on the matrix, then carrying out average pooling to obtain the overall similarity of the two tracks, and carrying out overall track comparison according to the similarity. Therefore, the detailed characteristics of the track are considered, the overall characteristics of the track are considered, the unique detailed information of a single frame in the track and the mutually-associated overall information among multiple frames are fully utilized, and the track matching precision is greatly improved; and the features are extracted based on the single-frame image, and then the inter-track features are matched, so that the scheme is more efficient and easy to realize. Furthermore, noise filtering is embedded in the matching process, so that the influence of noise on the matching result can be eliminated in real time.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
As shown in fig. 7, a target trajectory matching apparatus includes:
the detection andtracking module 71 is configured to perform target detection and tracking on video data acquired by the monitoring device to obtain a plurality of track segments;
thefirst matching module 72 is configured to perform frame-to-frame feature matching on any two track segments of the multiple track segments to obtain a feature similarity matrix;
aconversion module 73, configured to convert the feature similarity matrix to obtain a similarity score of two track segments;
the first judging module 74 is configured to judge whether the two track segments belong to the same target track based on the similarity between the two track segments.
The invention considers the detail characteristic of the track and the integral characteristic of the track, and fully utilizes the unique detail information of a single frame and the mutually related integral information among multiple frames in the track, so that the track matching precision is greatly improved.
In step S11, the video data may be obtained by one monitoring device or different monitoring devices. Taking different monitoring devices as examples, the monitoring sequence devices are respectively a first camera and a second camera, the first camera and the second camera respectively shoot a first video and a second video, and after a target moves from a shooting area of the first camera to a shooting area of the second camera, the target is included in both the first video and the second video. By comparing the moving tracks of the target, whether the target in the first video and the target in the second video are the moving tracks of the same target can be determined.
The first video shot by the first camera or the second video shot by the second camera comprises a plurality of frames of images, and one target in each frame of image in the plurality of frames of images can be regarded as one point. This point can be considered as the location of the object. The point may be a center point of the object or a location of the object. By the method, the track of the target in the first video shot by the first camera and the track of the target in the second video shot by the second camera can be determined.
Before feature matching is performed on the two track segments, feature extraction needs to be performed on the two track segments. The extraction of the track segments can be realized by carrying out feature extraction on the two track segments through a trained target re-recognition convolutional neural network model. In the process of re-identifying the convolutional neural network model for training the target, a single-frame pedestrian image is taken as input, and a feature with the length of D is taken as output. The process of extracting the features of the track segment by the target re-identification convolutional neural network model is shown in fig. 2.
And performing feature extraction on the track segment by using the trained target re-recognition convolutional neural network model to obtain feature representation of each frame of image. And then, performing frame-to-frame feature matching on the two track segments by using a feature matching algorithm to obtain a feature similarity matrix between the two tracks, wherein the feature similarity matrix is shown in fig. 3. And after the characteristic similarity matrix is obtained, converting the characteristic similarity matrix to obtain a similarity score, and judging whether the two track segments belong to the track of the same target or not according to the similarity score. For example, if the similarity score is greater than the set similarity score threshold, the two track segments are considered to belong to the same target track, and otherwise, the two track segments belong to different target tracks.
In the foregoing, whether two track segments belong to the same target track segment is determined, and if two track segments are includedTrack segments above the segment, assuming that the total track segment number is T, for any two track segments Ti、TjAssuming that the number of images contained in each is M, N, the track segment T is judgedi、TjIn the process of judging whether the tracks belong to the same target or not, a characteristic similarity matrix with the shape of M multiplied by N can be obtained, a score is obtained after the characteristic similarity matrix is converted, and finally, pairwise matching is carried out on all the tracks to obtain a characteristic similarity matrix S, S with the shape of T multiplied by Ti,jRepresenting a track TiAnd TjThe similarity score of (a).
In one embodiment, the apparatus further comprises:
the similarity obtaining module is used for obtaining the similarity score of each track segment and other track segments in the plurality of track segments;
and the second judging module is used for judging whether any two track segments belong to the track of the same target or not based on the similarity scores of each track segment and other track segments in the multiple track segments.
In one embodiment, as shown in fig. 8, the first matching module includes:
afeature extraction submodule 81, configured to perform feature extraction on any two segments of track segments to obtain a plurality of track feature vectors;
thesimilarity operator module 82 is configured to obtain similarities of each frame image of the first track segment in the any two track segments and all frame images of the second track segment, and determine the feature similarity matrix; each row of the characteristic similarity matrix represents the similarity between each frame of the first track segment and all frames of the second track segment, and each column represents the similarity between each frame of the second track segment and all frames of the first track segment.
In one embodiment, as shown in fig. 9, the conversion module includes:
themaximum pooling sub-module 91 is configured to perform maximum pooling operation on each row and each column of the feature similarity matrix to obtain a first similarity score vector and a second similarity score vector;
a first averagingsubmodule 92, configured to average the first similarity score vector and the second similarity score vector to obtain a first average similarity score and a second average similarity score;
and a second averaging submodule 93, configured to average the first average similarity score and the second average similarity score to obtain a similarity score of the two track segments.
Fig. 6 shows a process of converting the feature similarity matrix. As shown in fig. 6, each row of the feature similarity matrix represents the similarity between each frame in the track 1 and all frames in the track 2, and each column represents the similarity between each frame in the track 2 and all frames in the track 1. In order to comprehensively compare the two tracks, the feature similarity matrix is converted, that is, the maximum pooling operation is performed on each row and each column of the feature similarity matrix in sequence, that is, each row and each column take a maximum value, so as to obtain an M-dimensional and an N-dimensional similarity score vector, which indicates that for each frame of the track 1, a frame most similar to the frame is found in the track 2. And finally, averaging the two average similarity scores to obtain the final similarity score. Whether the two track segments belong to the track of the same target or not can be judged through the similarity score.
In one embodiment, the conversion module further comprises:
and the abnormal value removing sub-module is used for respectively calculating the abnormal values in the first similarity score vector and the second similarity score vector and removing the abnormal values.
And judging whether an abnormal value far away from the overall distribution exists in the image by calculating the abnormal value (for example, a score which is obviously lower than other scores in the vector indicates that the image of the frame is not similar to all frames in another track and also indicates that the image of the frame has low similarity with other frames in the track where the image of the frame is located and is an outlier), if the abnormal value exists, rejecting the abnormal value, and taking the average value of the remaining scores after removing the noise as the final similarity of the two tracks.
In an embodiment, the outlier in the first similarity score vector and the second similarity score is calculated by a 3 σ criterion.
The 3 σ criterion is as follows:
1. calculating a standard deviation;
2. comparing the absolute value of the difference between each sampling point and the mean value with 3 times of standard deviation, and rejecting the samples if the absolute value is more than 3 times of standard deviation;
3. and (4) repeating the step 1 until the circulation has no rejection.
When the number of pictures in a section of track is too small (N <10), a shielding effect caused by a single-side abnormal value is generated (namely, the average value is greatly influenced by the abnormal value, so that the standard deviation cannot reflect the actual distribution), and the abnormal value cannot be accurately removed, so that a median is selected to replace the average value in the implementation process.
For a section of track of a target, firstly, extracting features for each frame of image by using a target re-identification network model based on the image, not performing merging operation of averaging or maximum value acquisition, but retaining the features of all track images and performing retrieval comparison one by one; and then, respectively carrying out feature matching on the images of the two tracks to obtain a similarity matrix, carrying out row maximum pooling, column maximum pooling and noise filtering on the matrix, then carrying out average pooling to obtain the overall similarity of the two tracks, and carrying out overall track comparison according to the similarity. Therefore, the detailed characteristics of the track are considered, the overall characteristics of the track are considered, the unique detailed information of a single frame in the track and the mutually-associated overall information among multiple frames are fully utilized, and the track matching precision is greatly improved; and the features are extracted based on the single-frame image, and then the inter-track features are matched, so that the scheme is more efficient and easy to realize. Furthermore, noise filtering is embedded in the matching process, so that the influence of noise on the matching result can be eliminated in real time.
An embodiment of the present application further provides an apparatus, which may include: one or more processors; and one or more machine readable media having instructions stored thereon that, when executed by the one or more processors, cause the apparatus to perform the method of fig. 1. In practical applications, the device may be used as a terminal device, and may also be used as a server, where examples of the terminal device may include: the mobile terminal includes a smart phone, a tablet computer, an electronic book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop, a vehicle-mounted computer, a desktop computer, a set-top box, an intelligent television, a wearable device, and the like.
The present application further provides a non-transitory readable storage medium, where one or more modules (programs) are stored in the storage medium, and when the one or more modules are applied to a device, the device may be caused to execute instructions (instructions) of steps included in the method in fig. 1 according to the present application.
Fig. 10 is a schematic diagram of a hardware structure of a terminal device according to an embodiment of the present application. As shown, the terminal device may include: aninput device 1100, afirst processor 1101, anoutput device 1102, afirst memory 1103, and at least onecommunication bus 1104. Thecommunication bus 1104 is used to implement communication connections between the elements. Thefirst memory 1103 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and thefirst memory 1103 may store various programs for performing various processing functions and implementing the method steps of the present embodiment.
Alternatively, thefirst processor 1101 may be, for example, a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic components, and thefirst processor 1101 is coupled to theinput device 1100 and theoutput device 1102 through a wired or wireless connection.
Optionally, theinput device 1100 may include a variety of input devices, such as at least one of a user-oriented user interface, a device-oriented device interface, a software programmable interface, a camera, and a sensor. Optionally, the device interface facing the device may be a wired interface for data transmission between devices, or may be a hardware plug-in interface (e.g., a USB interface, a serial port, etc.) for data transmission between devices; optionally, the user-facing user interface may be, for example, a user-facing control key, a voice input device for receiving voice input, and a touch sensing device (e.g., a touch screen with a touch sensing function, a touch pad, etc.) for receiving user touch input; optionally, the programmable interface of the software may be, for example, an entry for a user to edit or modify a program, such as an input pin interface or an input interface of a chip; theoutput devices 1102 may include output devices such as a display, audio, and the like.
In this embodiment, the processor of the terminal device includes a module for executing functions of each module in each device, and specific functions and technical effects may refer to the foregoing embodiments, which are not described herein again.
Fig. 11 is a schematic hardware structure diagram of a terminal device according to an embodiment of the present application. FIG. 11 is a specific embodiment of the implementation of FIG. 10. As shown, the terminal device of the present embodiment may include asecond processor 1201 and asecond memory 1202.
Thesecond processor 1201 executes the computer program code stored in thesecond memory 1202 to implement the method described in fig. 1 in the above embodiment.
Thesecond memory 1202 is configured to store various types of data to support operations at the terminal device. Examples of such data include instructions for any application or method operating on the terminal device, such as messages, pictures, videos, and so forth. Thesecond memory 1202 may include a Random Access Memory (RAM) and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
Optionally, asecond processor 1201 is provided in theprocessing assembly 1200. The terminal device may further include:communication component 1203,power component 1204,multimedia component 1205,speech component 1206, input/output interfaces 1207, and/orsensor component 1208. The specific components included in the terminal device are set according to actual requirements, which is not limited in this embodiment.
Theprocessing component 1200 generally controls the overall operation of the terminal device. Theprocessing assembly 1200 may include one or moresecond processors 1201 to execute instructions to perform all or part of the steps of the data processing method described above. Further, theprocessing component 1200 can include one or more modules that facilitate interaction between theprocessing component 1200 and other components. For example, theprocessing component 1200 can include a multimedia module to facilitate interaction between themultimedia component 1205 and theprocessing component 1200.
Thepower supply component 1204 provides power to the various components of the terminal device. Thepower components 1204 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the terminal device.
Themultimedia components 1205 include a display screen that provides an output interface between the terminal device and the user. In some embodiments, the display screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the display screen includes a touch panel, the display screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.
Thevoice component 1206 is configured to output and/or input voice signals. For example, thevoice component 1206 includes a Microphone (MIC) configured to receive external voice signals when the terminal device is in an operational mode, such as a voice recognition mode. The received speech signal may further be stored in thesecond memory 1202 or transmitted via thecommunication component 1203. In some embodiments, thespeech component 1206 further comprises a speaker for outputting speech signals.
The input/output interface 1207 provides an interface between theprocessing component 1200 and peripheral interface modules, which may be click wheels, buttons, etc. These buttons may include, but are not limited to: a volume button, a start button, and a lock button.
Thesensor component 1208 includes one or more sensors for providing various aspects of status assessment for the terminal device. For example, thesensor component 1208 may detect an open/closed state of the terminal device, relative positioning of the components, presence or absence of user contact with the terminal device. Thesensor assembly 1208 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact, including detecting the distance between the user and the terminal device. In some embodiments, thesensor assembly 1208 may also include a camera or the like.
Thecommunication component 1203 is configured to facilitate communications between the terminal device and other devices in a wired or wireless manner. The terminal device may access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In one embodiment, the terminal device may include a SIM card slot therein for inserting a SIM card therein, so that the terminal device may log onto a GPRS network to establish communication with the server via the internet.
From the above, thecommunication component 1203, thevoice component 1206, the input/output interface 1207 and thesensor component 1208 involved in the embodiment of fig. 11 can be implemented as the input device in the embodiment of fig. 10.
The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical spirit of the present invention be covered by the claims of the present invention.