Detailed Description
Reference will now be made in detail to the preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts.
Fig. 1 is a block diagram illustrating a motion tracking system 10 according to one of the exemplary embodiments of the present invention. Referring to fig. 1, a motion tracking system 10 includes, but is not limited to, three or more multiple motion sensing devices 100 and a computing device 200. The motion tracking system 10 may be adapted for use with VR, AR, MR, XR or other reality-related technologies.
Each motion sensing device 100 includes, but is not limited to, a wireless transceiver 110 and a motion sensor 130. The motion sensing device 100 may be a handheld controller or a wearable device, such as a wearable controller, a smart watch, an ankle sensor, a belt, a head-mounted display (HMD), or the like. In one embodiment, each motion sensing device 100 may be worn on a human body part of a user. The body part may be a hand, head, ankle, leg, waist or other part.
The wireless transceiver 110 may be a communication transceiver compatible with bluetooth, wi-Fi, IR, RFID, or other wireless communication technologies. In one embodiment, the wireless transceiver 110 is used to transmit and/or receive wireless signals with the wireless transceivers 110 of other motion sensing devices 100, and the first sensed data of the sequence will be generated based on the wireless signals transmitted between the motion sensing devices 100. A detailed flow for generating the first sensed data of the sequence will be described later.
The motion sensor 130 may be an accelerometer, a gyroscope, a magnetometer, an Inertial Measurement Unit (IMU), or any combination of the foregoing. In an embodiment, the motion sensor 130 is configured to sense motion of a corresponding human body part of a user wearing the motion sensing device 100 over a period of time, to generate a sequence of first sensed data from sensing results (such as acceleration, rotation, magnetic force, etc.) of the motion sensor 130 at a plurality of time points over the period of time. For one example, the first sensed data includes 3-degree of freedom (3-degree of freedom; 3-DoF) data, and the 3-DoF data relates to orientation information of a human body part in three-dimensional (3D) space, such as acceleration of yaw, roll, and pitch.
The computing device 200 includes, but is not limited to, a memory 240 and a processor 250. The computing device 200 may be one of a computer, server, smart phone, tablet, or motion sensing device 100.
The Memory 240 may be any type of fixed or removable Random Access Memory (RAM), read-only Memory (ROM), flash Memory or the like, or a combination thereof. Memory 240 may be used to store program codes, device configurations, buffer data, or persistent data (such as sensed data, motion information, distance relationships, etc.), and such data will be described later.
The processor 250 is connected to the memory 240, and the processor 250 is configured to load program codes stored in the memory 240, thereby executing the programs of the exemplary embodiments of the present invention. In one embodiment, the functions of the processor 150 are implemented using programmable units such as a central processing unit (central processing unit; CPU), microprocessor, microcontroller, digital Signal Processing (DSP) chip, field programmable gate array (field programmable GATE ARRAY; FPGA), or the like. In some embodiments, the functions of the processor 250 may also be implemented by a separate electronic device or integrated circuit (INTEGRATED CIRCUIT; IC), and the operations of the processor 250 may also be implemented by software.
It should be noted that processor 250 may or may not be disposed on the same device as one, part of, or all of motion sensing device 100. However, devices equipped with the motion sensor 130 and the processor 250, respectively, may also include communication transceivers with compatible communication technologies such as Bluetooth, wi-Fi, IR, or physical transmission lines to transmit/receive data to/from each other.
In one embodiment, the motion tracking system 10 may also include a Head Mounted Display (HMD) 300.HMD 300 may be worn on the head of a user. HMD 300 includes, but is not limited to, a wireless transceiver 310 and an image sensor 360.
The description of wireless transceiver 310 may refer to the description of wireless transceiver 110 and will be omitted. This means that the HMD 300 may communicate with the motion sensing device 100 through the wireless transceiver 310.
The image sensor 360 may be a camera, such as a monochrome or color camera, a depth camera, a video recorder, or other sensor capable of capturing images.
Fig. 2 is a schematic diagram illustrating a motion tracking system 20 according to one of the exemplary embodiments of the present invention. Referring to fig. 2, the motion tracking system 20 includes an HMD 300 and four motion sensing devices 100 (the four devices are two ankle sensors worn on a human body part B1 and a human body part B2 (i.e., two ankles) and two hand-held controllers worn on a human body part B3 and a human body part B4 (i.e., two hands)). In some embodiments, the HMD 300 may also include another motion sensor 130 (not shown) to obtain orientation information of the human body part B5 (i.e., the head). The processor 250 is embedded in the HMD 300.
It should be noted that the motion tracking system 20 is merely an example to illustrate the manner in which the motion sensing device 100, HMD 300, and processor 250 are positioned. However, many other embodiments of behavior understanding system 10 exist and the invention is not limited thereto.
In order to better understand the operational flow provided in one or more embodiments of the present invention, several embodiments will be illustrated below to explain the operational flow of motion tracking system 10 or motion tracking system 20 in detail. The devices and modules in the motion tracking system 10 or the motion tracking system 20 are applied in the following embodiments to explain the control methods provided herein. Each step of the control method may be adjusted according to actual implementation and should not be limited to what is described herein.
Fig. 3 is a flowchart illustrating a motion tracking method according to one of the exemplary embodiments of the present invention. Referring to fig. 3, the processor 250 may obtain first sensing data based on the motion sensors 130 disposed on the three motion sensing apparatuses 100 (step S310). In particular, with respect to the different types of motion sensors 130, acceleration, rotation, magnetic force, orientation, and/or 3-DoF of motion of the corresponding human body part in 2D/3D space may be obtained, and one or more sensing results of the motion sensors 130 will become sequential first sensing data of the human body part.
On the other hand, the processor 250 may obtain second sensing data based on wireless signals transmitted between the three motion sensing devices 100 (step S330). In one embodiment, processor 250 may obtain signal strengths of wireless signals from three or more motion sensing devices 100 at multiple points in time, and each signal strength will be recorded in memory 240 along with its corresponding transmitter and receiver. The signal strength may be a received signal strength indication (RECEIVED SIGNAL STRENGTH indication; RSSI), a received channel power parameter (RECEIVED CHANNEL power indicator; RCPI), a reference signal received power (REFERENCE SIGNAL RECEIVED power; RSRP), or the like. In one embodiment, the motion sensing device 100 may monitor the signal strength of all detectable wireless signals, and each wireless signal includes a specific identifier of a transmitter and/or receiver. The motion sensing device 100 may further feed back the signal strength with the corresponding identifier to the computing device 200. In another embodiment, the computing device 200 may monitor the signal strength of all detectable wireless signals, and the processor 250 records the signal strength and the corresponding identifier of the transmitter in the memory 240. The signal strength will be recorded for a period of time to generate a sequence of second sensed data. This means that the second sensed data contains signal intensities of a time-ordered sequence.
In some embodiments, processor 250 may further obtain third sensed data based on an image captured from image sensor 360. The third sensed data may be the images of the sequence and/or the sensed result of the pixels in the images (such as brightness, color, depth, etc.).
Next, the processor 250 may determine motion information of the user from the determining factors including the first sensing data and the second sensing data (step S350). In one embodiment, the motion information may include location information and orientation information. First, with respect to location information, in one embodiment, the processor 250 may determine location information of the user from the first sensed data. In this embodiment, the determining factor includes the first sensing data. The displacement of the corresponding human body part may be estimated by double integration of the detected accelerations of the human body part in three axes (i.e., the second sensed data) to further determine the position information based on the displacement. For example, the positional information may be coordinates on two or three axes, a position relative to a reference, or the like.
In another embodiment, the processor 250 may obtain the location information from second sensed data based on wireless signals between the three motion sensing devices 100. In this embodiment, the determining factor includes the second sensing data, and it should be noted that the signal strength of the wireless signal is related to the relative distance between the two motion sensing devices 100. Additionally, based on trilateration, three distances between three points may be used to determine relative position information for the three points. Assuming three motion sensing devices 100 as the aforementioned three points, processor 250 may determine the relative distance between each two motion sensing devices 100 as a distance relationship between motion sensing devices 100. Processor 250 may then generate location information for the tracking device based on the distance relationship and the trilateration.
Taking the motion tracking system 20 as an example, the processor 250 may obtain signal strengths of wireless signals from the motion sensing device 100 for the human body part B3 to the HMD 300 for the human body part B5 (in this embodiment, one of the motion sensing devices 100), wireless signals from the motion sensing device 100 for the human body part B4 to the HMD 300 for the human body part B5, and wireless signals from the motion sensing device 100 for the human body part B3 to the motion sensing device 100 for the human body part B4. Processor 250 may determine its distance relationship from the signal strengths and then generate location information for human body part B3 based on the distance relationship. The location information may be coordinates or relative locations.
It should be noted that embodiments are not limited to selecting three motion sensing devices 100. For example, the signal strengths of the wireless signals from the motion sensing apparatus 100 for the human body part B2 to the motion sensing apparatus 100 for the human body part B3, the wireless signals from the motion sensing apparatus 100 for the human body part B3 to the motion sensing apparatus 100 for the human body part B1, and the wireless signals from the motion sensing apparatus 100 for the human body part B2 to the motion sensing apparatus 100 for the human body part B1 may be used to estimate the position information of the human body part B1. The combination of motion sensing devices 100 may vary as desired.
In another embodiment, the processor 250 may determine the location information of the user according to the third sensing data. In this embodiment, the determining factor includes third sensing data. The position and displacement of the body part in the image can be used to determine position information in the real environment. Taking fig. 2 as an example, the sensed intensities and pixel positions corresponding to the human body part B4 in the image may be used to estimate depth information of the human body part B4 (i.e., distance relative to the HMD 300) and to estimate the 2D position of the human body part B4 on a plane parallel to the image sensor 360.
It should be noted that the accuracy of the location information based on only one sensing approach (e.g., based on one of the wireless transceiver 110, the motion sensor 130, and the image sensor 360) may be different. Thus, two or more sensing approaches may be used to determine position information for the corresponding body part.
In one embodiment, the processor 250 may obtain first location information from the first sensed data, second location information from the second sensed data, and adjusted location information from the first location information and the second location information. In this embodiment, the determining factor includes first sensing data and second sensing data. Processor 250 may determine the location information based on a combination of the first location information and the second location information. In some embodiments, the combination is a weighted combination. And determining the adjusted position information according to the weighted first position information and the weighted second position information.
In one embodiment, the weight of the weighted combination of the first location information and the second location information may be fixed. In another embodiment, the weight of the weighted combination of the first location information and the second location information may be varied. The weight of the first position information may be a value from 0 to 100%, and the weight of the second position information may be a value from 0 to 100%. However, the weights of the first position information and the second position information cannot be 0 at the same time.
It should be noted that in some embodiments, the location information determined based on the third sensed data generated by the image of the image sensor 360 may be more accurate than the location information determined based on the wireless transceiver 110 and/or the motion sensor 130. Thus, in one embodiment, the determinants may include the second sensed data and the third sensed data. The processor 250 may determine the location information according to a combination of location information obtained based on the first sensing data, the second sensing data, and the third sensing data.
In one embodiment, the processor 250 may obtain a first portion of the positional information from the second sensed data for a first duration, obtain a second portion of the positional information from the third sensed data for a second duration, and combine the first portion and the second portion of the positional information into combined positional information. The third sensed data of the detected human body part may be used to correct the position information based on the second sensed data for the first duration and the second duration. Processor 250 may determine combined location information based on the first location and the second location of the location information for different durations. For example, a position (1, 1) is determined based on the second sensed data for a first duration, another position (2, 1) is determined based on the third sensed data for a second duration, and the combined position information may be a displacement from position (1, 1) to position (2, 1).
In some embodiments, the processor 250 may determine the location information from a weighted combination of the second location information and the third location information. The weights of the second and third location information may be varied or fixed based on the actual situation. For example, the weight of the third location information may be greater than the weight of the second location information. In another embodiment, the location information is a weighted combination if the human body part is present in the third sensed data and the location is the second location information if the human body part is not present in the third sensed data.
In one embodiment, the image sensor 360 may be designed with a particular field of view (SPECIFIC FIELD of view; FOV). If a body part is positioned outside of the field of view of the image sensor 360, the processor 250 may not be able to determine motion information for that body part using only the third sensed data, and either the first sensed data or the second sensed data should be considered.
In one embodiment, the processor 250 may determine whether one human body part of the user is present in the third sensing data of the sequence, and determine whether to use the distance relationship between the three motion sensing devices 100 according to the result of the determination of the presence of the human body part to determine the position information based on trilateration. Processor 250 may use machine learning techniques, such as deep learning, artificial neural networks (ARTIFICIAL NEURAL NETWORK; ANN) or support vector machines (support vector machine; SVM), etc., to identify the target human body part in the third sensed data.
Fig. 4 is a schematic diagram illustrating a motion tracking method according to one of the exemplary embodiments of the present invention. Referring to fig. 4, it is assumed that the motion sensing device 100 of the human body part B4 is a target device. In this figure, the human body part B4 is present in the field of view FOV of the HMD 300 (i.e., the human body part B4 is present in the third sensed data).
Fig. 5 is a schematic diagram illustrating a motion tracking method according to one of the exemplary embodiments of the present invention. Referring to fig. 5, it is assumed that the motion sensing device 100 of the human body part B3 is a target device. In this figure, the human body part B5 is not present in the field of view FOV of the HMD 300 (i.e., the human body part B3 is not present in the third sensed data).
It should be noted that the size and shape of the fields of view shown in fig. 4 and 5 are merely examples, and may be modified based on actual needs.
Accordingly, the field of view of the image sensor 360 is used to determine whether a human body part is present in the third sensed data. In one embodiment, it is assumed that the human body part is located outside the field of view (i.e., not present in the third sensed data) for a first duration and the human body part is located within the field of view of the image sensor 360 (i.e., present in the third sensed data) for a second duration. In some embodiments, it is assumed that the body part is positioned inside the field of view of the image sensor 360 for a first duration and a second duration.
In another embodiment, the processor 250 may obtain the first position information according to the first sensing data, obtain the second position information according to the second sensing data, obtain the third position information according to the third sensing data, and obtain the adjusted position information according to the first position information, the second position information, and the third position information. In this embodiment, the determining factors include first sensing data, second sensing data and third sensing data. Processor 250 may determine the adjusted location information based on a combination of the first motion information, the second motion information, and the third location information.
In one embodiment, the combination is a weighted combination. Processor 250 may determine a first weight for the first location information and a second weight for the second location information based on the third location information. In one embodiment, the first weight and the second weight are iteratively varied. The third position information will be regarded as corrected position information for a duration in which the human body part is present in the third sensing data, and a weighted combination of the first position information and the second position information having the first weight and the second weight will be adjusted according to the third position information. It should be noted that the processor 250 may obtain the first parameter by multiplying the first weight with the first position information, obtain the second parameter by multiplying the second weight with the second position information, and obtain the adjusted position information by adding the first parameter to the second parameter to obtain the weighted combination.
In one embodiment, the first weight and the second weight for the subsequent point in time may be determined based on an equation in which the third location information is equal to a weighted combination of the first location information and the second location information for the previous point in time. For example, at a third point in time, in the 3-dimensional coordinate system, the first weight is 0.5 and the second weight is 0.5, the first location information is (6, 6) and the second location information is (10,10,10), and the adjusted location information will be (8, 8). If the third location information is (7, 7), the first weight and the second weight at the fourth point in time will be determined to be 0.75 and 0.25, respectively. Next, at a fourth point in time, if the first location information is (7,6,6) and the second location information is (12,10,10) in the 3-dimensional coordinate system, the adjusted location information will be (8.25,7,7).
In another embodiment, the first weight and the second weight of the current point in time may be determined based on an equation in which the third location information is equal to a weighted combination of the first location information and the second location information of the current point in time. For example, at a second point in time, in a 3-dimensional coordinate system, the first location information is (6, 6) and the second location information is (10,10,10). If the third location information is (7, 7), the first weight and the second weight at the second point in time will be determined to be 0.75 and 0.25, respectively. Then, the adjusted position information at the second point in time will be determined as (7, 7).
In some embodiments, the first weight and the second weight are fixed if a human body part of the user is not present in the third sensed data. If the body part is located outside the field of view of the image sensor 360, the third weight and the second weight will be the same as the previous first weight and second weight of the previous point in time when the body part of the user is still present in the third sensed data. For example, at a first point in time, the human body part is located within the field of view of the image sensor 360, and the first weight is 0.5 and the second weight is 0.5. Next, at a second point in time, the human body part is located outside the field of view of the image sensor 360. As with the first weight and the second weight at the first point in time, at the second point in time the first weight will be 0.5 and the second weight will be 0.5. Until the human body part of the user exists in the third sensed data, the first weight and the second weight will vary according to the third sensed data.
In another embodiment, the processor 250 may determine the adjusted location information based on a weighted combination of the first location information, the second location information, and the third location information. And determining the adjusted position information according to the sum of the weighted first position information, the weighted second position information and the weighted third position information. The weights of the three pieces of position information may be varied or fixed based on actual conditions.
On the other hand, with respect to orientation information, in one embodiment, the processor 250 may use the first sensed data of the sequence directly as orientation information. For example, the orientation information may be acceleration, angular velocity in three axes, orientation, 3-DoF information, and/or 6-DoF information.
In another embodiment, the processor 250 may determine the orientation information from the third sensed data. Taking fig. 4 as an example, two poses of the human body part B4 in the image at different points in time can be used to estimate the orientation information.
In some embodiments, the processor 250 may determine the orientation information from the first sensed data and the third sensed data. The orientation information may be a weighted combination of the first sensed data and the third sensed data. For example, the position information is determined from the sum of the weighted first orientation information based on the motion sensor 130 and the weighted second orientation information based on the image sensor 360.
In one embodiment, the field of view of the image sensor 360 will be a condition as to whether orientation information is used in accordance with the third sensed data. If the human body part exists in the third sensed data, orientation information may be determined according to the first sensed data and the third sensed data. If the human body part is not present in the third sensing data, the orientation information may be determined according to only the first sensing data.
In one embodiment, processor 250 may determine movement information of the user based on the orientation information and the location information. The orientation information may be generated based on the first sensed data, the third sensed data, or a combination of the first sensed data and the third sensed data as described above. The position information may be generated based on the first sensing data, the second sensing data, and the third sensing data as described above. Taking the body part B1 or the body part B2 in fig. 2 as an example, the movement information may be related to a lifting, a point, a kick, a stepping, or a jumping movement.
In another embodiment, the processor 250 may determine the motion information of the user according to the orientation information based on the first sensed data and the adjusted position information based on the first position information and the second position information. Whether or not a human body part is present in the third sensed data, the processor 250 may predict the movement of the user.
In another embodiment, the processor 250 may determine the motion information of the user according to the orientation information based on the first sensed data and the combined position information based on the second sensed data and the third sensed data. This means that when a human body part is present and absent in the third sensed data, motion information may be determined based on the orientation information and the combined position information for two durations.
Taking fig. 4 and 5 as an example, the lifting hand movement of the human body part B4 is determined in fig. 4, and the lowering hand movement is determined in fig. 5. Next, a top-to-bottom swinging motion of the body part B4 is determined.
In one embodiment, the processor 250 may determine the motion information of the user based only on the location information based on the second sensed data. In another embodiment, the processor 250 may determine the motion information of the user based only on the combined location information based on the second sensed data and the third sensed data. In some embodiments, if the human body part is not present in the third sensed data, the processor 250 may determine the motion information of the user according to only the position information based on the second sensed data, and if the human body part is present in the third sensed data, the processor 250 may determine the motion information of the user according to only the position information or the combined position information based on the third sensed data.
The displacement or trajectory of the body part may be tracked, and motion information may be determined based on the displacement or trajectory. Taking fig. 4 and 5 as an example, the human body part B3 moves from top to bottom, and the human body part B4 is determined to swing from top to bottom.
In summary, in the motion tracking system and the motion tracking method for a plurality of operation positions according to the embodiments of the present invention, the motion of the human body part can be tracked based on the signal intensity, the sensing result of the motion sensor and/or the camera image. If the tracked human body part is not in the FOV according to the detection result of the camera image, the signal intensity between the motion sensing devices can be used for compensating the accuracy of the sensing result obtained by using the motion sensor in the position information determination. Furthermore, if the tracked body part is present within the FOV, the camera image may be used to correct the signal strength based position estimation. Therefore, a plurality of tracking methods can be used for different conditions, and the accuracy of the tracking result is improved accordingly.
It will be apparent to those skilled in the art that various modifications and variations can be made in the structure of the present invention without departing from the scope or spirit of the invention. In view of the above, it is intended that the present invention cover modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents.