Disclosure of Invention
The application mainly aims to provide a writing method, an intelligent pen, intelligent glasses and a computer readable storage medium, and aims to solve the technical problem that the writing efficiency of a user in a virtual space is low because real-time writing feedback cannot be truly and accurately provided for the user.
In order to achieve the above object, the present application provides a writing method, applied to an intelligent pen, comprising:
Establishing a communication connection link with the intelligent glasses;
based on the communication connection link, dynamically receiving the use state information of the intelligent pen sent by the intelligent glasses, wherein the use state information is generated by tracking the intelligent pen through a camera of the intelligent glasses;
and dynamically adjusting the vibration characteristics of the nib part of the intelligent pen according to the using state information so as to enable the vibration characteristics to be matched with the using state information, wherein the vibration characteristics comprise vibration frequency and vibration amplitude.
Optionally, the usage state information includes one of idle state information, writing state information, pen-down state information and pen-up state information, and the step of dynamically adjusting the vibration characteristic of the nib part of the smart pen according to the usage state information includes:
If the using state information is idle state information, setting the vibration characteristic as a first preset vibration characteristic;
If the use state information is writing state information, setting the vibration characteristic as a second preset vibration characteristic, wherein the vibration frequency of the second preset vibration characteristic is greater than that of the first preset vibration characteristic, and/or the vibration amplitude of the second preset vibration characteristic is greater than that of the first preset vibration characteristic;
If the using state information is pen-down state information, gradually increasing the vibration characteristics from a first preset vibration characteristic to a second preset vibration characteristic within a preset time period;
And if the using state information is pen lifting state information, gradually reducing the vibration characteristic from the second preset vibration characteristic to the first preset vibration characteristic within the preset time.
Optionally, before the step of dynamically receiving the usage status information of the smart pen sent by the smart glasses based on the communication connection link, the method further includes:
Dynamically detecting second pose information of the intelligent pen through an inertial sensor of the intelligent pen;
And sending the second pose information to the intelligent glasses based on the communication connection link so that the intelligent glasses respond to the second pose information and generate the using state information of the intelligent pen in combination with the first pose information determined by the camera tracking the intelligent pen.
Optionally, before the step of establishing a communication connection with the smart glasses, the method further includes:
Acquiring pressure parameters of the intelligent pen shell;
and if the pressure parameter is larger than a preset pressure parameter threshold, executing the step of establishing a communication connection link with the intelligent glasses.
In addition, in order to achieve the above object, the present application further provides a writing method, applied to intelligent glasses, including:
After a communication connection link is established with the intelligent glasses, tracking the feature objects on the shell of the intelligent pen through the camera of the intelligent glasses, and determining first pose information of the intelligent pen;
Constructing a three-dimensional virtual image of the intelligent pen according to the first pose information;
generating using state information of the intelligent pen according to the three-dimensional virtual image;
And sending the using state information to the intelligent pen so that the intelligent pen can dynamically adjust the vibration characteristics of the nib part of the intelligent pen according to the using state information, wherein the vibration characteristics comprise vibration frequency and vibration amplitude.
Optionally, the step of constructing a three-dimensional virtual image of the smart pen according to the first pose information includes:
Receiving second pose information sent by the intelligent pen, wherein the first pose information and the second pose information are the poses of the intelligent pen measured by the same time node;
And performing sum calculation on the first pose information of the first preset weight and the second pose information of the second preset weight to obtain third pose information, and constructing a three-dimensional virtual image of the intelligent pen according to the third pose information.
Optionally, the step of generating the usage status information of the smart pen according to the three-dimensional virtual image includes:
Determining the distance between the pen point part of the three-dimensional virtual image and the virtual image interaction interface and the change trend of the distance, wherein the change trend comprises an increasing trend and a decreasing trend;
if the distance is matched with the first preset distance, generating writing state information;
if the distance is matched with a second preset distance and the change trend is an increasing trend, generating pen lifting state information;
if the distance is matched with a second preset distance and the change trend is a decreasing trend, generating pen-down state information;
And if the distance is matched with a third preset distance, generating idle state information, wherein the second preset distance is larger than the first preset distance, and the third preset distance is larger than the second preset distance.
The application also provides a smart pen which is entity equipment and comprises a memory, a processor and a program of the writing method which is stored in the memory and can run on the processor, wherein the program of the writing method can realize the steps of the writing method when being executed by the processor.
The application also provides the intelligent glasses which are entity equipment, wherein the intelligent glasses comprise a memory, a processor and a program of the writing method which is stored in the memory and can run on the processor, and the program of the writing method can realize the steps of the writing method when being executed by the processor.
The application provides a writing method, an intelligent pen, intelligent glasses and a computer readable storage medium, wherein the intelligent pen is used for receiving the use state information of the intelligent pen generated by the intelligent glasses, and adjusting the vibration characteristics (the vibration characteristics comprise the vibration frequency and the vibration amplitude) of the nib part of the intelligent pen in real time according to the use state information, so that the reaction force given by a pen by a physical medium when a user writes on the physical medium is simulated, thereby giving tactile feedback to the user, and when the pen in the virtual world does not contact with a virtual interactive interface, the user can truly feel and timely accurately drop the nib of the pen in the virtual world onto the virtual interactive interface, and the need of adjusting the intelligent pen is not perceived until the written content is found to be not appear on the virtual interactive interface after a period of writing, thereby truly and accurately providing real-time writing feedback for the user, effectively avoiding invalid writing, reducing the times of repeated input of the content by the user, and further improving the writing efficiency of the user in the virtual space.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
With the development of virtual reality, augmented reality and other technologies, users can obtain more and more complete information in the virtual space. When information is obtained, information is required to be input, compared with keyboard input, the intelligent pen is used for inputting information by a user, so that the intelligent pen is more efficient and convenient, the requirements of a large number of information interaction scenes of characters and drawings can be met, and the user can write in a virtual space as early as possible by using the intelligent pen.
Generally, the virtual system needs the nib of the intelligent pen to touch the virtual interactive interface to identify information input by the user, however, the user uses the intelligent pen in the air when writing in the virtual space, and the user cannot feel the real touch when the pen touches the real object, so the user does not know whether the nib of the intelligent pen touches the virtual interactive interface, and therefore the problem of invalid writing is brought, namely the user waves the intelligent pen to write, but finds that the content is not successfully input in the virtual interactive interface, and therefore the user needs to re-input the content after repeatedly adjusting, so the efficiency of writing in the virtual space is reduced.
The application provides a writing method, an intelligent pen, intelligent glasses and a computer readable storage medium, wherein the intelligent pen is used for receiving the use state information of the intelligent pen generated by the intelligent glasses, and adjusting the vibration characteristics (the vibration characteristics comprise the vibration frequency and the vibration amplitude) of the nib part of the intelligent pen in real time according to the use state information, so that the reaction force given by a pen by a physical medium when a user writes on the physical medium is simulated, thereby giving tactile feedback to the user, and when the pen in the virtual world does not contact with a virtual interactive interface, the user can truly feel and timely accurately drop the nib of the pen in the virtual world onto the virtual interactive interface, and the need of adjusting the intelligent pen is not perceived until the written content is found to be not appear on the virtual interactive interface after a period of writing, thereby truly and accurately providing real-time writing feedback for the user, effectively avoiding invalid writing, reducing the times of repeated input of the content by the user, and further improving the writing efficiency of the user in the virtual space.
Embodiments of the present application will be further described with reference to the accompanying drawings.
Referring to fig. 1, fig. 1 is a block diagram of a system structure of a smart glasses according to an embodiment of the present application, where the smart glasses may include an application processor, such as a central processing unit (Central Processing Unit, CPU), a wireless communication module, a graphics renderer, a 6DoF (Degrees of Freedom, degree of freedom) camera module, and a virtual distance adjustment module, which are connected to the application processor through a communication bus, and further, the smart glasses may include a Memory, which may be a high-speed random access Memory (Random Access Memory, RAM) or a stable Non-Volatile Memory (NVM), such as a disk storage. The wireless communication module is used for carrying out data interaction with the intelligent pen, the graphic renderer is used for projecting a virtual picture into the display module of the intelligent glasses, the virtual image distance adjusting module is used for adjusting and displaying the virtual image distance, and the 6DoF camera module is used for tracking the intelligent pen to acquire pose information of the intelligent pen. It will be appreciated by those skilled in the art that the structure shown in fig. 1 is not limiting and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components. The smart glasses are not limited to intelligent products such as AR (Augmented Reality) glasses, XR (Extended Reality) glasses, MR (Mixed Reality) glasses, or other head display devices.
Referring to fig. 2, fig. 2 is a block diagram of a smart pen system according to an embodiment of the present application, where the smart pen may include a microprocessor, such as a central processing unit (Central Processing Unit, CPU), a wireless communication module, a haptic feedback module, and an IMU (Inertial Measurement Unit ) module, which are connected to the microprocessor through a communication bus, and may further include an IR LED (INFRARED LIGHT EMITTING Diode) array module, and may further include a memory, which may be a high-speed random access memory, or a stable nonvolatile memory, such as a magnetic disk storage. The wireless communication module is used for carrying out data interaction with the intelligent glasses, the touch feedback module is used for adjusting the vibration characteristics of the intelligent pen (the vibration characteristics comprise vibration frequency and vibration amplitude), so that the user is given touch feedback, and the IMU module is used for collecting the pose of the intelligent pen in real time. The IR LED array module may include a plurality of IR LED lights and drivers that are spatially distributed on the smart pen, so that a user can recognize the IR LED lights and drivers by the 6DOF camera module of the smart glasses in different pen holding postures or pen holding angles, and can be replaced by highly reflective graphics or features, so that the smart glasses can track the smart pen, and the 6DOF camera of the smart glasses can clearly recognize the smart pen even in a dark environment, and those skilled in the art will appreciate that the structure shown in fig. 2 does not constitute a limitation to the smart pen, and may include more or fewer components than those illustrated, or some components in combination, or different component arrangements.
Example 1
Referring to fig. 3, the present application provides a writing method, which is applied to a smart pen in the present embodiment in a first embodiment of the writing method, the method comprising the steps of:
Step S100, establishing a communication connection link with the intelligent glasses;
In this embodiment, the smart pen needs to perform data interaction with the smart glasses, and before performing data interaction, the smart pen needs to establish a communication connection link with the smart glasses, where the communication connection link may be a wireless connection. It should be noted that, the writing panel applied by the intelligent pen in the writing space may be a virtual writing panel, or may be a real writing panel, that is, after the intelligent glasses are communicated with the intelligent pen, a writing panel for writing by the intelligent pen is established, and writing actions of the intelligent pen on the writing panel are identified.
Step S200, based on the communication connection link, dynamically receiving the use state information of the intelligent pen sent by the intelligent glasses, wherein the use state information is generated by tracking the intelligent pen through a camera of the intelligent glasses;
In this embodiment, the smart glasses may use the graphics renderer to project the virtual image on the display module, build a virtual space with the aid of the unit engine, generate a virtual interaction interface in the virtual space by the smart glasses, and simultaneously reconstruct the smart pen in three dimensions in the virtual space to generate a three-dimensional virtual image, that is, the motion gesture of the smart pen in the real world is synchronous with the motion gesture of the three-dimensional virtual image of the smart pen in the virtual space. When the user prepares to input the content by handwriting, the virtual distance adjusting module adjusts the display virtual image from far to near, and for example, when the display virtual image is located far, the display function is mainly provided for the user, so that when the display virtual image is located far, the distance between the display virtual image and the intelligent glasses can be 2 meters, and when the display virtual image is located near, a virtual interaction interface can be generated, so that convenience is provided for the user to write in the virtual space, at this time, the distance between the virtual interaction interface and the intelligent glasses can be 0.5 meter, and the distance can be set to other values so as to better match the most comfortable writing distance of the user.
In this embodiment, the smart glasses track the smart pen through the camera, so as to dynamically acquire pose information of the smart pen, generate a three-dimensional virtual image of the smart pen in the virtual space according to the pose information, determine usage state information of the smart pen through the three-dimensional virtual image, and dynamically receive the usage state information sent by the smart glasses through the communication connection link, where the usage state information may be one of writing state information and idle state information. When the nib part of the three-dimensional virtual image contacts the virtual interactive interface, the intelligent glasses can generate writing state information, and if the using state information received by the intelligent pen is the writing state information, the writing content representing the user in the virtual space can be timely displayed on the virtual interactive interface. When the pen point part of the three-dimensional virtual image is not contacted with the virtual interactive interface and has a certain distance with the virtual interactive interface, the intelligent glasses can generate idle state information, and if the using state information received by the intelligent pen is the idle state information, the writing content representing the user in the virtual space cannot be displayed on the virtual interactive interface.
Alternatively, the usage status information may also be pen-down status information or pen-up status information, and illustratively, the process of moving the nib portion of the three-dimensional virtual image from a position far from the virtual interaction interface to a position contacting the virtual interaction interface is a pen-down process of the three-dimensional virtual image, the smart glasses may generate pen-down status information, and conversely, the process of moving the nib portion of the three-dimensional virtual image from a position contacting the virtual interaction interface to a position far from the virtual interaction interface is a pen-up process of the three-dimensional virtual image, the smart glasses may generate pen-up status information.
After the step S200, a step S300 is performed to dynamically adjust the vibration characteristics of the nib portion of the smart pen according to the usage status information, so that the vibration characteristics are matched with the usage status information, wherein the vibration characteristics include a vibration frequency and a vibration amplitude.
The haptic feedback module of the smart pen may include a piezoelectric feedback device or a vibration motor, the piezoelectric feedback device may be a piezoelectric ceramic having an inverse piezoelectric effect, the piezoelectric ceramic sheet may be mechanically deformed when an electric field is applied to the piezoelectric sheet, and its strain is proportional to the strength of the electric field, the stronger the strength of the applied electric field is, the greater the amplitude of vibration is, the vibration characteristics of the nib portion of the smart pen may be adjusted by the piezoelectric ceramic or the vibration motor, and the sound emitted when the piezoelectric ceramic vibrates is extremely clear, and has no noise due to its ultra-thin thickness and light weight without contact, and simultaneously has a small operating current and low power consumption characteristics.
In this embodiment, if the usage status information received by the smart pen is writing status information, which indicates that the user is writing on the virtual interactive interface, or is ready to write, in order to simulate the haptic sensation of the user when writing on the physical medium in reality, the nib portion of the smart pen may start to vibrate, simulate the reaction force of the physical medium to the smart pen in reality, and if the smart pen is already in a vibrating state, the original vibration characteristics may be maintained after receiving the writing status information. If the usage state information received by the intelligent pen is idle state information, the user stops writing on the virtual interactive interface, and the pen does not contact the virtual interactive interface in the virtual space, so that in order to simulate the touch feeling of the user in such a scene, the nib part of the intelligent pen can stop vibrating, or the vibration characteristics (including the vibration frequency and the vibration amplitude) can be set to a value that the user cannot feel the vibration of the intelligent pen.
Optionally, if the usage status information received by the smart pen is pen lifting status information, which indicates that the user is doing a motion of lifting the smart pen in the virtual space, in order to simulate a process of reducing a reaction force of the physical medium to the smart pen when the user lifts the pen from the physical medium in reality, the vibration frequency and/or the vibration amplitude of the nib portion of the smart pen may be gradually reduced. Conversely, if the usage status information received by the smart pen is pen-down status information, it indicates that the user is doing a pen-down action in the virtual space, in order to simulate the process of increasing the reaction force of the physical medium to the smart pen when the pen falls on the physical medium in reality, at this time, the vibration frequency and/or the vibration amplitude of the nib portion of the smart pen may be slowly increased.
Further, referring to fig. 4, the usage status information includes one of idle status information, writing status information, pen-down status information and pen-up status information, and the step S300 of dynamically adjusting the vibration characteristics of the nib portion of the smart pen according to the usage status information includes:
Step S310, if the use state information is idle state information, setting the vibration characteristic as a first preset vibration characteristic;
Step S320, if the usage status information is writing status information, setting the vibration characteristic as a second preset vibration characteristic, where the vibration frequency of the second preset vibration characteristic is greater than the vibration frequency of the first preset vibration characteristic, and/or the vibration amplitude of the second preset vibration characteristic is greater than the vibration amplitude of the first preset vibration characteristic;
Step S330, if the usage status information is pen-down status information, gradually increasing the vibration characteristic from the first preset vibration characteristic to the second preset vibration characteristic within a preset time period;
Step S340, if the usage status information is pen-lifting status information, gradually reducing the vibration characteristic from the second preset vibration characteristic to the first preset vibration characteristic within a preset time period.
In this embodiment, the first preset vibration characteristic is a vibration characteristic of a nib portion of the smart pen when the three-dimensional virtual image of the smart pen does not contact the virtual interactive interface, for example, the vibration frequency may be zero, or may be another value smaller than the preset vibration frequency threshold. And/or the vibration amplitude may be zero, or may be other value less than a preset vibration amplitude threshold, which is not limited herein. The second preset vibration characteristic is the vibration characteristic of the nib part of the intelligent pen when the three-dimensional virtual image of the intelligent pen contacts the virtual interactive interface, the vibration frequency of the second preset vibration characteristic is larger than that of the first preset vibration characteristic, and/or the vibration amplitude of the second preset vibration characteristic is larger than that of the first preset vibration characteristic, so that the intelligent pen can be set by a person skilled in the art by himself, and the actual touch feeling of the intelligent pen when the intelligent pen contacts the physical medium in reality is simulated truly. The preset time period can be 0.1 second or other values, and can be set by one skilled in the art. The idle state information is matched with the first preset vibration characteristic, the writing state information is matched with the second preset vibration characteristic, and when the using state information is the pen-down state information, in order to enable the vibration characteristic to be matched with the pen-down state information, the vibration characteristic of the nib part of the intelligent pen is gradually increased from the first preset vibration characteristic to the second preset vibration characteristic, so that the vibration characteristic of the nib part of the intelligent pen is smoothly increased (the vibration frequency and/or the vibration amplitude of the nib part of the intelligent pen are slowly increased), and the vibration characteristic is not directly set to the second preset vibration characteristic by the first preset vibration characteristic. When the usage state information is pen-lifting state information, in order to match the vibration characteristics with the pen-lifting state information, the vibration characteristics of the nib part of the intelligent pen are gradually reduced from the second preset vibration characteristics to the first preset vibration characteristics (the vibration frequency and/or the vibration amplitude of the nib part of the intelligent pen are slowly reduced), so that the vibration characteristics of the nib part of the intelligent pen are smoothly reduced, instead of being directly set as the first preset vibration characteristics by the second preset vibration characteristics. In this embodiment, the intelligent pen adjusts the vibration characteristics of the nib portion in real time according to the use state information sent by the intelligent glasses, and gives the user a tactile feedback, and meanwhile, when the three-dimensional virtual image is used for carrying out pen lifting or pen dropping actions, the intelligent pen smoothly adjusts the vibration characteristics of the nib portion, so as to reduce the feeling of pause caused by abrupt change of the vibration characteristics, realize the simulation of the tactile feeling when the user carries out pen lifting and pen dropping, and further ensure that real and accurate real-time writing feedback is provided for the user.
In a possible implementation manner, before the step of dynamically receiving the usage state information of the smart pen sent by the smart glasses based on the communication connection link, the method further includes:
Step A10, dynamically detecting second pose information of the intelligent pen through an inertial sensor of the intelligent pen;
as will be appreciated by those skilled in the art, the inertial sensor is an IMU (Inertial Measurement Unit ) sensor.
And step A20, transmitting the second pose information to the intelligent glasses based on the communication connection link so that the intelligent glasses respond to the second pose information and generate the using state information of the intelligent pen in combination with the first pose information determined by the camera tracking the intelligent pen.
In this embodiment, the intelligent pen is provided with the IMU module, the second pose information of the intelligent pen itself can be dynamically detected through the inertial sensor of the IMU module, the second pose information is sent to the intelligent glasses through the communication connection link, the intelligent glasses track the intelligent pen through the camera to obtain the first pose information of the intelligent pen, and then the three-dimensional virtual image of the intelligent pen is constructed by combining the second pose information sent by the intelligent pen. In this embodiment, the smart pen sends its pose information to the smart glasses through the communication connection link, and a data base is provided for the smart glasses to construct a three-dimensional virtual image of the smart pen.
In one embodiment, before the step of establishing a communication connection with the smart glasses, the method further includes:
step B10, obtaining pressure parameters of the intelligent pen shell;
and step B20, if the pressure parameter is larger than a preset pressure parameter threshold, executing the step of establishing a communication connection link with the intelligent glasses.
In this embodiment, it is required to determine whether the user is using the smart pen, and after the user is using the smart pen, the smart pen and the smart glasses establish a communication connection link, at this time, a pressure sensor may be added at the housing of the smart pen to detect a change of a pressure parameter at the housing of the smart pen in real time, and a preset pressure parameter threshold is set by a person skilled in the art to accurately determine whether the smart pen is being used, when the smart pen is picked up (held) by the user, the pressure parameter of the housing of the smart pen is greater than the preset pressure parameter threshold, and then it can be determined that the user is using the smart pen, and the smart pen and the smart glasses establish a communication connection link, and when the smart pen is put down (not held) by the user, and at this time, the pressure value of the housing of the smart pen is less than or equal to the preset pressure parameter threshold, and then it is determined that the smart pen is not being used, and the smart pen is disconnected from the smart glasses. Optionally, a button or switch may be installed on the smart pen, and when the user needs to use the smart pen, the user may manually open the smart pen to establish a communication connection link between the smart pen and the smart glasses. According to the embodiment, the communication connection state of the intelligent pen and the intelligent glasses is automatically controlled through detecting the pressure parameter of the intelligent pen shell, and the intelligent pen is opened and closed, so that the convenience of writing in the virtual space by a user is improved while the power consumption of the intelligent pen is reduced.
In this embodiment, a user uses an intelligent pen in a virtual space, the intelligent glasses generate use state information of the intelligent pen according to a three-dimensional virtual image of the intelligent pen in the virtual space, and the use state information is shared to the intelligent pen in real time, and the intelligent pen dynamically adjusts vibration characteristics of a pen point part according to the corresponding use state information, so that the user can feel touch feeling when the pen contacts with a physical medium during writing in a virtual world, and can accurately judge whether the user effectively writes according to change of touch feeling, and timely accurately drops a pen point of the pen in the virtual world on a virtual interactive interface.
Example two
Referring to fig. 5, the embodiment of the present application further provides a writing method of a second embodiment, where the writing method is applied to smart glasses, and the method includes the following steps:
Step S400, after a communication connection link is established with the intelligent glasses, tracking the feature on the shell of the intelligent pen through the camera of the intelligent glasses, and determining first pose information of the intelligent pen;
step S500, constructing a three-dimensional virtual image of the intelligent pen according to the first pose information;
step S600, generating the using state information of the intelligent pen according to the three-dimensional virtual image;
and step S700, the using state information is sent to the intelligent pen, so that the intelligent pen adjusts the vibration characteristics of the pen tip part of the intelligent pen according to the using state information, wherein the vibration characteristics comprise the vibration frequency and the vibration amplitude.
In this embodiment, the feature may be an IR LED array module on the smart pen housing, where the IR LED array module may include a plurality of IR LED lamps and drivers that are three-dimensionally distributed on the smart pen, so that the user can be identified by the 6DOF camera module of the smart glasses when the user holds the pen in different pen gestures or pen holding angles. In addition, can also replace with highly reflective graphics for smart glasses can track smart pen, even the 6DoF camera of smart glasses can clearly discern smart pen under the darker environment of light. The first pose information comprises the real-time pose of the intelligent pen measured by the intelligent glasses, the intelligent glasses track the intelligent pen through the 6DoF camera module, the intelligent pen is photographed or imaged, then the first pose information of the intelligent pen is obtained through SLAM (Simultaneous Localization AND MAPPING, instant positioning and map building) algorithm calculation, the intelligent glasses can firstly mark the time and the position of the 6DoF camera module and the display module in a display coordinate system of the intelligent glasses, the position of a three-dimensional virtual image of the intelligent pen in the display coordinate system can be the same as the display module initially, the intelligent glasses obtain the first pose information of the intelligent pen, the change condition of the position and the pose of the intelligent pen is obtained according to the information, and the three-dimensional virtual image of the intelligent pen is reconstructed. When a user wears intelligent glasses or other head display equipment, the position of a pen can not be seen, so that difficulty is brought to writing of the user, in a scene, the user writes contents on a virtual interactive interface, the user pauses writing for some reasons, when the user wants to continue writing, the position where the writing is paused last time can not be found, the three-dimensional virtual image of the intelligent pen is displayed in a virtual space in real time, and the position of the three-dimensional virtual image can be prompted.
When the intelligent pen moves in the real world and the posture is changed, the corresponding three-dimensional virtual image also moves in the display coordinate system and the same posture is changed, at the moment, the use state information of the intelligent pen can be determined by calculating the displacement of the nib part of the three-dimensional virtual image, preferably, the use state information of the intelligent pen can be determined by calculating the distance between the nib part of the three-dimensional virtual image and the virtual interactive interface, then the use state information is sent to the intelligent pen, and the intelligent pen adjusts the vibration characteristics of the nib part of the intelligent pen in real time according to the use state information.
According to the embodiment, the pose information of the intelligent pen is obtained in real time, the intelligent pen is reconstructed in the virtual space, so that a user can accurately determine the writing position through the three-dimensional virtual image of the intelligent pen, the convenience of writing in the virtual world is improved, in addition, the intelligent pen is enabled to adjust the vibration characteristics (including the vibration frequency and the vibration amplitude) of the nib part in real time by sending the use state information of the intelligent pen to the intelligent pen, the vibration characteristics are matched with the use state information, the user can accurately judge whether the user effectively writes in the virtual space, real and accurate real-time writing feedback is further realized for the user, and the writing efficiency of the user in the virtual space is improved.
In one possible embodiment, the step of constructing a three-dimensional virtual image of the smart pen according to the first pose information includes:
Step C10, receiving second pose information sent by the intelligent pen, wherein the first pose information and the second pose information are the poses of the intelligent pen measured at the same time node;
And step C20, performing sum calculation on the first pose information of the first preset weight and the second pose information of the second preset weight to obtain third pose information, and constructing a three-dimensional virtual image of the intelligent pen according to the third pose information.
In this embodiment, the second pose information is self pose information acquired by the intelligent pen in real time through the IMU module, the acquired self pose information is sent to the intelligent glasses, the first pose information and the second pose information respectively include poses of the intelligent glasses and the intelligent pen measured by the intelligent pen under the same time node, and because the first pose information and the second pose information may generate certain errors, the two sets of pose information can be comprehensively judged, preset weights can be respectively set for the two sets of poses, sum value calculation is performed on the first pose information of the first preset weight and the second pose information of the second preset weight, third pose information is obtained through calculation, and then three-dimensional virtual images of the intelligent pen are constructed by using the third pose information. The embodiment receives the second pose information measured by the intelligent pen, and determines the pose of the intelligent pen by combining the first pose information and the second pose information, so that the error is reduced, and the accuracy of constructing the three-dimensional virtual image of the intelligent pen is improved.
In another possible embodiment, the step of generating the usage status information of the smart pen according to the three-dimensional virtual image includes:
Step D10, simultaneously tracking the feature objects on the shell of the intelligent pen through a preset number of cameras on the intelligent glasses, and determining fourth pose information of the intelligent pen in a preset number of groups, wherein all the fourth pose information is the pose of the intelligent pen under the same time node;
And D20, respectively fusing all the fourth pose information based on the weights correspondingly distributed by the cameras, generating fifth pose information, and constructing a three-dimensional virtual image of the intelligent pen according to the fifth pose information.
In this embodiment, the preset number is at least two, the embodiment is not limited, a preset number of groups of 6DoF cameras can be set on the smart glasses to obtain fourth pose information generated by tracking the smart pen by the plurality of 6DoF cameras, corresponding preset weights are allocated to the fourth pose information acquired by each camera, sum value calculation is performed on the fourth pose information of the preset weights allocated by all cameras to obtain fifth pose information of the smart pen, and three-dimensional virtual images of the smart pen are constructed by using the fifth pose information, wherein the numerical value of each weight can be the same or different, and the sum of all weights can be one. According to the embodiment, the plurality of pose information of the intelligent pen at the same moment is collected, the pose of the intelligent pen is determined by integrating the plurality of pose information, errors are reduced, and the accuracy of constructing the three-dimensional virtual image of the intelligent pen is improved.
In another possible embodiment, the step of generating the usage status information of the smart pen according to the three-dimensional virtual image includes:
E10, determining the distance between the pen point part of the three-dimensional virtual image and the virtual image interaction interface and the change trend of the distance, wherein the change trend comprises an increasing trend and a decreasing trend;
E20, if the distance is matched with the first preset distance, generating writing state information;
Step E30, if the distance is matched with a second preset distance and the change trend is an increasing trend, generating pen lifting state information;
Step E40, if the distance is matched with the second preset distance and the change trend is a decreasing trend, generating pen-down state information;
And E50, if the distance is matched with a third preset distance, generating idle state information, wherein the second preset distance is larger than the first preset distance, and the third preset distance is larger than the second preset distance.
In this embodiment, whether the user is writing in the virtual space may be determined by determining the distance between the nib portion of the three-dimensional virtual image and the virtual image interaction interface, and the first preset distance may be zero when determining that the three-dimensional virtual image is in contact with the virtual interaction interface. The second preset distance is the distance between the three-dimensional virtual image and the virtual interaction interface when the three-dimensional virtual image is in the pen lifting action or the pen falling action, and the third preset distance is the distance between the three-dimensional virtual image and the virtual interaction interface when the three-dimensional virtual image is not written. Wherein, the second preset distance is greater than the first preset distance, and the third preset distance is greater than the second preset distance, and the numerical value can be set by the person skilled in the art. According to the embodiment, whether the user writes in the virtual space can be judged by determining the distance between the nib part of the intelligent pen and the virtual image interaction interface, and judging conditions are provided for adjusting the vibration characteristics of the nib part of the intelligent pen in real time.
For example, if the trend of the distance is an increasing trend, the three-dimensional virtual image is far away from the virtual interactive interface, and meanwhile, the distance is matched with a second preset distance, so that it can be determined that the three-dimensional virtual image is doing pen lifting actions, the intelligent glasses correspondingly generate pen lifting state information, if the trend of the distance is a decreasing trend, the three-dimensional virtual image is approaching the virtual interactive interface, and meanwhile, the distance is matched with the second preset distance, so that it can be determined that the three-dimensional virtual image is doing pen dropping actions, and the intelligent glasses correspondingly generate pen dropping state information. In this embodiment, the vibration characteristics of the smart pen nib portion are slowly increased, such as slowly increasing the vibration frequency and/or vibration amplitude of the smart pen nib portion, as the three-dimensional virtual image performs a pen-down motion. When the three-dimensional virtual image is used for carrying out the pen lifting action, the vibration characteristics of the nib part of the intelligent pen are slowly reduced, for example, the vibration frequency and/or the vibration amplitude of the nib part of the intelligent pen are slowly reduced, the change process of the vibration characteristics is smoother, the frustration caused by abrupt change of the vibration characteristics is reduced, and the embodiment can judge whether a user is in a virtual space for pen lifting or pen lifting by determining the change trend of the distance between the nib part of the three-dimensional virtual image and the virtual image interaction interface, so that judgment conditions are provided for adjusting the vibration characteristics of the nib part of the intelligent pen in real time.
In this embodiment, the three-dimensional virtual image of the intelligent pen is reconstructed by the intelligent glasses in the virtual space, so that the user can judge the writing position according to the three-dimensional virtual image, and can correctly find the position when the user pauses writing when continuing writing, thereby improving the convenience of writing in the virtual space for the user.
Example III
The embodiment of the invention provides intelligent glasses, which comprise at least one processor and a memory in communication connection with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the writing method in the first embodiment.
Referring now to fig. 6, a schematic diagram of a configuration of smart glasses suitable for use in implementing embodiments of the present disclosure is shown. The smart glasses in the embodiments of the present disclosure include, but are not limited to, mixed Reality (Mixed Reality) -MR glasses (, augmented Reality (Augmented Reality) -AR glasses, augmented Reality (Extended Reality) -XR glasses, or some combination thereof, etc. the smart glasses shown in fig. 6 are merely one example and should not impose any limitation on the functionality and scope of use of the embodiments of the present disclosure.
As shown in fig. 6, the smart glasses may include a processing device 1001 (e.g., a central processor, a graphic processor, etc.) that may perform various appropriate actions and processes according to programs stored in a read-only memory (ROM 1002) or programs loaded from a storage device into a random access memory (RAM 1004). In the RAM1004, various programs and data required for the operation of the smart glasses are also stored. The processing device 1001, the ROM1002, and the RAM1004 are connected to each other by a bus 1005. An input/output (I/O) interface is also connected to bus 1005.
In general, a system including an input device 1007 such as a touch screen, touch pad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc., an output device 1008 including a Liquid Crystal Display (LCD), speaker, vibrator, etc., a storage device 1003 including a magnetic tape, hard disk, etc., and a communication device 1009 may be connected to the I/O interface 1006. The communication means 1009 may allow the smart glasses to communicate wirelessly or by wire with other devices to exchange data. While smart glasses with various systems are shown in the figures, it should be understood that not all of the illustrated systems are required to be implemented or provided. More or fewer systems may alternatively be implemented or provided.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. The above-described functions defined in the method of the embodiment of the present disclosure are performed when the computer program is executed by the processing device 1001.
The intelligent glasses provided by the invention can solve the technical problem that the writing efficiency of the user in the virtual space is low due to the fact that real-time writing feedback cannot be truly and accurately provided for the user by adopting the writing method in the embodiment. Compared with the prior art, the beneficial effects of the intelligent glasses provided by the embodiment of the invention are the same as those of the writing method provided by the embodiment, and other technical features of the intelligent glasses are the same as those disclosed by the method of the previous embodiment, and are not repeated here.
It should be understood that portions of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof. In the description of the above embodiments, particular features, structures, materials, or characteristics may be combined in any suitable manner in any one or more embodiments or examples.
The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Example IV
The present embodiment also provides a computer-readable storage medium having computer-readable program instructions stored thereon for performing the writing method in the above-described embodiments.
The computer readable storage medium provided by the embodiments of the present invention may be, for example, a usb disk, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared system, or device, or a combination of any of the foregoing. More specific examples of a computer-readable storage medium may include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this embodiment, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, or device. Program code embodied on a computer readable storage medium may be transmitted using any appropriate medium, including but not limited to electrical wiring, fiber optic cable, RF (radio frequency), and the like, or any suitable combination of the foregoing.
The computer readable storage medium may be included in the smart pen or the smart glasses, or may exist alone without being assembled into the smart pen or the smart glasses.
The computer readable storage medium carries one or more programs, when the one or more programs are executed by the intelligent pen, the intelligent pen is enabled to establish a communication connection link with the intelligent glasses, the use state information of the intelligent pen sent by the intelligent glasses is dynamically received based on the communication connection link, wherein the use state information is generated by tracking the intelligent pen through a camera of the intelligent glasses, and vibration characteristics of a pen point part of the intelligent pen are dynamically adjusted according to the use state information so that the vibration characteristics are matched with the use state information, and the vibration characteristics comprise vibration frequency and vibration amplitude.
The computer readable storage medium carries one or more programs, when the one or more programs are executed by the intelligent glasses, the intelligent glasses are enabled to track the feature objects on the shell of the intelligent pen through the camera of the intelligent glasses after a communication connection link is established between the intelligent glasses and the intelligent glasses, first pose information of the intelligent pen is determined, a three-dimensional virtual image of the intelligent pen is built according to the first pose information, using state information of the intelligent pen is generated according to the three-dimensional virtual image, and the using state information is sent to the intelligent pen, so that the intelligent pen dynamically adjusts vibration characteristics of a pen point part of the intelligent pen according to the using state information.
Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present disclosure may be implemented in software or hardware. Wherein the name of the module does not constitute a limitation of the unit itself in some cases.
The computer readable storage medium provided by the invention stores the computer readable program instructions for executing the writing method, so that the technical problem of how to truly and accurately provide real-time writing feedback for a user and improve the writing efficiency of the user in a virtual space is solved. Compared with the prior art, the beneficial effects of the computer readable storage medium provided by the embodiment of the invention are the same as those of the writing method provided by the above embodiment, and are not described herein.
Example five
The embodiments of the present invention also provide a computer program product comprising a computer program which, when executed by a processor, implements the steps of the writing method as described above.
The computer program product provided by the application can solve the technical problem that the writing efficiency of the user in the virtual space is low because real-time writing feedback cannot be truly and accurately provided for the user. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as those of the writing method provided by the first embodiment or the second embodiment, and are not described herein.
The foregoing description is only of the preferred embodiments of the present application, and is not intended to limit the scope of the application, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein, or any application, directly or indirectly, within the scope of the application.