Disclosure of Invention
In order to solve the problems, the invention provides a remote labeling method, a terminal and a system based on mixed reality, and the method provided by the invention can realize remote mixed reality labeling between Hololens end and other equipment such as a PC end and has higher accuracy.
In order to achieve the above object, in a first aspect, the present invention provides a remote labeling method based on mixed reality, which adopts the following technical scheme:
A remote labeling method based on mixed reality comprises the following steps:
Acquiring a two-dimensional annotation array;
Converting the acquired two-dimensional annotation array into a three-dimensional coordinate array;
The method comprises the steps of obtaining a three-dimensional array, obtaining a barycentric coordinate according to the sum of all coordinates in the three-dimensional array in all directions, sending out rays at the barycentric coordinate, determining an object impacted by the rays as the object to be marked, subtracting the coordinates of the rays and the collision points of the object from the barycentric coordinate to obtain a transformation displacement, adding each point in the three-dimensional array with the transformation displacement to obtain a new three-dimensional array, and rendering a line segment between every two points of the new three-dimensional array to finish real-time three-dimensional marking in space.
Further, when the two-dimensional annotation array is obtained, a line renderer and an inverse interpolation algorithm are used for two-dimensional annotation.
Further, the interpolation lerp between two numbers y1,y2 is lerp=y1+(y2-y1) x weighy;
Wherein weight is a real number in interval [0,1], and the inverse interpolation algorithm uses the known interpolation lerp and two numbers y1,y2 to calculate the weight value
Further, screen space coordinates of each frame of the mouse are automatically captured and stored in a storage coordinate array, a point number is added, meanwhile, the last two-bit coordinates in the coordinate array form a new coordinate array, the last two-bit coordinates are (point 1, point 2), the method is circulated for a plurality of times, 0 is used as y1, 3 is used as y2 and 0 is used as interpolation for each circulation, a weight value is obtained, a supplementary coordinate= (1-weight) point1+weight point s2 is added, the supplementary coordinate is added to a new coordinate array, meanwhile, the interpolation value is added by 1 to enter the next circulation, the supplementary coordinate between 2 two points and the original two coordinate points are obtained after the circulation is finished for a plurality of times, and a smooth curve is obtained by connecting each point of the obtained coordinates until end point labeling is assisted.
Further, when converting the acquired two-dimensional annotation array into a three-dimensional coordinate array, converting the two-dimensional coordinate array of the screen into a three-dimensional coordinate array at the same position of the object picture used by the assisted terminal,
Where xm is the sum of the x-axes of all coordinates, ym is the sum of the y-axes of all coordinates, zm is the sum of the z-axes of all coordinates, and m is the number of coordinates.
In order to achieve the above purpose, in a second aspect, the present invention further provides a remote labeling terminal based on mixed reality, which adopts the following technical scheme:
a mixed reality based remote annotation terminal comprising at least a processor configured to:
Acquiring a two-dimensional annotation array;
Converting the acquired two-dimensional annotation array into a three-dimensional coordinate array;
The method comprises the steps of obtaining a three-dimensional array, obtaining a barycentric coordinate according to the sum of all coordinates in the three-dimensional array in all directions, sending out rays at the barycentric coordinate, determining an object impacted by the rays as the object to be marked, subtracting the coordinates of the rays and the collision points of the object from the barycentric coordinate to obtain a transformation displacement, adding each point in the three-dimensional array with the transformation displacement to obtain a new three-dimensional array, and rendering a line segment between every two points of the new three-dimensional array to finish real-time three-dimensional marking in space.
When initiating video voice call invitation, the remote server pulls the token of the labeling terminal and the token of the assisted terminal into the same room, and sends the internet protocol address and port of the other party to the two parties, so that the assisted terminal and the assisted terminal establish a user datagram protocol.
In order to achieve the above purpose, in a third aspect, the present invention further provides a remote labeling system based on mixed reality, which adopts the following technical scheme:
a mixed reality based remote labeling system, comprising:
a remote server, and at least one assisting terminal and at least one assisted terminal connected thereto;
When initiating video voice call invitation, the remote server pulls the token of the marked terminal and the token of the assisted terminal into the same room, and sends the internet protocol address and port of the other party to the two parties, so that the assisted terminal and the assisted terminal establish a user datagram protocol;
The labeling module is configured to calculate barycentric coordinates according to the sum of all coordinates in the three-dimensional coordinate array, send out rays at the barycentric coordinates, determine the object impacted by the rays as the object to be labeled, subtract the coordinates of the collision points of the rays and the object from the barycentric coordinates to obtain a transformation displacement, add each point in the three-dimensional coordinate array with the transformation displacement to obtain a new three-dimensional array, and render a line segment between every two points of the new three-dimensional array to complete real-time three-dimensional labeling in space.
In order to achieve the above object, in a fourth aspect, the present invention further provides a remote labeling system based on mixed reality, which adopts the following technical scheme:
A remote labeling system based on mixed reality comprises a data acquisition module, a conversion module and a labeling module;
the data acquisition module is configured to acquire a two-dimensional annotation array;
the conversion module is configured to convert the acquired two-dimensional annotation array into a three-dimensional coordinate array;
The labeling module is configured to calculate barycentric coordinates according to the sum of all coordinates in the three-dimensional coordinate array, send out rays at the barycentric coordinates, determine the object impacted by the rays as the object to be labeled, subtract the coordinates of the collision points of the rays and the object from the barycentric coordinates to obtain a transformation displacement, add each point in the three-dimensional coordinate array with the transformation displacement to obtain a new three-dimensional array, and render a line segment between every two points of the new three-dimensional array to complete real-time three-dimensional labeling in space.
In order to achieve the above object, the present invention also provides, in a fifth aspect, a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the mixed reality based remote labeling method of the first aspect.
In order to achieve the above object, the present invention further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the mixed reality based remote labeling method according to the first aspect when executing the program.
Compared with the prior art, the invention has the beneficial effects that:
The method comprises the steps of converting a received two-dimensional coordinate array into a three-dimensional coordinate array which is the same with an object picture used by a assisted end, solving a barycentric coordinate, sending a ray by using the barycentric coordinate, wherein an object which is impacted by the ray is the object to be marked, subtracting the coordinate of an impact point and the coordinate point of the barycentric coordinate to obtain a transformation displacement, adding each point in the three-dimensional coordinate array to the transformation displacement to obtain a new three-dimensional array, and rendering a line segment between every two points of the new three-dimensional array to finish real-time three-dimensional marking in space; the experimental results show that the method and the system provided by the invention can realize remote mixed reality labeling between Hololens end and other equipment such as PC end, and have higher accuracy.
Detailed Description
The invention will be further described with reference to the drawings and examples.
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
Example 1:
As shown in fig. 1, the present embodiment provides a remote labeling method based on mixed reality, including:
Acquiring a two-dimensional annotation array;
Converting the acquired two-dimensional annotation array into a three-dimensional coordinate array;
The method comprises the steps of obtaining a three-dimensional array, obtaining a barycentric coordinate according to the sum of all coordinates in the three-dimensional array in all directions, sending out rays at the barycentric coordinate, determining an object impacted by the rays as the object to be marked, subtracting the coordinates of the rays and the collision points of the object from the barycentric coordinate to obtain a transformation displacement, adding each point in the three-dimensional array with the transformation displacement to obtain a new three-dimensional array, and rendering a line segment between every two points of the new three-dimensional array to finish real-time three-dimensional marking in space.
The method in the embodiment can be realized according to a remote server, an assisting terminal (namely an assisting terminal) and a labeling terminal (namely an assisted terminal), wherein the server is connected with the assisting terminal and the assisted terminal, in the embodiment, the assisting terminal is configured to perform two-dimensional labeling and send the two-dimensional labeling to the assisted terminal, the assisted terminal is configured to convert the two-dimensional labeling into three-dimensional labeling and display the three-dimensional labeling in a specific position in real space in real time, convert the received two-dimensional coordinate array into a three-dimensional coordinate array which is the same with an object picture, calculate a barycentric coordinate, send out a ray by utilizing the barycentric coordinate, the object which is impacted by the ray is the object to be labeled, subtract the coordinate of an impact point and the barycentric coordinate point to obtain a transformation displacement, add each point in the three-dimensional coordinate array to obtain a new three-dimensional array, render a line segment between every two points of the new three-dimensional array, and finish the three-dimensional labeling in real time in space.
Specifically, as shown in fig. 2, the working process or principle of the remote labeling system based on mixed reality in this embodiment is as follows:
s1, constructing a remote server, and connecting an assisted end with an assisted end;
The method that two modes of Transmission Control Protocol (TCP) and user data packet protocol (UDP) are combined in the process of constructing the server is adopted, because the TCP protocol has very good reliability, before the TCP is adopted to transmit data, three handshakes are adopted to ensure the synchronous transmission and reception of a data end for tracking and negotiating the data transmitted each time, and in addition, mechanisms such as acknowledgement, timeout retransmission, sliding window, congestion control, flow control and delayed acknowledgement are adopted to ensure the integrity of the data in the process of data transmission, after the data transmission is completed, the connection is automatically disconnected, so that the system resources are saved, and the like.
As shown in fig. 1, the method of establishing a user pool (UserTokenPool) is used to achieve the improvement of communication efficiency by pulling the user into a Room (Room), the server uniformly processes the request of establishment and disconnection of two protocols by UnityNetWorkManager, the server delivers the received information to MESSAGEHANDLERCENTER to process the type and content of the identification information and execute the corresponding operation, the server uses MESSAGESENDMANAGER to send the request, and then the application layer is only responsible for interacting with MESSAGEHANDLERCENTER and MESSAGESENDMANAGER types, so that the encapsulation of the application layer communication can be achieved.
When the server is opened, the port and user pool are initialized, connection client events are registered, and receive and send events are registered for all users in all user pools. And then, each time a user is connected to the server, a user (UserToken) is popped (Pop) from the user pool, assigned as a socket of the current connected user, and after the server receives the user's request, a corresponding judgment is made for the user, and a result is returned. The server judges the heartbeat time, and the user sends heartbeat packets to the server at intervals of 30 seconds to judge whether the user is online, if the user is overtime, the user is disconnected, or the user is disconnected actively.
S2, establishing video connection;
When the assistance terminal and the assisted terminal initiate the video voice call invitation after establishing connection with the server, the remote server pulls the Token (Token) of the assistance terminal and the Token (Token) of the assisted terminal into the same Room (Room) and sends the internet protocol address (IP) and the Port (Port) of the other party to the two parties, thereby establishing the assistance terminal and the assisted terminal
User datagram protocol (User Datagram Protocol, UDP) connections.
After connection is established, a designated network camera is opened by using a Unity Application Programming Interface (API) to acquire video data, a real-time video input rendered Texture (WebConm Texture) is created, the real-time video input rendered Texture is required to be converted into a Texture2D format to acquire the video data, the Texture2D is created by taking RenderTexture as an intermediary, and an image is created by directly pointing to a memory address of the real-time camera rendered Texture and is stored in the Texture data, so that the memory is saved to finish conversion to acquire the video data.
S3, the assisting end marks and sends the two-dimensional mark to the assisted end;
as shown in fig. 3, a line renderer and an inverse interpolation algorithm may be used in the labeling process, specifically:
Interpolation lerp between two numbers y1,y2 is lerp=y1+(y2-y1) weigt, weigt is a real number in interval [0,1], and the inverse interpolation algorithm uses the known interpolation lerp and two numbers y1,y2 to find the weight value
After the auxiliary end clicks to start marking, every time the mouse presses the left button, the screen space coordinates of each frame of the mouse are automatically captured and saved into the stored coordinate array (Positions) and a point number is added, meanwhile, the last two coordinates (point 1, point 2) in the Positions array form a new coordinate array (NewPositions) to be circulated for a plurality of times, in this embodiment, 4 times, 0 is used as y1 and 3 is used as y2 and 0 is used as lerp, the value of weigt is obtained, the algorithm is used for obtaining the supplementary coordinates= (1-weigt) x point1+ weigt x point2, the calculated supplementary coordinates are added into a new coordinate array, meanwhile, the lerp value is added by 1 to enter the next circulation, after 4 times of circulation, the obtained supplementary coordinates between 2 points and the original two coordinates are obtained, as shown in table 1, the obtained coordinates are connected between each point until the auxiliary end is smoothed, and the auxiliary end marking is obtained until the auxiliary end is sent to the auxiliary end Positions.
Table 1 complements the coordinates and original two coordinate points
S4, the assisted end converts the two-dimensional annotation into a three-dimensional annotation, and displays the three-dimensional annotation at a specific position in a display space in real time to finish the annotation;
as shown in fig. 4, after the assisted terminal receives the coordinate array of the assisted terminal, the received coordinates are converted into a three-dimensional coordinate array at the same position from the user (camera) screen of the assisted terminal by using a Unity Application Programming Interface (API), and an algorithm is usedThe method comprises the steps of obtaining barycentric coordinates (x, y, z), utilizing barycentric coordinates to send out a ray forwards, storing coordinates (n, o, a) of a collision point when an object to be marked is the object to be marked, subtracting the coordinates of the collision point and the barycentric coordinates to obtain transformation displacement amounts (a, b, c), adding each point in a three-dimensional coordinate array to the transformation displacement amounts (a, b, c) to obtain a new three-dimensional array, and rendering a line segment between every two points of the new three-dimensional array to finish real-time three-dimensional marking in space, wherein xm is the sum of x axes of all coordinates, ym is the sum of y axes of all coordinates, zm is the sum of z axes of all coordinates, and m is the number of coordinates.
S5, as shown in fig. 5 and 6, the experimental results show that the method provided by the patent can realize remote labeling between the equipment of the assisted terminal and the equipment of the assisted terminal, and has higher accuracy.
Example 2:
the present embodiment provides a remote labeling terminal based on mixed reality, at least including a processor, where other settings of the terminal are conventional techniques, and the processor is configured to:
Acquiring a two-dimensional annotation array;
Converting the acquired two-dimensional annotation array into a three-dimensional coordinate array;
The method comprises the steps of obtaining a three-dimensional array, obtaining a barycentric coordinate according to the sum of all coordinates in the three-dimensional array in all directions, sending out rays at the barycentric coordinate, determining an object impacted by the rays as the object to be marked, subtracting the coordinates of the rays and the collision points of the object from the barycentric coordinate to obtain a transformation displacement, adding each point in the three-dimensional array with the transformation displacement to obtain a new three-dimensional array, and rendering a line segment between every two points of the new three-dimensional array to finish real-time three-dimensional marking in space.
When initiating video voice call invitation, the remote server pulls the token of the labeling terminal and the token of the assisted terminal into the same room, and sends the internet protocol address and port of the other party to the two parties, so that the assisted terminal and the assisted terminal establish a user datagram protocol.
The working method of the system is the same as that of the mixed reality-based remote labeling method in embodiment 1, and will not be described here again.
Example 3:
a mixed reality based remote labeling system, comprising:
a remote server, and at least one assisting terminal and at least one assisted terminal connected thereto;
When initiating video voice call invitation, the remote server pulls the token of the marked terminal and the token of the assisted terminal into the same room, and sends the internet protocol address and port of the other party to the two parties, so that the assisted terminal and the assisted terminal establish a user datagram protocol;
The labeling module is configured to calculate barycentric coordinates according to the sum of all coordinates in the three-dimensional coordinate array, send out rays at the barycentric coordinates, determine the object impacted by the rays as the object to be labeled, subtract the coordinates of the collision points of the rays and the object from the barycentric coordinates to obtain a transformation displacement, add each point in the three-dimensional coordinate array with the transformation displacement to obtain a new three-dimensional array, and render a line segment between every two points of the new three-dimensional array to complete real-time three-dimensional labeling in space.
The working method of the system is the same as that of the mixed reality-based remote labeling method in embodiment 1, and will not be described here again.
Example 4:
the embodiment provides a remote labeling system based on mixed reality, which comprises a data acquisition module, a conversion module and a labeling module;
the data acquisition module is configured to acquire a two-dimensional annotation array;
the conversion module is configured to convert the acquired two-dimensional annotation array into a three-dimensional coordinate array;
The labeling module is configured to calculate barycentric coordinates according to the sum of all coordinates in the three-dimensional coordinate array, send out rays at the barycentric coordinates, determine the object impacted by the rays as the object to be labeled, subtract the coordinates of the collision points of the rays and the object from the barycentric coordinates to obtain a transformation displacement, add each point in the three-dimensional coordinate array with the transformation displacement to obtain a new three-dimensional array, and render a line segment between every two points of the new three-dimensional array to complete real-time three-dimensional labeling in space.
The working method of the system is the same as that of the mixed reality-based remote labeling method in embodiment 1, and will not be described here again.
Example 5:
the present embodiment provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the mixed reality based remote labeling method described in embodiment 1.
Example 6:
The present embodiment provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the steps of the mixed reality-based remote labeling method described in embodiment 1 are implemented when the processor executes the program.
The above description is only a preferred embodiment of the present embodiment, and is not intended to limit the present embodiment, and various modifications and variations can be made to the present embodiment by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present embodiment should be included in the protection scope of the present embodiment.