100	system architecture	101、102、103	Terminal device
				104	Network	105	Server
4	Object sizemeasuring device	401	Image acquisition module
				402	Pointcloud fusion module	403	Three-dimensional reconstruction module
404	Size calculation module
				5	Computer equipment	51	Memory device
52	Processor with a memory having a plurality ofmemory cells	53	Network interface

Detailed Description

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "including" and "having," and any variations thereof, in the description and claims of this application and the description of the above figures are intended to cover non-exclusive inclusions. The terms "first," "second," and the like in the description and claims of this application or in the above-described drawings are used for distinguishing between different objects and not for describing a particular order.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.

In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

As shown in fig. 1, thesystem architecture 100 may include

terminal devices

101, 102, 103, anetwork 104, and aserver 105. Thenetwork 104 serves as a medium for providing communication links between the

terminal devices

101, 102, 103 and theserver 105.Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user can use the

terminal devices

101, 102, 103 to interact with theserver 105 via thenetwork 104 to receive or send messages or the like to operate the services and applications of the object dimension measuring method. The

terminal devices

101, 102, 103 may have various communication client applications installed thereon, such as a web browser application, a shopping application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like.

The

terminal devices

101, 102, 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to user devices, network devices, or devices formed by integrating user devices and network devices through a network. The user equipment includes, but is not limited to, any mobile electronic product, such as a smart phone, a tablet computer, and the like, which can perform human-computer interaction with a user through a touch panel, and the mobile electronic product may employ any operating system, such as an android operating system, an iOS operating system, and the like. The network device includes an electronic device capable of automatically performing numerical calculation and information processing according to preset or stored instructions, and the hardware includes but is not limited to a microprocessor, an Application Specific Integrated Circuit (ASIC), a programmable gate array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like. The network device comprises but is not limited to a computer, a network host, a single network server, a plurality of network server sets or a cloud formed by a plurality of servers; here, the Cloud is composed of a large number of computers or web servers based on Cloud Computing (Cloud Computing), which is a kind of distributed Computing, one virtual supercomputer consisting of a collection of loosely coupled computers. Including, but not limited to, the internet, a wide area network, a metropolitan area network, a local area network, a VPN network, a wireless Ad Hoc network (Ad Hoc network), etc. Of course, those skilled in the art should understand that the above terminal device is only an example, and other existing or future terminal devices may be applicable to the present application, and are included in the scope of the present application and are incorporated herein by reference.

Theserver 105 may be a server, a server cluster composed of several servers, or a cloud computing service center. It may also be a server providing various services, such as a background server providing support for pages displayed on the

terminal devices

101, 102, 103.

It should be noted that the object size measuring method provided in the embodiments of the present application is generally executed by a terminal device, and accordingly, the object size measuring apparatus is generally disposed in the terminal device.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

With the increasing functionality of terminal mobile devices (e.g., mobile phones, tablet computers, etc.), users will carry the mobile devices with them every day, but will not necessarily carry tools such as measuring scales with them. For example, sometimes, if an object is placed in a certain place, but the placing space is not enough, the specific length or volume of the object needs to be known. Or sometimes it is also desirable to know what the person's height is in order to fill in information, etc. However, the user does not have a tool for measuring the length of the object, and when the user does not have a measuring scale or other tools for measuring the length and other dimensions of the object or the size of the object is too large to be directly measured by the measuring scale, a method for easily and conveniently measuring the size information of the object by using the portable mobile terminal device is needed, so that the user can conveniently and quickly know the size of the object to be known.

In view of the problems in the background art, the following technical solutions are proposed in the present application. Referring to FIG. 2, a flow chart of one embodiment of a method of measuring dimensions of an object according to the present application is shown. The object size measuring method comprises the following steps:

101: acquiring a plurality of frames of images of the object to be detected and the posture and point cloud of each frame of image.

In the embodiment of the invention, the image of the object to be detected can be acquired through a mobile terminal, such as a mobile phone camera, a tablet personal computer and other equipment with a camera shooting function.

In the embodiment of the present invention, the point cloud is a set of point data on the surface of the object to be measured, and the point cloud data is represented in the form of three-dimensional coordinates of world coordinates X, Y, and Z, and in addition, the point cloud data may also represent RGB color, gray value, depth, segmentation result, and the like of one point in addition to the geometric location information represented by (X, Y, Z). Specifically, the point cloud is obtained by acquiring an image of an object to be measured, video image data and point cloud data in the image by using sensors such as an accelerometer, a gyroscope and a camera on mobile equipment, then estimating the relative position of the moving target at different times by using algorithms such as VIO (Visual Inertial odometer) or SLAM (Simultaneous localization and mapping, even if positioning and mapping are performed instantly) and processing the accumulated error of the estimation result of the Visual odometer, and performing three-dimensional coordinate construction on the object to be measured.

In this embodiment, when the image of the object to be detected is obtained instep 101, it is necessary to slowly move a mobile device such as a mobile phone, and obtain the image of the object to be detected as a continuously obtained offset image, and then acquire information of a large number of points on the surface of the target object through a VIO algorithm, as shown in fig. 3, the hashed points in the image are point clouds of the object to be detected, and store coordinates of the acquired point clouds.

Further, the mobile terminal equipment acquires multi-frame images of the object to be detected, acquires the posture of each frame image at the acquisition moment and corresponding sparse point cloud, and records and stores the number, three-dimensional coordinates and corresponding image characteristics of the point cloud.

102: and updating and expanding the point clouds of the images according to the posture change among the multiple frames of images, and fusing the same point clouds among the multiple frames of images.

Instep 101, mobile devices such as a mobile phone need to be slowly moved, the object to be measured is constantly observed at different angles or poses, and the point cloud is also continuously updated and expanded, so that observation results of different visions are accumulated.

The embodiment of the invention repeatedly positions and collects point clouds from a plurality of visual angles, updates and expands the point clouds of the images, transforms multi-frame point clouds to the same space coordinate system by using a point cloud splicing technology, and fuses the same point clouds among the multi-frame images so as to restore the whole scene or object. In the embodiment, the relative position of the object to be detected at different moments is estimated by acquiring the posture of each frame of image at the acquisition moment, the point cloud position change of the image is tracked according to the relative position change of the object to be detected, the point cloud information in each frame of image is recorded, and the number, the coordinates and the corresponding image characteristics of the point cloud are recorded; and fusing the same point clouds among the multiple frames of images according to the numbers, the coordinates and the corresponding image characteristics. The same point clouds among multiple frames of images are fused by updating and expanding the point clouds of the images, the three-dimensional point cloud information acquired by the embodiment is comprehensively covered, accurate data are provided for the subsequent point cloud data processing of three-dimensional reconstruction, and the data processing work is greatly facilitated.

Specifically, in this embodiment, an ICP (Iterative Closest Point) algorithm is used to fuse the overlapped Point clouds, and Point clouds in different visual environments are converted by a certain rotation and translation to unify Point cloud data in different coordinate systems into the same reference coordinate system.

Further, fusing the same point clouds among the images comprises the following processes:

(1) taking a point set P in a frame of image point cloud P_i∈P；

(2) From a corresponding point set Q in another frame of image point cloud Q_iBelongs to Q, so that | | | Q_i-p_i||＝min；

(3) Calculating a rotation matrix R and a translation matrix t to minimize an error function;

(4) carrying out rotation and translation transformation on pi by using the rotation matrix R and the translation matrix t obtained in the previous step to obtain a new corresponding point set p_i’＝{p_i’＝Rp_i+t，p_i∈P}；

(5) Calculating p_i' and corresponding point set q_iAverage distance of (d):

(6) if d is less than a given threshold or greater than a preset maximum number of iterations, the iterative computation is stopped. Otherwise, returning to the step (2) until the convergence condition is met.

In this embodiment, the error function E (R, t) in step (3) is:

where n is the number of nearest neighbor point pairs, p_iAnd (3) taking one point in the target point cloud P, qi as the closest point corresponding to pi in the source point cloud Q, R as a rotation matrix, and t as a translation vector.

In other embodiments, other algorithms may be used to fuse the point cloud, such as a feature matching method, a TSDF (spatial voxel based fusion method) algorithm, or a point cloud fusion method based on a clustering concept, etc.

103: and carrying out three-dimensional reconstruction on the scene of the object to be detected, and calculating the coordinates of the point cloud after reconstruction.

In the embodiment of the invention, the image of the object to be detected acquired instep 101 is a two-dimensional image, the same points among multiple frames of images are fused instep 102, the three-dimensional point clouds among the multiple frames of images form a corresponding relation, and the characteristics of the multiple frames of images correspond one to form stereo matching. And based on the point cloud information fused and matched among the multiple frames, carrying out three-dimensional scene reconstruction on the object to be detected, recovering the geometric structure of the object to be detected, and obtaining a three-dimensional model of the object to be detected. Furthermore, through the three-dimensional reconstruction of multiple visual angles, the coordinates of the point cloud in the object scene to be detected can be continuously updated along with the change of the posture, and errors caused by a single visual angle can be avoided.

In the embodiment of the invention, a scene is three-dimensionally reconstructed through sfm (structure from motion) and other algorithms, and new three-dimensional coordinates are sampled from the reconstructed scene according to the stored coordinates of the point cloud or image features. Specifically, in this embodiment, a first frame of image of the object to be measured is obtained, point cloud information of the object to be measured is recorded, the frame serving as the frame for recording the point cloud is recorded as a first key frame, along withstep 101, the terminal device is moved greatly, another frame of image is selected as a second key frame, new key frames are continuously updated, when more than two key frames are obtained, a three-dimensional reconstruction is performed on a scene of the object to be measured according to the image, the feature point, the point cloud of the key frame and the pose of the mobile terminal, and coordinates of the point cloud after the reconstruction are calculated.

More specifically, the sfm algorithm describes and extracts feature points and point clouds of the image by ORB (organized FAST and Rotated bright BRIEF, algorithm for FAST feature point extraction and description) by extracting point clouds and feature points of all key frames. Matching any two key frames, constraining the point clouds fused in thestep 102 by using a homography matrix (homography matrix) or a fundamental matrix (fundamental matrix), and rejecting misfused point clouds (for example, rejecting mismatching points by using a RANSAC method). Projecting the correct camera attitude (R, t) and point cloud to a zero system according to the error function in thestep 102, wherein the zero system is a camera coordinate system corresponding to the first key frame; and then, optimizing by using a Beam Adjustment (BA), projecting all three-dimensional points in a zero system onto each key frame according to the camera posture (R, t) of each key frame, calculating the projection error of each three-dimensional point, adjusting the camera posture (R, t) of each key frame, and performing multiple iterations to reduce the projection error to the minimum.

104: and selecting a point cloud range, and measuring the size of the object to be measured according to the point cloud coordinates in the range.

In the prior art, a ray is generally emitted into a scene from the position of an AR camera along the center of the screen or the direction of a click point of a user clicking the screen, an intersection point is calculated when the ray intersects with an estimated plane in the scene, the coordinates of a point in a current frame point cloud are calculated when the ray intersects with the point or the distance of the point is within a certain range, and a point closest to the camera is selected from the intersection results as a point selected by the user. This approach only considers the situation at the current viewing angle, and the selected point is not necessarily on the object surface, which is generally the case when the viewing angle is changed. Namely, the accuracy of the measurement method in the prior art depends on the accuracy of two point selection, the measurement result has high contingency, once the point is selected, the point cannot be updated and improved in the subsequent AR session, and the constraint update of multiple visual angles is not supported.

In the embodiment, when the point cloud is selected, a range is determined from the image of the object to be detected, the selected range can be rectangular, circular or other shapes, rays are projected into a scene from the position of the mobile terminal camera through the determined screen range to form a selected light cone, and the point cloud located in the light cone is selected. Further, the point cloud of the same ray is selected as the point cloud closest to the camera.

Three-dimensional coordinates of each point of the selected point cloud are acquired, and the size of the object is calculated according to the point clouds. Specifically, the height of the object to be measured is calculated by subtracting the minimum value from the maximum value of the y coordinates of the point clouds, and the width and the height of the object to be measured are determined by projecting the x and z coordinates of the point clouds to the main axis direction and selecting the maximum and minimum projection values. The main axis direction of the coordinates is obtained by decomposing the eigenvalues of the x and z coordinates, so that the three-dimensional bounding box can be determined according to the actual aspect ratio and the placement of the object, rather than the maximum and minimum x, y, and z in the world coordinate system.

In yet another embodiment, selecting the point cloud is back projecting the sparse point cloud onto a screen and then selecting the point cloud that lies within the range. The selected point clouds are displayed on the screen in different colors.

It should be noted that, after the object range is determined through the above steps, the range may be converted into a three-dimensional cube identifier, and the three-dimensional cube identifier may be drawn on a screen. As shown in fig. 3, the range of the object to be measured is a range of a cube in the drawing. With the continuous selection of the user, the three-dimensional surrounding frame is continuously updated, so that the measurement progress and the measurement precision can be mastered by the user in real time.

The embodiment of the application provides an object size measurement method, through right the point cloud of image updates the extension, through inciting somebody to action the same point cloud fuses between the multiframe image to the point cloud to the image updates the extension, and the three-dimensional point cloud information that obtains covers comprehensively, provides accurate data right the scene of image carries out three-dimensional reconstruction, and the coordinate in the scene can constantly be updated, can avoid the error that single visual angle caused, so that it is right to select the point cloud scope the measuring result is more accurate when the object that awaits measuring is measured.

In order to solve the above technical problem, as shown in fig. 4, an object dimension measuring device 4 is further provided in the embodiment of the present application.

An object dimension measuring apparatus 4, comprising:

theimage acquisition module 401 is configured to acquire a plurality of frames of images of an object to be detected and a posture and a point cloud of each frame of the images;

a pointcloud fusion module 402, configured to update and expand the point cloud of the image according to the change of the image pose, and fuse the same point cloud among the multiple frames of images;

a three-dimensional reconstruction module 403, configured to perform three-dimensional reconstruction on the scene of the image, and calculate coordinates of a point cloud after reconstruction;

and thesize calculation module 404 is configured to select a point cloud range to measure the object to be measured.

The embodiment of the application acquires multiframe images and each frame of an object to be measured through theimage acquisition module 401 the attitude and the point cloud of the image, the pointcloud fusion module 402 is used for covering the acquired three-dimensional point cloud information comprehensively according to the change of the image attitude, right the point cloud updating and expanding of the image, and the same point cloud fusion between the multiframe images, and through the point cloud updating and expanding of the image, the acquired three-dimensional point cloud information is covered comprehensively, so that the three-dimensional reconstruction module 403 is provided, the scene of the image is subjected to three-dimensional reconstruction, the coordinate in the scene is continuously updated, the error caused by a single visual angle can be avoided, and the point cloud range is selected by thesize calculation module 404 to be more accurate in the. The object size measuring device can measure the size of the object more conveniently and quickly, and has high measuring reliability and accuracy.

The object size measuring device 4 further comprises a display module, the display module is connected with the adjusting module, and the display module is used for displaying data information and an operation page of the object size measuring device.

The object size measuring device 4 may further include an input module, the input module is connected to the display module, the input module may include a key, and may be configured to input information such as an account number, a password, and a name of a user id, the object size measurement process operation page may be displayed on the display module in the object size measuring device, and the display module may further display other information of the user, and store the information, so that the user can view the information at any time.

It should be noted that the apparatus of this embodiment and the method embodiment belong to the same concept, and specific implementation processes thereof are detailed in the method embodiment, and technical features in the method embodiment are all correspondingly applicable in this embodiment, which is not described herein again.

In order to solve the above technical problem, the embodiment of the present application further provides a computer device 5. Referring to fig. 5, (Application Specific Integrated Circuit, ASIC), Programmable Gate Array (FPGA), Digital Signal Processor (DSP), embedded device, etc.

The computer device can be a desktop computer, a notebook, a palm computer, a cloud server and other computing devices. The computer equipment can carry out man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch panel or voice control equipment and the like.

Thememory 51 includes at least one type of readable storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, thememory 51 may be an internal storage unit of the computer device 5, such as a hard disk or a memory of the computer device 5. In other embodiments, thememory 51 may also be an external storage device of the computer device 5, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the computer device 5. Of course, thememory 51 may also comprise both an internal storage unit of the computer device 5 and an external storage device thereof. In this embodiment, thememory 51 is generally used for storing an operating fund system installed in the computer device 5 and various types of application software, such as program codes of an object dimension measuring method. Further, thememory 51 may also be used to temporarily store various types of data that have been output or are to be output.

Theprocessor 52 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. Theprocessor 52 is typically used to control the overall operation of the computer device 5. In this embodiment, theprocessor 52 is configured to execute the program code stored in thememory 51 or process data, for example, execute the program code of the object dimension measuring method.

Thenetwork interface 53 may comprise a wireless network interface or a wired network interface, and thenetwork interface 53 is generally used for establishing communication connections between the computer device 5 and other electronic devices.

The present application provides yet another embodiment, which provides a computer-readable storage medium storing an object dimension measurement program executable by at least one processor to cause the at least one processor to perform the steps of the object dimension measurement method as described above.

Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present application.

It is to be understood that the above-described embodiments are merely illustrative of some, but not restrictive, of the broad invention, and that the appended drawings illustrate preferred embodiments of the invention and do not limit the scope of the invention. This application is capable of embodiments in many different forms and is provided for the purpose of enabling a thorough understanding of the disclosure of the application. Although the present application has been described in detail with reference to the foregoing embodiments, it will be apparent to one skilled in the art that the present application may be practiced without modification or with equivalents of some of the features described in the foregoing embodiments. All equivalent structures made by using the contents of the specification and the drawings of the present application are directly or indirectly applied to other related technical fields and are within the protection scope of the present application.

Claims

1. A method of measuring the dimensions of an object, comprising the steps of:

2. The method for measuring the size of an object according to claim 1, wherein the updating and expanding of the point clouds of the images according to the posture change among the multiple frames of images and the fusion of the same point clouds among the multiple frames of images comprise

3. The object dimension measuring method according to claim 2, wherein the algorithm for fusing the same point cloud among the plurality of frames of images includes a feature matching algorithm or a point cloud ICP algorithm.

4. The object dimension measuring method of claim 1, wherein three-dimensionally reconstructing a scene of the image and calculating coordinates of a point cloud after reconstruction comprises

5. The method as claimed in claim 4, wherein the selecting the point cloud range to measure the object comprises

Selecting a range in the image;

projecting rays on the scene in the selected range to form a light cone;

6. The method as claimed in claim 4, wherein the selecting a point cloud range for measuring the object to be measured further comprises

7. The object dimension measuring method according to claim 5 or 6, characterized in that a three-dimensional solid mark is formed for the object to be measured at the time of measuring the dimension of the object to be measured.

8. An object dimension measuring apparatus, comprising:

the point cloud fusion module is used for updating and expanding the point cloud of the images according to the posture change among the multi-frame images and fusing the same point cloud among the multi-frame images;

9. A computer device, characterized by comprising a memory in which a computer program is stored, a processor and a network interface, the processor implementing the steps of the object dimension measuring method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that a computer program is stored thereon, which computer program, when being executed by a processor, carries out the steps of the object dimension measuring method according to any one of claims 1 to 7.