100	system architecture	101、102、103	Terminal equipment
				104	Network system	105	Server device
4	Object size measuring device	401	Image acquisition module
				402	Point cloud fusion module	403	Three-dimensional reconstruction module
404	Size calculation module
				5	Computer equipment	51	Memory device
52	Processor and method for controlling the same	53	Network interface

Detailed Description

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the applications herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having" and any variations thereof in the description of the application and the claims and the description of the drawings above are intended to cover a non-exclusive inclusion. The terms first, second and the like in the description and in the claims or in the above-described figures, are used for distinguishing between different objects and not necessarily for describing a sequential or chronological order.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.

In order to make the person skilled in the art better understand the solution of the present application, the technical solution of the embodiment of the present application will be clearly and completely described below with reference to the accompanying drawings.

As shown in fig. 1, a system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

A user may interact with the server 105 via the network 104 using the terminal devices 101, 102, 103 to receive or send messages or the like, to operate the services and applications of the object size measurement method. Various communication client applications, such as a web browser application, a shopping class application, a search class application, an instant messaging tool, a mailbox client, social platform software, etc., may be installed on the terminal devices 101, 102, 103.

The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to user devices, network devices, or devices formed by integrating user devices with network devices through a network. The user equipment comprises, but is not limited to, any mobile electronic product which can perform man-machine interaction with a user through a touch pad, such as a smart phone, a tablet computer and the like, and any operating system can be adopted by the mobile electronic product, such as an android operating system, an iOS operating system and the like. The network device comprises an electronic device capable of automatically performing numerical calculation and information processing according to a preset or stored instruction, and the hardware of the electronic device comprises, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a programmable gate array (FPGA), a digital processor (DSP), an embedded device and the like. The network device includes, but is not limited to, a computer, a network host, a single network server, a plurality of network server sets, or a cloud of servers; here, the Cloud is composed of a large number of computers or network servers based on Cloud Computing (Cloud Computing), which is a kind of distributed Computing, a virtual supercomputer composed of a group of loosely coupled computer sets. Including but not limited to the internet, wide area networks, metropolitan area networks, local area networks, VPN networks, wireless Ad Hoc networks (Ad Hoc networks), and the like. Of course, those skilled in the art will appreciate that the above-described terminal devices are merely examples, and that other terminal devices now known or hereafter developed, as applicable, are within the scope of the present application and are incorporated herein by reference.

The server 105 may be a server, a server cluster comprising a plurality of servers, or a cloud computing service center. It may also be a server providing various services, such as a background server providing support for pages displayed on the terminal devices 101, 102, 103.

It should be noted that, the object size measurement method provided in the embodiment of the present application is generally executed by a terminal device, and accordingly, the object size measurement device is generally disposed in the terminal device.

It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

With the increasing functions of terminal mobile devices (such as mobile phones and tablet computers), users can carry the mobile devices with them every day, but not necessarily carry tools such as measuring tapes with them. For example, it is sometimes necessary to know the specific length or volume of an object if the object is to be placed in a certain place but there is a fear of insufficient placement space. Or sometimes also to know what the height of the person is in order to fill in the information etc. However, there is no tool for measuring the length of the object, and when there is no tool such as a measuring tape to measure the length of the object or the size of the object is too large to be directly measured by the measuring tape, a way of easily and conveniently measuring the size information of the object by using portable mobile terminal equipment is needed, so that the user can conveniently and rapidly know the size of the object which the user wants to know.

In view of the problems existing in the background art, the present application proposes the following technical solutions. Referring to FIG. 2, a flow chart of one embodiment of a method of object size measurement according to the present application is shown. The object size measuring method comprises the following steps:

101: and acquiring multi-frame images of the object to be detected, and the gesture and the point cloud of each frame of images.

In the embodiment of the invention, the image of the object to be detected can be acquired through a mobile terminal, such as a mobile phone camera, a tablet personal computer and other equipment with a camera shooting function.

In the embodiment of the present invention, the point cloud is a set of point data on the surface of the object to be measured, where the point cloud data is represented in the form of three-dimensional coordinates of world coordinates X, Y, and Z, and besides the geometric position information represented by (X, Y, and Z), the point cloud data may also represent RGB color, gray value, depth, segmentation result, and the like of one point. Specifically, the point cloud is obtained by using an accelerometer, a gyroscope, a camera and other sensors on the mobile device to collect images of an object to be measured, video image data in the images, point cloud data and the like, then using algorithms such as VIO (Visual Inertial Odometry, visual inertial odometer) or SLAM (Simultaneous localization AND MAPPING, even if instant positioning and image construction) and the like to estimate the relative positions of moving targets at different moments and process the accumulated errors of estimation results of the visual odometer, and constructing three-dimensional coordinates of the object to be measured.

In this embodiment, when the image of the object to be measured is acquired in step 101, a mobile device such as a mobile phone needs to be slowly moved, the image of the object to be measured is acquired as an offset image that is continuously acquired, then a large amount of point information on the surface of the target object is acquired through a VIO algorithm, as shown in fig. 3, the hashed points in the figure are point clouds of the object to be measured, and the acquired point cloud coordinates are stored.

Further, the mobile terminal equipment acquires multi-frame images of the object to be detected, acquires the posture of each frame of image at the moment of acquisition and corresponding sparse point clouds, and records and stores the number, the three-dimensional coordinates and the corresponding image characteristics of the point clouds.

102: And updating and expanding the point cloud of the images according to the posture change among the multiple frames of images, and fusing the same point cloud among the multiple frames of images.

In step 101, the mobile device such as a mobile phone needs to be slowly moved, the object to be measured is continuously observed by adopting different angles or poses, the point cloud is continuously updated and expanded, and the observation results of different views are accumulated.

According to the embodiment of the invention, the point clouds are repeatedly positioned and collected from multiple view angles, the point clouds of the images are updated and expanded, the multiple frames of point clouds are transformed into the same space coordinate system by utilizing the point cloud splicing technology, and the same point clouds among the multiple frames of images are fused to restore the whole scene or object. According to the embodiment, the relative positions of the object to be detected at different moments are estimated by acquiring the gesture of each frame of image at the acquisition moment, the position change of the point cloud of the image is tracked according to the relative position change of the object to be detected, the point cloud information in each frame of image is recorded, and the number, the coordinates and the corresponding image characteristics of the point cloud are recorded; and fusing the same point cloud among the multi-frame images according to the numbers, the coordinates and the corresponding image features. The same point clouds among multiple frames of images are fused by updating and expanding the point clouds of the images, and the three-dimensional point cloud information acquired by the embodiment is comprehensive in coverage, so that accurate data is provided for the point cloud data processing of the subsequent three-dimensional reconstruction, and the data processing work is greatly facilitated.

Specifically, in this embodiment, an ICP (ITERATIVE CLOSEST POINT ) algorithm is used to fuse overlapping point clouds, and point clouds under different views are unified into the same reference coordinate system by certain rotation and translation transformation.

Further, fusing the same point cloud between the images comprises the following processes:

(1) A point set P_i epsilon P is acquired in a frame of image point cloud P;

(2) From a corresponding point set Q_i e Q in another frame image point cloud Q, so that ||q_i-p_i ||=min;

(3) Calculating a rotation matrix R and a translation matrix t so as to minimize an error function;

(4) Performing rotation and translation transformation on pi by using the rotation matrix R and the translation matrix t obtained in the previous step to a new corresponding point set P_i'＝{p_i'＝Rp_i+t,p_i epsilon P;

(5) The average distance of p_i' from the corresponding point set q_i is calculated:

(6) If d is less than a given threshold or greater than a preset maximum number of iterations, the iterative calculation is stopped. Otherwise, returning to the step (2) until the convergence condition is met.

In this embodiment, the error function in step (3) is E (R, t) as follows:

Wherein n is the number of nearest point pairs, P_i is one point in the target point cloud P, qi is the nearest point corresponding to pi in the source point cloud Q, R is a rotation matrix, and t is a translation vector.

In other embodiments, other algorithms may be used to fuse the point clouds, such as feature matching methods, TSDF (spatial voxel based fusion) algorithms, or clustering concept based point cloud fusion methods.

103: And carrying out three-dimensional reconstruction on the scene of the object to be detected, and calculating coordinates of the reconstructed point cloud.

In the embodiment of the invention, the object image to be detected obtained in step 101 is a two-dimensional image, the same points among multiple frames of images are fused through step 102, the three-dimensional point clouds among the multiple frames of images form a corresponding relationship, and the characteristics among the multiple frames of images correspond one by one to form three-dimensional matching. And carrying out three-dimensional scene reconstruction on the object to be detected based on the point cloud information of fusion matching among multiple frames, and recovering the geometric structure of the object to be detected to obtain a three-dimensional model of the object to be detected. Further, through three-dimensional reconstruction of multiple visual angles, coordinates of point clouds in a scene of an object to be detected are updated continuously along with the change of the gesture, so that errors caused by a single visual angle can be avoided.

In the embodiment of the invention, three-dimensional reconstruction is carried out on the scene through sfm (structure from motion, from a motion recovery structure) and other algorithms, and new three-dimensional coordinates are sampled from the reconstructed scene according to the stored coordinates or image features of the point cloud. Specifically, in this embodiment, when the first frame image of the object to be measured is acquired, the point cloud information of the object to be measured is recorded, the first frame image is recorded as a first key frame, along with the step 101, the mobile terminal device, when moving greatly, selects another frame image as a second key frame, continuously updates the new key frame, and when more than two key frames are acquired, three-dimensional reconstruction is performed on the scene of the object to be measured according to the images, feature points and point clouds of the key frames and the pose of the mobile terminal, and the coordinates of the reconstructed point cloud are calculated.

More specifically, the sfm algorithm describes and extracts feature points and point clouds of the image with ORB (Oriented FAST and Rotated BRIEF, algorithm for fast feature point extraction and description) by extracting point clouds and feature points of all key frames. Matching any two key frames, constraining the fused point clouds in the step 102 by using a homography matrix (homography matrix) or a basic matrix (fundamental matrix), and removing the misfused point clouds (for example, removing mismatching points by using a RANSAC method). Projecting the correct camera pose (R, t) and the point cloud to a zero system according to the error function in the step 102, wherein the zero system refers to a camera coordinate system corresponding to the first key frame; and then, optimizing by using a beam adjustment method (Bundle Adjustment, BA), projecting all three-dimensional points in the zero system onto each key frame according to the camera pose (R, t) of each key frame, calculating the projection error of each three-dimensional point, adjusting the camera pose (R, t) of each key frame, and carrying out multiple iterations to minimize the projection error.

104: And selecting a point cloud range, and measuring the size of the object to be measured according to the point cloud coordinates in the range.

In the prior art, when a point is selected, a ray is usually emitted from the position of an AR camera into a scene along the center of the screen or along the direction of the point where a user clicks the screen, an intersection point is calculated when the ray intersects with an estimated plane in the scene, the coordinates of a point cloud point in the current frame point cloud are calculated when the ray intersects with the point cloud point or the distance is within a certain range, and the point closest to the camera is selected from the intersection results as the point selected by the user at this time. This approach only takes into account the current view, and the selected points are not necessarily on the object surface, which is generally the case when changing view. That is, the accuracy of the measurement method in the prior art depends on the accuracy of two points selection, the measurement result is occasional, and once the point selection is performed, the point selection is not updated and improved in the subsequent AR session, and the constraint update of multiple views is not supported.

When the point cloud is selected, a range is determined from the image of the object to be detected, the selected range can be rectangular, circular or other shapes, rays are projected into the scene from the position of the mobile terminal camera through the determined screen range, a selected light cone is formed, and the point cloud positioned in the light cone is selected. Further, a point cloud closest to the camera is selected from the point clouds of the same ray.

Three-dimensional coordinates of each point of the selected point clouds are acquired, and the size of the object is calculated according to the point clouds. Specifically, the height of the object to be measured is calculated by subtracting the minimum value from the maximum value of the y coordinates of the point clouds, and the width and height of the object to be measured are determined by projecting the x, z coordinates of the point clouds in the principal axis direction and selecting the maximum and minimum projection values. The principal axis direction of the coordinates is obtained by decomposing the eigenvalues of the x, z coordinates, so that the three-dimensional bounding box can be determined according to the actual aspect ratio and placement of the object, instead of the maximum and minimum x, y and z in the world coordinate system.

In yet another embodiment, selecting the point cloud is back-projecting the sparse point cloud onto the screen, and then selecting the point cloud that is within range. The selected point cloud is displayed on the screen in a different color.

It should be noted that, after determining the range of the object through the above steps, the embodiment of the present invention may convert the range into a three-dimensional cube identifier and draw the three-dimensional cube identifier on a screen. As shown in fig. 3, the range of the object to be measured is the range of the cube in the figure. Along with the continuous selection of the user, the three-dimensional bounding box is also continuously updated, so that the measurement progress and accuracy which are mastered by the user in real time can be realized.

The embodiment of the application provides an object size measuring method, which comprises the steps of updating and expanding point clouds of images, fusing the same point clouds among multiple frames of images to update and expand the point clouds of the images, acquiring three-dimensional point cloud information, covering comprehensively, providing accurate data to reconstruct three-dimensional scenes of the images, continuously updating coordinates in the scenes, and avoiding errors caused by a single visual angle, so that a measuring result is more accurate when the object to be measured is measured in a selected point cloud range.

In order to solve the above technical problems, as shown in fig. 4, the embodiment of the present application further provides an object size measuring device 4.

An object size measuring device 4 comprising:

The image acquisition module 401 is configured to acquire a multi-frame image of an object to be detected, and a pose and a point cloud of each frame of the image;

The point cloud fusion module 402 is configured to update and expand a point cloud of the image according to the change of the image pose, and fuse the same point clouds among the multiple frames of images;

the three-dimensional reconstruction module 403 is configured to perform three-dimensional reconstruction on the scene of the image, and calculate coordinates of the reconstructed point cloud;

and the size calculation module 404 is configured to select a point cloud range to measure the object to be measured.

According to the embodiment of the application, the image acquisition module 401 is used for acquiring multi-frame images of the object to be detected, the pose of each frame of the images and the point cloud, the point cloud fusion module 402 updates and expands the point clouds of the images according to the change of the image poses, the same point clouds among the multi-frame images are fused, the acquired three-dimensional point cloud information is comprehensively covered by the point cloud updating and expanding of the images, accurate data are provided to enable the three-dimensional reconstruction module 403 to reconstruct the three-dimensional scene of the image, the coordinates in the scene are continuously updated, errors caused by a single visual angle can be avoided, and the measurement result of the object to be detected by the point cloud selection module 404 is more accurate. The object size measuring device can measure the object size more conveniently and rapidly, and has high measuring reliability and accuracy.

The object size measuring device 4 further comprises a display module, wherein the display module is connected with the adjusting module and is used for displaying data information and operation pages of the object size measuring device.

The object size measuring device 4 may further include an input module, where the input module is connected to the display module, the input module may include a key, and may be used to input information such as an account number, a password, a name, etc. of the user id, and the operation page in the object size measuring process may be displayed in the display module in the object size measuring device, and the display module may further display other information of the user, and store the information, so that the user may check at any time.

It should be noted that, the apparatus in this embodiment belongs to the same concept as the method embodiment, the specific implementation process of the apparatus in this embodiment is detailed in the method embodiment, and the technical features in the method embodiment are correspondingly applicable in this embodiment, which is not described herein again.

In order to solve the above technical problems, the embodiment of the present application further provides a computer device 5. Referring specifically to FIG. 5, (Application SPECIFIC INTEGRATED Circuit, ASIC), programmable gate array (Field-Programmable GATE ARRAY, FPGA), digital Processor (DIGITAL SIGNAL Processor, DSP), embedded devices, and the like.

The computer equipment can be a desktop computer, a notebook computer, a palm computer, a cloud server and other computing equipment. The computer equipment can perform man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch pad or voice control equipment and the like.

The memory 51 includes at least one type of readable storage medium including flash memory, hard disk, multimedia card, card memory (e.g., SD or DX memory, etc.), random Access Memory (RAM), static Random Access Memory (SRAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), programmable Read Only Memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the storage 51 may be an internal storage unit of the computer device 5, such as a hard disk or a memory of the computer device 5. In other embodiments, the memory 51 may also be an external storage device of the computer device 5, such as a plug-in hard disk, a smart memory card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD) or the like, which are provided on the computer device 5. Of course, the memory 51 may also comprise both an internal memory unit of the computer device 5 and an external memory device. In this embodiment, the memory 51 is typically used for storing an operation fund system and various kinds of application software installed in the computer device 5, such as program codes of an object size measuring method, and the like. Further, the memory 51 may be used to temporarily store various types of data that have been output or are to be output.

The processor 52 may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 52 is typically used to control the overall operation of the computer device 5. In this embodiment, the processor 52 is configured to execute the program code stored in the memory 51 or process data, such as the program code for executing the object size measuring method.

The network interface 53 may comprise a wireless network interface or a wired network interface, which network interface 53 is typically used to establish communication connections between the computer device 5 and other electronic devices.

The present application also provides another embodiment, namely, a computer-readable storage medium storing an object size measurement program executable by at least one processor to cause the at least one processor to perform the steps of the object size measurement method as described above.

From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present application.

It is apparent that the above-described embodiments are only some embodiments of the present application, but not all embodiments, and the preferred embodiments of the present application are shown in the drawings, which do not limit the scope of the patent claims. This application may be embodied in many different forms, but rather, embodiments are provided in order to provide a thorough and complete understanding of the present disclosure. Although the application has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that modifications may be made to the embodiments described in the foregoing description, or equivalents may be substituted for elements thereof. All equivalent structures made by the content of the specification and the drawings of the application are directly or indirectly applied to other related technical fields, and are also within the scope of the application.