CN115456898A - Method and device for building image of parking lot, vehicle and storage medium - Google Patents
Method and device for building image of parking lot, vehicle and storage mediumDownload PDFInfo
- Publication number
- CN115456898A CN115456898ACN202211107120.2ACN202211107120ACN115456898ACN 115456898 ACN115456898 ACN 115456898ACN 202211107120 ACN202211107120 ACN 202211107120ACN 115456898 ACN115456898 ACN 115456898A
- Authority
- CN
- China
- Prior art keywords
- parking lot
- vehicle
- image
- ground
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/10—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration
- G01C21/12—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning
- G01C21/16—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation
- G01C21/165—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation combined with non-inertial navigation instruments
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S19/00—Satellite radio beacon positioning systems; Determining position, velocity or attitude using signals transmitted by such systems
- G01S19/38—Determining a navigation solution using signals transmitted by a satellite radio beacon positioning system
- G01S19/39—Determining a navigation solution using signals transmitted by a satellite radio beacon positioning system the satellite radio beacon positioning system transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO
- G01S19/42—Determining position
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/11—Complex mathematical operations for solving equations, e.g. nonlinear equations, general mathematical optimization problems
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
- G06T3/047—Fisheye or wide-angle transformations
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/277—Analysis of motion involving stochastic approaches, e.g. using Kalman filters
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Remote Sensing (AREA)
- Radar, Positioning & Navigation (AREA)
- Mathematical Physics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Pure & Applied Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Analysis (AREA)
- Computational Mathematics (AREA)
- Mathematical Optimization (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Algebra (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Operations Research (AREA)
- Automation & Control Theory (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Navigation (AREA)
Abstract
Description
Translated fromChinese技术领域technical field
本申请涉及车辆融合定位技术领域，特别涉及一种停车场的建图方法、装置、车辆及存储介质。The present application relates to the technical field of vehicle fusion positioning, and in particular to a parking lot mapping method, device, vehicle and storage medium.
背景技术Background technique
智能驾驶技术是一种可以实现车辆在无人为干预的条件下，自动进行融合定位、环境感知、路径规划以及自动控制实现自主驾驶的技术。该技术基于相机、雷达、声波、惯导等传感器，以及众多的定位、识别、感知、规划和控制等算法，实现车辆自身的物理定位、环境感知、障碍识别、路径规划以及行车控制等功能，最终完成自主驾驶。Intelligent driving technology is a technology that enables vehicles to automatically perform fusion positioning, environment perception, path planning, and automatic control to achieve autonomous driving without human intervention. Based on sensors such as cameras, radars, acoustic waves, and inertial navigation, as well as numerous positioning, identification, perception, planning, and control algorithms, this technology realizes the vehicle's own physical positioning, environmental perception, obstacle identification, path planning, and driving control functions. Finally complete autonomous driving.
停车场建图与定位技术是智能驾驶的一项子技术，是实现车辆在停车场实现自动泊车的基础技术之一，车辆通过车身传感器获取周围环境图像、点云、自身角速度、加速度等数据，并通过算法处理传感器数据，估计自身位姿构建周围环境的点云地图，为后续的重定位以及自动泊车的路径规划提供依据。Parking lot mapping and positioning technology is a sub-technology of intelligent driving, and it is one of the basic technologies to realize automatic parking of vehicles in the parking lot. The vehicle obtains surrounding environment images, point clouds, own angular velocity, acceleration and other data through body sensors , and process the sensor data through the algorithm, estimate its own position and pose to construct the point cloud map of the surrounding environment, and provide a basis for subsequent relocation and automatic parking path planning.
相关技术中，大多采用基于激光雷达等传感器的融合方法或者多个相机图像的深度学习方法或者在停车场设置标志码来实现停车场融合建图与定位。In related technologies, fusion methods based on sensors such as lidar or deep learning methods of multiple camera images are mostly used, or marking codes are set in the parking lot to realize fusion mapping and positioning of the parking lot.
然而，通过上述方法在实现停车场融合建图与定位过程中，往往存在相机透视变换不准确、停车场标志物分割不稳定、建图定位精度低以及高成本的限制等问题，亟需解决。However, in the process of realizing fusion mapping and positioning of the parking lot through the above method, there are often problems such as inaccurate camera perspective transformation, unstable segmentation of parking lot markers, low accuracy of mapping and positioning, and high cost constraints, which need to be solved urgently.
发明内容Contents of the invention
本申请提供一种停车场的建图方法、装置、车辆及存储介质，以解决相关技术中环视相机逆透视变换不准确、基于鸟瞰图的停车场标志物分割不稳定以及单纯基于标志点云建图定位的精度低等问题，通过引入自适应逆透视变换算法，优化地面停车标志物的点云生成精度；引入深度学习图像分割算法，优化地面停车标志物的分割效果以及通过融合IMU(Inertial Measurement Unit，惯性测量单元)和GNSS(Global Navigation SatelliteSystem，全球导航卫星系统)提升建图和定位的精度。This application provides a parking lot mapping method, device, vehicle, and storage medium to solve the inaccurate inverse perspective transformation of the surround-view camera, unstable segmentation of parking lot markers based on bird's-eye views, and building points based solely on marker points in the related art. For problems such as low accuracy of map positioning, the point cloud generation accuracy of ground parking markers is optimized by introducing an adaptive inverse perspective transformation algorithm; the deep learning image segmentation algorithm is introduced to optimize the segmentation effect of ground parking markers and the integration of IMU (Inertial Measurement Unit, Inertial Measurement Unit) and GNSS (Global Navigation Satellite System, Global Navigation Satellite System) improve the accuracy of mapping and positioning.
本申请第一方面实施例提供一种停车场的建图方法，包括以下步骤：The embodiment of the first aspect of the present application provides a parking lot mapping method, including the following steps:
获取车辆周围的停车场图像，并畸变校正所述车辆周围的停车场图像，得到畸变校正后的车辆周围的停车场图像；Acquiring a parking lot image around the vehicle, and distortion correcting the parking lot image around the vehicle to obtain a distortion-corrected parking lot image around the vehicle;
对所述畸变校正后的车辆周围的停车场图像进行逆投影变换，生成地面鸟瞰图，并按照预设的分割要求分割所述地面鸟瞰图，得到地面标志图像分割图；以及performing inverse projection transformation on the distortion-corrected parking lot image around the vehicle to generate a ground bird's-eye view, and segmenting the ground bird's-eye view according to preset segmentation requirements to obtain a ground sign image segmentation map; and
基于预设的逆透视投影变换方法，对所述地面标志图像分割图进行变换，生成地面标志点云，并对所述地面标志点云进行帧图匹配，得到环视相机里程计，融合所述环视相机里程计和预设的IMU里程计得到所述车辆的位姿数据，并根据所述位姿数据对所述停车场建图和定位，得到所述停车场的最终全局地图。Based on a preset inverse perspective projection transformation method, transform the ground marker image segmentation map to generate a ground marker point cloud, and perform frame image matching on the ground marker point cloud to obtain a surround view camera odometer, and fuse the surround view The camera odometer and the preset IMU odometer obtain the pose data of the vehicle, and map and locate the parking lot according to the pose data to obtain the final global map of the parking lot.
根据本申请的一个实施例，在得到所述畸变校正后的车辆周围的停车场图像之后，还包括：According to an embodiment of the present application, after obtaining the image of the parking lot around the vehicle after the distortion correction, it further includes:
根据过惯导积分模型对IMU数据进行积分，得到所述预设的IMU里程计。The IMU data is integrated according to the inertial navigation integration model to obtain the preset IMU odometer.
根据本申请的一个实施例，所述畸变校正所述车辆周围的停车场图像，包括：According to an embodiment of the present application, the distortion correcting the parking lot image around the vehicle includes:
基于预设的四阶多项式参数模型，畸变校正所述车辆周围的停车场图像，其中，所述预设的四阶多项式参数模型为：Based on a preset fourth-order polynomial parameter model, the image of the parking lot around the vehicle is corrected for distortion, wherein the preset fourth-order polynomial parameter model is:
ρ(θ)＝k1*θ+k2*θ2+k3*θ3+k4*θ4；ρ(θ)=k1 *θ+k2 *θ2 +k3 *θ3 +k4 *θ4 ;
其中，θ是相对于光轴的入射角，ρ是图像中心和投影点之间的距离，k1、k2、k3和k4均为常数，在校准文件中给出。where θ is the angle of incidence relative to the optical axis, ρ is the distance between the image center and the projection point, and k1, k2, k3, and k4 are all constants given in the calibration file.
根据本申请的一个实施例，所述按照预设的分割要求分割所述地面鸟瞰图，得到地面标志图像分割图，包括：According to an embodiment of the present application, the segmentation of the ground bird's-eye view according to the preset segmentation requirements to obtain the segmentation image of the ground marker image includes:
对所述地面鸟瞰图进行车位标志线、行车引导线和停止线分割，得到所述地面标志图像分割图。The ground bird's-eye view image is segmented into the parking space marking line, the driving guide line and the stop line to obtain the ground marking image segmentation map.
根据本申请实施例的停车场的建图方法，通过获取车辆周围的停车场图像，并进行畸变校正，得到畸变校正后车辆周围的停车场图像进行逆投影变换，生成地面鸟瞰图，并按照预设的分割要求进行分割，得到地面标志图像分割图；基于预设的逆透视投影变换方法，对地面标志图像分割图进行变换，生成地面标志点云，并进行帧图匹配，得到环视相机里程计，并与预设的IMU里程计融合，得到车辆的位姿数据，进而对停车场建图和定位，得到停车场的最终全局地图。由此，解决了相机透视变换不准确、标志物分割不稳定、建图定位精度低以及高成本的限制等问题，通过引入新算法，在提高精度的同时降低车辆在停车场的建图与定位成本。According to the parking lot mapping method of the embodiment of the present application, by acquiring the parking lot image around the vehicle and performing distortion correction, the distortion-corrected parking lot image around the vehicle is back-projected to generate a bird's-eye view of the ground, and according to the preset Segmentation is performed according to the set segmentation requirements to obtain the ground marker image segmentation map; based on the preset inverse perspective projection transformation method, the ground marker image segmentation map is transformed to generate a ground marker point cloud, and the frame image matching is performed to obtain the surround view camera odometer , and fused with the preset IMU odometer to obtain the pose data of the vehicle, and then map and locate the parking lot to obtain the final global map of the parking lot. As a result, the problems of inaccurate camera perspective transformation, unstable landmark segmentation, low mapping positioning accuracy, and high cost constraints are solved. By introducing a new algorithm, the accuracy of the vehicle’s mapping and positioning in the parking lot is reduced while improving the accuracy. cost.
本申请第二方面实施例提供一种停车场的建图装置，包括：The embodiment of the second aspect of the present application provides a parking lot mapping device, including:
校正模块，用于获取车辆周围的停车场图像，并畸变校正所述车辆周围的停车场图像，得到畸变校正后的车辆周围的停车场图像；A correction module, configured to acquire a parking lot image around the vehicle, and distortion correct the parking lot image around the vehicle, to obtain a distortion-corrected parking lot image around the vehicle;
分割模块，用于对所述畸变校正后的车辆周围的停车场图像进行逆投影变换，生成地面鸟瞰图，并按照预设的分割要求分割所述地面鸟瞰图，得到地面标志图像分割图；以及A segmentation module, configured to perform inverse projection transformation on the distortion-corrected parking lot image around the vehicle to generate a ground bird's-eye view, and segment the ground bird's-eye view according to preset segmentation requirements to obtain a ground sign image segmentation map; and
建图模块，用于基于预设的逆透视投影变换方法，对所述地面标志图像分割图进行变换，生成地面标志点云，并对所述地面标志点云进行帧图匹配，得到环视相机里程计，融合所述环视相机里程计和预设的IMU里程计得到所述车辆的位姿数据，并根据所述位姿数据对所述停车场建图和定位，得到所述停车场的最终全局地图。The mapping module is used to transform the segmentation map of the ground marker image based on the preset inverse perspective projection transformation method to generate a ground marker point cloud, and perform frame image matching on the ground marker point cloud to obtain the mileage of the surround-view camera Calculate, fuse the look-around camera odometer and the preset IMU odometer to obtain the pose data of the vehicle, and build a map and locate the parking lot according to the pose data, and obtain the final overall situation of the parking lot map.
根据本申请的一个实施例，在得到所述畸变校正后的车辆周围的停车场图像之后，所述校正模块，还用于：According to an embodiment of the present application, after obtaining the distortion-corrected parking lot image around the vehicle, the correction module is further configured to:
根据过惯导积分模型对IMU数据进行积分，得到所述预设的IMU里程计。The IMU data is integrated according to the inertial navigation integration model to obtain the preset IMU odometer.
根据本申请的一个实施例，所述校正模块，具体用于：According to an embodiment of the present application, the correction module is specifically used for:
校正单元，用于基于预设的四阶多项式参数模型，畸变校正所述车辆周围的停车场图像，其中，所述预设的四阶多项式参数模型为：A correction unit, configured to correct the distortion of the parking lot image around the vehicle based on a preset fourth-order polynomial parameter model, wherein the preset fourth-order polynomial parameter model is:
ρ(θ)＝k1*θ+k2*θ2+k3*θ3+k4*θ4；ρ(θ)=k1 *θ+k2 *θ2 +k3 *θ3 +k4 *θ4 ;
其中，θ是相对于光轴的入射角，ρ是图像中心和投影点之间的距离，k1、k2、k3和k4均为常数，在校准文件中给出。where θ is the angle of incidence relative to the optical axis, ρ is the distance between the image center and the projection point, and k1, k2, k3, and k4 are all constants given in the calibration file.
根据本申请的一个实施例，所述分割模块，具体用于：According to an embodiment of the present application, the segmentation module is specifically used for:
对所述地面鸟瞰图进行车位标志线、行车引导线和停止线分割，得到所述地面标志图像分割图。The ground bird's-eye view image is segmented into the parking space marking line, the driving guide line and the stop line to obtain the ground marking image segmentation map.
根据本申请实施例的停车场的建图装置，通过获取车辆周围的停车场图像，并进行畸变校正，得到畸变校正后车辆周围的停车场图像进行逆投影变换，生成地面鸟瞰图，并按照预设的分割要求进行分割，得到地面标志图像分割图；基于预设的逆透视投影变换方法，对地面标志图像分割图进行变换，生成地面标志点云，并进行帧图匹配，得到环视相机里程计，并与预设的IMU里程计融合，得到车辆的位姿数据，进而对停车场建图和定位，得到停车场的最终全局地图。由此，解决了相机透视变换不准确、标志物分割不稳定、建图定位精度低以及高成本的限制等问题，通过引入新算法，在提高精度的同时降低车辆在停车场的建图与定位成本。According to the parking lot mapping device of the embodiment of the present application, by acquiring the parking lot image around the vehicle and performing distortion correction, the image of the parking lot around the vehicle after distortion correction is back-projected and transformed to generate a ground bird's-eye view, and according to the preset Segmentation is performed according to the set segmentation requirements to obtain the ground marker image segmentation map; based on the preset inverse perspective projection transformation method, the ground marker image segmentation map is transformed to generate a ground marker point cloud, and the frame image matching is performed to obtain the surround view camera odometer , and fused with the preset IMU odometer to obtain the pose data of the vehicle, and then map and locate the parking lot to obtain the final global map of the parking lot. As a result, the problems of inaccurate camera perspective transformation, unstable landmark segmentation, low mapping positioning accuracy, and high cost constraints are solved. By introducing a new algorithm, the accuracy of the vehicle’s mapping and positioning in the parking lot is reduced while improving the accuracy. cost.
本申请第三方面实施例提供一种车辆，包括：存储器、处理器及存储在所述存储器上并可在所述处理器上运行的计算机程序，所述处理器执行所述程序，以实现如上述实施例所述的停车场的建图方法。The embodiment of the third aspect of the present application provides a vehicle, including: a memory, a processor, and a computer program stored on the memory and operable on the processor, and the processor executes the program to implement the following: The method for constructing a map of a parking lot described in the foregoing embodiments.
本申请第四方面实施例提供一种计算机可读存储介质，所述计算机可读存储介质存储计算机指令，所述计算机指令用于使所述计算机执行如上述实施例所述的停车场的建图方法。The embodiment of the fourth aspect of the present application provides a computer-readable storage medium, the computer-readable storage medium stores computer instructions, and the computer instructions are used to make the computer execute the parking lot mapping described in the above-mentioned embodiments method.
本申请附加的方面和优点将在下面的描述中部分给出，部分将从下面的描述中变得明显，或通过本申请的实践了解到。Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.
附图说明Description of drawings
本申请上述的和/或附加的方面和优点从下面结合附图对实施例的描述中将变得明显和容易理解，其中：The above and/or additional aspects and advantages of the present application will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:
图1为根据本申请实施例提供的一种停车场的建图方法的流程图；FIG. 1 is a flowchart of a method for building a parking lot according to an embodiment of the present application;
图2为根据本申请一个实施例的整体流程示意图；FIG. 2 is a schematic diagram of an overall process according to an embodiment of the present application;
图3为根据本申请一个实施例的车辆环视相机的配置方式示意图；FIG. 3 is a schematic diagram of a configuration of a vehicle surround view camera according to an embodiment of the present application;
图4为根据本申请一个实施例的车辆环视相机获取的原始鱼眼图像示意图；4 is a schematic diagram of an original fisheye image obtained by a vehicle surround view camera according to an embodiment of the present application;
图5为根据本申请一个实施例的鱼眼图像经过畸变校正后的示意图；FIG. 5 is a schematic diagram of a fisheye image after distortion correction according to an embodiment of the present application;
图6为根据本申请一个实施例的逆透视投影变换后拼接生成的鸟瞰示意图；FIG. 6 is a schematic diagram of a bird's-eye view generated by splicing after inverse perspective projection transformation according to an embodiment of the present application;
图7为根据本申请一个实施例的深度学习网络对鸟瞰图的分割结果示意图；FIG. 7 is a schematic diagram of a segmentation result of a bird's-eye view by a deep learning network according to an embodiment of the present application;
图8为根据本申请一个实施例的基于分割后的鸟瞰图生成的地面标志点云示意图；FIG. 8 is a schematic diagram of a ground marker point cloud generated based on a segmented bird's-eye view according to an embodiment of the present application;
图9为根据本申请一个实施例的停车场的局部点云示意图；FIG. 9 is a schematic diagram of a local point cloud of a parking lot according to an embodiment of the present application;
图10为根据本申请一个实施例的停车场的整体点云示意图；FIG. 10 is a schematic diagram of an overall point cloud of a parking lot according to an embodiment of the present application;
图11为根据本申请实施例的停车场的建图装置的示例图；FIG. 11 is an example diagram of a mapping device for a parking lot according to an embodiment of the present application;
图12为根据本申请实施例提供的车辆的结构示意图。Fig. 12 is a schematic structural diagram of a vehicle provided according to an embodiment of the present application.
具体实施方式detailed description
下面详细描述本申请的实施例，实施例的示例在附图中示出，其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的，旨在用于解释本申请，而不能理解为对本申请的限制。Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary, and are intended to explain the present application, and should not be construed as limiting the present application.
下面参考附图描述本申请实施例的停车场的建图方法、装置、车辆及存储介质。针对上述背景技术中提到的在停车场的建图方法中，相关技术中环视相机逆透视变换不准确、基于鸟瞰图的停车场标志物分割不稳定以及单纯基于标志点云建图定位的精度低等问题，本申请提供了一种停车场的建图方法，在该方法中，通过获取车辆周围的停车场图像，并进行畸变校正，得到畸变校正后车辆周围的停车场图像进行逆投影变换，生成地面鸟瞰图，并按照预设的分割要求进行分割，得到地面标志图像分割图；基于预设的逆透视投影变换方法，对地面标志图像分割图进行变换，生成地面标志点云，并进行帧图匹配，得到环视相机里程计，并与预设的IMU里程计融合，得到车辆的位姿数据，进而对停车场建图和定位，得到停车场的最终全局地图。由此，解决了相机透视变换不准确、标志物分割不稳定、建图定位精度低以及高成本的限制等问题，通过引入新算法，在提高精度的同时降低车辆在停车场的建图与定位成本。The parking lot mapping method, device, vehicle, and storage medium of the embodiments of the present application will be described below with reference to the accompanying drawings. For the mapping method of the parking lot mentioned in the above background technology, the inverse perspective transformation of the surround view camera in the related technology is not accurate, the segmentation of the parking lot markers based on the bird's eye view is unstable, and the accuracy of the mapping and positioning based solely on the marker point cloud Low-level problem, this application provides a parking lot mapping method, in this method, by obtaining the parking lot image around the vehicle, and performing distortion correction, the image of the parking lot around the vehicle after distortion correction is back-projected , generate a bird's-eye view of the ground, and segment it according to the preset segmentation requirements to obtain a ground marker image segmentation map; based on the preset inverse perspective projection transformation method, transform the ground marker image segmentation map to generate a ground marker point cloud, and perform Frame map matching, to obtain the odometer of the surround-view camera, and fuse it with the preset IMU odometer to obtain the pose data of the vehicle, and then map and locate the parking lot to obtain the final global map of the parking lot. As a result, the problems of inaccurate camera perspective transformation, unstable landmark segmentation, low mapping positioning accuracy, and high cost constraints are solved. By introducing a new algorithm, the accuracy of the vehicle’s mapping and positioning in the parking lot is reduced while improving the accuracy. cost.
具体而言，图1为本申请实施例所提供的一种停车场的建图方法的流程示意图。Specifically, FIG. 1 is a schematic flowchart of a parking lot mapping method provided by an embodiment of the present application.
如图1所示，该停车场的建图方法包括以下步骤：As shown in Figure 1, the mapping method of the parking lot includes the following steps:
在步骤S101中，获取车辆周围的停车场图像，并畸变校正车辆周围的停车场图像，得到畸变校正后的车辆周围的停车场图像。In step S101 , a parking lot image around the vehicle is acquired, and the distortion-corrected image of the parking lot around the vehicle is obtained to obtain a distortion-corrected parking lot image around the vehicle.
进一步地，在一些实施例中，畸变校正车辆周围的停车场图像，包括：基于预设的四阶多项式参数模型，畸变校正车辆周围的停车场图像，其中，预设的四阶多项式参数模型为：Further, in some embodiments, the distortion correction of the parking lot image around the vehicle includes: based on a preset fourth-order polynomial parameter model, distortion correction of the parking lot image around the vehicle, wherein the preset fourth-order polynomial parameter model is :
ρ(θ)＝k1*θ+k2*θ2+k3*θ3+k4*θ4；ρ(θ)=k1 *θ+k2 *θ2 +k3 *θ3 +k4 *θ4 ;
其中，θ是相对于光轴的入射角，ρ是图像中心和投影点之间的距离，k1、k2、k3和k4均为常数，在校准文件中给出。where θ is the angle of incidence relative to the optical axis, ρ is the distance between the image center and the projection point, and k1, k2, k3, and k4 are all constants given in the calibration file.
具体地，如图2所示，本申请实施例通过车辆自身携带的环视相机获取车辆周围的停车场图像，需要尽量包含车位标志线、行车引导线、车道线和停止线等地面固定标志物。其中，由于车辆自身携带的环视相机一般为鱼眼相机，因此，获取到的图像大部分为车身周围包含地面行车交通特征的鱼眼图像，采集到的图像如图3所示。Specifically, as shown in FIG. 2 , in the embodiment of the present application, the image of the parking lot around the vehicle is acquired through the surround-view camera carried by the vehicle itself, and it is necessary to include fixed ground markers such as parking space marking lines, driving guidance lines, lane lines, and stop lines as much as possible. Among them, since the surround-view camera carried by the vehicle itself is generally a fisheye camera, most of the acquired images are fisheye images around the vehicle body including ground traffic characteristics. The collected images are shown in Figure 3.
进一步地，本申请实施例通过鱼眼相机获取到车辆周围的停车场信息后，需要对获取的鱼眼图像进行畸变校正。其中，环视相机的内外参数需要提前进行标定，该部分包含对鱼眼相机用于畸变校正的多项式模型参数进行标定，以获取环视相机的初始外参。Further, in the embodiment of the present application, after acquiring the parking lot information around the vehicle through the fisheye camera, it is necessary to perform distortion correction on the acquired fisheye image. Among them, the internal and external parameters of the surround-view camera need to be calibrated in advance. This part includes the calibration of the polynomial model parameters used for distortion correction of the fisheye camera to obtain the initial external parameters of the surround-view camera.
具体地，本申请实施例的标定参数可以基于径向四阶多项式参数模型对鱼眼图像进行畸变校正，校正前后的图像对比如图4和图5所示。Specifically, the calibration parameters in the embodiment of the present application can correct the distortion of the fisheye image based on the radial fourth-order polynomial parameter model, and the image comparison before and after correction is shown in FIG. 4 and FIG. 5 .
进一步地，本申请实施例所采用的预设的四阶多项式参数模型为：Further, the preset fourth-order polynomial parameter model adopted in the embodiment of the present application is:
ρ(θ)＝k1*θ+k2*θ2+k3*θ3+k4*θ4；ρ(θ)=k1 *θ+k2 *θ2 +k3 *θ3 +k4 *θ4 ;
其中，θ是相对于光轴的入射角，ρ是图像中心和投影点之间的距离，系数k1、k2、k3和k4在校准文件中给出。需要说明的是，主点的图像宽度和高度以及偏移量(cx，cy)以像素为单位。where θ is the angle of incidence relative to the optical axis, ρ is the distance between the image center and the projection point, and the coefficients k1, k2, k3, and k4 are given in the calibration file. It should be noted that the image width, height and offset (cx, cy) of the main point are in pixels.
举例而言，以相机坐标给出的3D点和图像坐标为例，相机坐标给出的3D点(X，Y，Z)到图像坐标(u，v)的投影如下所示：For example, taking the 3D point given by the camera coordinates and the image coordinates as an example, the projection of the 3D point (X, Y, Z) given by the camera coordinates to the image coordinates (u, v) is as follows:
θ＝arctan2(χ，Z)＝pi/2-arctan2(Z，χ)θ=arctan2(χ,Z)=pi/2-arctan2(Z,χ)
ρ＝ρ(θ)ρ=ρ(θ)
u′＝ρ*X/χ if χ≠0 else 0u′=ρ*X/χ if χ≠0 else 0
v′＝ρ*Y/χ if χ≠0 else 0v′=ρ*Y/χ if χ≠0 else 0
u＝u′+cx+w/2-0.5u=u'+cx+w/2-0.5
v＝v′*aspect.ratio+cy+h/2-0.5v=v'*aspect.ratio+cy+h/2-0.5
其中，cx逆为图像主点u向偏移量，cy为图像主点v向偏移量，w为图像宽度，h为图像高度，aspect.ratio为图像纵横比，χ，、u’、v’均为计算中间量，最后两行显示了图像坐标系的最终转换，假设图像坐标系的原点位于左上角，则左上角像素为(0，0)。Among them, cx is the offset of the main point of the image in the u direction, cy is the offset of the main point of the image in the v direction, w is the width of the image, h is the height of the image, aspect.ratio is the aspect ratio of the image, χ,, u', v ' Both calculate the intermediate amount, and the last two lines show the final transformation of the image coordinate system. Assuming that the origin of the image coordinate system is located in the upper left corner, the upper left pixel is (0, 0).
进一步地，在一些实施例中，在得到畸变校正后的车辆周围的停车场图像之后，还包括：根据惯导积分模型对IMU数据进行积分，得到预设的IMU里程计。Further, in some embodiments, after obtaining the distortion-corrected image of the parking lot around the vehicle, the method further includes: integrating the IMU data according to the inertial navigation integration model to obtain a preset IMU odometer.
具体地，本申请实施例在得到畸变后的车辆周围的停车场图像之后，需要根据惯导积分模型对IMU数据进行积分，得到车辆的预测位姿，即IMU里程计，其中，预测位姿包括车辆位置、速度、姿态等，主要积分过程如下式所示：Specifically, in the embodiment of the present application, after obtaining the distorted image of the parking lot around the vehicle, it is necessary to integrate the IMU data according to the inertial navigation integral model to obtain the predicted pose of the vehicle, that is, the IMU odometer, wherein the predicted pose includes Vehicle position, speed, attitude, etc., the main integration process is shown in the following formula:
v←v+(R(am-ab)+g)Δtv←v+(R(am -ab )+g)Δt
ab←abab ← ab
wb←ωbwb ←ωb
g←g，g←g,
其中，p为imu位置向量，v为imu速度向量，q为imu姿态四元数，am为imu加速度，ωm为imu角速度，ab为imu加速度偏置，ωb为imu角速度偏置，R为imu姿态矩阵，Δt为两帧imu之间的间隔时间，g为重力加速度向量。Among them, p is the imu position vector, v is the imu velocity vector, q is the imu attitude quaternion, am is the imu acceleration, ωm is the imu angular velocity, ab is the imu acceleration bias, ωb is the imu angular velocity bias, R is the imu attitude matrix, Δt is the interval time between two frames of imu, and g is the gravity acceleration vector.
在步骤S102中，对畸变校正后的车辆周围的停车场图像进行逆投影变换，生成地面鸟瞰图，并按照预设的分割要求分割地面鸟瞰图，得到地面标志图像分割图。In step S102, the distortion-corrected parking lot image around the vehicle is back-projected to generate a ground bird's-eye view, and the ground bird's-eye view is segmented according to preset segmentation requirements to obtain a ground sign image segmentation map.
进一步地，在一些实施例中，按照预设的分割要求分割地面鸟瞰图，得到地面标志图像分割图，包括：对地面鸟瞰图进行车位标志线、行车引导线和停止线分割，得到地面标志图像分割图。Further, in some embodiments, the ground bird's-eye view is segmented according to the preset segmentation requirements to obtain the segmentation map of the ground sign image, including: segmenting the ground bird's-eye view with parking space marking lines, driving guide lines and stop lines to obtain the ground sign image Split graph.
具体地，如图6所示，本申请实施例首先通过自适应IPM算法对畸变校正后的车辆周围环境进行逆投影变换，从而生成单个的地面鸟瞰图，并通过各个相机之间的外参拼接成整体的鸟瞰图。其中，自适应IPM(Inverse Perspective Mapping，逆透视投影)利用IMU里程计中对车辆位姿的预测得到相机位姿，以得到相机在当前环境中的动态俯仰角和偏航角，并在相邻帧中加入了俯仰角的修正，其具体过程如下：Specifically, as shown in Fig. 6, the embodiment of the present application first uses the adaptive IPM algorithm to perform inverse projection transformation on the surrounding environment of the vehicle after distortion correction, thereby generating a single ground bird's-eye view, and splicing the external parameters between each camera Aerial view of the whole. Among them, adaptive IPM (Inverse Perspective Mapping, inverse perspective projection) uses the prediction of the vehicle pose in the IMU odometer to obtain the camera pose, so as to obtain the dynamic pitch angle and yaw angle of the camera in the current environment, and in the adjacent The correction of the pitch angle is added to the frame, and the specific process is as follows:
其中，θo为相机初始俯仰角即光轴与水平面的夹角，θp为相机动态增量俯仰角，αr为垂直视场角的一半，αc水平视场角的一半，m为图像高度，n为图像宽度。Among them, θo is the initial pitch angle of the camera, that is, the angle between the optical axis and the horizontal plane, θp is the dynamic incremental pitch angle of the camera, αr is half of the vertical field of view, αc is half of the horizontal field of view, and m is the image height, n is the image width.
进一步地，本申请实施例在得到整体的地面鸟瞰图后，基于UNET(Network，网络)网络进行车位标志线、行车引导线和停止线等地面标志的分割，在分割过程中，需要事先使用包含停车场各种类型和情况地面标志的鸟瞰图对网络进行训练，以获得权重参数初始化网络用于在线分割，其分割结果如图7所示。Further, in the embodiment of the present application, after obtaining the overall ground bird's-eye view, the segmentation of ground signs such as parking space marking lines, driving guidance lines and stop lines is performed based on the UNET (Network, network). The bird's-eye view of ground signs of various types and situations in the parking lot is used to train the network to obtain weight parameters to initialize the network for online segmentation, and the segmentation results are shown in Figure 7.
具体而言，UNET网络是一种典型的分割网络，针对单调的地面环境具有快速、准确、鲁棒的优点。针对停车场鸟瞰图所具有的车道线、停车线、引导符、减速带、障碍物和自由空间的特征，使用停车场数据集进行预先训练，获得用于识别的权重，之后基于训练得到权重设置网络对整体鸟瞰图进行分割。Specifically, the UNET network is a typical segmentation network, which has the advantages of being fast, accurate and robust against monotonous ground environments. Aiming at the characteristics of lane lines, parking lines, guide signs, speed bumps, obstacles and free spaces in the bird's-eye view of the parking lot, the parking lot dataset is used for pre-training to obtain weights for recognition, and then weight settings are obtained based on training The network segments the overall bird's-eye view.
在步骤S103中，基于预设的逆透视投影变换方法，对地面标志图像分割图进行变换，生成地面标志点云，并对地面标志点云进行帧图匹配，得到环视相机里程计，融合环视相机里程计和预设的IMU里程计得到车辆的位姿数据，并根据位姿数据对停车场建图和定位，得到停车场的最终全局地图。In step S103, based on the preset inverse perspective projection transformation method, transform the ground marker image segmentation map to generate a ground marker point cloud, and perform frame image matching on the ground marker point cloud to obtain the odometer of the surround-view camera, and fuse the surround-view camera The odometer and the preset IMU odometer obtain the pose data of the vehicle, and map and locate the parking lot according to the pose data to obtain the final global map of the parking lot.
具体地，如图8所示，本申请实施例通过提取分割鸟瞰图中对应车道线、停车线、引导符、减速带和障碍物的部分，基于鸟瞰图虚拟相机到车身坐标系的位姿关系，并通过逆透视投影的逆向变换生成基于车辆的自身坐标系的地面标志点云。Specifically, as shown in FIG. 8, the embodiment of the present application extracts parts corresponding to lane lines, stop lines, guide signs, speed bumps and obstacles in the segmented bird's-eye view, based on the pose relationship between the bird's-eye view virtual camera and the vehicle body coordinate system , and generate a ground marker point cloud based on the vehicle's own coordinate system through the inverse transformation of the inverse perspective projection.
进一步地，地面标志点云首先通过NDT(Normal Distributions Transform，正态分布变换)帧图匹配算法获得点云的环视相机里程计，其中，帧图匹配采用NDT算法；其次，在进行帧图匹配时，局部地图采用基于距离的滑动窗口原理进行提取，以确保精度和效率达到平衡。其中，匹配时的预测位姿采用IMU的积分结果，并基于时间进行对齐和插值。Further, the ground marker point cloud first obtains the odometer of the point cloud's surround-view camera through the NDT (Normal Distributions Transform) frame-image matching algorithm, wherein the frame-image matching uses the NDT algorithm; secondly, when performing frame-image matching , the partial map is extracted using a distance-based sliding window principle to ensure a balance between accuracy and efficiency. Among them, the predicted pose during matching uses the integral result of the IMU, and is aligned and interpolated based on time.
需要说明的是，由于单纯基于地面特征点云帧图匹配得到的点云里程计误差较大，因此，本申请实施例使用ESKF(Error State Kalman Filter，误差状态卡尔曼滤波)算法对IMU里程计和环视相机地面标志点云里程计进行融合，从而获得车身的高精度位姿数据，进行建图和定位。其中，如图9所示，本申请实施例的建图可以采用SLAM(SimultaneousLocalization And Mapping，同时定位遇见图)方法通过局部地图根据融合后的全局位姿进行构建。It should be noted that, because the point cloud odometer error obtained purely based on ground feature point cloud frame image matching is relatively large, the embodiment of the present application uses the ESKF (Error State Kalman Filter, error state Kalman filter) algorithm to analyze the IMU odometer It is fused with the ground marker point cloud odometer of the surround-view camera to obtain high-precision pose data of the car body for mapping and positioning. Wherein, as shown in FIG. 9 , the mapping of the embodiment of the present application can be constructed by using the SLAM (Simultaneous Localization And Mapping, simultaneous localization and meeting map) method through the local map according to the fused global pose.
进一步地，ESKF整体分为预测和观测两个部分，其中，预测中的误差状态系统方程为：Furthermore, the ESKF is divided into two parts: prediction and observation, where the error state system equation in prediction is:
δx←f(x，δx，um，i)＝Fx(x，um)·δx+Fi·i；δx←f(x, δx, um , i) = Fx (x, um )·δx+Fi ·i;
其中，δx为误差状态，x为名义状态，um为imu测量结果，i为扰动，Fx为f()关于误差状态的雅克比矩阵，Fi为f()关于扰动的雅克比矩阵。where δx is the error state, x is the nominal state, um is the imu measurement result, i is the disturbance, Fx is the Jacobian matrix of f() with respect to the error state, and Fi is the Jacobian matrix of f() with respect to the disturbance.
预测中的误差状态方程为：The error state equation in prediction is:
其中，
预测中的误差状态协方差矩阵为：The error state covariance matrix in the forecast is:
其中，P为误差状态协方差矩阵，Qi为扰动的协方差矩阵。Among them, P is the covariance matrix of the error state, and Qi is the covariance matrix of the disturbance.
观测中的观测方程为：The observation equation in the observation is:
y＝h(xt)+v；y=h(xt )+v;
其中，y为观测状态，xt为真值状态，()为一般的非线性函数，v为高斯噪声。Among them, y is the observation state, xt is the truth state, () is a general nonlinear function, and v is Gaussian noise.
观测中的卡尔曼增益更新Kalman gain update in observations
其中，H为()关于误差状态的雅克比矩阵，V为观测高斯噪声协方差矩阵。Among them, H is the Jacobian matrix of ( ) about the error state, and V is the observed Gaussian noise covariance matrix.
观测中的误差状态更新：Error status updates in observations:
其中，K为卡尔曼增益，
观测中的误差状态协方差矩阵更新：Error state covariance matrix update in observations:
P←(I-KH)P；P←(I-KH)P;
其中，I为单位矩阵。Among them, I is the identity matrix.
具体地，由于点云里程计存在累积误差，在地图较大、转弯较多后必然存在位姿的飘逸。因此，本申请实施例需要基于距离和时间的限制，检测潜在的回环，通过点云ICP(Iterative Closest Point，迭代最近点)算法匹配当前帧的地面标志点云和回环局部地图，进一步确认是否发生回环，如果回环成功，则当前帧的ICP匹配结果即为回环约束。Specifically, due to the cumulative error of the point cloud odometry, there must be pose drift after a large map and many turns. Therefore, the embodiment of the present application needs to detect potential loop closures based on distance and time constraints, and use the point cloud ICP (Iterative Closest Point, Iterative Closest Point) algorithm to match the ground marker point cloud of the current frame and the local loop closure map to further confirm whether it occurs Loopback, if the loopback is successful, the ICP matching result of the current frame is the loopback constraint.
进一步地，如图10所示，全局位姿的优化基于GTSAM图优化库进行，当回环约束发生时，将当前位姿约束加入优化图中，进行更新操作后即可获得新的全局位姿，进而更新全局地图，从而得到最终的全局地图。Further, as shown in Figure 10, the optimization of the global pose is based on the GTSAM graph optimization library. When the loopback constraint occurs, the current pose constraint is added to the optimization graph, and the new global pose can be obtained after the update operation. Then the global map is updated to obtain the final global map.
综上，本申请实施例具有如下技术效果：To sum up, the embodiment of the present application has the following technical effects:
(1)利用低成本的车身环视相机和惯性导航设备，在视觉特征多变GNSS信号较弱且无需提前设置视觉标记或是进行其他改造的停车场环境中，提供相对稳定且具有较高精度定位和建图。(1) Use low-cost body surround view cameras and inertial navigation equipment to provide relatively stable and high-precision positioning in a parking lot environment where visual features are changing and GNSS signals are weak and there is no need to set visual markers in advance or perform other modifications and map.
(2)使用低成本的环视相机和惯性导航设备以及GNSS定位设备，无需额外的传感器且不需要对停车场进行预先设置视觉标记等其他改造，成本低且实用性强。(2) Using low-cost surround-view cameras, inertial navigation equipment, and GNSS positioning equipment, no additional sensors are required and no other modifications such as pre-set visual markers are required for the parking lot. The cost is low and the practicability is strong.
(3)在各步骤引入精度和效率适合的算法，提高各步骤处理结果的精度，最终保证建图与定位过程中的精度与实时性，同时地图在使用过程中可以进行持续迭代优化。(3) Introduce algorithms with appropriate accuracy and efficiency in each step to improve the accuracy of the processing results of each step, and finally ensure the accuracy and real-time performance in the process of mapping and positioning. At the same time, the map can be continuously iteratively optimized during use.
根据本申请实施例的停车场的建图方法，通过获取车辆周围的停车场图像，并进行畸变校正，得到畸变校正后车辆周围的停车场图像进行逆投影变换，生成地面鸟瞰图，并按照预设的分割要求进行分割，得到地面标志图像分割图；基于预设的逆透视投影变换方法，对地面标志图像分割图进行变换，生成地面标志点云，并进行帧图匹配，得到环视相机里程计，并与预设的IMU里程计融合，得到车辆的位姿数据，进而对停车场建图和定位，得到停车场的最终全局地图。由此，解决了相机透视变换不准确、标志物分割不稳定、建图定位精度低以及高成本的限制等问题，通过引入新算法，在提高精度的同时降低车辆在停车场的建图与定位成本。According to the parking lot mapping method of the embodiment of the present application, by acquiring the parking lot image around the vehicle and performing distortion correction, the distortion-corrected parking lot image around the vehicle is back-projected to generate a bird's-eye view of the ground, and according to the preset Segmentation is performed according to the set segmentation requirements to obtain the ground marker image segmentation map; based on the preset inverse perspective projection transformation method, the ground marker image segmentation map is transformed to generate a ground marker point cloud, and the frame image matching is performed to obtain the surround view camera odometer , and fused with the preset IMU odometer to obtain the pose data of the vehicle, and then map and locate the parking lot to obtain the final global map of the parking lot. As a result, the problems of inaccurate camera perspective transformation, unstable landmark segmentation, low mapping positioning accuracy, and high cost constraints are solved. By introducing a new algorithm, the accuracy of the vehicle’s mapping and positioning in the parking lot is reduced while improving the accuracy. cost.
其次参照附图描述根据本申请实施例提出的停车场的建图装置。Next, a mapping device for a parking lot proposed according to an embodiment of the present application will be described with reference to the accompanying drawings.
图11是本申请实施例的停车场的建图装置的方框示意图。FIG. 11 is a schematic block diagram of a mapping device for a parking lot according to an embodiment of the present application.
如图11所示，该停车场的建图装置10包括：校正模块100、分割模块200和建图模块300。As shown in FIG. 11 , the
其中，校正模块100，用于获取车辆周围的停车场图像，并畸变校正车辆周围的停车场图像，得到畸变校正后的车辆周围的停车场图像；Wherein, the
分割模块200，用于对畸变校正后的车辆周围的停车场图像进行逆投影变换，生成地面鸟瞰图，并按照预设的分割要求分割地面鸟瞰图，得到地面标志图像分割图；以及The
建图模块300，用于基于预设的逆透视投影变换方法，对地面标志图像分割图进行变换，生成地面标志点云，并对地面标志点云进行帧图匹配，得到环视相机里程计，融合环视相机里程计和预设的IMU里程计得到车辆的位姿数据，并根据位姿数据对停车场建图和定位，得到停车场的最终全局地图。The
进一步地，在一些实施例中，在得到畸变校正后的车辆周围的停车场图像之后，校正模块100，还用于：Further, in some embodiments, after obtaining the distortion-corrected image of the parking lot around the vehicle, the
根据过惯导积分模型对IMU数据进行积分，得到预设的IMU里程计。Integrate the IMU data according to the inertial navigation integration model to obtain the preset IMU odometer.
进一步地，在一些实施例中，校正模块100，具体用于：Further, in some embodiments, the
校正单元，用于基于预设的四阶多项式参数模型，畸变校正车辆周围的停车场图像，其中，预设的四阶多项式参数模型为：The correction unit is used for distorting and correcting the parking lot image around the vehicle based on a preset fourth-order polynomial parameter model, wherein the preset fourth-order polynomial parameter model is:
ρ(θ)＝k1*θ+k2*θ2+k3*θ3+k4*θ4；ρ(θ)=k1 *θ+k2 *θ2 +k3 *θ3 +k4 *θ4 ;
其中，θ是相对于光轴的入射角，ρ是图像中心和投影点之间的距离，k1、k2、k3和k4均为常数，在校准文件中给出。where θ is the angle of incidence relative to the optical axis, ρ is the distance between the image center and the projection point, and k1, k2, k3, and k4 are all constants given in the calibration file.
进一步地，在一些实施例中，分割模块200，具体用于：Further, in some embodiments, the
对地面鸟瞰图进行车位标志线、行车引导线和停止线分割，得到地面标志图像分割图。Segment the parking space marking line, driving guide line and stop line on the ground bird's-eye view to obtain the ground marking image segmentation map.
根据本申请实施例的停车场的建图装置，通过获取车辆周围的停车场图像，并进行畸变校正，得到畸变校正后车辆周围的停车场图像进行逆投影变换，生成地面鸟瞰图，并按照预设的分割要求进行分割，得到地面标志图像分割图；基于预设的逆透视投影变换方法，对地面标志图像分割图进行变换，生成地面标志点云，并进行帧图匹配，得到环视相机里程计，并与预设的IMU里程计融合，得到车辆的位姿数据，进而对停车场建图和定位，得到停车场的最终全局地图。由此，解决了相机透视变换不准确、标志物分割不稳定、建图定位精度低以及高成本的限制等问题，通过引入新算法，在提高精度的同时降低车辆在停车场的建图与定位成本。According to the parking lot mapping device of the embodiment of the present application, by acquiring the parking lot image around the vehicle and performing distortion correction, the image of the parking lot around the vehicle after distortion correction is back-projected and transformed to generate a ground bird's-eye view, and according to the preset Segmentation is performed according to the set segmentation requirements to obtain the ground marker image segmentation map; based on the preset inverse perspective projection transformation method, the ground marker image segmentation map is transformed to generate a ground marker point cloud, and the frame image matching is performed to obtain the surround view camera odometer , and fused with the preset IMU odometer to obtain the pose data of the vehicle, and then map and locate the parking lot to obtain the final global map of the parking lot. As a result, the problems of inaccurate camera perspective transformation, unstable landmark segmentation, low mapping positioning accuracy, and high cost constraints are solved. By introducing a new algorithm, the accuracy of the vehicle’s mapping and positioning in the parking lot is reduced while improving the accuracy. cost.
图12为本申请实施例提供的车辆的结构示意图。该车辆可以包括：Fig. 12 is a schematic structural diagram of a vehicle provided by an embodiment of the present application. The vehicle can include:
存储器1201、处理器1202及存储在存储器1201上并可在处理器1202上运行的计算机程序。A
处理器1202执行程序时实现上述实施例中提供的停车场的建图方法。When the
进一步地，车辆还包括：Further, the vehicle also includes:
通信接口1203，用于存储器1201和处理器1202之间的通信。The
存储器1201，用于存放可在处理器1202上运行的计算机程序。The
存储器1201可能包含高速RAM存储器，也可能还包括非易失性存储器(non-volatile memory)，例如至少一个磁盘存储器。The
如果存储器1201、处理器1202和通信接口1203独立实现，则通信接口1203、存储器1201和处理器1202可以通过总线相互连接并完成相互间的通信。总线可以是工业标准体系结构(Industry Standard Architecture，简称为ISA)总线、外部设备互连(PeripheralComponent，简称为PCI)总线或扩展工业标准体系结构(Extended Industry StandardArchitecture，简称为EISA)总线等。总线可以分为地址总线、数据总线、控制总线等。为便于表示，图12中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。If the
可选地，在具体实现上，如果存储器1201、处理器1202及通信接口1203，集成在一块芯片上实现，则存储器1201、处理器1202及通信接口1203可以通过内部接口完成相互间的通信。Optionally, in specific implementation, if the
处理器1202可能是一个中央处理器(Central Processing Unit，简称为CPU)，或者是特定集成电路(Application Specific Integrated Circuit，简称为ASIC)，或者是被配置成实施本申请实施例的一个或多个集成电路。The
本实施例还提供一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如上的停车场的建图方法。This embodiment also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above parking lot mapping method is realized.
在本说明书的描述中，参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本申请的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不必须针对的是相同的实施例或示例。而且，描述的具体特征、结构、材料或者特点可以在任一个或N个实施例或示例中以合适的方式结合。此外，在不相互矛盾的情况下，本领域的技术人员可以将本说明书中描述的不同实施例或示例以及不同实施例或示例的特征进行结合和组合。In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the described specific features, structures, materials or characteristics may be combined in any one or N embodiments or examples in an appropriate manner. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.
此外，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括至少一个该特征。在本申请的描述中，“N个”的含义是至少两个，例如两个，三个等，除非另有明确具体的限定。In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present application, "N" means at least two, such as two, three, etc., unless otherwise specifically defined.
流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为，表示包括一个或更N个用于实现定制逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分，并且本申请的优选实施方式的范围包括另外的实现，其中可以不按所示出或讨论的顺序，包括根据所涉及的功能按基本同时的方式或按相反的顺序，来执行功能，这应被本申请的实施例所属技术领域的技术人员所理解。Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing a custom logical function or step of a process , and the scope of preferred embodiments of the present application includes additional implementations in which functions may be performed out of the order shown or discussed, including in substantially simultaneous fashion or in reverse order depending on the functions involved, which shall It should be understood by those skilled in the art to which the embodiments of the present application belong.
在流程图中表示或在此以其他方式描述的逻辑和/或步骤，例如，可以被认为是用于实现逻辑功能的可执行指令的定序列表，可以具体实现在任何计算机可读介质中，以供指令执行系统、装置或设备(如基于计算机的系统、包括处理器的系统或其他可以从指令执行系统、装置或设备取指令并执行指令的系统)使用，或结合这些指令执行系统、装置或设备而使用。就本说明书而言，"计算机可读介质"可以是任何可以包含、存储、通信、传播或传输程序以供指令执行系统、装置或设备或结合这些指令执行系统、装置或设备而使用的装置。计算机可读介质的更具体的示例(非穷尽性列表)包括以下：具有一个或N个布线的电连接部(电子装置)，便携式计算机盘盒(磁装置)，随机存取存储器(RAM)，只读存储器(ROM)，可擦除可编辑只读存储器(EPROM或闪速存储器)，光纤装置，以及便携式光盘只读存储器(CDROM)。另外，计算机可读介质甚至可以是可在其上打印程序的纸或其他合适的介质，因为可以例如通过对纸或其他介质进行光学扫描，接着进行编辑、解译或必要时以其他合适方式进行处理来以电子方式获得程序，然后将其存储在计算机存储器中。The logic and/or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium, For use with instruction execution systems, devices, or devices (such as computer-based systems, systems including processors, or other systems that can fetch instructions from instruction execution systems, devices, or devices and execute instructions), or in conjunction with these instruction execution systems, devices or equipment for use. For the purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use in or in conjunction with an instruction execution system, device or device. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connection with one or N wires (electronic device), portable computer disk case (magnetic device), random access memory (RAM), Read Only Memory (ROM), Erasable and Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program may be printed, as it may be possible, for example, by optically scanning the paper or other medium, followed by editing, interpretation or other suitable means if necessary. Processing to obtain programs electronically and store them in computer memory.
应当理解，本申请的各部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中，N个步骤或方法可以用存储在存储器中且由合适的指令执行系统执行的软件或固件来实现。如，如果用硬件来实现和在另一实施方式中一样，可用本领域公知的下列技术中的任一项或他们的组合来实现：具有用于对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路，具有合适的组合逻辑门电路的专用集成电路，可编程门阵列(PGA)，现场可编程门阵列(FPGA)等。It should be understood that each part of the present application may be realized by hardware, software, firmware or a combination thereof. In the above embodiments, the N steps or methods may be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented by any one or a combination of the following techniques known in the art: a discrete Logic circuits, ASICs with suitable combinational logic gates, Programmable Gate Arrays (PGA), Field Programmable Gate Arrays (FPGA), etc.
本技术领域的普通技术人员可以理解实现上述实施例方法携带的全部或部分步骤是可以通过程序来指令相关的硬件完成，的程序可以存储于一种计算机可读存储介质中，该程序在执行时，包括方法实施例的步骤之一或其组合。Those of ordinary skill in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed , including one or a combination of the steps of the method embodiment.
此外，在本申请各个实施例中的各功能单元可以集成在一个处理模块中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。集成的模块如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, each unit may exist separately physically, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. If the integrated modules are realized in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium.
上述提到的存储介质可以是只读存储器，磁盘或光盘等。尽管上面已经示出和描述了本申请的实施例，可以理解的是，上述实施例是示例性的，不能理解为对本申请的限制，本领域的普通技术人员在本申请的范围内可以对上述实施例进行变化、修改、替换和变型。The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, and the like. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limitations on the present application, and those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211107120.2ACN115456898A (en) | 2022-09-09 | 2022-09-09 | Method and device for building image of parking lot, vehicle and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211107120.2ACN115456898A (en) | 2022-09-09 | 2022-09-09 | Method and device for building image of parking lot, vehicle and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115456898Atrue CN115456898A (en) | 2022-12-09 |
Family
ID=84304005
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211107120.2APendingCN115456898A (en) | 2022-09-09 | 2022-09-09 | Method and device for building image of parking lot, vehicle and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115456898A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116499470A (en)* | 2023-06-28 | 2023-07-28 | 苏州中德睿博智能科技有限公司 | Optimal control method, device and system for positioning system of looking-around camera |
| CN116659525A (en)* | 2023-05-30 | 2023-08-29 | 北京易航远智科技有限公司 | Real-time mapping method, device, electronic equipment and storage medium combined with navigation map |
| CN117237237A (en)* | 2023-11-13 | 2023-12-15 | 深圳元戎启行科技有限公司 | Luminosity balancing method and device for vehicle-mounted 360-degree panoramic image |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108263283A (en)* | 2018-01-25 | 2018-07-10 | 长沙立中汽车设计开发股份有限公司 | More marshalling varied angle vehicle panoramic viewing system calibration and joining method |
| CN110243358A (en)* | 2019-04-29 | 2019-09-17 | 武汉理工大学 | Indoor and outdoor positioning method and system for unmanned vehicles based on multi-source fusion |
| US20190368879A1 (en)* | 2018-05-29 | 2019-12-05 | Regents Of The University Of Minnesota | Vision-aided inertial navigation system for ground vehicle localization |
| CN113903011A (en)* | 2021-10-26 | 2022-01-07 | 江苏大学 | Semantic map construction and positioning method suitable for indoor parking lot |
| CN114863096A (en)* | 2022-04-02 | 2022-08-05 | 合众新能源汽车有限公司 | Semantic map construction and positioning method and device for indoor parking lot |
| CN114964236A (en)* | 2022-05-25 | 2022-08-30 | 重庆长安汽车股份有限公司 | Mapping and vehicle positioning system and method for underground parking lot environment |
- 2022
- 2022-09-09CNCN202211107120.2Apatent/CN115456898A/enactivePending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108263283A (en)* | 2018-01-25 | 2018-07-10 | 长沙立中汽车设计开发股份有限公司 | More marshalling varied angle vehicle panoramic viewing system calibration and joining method |
| US20190368879A1 (en)* | 2018-05-29 | 2019-12-05 | Regents Of The University Of Minnesota | Vision-aided inertial navigation system for ground vehicle localization |
| CN110243358A (en)* | 2019-04-29 | 2019-09-17 | 武汉理工大学 | Indoor and outdoor positioning method and system for unmanned vehicles based on multi-source fusion |
| CN113903011A (en)* | 2021-10-26 | 2022-01-07 | 江苏大学 | Semantic map construction and positioning method suitable for indoor parking lot |
| CN114863096A (en)* | 2022-04-02 | 2022-08-05 | 合众新能源汽车有限公司 | Semantic map construction and positioning method and device for indoor parking lot |
| CN114964236A (en)* | 2022-05-25 | 2022-08-30 | 重庆长安汽车股份有限公司 | Mapping and vehicle positioning system and method for underground parking lot environment |
Non-Patent Citations (3)
| Title |
|---|
| 徐玲;蔡慧;郑恩辉;花江峰;刘政;王谈谈;: "基于关键帧的视觉惯性定位算法", 科技通报, no. 05, 31 May 2020 (2020-05-31)* |
| 王晨捷;张云;赵青;王伟;尹露;罗斌;张良培;: "分裂合并运动分割的多运动视觉里程计方法", 中国图象图形学报, no. 09, 16 September 2020 (2020-09-16)* |
| 逯建军;任晓军;孙伟;郭元江;李群;: "惯性/双目视觉里程计深组合导航方法", 导航定位与授时, no. 03, 15 May 2016 (2016-05-15)* |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116659525A (en)* | 2023-05-30 | 2023-08-29 | 北京易航远智科技有限公司 | Real-time mapping method, device, electronic equipment and storage medium combined with navigation map |
| CN116499470A (en)* | 2023-06-28 | 2023-07-28 | 苏州中德睿博智能科技有限公司 | Optimal control method, device and system for positioning system of looking-around camera |
| CN116499470B (en)* | 2023-06-28 | 2023-09-05 | 苏州中德睿博智能科技有限公司 | Optimal control method, device and system for positioning system of looking-around camera |
| CN117237237A (en)* | 2023-11-13 | 2023-12-15 | 深圳元戎启行科技有限公司 | Luminosity balancing method and device for vehicle-mounted 360-degree panoramic image |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109084782B (en) | Lane line map construction method and system based on camera sensors | |
| CN111912416B (en) | Method, device and device for device positioning | |
| CN108362295B (en) | Vehicle route guidance device and method | |
| JP7404010B2 (en) | Position estimation device and method | |
| US11625851B2 (en) | Geographic object detection apparatus and geographic object detection method | |
| CN111121754A (en) | Mobile robot positioning and navigation method, device, mobile robot and storage medium | |
| JP2022517961A (en) | Method and device for automatically annotating image data | |
| CN111508018B (en) | Method and device for executing self vehicle positioning by utilizing V2X information fusion | |
| CN115456898A (en) | Method and device for building image of parking lot, vehicle and storage medium | |
| CN110147382A (en) | Lane line update method, device, equipment, system and readable storage medium storing program for executing | |
| US12422535B2 (en) | Method for calibration of camera and lidar, and computer program recorded on recording medium for executing method therefor | |
| JP2018124787A (en) | Information processing device, data managing device, data managing system, method, and program | |
| CN112232275B (en) | Obstacle detection method, system, equipment and storage medium based on binocular recognition | |
| US20210407128A1 (en) | Learnable localization using images | |
| CN112739983B (en) | Method and related device for correcting point cloud data | |
| US20240427019A1 (en) | Visual mapping method, and computer program recorded on recording medium for executing method therefor | |
| WO2020156923A2 (en) | Map and method for creating a map | |
| US12366643B2 (en) | Method for calibration of lidar and IMU, and computer program recorded on recording medium for executing method therefor | |
| US20240426987A1 (en) | Method for calibration of multiple lidars, and computer program recorded on record-medium for executing method therefor | |
| JP2017181476A (en) | Vehicle position detection device, vehicle position detection method, and computer program for vehicle position detection | |
| CN115790568A (en) | Map generation method based on semantic information and related equipment | |
| CN116997771A (en) | Vehicle, positioning method, device, equipment and computer readable storage medium thereof | |
| CN117893634A (en) | Simultaneous positioning and map construction method and related equipment | |
| CN114120701B (en) | Parking positioning method and device | |
| WO2022133986A1 (en) | Accuracy estimation method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |