CN112630749B - Method and device for outputting prompt information - Google Patents
Method and device for outputting prompt informationDownload PDFInfo
- Publication number
- CN112630749B CN112630749BCN201910903452.3ACN201910903452ACN112630749BCN 112630749 BCN112630749 BCN 112630749BCN 201910903452 ACN201910903452 ACN 201910903452ACN 112630749 BCN112630749 BCN 112630749B
- Authority
- CN
- China
- Prior art keywords
- point cloud
- distribution
- frame
- lidars
- distribution histogram
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/497—Means for monitoring or calibrating
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Traffic Control Systems (AREA)
- Optical Radar Systems And Details Thereof (AREA)
Abstract
Description
Translated fromChinese技术领域technical field
本申请实施例涉及计算机技术领域，具体涉及用于输出提示信息的方法和装置。The embodiments of the present application relate to the field of computer technology, and in particular to a method and device for outputting prompt information.
背景技术Background technique
无人驾驶汽车是一种新型的智能汽车，也称之为“轮式移动机器人”，主要通过传感器(激光雷达、摄像头等)来获取其周边环境数据，并对这些数据进行综合分析运算后，发出指令来分别控制无人驾驶汽车中的不同设备，从而实现汽车的全自动运行，达到汽车无人驾驶的目的。A driverless car is a new type of smart car, also known as a "wheeled mobile robot". It mainly uses sensors (lidar, camera, etc.) Send instructions to control different devices in the driverless car separately, so as to realize the fully automatic operation of the car and achieve the goal of driverless cars.
通常，无人驾驶汽车通过安装多个激光雷达实现感知。并且，这些激光雷达被固定装置固定在无人驾驶汽车的车顶，并标定外参。我们期望的是外参标定好之后就不再变动。然而，在无人驾驶汽车实际运行过程中，无人驾驶汽车的颠簸、人为因素或其他未知情况均会导致激光雷达的位姿发生改变，造成后续感知到的障碍物的位姿错误。Typically, self-driving cars achieve perception by installing multiple lidars. Moreover, these lidars are fixed on the roof of the driverless car by fixtures, and the external parameters are calibrated. What we expect is that the external parameters will not change after they are calibrated. However, during the actual operation of the unmanned vehicle, the bumps of the unmanned vehicle, human factors or other unknown conditions will cause the pose of the lidar to change, resulting in errors in the subsequent perception of obstacles.
发明内容Contents of the invention
本申请实施例提出了用于输出提示信息的方法和装置。The embodiments of the present application propose a method and an apparatus for outputting prompt information.
第一方面，本申请实施例提供了一种用于输出提示信息的方法，包括：获取在预设时间段内采集的至少一帧点云，其中，同一帧点云是无人车上的多个激光雷达在同一时刻采集的；以高度区间为横坐标，以数目为纵坐标，绘制至少一帧点云的分布直方图；基于分布直方图中的点云点的分布情况输出标定提示信息。In the first aspect, the embodiment of the present application provides a method for outputting prompt information, including: acquiring at least one frame of point cloud collected within a preset period of time, wherein the same frame of point cloud is multiple points on the unmanned vehicle Lidars are collected at the same time; with the height interval as the abscissa and the number as the ordinate, draw a distribution histogram of at least one frame of point cloud; output calibration prompt information based on the distribution of point cloud points in the distribution histogram.
在一些实施例中，基于分布直方图的分布情况输出标定提示信息，包括：若分布直方图服从单峰分布，输出用于提示多个激光雷达标定正确的信息；若分布直方图服从多峰分布，输出用于提示多个激光雷达标定错误的信息。In some embodiments, the calibration prompt information is output based on the distribution of the distribution histogram, including: if the distribution histogram obeys a unimodal distribution, output information for prompting that multiple lidars are calibrated correctly; if the distribution histogram obeys a multimodal distribution , output the information used to prompt multiple lidar calibration errors.
在一些实施例中，该方法还包括：若分布直方图的峰的数目不小于2，且小于全部激光雷达的数目，确定存在峰的数目减1个激光雷达标定错误；若分布直方图的峰的数目等于全部激光雷达的数目，确定存在峰的数目减1个激光雷达标定错误或全部激光雷达标定错误。In some embodiments, the method also includes: if the number of peaks in the distribution histogram is not less than 2 and is less than the number of all lidars, determining that there is a calibration error of the number of peaks minus 1 lidar; if the peak in the distribution histogram The number of is equal to the number of all lidars, and it is determined that there are peaks minus 1 lidar calibration error or all lidar calibration errors.
在一些实施例中，至少一帧点云包括以下一项：一帧点云、连续帧点云、间隔帧点云。In some embodiments, at least one frame of point cloud includes one of the following: one frame of point cloud, continuous frame of point cloud, and interval frame of point cloud.
在一些实施例中，多个激光雷达采集的同一帧点云的点云点均是在同一预设坐标系中的坐标点，其中，预设坐标系包括以下至少一项：惯性测量单元坐标系、车辆坐标系、激光雷达坐标系。In some embodiments, the point cloud points of the same frame of point cloud collected by multiple lidars are all coordinate points in the same preset coordinate system, wherein the preset coordinate system includes at least one of the following: inertial measurement unit coordinate system , vehicle coordinate system, lidar coordinate system.
第二方面，本申请实施例提供了一种用于输出提示信息的装置，包括：获取单元，被配置成获取在预设时间段内采集的至少一帧点云，其中，同一帧点云是无人车上的多个激光雷达在同一时刻采集的；绘制单元，被配置成以高度区间为横坐标，以数目为纵坐标，绘制至少一帧点云的分布直方图；输出单元，被配置成基于分布直方图中的点云点的分布情况输出标定提示信息。In a second aspect, the embodiment of the present application provides a device for outputting prompt information, including: an acquisition unit configured to acquire at least one frame of point cloud collected within a preset time period, wherein the same frame of point cloud is Multiple lidars on the unmanned vehicle are collected at the same time; the drawing unit is configured to draw the distribution histogram of at least one frame of point cloud with the height interval as the abscissa and the number as the ordinate; the output unit is configured Output calibration prompt information based on the distribution of point cloud points in the distribution histogram.
在一些实施例中，输出单元进一步被配置成：若分布直方图服从单峰分布，输出用于提示多个激光雷达标定正确的信息；若分布直方图服从多峰分布，输出用于提示多个激光雷达标定错误的信息。In some embodiments, the output unit is further configured to: if the distribution histogram obeys a unimodal distribution, output information for prompting that multiple laser radars are calibrated correctly; if the distribution histogram obeys a multimodal distribution, output for prompting multiple Lidar calibration error information.
在一些实施例中，该装置还包括：第一确定单元，被配置成若分布直方图的峰的数目不小于2，且小于全部激光雷达的数目，确定存在峰的数目减1个激光雷达标定错误；第二确定单元，被配置成若分布直方图的峰的数目等于全部激光雷达的数目，确定存在峰的数目减1个激光雷达标定错误或全部激光雷达标定错误。In some embodiments, the device further includes: a first determination unit, configured to determine that the number of peaks minus 1 lidar calibration is present if the number of peaks in the distribution histogram is not less than 2 and is less than the number of all lidars Error; the second determining unit is configured to determine that if the number of peaks in the distribution histogram is equal to the number of all lidars, there is a number of peaks minus 1 lidar calibration error or all lidar calibration errors.
在一些实施例中，至少一帧点云包括以下一项：一帧点云、连续帧点云、间隔帧点云。In some embodiments, at least one frame of point cloud includes one of the following: one frame of point cloud, continuous frame of point cloud, and interval frame of point cloud.
在一些实施例中，多个激光雷达采集的同一帧点云的点云点均是在同一预设坐标系中的坐标点，其中，预设坐标系包括以下至少一项：惯性测量单元坐标系、车辆坐标系、激光雷达坐标系。In some embodiments, the point cloud points of the same frame of point cloud collected by multiple lidars are all coordinate points in the same preset coordinate system, wherein the preset coordinate system includes at least one of the following: inertial measurement unit coordinate system , vehicle coordinate system, lidar coordinate system.
第三方面，本申请实施例提供了一种电子设备，该电子设备包括：一个或多个处理器；存储装置，其上存储有一个或多个程序；当一个或多个程序被一个或多个处理器执行，使得一个或多个处理器实现如第一方面中任一实现方式描述的方法。In the third aspect, the embodiment of the present application provides an electronic device, the electronic device includes: one or more processors; a storage device, on which one or more programs are stored; when one or more programs are used by one or more processors, so that one or more processors implement the method described in any implementation manner in the first aspect.
第四方面，本申请实施例提供了一种计算机可读介质，其上存储有计算机程序，该计算机程序被处理器执行时实现如第一方面中任一实现方式描述的方法。In a fourth aspect, an embodiment of the present application provides a computer-readable medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in any implementation manner in the first aspect is implemented.
本申请实施例提供的用于输出提示信息的方法和装置，首先获取在预设时间段内采集的至少一帧点云；然后以高度区间为横坐标，以数目为纵坐标，绘制至少一帧点云的分布直方图；最后基于分布直方图中的点云点的分布情况输出标定提示信息。基于分布直方图中的点云点的分布情况，能够快速地确定无人车上安装的激光雷达是否需要重新标定，有助于减少障碍物的位姿感知错误的情况。In the method and device for outputting prompt information provided by the embodiments of the present application, firstly obtain at least one frame of point cloud collected within a preset time period; then draw at least one frame with the height interval as the abscissa and the number as the ordinate The distribution histogram of the point cloud; finally output the calibration prompt information based on the distribution of the point cloud points in the distribution histogram. Based on the distribution of point cloud points in the distribution histogram, it is possible to quickly determine whether the lidar installed on the unmanned vehicle needs to be recalibrated, which helps to reduce the situation of obstacle perception errors.
附图说明Description of drawings
通过阅读参照以下附图所作的对非限制性实施例所作的详细描述，本申请的其它特征、目的和优点将会变得更明显：Other characteristics, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:
图1是本申请可以应用于其中的示例性系统架构；FIG. 1 is an exemplary system architecture to which the present application can be applied;
图2是根据本申请的用于输出提示信息的方法的一个实施例的流程图；FIG. 2 is a flowchart of an embodiment of a method for outputting prompt information according to the present application;
图3是根据本申请的用于输出提示信息的方法的又一个实施例的流程图；Fig. 3 is a flow chart of another embodiment of the method for outputting prompt information according to the present application;
图4是根据本申请的用于输出提示信息的装置的一个实施例的结构示意图；Fig. 4 is a schematic structural diagram of an embodiment of a device for outputting prompt information according to the present application;
图5是适于用来实现本申请实施例的电子设备的计算机系统的结构示意图。Fig. 5 is a schematic structural diagram of a computer system suitable for implementing the electronic device of the embodiment of the present application.
具体实施方式Detailed ways
下面结合附图和实施例对本申请作进一步的详细说明。可以理解的是，此处所描述的具体实施例仅仅用于解释相关发明，而非对该发明的限定。另外还需要说明的是，为了便于描述，附图中仅示出了与有关发明相关的部分。The application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain related inventions, rather than to limit the invention. It should also be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.
需要说明的是，在不冲突的情况下，本申请中的实施例及实施例中的特征可以相互组合。下面将参考附图并结合实施例来详细说明本申请。It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and embodiments.
图1示出了可以应用本申请的用于输出提示信息的方法或用于输出提示信息的装置的实施例的示例性系统架构100。FIG. 1 shows an
如图1所示，系统架构100中可以包括无人驾驶汽车101。无人驾驶汽车101上可以安装有激光雷达1011、1012、1013，网络1014和驾驶控制设备1015。网络1014用以在激光雷达1011、1012、1013和驾驶控制设备1015之间提供通信链路的介质。网络1014可以包括各种连接类型，例如有线、无线通信链路或者光纤电缆等等。As shown in FIG. 1 , an
激光雷达1011、1012、1013可以通过网络1014与驾驶控制设备1015交互，以接收或发送消息等。The lidars 1011, 1012, 1013 can interact with the driving control device 1015 through the
激光雷达1011、1012、1013可以是以发射激光束探测目标的位置、速度等特征量的雷达系统。具体地，当激光雷达1011、1012、1013发射的激光束照射到目标表面时，所反射的激光会携带方位、距离等信息。若激光雷达1011、1012、1013发射的激光束按照某种轨迹进行扫描，便会边扫描边记录到反射的激光点信息，由于扫描极为精细，则能够得到大量的激光点，因而就可形成激光点云。The laser radars 1011, 1012, and 1013 may be radar systems that emit laser beams to detect feature quantities such as the position and speed of targets. Specifically, when the laser beams emitted by the lidars 1011 , 1012 , and 1013 irradiate the target surface, the reflected laser light will carry information such as azimuth and distance. If the laser beams emitted by the lidars 1011, 1012, and 1013 are scanned according to a certain trajectory, the reflected laser point information will be recorded while scanning. Since the scanning is extremely fine, a large number of laser points can be obtained, thus forming a laser beam. point cloud.
驾驶控制设备1015，又称为车载大脑，负责无人驾驶汽车101的智能控制。驾驶控制设备1015可以是单独设置的控制器，例如可编程逻辑控制器、单片机、工业控制机等；也可以是由其他具有输入/输出端口，并具有运算控制功能的电子器件组成的设备；还可以是安装有车辆驾驶控制类应用的计算机设备。The driving control device 1015 , also known as the on-board brain, is responsible for the intelligent control of the
需要说明的是，本申请实施例所提供的用于输出提示信息的方法一般由驾驶控制设备1015执行，相应地，用于输出提示信息的装置一般设置于驾驶控制设备1015中。It should be noted that the method for outputting prompt information provided in the embodiment of the present application is generally executed by the driving control device 1015 , and correspondingly, the device for outputting prompt information is generally set in the driving control device 1015 .
应该理解，图1中的驾驶控制设备、网络和激光雷达的数目仅仅是示意性的。根据实现需要，可以具有任意数目的驾驶控制设备、网络和激光雷达。It should be understood that the numbers of driving control devices, networks and lidars in Figure 1 are merely illustrative. There can be any number of driving control devices, networks, and lidars depending on implementation needs.
继续参考图2，其示出了根据本申请的用于输出提示信息的方法的一个实施例的流程200。该用于输出提示信息的方法包括以下步骤：Continue to refer to FIG. 2 , which shows a
步骤201，获取在预设时间段内采集的至少一帧点云。
在本实施例中，用于输出提示信息的方法的执行主体(例如图1所示的驾驶控制设备1015)可以获取激光雷达(例如图1所示的激光雷达1011、1012、1013)在预设时间段内(例如前1分钟内)采集的至少一帧点云。通常，当采集到每帧点云时，激光雷达会将采集到的该帧点云实时发送至上述执行主体。In this embodiment, the executing subject of the method for outputting prompt information (for example, the driving control device 1015 shown in FIG. At least one frame of point cloud collected within a time period (for example, within the previous 1 minute). Usually, when each frame of point cloud is collected, the lidar will send the collected frame of point cloud to the above-mentioned execution subject in real time.
实践中，激光雷达可以安装在无人驾驶汽车(例如图1所示的无人驾驶汽车101)的车顶，用于采集无人驾驶汽车周围的物体的点云。并且，无人驾驶汽车可以安装多个激光雷达，同一帧点云是无人车上的多个激光雷达在同一时刻采集的。其中，每一帧点云可以由大量点云点(激光点)组成，每个点云点可以包括三维坐标和激光反射强度。通常，在对激光雷达标定外参时，会预设坐标系。而同一无人驾驶汽车上的多个激光雷达采集的同一帧点云中的点云点均会是在同一预设坐标系中的坐标点。预设坐标系可以随着无人驾驶汽车的运动而运动，例如车辆坐标系、IMU(Inertial measurement unit，惯性测量单元)坐标系、激光雷达坐标系等。需要说明的是，当预设坐标系是激光雷达坐标系时，同一无人驾驶汽车上安装的不同激光雷达采集的点云点均是在同一激光雷达坐标系中的坐标点。In practice, the lidar can be installed on the roof of an unmanned vehicle (such as the
在本实施例的一些可选的实现方式中，上述执行主体获取到的至少一帧点云可以包括以下一项：多个激光雷达在预设时间段内采集的一帧点云、多个激光雷达在预设时间段内采集的连续帧点云、多个激光雷达在预设时间段内采集的间隔帧点云等等。In some optional implementations of this embodiment, the at least one frame of point cloud acquired by the above execution subject may include the following items: a frame of point cloud collected by multiple laser radars within a preset time period, multiple laser Continuous frame point clouds collected by radar within a preset time period, interval frame point clouds collected by multiple lidars within a preset time period, etc.
通常，点云帧数越多，分布直方图的分布情况越真实。为了确保分布的真实性，上述执行主体通常会获取多帧点云进行分析。例如，上述执行主体可以读取连续的一段时间内的点云。实践中，上述执行主体读取的是30秒以上的时间段内的点云，例如，读取1分钟内的点云。此时，若激光雷达的采集频率是10帧/秒，那么上述执行主体可以读取出600帧点云。其中，每一帧点云可以包括两万多个点云点。Generally, the more the number of point cloud frames, the more realistic the distribution of the distribution histogram. In order to ensure the authenticity of the distribution, the above execution subjects usually obtain multi-frame point clouds for analysis. For example, the above execution subject may read the point cloud in a continuous period of time. In practice, the above execution subject reads the point cloud within a time period of more than 30 seconds, for example, reads the point cloud within 1 minute. At this time, if the acquisition frequency of the lidar is 10 frames per second, then the above execution subject can read out 600 frames of point clouds. Among them, each frame of point cloud can include more than 20,000 point cloud points.
步骤202，以高度区间为横坐标，以数目为纵坐标，绘制至少一帧点云的分布直方图。
在本实施例中，上述执行主体可以以高度区间为横坐标，以数目为纵坐标，绘制至少一帧点云的分布直方图。其中，至少一帧点云中的点云点的高度可以等于该点云点的三维坐标中的Z坐标的值。In this embodiment, the execution subject may draw a distribution histogram of at least one frame of point clouds with the height interval as the abscissa and the number as the ordinate. Wherein, the height of the point cloud point in at least one frame of point cloud may be equal to the value of the Z coordinate in the three-dimensional coordinates of the point cloud point.
这里，上述执行主体可以将高度划分为多个细粒度区间，例如，高度区间可以包括0-0.1米、0.1-0.2米、0.2-0.3米等等。随后，上述执行主体可以统计落入到每个高度区间的点云点的数目，并绘制出分布直方图。Here, the aforementioned executive body may divide the height into multiple fine-grained intervals, for example, the height interval may include 0-0.1 meters, 0.1-0.2 meters, 0.2-0.3 meters and so on. Subsequently, the above execution subject can count the number of point cloud points falling into each height interval, and draw a distribution histogram.
步骤203，基于分布直方图中的点云点的分布情况输出标定提示信息。
在本实施例中，上述执行主体可以基于分布直方图中的点云点的分布情况输出标定提示信息。In this embodiment, the execution subject may output calibration prompt information based on the distribution of point cloud points in the distribution histogram.
通常，无人驾驶汽车上安装的激光雷达发射的激光束大都会照射到地面上。也就是说，至少一帧点云中会有很大部分点云点属于地面。虽然地面不是完全平整的，但是地面上的点云点的高度误差却在2-3厘米之间。因此，多个激光雷达采集的至少一帧点云中的绝大多数的点云点是地面点云点。若多个激光雷达标定正确，不同激光雷达采集的地面点云点均会落入在同一个高度区间。并且，落入这个高度区间的点云点的数目明显高于落入其相邻区间的点云点的数目。也就是说，分布直方图服从正态单峰分布，地面点云点落入的高度区间是峰值区间。Usually, most of the laser beams emitted by the lidar installed on the self-driving car shine on the ground. That is to say, at least a large part of point cloud points in one frame of point cloud belong to the ground. Although the ground is not completely flat, the height error of the point cloud points on the ground is between 2-3 cm. Therefore, most of the point cloud points in at least one frame of point cloud collected by multiple lidars are ground point cloud points. If multiple lidars are calibrated correctly, the ground point cloud points collected by different lidars will all fall into the same height interval. Also, the number of point cloud points falling into this height interval is significantly higher than the number of point cloud points falling into its adjacent intervals. That is to say, the distribution histogram obeys the normal unimodal distribution, and the height interval where the ground point cloud points fall is the peak interval.
在本实施例的一些可选的实现方式中，若分布直方图服从单峰分布，说明无人驾驶汽车上安装的多个激光雷达采集点云点均是在同一预设坐标系中的坐标点。此时，上述执行主体可以输出用于提示多个激光雷达标定正确的信息，以提示用户无人驾驶汽车上安装的激光雷达无需重新标定。若分布直方图服从多峰分布，说明无人驾驶汽车上安装的多个激光雷达采集点云点不是在同一预设坐标系中的坐标点。此时，上述执行主体可以输出用于提示多个激光雷达标定错误的信息，以提示用户无人驾驶汽车上安装的激光雷达需要重新标定。In some optional implementations of this embodiment, if the distribution histogram obeys a unimodal distribution, it means that the point cloud points collected by multiple lidars installed on the driverless car are all coordinate points in the same preset coordinate system . At this time, the above-mentioned executive body can output information for prompting that multiple laser radars are calibrated correctly, so as to remind the user that the laser radars installed on the driverless car do not need to be re-calibrated. If the distribution histogram obeys the multimodal distribution, it means that the point cloud points collected by multiple lidars installed on the driverless car are not coordinate points in the same preset coordinate system. At this time, the above-mentioned executive body can output information for prompting multiple lidar calibration errors, so as to remind the user that the lidar installed on the driverless car needs to be re-calibrated.
应当理解的是，在实际应用中，安装在同一无人驾驶汽车上的多个激光雷达通常是相同线束的激光雷达，例如4线、16线、40线、128线等等。此外，本申请实施例提供的用于输出提示信息的方法对于同一无人驾驶汽车上安装多个不同线束的激光雷达同样适用。It should be understood that in practical applications, the multiple lidars installed on the same unmanned vehicle are usually lidars of the same wiring harness, such as 4-wire, 16-wire, 40-wire, 128-wire and so on. In addition, the method for outputting prompt information provided in the embodiment of the present application is also applicable to lidars with multiple different wiring harnesses installed on the same driverless car.
本申请实施例提供的用于输出提示信息的方法，首先获取在预设时间段内采集的至少一帧点云；然后以高度区间为横坐标，以数目为纵坐标，绘制至少一帧点云的分布直方图；最后基于分布直方图中的点云点的分布情况输出标定提示信息。基于分布直方图中的点云点的分布情况，能够快速地确定无人车上安装的激光雷达是否需要重新标定，有助于减少障碍物的位姿感知错误的情况。The method for outputting prompt information provided by the embodiment of the present application first obtains at least one frame of point cloud collected within a preset time period; then draws at least one frame of point cloud with the height interval as the abscissa and the number as the ordinate The distribution histogram of the distribution histogram; finally output the calibration prompt information based on the distribution of the point cloud points in the distribution histogram. Based on the distribution of point cloud points in the distribution histogram, it is possible to quickly determine whether the lidar installed on the unmanned vehicle needs to be recalibrated, which helps to reduce the situation of obstacle perception errors.
进一步参考图3，其示出了根据本申请的用于输出提示信息的方法的又一个实施例的流程300。该用于输出提示信息的方法包括以下步骤：Further referring to FIG. 3 , it shows a
步骤301，获取在预设时间段内采集的至少一帧点云。
步骤302，以高度区间为横坐标，以数目为纵坐标，绘制至少一帧点云的分布直方图。
在本实施例中，步骤301-302的具体操作已在图2所示的实施例中步骤201-202中进行了详细的介绍，在此不再赘述。In this embodiment, the specific operations of steps 301-302 have been introduced in detail in steps 201-202 in the embodiment shown in FIG. 2 , and will not be repeated here.
步骤303，若分布直方图服从单峰分布，输出用于提示多个激光雷达标定正确的信息。
在本实施例中，用于输出提示信息的方法的执行主体(例如图1所示的驾驶控制设备1015)可以分析分布直方图的分布情况。若分布直方图服从单峰分布，说明无人驾驶汽车上安装的多个激光雷达采集点云点均是在同一预设坐标系中的坐标点。此时，上述执行主体可以输出用于提示多个激光雷达标定正确的信息，以提示用户无人驾驶汽车上安装的激光雷达无需重新标定。In this embodiment, the executing subject of the method for outputting prompt information (for example, the driving control device 1015 shown in FIG. 1 ) may analyze the distribution of the distribution histogram. If the distribution histogram obeys the unimodal distribution, it means that the point cloud points collected by multiple lidars installed on the driverless car are all coordinate points in the same preset coordinate system. At this time, the above-mentioned executive body can output information for prompting that multiple laser radars are calibrated correctly, so as to remind the user that the laser radars installed on the driverless car do not need to be re-calibrated.
步骤304，若分布直方图服从多峰分布，输出用于提示多个激光雷达标定错误的信息。
在本实施例中，若分布直方图服从多峰分布，说明无人驾驶汽车上安装的多个激光雷达采集点云点不是在同一预设坐标系中的坐标点。此时，上述执行主体可以输出用于提示多个激光雷达标定错误的信息，以提示用户无人驾驶汽车上安装的激光雷达需要重新标定。In this embodiment, if the distribution histogram obeys the multimodal distribution, it means that the point cloud points collected by multiple lidars installed on the driverless car are not coordinate points in the same preset coordinate system. At this time, the above-mentioned executive body can output information for prompting multiple lidar calibration errors, so as to remind the user that the lidar installed on the driverless car needs to be re-calibrated.
步骤305，若分布直方图的峰的数目不小于2，且小于全部激光雷达的数目，确定存在峰的数目减1个激光雷达标定错误。
在本实施例中，若分布直方图的峰的数目不小于2，且小于全部激光雷达的数目，上述执行主体可以确定存在峰的数目减1个激光雷达标定错误。例如，无人驾驶汽车上安装有3个激光雷达，在激光雷达标定正确的情况下，分布直方图存在1个峰。当存在1个激光雷达标定错误的情况下，这个标定错误的激光雷达采集的点云点的分布情况会发生移位，此时，分布直方图会存在2个峰。In this embodiment, if the number of peaks in the distribution histogram is not less than 2 and less than the number of all lidars, the execution subject may determine that there is a calibration error of the number of peaks minus 1 lidar. For example, there are 3 lidars installed on the driverless car, and when the lidars are calibrated correctly, there is a peak in the distribution histogram. When there is one lidar calibration error, the distribution of point cloud points collected by the wrong calibration lidar will shift, and at this time, there will be two peaks in the distribution histogram.
步骤306，若分布直方图的峰的数目等于全部激光雷达的数目，确定存在峰的数目减1个激光雷达标定错误或全部激光雷达标定错误。
在本实施例中，若分布直方图的峰的数目等于全部激光雷达的数目，上述执行主体可以确定存在峰的数目减1个激光雷达标定错误或全部激光雷达标定错误。例如，无人驾驶汽车上安装有3个激光雷达，在激光雷达标定正确的情况下，分布直方图存在1个峰。当存在2个激光雷达标定错误的情况下，这2个标定错误的激光雷达采集的点云点的分布情况均会发生移位，此时，分布直方图会存在3个峰。此外，当存在3个激光雷达标定错误的情况下，这3个标定错误的激光雷达采集的点云点的分布情况均会发生移位，此时，分布直方图会存在3个峰。In this embodiment, if the number of peaks in the distribution histogram is equal to the number of all lidars, the execution subject may determine that there is a number of peaks minus 1 lidar calibration error or all lidar calibration errors. For example, there are 3 lidars installed on the driverless car, and when the lidars are calibrated correctly, there is a peak in the distribution histogram. When there are two lidar calibration errors, the distribution of point cloud points collected by the two incorrectly calibrated lidars will shift. At this time, there will be three peaks in the distribution histogram. In addition, when there are three lidar calibration errors, the distribution of point cloud points collected by the three incorrectly calibrated lidars will shift, and at this time, there will be three peaks in the distribution histogram.
从图3中可以看出，与图2对应的实施例相比，本实施例中的用于输出提示信息的方法的流程300增加了确定标定错误的激光雷达的数目的步骤。由此，本实施例描述的方案通过分布直方图的峰的数目，可以快速地确定出标定错误的激光雷达的数目。It can be seen from FIG. 3 that, compared with the embodiment corresponding to FIG. 2 , the
进一步参考图4，作为对上述各图所示的方法的实现，本申请提供了一种用于输出提示信息的装置的一个实施例，该装置实施例与图2所示的方法实施例相对应，该装置具体可以应用于各种电子设备中。Further referring to FIG. 4 , as an implementation of the methods shown in the above figures, the present application provides an embodiment of a device for outputting prompt information, which corresponds to the method embodiment shown in FIG. 2 , the device can be specifically applied to various electronic devices.
如图4所示，本实施例的用于输出提示信息的装置400可以包括：获取单元401、绘制单元402和输出单元403。其中，获取单元401，被配置成获取在预设时间段内采集的至少一帧点云，其中，同一帧点云是无人车上的多个激光雷达在同一时刻采集的；绘制单元402，被配置成以高度区间为横坐标，以数目为纵坐标，绘制至少一帧点云的分布直方图；输出单元403，被配置成基于分布直方图中的点云点的分布情况输出标定提示信息。As shown in FIG. 4 , the
在本实施例中，用于输出提示信息的装置400中：获取单元401、绘制单元402和输出单元403的具体处理及其所带来的技术效果可分别参考图2对应实施例中的步骤201-203的相关说明，在此不再赘述。In this embodiment, in the
在本实施例的一些可选的实现方式中，输出单元403进一步被配置成：若分布直方图服从单峰分布，输出用于提示多个激光雷达标定正确的信息；若分布直方图服从多峰分布，输出用于提示多个激光雷达标定错误的信息。In some optional implementations of this embodiment, the
在本实施例的一些可选的实现方式中，用于输出提示信息的装置400还包括：第一确定单元(图中未示出)，被配置成若分布直方图的峰的数目不小于2，且小于全部激光雷达的数目，确定存在峰的数目减1个激光雷达标定错误；第二确定单元(图中未示出)，被配置成若分布直方图的峰的数目等于全部激光雷达的数目，确定存在峰的数目减1个激光雷达标定错误或全部激光雷达标定错误。In some optional implementations of this embodiment, the
在本实施例的一些可选的实现方式中，至少一帧点云包括以下一项：一帧点云、连续帧点云、间隔帧点云。In some optional implementation manners of this embodiment, at least one frame of point cloud includes one of the following: one frame of point cloud, continuous frame of point cloud, and interval frame of point cloud.
在本实施例的一些可选的实现方式中，多个激光雷达采集的同一帧点云的点云点均是在同一预设坐标系中的坐标点，其中，预设坐标系包括以下至少一项：惯性测量单元坐标系、车辆坐标系、激光雷达坐标系。In some optional implementations of this embodiment, the point cloud points of the same frame of point cloud collected by multiple lidars are all coordinate points in the same preset coordinate system, wherein the preset coordinate system includes at least one of the following Item: IMU coordinate system, vehicle coordinate system, lidar coordinate system.
下面参考图5，其示出了适于用来实现本申请实施例的电子设备(例如图1所示的驾驶控制设备1015)的计算机系统500的结构示意图。图5示出的电子设备仅仅是一个示例，不应对本申请实施例的功能和使用范围带来任何限制。Referring now to FIG. 5 , it shows a schematic structural diagram of a
如图5所示，计算机系统500包括中央处理单元(CPU)501，其可以根据存储在只读存储器(ROM)502中的程序或者从存储部分508加载到随机访问存储器(RAM)503中的程序而执行各种适当的动作和处理。在RAM 503中，还存储有系统500操作所需的各种程序和数据。CPU 501、ROM 502以及RAM 503通过总线504彼此相连。输入/输出(I/O)接口505也连接至总线504。As shown in FIG. 5 , a
以下部件连接至I/O接口505：包括键盘、鼠标等的输入部分506；包括诸如阴极射线管(CRT)、液晶显示器(LCD)等以及扬声器等的输出部分507；包括硬盘等的存储部分508；以及包括诸如LAN卡、调制解调器等的网络接口卡的通信部分509。通信部分509经由诸如因特网的网络执行通信处理。驱动器510也根据需要连接至I/O接口505。可拆卸介质511，诸如磁盘、光盘、磁光盘、半导体存储器等等，根据需要安装在驱动器510上，以便于从其上读出的计算机程序根据需要被安装入存储部分508。The following components are connected to the I/O interface 505: an
特别地，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信部分509从网络上被下载和安装，和/或从可拆卸介质511被安装。在该计算机程序被中央处理单元(CPU)501执行时，执行本申请的方法中限定的上述功能。In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, where the computer program includes program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via
需要说明的是，本申请所述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本申请中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本申请中，计算机可读的信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：无线、电线、光缆、RF等等，或者上述的任意合适的组合。It should be noted that the computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, in which computer-readable program codes are carried. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. . Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
可以以一种或多种程序设计语言或其组合来编写用于执行本申请的操作的计算机程序代码，所述程序设计语言包括面向目标的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言—诸如”C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或电子设备上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。Computer program code for carrying out the operations of this application can be written in one or more programming languages, or combinations thereof, including object-oriented programming languages—such as Java, Smalltalk, C++, and conventional A procedural programming language—such as "C" or a similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or electronic device. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as through an Internet service provider). Internet connection).
附图中的流程图和框图，图示了按照本申请各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that contains one or more logical functions for implementing specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.
描述于本申请实施例中所涉及到的单元可以通过软件的方式实现，也可以通过硬件的方式来实现。所描述的单元也可以设置在处理器中，例如，可以描述为：一种处理器包括获取单元、绘制单元和输出单元。其中，这些单元的名称在种情况下并不构成对该单元本身的限定，例如，获取单元还可以被描述为“获取在预设时间段内采集的至少一帧点云的单元”。The units involved in the embodiments described in the present application may be implemented by means of software or by means of hardware. The described units may also be set in a processor, for example, it may be described as: a processor includes an acquisition unit, a rendering unit, and an output unit. Wherein, the names of these units do not constitute a limitation on the unit itself in this case, for example, the acquisition unit may also be described as "a unit that acquires at least one frame of point cloud collected within a preset time period".
作为另一方面，本申请还提供了一种计算机可读介质，该计算机可读介质可以是上述实施例中描述的电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该电子设备执行时，使得该电子设备：获取在预设时间段内采集的至少一帧点云，其中，同一帧点云是无人车上的多个激光雷达在同一时刻采集的；以高度区间为横坐标，以数目为纵坐标，绘制至少一帧点云的分布直方图；基于分布直方图中的点云点的分布情况输出标定提示信息。As another aspect, the present application also provides a computer-readable medium. The computer-readable medium may be included in the electronic device described in the above embodiments; it may also exist independently without being assembled into the electronic device. middle. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device: acquires at least one frame of point cloud collected within a preset time period, wherein the same The frame point cloud is collected by multiple lidars on the unmanned vehicle at the same time; with the height interval as the abscissa and the number as the ordinate, draw the distribution histogram of at least one frame of point cloud; based on the points in the distribution histogram The distribution of cloud points outputs calibration prompt information.
以上描述仅为本申请的较佳实施例以及对所运用技术原理的说明。本领域技术人员应当理解，本申请中所涉及的发明范围，并不限于上述技术特征的特定组合而成的技术方案，同时也应涵盖在不脱离上述发明构思的情况下，由上述技术特征或其等同特征进行任意组合而形成的其它技术方案。例如上述特征与本申请中公开的(但不限于)具有类似功能的技术特征进行互相替换而形成的技术方案。The above description is only a preferred embodiment of the present application and an illustration of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, and should also cover the technical solution formed by the above-mentioned technical features or Other technical solutions formed by any combination of equivalent features. For example, a technical solution formed by replacing the above-mentioned features with technical features with similar functions disclosed in (but not limited to) this application.
Claims (10)
Translated fromChinesePriority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910903452.3ACN112630749B (en) | 2019-09-24 | 2019-09-24 | Method and device for outputting prompt information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910903452.3ACN112630749B (en) | 2019-09-24 | 2019-09-24 | Method and device for outputting prompt information |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112630749A CN112630749A (en) | 2021-04-09 |
| CN112630749Btrue CN112630749B (en) | 2023-06-09 |
Family
ID=75282619
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910903452.3AActiveCN112630749B (en) | 2019-09-24 | 2019-09-24 | Method and device for outputting prompt information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112630749B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116309199A (en)* | 2021-12-03 | 2023-06-23 | 咪咕文化科技有限公司 | A point cloud processing method, device and electronic equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| AU741352B3 (en)* | 2001-02-09 | 2001-11-29 | Commonwealth Scientific And Industrial Research Organisation | Lidar system and method |
| EP3370078A1 (en)* | 2017-03-01 | 2018-09-05 | STMicroelectronics (Grenoble 2) SAS | Range and parameter extraction using processed histograms generated from a time of flight sensor - crosstalk correction |
| CN109143207A (en)* | 2018-09-06 | 2019-01-04 | 百度在线网络技术（北京）有限公司 | Laser radar internal reference precision test method, apparatus, equipment and medium |
| WO2019109142A1 (en)* | 2017-12-06 | 2019-06-13 | University Of Technology Sydney | Monitoring systems, and computer implemented methods for processing data in monitoring systems, programmed to enable identification and tracking of human targets in crowded environments |
| CN109949372A (en)* | 2019-03-18 | 2019-06-28 | 北京智行者科技有限公司 | A kind of laser radar and vision combined calibrating method |
| CN110168311A (en)* | 2016-11-29 | 2019-08-23 | 布莱克莫尔传感器和分析公司 | Method and system for classifying objects in a point cloud data set |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100066587A1 (en)* | 2006-07-14 | 2010-03-18 | Brian Masao Yamauchi | Method and System for Controlling a Remote Vehicle |
- 2019
- 2019-09-24CNCN201910903452.3Apatent/CN112630749B/enactiveActive
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| AU741352B3 (en)* | 2001-02-09 | 2001-11-29 | Commonwealth Scientific And Industrial Research Organisation | Lidar system and method |
| CN110168311A (en)* | 2016-11-29 | 2019-08-23 | 布莱克莫尔传感器和分析公司 | Method and system for classifying objects in a point cloud data set |
| EP3370078A1 (en)* | 2017-03-01 | 2018-09-05 | STMicroelectronics (Grenoble 2) SAS | Range and parameter extraction using processed histograms generated from a time of flight sensor - crosstalk correction |
| WO2019109142A1 (en)* | 2017-12-06 | 2019-06-13 | University Of Technology Sydney | Monitoring systems, and computer implemented methods for processing data in monitoring systems, programmed to enable identification and tracking of human targets in crowded environments |
| CN109143207A (en)* | 2018-09-06 | 2019-01-04 | 百度在线网络技术（北京）有限公司 | Laser radar internal reference precision test method, apparatus, equipment and medium |
| CN109949372A (en)* | 2019-03-18 | 2019-06-28 | 北京智行者科技有限公司 | A kind of laser radar and vision combined calibrating method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112630749A (en) | 2021-04-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11328429B2 (en) | Method and apparatus for detecting ground point cloud points | |
| CN110687549B (en) | Obstacle detection method and device | |
| CN109508580B (en) | Traffic signal identification method and device | |
| US11734935B2 (en) | Transferring synthetic lidar system data to real world domain for autonomous vehicle training applications | |
| EP3627180A1 (en) | Sensor calibration method and device, computer device, medium, and vehicle | |
| CN112590813B (en) | Method, device, electronic device and medium for generating information of automatic driving vehicle | |
| US20210132614A1 (en) | Control method and apparatus for autonomous vehicle | |
| CN110696826B (en) | Method and device for controlling a vehicle | |
| JP2020080149A (en) | Method and apparatus for controlling unmanned vehicle for route verification | |
| CN113269168B (en) | Obstacle data processing method and device, electronic equipment and computer readable medium | |
| CN108495113B (en) | Control method and device for binocular vision system | |
| CN112622923A (en) | Method and device for controlling a vehicle | |
| CN116710809A (en) | Systems and methods for monitoring LiDAR sensor health | |
| US20240183983A1 (en) | Systems and methods for pose determination of a mobile subject | |
| CN110654380A (en) | Method and apparatus for controlling a vehicle | |
| CN112630749B (en) | Method and device for outputting prompt information | |
| CN109827610B (en) | Method and device for verifying sensor fusion result | |
| JP6894992B2 (en) | Methods and equipment for processing information | |
| CN112630798B (en) | Method and apparatus for estimating ground | |
| CN113034603A (en) | Method and device for determining calibration parameters | |
| CN112558035B (en) | Method and apparatus for estimating ground | |
| US20190279339A1 (en) | Generating a super-resolution depth-map | |
| CN117392241A (en) | Sensor calibration method and device in automatic driving and electronic equipment | |
| CN116805331A (en) | Vehicle heading angle calculation method, device, equipment and storage medium | |
| CN112526477A (en) | Method and apparatus for processing information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |