



技术领域technical field
本发明属于航空摄影测量技术领域,尤其涉及一种多光轴平行的测绘相机单中心投影转换方法。The invention belongs to the technical field of aerial photogrammetry, in particular to a single-center projection conversion method for a surveying and mapping camera with parallel multiple optical axes.
背景技术Background technique
测绘在自然灾害应急监测、土地资源调查、生态环境监测、城市精细化管理等领域应用广泛。Surveying and mapping are widely used in natural disaster emergency monitoring, land resource survey, ecological environment monitoring, and refined urban management.
大幅面的测绘相机是面阵测绘装备未来的发展方向,由于单个CCD或CMOS器件像素规模限制难以实现大幅面成像,通常解决方法是采取多面阵拼接技术,通常有内视场拼接、外视场拼接、内外视场混合拼接等方法。一般内视场拼接采用单一镜头,是单中心摄影测量;外视场拼接一般采用倾斜式摄影,各镜头主光轴与竖直方向有一定角度,通过数据处理获取近似单中心摄影测量拼接图像;内外视场混合拼接一般镜头“一”字排列,通过延时同地曝光实现高度近似单中心摄影测量,上述拼接方法数据处理方法相对成熟。Large-format surveying and mapping cameras are the future development direction of area-array surveying and mapping equipment. Due to the limitation of the pixel size of a single CCD or CMOS device, it is difficult to achieve large-format imaging. The usual solution is to adopt multi-area array splicing technology, usually with inner field of view stitching and outer field of view. splicing, mixed splicing of internal and external fields of view, etc. Generally, a single lens is used for the stitching of the inner field of view, which is a single-center photogrammetry; the stitching of the outer field of view generally adopts oblique photography, and the main optical axis of each lens has a certain angle with the vertical direction, and an approximate single-center photogrammetry stitching image is obtained through data processing; The mixed splicing of the internal and external fields of view is generally arranged in the "one" character of the lens, and a highly approximate single-center photogrammetry is achieved through time-delayed co-location exposure. The data processing method of the above splicing method is relatively mature.
在器件规模受限以及测绘装备安装平台限制下,上述拼接方法难以进一步增大成像幅面。主光轴平行的多镜头多探测器拼接,可以实现更多成像器件的拼接集成,实现更大幅面的成像。但主光轴平行的多镜头多探测器拼接相机在测绘领域应用中,其单中心投影转换流程,虚拟单中心影像生成方法对相机成图精度和应用至关重要。Under the limitations of the device scale and the installation platform of the surveying and mapping equipment, it is difficult for the above-mentioned splicing method to further increase the imaging size. The multi-lens and multi-detector splicing with the main optical axis parallel can realize the splicing and integration of more imaging devices and achieve larger-scale imaging. However, in the application of multi-lens multi-detector splicing cameras with parallel main optical axes in the field of surveying and mapping, the single-center projection conversion process and the virtual single-center image generation method are very important for the camera's mapping accuracy and application.
针对主光轴平行的多镜头多探测器拼接测绘装备,现有的单中心投影转换方法已不能准确描述多光轴平行的测绘相机的转换模型。For the multi-lens and multi-detector splicing surveying and mapping equipment with parallel main optical axes, the existing single-center projection conversion method can no longer accurately describe the conversion model of the multi-optical axis-parallel surveying and mapping cameras.
发明内容SUMMARY OF THE INVENTION
本发明解决的技术问题是:克服现有技术的不足,提供了一种多光轴平行的测绘相机单中心投影转换方法,解决了现有的单中心投影转换方法已不能准确描述多光轴平行的测绘相机转换模型的问题,通过光轴平行多面阵拼接实现了大幅面航测相机的核心问题。The technical problem solved by the present invention is: overcoming the deficiencies of the prior art, providing a single-center projection conversion method for a multi-optical axis parallel surveying and mapping camera, and solving the problem that the existing single-center projection conversion method cannot accurately describe the multi-optical axis parallelism The problem of converting the model of the surveying and mapping camera to the core problem of the large-format aerial surveying camera is realized through the splicing of the optical axis parallel multi-area array.
本发明目的通过以下技术方案予以实现:一种多光轴平行的测绘相机单中心投影转换方法,所述方法包括如下步骤:The object of the present invention is achieved through the following technical solutions: a single-center projection conversion method for a multi-optical axis parallel surveying and mapping camera, the method comprising the following steps:
步骤一:预设相机有S个相机单元,每个相机单元中的镜头参数相同,且各镜头的主光轴平行;相机有M×N个焦面组件参与拼接,每个焦面组件有效拼接像元数m×n,则每个相机单元的镜头焦面处虚拟焦面的大小为Mm×Nn;其中,M为参与拼接焦面组件阵列的行数,N为参与拼接焦面组件阵列的列数,m为每个焦面组件有效拼接像元数行数,n为每个焦面组件有效拼接像元数列数;Step 1: The preset camera has S camera units, the lens parameters in each camera unit are the same, and the main optical axes of each lens are parallel; the camera has M×N focal plane components participating in the stitching, and each focal plane component is effectively stitched The number of pixels is m×n, then the size of the virtual focal plane at the lens focal plane of each camera unit is Mm×Nn; where M is the number of rows participating in the focal plane assembly array, and N is the number of rows participating in the focal plane assembly array. Number of columns, m is the number of rows and rows of pixels effectively stitched for each focal plane component, and n is the number of columns and columns of effectively stitched pixels for each focal plane component;
以第i个相机单元的镜头的虚拟焦面像素中心为坐标原点,以虚拟焦面像素行方向为u轴、以虚拟焦面像素列方向为v轴,建立第i个相机单元的镜头的焦面坐标系oi-uivi,其中i=1,2,…S;Taking the virtual focal plane pixel center of the lens of the ith camera unit as the coordinate origin, taking the virtual focal plane pixel row direction as the u-axis, and taking the virtual focal plane pixel column direction as the v-axis, establish the focal point of the lens of the ith camera unit. Surface coordinate system oi -ui vi , where i=1,2,...S;
以第i个相机单元的摄站点为坐标原点Si,以平行虚拟焦面像素行方向为x轴,以平行虚拟焦面像素列方向为y轴,按右手坐标系建立第i个相机单元的像空间坐标系Si-xiyizi,其中i=1,2,…S;Taking the camera station of the i-th camera unit as the coordinate origin Si , taking the pixel row direction parallel to the virtual focal plane as the x-axis, and taking the pixel column direction parallel to the virtual focal plane as the y-axis, the right-hand coordinate system is used to establish the ith camera unit. Like the space coordinate system Si -xi yi zi , where i=1,2,...S;
步骤二:建立地面辅助坐标系O-XYZ;其中,以第1个相机单元的摄站点S1在地面的竖直投影点为O点,与第1个相机单元的像空间坐标系的x1轴平行的轴为地面辅助坐标系的X轴,与第1个相机单元的像空间坐标系的y1轴平行的轴为地面辅助坐标系的y轴;Step 2: Establish a ground auxiliary coordinate system O-XYZ; wherein, the vertical projection point of the camera station S1 of the first camera unit on the ground is point O, which is different from the image space coordinate system of the first camera unit x1 The axis parallel to the axis is the X axis of the ground auxiliary coordinate system, and the axis parallel to the y1 axis of the image space coordinate system of the first camera unit is the y axis of the ground auxiliary coordinate system;
步骤三:根据除第1个相机单元外的各相机单元相对第1个相机单元的线元素和除第1个相机单元外的各相机单元相对第1个相机单元的角元素得到除第1个相机单元外的各相机单元相对第1个相机单元的旋转矩阵;Step 3: According to the line elements of each camera unit except the first camera unit relative to the first camera unit and the corner elements of each camera unit except the first camera unit relative to the first camera unit The rotation matrix of each camera unit outside the camera unit relative to the first camera unit;
步骤四:以第1个相机单元的像空间坐标系S1-x1y1z1为虚拟像空间坐标系,根据多光轴平行的测绘相机单中心投影转换公式将除第1个相机单元外的各相机单元的坐标点向虚拟像空间坐标系投影转换得到等效单中心投影的大幅面虚拟影像。Step 4: Taking the image space coordinate system S1 -x1 y1 z1 of the first camera unit as the virtual image space coordinate system, divide the first camera unit according to the single-center projection conversion formula of the multi-optical axis parallel surveying and mapping camera The coordinate points of the external camera units are projected and transformed to the virtual image space coordinate system to obtain a large-scale virtual image equivalent to single-center projection.
上述多光轴平行的测绘相机单中心投影转换方法中,在步骤三中,根据第i个相机单元相对第1个相机单元的线元素和第i个相机单元相对第1个相机单元的角元素得到第i个相机单元相对第1个相机单元的旋转矩阵。In the above-mentioned single-center projection conversion method for a surveying and mapping camera with parallel multiple optical axes, in
上述多光轴平行的测绘相机单中心投影转换方法中,第i个相机单元相对第1个相机单元的旋转矩阵通过以下公式得到:In the above single-center projection conversion method for a surveying and mapping camera with parallel multiple optical axes, the rotation matrix of the i-th camera unit relative to the first camera unit is obtained by the following formula:
其中,ai1、ai2、ai3、bi1、bi2、bi3、ci1、ci2和ci3均为第i个相机单元相对第1个相机单元的旋转矩阵中的系数,ΔXi为第i个相机单元的摄站点在面辅助坐标系O-XYZ中相对第1个相机单元的横坐标,ΔYi为第i个相机单元的摄站点在地面辅助坐标系O-XYZ中相对第1个相机单元的纵坐标,ΔZi为第i个相机单元的摄站点在地面辅助坐标系O-XYZ中相对第1个相机单元的竖坐标,φi为第i个相机单元的像空间坐标系Si-xiyizi相对第1个相机单元的像空间坐标系S1-x1y1z1的x轴向旋转角,ωi为相第i个相机单元的像空间坐标系Si-xiyizi相对第1个相机单元的像空间坐标系S1-x1y1z1的y轴向旋转角,κi为第i个相机单元的像空间坐标系Si-xiyizi相对第1个相机单元的像空间坐标系S1-x1y1z1的z轴向旋转角。Among them, ai1 , ai2 , ai3 , bi1 , bi2 , bi3 , ci1 , ci2 and ci3 are the coefficients in the rotation matrix of the i-th camera unit relative to the first camera unit, ΔXi is the abscissa of the camera point of the i-th camera unit relative to the first camera unit in the surface auxiliary coordinate system O-XYZ, and ΔYi is the relative position of the camera point of the i-th camera unit in the ground auxiliary coordinate system O-XYZ. The ordinate of a camera unit, ΔZi is the vertical coordinate of the camera site of the ith camera unit relative to the first camera unit in the ground auxiliary coordinate system O-XYZ, φi is the image space coordinate of the ith camera unit The x-axis rotation angle of the system Si -xi yi zi relative to the image space coordinate system S1 -x1 y1 z1 of the first camera unit, ωi is the image space coordinate of the i-th camera unit System Si -xi yi zi relative to the y-axis rotation angle of the image space coordinate system S1 -x1 y1 z1 of the first camera unit, κi is the image space coordinate system of the i-th camera unit The z-axis rotation angle of Si -xi yi zi relative to the image space coordinate system S1 -x1 y1 z1 of the first camera unit.
上述多光轴平行的测绘相机单中心投影转换方法中,在步骤四中,多光轴平行的测绘相机单中心投影转换公式为:In the above-mentioned single-center projection conversion method for a surveying and mapping camera with parallel multiple optical axes, in
其中,x1为地面辅助坐标系O-XYZ中地面点A在第1个相机单元的虚拟的理想成像点a1的横坐标,y1为地面辅助坐标系O-XYZ中地面点A在第1个相机单元的虚拟的理想成像点a1的纵坐标,H为第1个相机单元的摄站点S1在地面辅助坐标系O-XYZ中的竖坐标;f1为第1个相机单元的主距,ai1、ai2、ai3、bi1、bi2和bi3均为第i个相机单元相对第1个相机单元的旋转矩阵中的系数,fi为第i个相机单元的主距,xi0为第i个相机单元的主点横坐标,yi0为第i个相机单元的主点纵坐标。Among them, x1 is the abscissa of the virtual ideal imaging point a1 of the ground point A in the first camera unit in the ground auxiliary coordinate system O-XYZ, and y1 is the ground point A in the ground auxiliary coordinate system O-XYZ. The ordinate of the virtual ideal imaging point a1 of one camera unit, H is the vertical coordinate of the imaging point S1 of the first camera unit in the ground auxiliary coordinate system O-XYZ; f1 is the first camera unit's vertical coordinate The principal distance, ai1 , ai2 , ai3 , bi1 , bi2 and bi3 are the coefficients in the rotation matrix of the ith camera unit relative to the first camera unit, and fi is the principal distance of the ith camera unit. distance, xi0 is the abscissa of the principal point of the ith camera unit, and yi0 is the ordinate of the principal point of the ith camera unit.
上述多光轴平行的测绘相机单中心投影转换方法中,还包括如下步骤:计算等效单中心投影的大幅面虚拟影像存在的理论转换误差。In the above-mentioned single-center projection conversion method for a surveying and mapping camera with parallel multiple optical axes, the method further includes the following steps: calculating the theoretical conversion error existing in the large-format virtual image of the equivalent single-center projection.
上述多光轴平行的测绘相机单中心投影转换方法中,转换误差为:In the above-mentioned single-center projection conversion method of a surveying and mapping camera with parallel multiple optical axes, the conversion error is:
其中,Δx为横坐标的转换误差,Δy为纵坐标的转换误差,H0为地面点A的实际高程,f1为第1个相机单元的主距,H为第1个相机单元的摄站点S1在地面辅助坐标系O-XYZ中的竖坐标,ΔXi为第i个相机单元的摄站点在地面辅助坐标系O-XYZ中相对第1个相机单元的横坐标,ΔYi为第i个相机单元的摄站点在地面辅助坐标系O-XYZ中相对第1个相机单元的纵坐标。Among them, Δx is the conversion error of the abscissa, Δy is the conversion error of the ordinate, H0 is the actual elevation of the ground point A, f1 is the principal distance of the first camera unit, and H is the camera site of the first camera unit The vertical coordinate of S1 in the ground auxiliary coordinate system O-XYZ, ΔXi is the abscissa of the camera site of the ith camera unit relative to the first camera unit in the ground auxiliary coordinate system O-XYZ, ΔYi is the ith ith camera unit The ordinate of the camera station of each camera unit relative to the first camera unit in the ground auxiliary coordinate system O-XYZ.
本发明与现有技术相比具有如下有益效果:Compared with the prior art, the present invention has the following beneficial effects:
主光轴平行测绘系统是新型的测绘成像体制,本发明提出一种多光轴平行的测绘相机单中心投影转换方法,从原理层面给出多中心投影图像至单中心投影图像的转换模型,并给出转换误差计算公式,解决了现有的单中心投影转换方法已不能准确描述多光轴平行的测绘相机转换模型的问题,通过光轴平行多面阵拼接实现了大幅面航测相机的核心问题。并且依据本发明可以输出近似单中心投影影像,与现有单中心航测相机输出图像接口相同,实现与现有航测系统处理软件的无缝对接。The main optical axis parallel surveying and mapping system is a new type of surveying and mapping imaging system. The present invention proposes a single-center projection conversion method for a multi-optical axis parallel surveying and mapping camera, and provides a conversion model from a multi-center projection image to a single-center projection image from a principle level, and The conversion error calculation formula is given, which solves the problem that the existing single-center projection conversion method cannot accurately describe the conversion model of the multi-optical axis-parallel surveying and mapping camera. And according to the present invention, an approximate single-center projection image can be output, which is the same as the output image interface of the existing single-center aerial survey camera, and realizes seamless connection with the processing software of the existing aerial survey system.
附图说明Description of drawings
通过阅读下文优选实施方式的详细描述,各种其他的优点和益处对于本领域普通技术人员将变得清楚明了。附图仅用于示出优选实施方式的目的,而并不认为是对本发明的限制。而且在整个附图中,用相同的参考符号表示相同的部件。在附图中:Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are for the purpose of illustrating preferred embodiments only and are not to be considered limiting of the invention. Also, the same components are denoted by the same reference numerals throughout the drawings. In the attached image:
图1是本发明实施例提供的像空间坐标系的示意图;1 is a schematic diagram of an image space coordinate system provided by an embodiment of the present invention;
图2是本发明实施例提供的地面辅助坐标系的示意图;2 is a schematic diagram of a ground-aided coordinate system provided by an embodiment of the present invention;
图3是本发明实施例提供的多中心向单中心投影转换模型的示意图;3 is a schematic diagram of a multi-center to single-center projection conversion model provided by an embodiment of the present invention;
图4是本发明实施例提供的成像系统的原理示意图。FIG. 4 is a schematic diagram of the principle of an imaging system provided by an embodiment of the present invention.
具体实施方式Detailed ways
下面将参照附图更详细地描述本公开的示例性实施例。虽然附图中显示了本公开的示例性实施例,然而应当理解,可以以各种形式实现本公开而不应被这里阐述的实施例所限制。相反,提供这些实施例是为了能够更透彻地理解本公开,并且能够将本公开的范围完整的传达给本领域的技术人员。需要说明的是,在不冲突的情况下,本发明中的实施例及实施例中的特征可以相互组合。下面将参考附图并结合实施例来详细说明本发明。Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be more thoroughly understood, and will fully convey the scope of the present disclosure to those skilled in the art. It should be noted that the embodiments of the present invention and the features of the embodiments may be combined with each other under the condition of no conflict. The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.
本实施例提供了一种多光轴平行的测绘相机单中心投影转换方法,该方法包括如下步骤:This embodiment provides a single-center projection conversion method for a surveying and mapping camera with parallel multiple optical axes, and the method includes the following steps:
步骤一:预设相机有S个相机单元,每个相机单元中的镜头参数相同,且各镜头的主光轴平行;相机有M×N个焦面组件参与拼接,每个焦面组件有效拼接像元数m×n,则每个相机单元的镜头焦面处虚拟焦面的大小为Mm×Nn;其中,M为参与拼接焦面组件阵列的行数,N为参与拼接焦面组件阵列的列数,m为每个焦面组件有效拼接像元数行数,n为每个焦面组件有效拼接像元数列数;Step 1: The preset camera has S camera units, the lens parameters in each camera unit are the same, and the main optical axes of each lens are parallel; the camera has M×N focal plane components participating in the stitching, and each focal plane component is effectively stitched The number of pixels is m×n, then the size of the virtual focal plane at the lens focal plane of each camera unit is Mm×Nn; where M is the number of rows participating in the focal plane assembly array, and N is the number of rows participating in the focal plane assembly array. Number of columns, m is the number of rows and rows of pixels effectively stitched for each focal plane component, and n is the number of columns and columns of effectively stitched pixels for each focal plane component;
以第i个相机单元的镜头的虚拟焦面像素中心为坐标原点,以虚拟焦面像素行方向为u轴、以虚拟焦面像素列方向为v轴,建立第i个相机单元的镜头的焦面坐标系oi-uivi,其中i=1,2,…S;Taking the virtual focal plane pixel center of the lens of the ith camera unit as the coordinate origin, taking the virtual focal plane pixel row direction as the u-axis, and taking the virtual focal plane pixel column direction as the v-axis, establish the focal point of the lens of the ith camera unit. Surface coordinate system oi -ui vi , where i=1,2,...S;
以第i个相机单元的摄站点为坐标原点Si,以平行虚拟焦面像素行方向为x轴,以平行虚拟焦面像素列方向为y轴,按右手坐标系建立第i个相机单元的像空间坐标系Si-xiyizi,其中i=1,2,…S;Taking the camera station of the i-th camera unit as the coordinate origin Si , taking the pixel row direction parallel to the virtual focal plane as the x-axis, and taking the pixel column direction parallel to the virtual focal plane as the y-axis, the right-hand coordinate system is used to establish the ith camera unit. Like the space coordinate system Si -xi yi zi , where i=1,2,...S;
步骤二:建立地面辅助坐标系O-XYZ;其中,以第1个相机单元的摄站点S1在地面的竖直投影点为O点,与第1个相机单元的像空间坐标系的x1轴平行的轴为地面辅助坐标系的X轴,与第1个相机单元的像空间坐标系的y1轴平行的轴为地面辅助坐标系的y轴;如图2所示。Step 2: Establish a ground auxiliary coordinate system O-XYZ; wherein, the vertical projection point of the camera station S1 of the first camera unit on the ground is point O, which is different from the image space coordinate system of the first camera unit x1 The axis parallel to the axis is the X axis of the ground auxiliary coordinate system, and the axis parallel to the y1 axis of the image space coordinate system of the first camera unit is the y axis of the ground auxiliary coordinate system; as shown in Figure 2.
步骤三:根据除第1个相机单元外的各相机单元相对第1个相机单元的线元素和除第1个相机单元外的各相机单元相对第1个相机单元的角元素得到除第1个相机单元外的各相机单元相对第1个相机单元的旋转矩阵;Step 3: According to the line elements of each camera unit except the first camera unit relative to the first camera unit and the corner elements of each camera unit except the first camera unit relative to the first camera unit The rotation matrix of each camera unit outside the camera unit relative to the first camera unit;
步骤四:以第1个相机单元的镜头的像空间坐标系S1-x1y1z1为虚拟像空间坐标系,根据多光轴平行的测绘相机单中心投影转换公式将除第1个相机单元外的各相机单元的坐标点向虚拟像空间坐标系投影转换得到等效单中心投影的大幅面虚拟影像。Step 4: Taking the image space coordinate system S1 -x1 y1 z1 of the lens of the first camera unit as the virtual image space coordinate system, divide the first The coordinate points of each camera unit outside the camera unit are projected and transformed to the virtual image space coordinate system to obtain a large-scale virtual image equivalent to single-center projection.
具体的,第一步,每个焦面像元重新命名。每个相机单元中的镜头焦面处虚拟焦面的大小为Mm×Nn,如图1,第A行B列处焦面,其中点P(i,j)重新命名为P((A-1)m+i,(B-1)n+j)。Specifically, in the first step, each focal plane pixel is renamed. The size of the virtual focal plane at the lens focal plane in each camera unit is Mm×Nn, as shown in Figure 1, the focal plane at row A, column B, where point P(i,j) is renamed P((A-1 )m+i, (B-1)n+j).
以虚拟焦面像素中心为坐标原点,以虚拟焦面像素行方向为u轴、以虚拟焦面像素列方向为v轴,建立焦面坐标系oi-uivi,其中i=1,2,…S。Take the virtual focal plane pixel center as the coordinate origin, take the virtual focal plane pixel row direction as the u-axis, and take the virtual focal plane pixel column direction as the v-axis, establish a focal plane coordinate system oi -ui vi , where i=1, 2,…S.
以摄站点为坐标原点Si,x轴平行虚拟焦面像素行方向,y轴平行虚拟焦面像素列方向,按右手坐标系建立像空间坐标系Si-xiyizi,其中i=1,2,…S。Taking the camera station as the coordinate origin Si , the x-axis is parallel to the pixel row direction of the virtual focal plane, and the y-axis is parallel to the virtual focal plane pixel column direction, and the image space coordinate system Si -xi yi zi is established according to the right-hand coordinate system, where i =1,2,...S.
第二步,第1个相机单元的摄站点为S1,O点为S1在地面辅助坐标系投影,以O为原点建立虚地面辅助坐标系,其X、Y轴与像空间坐标系x、y轴平行,按右手坐标系建立地面辅助坐标系投影O-XYZ。In the second step, the camera station of the first camera unit is S1 , the point O is the projection of S1 on the ground auxiliary coordinate system, and the virtual ground auxiliary coordinate system is established with O as the origin, and its X and Y axes are related to the image space coordinate system x , the y axis is parallel, and the ground auxiliary coordinate system projection O-XYZ is established according to the right-hand coordinate system.
第三步,内方位元素包含主点、主距和畸变参数。测量各相机单元中的内方位元素,主点为(xi0,yi0)、主距fi和畸变参数,其中i=1,2,…S。In the third step, the inner orientation element contains the principal point, principal distance and distortion parameters. Measure the inner orientation elements in each camera unit, the principal points are (xi0 , yi0 ), principal distance fi and distortion parameters, where i=1, 2, . . . S.
第四步,测量各相机单元中的相对外方位元素,各相机单元中的相对外方位元素含相对外方位角元素和外方位线元素。各相机单元相对第1相机单元的线元素为(ΔXi,ΔYi,ΔZi),各相机单元相对第1相机单元的角元素φi、ωi、κi,对应的旋转矩阵为Ri,定义为:The fourth step is to measure the relative outer azimuth elements in each camera unit, and the relative outer azimuth elements in each camera unit include a relative outer azimuth angle element and an outer azimuth line element. The line elements of each camera unit relative to the first camera unit are (ΔXi , ΔYi , ΔZi ), the angle elements φi , ωi , κi of each camera unit relative to the first camera unit, the corresponding rotation matrix is Ri ,defined as:
其中,i=2,3,…S。ai1、ai2、ai3、bi1、bi2、bi3、ci1、ci2和ci3均为第i个相机单元相对第1个相机单元的旋转矩阵中的系数,ΔXi为第i个相机单元的摄站点在地面辅助坐标系O-XYZ中相对第1个相机单元的横坐标,ΔYi为第i个相机单元的摄站点在地面辅助坐标系O-XYZ中相对第1个相机单元的纵坐标,ΔZi为第i个相机单元的摄站点在地面辅助坐标系O-XYZ中相对第1个相机单元的竖坐标,φi为第i个相机单元的像空间坐标系Si-xiyizi相对第1个相机单元的像空间坐标系S1-x1y1z1的x轴向旋转角,ωi为相第i个相机单元的像空间坐标系Si-xiyizi相对第1个相机单元的像空间坐标系S1-x1y1z1的y轴向旋转角,κi为第i个相机单元的像空间坐标系Si-xiyizi相对第1个相机单元的像空间坐标系S1-x1y1z1的z轴向旋转角。where i=2,3,...S. ai1 , ai2 , ai3 , bi1 , bi2 , bi3 , ci1 , ci2 and ci3 are the coefficients in the rotation matrix of the i-th camera unit relative to the first camera unit, and ΔXi is the The camera point of the i camera unit is relative to the abscissa of the first camera unit in the ground auxiliary coordinate system O-XYZ, ΔYi is the camera point of the ith camera unit relative to the first camera unit in the ground auxiliary coordinate system O-XYZ The vertical coordinate of the camera unit, ΔZi is the vertical coordinate of the camera site of the ith camera unit relative to the first camera unit in the ground auxiliary coordinate system O-XYZ, φi is the image space coordinate system S of the ith camera uniti - xi yi zi relative to the image space coordinate system S1 -x1 y1 z1 of the first camera unit x-axis rotation angle, ωi is the image space coordinate system S of the ith camera uniti - xi yi zi relative to the y-axis rotation angle of the image space coordinate system S1 -x1 y1 z1 of the first camera unit, κi is the image space coordinate system Si of the ith camera unit -xi yi zi relative to the z-axis rotation angle of the first camera unit's image space coordinate system S1 -x1 y1 z1 .
第五步以第1个相机单元的像空间坐标系S1-x1y1z1为虚拟像空间坐标系,根据多光轴平行的测绘相机单中心投影转换公式将除第1个相机单元外的各相机单元的坐标点向虚拟像空间坐标系投影转换得到等效单中心投影的大幅面虚拟影像。多光轴平行的测绘相机单中心投影转换公式(6)推导过程如下:The fifth step takes the image space coordinate system S1 -x1 y1 z1 of the first camera unit as the virtual image space coordinate system, and divides the first camera unit according to the single-center projection conversion formula of the multi-optical axis parallel surveying and mapping camera. The coordinate points of the external camera units are projected and transformed to the virtual image space coordinate system to obtain a large-scale virtual image equivalent to single-center projection. The derivation process of the single-center projection conversion formula (6) for a surveying and mapping camera with parallel multiple optical axes is as follows:
如图3,假设地面辅助坐标系O-XYZ中,第1个相机单元的摄站点S1坐标为S1(0,0,H),其中H为S1在地面辅助坐标系O-XYZ中的竖坐标;As shown in Figure 3, it is assumed that in the ground auxiliary coordinate system O-XYZ, the coordinate of the camera station S1 of the first camera unit is S1 (0,0,H), where H is S1 in the ground auxiliary coordinate system O-XYZ the vertical coordinates of ;
由第四步第i个相机单元相对第1个相机单元的线元素为(ΔXi,ΔYi,ΔZi),易得在地面辅助坐标系O-XYZ中,第i个相机单元的摄站点Si坐标为Si(ΔXi,ΔYi,H+ΔZi);From the fourth step, the line element of the ith camera unit relative to the first camera unit is (ΔXi , ΔYi , ΔZi ), it is easy to get the camera point of the ith camera unit in the ground auxiliary coordinate system O-XYZ The coordinate of Si is Si (ΔXi ,ΔYi ,H+ΔZi );
在地面辅助坐标系O-XYZ中地面点A(X,Y,0),其在第i个相机单元中成像点ai(xi,yi,-fi),在第1个相机单元中虚拟成像点a1(x1,y1,-f1)。The ground point A(X,Y,0) in the ground auxiliary coordinate system O-XYZ, which images the point ai (xi ,yi ,-fi ) in the ith camera unit, in the 1th camera unit In the virtual imaging point a1 (x1 , y1 , -f1 ).
地面辅助坐标系O-XYZ中第i个相机单元的摄站点Si、地面辅助坐标系O-XYZ中地面点A(X,Y,0)在第i个相机单元中像空间坐标系Si-xiyizi中实际成像点ai、地面辅助坐标系O-XYZ中地面点A,这三点共线,由共线方程:The camera point Si of the ith camera unit in the ground auxiliary coordinate system O-XYZ, the ground point A(X, Y, 0) in the ground auxiliary coordinate system O-XYZ is in the image space coordinate system Si of the ith camera unit The actual imaging point ai in -xi yi zi and the ground point A in the ground auxiliary coordinate system O-XYZ, these three points are collinear, according to the collinear equation:
xi为ai成像点的横坐标,yi为ai成像点的纵坐标,xi0为第i个相机单元的主点横坐标,yi0为第i个相机单元的主点纵坐标,fi为第i个相机单元的主距,为式(1)中旋转矩阵,是过渡矩阵。xi is the abscissa of the imaging point of ai , yi is the ordinate of the imaging point of ai , xi0 is the abscissa of the principal point of the ith camera unit, yi0 is the ordinate of the principal point of the ith camera unit, fi is the principal distance of the i-th camera unit, is the rotation matrix in formula (1), is the transition matrix.
可得,Available,
地面辅助坐标系O-XYZ中第1个相机单元的摄站点S1、地面辅助坐标系O-XYZ中地面点A(X,Y,0)在第1个相机单元中虚拟的理想成像点a1、地面辅助坐标系O-XYZ中地面点A,这三点共线,由共线方程:The imaging point S1 of the first camera unit in the ground auxiliary coordinate system O-XYZ, and the virtual ideal imaging point a in the first camera unit of the ground point A(X, Y, 0) in the ground auxiliary coordinate system O-XYZ1. The ground point A in the ground auxiliary coordinate system O-XYZ, these three points are collinear, according to the collinear equation:
x1为a1成像点的横坐标,y1为a1成像点的纵坐标,f1为相机1的主距。x1 is the abscissa of the imaging point of a1 , y1 is the ordinate of the imaging point of a1 , and f1 is the principal distance of the
将公式(4)带人公式(5)有Bringing formula (4) to formula (5) has
公式(6)为多光轴平行的测绘相机单中心投影转换公式。The formula (6) is the single-center projection conversion formula of the multi-optical axis-parallel mapping camera.
第六步,为实现拼接构建近似单中心的虚拟影像,公式(6)将地面假设为一个平面进行纠正。实际地面的起伏会造成转换中拼接的理论误差,如图3,假设A点的实际高程为H0,实际工程实现中相对H和H-H0来说ΔZi十分微小,令H+ΔZi≈H,H-H0+ΔZi≈H-H0可得误差为In the sixth step, in order to realize the splicing to construct a virtual image with an approximate single center, the formula (6) assumes the ground to be a plane for correction. The fluctuation of the actual ground will cause the theoretical error of splicing in the conversion, as shown in Figure 3. Assuming that the actual elevation of point A is H0 , ΔZi is very small compared to H and HH0 in actual engineering implementation, so that H+ΔZi ≈ H , HH0 +ΔZi ≈HH0 , the error can be obtained as
实施例中一种多光轴平行的测绘相机单中心投影转换方法,对应成像系统4个镜头光轴平行,每个镜头对应一组焦面,即有2×2个焦面组件参与拼接,每个焦面组件有效拼接像元数5120×5120,拼接后总有效拼接像元数2×5120×2×5120。具体步骤如下:In the embodiment, a single-center projection conversion method for a surveying and mapping camera with parallel multiple optical axes corresponds to the parallel optical axes of the four lenses of the imaging system, and each lens corresponds to a group of focal planes, that is, 2×2 focal plane components participate in splicing, and each lens is spliced. The number of effectively stitched pixels for each focal plane component is 5120×5120, and the total number of effective stitched pixels after stitching is 2×5120×2×5120. Specific steps are as follows:
第一步,每个焦面处像元重新命名。每个镜头焦面处虚拟焦面的大小为2×5120×2×5120,如图4,第2行2列处焦面,其中点P(i,j)重新命名为P((2-1)×5120+i,(2-1)×5120+j)。In the first step, the pixels at each focal plane are renamed. The size of the virtual focal plane at the focal plane of each lens is 2×5120×2×5120, as shown in Figure 4, the focal plane at
以焦面像素中心为坐标原点,像素排列方向分别为u、v轴,建立焦面坐标系oi-uivi,其中i=1,2,…4。Taking the center of the focal plane pixel as the coordinate origin, and the pixel arrangement directions are the u and v axes respectively, a focal plane coordinate system oi -ui vi is established, where i=1, 2, . . . 4.
以摄站点为坐标原点Si,像素排列方向分别为x、y轴,按右手坐标系建立像空间坐标系Si-xiyizi,其中i=1,2,…4。Taking the camera station as the coordinate origin Si , and the pixel arrangement directions are the x and y axes respectively, the image space coordinate system Si -xi yi zii is established according to the right-handed coordinate system, where i=1, 2, . . . 4.
第二步,O点为S1在地面辅助坐标系投影,以O为原点建立虚地面辅助坐标系,其X、Y轴与像空间坐标系x、y轴平行,按右手坐标系建立地面辅助坐标系投影O-XYZ。In the second step, the O point is the projection of S1 on the ground auxiliary coordinate system, and the virtual ground auxiliary coordinate system is established with O as the origin, and its X and Y axes are parallel to the x and y axes of the image space coordinate system. Coordinate system projection O-XYZ.
第三步,测量各相机内方位元素,主点为(xi0,yi0)、主距fi和畸变参数,其中i=1,2,…4。The third step is to measure the azimuth elements in each camera, the principal points are (xi0 , yi0 ), principal distance fi and distortion parameters, where i=1, 2, . . . 4.
第四步,测量各相机相对外方位元素,各相机相对外方位元素含相对外方位角元素和外方位线元素。各相机单元相对第1个相机单元的线元素为(ΔXi,ΔYi,ΔZi),各相机单元相对第1个相机单元的角元素φi、ωi、κi,对应的旋转矩阵为Ri,定义为:The fourth step is to measure the relative outer azimuth elements of each camera, and the relative outer azimuth elements of each camera include a relative outer azimuth angle element and an outer azimuth line element. The line elements of each camera unit relative to the first camera unit are (ΔXi , ΔYi , ΔZi ), and the angle elements φi , ωi , κi of each camera unit relative to the first camera unit, the corresponding rotation matrix is Ri , defined as:
其中,i=2,3,…4。where i=2,3,...4.
第五步,第1个相机单元的像空间坐标系为虚拟像空间坐标系,其它相机单元向第1个相机单元转换的公式推导。如图3,假设摄站点S1(0,0,H),由第二步可知Si(ΔXi,ΔYi,H+ΔZi),地面点A(X,Y,0),其在第i个相机单元中成像点ai(xi,yi,-fi),在第1个相机单元中虚拟成像点a1(x1,y1,-f1)。In the fifth step, the image space coordinate system of the first camera unit is a virtual image space coordinate system, and the formula for converting other camera units to the first camera unit is derived. As shown in Figure 3, assuming the camera site S1 (0,0,H), from the second step we can know that Si (ΔXi ,ΔYi ,H+ΔZi ), the ground point A(X,Y,0), which is in Image point ai (xi , yi , -fi ) in the ith camera unit, and virtual image point a1 (x1 , y1 , -f1 ) in the first camera unit.
Si、a、A共线,由共线方程:Si , a, and A are collinear by the collinear equation:
其中是过渡矩阵。in is the transition matrix.
可得,Available,
S1、a1、A共线,由共线方程:S1 , a1 , and A are collinear, according to the collinear equation:
将上述两个公式得到多光轴平行的测绘相机单中心投影转换公式The above two formulas are used to obtain the single-center projection conversion formula of the multi-optical axis-parallel mapping camera
多光轴平行的测绘相机单中心投影转换公式为多光轴平行的测绘相机单中心投影转换公式。多光轴平行测绘相机单中心投影转换最终投影变换至虚拟像平面S1-x1y1的基准内,通过拼接转换生成等效单中心投影的大幅面虚拟影像。The single-center projection conversion formula of a surveying and mapping camera with parallel multiple optical axes is the single-center projection conversion formula of a surveying and mapping camera with parallel multiple optical axes. The single-center projection transformation of the multi-optical-axis parallel mapping camera is finally transformed into the virtual image plane S1 -x1 y1 reference, and a large-scale virtual image equivalent to single-center projection is generated through splicing and transformation.
第六步,为实现拼接构建近似单中心的虚拟影像,多光轴平行的测绘相机单中心投影转换公式将地面假设为一个平面进行纠正。实际地面的起伏会造成转换中拼接的误差,如图3,假设A点的实际高程为H0,实际工程实现中相对H和H-H0来说ΔZi十分微小,令H+ΔZi≈H,H-H0+ΔZi≈H-H0可得误差为In the sixth step, in order to realize the splicing and constructing a virtual image with an approximate single center, the single-center projection conversion formula of the multi-optical axis-parallel surveying and mapping camera assumes the ground to be a plane for correction. The fluctuation of the actual ground will cause the error of splicing in the conversion, as shown in Figure 3. Assuming that the actual elevation of point A is H0 , ΔZi is very small compared to H and HH0 in the actual engineering implementation, so that H+ΔZi ≈ H, HH0 +ΔZi ≈HH0 , the error can be obtained as
主光轴平行测绘系统是新型的测绘成像体制,本实施例提出一种多光轴平行的测绘相机单中心投影转换方法,从原理层面给出多投影图像至单中心投影图像的转换模型,并给出转换误差计算公式,解决了现有的单中心投影转换方法已不能准确描述多光轴平行的测绘相机转换模型的问题,通过光轴平行多面阵拼接实现了大幅面航测相机的核心问题。并且依据本发明可以输出近似单中心投影影像,与现有单中心航测相机输出图像接口相同,实现与现有航测系统处理软件的无缝对接。The main optical axis parallel surveying and mapping system is a new type of surveying and mapping imaging system. This embodiment proposes a single-center projection conversion method for a multi-optical axis-parallel surveying and mapping camera, and provides a conversion model from a multi-projection image to a single-center projection image from the principle level. The conversion error calculation formula is given, which solves the problem that the existing single-center projection conversion method cannot accurately describe the conversion model of the multi-optical axis-parallel surveying and mapping camera. And according to the present invention, an approximate single-center projection image can be output, which is the same as the output image interface of the existing single-center aerial survey camera, and realizes seamless connection with the processing software of the existing aerial survey system.
本发明虽然已以较佳实施例公开如上,但其并不是用来限定本发明,任何本领域技术人员在不脱离本发明的精神和范围内,都可以利用上述揭示的方法和技术内容对本发明技术方案做出可能的变动和修改,因此,凡是未脱离本发明技术方案的内容,依据本发明的技术实质对以上实施例所作的任何简单修改、等同变化及修饰,均属于本发明技术方案的保护范围。Although the present invention has been disclosed above with preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can use the methods and technical contents disclosed above to improve the present invention without departing from the spirit and scope of the present invention. The technical solutions are subject to possible changes and modifications. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solutions of the present invention belong to the technical solutions of the present invention. protected range.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010724024.7ACN112066950B (en) | 2020-07-24 | 2020-07-24 | Multi-optical-axis parallel mapping camera single-center projection conversion method |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010724024.7ACN112066950B (en) | 2020-07-24 | 2020-07-24 | Multi-optical-axis parallel mapping camera single-center projection conversion method |
| Publication Number | Publication Date |
|---|---|
| CN112066950A CN112066950A (en) | 2020-12-11 |
| CN112066950Btrue CN112066950B (en) | 2022-10-14 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010724024.7AActiveCN112066950B (en) | 2020-07-24 | 2020-07-24 | Multi-optical-axis parallel mapping camera single-center projection conversion method |
| Country | Link |
|---|---|
| CN (1) | CN112066950B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2000136905A (en)* | 1998-11-02 | 2000-05-16 | Nissan Motor Co Ltd | Object edge position measuring device and moving object traffic judging device |
| CN102042825A (en)* | 2010-11-09 | 2011-05-04 | 青岛市光电工程技术研究院 | Three-dimensional imaging measurement system combining planar array imaging with laser scanning |
| CN102313536A (en)* | 2011-07-21 | 2012-01-11 | 清华大学 | Method for barrier perception based on airborne binocular vision |
| CN102538763A (en)* | 2012-02-14 | 2012-07-04 | 清华大学 | Method for measuring three-dimensional terrain in river model test |
| CN108827245A (en)* | 2018-05-17 | 2018-11-16 | 北京林业大学 | A kind of twin-lens portable forestry investigation apparatus |
| CN110836662A (en)* | 2019-11-04 | 2020-02-25 | 南京理工大学 | Slope displacement monitoring method based on relative orientation and absolute orientation algorithm |
| CN111339826A (en)* | 2020-05-06 | 2020-06-26 | 山西大学 | A Linear Sensor Network Framework Detection System for Landslide UAV |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPS59182688A (en)* | 1983-03-31 | 1984-10-17 | Toshiba Corp | Stereo vision processing device |
| JP5311365B2 (en)* | 2007-10-23 | 2013-10-09 | 独立行政法人産業技術総合研究所 | Stereo camera calibration method and calibration system |
| CN101441076B (en)* | 2008-12-29 | 2010-06-02 | 东软集团股份有限公司 | Method and device for detecting barrier |
| CN102062599B (en)* | 2010-11-23 | 2012-09-26 | 中国科学院遥感应用研究所 | Spliced imaging system based on axis-shifting principle |
| JP5787695B2 (en)* | 2011-09-28 | 2015-09-30 | 株式会社トプコン | Image acquisition device |
| JP6151902B2 (en)* | 2012-09-20 | 2017-06-21 | 株式会社トプコン | Camera for photo measurement and aerial photographic equipment |
| CN103115613B (en)* | 2013-02-04 | 2015-04-08 | 安徽大学 | Three-dimensional space positioning method |
| CN103245322B (en)* | 2013-04-10 | 2015-11-11 | 南京航空航天大学 | A kind of distance-finding method based on binocular stereo vision and system |
| CN103323028B (en)* | 2013-06-14 | 2015-10-21 | 武汉大学 | One locates conforming satellite multispectral image method for registering based on object space |
| CN105627926B (en)* | 2016-01-22 | 2017-02-08 | 尹兴 | Four-camera group planar array feature point three-dimensional measurement system and measurement method |
| CN108627142B (en)* | 2018-05-02 | 2020-07-17 | 成都纵横自动化技术股份有限公司 | Target positioning method combining offline elevation and airborne photoelectric pod |
| CN108955642B (en)* | 2018-05-07 | 2020-09-01 | 江苏师范大学 | A seamless stitching method for large-format equivalent central projection images |
| CN109668525B (en)* | 2019-01-30 | 2020-08-07 | 哈尔滨超精密装备工程技术中心有限公司 | High-precision three-dimensional angle measuring method and device based on reflection grating |
| CN109931906B (en)* | 2019-03-28 | 2021-02-23 | 华雁智科(杭州)信息技术有限公司 | Camera ranging method and device and electronic equipment |
| CN112082571B (en)* | 2020-07-24 | 2022-09-23 | 北京空间机电研究所 | A large-format surveying and mapping camera system and calibration method |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2000136905A (en)* | 1998-11-02 | 2000-05-16 | Nissan Motor Co Ltd | Object edge position measuring device and moving object traffic judging device |
| CN102042825A (en)* | 2010-11-09 | 2011-05-04 | 青岛市光电工程技术研究院 | Three-dimensional imaging measurement system combining planar array imaging with laser scanning |
| CN102313536A (en)* | 2011-07-21 | 2012-01-11 | 清华大学 | Method for barrier perception based on airborne binocular vision |
| CN102538763A (en)* | 2012-02-14 | 2012-07-04 | 清华大学 | Method for measuring three-dimensional terrain in river model test |
| CN108827245A (en)* | 2018-05-17 | 2018-11-16 | 北京林业大学 | A kind of twin-lens portable forestry investigation apparatus |
| CN110836662A (en)* | 2019-11-04 | 2020-02-25 | 南京理工大学 | Slope displacement monitoring method based on relative orientation and absolute orientation algorithm |
| CN111339826A (en)* | 2020-05-06 | 2020-06-26 | 山西大学 | A Linear Sensor Network Framework Detection System for Landslide UAV |
| Title |
|---|
| 一种多视点平行阵列摄像机标定方法;赵琨等;《信息技术》;20161025(第10期);178-183* |
| Publication number | Publication date |
|---|---|
| CN112066950A (en) | 2020-12-11 |
| Publication | Publication Date | Title |
|---|---|---|
| CN116051659B (en) | A joint calibration method of line scan camera and 2D laser scanner | |
| CN103646394B (en) | A kind of mixing vision system calibration method based on Kinect video camera | |
| CN101852623B (en) | On-track calibration method for internal element of satellite optical remote sensing camera | |
| CN110398208A (en) | Big data deformation monitoring method based on photographic total station system | |
| CN103268610A (en) | A Unified Model of Catadioptric Omnidirectional Camera and Its Calibration Method | |
| CN107967700B (en) | On-orbit geometric correction and precision verification method for large-view-field wide-working-distance binocular camera | |
| CN104778656A (en) | Fisheye image correction method on basis of spherical perspective projection | |
| CN117308889B (en) | High-precision astronomical measurement method by jointly splicing all chip units of CCD | |
| CN101882306A (en) | A High-Precision Stitching Method for Photos of Concave-Convex Objects | |
| CN104320565B (en) | A kind of more shot detection device array curved surface image planes joining methods | |
| CN103869595A (en) | Focal plane adjustment method for off-axis three-lens camera | |
| CN101692447B (en) | Multi-CCD super field of view image mosaic photoelectric system | |
| CN108269234A (en) | A kind of lens of panoramic camera Attitude estimation method and panorama camera | |
| CN118623782A (en) | A slope deformation monitoring method, device, electronic equipment and medium | |
| CN101598550B (en) | Rapid rearrangement method of satellite stereoscopic image approximate epipolar line based on projection trajectory | |
| CN101545775A (en) | Method for calculating orientation elements of photo and the height of building by utilizing digital map | |
| CN103778610A (en) | Geometric pretreatment method for vertical rail swing images of satellite-borne linear array sensor | |
| CN102589529A (en) | Scanning close-range photogrammetry method | |
| CN116576827A (en) | Remote sensing terrain three-dimensional mapping method based on polarization three-dimensional imaging | |
| CN108955642A (en) | A kind of large format equivalent center projection image seamless joint method | |
| CN103148839B (en) | Lens light splitting mode-based focal plane splicing aerial surveying camera with extra large plane | |
| CN112066950B (en) | Multi-optical-axis parallel mapping camera single-center projection conversion method | |
| CN118898541A (en) | A method and system for stitching the internal field of view of a high-orbit multi-array camera with consistent object space | |
| CN106066207A (en) | A kind of parallel light path combination type multi-source information acquiring processing means and method | |
| CN109696155A (en) | The weak intersection optical satellite image simultaneous adjustment method and system of light coplanar constraint |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |