CN109427199B - Augmented reality method and device for assisted driving - Google Patents

Abstract

The embodiment of the invention provides a method and a device for augmented reality for driving assistance, which are applied to the technical field of augmented reality, and the method comprises the following steps: the driving assistance information is determined based on the information acquired in the driving process, and then the virtual three-dimensional display information corresponding to the driving assistance information is displayed.

Description

Translated fromChinese

用于辅助驾驶的增强现实的方法及装置Augmented reality method and device for assisted driving

技术领域technical field

本发明涉及增强现实技术领域，具体而言，本发明涉及一种用于辅助驾驶的增强现实的方法及装置。The present invention relates to the technical field of augmented reality, in particular, the present invention relates to an augmented reality method and device for assisting driving.

背景技术Background technique

AR(Augmented Reality，增强现实)技术可以将虚拟物体和/或虚拟信息叠加到真实场景中，从而使得用户获得超过现实的感官体验，即用户可以感知到一个同时存在真实物体和虚拟物体和/或虚拟信息的场景。AR (Augmented Reality, Augmented Reality) technology can superimpose virtual objects and/or virtual information on real scenes, so that users can obtain a sensory experience beyond reality, that is, users can perceive a real object and virtual objects and/or The scene of virtual information.

在驾驶车辆的过程中，由于路况复杂以及驾驶人自身的一些限制，驾驶人难以完全掌握车辆行驶过程中的行车信息，从而可能导致事故的发生。通过将AR技术应用于车辆驾驶，可以帮助驾驶人更好的掌握车辆行驶过程中的行车信息，从而更安全的驾驶机动车辆，减少车辆行驶过程中事故的发生。因此如何在驾驶人驾驶车辆过程中应用AR技术成为一个关键问题。In the process of driving a vehicle, due to complex road conditions and some limitations of the driver himself, it is difficult for the driver to fully grasp the driving information during the driving process of the vehicle, which may lead to accidents. By applying AR technology to vehicle driving, it can help drivers better grasp the driving information during vehicle driving, so as to drive motor vehicles more safely and reduce the occurrence of accidents during vehicle driving. Therefore, how to apply AR technology in the process of driving a vehicle has become a key issue.

发明内容Contents of the invention

为克服上述技术问题或者至少部分地解决上述技术问题，特提出以下技术方案：In order to overcome the above-mentioned technical problems or at least partially solve the above-mentioned technical problems, the following technical solutions are proposed:

本发明的实施例根据一个方面，提供了一种用于辅助驾驶的增强现实的方法，包括：According to one aspect, an embodiment of the present invention provides an augmented reality method for assisting driving, including:

基于行车过程中获取的信息来确定行车辅助信息；Determining driving assistance information based on information acquired during driving;

显示所述行车辅助信息对应的虚拟三维显示信息。and displaying virtual three-dimensional display information corresponding to the driving assistance information.

本发明的实施例根据另一个方面，还提供了一种用于辅助驾驶的增强现实的装置，包括：According to another aspect, the embodiment of the present invention also provides an augmented reality device for assisting driving, including:

确定模块，用于基于行车过程中获取的信息来确定行车辅助信息；A determining module, configured to determine driving assistance information based on information acquired during driving;

显示模块，用于显示所述确定模块确定的行车辅助信息对应的虚拟三维显示信息。A display module, configured to display virtual three-dimensional display information corresponding to the driving assistance information determined by the determination module.

本发明提供了一种用于辅助驾驶的增强现实的方法及装置，与现有技术相比，本发明基于行车过程中获取的信息来确定行车辅助信息，然后显示行车辅助信息对应的虚拟三维显示信息，即本发明通过车辆行驶过程中获取到的信息，确定出行车过程中的行车辅助信息，并将行车过程中的行车辅助信息对应的虚拟三维显示信息，通过视觉和/或听觉的方式呈现给驾驶人，以通知或者警示驾驶人，从而可以实现在车辆行驶过程中应用增强现实技术帮助驾驶人更好的掌握车辆行驶过程中的行车信息，进而可以提升用户体验。The present invention provides an augmented reality method and device for assisting driving. Compared with the prior art, the present invention determines the driving assistance information based on the information obtained during driving, and then displays the virtual three-dimensional display corresponding to the driving assistance information. Information, that is, the present invention determines the driving auxiliary information during the driving process through the information obtained during the driving process of the vehicle, and presents the virtual three-dimensional display information corresponding to the driving auxiliary information during the driving process through visual and/or auditory means To the driver, to notify or warn the driver, so that the application of augmented reality technology in the driving process of the vehicle can help the driver better grasp the driving information during the driving process of the vehicle, thereby improving the user experience.

本发明附加的方面和优点将在下面的描述中部分给出，这些将从下面的描述中变得明显，或通过本发明的实践了解到。Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description, or may be learned by practice of the invention.

附图说明Description of drawings

本发明上述的和/或附加的方面和优点从下面结合附图对实施例的描述中将变得明显和容易理解，其中：The above and/or additional aspects and advantages of the present invention will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

图1为本发明实施例的一种用于辅助驾驶的增强现实的方法流程示意图；FIG. 1 is a schematic flowchart of an augmented reality method for driving assistance according to an embodiment of the present invention;

图2为本发明实施例中当道路表面未被完全覆盖，确定道路信息的示意图；Fig. 2 is a schematic diagram of determining road information when the road surface is not completely covered in an embodiment of the present invention;

图3为本发明实施例中当道路表面被完全覆盖，且中间车道隔离栏杆可见时，确定道路信息的示意图；Fig. 3 is a schematic diagram of determining road information when the road surface is completely covered and the separation railing of the middle lane is visible in an embodiment of the present invention;

图4为本发明实施例中道路被完全覆盖，无法区分道路表面和道路边缘的示意图；Fig. 4 is the schematic diagram that road is covered completely in the embodiment of the present invention, can't distinguish road surface and road edge;

图5为本发明实施例中当道路表面被完全覆盖，且中间车道隔离栏杆不可见时，确定道路信息的示意图；Fig. 5 is a schematic diagram of determining road information when the road surface is completely covered and the middle lane separation railing is not visible in an embodiment of the present invention;

图6为本发明实施例中显示车辙的示意图；Fig. 6 is a schematic diagram showing ruts in an embodiment of the present invention;

图7为本发明实施例中当车辙不清楚时，增强显示车辙的示意图；Fig. 7 is a schematic diagram of enhanced display of ruts when the ruts are not clear in the embodiment of the present invention;

图8为本发明实施例中显示AR信息与驾驶人视线的关系示意图；Fig. 8 is a schematic diagram showing the relationship between AR information and the driver's line of sight in an embodiment of the present invention;

图9为本发明实施例中当交通标志/指示标志被部分或者全部覆盖时，显示完整的交通标志/指示标志的示意图；FIG. 9 is a schematic diagram showing a complete traffic sign/indicative sign when the traffic sign/indicative sign is partially or fully covered in an embodiment of the present invention;

图10为本发明实施例中根据历史记录，确定当前位置对应的交通标志和/或指示标志的示意图；Fig. 10 is a schematic diagram of determining the traffic sign and/or indicator sign corresponding to the current location according to the historical records in the embodiment of the present invention;

图11为本发明实施例中确定侧方后视镜扩展区域的示意图；Fig. 11 is a schematic diagram of determining the expansion area of the side rearview mirror in an embodiment of the present invention;

图12为本发明实施例中物理的侧方后视镜的可视区域以及侧方后视镜扩展区域的示意图；12 is a schematic diagram of the visible area of the physical side rearview mirror and the expansion area of the side rearview mirror in an embodiment of the present invention;

图13为本发明实施例中内视镜扩展区域示意图；Fig. 13 is a schematic diagram of the expansion area of the endoscope in the embodiment of the present invention;

图14为本发明实施例中显示虚拟交通灯的示意图；Fig. 14 is a schematic diagram showing virtual traffic lights in an embodiment of the present invention;

图15为本发明实施例中根据交通警察手势，显示对应的AR信息的示意图；Fig. 15 is a schematic diagram of displaying corresponding AR information according to a traffic policeman's gesture in an embodiment of the present invention;

图16为本发明实施例中操作键盘所需按键的AR信息显示方式示意图；Fig. 16 is a schematic diagram of the AR information display mode of the keys required to operate the keyboard in the embodiment of the present invention;

图17为本发明实施例中对适合停车和不适合停车的区域进行显示并显示相应的增强现实辅助驾驶显示信息的示意图；Fig. 17 is a schematic diagram of displaying areas suitable for parking and unsuitable for parking and displaying corresponding augmented reality assisted driving display information in an embodiment of the present invention;

图18为本发明实施例中设备预先准备和渲染较大范围AR信息，以降低延迟的示意图；Fig. 18 is a schematic diagram of the device pre-preparing and rendering large-scale AR information in order to reduce delay in the embodiment of the present invention;

图19为本发明实施例中由于车速不同，行车区域显示方式不同的示意图；Fig. 19 is a schematic diagram of different display modes of driving areas due to different vehicle speeds in the embodiment of the present invention;

图20为本发明实施例中当同时存在多个AR信息需要显示时，多个AR信息显示方式示意图；Fig. 20 is a schematic diagram of a display mode of multiple AR information when multiple AR information needs to be displayed at the same time in the embodiment of the present invention;

图21为本发明实施例中当驾驶人的视线统计显示驾驶人的注意力偏向左侧时，设备将AR信息显示在右侧的示意图；Fig. 21 is a schematic diagram of the device displaying AR information on the right side when the driver's sight statistics show that the driver's attention is biased to the left in the embodiment of the present invention;

图22为本发明实施例中的一种用于辅助驾驶的增强现实的装置结构示意图。Fig. 22 is a schematic structural diagram of an augmented reality device for assisting driving in an embodiment of the present invention.

具体实施方式Detailed ways

下面详细描述本发明的实施例，所述实施例的示例在附图中示出，其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的，仅用于解释本发明，而不能解释为对本发明的限制。Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

本技术领域技术人员可以理解，除非特意声明，这里使用的单数形式“一”、“一个”、“单个”、“所述”和“该”也可包括复数形式。应该进一步理解的是，本发明的说明书中使用的措辞“包括”是指存在所述特征、整数、步骤、操作、元件和/或组件，但是并不排除存在或添加一个或多个其他特征、整数、步骤、操作、元件、组件和/或它们的组。应该理解，当我们称元件被“连接”或“耦接”到另一元件时，它可以直接连接或耦接到其他元件，或者也可以存在中间元件。此外，这里使用的“连接”或“耦接”可以包括无线连接或无线耦接。这里使用的措辞“和/或”包括一个或更多个相关联的列出项的全部或任一单元和全部组合。Those skilled in the art will understand that the singular forms "a", "an", "single", "said" and "the" used herein may also include plural forms unless otherwise stated. It should be further understood that the word "comprising" used in the description of the present invention refers to the presence of said features, integers, steps, operations, elements and/or components, but does not exclude the presence or addition of one or more other features, Integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Additionally, "connected" or "coupled" as used herein may include wireless connection or wireless coupling. The expression "and/or" used herein includes all or any elements and all combinations of one or more associated listed items.

本技术领域技术人员可以理解，除非另外定义，这里使用的所有术语(包括技术术语和科学术语)，具有与本发明所属领域中的普通技术人员的一般理解相同的意义。还应该理解的是，诸如通用字典中定义的那些术语，应该被理解为具有与现有技术的上下文中的意义一致的意义，并且除非像这里一样被特定定义，否则不会用理想化或过于正式的含义来解释。Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this invention belongs. It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood to have meanings consistent with their meaning in the context of the prior art, and unless specifically defined as herein, are not intended to be idealized or overly Formal meaning to explain.

本技术领域技术人员可以理解，这里所使用的“终端”、“终端设备”既包括无线信号接收器的设备，其仅具备无发射能力的无线信号接收器的设备，又包括接收和发射硬件的设备，其具有能够在双向通信链路上，进行双向通信的接收和发射硬件的设备。这种设备可以包括：蜂窝或其他通信设备，其具有单线路显示器或多线路显示器或没有多线路显示器的蜂窝或其他通信设备；PCS(Personal Communications Service，个人通信系统)，其可以组合语音、数据处理、传真和/或数据通信能力；PDA(Personal Digital Assistant，个人数字助理)，其可以包括射频接收器、寻呼机、互联网/内联网访问、网络浏览器、记事本、日历和/或GPS(Global Positioning System，全球定位系统)接收器；常规膝上型和/或掌上型计算机或其他设备，其具有和/或包括射频接收器的常规膝上型和/或掌上型计算机或其他设备。这里所使用的“终端”、“终端设备”可以是便携式、可运输、安装在交通工具(航空、海运和/或陆地)中的，或者适合于和/或配置为在本地运行，和/或以分布形式，运行在地球和/或空间的任何其他位置运行。这里所使用的“终端”、“终端设备”还可以是通信终端、上网终端、音乐/视频播放终端，例如可以是PDA、MID(Mobile Internet Device，移动互联网设备)和/或具有音乐/视频播放功能的移动电话，也可以是智能电视、机顶盒等设备。Those skilled in the art can understand that the "terminal" and "terminal equipment" used here not only include wireless signal receiver equipment, which only has wireless signal receiver equipment without transmission capabilities, but also include receiving and transmitting hardware. A device having receive and transmit hardware capable of bi-directional communication over a bi-directional communication link. Such equipment may include: cellular or other communication equipment, which has a single-line display or a multi-line display or a cellular or other communication equipment without a multi-line display; PCS (Personal Communications Service, personal communication system), which can combine voice, data Processing, facsimile and/or data communication capabilities; PDA (Personal Digital Assistant, Personal Digital Assistant), which may include radio frequency receiver, pager, Internet/Intranet access, web browser, notepad, calendar and/or GPS (Global Positioning System (Global Positioning System) receiver; a conventional laptop and/or palmtop computer or other device having and/or including a radio frequency receiver. As used herein, a "terminal", "terminal device" may be portable, transportable, installed in a vehicle (air, sea, and/or land), or adapted and/or configured to operate locally, and/or In distributed form, the operation operates at any other location on Earth and/or in space. The "terminal" and "terminal equipment" used here can also be communication terminals, Internet terminals, music/video playback terminals, such as PDAs, MIDs (Mobile Internet Devices, mobile Internet devices) and/or with music/video playback terminals. Functional mobile phones, smart TVs, set-top boxes and other devices.

ADAS(Advanced Driver Assistance System，高级驾驶人辅助系统)旨在帮助驾驶人安全的驾驶机动车辆，减少事故发生，ADAS可以基于路况通过对驾驶人用视觉、听觉或者触觉的方式进行反馈，以通知或警示驾驶人，可以包括但不限于车道偏离预警，车道保持系统等。ADAS (Advanced Driver Assistance System, Advanced Driver Assistance System) is designed to help drivers drive motor vehicles safely and reduce accidents. ADAS can provide visual, auditory or tactile feedback to drivers based on road conditions to notify or Alerting the driver may include but not limited to lane departure warning, lane keeping system, etc.

现有的ADAS主要致力于在路面条件良好的条件下为驾驶人提供帮助，但对于挑战性的环境，如积雪路面和泥泞路面等，缺少有效的解决方案。Existing ADAS is mainly focused on assisting drivers in good road conditions, but there is a lack of effective solutions for challenging environments, such as snowy roads and muddy roads.

现有的采用增强现实技术的高级驾驶人辅助系统使用的通常是车载的单独屏幕，且屏幕比较小，显示的信息类型固定，驾驶人观感不自然。且延迟比较大，在比较有挑战性的驾驶场景无法有效的帮助驾驶人。因此如何为驾驶人自适应的选择要显示的物体/信息，如何以观感自然的方式呈现，如何低延迟的显示信息，以及如何同时显示多个物体/信息，是需要解决的问题。Existing advanced driver assistance systems using augmented reality technology usually use a separate screen in the vehicle, and the screen is relatively small, the type of information displayed is fixed, and the driver's perception is unnatural. And the delay is relatively large, and it cannot effectively help the driver in more challenging driving scenarios. Therefore, how to adaptively select the object/information to be displayed for the driver, how to present it in a natural way, how to display information with low delay, and how to display multiple objects/information at the same time are problems that need to be solved.

对于本发明实施例，对于挑战性的环境，如积雪路面和泥泞路面等，本发明实施例通过感知道路环境和/或道路地图信息可以估计出包括车道范围、车道线位置、道路边缘线位置、路面交通标志和非路面交通标志至少之一，并生成相应的AR信息(即虚拟三维显示信息)。For the embodiments of the present invention, for challenging environments, such as snow-covered roads and muddy roads, etc., the embodiments of the present invention can estimate the range of lanes, the position of lane lines, and the position of road edge lines by sensing the road environment and/or road map information. , at least one of road traffic signs and non-road traffic signs, and generate corresponding AR information (that is, virtual three-dimensional display information).

其中，对于感知道路环境，设备可以通过以下方式至少之一感知道路环境：使用设备所承载的至少一个传感器感知；使用本车所载的至少一个传感器感知；使用通信方式从其他至少一个：同类设备、不同类设备、其他车辆获得信息；使用GPS(Global PositioningSystem，全球定位系统)获得信息。Among them, for sensing the road environment, the device can perceive the road environment in at least one of the following ways: use at least one sensor carried by the device to sense; use at least one sensor carried by the vehicle to sense; use at least one other communication method: similar devices , different types of equipment, and other vehicles to obtain information; use GPS (Global Positioning System, Global Positioning System) to obtain information.

进一步的，设备的可感知区域可以为使用以上所述各种方式感知范围的集合。Further, the perceivable area of the device may be a collection of perceivable ranges using the above-mentioned various methods.

对于本发明实施例，AR信息(即虚拟三维显示信息)可以包括但不限于以下至少之一：AR物体、AR文字、AR图片和AR动画。在本发明实施例中不做限定。For the embodiment of the present invention, the AR information (that is, the virtual three-dimensional display information) may include but not limited to at least one of the following: AR objects, AR text, AR pictures, and AR animations. It is not limited in the embodiment of the present invention.

对于本发明实施例，对于自适应的选择要显示的AR信息，在本发明实施例中通过包括感知道路环境、本车状态，和驾驶人意图在内的至少之一种方式，自适应的判断某种或多种AR信息是否需要被显示，并生成相应的内容。For the embodiment of the present invention, for the adaptive selection of the AR information to be displayed, in the embodiment of the present invention, through at least one method including sensing the road environment, the state of the vehicle, and the driver's intention, the adaptive judgment Whether one or more types of AR information need to be displayed and generate corresponding content.

对于本发明实施例，对于以观感自然的方式呈现AR信息，在本发明实施例中通过将AR信息显示在物理正确的位置(如与相应的真实物体相对位置姿态尺度遮挡等关系正确的位置)，和/或驾驶人习惯的位置(即驾驶人不需要改变驾驶习惯的位置)。For the embodiment of the present invention, for presenting AR information in a natural way, in the embodiment of the present invention, the AR information is displayed in a physically correct position (such as a position with a correct relationship with the corresponding real object relative position, attitude, scale, etc.) , and/or the driver's habitual position (that is, the driver does not need to change the driving habit's position).

本发明实施例可以配合头戴式显示装置(如3D增强现实/混合现实眼镜)，和/或放置在车辆上的车载显示设备(如3D抬头显示器)使用。特别的，通过使用头戴式显示装置，设备可以将AR信息显示空间扩展为整个三维空间。The embodiment of the present invention can be used with a head-mounted display device (such as 3D augmented reality/mixed reality glasses), and/or a vehicle-mounted display device placed on a vehicle (such as a 3D head-up display). In particular, by using a head-mounted display device, the device can expand the AR information display space to the entire three-dimensional space.

其中，对于降低延迟，本发明实施例的设备和方法自适应减少两种延迟：一种是注意力延迟，另一种是显示延迟。注意力延迟定义为从设备显示AR信息到驾驶人注意到AR信息的时间延迟；显示延迟定义为设备在生成，渲染和显示AR信息上花费的时间。Among them, for reducing delay, the device and method of the embodiment of the present invention adaptively reduce two delays: one is attention delay, and the other is display delay. Attention latency is defined as the time delay from when the device displays AR information to when the driver notices the AR information; display latency is defined as the time the device spends generating, rendering, and displaying the AR information.

图1为本发明一个实施例提供的用于辅助驾驶的增强现实的方法流程示意图。Fig. 1 is a schematic flowchart of an augmented reality method for driving assistance provided by an embodiment of the present invention.

步骤101、基于行车过程中获取的信息来确定行车辅助信息；步骤102、显示行车辅助信息对应的虚拟三维显示信息。Step 101: Determine driving assistance information based on information acquired during driving; Step 102: Display virtual three-dimensional display information corresponding to the driving assistance information.

进一步地，步骤101包括步骤1011：步骤102包括步骤1021。Further,step 101 includes step 1011 ;step 102 includes step 1021 .

其中，步骤1011、基于行车过程中获取的可感知区域的信息确定被遮挡的行车辅助信息；步骤1021、显示被遮挡的行车辅助信息对应的虚拟三维显示信息。Wherein, step 1011, determine the blocked driving assistance information based on the perceptible area information obtained during driving; step 1021, display the virtual three-dimensional display information corresponding to the blocked driving assistance information.

其中，被遮挡的行车辅助信息包括以下至少之一：路面道路信息、非路面交通标志信息、盲区信息。Wherein, the blocked driving assistance information includes at least one of the following: road information, non-road traffic sign information, and blind spot information.

其中，路面道路信息包括以下至少之一：车道、车道线、道路边缘线、路面交通标志、路面交通标线。Wherein, the road surface information includes at least one of the following: lanes, lane lines, road edge lines, road traffic signs, and road traffic markings.

其中，非路面交通标志信息包括以下至少之一：路旁交通标志、道路上方交通标志。Wherein, the non-road surface traffic sign information includes at least one of the following: roadside traffic signs and road above traffic signs.

其中，盲区信息包括：后视镜盲区中的信息。Wherein, the blind spot information includes: information in the blind spot of the rearview mirror.

其中，交通标志包括以下至少之一：警告标志、禁令标志、指示标志、指路标志、旅游区标志、作业区标志、辅助标志、告示标志。Wherein, the traffic signs include at least one of the following: warning signs, prohibition signs, instruction signs, guide signs, tourist area signs, work area signs, auxiliary signs, and notice signs.

其中，交通标线包括以下至少之一：指示标线、禁止标线、警告标线。Wherein, the traffic markings include at least one of the following: indicating markings, prohibiting markings, and warning markings.

进一步地，当被遮挡的行车辅助信息包括：路面道路信息、和/或非路面交通标志信息时，显示被遮挡的行车辅助信息对应的虚拟三维显示信息，包括：在被遮挡的行车辅助信息的位置上，显示被遮挡的行车辅助信息对应的虚拟三维显示信息。Further, when the blocked driving assistance information includes: road information and/or non-road traffic sign information, the virtual three-dimensional display information corresponding to the blocked driving assistance information is displayed, including: In the position, the virtual three-dimensional display information corresponding to the blocked driving assistance information is displayed.

进一步地，基于行车过程中获取的可感知区域的信息确定被遮挡的行车辅助信息可以采用以下至少一种方式实现：若被遮挡的行车辅助信息只有部分被遮挡，则根据行车辅助信息的可感知部分，确定被遮挡的行车辅助信息；基于当前车辆的位置以及当前行车中可感知区域的参照物信息，确定被遮挡的行车辅助信息；基于从驾驶人视角以外其它角度获取到的被遮挡的行车辅助信息的多媒体信息，确定被遮挡的行车辅助信息；基于行车过程中获取的可感知区域中被遮挡的行车辅助信息的多媒体信息，对多媒体信息进行增强和/或复原，确定出被遮挡的行车辅助信息；被遮挡的行车辅助信息包括路面道路信息时，通过将当前道路与当前道路的地图进行对齐，根据地图确定被遮挡的行车辅助信息；根据其他的行车辅助信息，确定当前被遮挡的行车辅助信息。Further, determining the blocked driving assistance information based on the perceivable area information obtained during driving can be implemented in at least one of the following ways: if the blocked driving assistance information is only partially blocked, then according to the perceivable part, determine the blocked driving assistance information; determine the blocked driving assistance information based on the current vehicle position and the reference information of the perceptible area in the current driving; The multimedia information of the auxiliary information determines the blocked driving auxiliary information; based on the multimedia information of the blocked driving auxiliary information in the perceivable area acquired during driving, the multimedia information is enhanced and/or restored to determine the blocked driving Auxiliary information; when the blocked driving auxiliary information includes road information, by aligning the current road with the map of the current road, the blocked driving auxiliary information is determined according to the map; according to other driving auxiliary information, the currently blocked driving is determined Supplementary information.

进一步地，在基于行车过程中获取的可感知区域的信息确定被遮挡的行车辅助信息之后，方法还包括：对确定的被遮挡的行车辅助信息进行校正；Further, after determining the blocked driving assistance information based on the perceivable area information acquired during driving, the method further includes: correcting the determined blocked driving assistance information;

其中，显示被遮挡的行车辅助信息对应的虚拟三维显示信息，包括：在校正的位置处显示校正后的行车辅助信息对应的虚拟三维显示信息。Wherein, displaying the virtual three-dimensional display information corresponding to the blocked driving assistance information includes: displaying the virtual three-dimensional display information corresponding to the corrected driving assistance information at the corrected position.

进一步地，对确定的被遮挡的行车辅助信息进行校正可以采用以下至少一种方式实现：当被遮挡的行车辅助信息包括车道相关信息时，基于当前车辆预设范围内其他车辆的行车轨迹和/或路面车辙信息，校正被遮挡的行车辅助信息的位置；当被遮挡的行车辅助信息包括路面道路信息时，通过将当前道路与当前道路的地图进行对齐，根据地图校正被遮挡的行车辅助信息的位置。Further, correcting the determined blocked driving assistance information may be implemented in at least one of the following ways: when the blocked driving assistance information includes lane-related information, based on the driving trajectories of other vehicles within the preset range of the current vehicle and/or or road rut information, correct the position of the blocked driving assistance information; when the blocked driving assistance information includes road information, by aligning the current road with the map of the current road, correct the position of the blocked driving assistance information according to the map Location.

进一步地，当被遮挡的行车辅助信息包括：车道相关信息时，显示的车道宽度小于实际的车道宽度。Further, when the blocked driving assistance information includes: lane-related information, the displayed lane width is smaller than the actual lane width.

其中，该车道相关信息包括：车道、车道线、道路边缘线、路面交通标志、以及路面交通标线中至少之一。Wherein, the lane-related information includes: at least one of lanes, lane lines, road edge lines, road traffic signs, and road traffic markings.

进一步地，当被遮挡的行车辅助信息包括：盲区信息时，显示被遮挡的行车辅助信息对应的虚拟三维显示信息，包括：在后视镜的扩展区域显示盲区信息对应的虚拟三维显示信息。Further, when the blocked driving assistance information includes: blind spot information, displaying virtual three-dimensional display information corresponding to the blocked driving assistance information includes: displaying virtual three-dimensional display information corresponding to blind spot information in an extended area of the rearview mirror.

其中，当后视镜是侧方后视镜时，在扩展区域显示的虚拟三维显示信息是该虚拟三维显示信息对应的真实物体按照侧方后视镜的镜面属性和驾驶人视点生成的。Wherein, when the rearview mirror is a side rearview mirror, the virtual 3D display information displayed in the extended area is generated by the real object corresponding to the virtual 3D display information according to the mirror properties of the side rearview mirror and the driver's viewpoint.

进一步地，步骤101包括：获取当前路段的交通规则和/或交警动作信息，并对确定的当前路段的交通规则和/或交警动作信息进行呈现方式的转换；步骤102包括：显示转换后的当前路段的交通规则和/或交警动作信息对应的虚拟三维显示信息。Further,step 101 includes: obtaining traffic rules and/or traffic police action information of the current road segment, and converting the presentation mode of the determined traffic rules and/or traffic police action information of the current road segment;step 102 includes: displaying the converted current Virtual three-dimensional display information corresponding to road section traffic rules and/or traffic police action information.

进一步地，显示行车辅助信息对应的虚拟三维显示信息可以采用以下至少一种方式实现：当感知到异常车辙信息时，显示确定出的异常车辙区域对应的虚拟三维显示信息和/或该区域为异常车辙区域的警示信息的虚拟三维显示信息；当需要显示当前车辆已行驶过的道路区域的交通标志时，显示已获取到的当前车辆已行驶过的道路区域的交通标志对应的虚拟三维显示信息；当感知到当前车辆所在路口存在交通标志和/或交通信号灯，且交通标志和/或交通信号灯满足预定显示条件，显示该路口交通标志和/或交通信号灯对应的虚拟增强显示信息；当需要显示操作表盘中的按键信息时，显示以下信息至少之一对应的虚拟三维显示信息：按键的位置信息、按键的功能名称信息、按键的操作指示信息、和按键；当需要显示停车区域信息时，显示允许停车且适合停车区域、允许停车但不适合停车区域、不允许停车区域至少之一对应的虚拟三维显示信息。Further, displaying the virtual three-dimensional display information corresponding to the driving assistance information can be realized in at least one of the following ways: when the abnormal rut information is sensed, the virtual three-dimensional display information corresponding to the determined abnormal rut area is displayed and/or the area is abnormal Virtual three-dimensional display information of the warning information in the rut area; when it is necessary to display the traffic signs of the road area that the current vehicle has traveled, display the obtained virtual three-dimensional display information corresponding to the traffic signs of the road area that the current vehicle has traveled; When it is sensed that there are traffic signs and/or traffic lights at the intersection where the current vehicle is located, and the traffic signs and/or traffic lights meet the predetermined display conditions, display the virtual enhanced display information corresponding to the traffic signs and/or traffic lights at the intersection; when the display operation is required When the button information in the dial is displayed, the virtual three-dimensional display information corresponding to at least one of the following information is displayed: the position information of the button, the function name information of the button, the operation instruction information of the button, and the button; when the parking area information needs to be displayed, the display allows Virtual three-dimensional display information corresponding to at least one of parking and suitable parking area, parking allowed but not suitable for parking area, and parking area not allowed.

其中，感知可以包括机器设备识别、探测、检测中的至少一种，在此不再赘述。Wherein, perception may include at least one of machine equipment identification, detection, and detection, which will not be repeated here.

其中，异常车辙信息为满足异常车辙判断条件的车辆在行驶状态下的行车印迹，异常车辙区域为存在异常车辙信息的区域。Wherein, the abnormal rut information is the driving trace of the vehicle in the driving state that satisfies the abnormal rut judgment condition, and the abnormal rut area is the area where the abnormal rut information exists.

其中，该异常车辙信息包括以下至少之一：车辙边缘线的方向与车道线和/或车道边缘线的方向不一致的车辙信息；车辙边缘线与整体车辙边缘线的方向不一致的车辙信息；存在刹车痕迹的车辙信息。Wherein, the abnormal rut information includes at least one of the following: rut information in which the direction of the rut edge line is inconsistent with the direction of the lane line and/or the direction of the lane edge line; rut information in which the direction of the rut edge line is inconsistent with the direction of the overall rut edge line; there is braking Rut information for traces.

其中，生成的车辙边缘性是否异常可以通过以下方法判断：判断某一时刻的生成的车辙边缘线构建的向量场是否显著区别于其他/整体车辙边缘线构建的向量场的方向；若有显著区别，则确定生成的车辙边缘线属于异常车辙边缘线；和/或，判断车辙边缘线是否有明显的刹车痕迹；若有明显刹车痕迹，则确定生成的车辙边缘线属于异常车辙边缘线。Among them, whether the generated rut edge is abnormal can be judged by the following method: judging whether the vector field constructed by the generated rut edge line at a certain moment is significantly different from the direction of the vector field constructed by other/overall rut edge lines; if there is a significant difference , it is determined that the generated rut edge line belongs to the abnormal rut edge line; and/or, it is judged whether there are obvious braking marks on the rut edge line; if there are obvious braking marks, it is determined that the generated rut edge line belongs to the abnormal rut edge line.

其中，预定显示条件包括：交通标志和/或交通信号灯损坏、交通标志和/或交通信号灯显示不清晰、交通标志和/或交通信号灯不完整位于驾驶人的当前可视范围之内，以及根据驾驶人指令中至少之一。Among them, the predetermined display conditions include: the traffic signs and/or traffic lights are damaged, the traffic signs and/or traffic lights are not clearly displayed, the traffic signs and/or traffic lights are not completely within the driver's current visual range, and at least one of the human instructions.

进一步地，基于行车过程中获取的信息来确定行车辅助信息可以采用以下至少一种方式实现：确定路面车辙信息是否存在异常车辙信息，若存在异常车辙信息，则确定存在异常车辙区域；当需要显示当前车辆已行驶过的道路区域的交通标志时，从获取到的多媒体信息中和/或交通标志数据库中，确定当前已行驶过的道路区域的交通标志；当需要显示停车区域信息时，根据当前车辆周围区域是否存在禁止停车标识，当前车辆的体积，以及当前路面情况中的至少一项，确定允许停车且适合停车区域、允许停车但不适合停车区域、不允许停车区域中的至少一项。Further, determining the driving assistance information based on the information obtained during driving can be implemented in at least one of the following ways: determine whether there is abnormal rutting information in the road surface rutting information, and if there is abnormal rutting information, determine that there is an abnormal rutting area; when it is necessary to display When the traffic sign of the road area that the current vehicle has traveled, determine the traffic sign of the road area that the current vehicle has traveled from the acquired multimedia information and/or the traffic sign database; when it is necessary to display the parking area information, according to the current Whether there is no parking sign in the surrounding area of the vehicle, the volume of the current vehicle, and at least one of the current road conditions, determine at least one of the areas where parking is allowed and suitable for the parking area, parking is allowed but not suitable for the parking area, and parking area is not allowed.

进一步地，步骤102包括：增强显示车辙信息对应的虚拟三维显示信息。Further,step 102 includes: enhancing the display of virtual three-dimensional display information corresponding to the rut information.

进一步地，显示已获取到的当前车辆已行驶过的道路区域的交通标志对应的虚拟三维显示信息，包括：根据车辆当前位置及已行驶过的道路区域的交通标志对应的虚拟三维显示信息，调整当前已行驶过的道路区域的交通标志对应的虚拟三维显示信息，并显示调整后的交通标志对应的虚拟三维显示信息。Further, displaying the acquired virtual three-dimensional display information corresponding to the traffic signs in the road area that the current vehicle has driven includes: adjusting The virtual three-dimensional display information corresponding to the traffic sign in the road area that has been driven at present is displayed, and the adjusted virtual three-dimensional display information corresponding to the traffic sign is displayed.

进一步地，步骤102包括：确定虚拟三维显示信息对应的显示方式；基于确定的显示方式，显示行车辅助信息对应的虚拟三维显示信息。Further,step 102 includes: determining a display manner corresponding to the virtual three-dimensional display information; based on the determined display manner, displaying the virtual three-dimensional display information corresponding to the driving assistance information.

其中显示方式包括以下至少一项：虚拟三维显示信息的显示位置；虚拟三维显示信息的显示姿态；虚拟三维显示信息的显示尺寸大小；虚拟三维显示信息的显示开始时间；虚拟三维显示信息的显示结束时间；虚拟三维显示信息的显示时长；虚拟三维显示信息的显示内容详细程度；虚拟三维显示信息的呈现方式；多个虚拟三维显示信息间显示的相互关系。The display method includes at least one of the following: the display position of the virtual three-dimensional display information; the display posture of the virtual three-dimensional display information; the display size of the virtual three-dimensional display information; the display start time of the virtual three-dimensional display information; the display end of the virtual three-dimensional display information time; the display duration of the virtual three-dimensional display information; the detail degree of the display content of the virtual three-dimensional display information; the presentation mode of the virtual three-dimensional display information;

其中，呈现方式包括以下至少一项：文字方式；图标方式；动画方式；音效方式；灯光方式；震动方式。Wherein, the presentation mode includes at least one of the following: text mode; icon mode; animation mode; sound effect mode; light mode; vibration mode.

进一步地，该方法还包括以下至少一种方式：当同时存在多个待显示的虚拟三维显示信息时，将多个待显示的虚拟三维显示信息进行合并处理，并显示处理后的虚拟三维显示信息；当同时显示多个待显示的虚拟三维显示信息时，将多个待显示的虚拟三维显示信息，基于语义进行整合处理，并显示处理后的虚拟三维显示信息。Further, the method further includes at least one of the following ways: when there are multiple virtual three-dimensional display information to be displayed at the same time, combining the multiple virtual three-dimensional display information to be displayed, and displaying the processed virtual three-dimensional display information ; When simultaneously displaying multiple virtual three-dimensional display information to be displayed, the multiple virtual three-dimensional display information to be displayed is integrated based on semantics, and the processed virtual three-dimensional display information is displayed.

进一步地，该方法还包括以下至少一种方式：将高于第一预设优先级的行车辅助信息对应的虚拟三维显示信息显示在驾驶人当前视野的显著位置，并实时根据驾驶人的视线位置，调整显示该虚拟三维显示信息的位置；显示高于第一预设优先级的行车辅助信息对应的虚拟三维显示信息，暂停和/或停止显示低于第二预设优先级的行车辅助信息对应的虚拟三维显示信息。Further, the method further includes at least one of the following methods: displaying the virtual three-dimensional display information corresponding to the driving assistance information higher than the first preset priority in a prominent position of the driver's current field of vision, and real-time according to the driver's line of sight position , adjust the position where the virtual three-dimensional display information is displayed; display the virtual three-dimensional display information corresponding to the driving assistance information higher than the first preset priority, and pause and/or stop displaying the corresponding driving assistance information lower than the second preset priority virtual 3D display information.

其中，显著位置可以是驾驶人当前视野的中心区域、驾驶人视线聚焦区域、驾驶人视线停留时间长的区域、正面面对驾驶人的区域中的至少一处。Wherein, the conspicuous position may be at least one of the central area of the driver's current field of vision, the area where the driver's line of sight is focused, the area where the driver's line of sight stays for a long time, and the area that faces the driver directly.

其中，第一预设优先级，和/或第二预设优先级可以根据驾驶人指令定义；也可以根据感知到的道路情况、本车情况、驾驶人意图、行车辅助信息的语义分析至少之一，自适应的对行车辅助信息进行分级。Wherein, the first preset priority, and/or the second preset priority can be defined according to the driver's instruction; it can also be defined according to at least one of the perceived road conditions, vehicle conditions, driver intentions, and semantic analysis of driving assistance information. First, adaptively grade the driving assistance information.

进一步地，步骤102包括：根据车辆的当前状态、当前的路况信息以及设备的系统延迟情况中的至少之一，确定显示虚拟三维显示信息的显示开始时间、显示结束时间以及显示持续时长中的至少一项；根据确定的显示虚拟三维显示信息的显示开始时间、显示结束时间以及显示持续时长中的至少一项，显示行车辅助信息对应的虚拟三维显示信息。Further,step 102 includes: according to at least one of the current state of the vehicle, the current road condition information, and the system delay of the device, determine at least one of the display start time, display end time, and display duration of the virtual three-dimensional display information. One item: display the virtual three-dimensional display information corresponding to the driving assistance information according to at least one of the determined display start time, display end time, and display duration of the virtual three-dimensional display information.

进一步地，当同时存在多个待显示的行车辅助信息对应的虚拟三维显示信息，并且待显示的多个虚拟三维显示信息之间存在遮挡关系时，还可以包括以下至少一种方式：根据存在遮挡关系的多个虚拟三维显示信息之间的位置关系，仅显示虚拟三维显示信息中未被遮挡的部分；在不同的显示时间，分别显示存在遮挡关系的多个虚拟三维显示信息中虚拟三维显示信息；调整存在遮挡关系的多个虚拟三维显示信息中至少一个虚拟三维显示信息的显示位置、内容详细程度、呈现方式中的至少一项，并根据调整的方式，显示存在遮挡关系的多个虚拟三维显示信息中每个虚拟三维显示信息。Further, when there is a plurality of virtual 3D display information corresponding to the driving assistance information to be displayed at the same time, and there is an occlusion relationship among the plurality of virtual 3D display information to be displayed, at least one of the following methods may also be included: The positional relationship between multiple virtual 3D display information related to each other, only the unoccluded part of the virtual 3D display information is displayed; at different display times, the virtual 3D display information among the multiple virtual 3D display information with occlusion relationship is displayed respectively ;Adjust at least one of the display position, level of content detail, and presentation method of at least one of the virtual three-dimensional display information in the plurality of virtual three-dimensional display information with an occlusion relationship, and display the plurality of virtual three-dimensional display information with an occlusion relationship according to the adjustment method Each virtual three-dimensional display information in the display information.

进一步地，步骤102包括：在预设的显示位置，显示待显示的行车辅助信息对应的虚拟三维显示信息。Further,step 102 includes: displaying virtual three-dimensional display information corresponding to the driving assistance information to be displayed at a preset display position.

其中，预设的显示位置包括以下至少一项：Wherein, the preset display position includes at least one of the following:

与真实的行车辅助信息对齐的显示位置；不干扰驾驶人驾驶的区域位置；驾驶人当前视野的显著位置；驾驶人视野相对开阔的位置；驾驶人注意力不足的位置。The display position aligned with the real driving assistance information; the area position that does not interfere with the driver's driving; the prominent position of the driver's current field of vision; the position where the driver's field of vision is relatively open; the position where the driver's attention is insufficient.

进一步地，该方法还包括：预先渲染待显示的虚拟三维显示信息；当满足预设显示触发条件时，从预先渲染的虚拟三维显示信息中，获取待显示的虚拟三维显示信息，并根据当前环境调整虚拟三维显示信息的呈现方式，并根据调整后的呈现方式，显示虚拟三维显示信息；根据当前环境实时调整虚拟三维显示信息的显示方式，并根据调整后的显示方式，显示虚拟三维显示信息。Further, the method further includes: pre-rendering the virtual three-dimensional display information to be displayed; when the preset display trigger condition is satisfied, acquiring the virtual three-dimensional display information to be displayed from the pre-rendered virtual three-dimensional display information, and according to the current environment Adjust the presentation mode of the virtual three-dimensional display information, and display the virtual three-dimensional display information according to the adjusted presentation mode; adjust the display mode of the virtual three-dimensional display information in real time according to the current environment, and display the virtual three-dimensional display information according to the adjusted display mode.

其中，预设显示触发条件可以根据驾驶人的指令进行定义；也可以根据感知到的道路情况、本车情况、驾驶人意图、行车辅助信息的语义分析至少之一，自适应的定义预设显示触发条件。Among them, the preset display trigger condition can be defined according to the driver's instruction; it can also be based on at least one of the perceived road conditions, vehicle conditions, driver intentions, and semantic analysis of driving assistance information, and the preset display can be adaptively defined. Triggering conditions.

本发明实施例提供了一种用于辅助驾驶的增强现实的方法，与现有技术相比，本发明实施例基于行车过程中获取的信息来确定行车辅助信息，然后显示行车辅助信息对应的虚拟三维显示信息，即通过车辆行驶过程中获取到的信息，确定出行车过程中的行车辅助信息，并将行车过程中的行车辅助信息对应的虚拟三维显示信息，通过视觉和/或听觉的方式呈现给驾驶人，以通知或者警示驾驶人，从而可以实现在车辆行驶过程中应用增强现实技术帮助驾驶人更好的掌握车辆行驶过程中的行车信息，进而可以提升用户体验。The embodiment of the present invention provides an augmented reality method for assisting driving. Compared with the prior art, the embodiment of the present invention determines the driving assistance information based on the information obtained during driving, and then displays the virtual reality corresponding to the driving assistance information. Three-dimensional display information, that is, determine the driving assistance information during the driving process through the information obtained during the driving process of the vehicle, and present the virtual three-dimensional display information corresponding to the driving auxiliary information during the driving process through visual and/or auditory means To the driver, to notify or warn the driver, so that the application of augmented reality technology in the driving process of the vehicle can help the driver better grasp the driving information during the driving process of the vehicle, thereby improving the user experience.

对于本发明实施例，图1示出了本发明中描述的辅助驾驶设备(下文简称设备)的显示方式的整体流程示意图，该方法可以应用于行车过程中驾驶人佩戴的增强/混合现实的头戴式显示设备(近眼显示设备)，如3D增强现实眼镜，和/或放置在车辆上的车载显示设备，如3D抬头显示器。请注意到设备可以包括多个同种或不同种显示设备，当显示设备不同时，实现方式有所不同，如下文详述。For the embodiment of the present invention, Fig. 1 shows the overall flowchart of the display mode of the driving assistance device (hereinafter referred to as the device) described in the present invention, which can be applied to the augmented/mixed reality headset worn by the driver during driving. A wearable display device (near-eye display device), such as 3D augmented reality glasses, and/or a vehicle-mounted display device placed on a vehicle, such as a 3D head-up display. Please note that the device may include multiple display devices of the same type or different types. When the display devices are different, the implementation methods are different, as detailed below.

本发明实施例的设备整体流程示意图中，各步骤所执行的内容概述如下：步骤S110(图中未标注)：设备确定需要显示的一个或多个目标行车辅助信息；步骤S120(图中未标注)：获取信息，处理信息，生成目标行车辅助信息内容；步骤S130(图中未标注)：确定一个或多个目标行车辅助信息的显示方式；步骤S140(图中未标注)：显示一个或多个目标行车辅助信息对应的虚拟三维AR信息。In the schematic diagram of the overall flow of the device in the embodiment of the present invention, the contents executed by each step are summarized as follows: Step S110 (not marked in the figure): the device determines one or more target driving assistance information to be displayed; step S120 (not marked in the figure) ): acquire information, process information, and generate target driving assistance information content; step S130 (not marked in the figure): determine the display mode of one or more target driving assistance information; step S140 (not marked in the figure): display one or more The virtual three-dimensional AR information corresponding to the target driving assistance information.

其中，除AR物体外的AR信息以文字方式、图标方式、动画方式、音效方式、灯光方式、震动方式等至少一项呈现方式呈现的信息，例如一个配有文字的箭头图标；AR物体可以包括但不限于以真实物体形态呈现的信息，例如一个虚拟的交通红绿灯。Among them, the AR information other than the AR object is information presented in at least one presentation mode such as text, icon, animation, sound effect, light, vibration, etc., for example, an arrow icon with text; the AR object can include But not limited to information presented in the form of real objects, such as a virtual traffic light.

其中，在下文中，虽然AR信息和AR物体经常同时提及，AR信息中的AR物体，虽然不是全都需要和真实物体对齐，但通常指代在显示时需要和真实物体对齐的虚拟物体；而其它AR信息在显示时，通常不需要与真实物体对齐。Among them, in the following, although AR information and AR objects are often mentioned at the same time, although not all AR objects in AR information need to be aligned with real objects, they usually refer to virtual objects that need to be aligned with real objects when displayed; while other When AR information is displayed, it usually does not need to be aligned with real objects.

其中，在步骤S110，设备可以选择零个目标行车辅助信息，即当前场景下不显示行车辅助信息对应的虚拟三维显示信息。Wherein, in step S110, the device may select zero target driving assistance information, that is, the virtual three-dimensional display information corresponding to the driving assistance information is not displayed in the current scene.

其中，在步骤S110，设备可以通过识别场景自适应判断需要显示的目标行车辅助信息，也可以通过用户交互的方式获取需要显示的目标行车辅助信息，也可以两种方式组合使用。Wherein, in step S110, the device can adaptively determine the target driving assistance information to be displayed by identifying the scene, or obtain the target driving assistance information to be displayed through user interaction, or use a combination of the two methods.

其中，目标行车辅助信息可以包括但不限于行车过程中与行车安全、行车环境、交通和路况等信息标识、交通规则以及车内信息等相关联的提示信息。Wherein, the target driving assistance information may include but not limited to prompt information associated with driving safety, driving environment, traffic and road conditions and other information signs, traffic rules, and in-vehicle information during driving.

具体地，与目标行车辅助信息相关的行车过程中的目标对象可以包括但不限于：车道线、车道分隔障碍物、车道、周围机动车辆、周围非机动车辆、周围行人、周围树木、周围建筑物、路面车辙、交通和路况等信息标识、交通警察、侧方后视镜的盲区物体、车内后排座位的物体、车尾外部区域，以及驾驶操作台等。Specifically, the target objects in the driving process related to the target driving assistance information may include but not limited to: lane lines, lane separation obstacles, lanes, surrounding motor vehicles, surrounding non-motor vehicles, surrounding pedestrians, surrounding trees, surrounding buildings , road ruts, traffic and road conditions and other information signs, traffic police, objects in the blind area of the side rearview mirror, objects in the rear seats in the car, the outer area of the rear of the car, and the driving console, etc.

其中，在步骤S130，显示方式的确定包括以下至少一项：一个或多个AR信息的显示位置、显示姿态、显示尺寸大小、显示开始时间、显示结束时间、和/或显示多个AR信息时的位置和/或姿态，显示的开始时机、显示的结束时机、显示时长，显示内容详细程度、呈现方式，以及多个AR信息间显示的相互关系。Wherein, in step S130, the determination of the display mode includes at least one of the following: display position, display posture, display size, display start time, display end time of one or more AR information, and/or when multiple AR information is displayed The location and/or posture of the AR information, the start timing of the display, the end timing of the display, the display duration, the detail level of the display content, the presentation method, and the relationship between multiple AR information displays.

实施例一Embodiment one

本实施例提供了一种行车辅助信息的显示方法，该方法用于在路面被部分覆盖遮挡的道路，对包括以下至少之一：车道线、道路边缘线、路面交通标志、路面交通标线等信息标识的提示信息进行显示，并显示相应的增强现实辅助驾驶信息，如图2所示。其中，遮挡道路上车道线的可以是但不限于落叶、积雪、积水、泥污、油等，或组合。This embodiment provides a method for displaying driving assistance information. The method is used for roads that are partially covered and blocked, and includes at least one of the following: lane lines, road edge lines, road traffic signs, road traffic markings, etc. The prompt information of the information identification is displayed, and the corresponding augmented reality assisted driving information is displayed, as shown in FIG. 2 . Wherein, what blocks the lane line on the road may be but not limited to fallen leaves, snow, water, mud, oil, etc., or a combination thereof.

本实施例中的方法包括：The method in this embodiment includes:

步骤S1101(图中未标注)、设备确定是否需要显示道路信息。Step S1101 (not marked in the figure), the device determines whether road information needs to be displayed.

该步骤S1101可以是设备确定需要显示的一个或多个目标行车辅助信息的一种实现方式。This step S1101 may be an implementation manner for the device to determine one or more target driving assistance information to be displayed.

对于本发明实施例，设备通过图像检测识别的方式确定本车周围的道路信息，可以包括但不限于车道线、道路边缘线、路面交通标志、路面交通标线。其中，设备可以是一直保持检测识别的状态，在部分遮挡覆盖的路面情况下自适应开启显示功能，也可以根据用户的指令开始检测识别和/或显示的功能。其中，用户指令可以是通过手势、语音、物理按键、指纹等生物识别标识来承载的。For the embodiment of the present invention, the device determines the road information around the vehicle through image detection and recognition, which may include but not limited to lane lines, road edge lines, road traffic signs, and road traffic markings. Among them, the device can always keep the state of detection and recognition, and adaptively turn on the display function when the road surface is partially covered, or start the function of detection, recognition and/or display according to the user's instruction. Among them, user instructions can be carried by gestures, voice, physical buttons, fingerprints and other biometric identifications.

步骤S1201(图中未标注)、设备检测识别道路信息，生成目标行车辅助信息内容。Step S1201 (not marked in the figure), the device detects and recognizes road information, and generates target driving assistance information content.

该步骤S1201可以是获取信息，处理信息，生成目标行车辅助信息内容的一种实现方式。This step S1201 may be an implementation manner of acquiring information, processing information, and generating target driving assistance information content.

对于本发明实施例，设备通过图像处理技术和识别技术从一个或多个图像/视频中定位识别出包括车道线、道路边缘线、路面交通标志，和路面交通标线至少之一的可见部分，即未被完全覆盖遮挡的部分；并根据车道线的可见线段连接成完整的车道线(若车道线为虚线，则将虚线补全)；将道路边缘线的可见线段连接成完整的道路边缘线，以及根据路面交通标志和/或路面交通标线的可见部分识别出路面交通标志和/或路面交通标线的类型。For the embodiment of the present invention, the device uses image processing technology and recognition technology to locate and identify the visible part including at least one of lane lines, road edge lines, road traffic signs, and road traffic markings from one or more images/videos, That is, the part that is not completely covered and blocked; and connect the visible line segments of the lane line into a complete lane line (if the lane line is a dotted line, the dotted line will be completed); connect the visible line segments of the road edge line into a complete road edge line , and identifying the type of the traffic sign and/or traffic marking based on the visible portion of the traffic sign and/or traffic marking.

具体的，在单幅图像中，车道线、道路边缘线的可见线段可以通过图像边缘提取算法和/或颜色聚类算法提取，并且可以使用车道线和道路边缘线通常为规则直线或平滑弧线的先验知识去除错误的线段；部分遮挡的路面交通标志和/或路面交通标线的轮廓可以通过图像边缘提取算法和/或颜色聚类算法提取，并在路面交通标志数据库中匹配获取完整的路面交通标志和/或路面交通标线。Specifically, in a single image, the visible line segments of lane lines and road edge lines can be extracted by image edge extraction algorithms and/or color clustering algorithms, and lane lines and road edge lines are usually regular straight lines or smooth arcs. The prior knowledge to remove wrong line segments; the outline of partially occluded road traffic signs and/or road traffic markings can be extracted by image edge extraction algorithm and/or color clustering algorithm, and matched in the road traffic sign database to obtain a complete Road traffic signs and/or road traffic markings.

对于本发明实施例，车道线、道路边缘线、路面交通标志，和路面交通标线也可以直接由检测识别算法直接识别定位。识别的过程中，可以使用道路交通的领域知识，和/或道路地图，辅助检测识别。例如，白色单虚线和白色单实线在部分覆盖遮挡的路面均表现为不规则的白色单虚线，基于道路交通的领域知识，设备可以通过判断检测到的白色单虚线中是否存在长度超过规定长度的线段，来判断检测到的白色单虚线对应于真实的白色单虚线还是真实的白色单实线。进一步地，基于道路交通的领域知识和道路地图，设备可以根据检测到的白色单虚线的所处位置和交通含义判断其对应于真实的白色单虚线还是真实的白色单实线。For the embodiment of the present invention, lane lines, road edge lines, road traffic signs, and road traffic markings can also be directly identified and positioned by a detection and recognition algorithm. During the recognition process, domain knowledge of road traffic and/or road maps can be used to assist detection and recognition. For example, white single-dotted lines and white single-solid lines appear as irregular white single-dotted lines on partially covered roads. Based on the domain knowledge of road traffic, the device can judge whether there is a length exceeding the specified length among the detected white single-dotted lines. to determine whether the detected white single-dashed line corresponds to a real white single-dashed line or a real white single-dotted line. Further, based on the domain knowledge of road traffic and the road map, the device can judge whether the detected white single dashed line corresponds to a real white single dashed line or a real white single solid line according to the location and traffic meaning of the detected white single dashed line.

例如，检测到的白色单虚线位于道路正中间，根据道路交通的领域知识其应对应于真实的白色单实线。For example, the detected white single dashed line is located in the middle of the road, which should correspond to the real white single solid line according to the domain knowledge of road traffic.

进一步地，识别的过程中，可以使用道路交通的领域知识，和/或道路地图，生成正确的路面交通标志和/或路面交通标线。例如，路面的直行方向箭头和右转弯方向箭头在箭头部分覆盖遮挡的情况下无法区分，均表现为长方形；但若根据检测到的长方形所在车道，在道路地图显示为右转弯车道，则设备可区分出该路面交通标志为右转弯方向箭头。Furthermore, during the recognition process, the domain knowledge of road traffic and/or road maps may be used to generate correct road traffic signs and/or road traffic markings. For example, the straight direction arrow and the right-turn direction arrow on the road surface cannot be distinguished when the arrows are partially covered and occluded, and both appear as rectangles; however, if the detected lane of the rectangle is displayed on the road map as a right-turn lane, the device can Distinguish the road traffic sign as a right-turn direction arrow.

特别的，当同时有空间上的多幅图像，和/或时间上的多幅图像时，设备可以使用多幅图像协同识别包括车道线、道路边缘线，路面交通标志，和路面交通标线至少之一，去除使用单幅图像时的错误，并使得识别结果保持空间，和/或时间上的一致。In particular, when there are multiple images in space and/or multiple images in time, the device can use multiple images to collaboratively identify lane lines, road edge lines, road traffic signs, and road traffic markings at least One is to remove errors when using a single image, and make the recognition results consistent in space and/or time.

例如，设备可以在某一段路面识别出该车道线对应于真实的白色单实线，在之后的行驶中对同一车道线进行跟踪，并保持识别为白色单实线。For example, the device can recognize that the lane line corresponds to a real white single solid line on a certain road surface, track the same lane line in the subsequent driving, and keep recognizing it as a white single solid line.

步骤S1301(图中未标注)、确定目标行车辅助信息内容的显示方式。Step S1301 (not marked in the figure), determining the display mode of the content of the target driving assistance information.

该步骤S1301可以是确定一个或多个目标行车辅助信息的显示方式的一种实现方式。This step S1301 may be an implementation of determining a display manner of one or more target driving assistance information.

对于本发明实施例，设备通过定位算法获取真实物体相对于显示装置的位置和姿态，从而可以将显示的AR信息和对应的真实物体对齐。For the embodiment of the present invention, the device obtains the position and posture of the real object relative to the display device through a positioning algorithm, so that the displayed AR information can be aligned with the corresponding real object.

特别的，为减小延迟，对于同一真实物体，设备可以根据本车的运动模型预测与此真实物体当前和本车的相对位置、姿态和/或尺度关系，预测此真实物体在未来时刻和本车的相对位置、姿态和/或尺度关系，从而提前准备目标行车辅助信息对应的AR信息。In particular, in order to reduce the delay, for the same real object, the device can predict the relative position, attitude and/or scale relationship between the real object and the current vehicle based on the motion model of the vehicle, and predict the future time of the real object and the vehicle. The relative position, attitude and/or scale relationship of the car, so as to prepare the AR information corresponding to the target driving assistance information in advance.

具体的，当使用单个相机获取本车周围的道路信息时，近似的，考虑到本车周围局部道路可以认为是一个平面，可以对道路图像提取特征点，通过求解单应性矩阵获取道路和相机的相对位置和姿态关系；更为精确的，可以使用视觉里程计的方式对图像序列进行特征跟踪，其中特征采集于车道线线段、道路边缘线段、路面交通标志轮廓，和路面交通标线轮廓等需要对齐的真实物体，从而获取真实物体相对于相机的相对位置和姿态关系。特别的，特征点的提取和跟踪可以用图像识别分割辅助，从而去除错误匹配并加快速度。Specifically, when using a single camera to obtain road information around the vehicle, approximately, considering that the local road around the vehicle can be considered as a plane, feature points can be extracted from the road image, and the road and camera can be obtained by solving the homography matrix The relative position and attitude relationship of the image; more precisely, the visual odometer can be used to track the features of the image sequence, in which the features are collected from the lane line segment, road edge line segment, road traffic sign outline, and road traffic marking outline, etc. The real object that needs to be aligned, so as to obtain the relative position and pose relationship of the real object relative to the camera. In particular, the extraction and tracking of feature points can be assisted by image recognition segmentation to remove false matches and speed up.

对于本发明实施例，运用单相机时，真实物体的尺度信息可以通过如下三种方式获得(该三种方式可以单独使用也可以组合使用)；1)设备可以通过事先标定相机在本车辆的安装固定高度获取尺度信息；2)设备可以根据道路交通的领域的先验知识获取真实物体的物理尺寸，继而获得尺度信息，例如，可以根据先验知识获取当地车道的规定宽度；3)当设备使用图像序列的时候，可通过车辆的实际运动速度、距离等信息获取尺度。For the embodiment of the present invention, when using a single camera, the scale information of the real object can be obtained in the following three ways (these three ways can be used alone or in combination); Fixed height to obtain scale information; 2) The device can obtain the physical size of the real object according to the prior knowledge of the road traffic field, and then obtain the scale information, for example, the specified width of the local lane can be obtained according to the prior knowledge; 3) When the device is used During the image sequence, the scale can be obtained through information such as the actual speed and distance of the vehicle.

对于本发明实施例，当使用单相机配合立体相机、深度相机、激光传感器、雷达传感器、超声传感器中至少之一获取本车周围的道路信息时，真实物体和相机的相对位置和姿态关系可以通过与上述使用单相机的情况相似的方式获取，在此不再赘述。特别的，当使用标定过的立体相机、深度相机、激光传感器、雷达传感器、超声传感器至少之一时，真实物体的尺度信息可以直接获取，也可以用上述使用单相机时的方式获得的尺度估计值进行交叉验证。For the embodiment of the present invention, when a single camera is used to cooperate with at least one of a stereo camera, a depth camera, a laser sensor, a radar sensor, and an ultrasonic sensor to obtain road information around the vehicle, the relative position and attitude relationship between the real object and the camera can be obtained through It is obtained in a manner similar to the above-mentioned case of using a single camera, and will not be repeated here. In particular, when using at least one of the calibrated stereo camera, depth camera, laser sensor, radar sensor, and ultrasonic sensor, the scale information of the real object can be obtained directly, or the scale estimated value obtained in the above-mentioned way when using a single camera Do cross-validation.

对于本发明实施例，尺度信息也可以由其他传感器和单相机配合获取，例如，码盘和单相机数据融合可以估计出尺度信息；再例如，惯性传感器单元(包括加速度计和陀螺仪)和单相机数据融合可以估计出尺度信息；设备也可以组合使用这些传感器的数据，以获取尺度信息。For the embodiment of the present invention, scale information can also be acquired by cooperation of other sensors and a single camera, for example, code disc and single camera data fusion can estimate scale information; Camera data fusion can estimate scale information; devices can also combine data from these sensors to obtain scale information.

对于本发明实施例，通过如上方式，我们可以获取真实物体和本车上，或设备上的至少一个外视相机(即拍摄本车外部环境的相机)的位置、姿态、以及尺度关系。For the embodiment of the present invention, through the above method, we can obtain the position, attitude, and scale relationship between the real object and at least one external-view camera on the vehicle or on the device (ie, the camera that captures the external environment of the vehicle).

对于本发明实施例，为了将AR信息和真实物体对齐，设备需要进一步估计眼睛和真实物体之间的相对位置、姿态、以及尺度关系。其中，估计的步骤和方式与设备的显示装置的种类有关，如下对显示装置为单个头戴式显示装置或单个车载显示装置这两种情况分别进行表述。当设备的显示装置包括多个的头戴式显示设备，和/或多个车载显示器时，如下方法可以通过较为直接的组合、调整被应用，不再赘述。For the embodiment of the present invention, in order to align the AR information with the real object, the device needs to further estimate the relative position, pose, and scale relationship between the eyes and the real object. Wherein, the steps and methods of estimation are related to the type of the display device of the equipment, and the two cases of the display device being a single head-mounted display device or a single vehicle-mounted display device are respectively described as follows. When the display device of the device includes multiple head-mounted display devices and/or multiple vehicle-mounted displays, the following methods can be applied through relatively direct combination and adjustment, and will not be repeated here.

1)对于显示装置为单个头戴式显示装置的情况，眼睛和显示装置的相对位置和姿态关系相对固定，可以事先校准(使用期偶尔需要重新校准，例如使用人调整了头戴式显示设备的位置之后需要重新校准位置姿态和尺度关系)；1) For the case where the display device is a single head-mounted display device, the relative position and posture relationship between the eyes and the display device are relatively fixed, which can be calibrated in advance (recalibration is occasionally required during the use period, for example, the user adjusts the head-mounted display device After the position, the position, attitude and scale relationship need to be recalibrated);

1.1)对于已经获取真实物体和设备上的一个外视相机(即拍摄本车外部环境的相机)的位置、姿态、以及尺度关系的情况，由于设备上的一个外视相机和显示装置位置姿态尺度关系相对固定，设备可以计算获取到眼睛和真实物体之间的相对位置、姿态、以及尺度关系(眼睛←校准→显示装置←校准→设备上的外视相机←估计→真实物体)；1.1) For the situation that the position, attitude, and scale relationship of an external-looking camera (that is, the camera that captures the external environment of the vehicle) on the real object and the equipment have been obtained, since the position and attitude scale of an external-looking camera on the equipment and the display device The relationship is relatively fixed, and the device can calculate and obtain the relative position, attitude, and scale relationship between the eye and the real object (eye ← calibration → display device ← calibration → external camera on the device ← estimation → real object);

1.2)对于已经获取真实物体和本车上的一个外视相机(即拍摄本车外部环境的相机)的位置、姿态、以及尺度关系的情况，设置仍需要获取显示装置和外视相机之间的相对位置姿态关系，根据设备硬件实现方式不同，有如下两种方式1.2.1)以及1.2.2)：1.2) For the situation where the position, attitude, and scale relationship between the real object and an external-view camera on the vehicle (that is, the camera that captures the external environment of the vehicle) have been obtained, the setting still needs to obtain the distance between the display device and the external-view camera. Relative position and posture relationship, depending on the hardware implementation of the device, there are the following two methods 1.2.1) and 1.2.2):

1.2.1)设备可以使用设备上的外视相机获取本车上的一个外视相机和设备上的一个外视相机的相对位置姿态尺度关系。其中，获取的方式可以通过在与车载显示装置相对位置姿态尺度固定的地方贴定位标志物，设备上的外视相机通过定位标志物计算出和本车外视相机的相对位置姿态尺度关系；获取的方式也可以是将本车上的外视相机作为场景中的物体，用上文描述的基于图像和/或多种传感器融合的提取特征点跟踪方式，和/或检测等方式的，如SLAM(Simultaneous Localization and Mapping，同步定位和建图)技术和目标跟踪技术。即(眼睛←校准→显示装置←校准→设备上的外视相机←估计→本车上的外视相机←估计→真实物体)；1.2.1) The device can use the external camera on the device to obtain the relative position, attitude and scale relationship between an external camera on the vehicle and an external camera on the device. Among them, the acquisition method can be by pasting a positioning marker on a place where the relative position and attitude scale of the vehicle-mounted display device is fixed, and the external-view camera on the device calculates the relative position and attitude-scale relationship with the external-view camera of the vehicle through the positioning marker; The method can also be to use the external camera on the vehicle as an object in the scene, and use the above-described method of extracting feature points based on image and/or fusion of multiple sensors to track and/or detect, such as SLAM (Simultaneous Localization and Mapping, simultaneous positioning and mapping) technology and target tracking technology. That is (eye ← calibration → display device ← calibration → external camera on the device ← estimation → external camera on the vehicle ← estimation → real object);

1.2.2)设备可以使用本车上的内视相机(即拍摄本车内部环境，如驾驶人位置的相机)获取显示装置和本车上的内视相机的相对位置姿态尺度关系。其中获取的方式可以是基于显示装置上的定位标志物的，和/或基于图像和/或多种传感器融合的提取特征点跟踪方式，和/或检测等方式的，如SLAM(Simultaneous Localization and Mapping，同步定位和建图)技术和目标跟踪技术。本车上的内视相机和本车上的外视相机的相对位置姿态尺度关系相对固定，可以事先校准(使用期偶尔需要重新校准，例如车辆剧烈颠簸后，需要重新校准位置姿态和尺度关系)。即(眼睛←校准→显示装置←估计→本车上的内视相机←校准→本车上的外视相机←估计→真实物体)；1.2.2) The device can use the internal camera on the vehicle (that is, the camera that captures the internal environment of the vehicle, such as the driver's position) to obtain the relative position, posture, and scale relationship between the display device and the internal camera on the vehicle. The acquisition method can be based on the positioning markers on the display device, and/or based on image and/or multiple sensor fusion extraction feature point tracking methods, and/or detection methods, such as SLAM (Simultaneous Localization and Mapping , simultaneous positioning and mapping) technology and target tracking technology. The relative position, attitude, and scale relationship of the interior camera on this vehicle and the exterior camera on this vehicle are relatively fixed, and can be calibrated in advance (occasionally need to be recalibrated during the service period, for example, after the vehicle is violently bumped, the position, attitude, and scale relationship need to be recalibrated) . Namely (eye ← calibration → display device ← estimation → internal camera on the vehicle ← calibration → external camera on the vehicle ← estimation → real object);

2)对于显示装置为单个车载显示装置的情况，显示装置和本车上的一个外视相机的相对位置和姿态关系相对固定，可以事先校准(使用期偶尔需要重新校准，例如车辆剧烈颠簸后，需要重新校准位置姿态和尺度关系)；特别的，设备上的一个外视相机和本车上的一个外视相机相对位置姿态尺度关系也可以认为是相对固定，两个外置相机也可以是同一个，因此这种情况下不需要区分外视相机为本车上的外视相机还是设备上的外视相机；为了获取眼睛和真实物体之间的相对位置、姿态、以及尺度关系，设备只需要获取眼睛和显示装置的相对位置、姿态、以及尺度关系。根据设备硬件实现方式不同，有如下两种方式：1.3)以及1.4)，其中，2) For the case where the display device is a single vehicle-mounted display device, the relative position and attitude relationship between the display device and an external-view camera on the vehicle are relatively fixed, and can be calibrated in advance (occasionally need to be re-calibrated during the service period, for example, after the vehicle is violently bumped, It is necessary to recalibrate the position, attitude and scale relationship); in particular, the relative position, attitude and scale relationship between an external camera on the device and an external camera on the vehicle can also be considered relatively fixed, and the two external cameras can also be the same One, so in this case, it is not necessary to distinguish whether the external-view camera is the external-view camera on the vehicle or the external-view camera on the device; in order to obtain the relative position, posture, and scale relationship between the eyes and the real object, the device only needs to Obtain the relative position, pose, and scale relationship of the eye and the display device. Depending on the hardware implementation of the device, there are two methods as follows: 1.3) and 1.4), among which,

1.3)设备可以使用驾驶人头戴的外视相机获取车载显示装置和头戴的外视相机的相对位置姿态尺度关系。其中，头戴的外视相机和眼睛的相对位置姿态尺度关系可以认为是相对固定的，可以事先校准(使用期偶尔需要重新校准，例如使用人调整了头戴式相机的位置之后需要重新校准位置姿态和尺度关系)。其中，获取的方式可以通过在与车载显示装置相对位置固定的地方贴定位标志物，头戴的外视相机通过定位标志物计算出和车载显示装置的相对位置姿态关系；获取的方式也可以是将本车上的车载显示装置作为场景中的物体，用上文描述的基于图像和/或多种传感器融合的提取特征点跟踪方式，和/或检测等方式的，如SLAM(Simultaneous Localization and Mapping，同步定位和建图)技术和目标跟踪技术。即(眼睛←校准→头戴式外置相机←估计→显示装置←校准→外视相机←估计→真实物体)；1.3) The device can use the external-view camera worn by the driver to obtain the relative position, attitude, and scale relationship between the vehicle-mounted display device and the external-view camera worn by the driver. Among them, the relative position and attitude scale relationship between the head-mounted external-view camera and the eyes can be considered relatively fixed, and can be calibrated in advance (occasionally need to be re-calibrated during use, for example, the user needs to re-calibrate the position after adjusting the position of the head-mounted camera posture and scale relations). Among them, the acquisition method can be by pasting a positioning marker on a fixed place relative to the vehicle display device, and the head-mounted external-view camera calculates the relative position and posture relationship with the vehicle display device through the positioning marker; the acquisition method can also be Using the vehicle-mounted display device on the vehicle as an object in the scene, use the above-described extraction feature point tracking method based on image and/or multiple sensor fusion, and/or detection methods, such as SLAM (Simultaneous Localization and Mapping , simultaneous positioning and mapping) technology and target tracking technology. That is (eye ← calibration → head-mounted external camera ← estimation → display device ← calibration → external camera ← estimation → real object);

1.4)设备可以使用本车上的内视相机(即拍摄本车内部环境，如驾驶人位置的相机)获取眼睛和本车上的内视相机的相对位置姿态尺度关系。其中获取的方式可以是基于驾驶人头戴的定位标志物，其中眼睛和头戴的定位标志物的相对位置姿态尺度关系可以认为是相对固定的，可以事先校准(使用期偶尔需要重新校准，例如使用人调整了头戴式定位标志物的位置之后需要重新校准位置姿态和尺度关系)；获取的方式还可以是基于图像的头部/眼睛/视线定位跟踪技术，设备通过本车上的内视相机的图像或视频定位驾驶人的头部，基于头部的定位结果定位眼睛和内视相机之间的相对位置姿态关系；获取的方式还可以是基于图像的眼睛定位跟踪技术，设备通过本车上的内视相机的图像或视频直接定位眼睛和内视相机之间的相对位置、姿态关系。本车上的内视相机和本车上的车载显示装置的相对位置、姿态、尺度关系相对固定，可以事先校准(使用期偶尔需要重新校准，例如车辆剧烈颠簸后，需要重新校准位置姿态和尺度关系)。即(眼睛←估计→本车上的内视相机←校准→显示装置←校准→本车上的外视相机←估计→真实物体)。1.4) The device can use the internal camera on the vehicle (that is, the camera that captures the internal environment of the vehicle, such as the driver's position) to obtain the relative position, posture, and scale relationship between the eyes and the internal camera on the vehicle. The method of acquisition can be based on the positioning markers worn by the driver, where the relative position, attitude, and scale relationship between the eyes and the positioning markers worn on the head can be considered to be relatively fixed, and can be calibrated in advance (recalibration is occasionally required during use, such as After the user adjusts the position of the head-mounted positioning marker, the position, posture and scale relationship need to be recalibrated); the acquisition method can also be based on the image-based head/eye/sight positioning tracking technology, and the device uses the internal vision on the vehicle The image or video of the camera locates the driver's head, and the relative position and posture relationship between the eyes and the internal view camera is located based on the positioning result of the head; the acquisition method can also be based on the image-based eye positioning and tracking technology. The image or video of the endoscopic camera directly locates the relative position and posture relationship between the eyes and the endoscopic camera. The relative position, attitude, and scale relationship between the internal camera on this vehicle and the on-vehicle display device on this vehicle are relatively fixed, and can be calibrated in advance (occasionally need to be recalibrated during the use period, for example, after the vehicle is violently bumped, the position, attitude, and scale need to be recalibrated relation). That is (eye ← estimation → internal camera on the vehicle ← calibration → display device ← calibration → external camera on the vehicle ← estimation → real object).

对于本发明实施例，设备除了确定AR信息的显示位置、姿态、尺度，仍需要确定AR信息的呈现方式，可以包括但不限于颜色，亮度，透明度以及图标形式等。对于需要和真实物体对齐的AR信息，设备优先以和真实物体一致的颜色和形式呈现AR信息。For the embodiment of the present invention, in addition to determining the display position, posture, and scale of the AR information, the device still needs to determine the presentation mode of the AR information, which may include but not limited to color, brightness, transparency, and icon form. For AR information that needs to be aligned with the real object, the device gives priority to presenting the AR information in the same color and form as the real object.

例如，若道路上的真实车道线应该为白色单虚线，该设备优先以白色单虚线的方式呈现AR车道线；若以和真实物体一致的颜色和形式呈现AR信息会导致AR信息无法被驾驶人清晰识别理解，则设备自适应的选择更优的颜色和形式。设备通过一个或多个外视相机采集道路图像或视频，将AR信息以预计呈现形式根据估计的位置姿态尺度投影到图像或视频上，通过图像视频分析技术，可以获取AR信息和周围场景的对比度，进而判断出预计的呈现形式是否合适，如不合适则根据场景亮度颜色等更换识别度更高的呈现方式。例如，对于道路被积雪部分遮挡的情况，以白色单虚线的方式呈现AR车道线会导致AR车道线和路面差异过小，设备可以自适应的选择，例如以蓝色单虚线的方式呈现AR车道线。For example, if the real lane lines on the road should be white single-dashed lines, the device will give priority to presenting AR lane lines in the form of white single-dashed lines; if the AR information is presented in the same color and form as the real object, the AR information will not be recognized by the driver. Clear recognition and understanding, the device adaptively chooses better colors and forms. The device collects road images or videos through one or more external cameras, and projects AR information onto the images or videos in the expected presentation form based on the estimated position and attitude scale. Through image and video analysis technology, the contrast between AR information and surrounding scenes can be obtained , and then determine whether the expected presentation form is suitable, and if not, replace it with a more recognizable presentation form according to the brightness and color of the scene. For example, when the road is partially covered by snow, presenting AR lane lines in the form of white single-dashed lines will cause the difference between AR lane lines and the road surface to be too small. The device can make adaptive choices, such as presenting AR in the form of blue single-dashed lines. lane line.

步骤S1401(图中未标注)、显示生成的目标行车辅助信息对应的AR信息。Step S1401 (not marked in the figure), displaying AR information corresponding to the generated target driving assistance information.

该步骤S1401可以是显示一个或多个目标行车辅助信息对应的虚拟三维AR信息的一种实现方式。This step S1401 may be an implementation manner of displaying virtual 3D AR information corresponding to one or more target driving assistance information.

对于本发明实施例，设备根据眼睛和真实物体之间的相对位置、姿态、尺度关系，将AR信息、以在步骤S1301确定的呈现方式显示在显示装置，从而可以将显示的AR信息和对应的真实物体对齐。For the embodiment of the present invention, the device displays the AR information on the display device in the presentation manner determined in step S1301 according to the relative position, posture, and scale relationship between the eyes and the real object, so that the displayed AR information and the corresponding Real object alignment.

实施例二Embodiment two

本发明实施例提供了一种行车辅助信息的显示方法，该方法用于在路面被完全覆盖遮挡，但中间车道隔离栏杆(或者其他可见交通标志障碍物)仍然可感知的道路，对车道线、车道、车辙、路面交通标志、路面交通标线，和路况等信息标识的提示信息进行显示并显示相应的增强现实辅助驾驶信息，如图3所示。其中，遮挡道路上车道线的可以是但不限于积雪、积水、泥污、油等，或组合。An embodiment of the present invention provides a method for displaying driving assistance information. The method is used for roads where the road surface is completely covered and blocked, but the middle lane separation railing (or other visible traffic sign obstacles) is still perceivable. The prompt information of information signs such as lanes, ruts, road traffic signs, road traffic markings, and road conditions are displayed and the corresponding augmented reality assisted driving information is displayed, as shown in Figure 3. Wherein, what blocks the lane line on the road may be but not limited to snow, water, mud, oil, etc., or a combination thereof.

本发明实施例的方法包括：The method of the embodiment of the present invention comprises:

步骤S1102(图中未标注)、设备确定是否需要显示道路信息。In step S1102 (not marked in the figure), the device determines whether road information needs to be displayed.

该步骤S1102可以是设备确定需要显示的一个或多个目标行车辅助信息的一种实现方式。This step S1102 may be an implementation manner for the device to determine one or more target driving assistance information to be displayed.

对于本发明实施例，设备通过图像识别的方式确定本车周围的道路信息，可以包括但不限于车道线、道路边缘线，以及路面交通和路况等信息标识。其中，设备可以是一直保持识别的状态，在道路被完全遮挡覆盖但中间车道隔离栏杆(或者其他可见交通标志障碍物)仍然可感知的路面情况自适应开启显示功能，也可以根据用户的指令开始识别和/或显示的功能。其中，用户指令可以是通过手势、语音、物理按键、指纹等生物识别标识来承载的。For the embodiment of the present invention, the device determines the road information around the vehicle through image recognition, which may include but not limited to lane lines, road edge lines, and information signs such as road traffic and road conditions. Among them, the device can always keep the state of recognition, and when the road is completely covered but the middle lane separation railing (or other visible traffic sign obstacles) is still perceivable, the display function can be adaptively turned on, or it can be started according to the user's instruction. Identify and/or display features. Among them, user instructions can be carried by gestures, voice, physical buttons, fingerprints and other biometric identifications.

步骤S1202(图中未标注)、设备感知道路信息，生成目标行车辅助信息内容。Step S1202 (not marked in the figure), the device senses road information, and generates target driving assistance information content.

该步骤S1202可以是获取信息，处理信息，生成目标行车辅助信息内容的一种实现方式。This step S1202 may be an implementation manner of acquiring information, processing information, and generating target driving assistance information content.

对于本发明实施例，设备检测识别定位中间车道隔离栏杆(或者其他可见交通标志障碍物)，估算本车行驶方向一侧的路面宽度，并估计车道线和道路边缘的位置。For the embodiment of the present invention, the device detects, identifies and locates the middle lane separation railing (or other visible traffic sign obstacles), estimates the road width on one side of the driving direction of the vehicle, and estimates the position of the lane line and the edge of the road.

具体地，首先，设备通过图像处理技术和识别技术从一个或多个图像/视频中识别定位出仍然可见的中间车道隔离栏杆(或者其他可见交通标志障碍物)，并以此为道路中间位置参照物；其次，当道路边缘也存在车道隔离栏杆(或者其他可见交通标志障碍物)时，设备也可以通过类似的技术识别定位出来。其中，道路边缘隔离栏杆(或者其他可见交通标志障碍物)与中间车道隔离栏杆(或者其他可见交通标志障碍物)之间的区域即为本车行驶方向行车区域。Specifically, first, the device uses image processing technology and recognition technology to identify and locate the still visible middle lane separation railing (or other visible traffic sign obstacles) from one or more images/videos, and use this as a reference for the middle position of the road Secondly, when there are lane separation railings (or other visible traffic sign obstacles) on the edge of the road, the device can also identify and locate them through similar technology. Wherein, the area between the road edge separation railing (or other visible traffic sign obstacles) and the middle lane separation railing (or other visible traffic sign obstacles) is the driving area in the driving direction of the vehicle.

对于本发明实施例，当道路边缘不存在车道隔离栏杆(或者其他可见交通标志障碍物)时，设备可以通过使用单相机、立体相机、深度相机、激光传感器、雷达传感器、超声传感器中至少之一获取中间车道隔离栏杆(或者其他可见交通标志障碍物)与车道外物体(例如自行车，树，房屋)的距离，在获取距离时考虑到相机和/或其他传感器的朝向，将测量距离校正到与中间车道线垂直的方向，即获取中间车道隔离栏杆(或者其他可见交通标志障碍物)与车道外物体(例如行人，自行车，树，房屋等)的鲁棒统计最短距离，此鲁棒统计最短距离应为本车在行进中在多个位置多次测量的统计距离。此鲁棒统计最短距离为本车行驶方向一侧的车道宽度上界限。当存在道路地图时，此上界限可以根据地图所示道路宽度进一步验证优化。For the embodiment of the present invention, when there is no lane separation railing (or other visible traffic sign obstacles) at the edge of the road, the device can use at least one of a single camera, a stereo camera, a depth camera, a laser sensor, a radar sensor, and an ultrasonic sensor Obtain the distance between the middle lane separation railing (or other visible traffic sign obstacles) and objects outside the lane (such as bicycles, trees, houses), take into account the orientation of the camera and/or other sensors when obtaining the distance, and correct the measured distance to the same The direction perpendicular to the middle lane line is to obtain the robust statistical shortest distance between the middle lane separation railing (or other visible traffic sign obstacles) and objects outside the lane (such as pedestrians, bicycles, trees, houses, etc.). This robust statistical shortest distance It should be the statistical distance measured multiple times at multiple positions by the vehicle during travel. This robust statistical shortest distance is the upper limit of the lane width on one side of the driving direction of the vehicle. When a road map is present, this upper bound can be further validated for optimization based on the road width shown in the map.

其中，相机和/或其他传感器的朝向可以由如下方式获得，设备首先从图像序列里识别出中间车道隔离栏杆(或者其他可见交通标志障碍物)的延伸方向，计算出相机和这个方向的朝向关系；相机和本车上的其他传感器的相对位置姿态尺度关系可以认为是固定的，可以事先校准(使用期偶尔需要重新校准，例如车辆剧烈颠簸后，需要重新校准位置姿态和尺度关系)，从而可以得到相机和/或其他传感器的朝向。Among them, the orientation of the camera and/or other sensors can be obtained in the following manner. The device first identifies the extension direction of the middle lane barrier (or other visible traffic sign obstacles) from the image sequence, and calculates the orientation relationship between the camera and this direction ; The relative position, attitude, and scale relationship between the camera and other sensors on the vehicle can be considered fixed, and can be calibrated in advance (occasionally need to be recalibrated during the use period, for example, after the vehicle is violently bumped, the position, attitude, and scale relationship need to be recalibrated), so that Get the orientation of the camera and/or other sensors.

对于本发明实施例，在车道宽度上界限内，基于道路交通的领域知识，和/或道路地图，设备可以根据车道线宽度和/或车道线数量等先验知识预测出每条车道线的大致位置以及道路边缘线的大致位置。其中道路地图的使用可以结合包括但不限于无线信号定位，GPS(Global Positioning System，全球定位系统)定位等方式确定本车所处地图范围；道路地图可以是事先在存储在设备或者本车的存储空间里的，也可以是通过网络通信的方式获取的。For the embodiment of the present invention, within the upper limit of the lane width, based on the domain knowledge of road traffic and/or road maps, the device can predict the approximate width of each lane line based on prior knowledge such as the width of the lane line and/or the number of lane lines. location and the approximate location of the road edge. The use of the road map can be combined with including but not limited to wireless signal positioning, GPS (Global Positioning System, Global Positioning System) positioning and other methods to determine the map range where the vehicle is located; the road map can be stored in the device or the storage of the vehicle in advance. In space, it can also be obtained through network communication.

对于本发明实施例，设备可以通过一个或多个相机，和/或立体相机，和/或深度相机采集周围车辆和被完全覆盖的道路路面的图像和/或视频，利用物体检测、物体识别和跟踪技术分析当前车辆预设范围内其他车辆的行车轨迹和/或路面上的车辙，更为精确的校正车道线和道路边缘线的预测位置；在有高精度GPS和高精度道路地图的时候，设备可以通过视觉定位信息和GPS定位信息将当前道路与道路地图对齐，提升车道线和道路边缘线位置的预测精度。其中车辙的识别检测方式详见实施例四。For the embodiment of the present invention, the device can collect images and/or videos of surrounding vehicles and fully covered road surfaces through one or more cameras, and/or stereo cameras, and/or depth cameras, and use object detection, object recognition and Tracking technology analyzes the trajectories of other vehicles and/or ruts on the road surface within the preset range of the current vehicle, and more accurately corrects the predicted position of lane lines and road edge lines; when there are high-precision GPS and high-precision road maps, The device can align the current road with the road map through visual positioning information and GPS positioning information to improve the prediction accuracy of lane lines and road edge lines. For the identification and detection method of ruts, see Embodiment 4 for details.

对于本发明实施例，在有高精度道路地图的时候，设备可以通过将感知到的道路环境通过视觉定位信息和/或GPS定位信息将当前道路与道路地图对齐，从而设备可以通过道路地图获取路面道路信息，并生成相应的目标行车辅助信息。For the embodiment of the present invention, when there is a high-precision road map, the device can align the current road with the road map by using the perceived road environment through visual positioning information and/or GPS positioning information, so that the device can obtain the road surface through the road map Road information, and generate corresponding target driving assistance information.

步骤S1302(图中未标注)、确定目标行车辅助信息内容的显示方式。Step S1302 (not marked in the figure), determining the display mode of the content of the target driving assistance information.

该步骤S1302可以是确定一个或多个目标行车辅助信息的显示方式的一种实现方式。This step S1302 may be an implementation of determining a display manner of one or more target driving assistance information.

类似于实施例一的步骤S1301，不再赘述。特别的，考虑到车道范围的估计可能存在误差，即估计得到的车道范围可能包括了一部分相邻的真实车道或道路边缘，为避免让车辆行驶在两条车道的分界区域，或车道和道路边缘的分界区域，设备将估计的车道区域的两侧向内收窄，只对估计车道区域的中间区域准备目标行车辅助信息。It is similar to step S1301 inEmbodiment 1, and will not be repeated here. In particular, considering that there may be errors in the estimation of the lane range, that is, the estimated lane range may include a part of the adjacent real lane or road edge. The device narrows both sides of the estimated lane area inward, and only prepares target driving assistance information for the middle area of the estimated lane area.

步骤S1402(图中未标注)、显示生成的目标行车辅助信息对应的AR信息。Step S1402 (not marked in the figure), displaying AR information corresponding to the generated target driving assistance information.

该步骤S1402可以是显示一个或多个目标行车辅助信息对应的虚拟三维AR信息的一种实现方式。This step S1402 may be an implementation manner of displaying virtual 3D AR information corresponding to one or more target driving assistance information.

类似于实施例一的步骤S1401，不再赘述。It is similar to step S1401 inEmbodiment 1, and will not be repeated here.

实施例三Embodiment Three

本实施例提供了一种行车辅助信息的显示方法，该方法用于在路面被完全覆盖遮挡，也不存在中间车道隔离栏杆(或者其他可见交通标志障碍物)的道路，对车道线、车道、车辙、路面交通标志，和路面交通标线和路况等信息标识的提示信息进行显示并显示相应的增强现实辅助驾驶信息。其中，遮挡道路上车道线的可以是但不限于积雪，积水，泥污，油等。This embodiment provides a method for displaying driving assistance information. This method is used for roads where the road surface is completely covered and blocked, and there is no middle lane separation railing (or other visible traffic sign obstacles). The prompt information of ruts, road traffic signs, road traffic markings and road conditions and other information signs are displayed and corresponding augmented reality assisted driving information is displayed. Wherein, what blocks the lane line on the road may be but not limited to snow, water, mud, oil and the like.

本发明实施例的方法包括：The method of the embodiment of the present invention comprises:

步骤S1103(图中未标注)、设备确定是否需要显示道路信息。Step S1103 (not marked in the figure), the device determines whether road information needs to be displayed.

该步骤S1103可以是设备确定需要显示的一个或多个目标行车辅助信息的一种实现方式。This step S1103 may be an implementation manner for the device to determine one or more target driving assistance information to be displayed.

对于本发明实施例，设备通过图像识别的方式确定本车周围的道路信息，可以包括但不限于车道线、道路边缘线，以及路面交通和路况等信息标识。其中，设备可以一直保持识别的状态，在道路被完全遮挡覆盖也不存在中间车道隔离栏杆(或者其他可见交通标志障碍物)的路面情况自适应开启显示功能，也可以根据用户的指令开始识别和/或显示的功能。其中，用户指令可以是通过手势、语音、物理按键、指纹等生物识别标识来承载的。For the embodiment of the present invention, the device determines the road information around the vehicle through image recognition, which may include but not limited to lane lines, road edge lines, and information signs such as road traffic and road conditions. Among them, the device can always keep the recognition state, and when the road is completely covered and there is no middle lane separation railing (or other visible traffic sign obstacles), the display function can be adaptively turned on, and the recognition and recognition can also be started according to the user's instructions. /or display function. Among them, user instructions can be carried by gestures, voice, physical buttons, fingerprints and other biometric identifications.

步骤S1203(图中未标注)、设备感知道路信息，生成目标行车辅助信息内容。Step S1203 (not marked in the figure), the device senses road information, and generates target driving assistance information content.

该步骤S1203可以是获取信息，处理信息，生成目标行车辅助信息内容的一种实现方式。This step S1203 may be an implementation manner of acquiring information, processing information, and generating target driving assistance information content.

对于本发明实施例，设备估算路面宽度，估计分隔双向车道的中间隔离线所在位置(如果不是单向行驶道路)，估计车道线和道路边缘的位置。For the embodiment of the present invention, the device estimates the width of the road surface, estimates the position of the median separating the two-way lanes (if it is not a one-way road), and estimates the positions of the lane line and the edge of the road.

首先，设备估算路面宽度。当道路两侧边缘存在车道隔离栏杆(或者其他可见交通标志障碍物)时，设备可以通过图像处理技术和识别技术从一个或多个图像/视频中定位识别出两侧的道路边缘隔离栏杆(或者其他可见交通标志障碍物)，两者之间的区域即为路面区域。First, the device estimates the road width. When there are lane separation railings (or other visible traffic sign obstacles) on both sides of the road, the device can locate and identify the road edge separation railings on both sides (or other visible traffic sign obstacles) from one or more images/videos through image processing technology and recognition technology Other visible traffic sign obstacles), the area between the two is the road surface area.

当只有一侧道路边缘存在车道隔离栏杆(或者其他可见交通标志障碍物)时，设备可以通过图像处理技术和识别技术识别定位出这一侧的道路边缘位置，作为参照物；对于不存在车道隔离栏杆(或者其他可见交通标志障碍物)的一侧，设备可以通过使用单相机、立体相机、深度相机、激光传感器、雷达传感器、超声传感器中至少一个获取参照物和另一侧道路边缘外的物体(例如自行车，树，房屋)的距离，获取距离时考虑到相机和/或其他传感器的朝向，将测量距离校正到和参照物垂直的方向，即获取存在隔离栏杆(或者其他可见交通标志障碍物)的一侧道路边缘和不存在隔离栏杆(或者其他可见交通标志障碍物)的另一侧道路边缘外物体(例如行人，自行车，树，房屋等)的鲁棒统计最短距离，此鲁棒统计最短距离应为本车在行进中在多个位置多次测量的统计距离，并且此鲁棒统计最短距离为路面宽度上界限。当存在道路地图时，此上界限可以根据地图所示道路宽度进一步验证优化。其中，相机和/或其他传感器的朝向可以由如下方式获得，设备首先从图像序列里识别出道路一侧的车道隔离栏杆(或者其他可见交通标志障碍物)的延伸方向，计算出相机和这个方向的朝向关系；相机和本车上的其他传感器的相对位置姿态尺度关系可以认为是固定的，可以事先校准(使用期偶尔需要重新校准，例如车辆剧烈颠簸后，需要重新校准位置姿态和尺度关系)，从而可以得到相机和/或其他传感器的朝向。When there is a lane separation railing (or other visible traffic sign obstacles) on only one side of the road edge, the device can use image processing technology and recognition technology to identify and locate the position of the road edge on this side as a reference; if there is no lane separation On one side of the railing (or other visible traffic sign obstacles), the device can use at least one of a single camera, a stereo camera, a depth camera, a laser sensor, a radar sensor, and an ultrasonic sensor to obtain reference objects and objects outside the edge of the road on the other side (such as bicycles, trees, houses), take into account the orientation of the camera and/or other sensors when obtaining the distance, and correct the measured distance to the direction perpendicular to the reference object, that is, obtain the separation railing (or other visible traffic sign obstacles) The shortest distance between the edge of the road on one side of the road and the object (such as pedestrians, bicycles, trees, houses, etc.) The shortest distance should be the statistical distance measured multiple times by the vehicle at multiple locations during travel, and this robust statistical shortest distance is the upper limit of the road width. When a road map is present, this upper bound can be further validated for optimization based on the road width shown in the map. Among them, the orientation of the camera and/or other sensors can be obtained in the following way. The device first recognizes the extension direction of the lane barrier (or other visible traffic sign obstacles) on the side of the road from the image sequence, and calculates the direction of the camera and this direction. orientation relationship; the relative position, attitude, and scale relationship between the camera and other sensors on the vehicle can be considered fixed, and can be calibrated in advance (occasionally need to be recalibrated during the use period, for example, after the vehicle is violently bumped, the position, attitude, and scale relationship need to be recalibrated) , so that the orientation of the camera and/or other sensors can be obtained.

当其中两侧道路边缘均不存在车道隔离栏杆(或者其他可见交通标志障碍物)时，如图4所示，设备可以通过使用单相机、立体相机、深度相机、激光传感器、雷达传感器、超声传感器中至少之一获取本车和两侧道路边缘外物体(例如行人，自行车，树，房屋等)的距离之和，获取距离之和时考虑到相机的朝向和本车两侧传感器之间的宽度，将测量距离校正到与道路方向垂直的方向，即获取两侧道路边缘外物体(例如行人，自行车，树，房屋等)之间的鲁棒统计最短距离，此鲁棒统计最短距离应为本车在行进中在多个位置多次测量的统计距离；此鲁棒统计最短距离为路面宽度上界限。其中，相机和/或其他传感器的朝向可以由如下方式获得。When there are no lane separation railings (or other visible traffic sign obstacles) on the edge of the road on both sides, as shown in Figure 4, the device can use a single camera, stereo camera, depth camera, laser sensor, radar sensor, ultrasonic sensor At least one of them obtains the sum of distances between the vehicle and objects outside the road edges on both sides (such as pedestrians, bicycles, trees, houses, etc.), taking into account the orientation of the camera and the width between the sensors on both sides of the vehicle when obtaining the sum of distances , to correct the measurement distance to the direction perpendicular to the road direction, that is, to obtain the robust statistical shortest distance between objects (such as pedestrians, bicycles, trees, houses, etc.) outside the road edge on both sides. This robust statistical shortest distance should be The statistical distance measured multiple times at multiple locations while the car is moving; this robust statistical shortest distance is the upper limit of the road width. Wherein, the orientation of the camera and/or other sensors may be obtained in the following manner.

具体地，设备首先从图像序列里识别出路边树木排列，和/或建筑物表面的延伸方向，计算出相机和这个方向的朝向关系；相机和本车上的其他传感器的相对位置姿态尺度关系可以认为是固定的，可以事先校准(使用期偶尔需要重新校准，例如车辆剧烈颠簸后，需要重新校准位置姿态和尺度关系)，从而可以得到相机和/或其他传感器的朝向。Specifically, the device first recognizes the arrangement of roadside trees and/or the extension direction of the building surface from the image sequence, and calculates the orientation relationship between the camera and this direction; the relative position, attitude, and scale relationship between the camera and other sensors on the vehicle can be It is considered fixed and can be calibrated in advance (recalibration is occasionally required during the use period, for example, the position, attitude and scale relationship need to be recalibrated after the vehicle is violently bumped), so that the orientation of the camera and/or other sensors can be obtained.

对于本发明实施例，如图5所示，在估计得路面范围上界限内，基于道路交通的领域知识，和/或道路地图，设备可以根据各方向的车道线宽度和/或车道线数量预测出道路中间线的大致位置，并以此中间线为参照，估计出每条车道线的大致位置以及道路边缘线的大致位置；For the embodiment of the present invention, as shown in Figure 5, within the estimated upper limit of the road surface, based on the domain knowledge of road traffic and/or road maps, the device can predict the width of lane lines and/or the number of lane lines in each direction Find out the approximate position of the middle line of the road, and use this middle line as a reference to estimate the approximate position of each lane line and the approximate position of the road edge line;

考虑到如果估计的车道范围是正确的，那么这个范围不会同时存在两个方向行驶的车辆或车辙，所以设备可以通过一个或多个相机，和/或立体相机，和/或深度相机采集周围车辆和被完全覆盖的道路路面的图像和/或视频，利用物体检测、物体识别和跟踪技术分析当前车辆预设范围内其他车辆的行车轨迹和/或路面上的车辙，更为精确的校正车道线和道路边缘线的预测位置；在有高精度GPS和高精度道路地图的时候，设备可以通过视觉定位信息和GPS定位信息将当前道路与道路地图上对齐，提升车道线和道路边缘线位置的估计精度。其中车辙的识别检测方式详见实施例四。Considering that if the estimated lane range is correct, there will not be vehicles or ruts traveling in two directions at the same time in this range, so the device can use one or more cameras, and/or stereo cameras, and/or depth cameras to capture the surrounding Images and/or videos of the vehicle and the fully covered road surface, using object detection, object recognition and tracking technology to analyze the trajectories of other vehicles within the preset range of the current vehicle and/or ruts on the road surface, and correct the lane more accurately Predicted positions of lane lines and road edge lines; when there are high-precision GPS and high-precision road maps, the device can align the current road with the road map through visual positioning information and GPS positioning information, and improve the accuracy of lane lines and road edge lines. Estimated accuracy. For the identification and detection method of ruts, see Embodiment 4 for details.

对于本发明实施例，在有高精度道路地图的时候，设备可以通过将感知到的道路环境通过视觉定位信息和/或GPS定位信息将当前道路与道路地图对齐，从而设备可以通过道路地图获取路面道路信息，并生成相应的目标行车辅助信息。For the embodiment of the present invention, when there is a high-precision road map, the device can align the current road with the road map by using the perceived road environment through visual positioning information and/or GPS positioning information, so that the device can obtain the road surface through the road map Road information, and generate corresponding target driving assistance information.

步骤S1303(图中未标注)、确定目标行车辅助信息内容的显示方式。Step S1303 (not marked in the figure), determine the display mode of the content of the target driving assistance information.

该步骤S1303可以是确定一个或多个目标行车辅助信息的显示方式的一种实现方式。This step S1303 may be an implementation manner of determining a display manner of one or more target driving assistance information.

类似于实施例一的步骤S1301，不再赘述。特别的，考虑到车道范围的估计可能存在误差，即估计得车道范围可能包括了一部分相邻的真实车道或道路边缘，为避免让车辆行驶在两条车道的分界区域，或车道和道路边缘的分界区域，设备将估计的车道区域的两侧向内收窄，只对估计车道区域的中间区域准备目标行车辅助信息。It is similar to step S1301 inEmbodiment 1, and will not be repeated here. In particular, considering that there may be errors in the estimation of the lane range, that is, the estimated lane range may include a part of the adjacent real lane or road edge. In the boundary area, the device narrows both sides of the estimated lane area inward, and only prepares target driving assistance information for the middle area of the estimated lane area.

步骤S1403(图中未标注)、显示生成的目标行车辅助信息对应的AR信息。Step S1403 (not marked in the figure), displaying AR information corresponding to the generated target driving assistance information.

该步骤S1403可以是显示一个或多个目标行车辅助信息对应的虚拟三维AR信息的一种实现方式。This step S1403 may be an implementation manner of displaying virtual 3D AR information corresponding to one or more target driving assistance information.

类似于实施例一的步骤S1401，不再赘述。It is similar to step S1401 inEmbodiment 1, and will not be repeated here.

实施例四Embodiment four

本发明实施例提供了一种行车辅助信息的显示方法，该方法用于在路面被完全覆盖遮挡的道路，对车辙信息标识的提示信息进行显示并显示相应的增强现实辅助驾驶显示信息。其中，遮挡道路上车道线的可以是但不限于积雪，积水，泥污，油等。An embodiment of the present invention provides a method for displaying driving assistance information. The method is used to display the prompt information of the rut information mark and display the corresponding augmented reality assisted driving display information on a road that is completely covered and blocked by the road surface. Wherein, what blocks the lane line on the road may be but not limited to snow, water, mud, oil and the like.

本发明实施例的方法包括：The method of the embodiment of the present invention comprises:

步骤S1104(图中未标注)、设备确定是否需要显示车辙相关信息。Step S1104 (not marked in the figure), the device determines whether it is necessary to display rut related information.

该步骤S1104可以是设备确定需要显示的一个或多个目标行车辅助信息的一种实现方式。This step S1104 may be an implementation manner for the device to determine one or more target driving assistance information to be displayed.

对于本发明实施例，设备通过图像识别的方式确定本车周围的道路信息，可以包括但不限于车道线、道路边缘线、路面交通标志，和路面交通标线和路况等信息标识。其中，设备可以是一直保持检测识别的状态，在道路被完全遮挡覆盖的路面情况自适应开启显示功能，也可以根据用户的指令开始识别和/或显示的功能。其中，用户指令可以是通过手势、语音、物理按键、指纹等生物识别标识来承载的。For the embodiment of the present invention, the device determines the road information around the vehicle through image recognition, which may include but not limited to lane lines, road edge lines, road traffic signs, road traffic markings and road conditions. Among them, the device can always keep the state of detection and recognition, adaptively turn on the display function when the road is completely covered by the road surface, or start the recognition and/or display function according to the user's instruction. Among them, user instructions can be carried by gestures, voice, physical buttons, fingerprints and other biometric identifications.

步骤S1204(图中未标注)、设备检测路面车辙。Step S1204 (not marked in the figure), the device detects road rutting.

该步骤S1204可以是获取信息，处理信息，生成目标行车辅助信息内容的一种实现方式。This step S1204 may be an implementation manner of acquiring information, processing information, and generating target driving assistance information content.

对于本发明实施例，首先设备使用一个或多个相机采集路面图像或者视频，使用图像处理技术和识别技术检测定位路面上的车辙。For the embodiment of the present invention, first, the device uses one or more cameras to collect road surface images or videos, and uses image processing technology and recognition technology to detect and locate ruts on the road surface.

具体地，在单幅图像中，设备可以使用图像识别技术定位车辙区域，车辙可以通过图像边缘提取算法和/或颜色聚类算法进行提取。设备可以将检测到的车辙边缘线段根据方向连接成连续的车辙边缘线，也可以将车辙边缘线段构建成向量场；当同时有空间上的多幅图像，和/或时间上的多幅图像时，多幅图像可以协同识别车辙，并把车辙边缘线在多幅图像中通过特征跟踪和模式匹配的方式连接起来，去除使用单幅图像时的错误，并使得识别结果保持空间，和/或时间上的一致。模式匹配时可以考虑到车辙的遗留时间长短(例如从车辙深浅推断遗留时间)去除错误匹配。Specifically, in a single image, the device can use image recognition technology to locate the rut area, and the rut can be extracted by using an image edge extraction algorithm and/or a color clustering algorithm. The device can connect the detected rut edge line segments into continuous rut edge lines according to the direction, and can also construct the rut edge line segments into a vector field; when there are multiple images in space and/or multiple images in time , multiple images can collaboratively identify ruts, and connect the rut edge lines in multiple images through feature tracking and pattern matching, remove errors when using a single image, and keep the recognition results in space and/or time Consistent with above. During pattern matching, the remaining time of the ruts can be considered (for example, the remaining time can be inferred from the depth of the ruts) to remove false matches.

步骤S1304(图中未标注)、确定车辙相关行车辅助信息内容的显示方式。Step S1304 (not marked in the figure), determining the display mode of the rut-related driving assistance information content.

该步骤S1304可以是确定一个或多个目标行车辅助信息的显示方式的一种实现方式。This step S1304 may be an implementation of determining a display manner of one or more target driving assistance information.

对于本发明实施例，设备判断是否存在异常车辙。其中异常车辙可以通过车辙边缘线构建出的向量场是否平滑，大体趋势和车道线方向以及道路边缘线是否一致来判断；也可以判断通过某一时刻的生成的车辙边缘线构建的向量场是否显著区别于其他/整体车辙边缘线构建的向量场的方向；若有显著区别，则确定生成的车辙边缘线属于异常车辙边缘线；通常异常车辙相对于车道线和道路边缘有明显的方向冲突，且很可能存在刹车痕迹。当存在异常车辙的时，设备用警示的方式增强显示异常车辙所在的路面区域，并生成行车警告AR信息，如图6所示；当不存在异常车辙的时候，设备根据车辙在图像或视频上的颜色的对比度和/或边缘的清晰度，判断车辙是否明显可见。如清晰可见，则不增强显示；对于所有车辙均不明显可见的路面，设备根据本车的行车路线选择相对最为明显的车辙，进行增强显示，如图7所示。特别的，设备根据本车行驶状态(如速度等)和路面环境(如路面是否湿滑)等因素，提前检测观察前方足够距离的路面，动态调整预警时间，为驾驶人预留足够的反应时间。For the embodiment of the present invention, the device judges whether there is an abnormal rut. Among them, the abnormal rut can be judged by whether the vector field constructed by the rut edge line is smooth, whether the general trend is consistent with the direction of the lane line and the road edge line; it can also be judged whether the vector field constructed by the generated rut edge line at a certain moment is significant It is different from the direction of the vector field constructed by other/overall rut edge lines; if there is a significant difference, it is determined that the generated rut edge line belongs to the abnormal rut edge line; usually the abnormal rut has obvious direction conflict with respect to the lane line and the road edge, and Most likely there are brake marks. When there are abnormal ruts, the device uses a warning method to enhance the display of the road area where the abnormal ruts are located, and generates driving warning AR information, as shown in Figure 6; when there are no abnormal ruts, the device uses the ruts on the image or video The contrast of the colors and/or the sharpness of the edges will determine whether the ruts are clearly visible. If it is clearly visible, the display will not be enhanced; for the road surface where all ruts are not clearly visible, the device will select the relatively most obvious rut according to the driving route of the vehicle for enhanced display, as shown in Figure 7. In particular, according to factors such as the driving state of the vehicle (such as speed, etc.) and the road environment (such as whether the road is slippery), the device detects and observes the road ahead with a sufficient distance in advance, dynamically adjusts the warning time, and reserves enough reaction time for the driver .

其中，用于增强显示异常车辙所在路面区域和增强显示不明显车辙的AR物体需要和真实路面和真实车辙对齐，显示方式类似于实施例一的步骤S1301，不再赘述。Wherein, the AR objects used to enhance the display of the road surface area where the abnormal rut is located and the enhanced display of the non-obvious rut need to be aligned with the real road surface and the real rut.

对于本发明实施例，对于异常车辙警告的AR信息不需要和真实物体对齐。当显示装置为头戴式显示装置时，设备可以直接将AR信息以醒目的方式呈现在驾驶人的当前视野的显著位置，且根据驾驶人的聚焦深度确定显示方式，且AR信息正面对于驾驶人的视线，驾驶人眼睛的聚焦深度可以通过视线跟踪的方式获得；当显示装置为车载显示装置时，设备可以将AR信息以醒目的方式呈现在显示屏幕的区域内，根据驾驶人的视线，将AR信息的姿态和深度设置为正面对于驾驶人的视线，位于驾驶人眼睛的聚焦深度，如图8所示。并且设备可以通过配合动画、音效等方式吸引驾驶人注意力，降低延迟。For the embodiment of the present invention, the AR information for abnormal rut warning does not need to be aligned with the real object. When the display device is a head-mounted display device, the device can directly present the AR information in a prominent position in the driver's current field of vision in an eye-catching manner, and the display method is determined according to the driver's focus depth, and the AR information faces the driver directly. line of sight, the focus depth of the driver's eyes can be obtained through line of sight tracking; when the display device is a vehicle-mounted display device, the device can present AR information in the area of the display screen in an eye-catching way, and according to the driver's line of sight, the The posture and depth of the AR information are set to face the driver's line of sight in the front, at the focus depth of the driver's eyes, as shown in Figure 8. And the device can attract the driver's attention by cooperating with animation, sound effects, etc., and reduce the delay.

步骤S1404(图中未标注)、显示生成的目标行车辅助信息对应的AR信息。Step S1404 (not marked in the figure), displaying AR information corresponding to the generated target driving assistance information.

该步骤S1404可以是显示一个或多个目标行车辅助信息对应的虚拟三维AR信息的一种实现方式。This step S1404 may be an implementation manner of displaying virtual 3D AR information corresponding to one or more target driving assistance information.

类似于实施例一的步骤S1401，不再赘述。It is similar to step S1401 inEmbodiment 1, and will not be repeated here.

实施例五Embodiment five

本发明实施例提供了一种行车辅助信息的显示方法，该方法用于在交通标志被部分或完全覆盖遮挡的道路，对被遮挡的交通标志的提示信息进行显示，并显示相应的增强现实辅助驾驶显示信息，如图9所示。其中，遮挡交通标志的可以是但不限于积雪，泥污，油，印刷品，树叶等。其中，遮挡可以广义的理解为损坏，或脱落，或油漆退色，或雨/雾/霾，或传感器处于不理想状态(如镜头污渍)导致的不完整清晰可见。An embodiment of the present invention provides a method for displaying driving assistance information. The method is used to display the prompt information of the blocked traffic sign on a road where the traffic sign is partially or completely covered, and display the corresponding augmented reality assistance. Driving display information, as shown in Figure 9. Wherein, what blocks the traffic sign may be but not limited to snow, mud, oil, printed matter, leaves and the like. Among them, occlusion can be broadly understood as damage, or peeling, or paint fading, or rain/fog/haze, or incomplete and clear visibility caused by the sensor being in an unsatisfactory state (such as lens stains).

本发明实施例的方法包括：The method of the embodiment of the present invention comprises:

步骤S1105(图中未标注)、设备确定是否需要显示交通标志相关信息。Step S1105 (not marked in the figure), the device determines whether it is necessary to display traffic sign related information.

该步骤S1105可以是设备确定需要显示的一个或多个目标行车辅助信息的一种实现方式。This step S1105 may be an implementation manner for the device to determine one or more target driving assistance information to be displayed.

对于本发明实施例，设备通过图像识别的方式确定本车周围的道路信息，可以包括但不限于车道线、道路边缘线，以及路面交通和路况等信息标识。其中，设备可以是一直保持识别的状态，在出现被部分或者完全遮挡交通标志和/或指示标志的路面情况自适应开启显示功能，也可以根据用户的指令开始识别和/或显示的功能。其中，用户指令可以是通过手势、语音、物理按键、指纹等生物识别标识来承载的。For the embodiment of the present invention, the device determines the road information around the vehicle through image recognition, which may include but not limited to lane lines, road edge lines, and information signs such as road traffic and road conditions. Among them, the device can always keep the recognition state, adaptively turn on the display function when there are road conditions that are partially or completely blocked by traffic signs and/or indicator signs, or start the recognition and/or display function according to the user's instruction. Among them, user instructions can be carried by gestures, voice, physical buttons, fingerprints and other biometric identifications.

步骤S1205(图中未标注)、设备判断是否需要增强显示。Step S1205 (not marked in the figure), the device judges whether enhanced display is required.

该步骤S1205可以是获取信息，处理信息，生成目标行车辅助信息内容的一种实现方式。This step S1205 may be an implementation manner of acquiring information, processing information, and generating target driving assistance information content.

对于本发明实施例，设备使用一个或多个相机采集路面图像或者视频，使用图像处理技术和识别技术检测道路两侧和上方的交通标志。设备通过图像或视频分析判断交通标志上的内容是否清晰完整。具体的，设备可以通过图像检测技术检测交通标志在图像上的位置和包围框(通常为矩形框/圆形框/三角形框等)。通过判断包围框内的图像是否清晰锐利，和/或颜色分布，判断内容是否清晰。对于不清晰的交通标志，设备可以通过如下方式至少之一获得交通标志的完整信息和图标形式：设备可以根据本车所在位置检索本地地图对应的数据库，通过模式匹配采集的图像和数据库获取交通标志的完整信息和图标形式；设备可以利用图像算法对不清晰的图像进行图像增强(例如，对于由于雾遮档的交通标志，可用图像去雾算法进行图像增强，获得相对清晰的图像)，获得交通标志的完整信息和图标形式；设备可以通过从其他角度获得的交通标志的图像中获取相应的完整信息和图标形式；设备可以通过其他的交通标志的信息，和/或行驶信息，获得完整信息和图标形式，例如，根据之前遇到的交通标志内容为“距离出口200米”和本车行驶了100米的行驶信息，可推断出当前的交通标志内容为“距离出口100米”。For the embodiment of the present invention, the device uses one or more cameras to collect road images or videos, and uses image processing technology and recognition technology to detect traffic signs on both sides and above the road. The device judges whether the content on the traffic sign is clear and complete through image or video analysis. Specifically, the device may detect the position and bounding frame (usually a rectangular frame/circular frame/triangular frame, etc.) of the traffic sign on the image through an image detection technology. Determine whether the content is clear by judging whether the image in the bounding box is clear and sharp, and/or the color distribution. For unclear traffic signs, the device can obtain the complete information and icon form of the traffic signs through at least one of the following methods: the device can retrieve the database corresponding to the local map according to the location of the vehicle, and obtain the traffic signs by pattern matching the collected images and the database The complete information and icon form; the device can use the image algorithm to enhance the image of the unclear image (for example, for the traffic sign blocked by the fog, the image can be enhanced by the image defogging algorithm to obtain a relatively clear image), and the traffic The complete information and icon form of the sign; the device can obtain the corresponding complete information and icon form from the image of the traffic sign obtained from other angles; the device can obtain the complete information and icon form from the information of other traffic signs and/or driving information In the form of icons, for example, according to the previously encountered traffic sign content as "200 meters from the exit" and the driving information that the vehicle has traveled 100 meters, it can be deduced that the current traffic sign content is "100 meters from the exit".

步骤S1305(图中未标注)、确定目标行车辅助信息内容的显示方式。Step S1305 (not marked in the figure), determining the display mode of the content of the target driving assistance information.

该步骤S1305可以是确定一个或多个目标行车辅助信息的显示方式的一种实现方式。This step S1305 may be an implementation manner of determining a display manner of one or more target driving assistance information.

类似于实施例一的步骤S1301，不再赘述。特别的，生成的AR交通标志需要和真实的交通标志对齐。设备通过图像定位算法获取真实物体相对于显示装置的位置和姿态。具体的，当使用单个相机获取本车周围的交通标志时，近似的，考虑到交通标志可以近似认为是一个平面，可以对交通标志轮廓提取特征点，通过求解单应性矩阵获取交通标志和相机的相对位置和姿态关系；更为精确的，可以使用视觉里程计的方式对图像序列进行特征跟踪，其中特征采集自交通标志的轮廓等需要对齐的真实物体，从而获取真实物体相对于相机的相对位置和姿态关系。特别的，特征点的提取和跟踪可以用图像识别分割辅助，从而去除错误匹配并加快速度。It is similar to step S1301 inEmbodiment 1, and will not be repeated here. In particular, the generated AR traffic signs need to be aligned with real traffic signs. The device obtains the position and attitude of the real object relative to the display device through the image positioning algorithm. Specifically, when using a single camera to obtain traffic signs around the vehicle, approximately, considering that traffic signs can be approximately considered as a plane, feature points can be extracted from the outline of traffic signs, and traffic signs and cameras can be obtained by solving the homography matrix The relative position and attitude relationship of the image; more precisely, visual odometry can be used to track the features of the image sequence, in which the features are collected from real objects that need to be aligned, such as the outline of traffic signs, so as to obtain the relative position of the real object relative to the camera. position and attitude relationship. In particular, the extraction and tracking of feature points can be assisted by image recognition segmentation to remove false matches and speed up.

步骤S1405(图中未标注)、显示生成的目标行车辅助信息对应的AR信息。Step S1405 (not marked in the figure), displaying AR information corresponding to the generated target driving assistance information.

该步骤S1405可以是显示一个或多个目标行车辅助信息对应的虚拟三维AR信息的一种实现方式。This step S1405 may be an implementation manner of displaying virtual 3D AR information corresponding to one or more target driving assistance information.

类似于实施例一的步骤S1401，不再赘述。It is similar to step S1401 inEmbodiment 1, and will not be repeated here.

实施例六Embodiment six

本发明实施例提供了一种行车辅助信息的显示方法，该方法用于对已经驶过区域的交通标志的提示信息进行显示并显示相应的增强现实辅助驾驶显示信息。An embodiment of the present invention provides a method for displaying driving assistance information. The method is used for displaying prompt information of traffic signs that have passed an area and displaying corresponding augmented reality assisted driving display information.

本发明实施例的方法包括：The method of the embodiment of the present invention comprises:

步骤S1106(图中未标注)、设备确定是否需要显示交通标志相关信息。Step S1106 (not marked in the figure), the device determines whether it is necessary to display traffic sign related information.

该步骤S1106可以是设备确定需要显示的一个或多个目标行车辅助信息的一种实现方式。This step S1106 may be an implementation manner for the device to determine one or more target driving assistance information to be displayed.

对于本发明实施例，设备通过用户的指令开始或者结束显示的功能。其中，用户指令可以是通过手势、语音、物理按键、指纹等生物识别标识来承载的；设备也可以是根据统计用户视线在AR交通标志停留的时间自适应的结束显示；也可以是两种方式的组合。For the embodiment of the present invention, the device starts or ends the display function through a user's instruction. Among them, user instructions can be carried by gestures, voices, physical keys, fingerprints and other biometric identifications; the device can also be based on the statistics of the time the user's sight stays on the AR traffic signs, and the end display can be self-adaptive; there can also be two ways The combination.

步骤S1206(图中未标注)、设备生成需要显示的内容。Step S1206 (not marked in the figure), the device generates content to be displayed.

该步骤S1206可以是获取信息，处理信息，生成目标行车辅助信息内容的一种实现方式。This step S1206 may be an implementation manner of acquiring information, processing information, and generating target driving assistance information content.

对于本发明实施例，设备检索本车过去一段时间内经过的交通标志。设备既可以检索所有经过的交通标志，也可以根据用户指令里的关键词检索相关的交通标志。在有高精度道路地图的时，设备可以根据本车当前定位在地图里检索刚刚经过区域内的交通标志；当没有地图时，设备使用一个或多个相机采集的历史路面图像或者视频，使用图像处理技术和检测识别技术检测识别道路两侧和上方的交通标志，获取符合检索要求的交通标志。设备从检索到的交通标志中提取交通标志的完整信息和图标形式。特别的，设备可以根据当前位置调整交通标志里的具体内容。例如，原来交通标志内容为“距离高速公路出口5公里”，但由于本车相对交通标志已经行进了300米，设备可以因此将指示标志的内容改为“距离高速公路出口4.7公里”，如图10所示，以适应当前情况。特别的，设备可以检索，和/或生成一个或多个交通标志。For the embodiment of the present invention, the device retrieves the traffic signs passed by the vehicle in the past period of time. The device can not only retrieve all passing traffic signs, but also retrieve relevant traffic signs according to keywords in user instructions. When there is a high-precision road map, the device can retrieve the traffic signs in the area just passed in the map according to the current location of the vehicle; when there is no map, the device uses the historical road images or videos collected by one or more cameras, and uses the image Processing technology and detection and recognition technology detect and recognize traffic signs on both sides and above the road, and obtain traffic signs that meet the retrieval requirements. The device extracts the complete information and icon form of the traffic sign from the retrieved traffic signs. In particular, the device can adjust the specific content of traffic signs according to the current location. For example, the original content of the traffic sign is "5 kilometers from the expressway exit", but since the car has traveled 300 meters relative to the traffic sign, the device can change the content of the sign to "4.7 kilometers from the expressway exit", as shown in the figure 10 to suit the current situation. In particular, the device may retrieve, and/or generate one or more traffic signs.

步骤S1306(图中未标注)、确定目标行车辅助信息内容的显示方式。Step S1306 (not marked in the figure), determining the display mode of the content of the target driving assistance information.

该步骤S1306可以是确定一个或多个目标行车辅助信息的显示方式的一种实现方式。This step S1306 may be an implementation manner of determining a display manner of one or more target driving assistance information.

类似于实施例一的步骤S1301，不再赘述。特别的，当检索到的交通标志多余一个，设备既可以同时显示多个交通标志，也可以依次轮流显示。设备优先以真实交通标志的形式呈现相应的AR交通标志。由于不需要和真实物体对齐，设备优先将AR交通标志根据驾驶人的注视点距离显示，且靠近天空部分，避免妨碍驾驶人观察真实路面。It is similar to step S1301 inEmbodiment 1, and will not be repeated here. In particular, when more than one traffic sign is retrieved, the device can display multiple traffic signs at the same time, or display them in turn. The device preferentially presents the corresponding AR traffic signs in the form of real traffic signs. Since it does not need to be aligned with real objects, the device gives priority to displaying AR traffic signs according to the distance of the driver's gaze point, and close to the sky to avoid obstructing the driver's observation of the real road.

步骤S1406(图中未标注)、显示生成的目标行车辅助信息对应的AR信息。Step S1406 (not marked in the figure), displaying AR information corresponding to the generated target driving assistance information.

该步骤S1406可以是显示一个或多个目标行车辅助信息对应的虚拟三维AR信息的一种实现方式。This step S1406 may be an implementation manner of displaying virtual 3D AR information corresponding to one or more target driving assistance information.

类似于实施例一的步骤S1401，不再赘述。It is similar to step S1401 inEmbodiment 1, and will not be repeated here.

实施例七Embodiment seven

本发明实施例提供了一种行车辅助信息的显示方法，该方法用于对本车两侧的侧方后视镜的扩展区域进行显示并显示相应的增强现实辅助驾驶显示信息。An embodiment of the present invention provides a method for displaying driving assistance information, which is used to display the extended areas of side rearview mirrors on both sides of the vehicle and display corresponding augmented reality assisted driving display information.

本发明实施例的方法包括：The method of the embodiment of the present invention comprises:

步骤S1107(图中未标注)、设备确定是否需要显示侧方后视镜的扩展区域相关信息。Step S1107 (not marked in the figure), the device determines whether it is necessary to display information about the extended area of the side rearview mirror.

该步骤S1107可以是设备确定需要显示的一个或多个目标行车辅助信息的一种实现方式。This step S1107 may be an implementation manner for the device to determine one or more target driving assistance information to be displayed.

对于本发明实施例，设备通过用户的指令开始或者结束显示的功能。其中，用户指令可以是通过手势、语音、物理按键、指纹等生物识别标识来承载的；设备也可以通过检测驾驶人的注视点是否在侧方后视镜，和/或侧方后视镜是否在驾驶人的视野内来自适应的判断是否开始或者结束显示的功能；也可以是两种方式的组合。当使用的相机是头戴式显示装置上的外视相机时，检测的方式可以是通过检测图像中是否有侧方后视镜来判断侧方后视镜是否在驾驶人的视野内，继而设备可以通过视线追踪的方式获得驾驶人当前的注视区域，判断注视点是否在侧方后视镜区域；当使用的相机是本车的内视相机时，设备可以通过检测驾驶人头部的朝向，和/或眼睛的朝向，和/或视线的方向判断侧方后视镜是否在驾驶人的视野范围内，以及驾驶人的视线聚焦区域。For the embodiment of the present invention, the device starts or ends the display function through a user's instruction. Among them, user instructions can be carried by gestures, voice, physical buttons, fingerprints and other biometric identifications; the device can also detect whether the driver's gaze is on the side rearview mirror, and/or whether the side rearview mirror The function of adaptively judging whether to start or end the display within the driver's field of vision; it can also be a combination of the two methods. When the camera used is an external-view camera on a head-mounted display device, the detection method may be to determine whether the side rear-view mirror is in the driver's field of vision by detecting whether there is a side rear-view mirror in the image, and then the device The driver's current gaze area can be obtained through eye-tracking, and it can be judged whether the gaze point is in the side rearview mirror area; when the camera used is the interior camera of the vehicle, the device can detect the driver's head orientation, And/or the orientation of the eyes, and/or the direction of the line of sight to determine whether the side rearview mirror is within the driver's field of vision, and the focus area of the driver's line of sight.

步骤S1207(图中未标注)、设备生成需要显示的内容。Step S1207 (not marked in the figure), the device generates content to be displayed.

该步骤S1207可以是获取信息，处理信息，生成目标行车辅助信息内容的一种实现方式。This step S1207 may be an implementation manner of acquiring information, processing information, and generating target driving assistance information content.

首先，设备估计驾驶人头部/眼睛/视线和侧方后视镜的相对位置和姿态。First, the device estimates the relative position and attitude of the driver's head/eyes/sight and side mirrors.

具体的，当使用的相机是头戴式显示装置上的外视相机时，此外视相机和驾驶人头部相对位置固定，与眼睛的相对位置姿态尺度关系可事先标定(使用期偶尔需要重新校准，例如使用人调整了头戴式显示装置的位置之后需要重新校准位置姿态和尺度关系)，因此只需要获得侧方后视镜和此外视相机的相对位置关系。具体的，设备可以通过图像识别技术分割出侧方后视镜，侧方后视镜可以被看作是一个尺度固定的平面，因此通过求解单应性矩阵获取侧方后视镜和相机的相对位置和姿态关系；设备也可以使用视觉里程计的方式对侧方后视镜的图像序列进行特征跟踪，其中特征采集侧方后视镜的边缘轮廓，从而获取平滑的侧方后视镜相对于相机的相对位置和姿态关系；设备也可以在侧方后视镜屏幕上贴定位标志物，设备上的外视相机通过定位标志物计算出侧方后视镜和本车外视相机的相对位置姿态尺度关系；即(眼睛←校准→显示装置←校准→设备上的外视相机←估计→侧方后视镜)。Specifically, when the camera used is the external-view camera on the head-mounted display device, the relative position of the external-view camera and the driver's head is fixed, and the relative position, attitude, and scale relationship with the eyes can be calibrated in advance (recalibration is occasionally required during use) , for example, after the user adjusts the position of the head-mounted display device, the position, posture and scale relationship need to be recalibrated), so only the relative position relationship between the side rearview mirror and the external view camera needs to be obtained. Specifically, the device can segment the side rearview mirror through image recognition technology. The side rearview mirror can be regarded as a plane with a fixed scale. Therefore, the relative relationship between the side rearview mirror and the camera can be obtained by solving the homography matrix. The position and attitude relationship; the device can also use the visual odometer to perform feature tracking on the image sequence of the side rearview mirror, in which the feature collects the edge contour of the side rearview mirror, so as to obtain a smooth side rearview mirror relative to The relative position and posture relationship of the camera; the device can also attach positioning markers on the side rearview mirror screen, and the exterior camera on the device can calculate the relative position of the side rearview mirror and the car's exterior camera through the positioning markers Attitude-scale relationship; ie (eye ← calibration → display device ← calibration → external looking camera on device ← estimate → side mirror).

当使用的相机是本车上的内视相机时，内视相机和侧方后视镜的相对位置姿态关系可以认为是固定的，可以事先标定(使用期偶尔需要重新校准，例如驾驶人调节侧方后视镜后，需要重新校准位置姿态和尺度关系)。因此只需要获得内视相机和显示设备的相对位置姿态关系。具体的，设备可以使用本车上的内视相机(即拍摄本车内部环境，如驾驶人位置，的相机)获取显示装置和本车上的内视相机的相对位置姿态尺度关系。获取的方式可以基于显示装置上的定位标志物，和/或基于图像和/或多种传感器融合的提取特征点跟踪方式，和/或检测等方式，如SLAM(Simultaneous Localization and Mapping，同步定位和建图)技术和目标跟踪技术。眼睛和显示装置的相对位置和姿态关系相对固定，可以事先校准(使用期偶尔需要重新校准，例如使用人调整了头戴式显示设备的位置之后需要重新校准位置姿态和尺度关系)；即(眼睛←校准→显示装置←估计→本车上的内视相机←校准→侧方后视镜)。When the camera used is the interior camera on the vehicle, the relative position and attitude relationship between the interior camera and the side rearview mirror can be considered fixed and can be calibrated in advance (occasionally need to be recalibrated during the service period, for example, the driver adjusts the side mirror) After the square rearview mirror, the position, attitude and scale relationship need to be recalibrated). Therefore, it is only necessary to obtain the relative position and posture relationship between the internal view camera and the display device. Specifically, the device can use the internal camera on the vehicle (that is, the camera that captures the internal environment of the vehicle, such as the driver's position), to obtain the relative position, attitude, and scale relationship between the display device and the internal camera on the vehicle. The acquisition method can be based on the positioning markers on the display device, and/or based on image and/or multiple sensor fusion extraction feature point tracking methods, and/or detection methods, such as SLAM (Simultaneous Localization and Mapping, simultaneous positioning and Mapping) technology and target tracking technology. The relative position and attitude relationship between the eyes and the display device are relatively fixed and can be calibrated in advance (recalibration is occasionally required during the use period, for example, the position, attitude and scale relationship need to be recalibrated after the user adjusts the position of the head-mounted display device); ← Calibration → Display Device ← Estimation → Interior Camera on the Vehicle ← Calibration → Side Mirrors).

其次，当设备估计出驾驶人头部眼睛和侧方后视镜的相对位置关系后，根据侧方后视镜的镜面属性，获得侧方后视镜的镜面图像的虚拟视点，根据虚拟视点和扩展后视镜的面积，设备可以通过车载的一个或多个相机，和/或立体相机，和/或深度相机采集车辆周围的图像信息，采集范围包含且大于侧方后视镜所覆盖区域，如图11所示；设备可以通过将其他车载相机的图像获得后视镜扩展区域的内容，由于本车上的外视相机和侧方后视镜的相对位置姿态关系相对固定，可以事先标定，因此设备可以用本车上的外视相机的图像通过基于图像的绘制技术(image based rendering)生成侧方后视镜的扩展区域的内容；当本车的外视相机是立体相机和/或配有深度相机时，设备也可以用基于深度图的绘制技术(depth image based rendering)从外视相机的图像生成侧方后视镜的扩展区域的内容。包含扩展区域的侧方后视镜的范围如图12所示。Secondly, after the device estimates the relative positional relationship between the driver's head and eyes and the side rearview mirror, it obtains the virtual viewpoint of the mirror image of the side rearview mirror according to the mirror properties of the side rearview mirror. To expand the area of the rearview mirror, the device can collect image information around the vehicle through one or more cameras, and/or stereo cameras, and/or depth cameras. The collection range includes and is larger than the area covered by the side rearview mirror. As shown in Figure 11; the device can obtain the content of the extended area of the rearview mirror by using the images of other on-board cameras. Since the relative position and posture relationship between the external-view camera and the side rearview mirror on the car is relatively fixed, it can be calibrated in advance. Therefore, the device can use the image of the exterior camera on the vehicle to generate the content of the extended area of the side rearview mirror through image based rendering technology; when the exterior camera of the vehicle is a stereo camera and/or equipped with When there is a depth camera, the device can also use the depth image based rendering technology (depth image based rendering) to generate the content of the extended area of the side rearview mirror from the image of the exterior camera. The extent of the side mirrors including the extended area is shown in Figure 12.

步骤1307(图中未标注)、确定侧方后视镜的扩展区域的显示方式。Step 1307 (not marked in the figure), determine the display mode of the extended area of the side rearview mirror.

该步骤S1307可以是确定一个或多个目标行车辅助信息的显示方式的一种实现方式。This step S1307 may be an implementation manner of determining a display manner of one or more target driving assistance information.

类似于实施例一的步骤S1301，不再赘述。特别的，It is similar to step S1301 inEmbodiment 1, and will not be repeated here. special,

为了使驾驶人观感自然，设备可以使用以下方式至少之一：侧方后视镜的扩展区域显示的虚拟三维显示信息和该虚拟三维显示信息对应的真实物体相对于所述扩展区域是对称的；在扩展区域显示的虚拟三维显示信息是该虚拟三维显示信息对应的真实物体相对于所述侧方后视镜的镜面成像；在扩展区域与所述侧方后视镜显示的内容是连续的；在扩展区域与所述侧方后视镜显示的内容有一定的重叠和/或过渡。In order to make the driver look and feel natural, the device can use at least one of the following methods: the virtual three-dimensional display information displayed in the extended area of the side rearview mirror and the real object corresponding to the virtual three-dimensional display information are symmetrical with respect to the extended area; The virtual three-dimensional display information displayed in the extended area is the mirror image of the real object corresponding to the virtual three-dimensional displayed information relative to the side rearview mirror; the content displayed in the extended area and the side rearview mirror is continuous; There is a certain overlap and/or transition between the expanded area and the content displayed in the side rearview mirror.

步骤S1407(图中未标注)、显示生成的目标行车辅助信息对应的AR信息。Step S1407 (not marked in the figure), displaying AR information corresponding to the generated target driving assistance information.

该步骤S1407可以是显示一个或多个目标行车辅助信息对应的虚拟三维AR信息的一种实现方式。This step S1407 may be an implementation manner of displaying virtual 3D AR information corresponding to one or more target driving assistance information.

类似于实施例一的步骤S1401，不再赘述。It is similar to step S1401 inEmbodiment 1, and will not be repeated here.

实施例八Embodiment eight

本发明实施例提供了一种行车辅助信息的显示方法，该方法用于对本车内部的内后视镜的扩展区域进行显示并显示相应的增强现实辅助驾驶显示信息。An embodiment of the present invention provides a method for displaying driving assistance information. The method is used for displaying an extended area of an interior rearview mirror inside a vehicle and displaying corresponding augmented reality auxiliary driving display information.

本发明实施例的方法包括：The method of the embodiment of the present invention comprises:

步骤S1108(图中未标注)、设备确定是否需要显示车内后视镜的扩展区域相关信息。Step S1108 (not marked in the figure), the device determines whether it is necessary to display information about the extended area of the interior rearview mirror.

该步骤S1108可以是设备确定需要显示的一个或多个目标行车辅助信息的一种实现方式。This step S1108 may be an implementation manner for the device to determine one or more target driving assistance information to be displayed.

对于本发明实施例，设备通过用户的指令开始或者结束显示的功能。其中，用户指令可以是通过手势、语音、物理按键、指纹等生物识别标识来承载的；设备也可以通过检测驾驶人的注视点是否在车内后视镜来自适应的判断是否开始或者结束显示的功能；也可以是两种方式的组合。For the embodiment of the present invention, the device starts or ends the display function through a user's instruction. Among them, user instructions can be carried by gestures, voice, physical buttons, fingerprints and other biometric identifications; the device can also adaptively judge whether to start or end the display by detecting whether the driver's gaze point is on the rearview mirror. function; it can also be a combination of the two.

其中，当使用的相机是头戴式显示装置上的外视相机时，检测的方式可以是通过检测图像中是否有车内后视镜来判断车内后视镜是否在驾驶人的视野内，继而设备可以通过视线追踪的方式获得驾驶人当前的注视区域，判断注视点是否在车内后视镜区域；当使用的相机是本车的内视相机时，设备可以通过检测驾驶人头部的朝向，和/或眼睛的朝向，和/或视线的方向判断车内后视镜是否在驾驶人的视野范围内，以及驾驶人的视线聚焦区域。设备也可以是根据统计用户视线在AR信息上的注视/停留的时间自适应的结束显示。Wherein, when the camera used is an external-view camera on a head-mounted display device, the detection method may be to determine whether the interior rear-view mirror is within the driver's field of vision by detecting whether there is an interior rear-view mirror in the image, Then the device can obtain the driver's current gaze area through eye tracking, and determine whether the gaze point is in the rearview mirror area of the car; when the camera used is the interior camera of the car, the device can detect the driver's head The orientation, and/or the orientation of the eyes, and/or the direction of the line of sight determine whether the interior rearview mirror is within the driver's field of vision, and the focus area of the driver's line of sight. The device may also self-adaptively end the display based on statistics of the user's gaze/stay time on the AR information.

步骤S1208(图中未标注)、设备生成需要显示的内容。Step S1208 (not marked in the figure), the device generates content to be displayed.

该步骤S1208可以是获取信息，处理信息，生成目标行车辅助信息内容的一种实现方式。This step S1208 may be an implementation manner of acquiring information, processing information, and generating target driving assistance information content.

首先，设备估计驾驶人头部/眼睛/视线和车内后视镜的相对位置和姿态。First, the device estimates the relative position and attitude of the driver's head/eyes/sight and the interior mirror.

对于显示装置为头戴式显示装置的情况，当使用的相机是头戴式显示装置上的外视相机时，此外视相机和驾驶人头部相对位置固定，与眼睛的相对位置姿态尺度关系可事先标定(使用期偶尔需要重新校准，例如使用人调整了头戴式显示装置的位置之后需要重新校准位置姿态和尺度关系)，因此只需要获得车内后视镜和此外视相机的相对位置关系。具体的，设备可以通过图像识别技术分割出车内后视镜，车内后视镜可以被看作是一个尺度固定的平面，因此通过求解单应性矩阵获取车内后视镜和相机的相对位置和姿态关系；设备也可以使用视觉里程计的方式对车内后视镜的图像序列进行特征跟踪，其中特征采集车内后视镜的边缘轮廓，从而获取平滑的车内后视镜相对于相机的相对位置和姿态关系；设备也可以在车内后视镜屏幕上贴定位标志物，设备上的外视相机通过定位标志物计算出车内后视镜和本车外视相机的相对位置姿态尺度关系；即(眼睛←校准→显示装置←校准→设备上的外视相机←估计→车内后视镜)。For the case where the display device is a head-mounted display device, when the camera used is the external-view camera on the head-mounted display device, the relative position of the external-view camera and the driver's head is fixed, and the relative position and attitude scale relationship with the eyes can be Calibration in advance (recalibration is occasionally required during the use period, for example, the position, attitude and scale relationship need to be recalibrated after the user adjusts the position of the head-mounted display device), so only the relative position relationship between the interior rearview mirror and the external view camera needs to be obtained . Specifically, the device can segment the interior rearview mirror through image recognition technology. The interior rearview mirror can be regarded as a plane with a fixed scale. Therefore, the relative relationship between the interior rearview mirror and the camera can be obtained by solving the homography matrix. The position and attitude relationship; the device can also use the visual odometer to perform feature tracking on the image sequence of the interior rearview mirror, in which the feature collects the edge contour of the interior rearview mirror, so as to obtain a smooth interior rearview mirror relative to The relative position and posture relationship of the camera; the device can also paste positioning markers on the screen of the interior rearview mirror, and the exterior camera on the device can calculate the relative position of the interior rearview mirror and the exterior camera of the vehicle through the positioning landmarks Attitude-scale relationship; ie (eye ← calibration → display device ← calibration → exterior camera on device ← estimation → interior mirror).

对于显示装置为头戴式显示装置的情况，当使用的相机是本车上的内视相机时，内视相机和车内后视镜的相对位置姿态关系可以认为是固定的，可以事先标定(使用期偶尔需要重新校准，例如驾驶人调节车内后视镜后，需要重新校准位置姿态和尺度关系)。因此只需要获得内视相机和显示设备的相对位置姿态关系。具体的，设备可以使用本车上的内视相机(即拍摄本车内部环境，如驾驶人位置的相机)获取显示装置和本车上的内视相机的相对位置姿态尺度关系。获取的方式可以是基于显示装置上的定位标志物的，和/或基于图像和/或多种传感器融合的提取特征点跟踪方式，和/或检测等方式的，如SLAM(Simultaneous Localization and Mapping，同步定位和建图)技术和目标跟踪技术。眼睛和显示装置的相对位置和姿态关系相对固定，可以事先校准(使用期偶尔需要重新校准，例如使用人调整了头戴式显示设备的位置之后需要重新校准位置姿态和尺度关系)；即(眼睛←校准→显示装置←估计→本车上的内视相机←校准→车内后视镜)。For the case where the display device is a head-mounted display device, when the camera used is the interior camera on the car, the relative position and posture relationship between the interior camera and the interior rearview mirror can be considered fixed, and can be calibrated in advance ( Occasionally recalibration is required during use, for example, after the driver adjusts the interior rearview mirror, the position, attitude and scale relationship need to be recalibrated). Therefore, it is only necessary to obtain the relative position and posture relationship between the internal view camera and the display device. Specifically, the device can use the internal camera on the vehicle (that is, the camera that captures the internal environment of the vehicle, such as the driver's position) to obtain the relative position, attitude, and scale relationship between the display device and the internal camera on the vehicle. The acquisition method can be based on the positioning markers on the display device, and/or based on image and/or multiple sensor fusion extraction feature point tracking methods, and/or detection methods, such as SLAM (Simultaneous Localization and Mapping, Synchronous positioning and mapping) technology and target tracking technology. The relative position and attitude relationship between the eyes and the display device are relatively fixed and can be calibrated in advance (recalibration is occasionally required during the use period, for example, the position, attitude and scale relationship need to be recalibrated after the user adjusts the position of the head-mounted display device); ←calibration→display device←estimation→interior view camera on the vehicle ←calibration→interior rearview mirror).

对于显示装置为车载显示装置的情况，设备可以使用本车上的内视相机(即拍摄本车内部环境，如驾驶人位置的相机)获取眼睛和本车上的内视相机的相对位置姿态尺度关系。其中获取的方式可以是基于驾驶人头戴的定位标志物，其中眼睛和头戴的定位标志物的相对位置姿态尺度关系可以认为是相对固定的，可以事先校准(使用期偶尔需要重新校准，例如使用人调整了头戴式定位标志物的位置之后需要重新校准位置姿态和尺度关系)；获取的方式还可以是基于图像的头部/眼睛/视线定位跟踪技术，设备通过本车上的内视相机的图像或视频定位驾驶人的头部，基于头部的定位结果定位眼睛和内视相机之间的相对位置姿态关系；获取的方式还可以是基于图像的眼睛定位跟踪技术，设备通过本车上的内视相机的图像或视频直接定位眼睛和内视相机之间的相对位置、姿态关系。For the case where the display device is a vehicle-mounted display device, the device can use the internal view camera on the vehicle (that is, the camera that captures the internal environment of the vehicle, such as the driver's position) to obtain the relative position and attitude scale of the eyes and the internal view camera on the vehicle relation. The method of acquisition can be based on the positioning markers worn by the driver, where the relative position, attitude, and scale relationship between the eyes and the positioning markers worn on the head can be considered to be relatively fixed, and can be calibrated in advance (recalibration is occasionally required during use, such as After the user adjusts the position of the head-mounted positioning marker, the position, posture and scale relationship need to be recalibrated); the acquisition method can also be based on the image-based head/eye/sight positioning tracking technology, and the device uses the internal vision on the vehicle The image or video of the camera locates the driver's head, and the relative position and posture relationship between the eyes and the internal view camera is located based on the positioning result of the head; the acquisition method can also be based on the image-based eye positioning and tracking technology. The image or video of the endoscopic camera directly locates the relative position and posture relationship between the eyes and the endoscopic camera.

其次，当设备估计出驾驶人眼睛和车内后视镜的相对位置姿态尺度关系后，设备可以通过车内内视相机拍摄车内后排位置，和/或车尾外部区域，的图像或者视频，将图像或者视频内容缩小/放大后作为扩展内容，如图13所示。Secondly, after the device estimates the relative position, attitude and scale relationship between the driver's eyes and the interior rearview mirror, the device can take images or videos of the rear position in the car and/or the outer area of the rear of the car through the interior view camera , reducing/enlarging the image or video content as the extended content, as shown in FIG. 13 .

步骤S1308(图中未标注)、确定车内后视镜的扩展区域的显示方式。Step S1308 (not marked in the figure), determine the display mode of the extended area of the interior rearview mirror.

该步骤S1308可以是确定一个或多个目标行车辅助信息的显示方式的一种实现方式。This step S1308 may be an implementation manner of determining a display manner of one or more target driving assistance information.

类似于实施例一的步骤S1301，不再赘述。特别的，设备可以使得AR扩展内容在一定角度范围内调整，使得AR扩展内容可面对驾驶人视线方向。对于显示装置为头戴式显示装置的情况，考虑到驾驶人的习惯，设备优先将AR信息呈现在车内后视镜区域或其下方；对于显示装置为车载显示装置的情况，设备优先将AR信息根据驾驶人的注视点距离显示，且靠近天空部分，避免妨碍驾驶人观察真实路面。It is similar to step S1301 inEmbodiment 1, and will not be repeated here. In particular, the device can adjust the AR extension content within a certain angle range, so that the AR extension content can face the driver's line of sight. For the case where the display device is a head-mounted display device, taking into account the driver's habits, the device gives priority to presenting AR information on or below the rearview mirror area of the car; The information is displayed according to the distance of the driver's gaze point, and it is close to the sky to avoid hindering the driver's observation of the real road.

步骤S1408(图中未标注)、显示生成的目标行车辅助信息对应的AR信息。Step S1408 (not marked in the figure), displaying AR information corresponding to the generated target driving assistance information.

该步骤S1408可以是显示一个或多个目标行车辅助信息对应的虚拟三维AR信息的一种实现方式。This step S1408 may be an implementation manner of displaying virtual 3D AR information corresponding to one or more target driving assistance information.

类似于实施例一的步骤S1401，不再赘述。It is similar to step S1401 inEmbodiment 1, and will not be repeated here.

实施例九Embodiment nine

本发明实施例提供了一种行车辅助信息的显示方法，该方法用于对包括但不限于没有交通信号灯的路口，或交通信号灯损坏，或交通信号灯不明显，或被部分/完全遮挡的路口进行显示，并显示相应的增强现实辅助驾驶显示信息。如上情形在下文统称为没有交通信号灯的路口。An embodiment of the present invention provides a method for displaying driving assistance information, which is used to display but not limited to intersections without traffic lights, or damaged traffic lights, or traffic lights that are not obvious, or partially/completely blocked intersections display, and display the corresponding augmented reality assisted driving display information. Such situations are collectively referred to as intersections without traffic lights hereinafter.

本发明实施例的方法包括：The method of the embodiment of the present invention comprises:

步骤S1109(图中未标注)、设备确定是否接近没有交通信号灯的路口相关信息。In step S1109 (not marked in the figure), the device determines whether it is close to information related to intersections without traffic lights.

该步骤S1109可以是设备确定需要显示的一个或多个目标行车辅助信息的一种实现方式。This step S1109 may be an implementation manner in which the device determines one or more target driving assistance information to be displayed.

对于本发明实施例，设备通过确定本车所在位置，在地图上检查前方路口是否存在交通信号灯，自适应开启显示功能，也可以根据用户的指令开始识别和/或显示的功能。其中，用户指令可以是通过手势、语音、物理按键、指纹等生物识别标识来承载的。For the embodiment of the present invention, the device determines the location of the vehicle, checks on the map whether there are traffic lights at the intersection ahead, and adaptively turns on the display function, or starts the recognition and/or display function according to the user's instruction. Among them, user instructions can be carried by gestures, voice, physical buttons, fingerprints and other biometric identifications.

步骤S1209(图中未标注)、设备生成需要显示的内容。Step S1209 (not marked in the figure), the device generates content to be displayed.

该步骤S1209可以是获取信息，处理信息，生成目标行车辅助信息内容的一种实现方式。This step S1209 may be an implementation manner of acquiring information, processing information, and generating target driving assistance information content.

对于本发明实施例，设备通过车载的一个或多个相机监测路口信息，通过图像检查技术和识别技术确定出路口处其他车辆的到达顺序。设备从地图中获取此路口适用的交通规则，生成虚拟AR交通信号灯，指示驾驶人的驾驶操作，如图14所示。例如，一个十字路口适用交通规则为到达后停车3秒钟，先停先走；设备可以通过图像技术监测到路口车辆的停车顺序，结合交通规则，生成3秒钟后变为绿灯的虚拟红灯。For the embodiment of the present invention, the device monitors intersection information through one or more cameras on the vehicle, and determines the arrival sequence of other vehicles at the intersection through image inspection technology and recognition technology. The device obtains the applicable traffic rules of this intersection from the map, generates virtual AR traffic lights, and instructs the driver to drive, as shown in Figure 14. For example, the applicable traffic rule at an intersection is to stop for 3 seconds after arriving, stop first and go first; the device can monitor the parking sequence of vehicles at the intersection through image technology, and combine traffic rules to generate a virtual red light that turns green after 3 seconds .

对于存在交通信号灯的路口，根据驾驶人指令，设备可以通过至少一个相机获取真实交通信号灯的视频，通过复制真实交通信号灯的方式生成AR交通信号灯。For intersections with traffic lights, according to the driver's instructions, the device can capture video of real traffic lights through at least one camera, and generate AR traffic lights by replicating real traffic lights.

步骤S1309(图中未标注)、确定目标行车辅助信息内容的显示方式。Step S1309 (not marked in the figure), determine the display mode of the content of the target driving assistance information.

该步骤S1309可以是确定一个或多个目标行车辅助信息的显示方式的一种实现方式。This step S1309 may be an implementation manner of determining a display manner of one or more target driving assistance information.

类似于实施例六的步骤S1306，将AR交通标志替换为AR交通信号灯，不再赘述。Similar to step S1306 of the sixth embodiment, the AR traffic sign is replaced with the AR traffic signal light, which will not be repeated here.

步骤S1409(图中未标注)、显示生成的目标行车辅助信息对应的AR信息。Step S1409 (not marked in the figure), displaying AR information corresponding to the generated target driving assistance information.

该步骤S1409可以是显示一个或多个目标行车辅助信息对应的虚拟三维AR信息的一种实现方式。This step S1409 may be an implementation manner of displaying virtual 3D AR information corresponding to one or more target driving assistance information.

类似于实施例一的步骤S1401，不再赘述。It is similar to step S1401 inEmbodiment 1, and will not be repeated here.

实施例十Embodiment ten

本发明实施例提供了一种行车辅助信息的显示方法，该方法用于对交通警察进行显示并显示相应的增强现实辅助驾驶显示信息。An embodiment of the present invention provides a method for displaying driving assistance information, which is used for displaying traffic police and corresponding augmented reality assisted driving display information.

本发明实施例的方法包括：The method of the embodiment of the present invention comprises:

步骤S1110(图中未标注)、设备确定是否存在交通警察。Step S1110 (not marked in the figure), the device determines whether there is a traffic policeman.

该步骤S1110可以是设备确定需要显示的一个或多个目标行车辅助信息的一种实现方式。This step S1110 may be an implementation manner for the device to determine one or more target driving assistance information to be displayed.

对于本发明实施例，设备通过一个或多个相机采集本车周围环境，通过图像检测识别技术判断是否存在交通警察，和/或是否存在交通警察放置的道路限制路障，自适应开启显示功能；也可以根据用户的指令开始识别和/或显示的功能；也可以是以上方式的组合。其中，用户指令可以是通过手势、语音、物理按键、指纹等生物识别标识来承载的。For the embodiment of the present invention, the device collects the surrounding environment of the vehicle through one or more cameras, judges whether there is a traffic policeman through image detection and recognition technology, and/or whether there is a road restriction barricade placed by the traffic policeman, and adaptively turns on the display function; The recognition and/or display function can be started according to the user's instruction; it can also be a combination of the above methods. Among them, user instructions can be carried by gestures, voice, physical buttons, fingerprints and other biometric identifications.

步骤S1210(图中未标注)、设备生成需要显示的内容。Step S1210 (not marked in the figure), the device generates content to be displayed.

该步骤S1210可以是获取信息，处理信息，生成目标行车辅助信息内容的一种实现方式。This step S1210 may be an implementation manner of acquiring information, processing information, and generating target driving assistance information content.

对于本发明实施例，设备通过一个或多个相机检测、识别、跟踪交通警察的姿态，手势，以及指挥棒等指示工具。根据当地交通警察手势规则生成相应的AR消息。例如，交通警察的手势指示向左转弯，则生成向左转弯的AR消息，如图15所示。For the embodiment of the present invention, the device uses one or more cameras to detect, recognize, and track traffic policeman's posture, hand gestures, and pointing tools such as batons. Generate corresponding AR messages according to the gesture rules of the local traffic police. For example, if a traffic policeman's gesture indicates to turn left, an AR message for turning left is generated, as shown in FIG. 15 .

步骤S1310(图中未标注)、确定目标行车辅助信息内容的显示方式。Step S1310 (not marked in the figure), determining the display mode of the content of the target driving assistance information.

该步骤S1310可以是确定一个或多个目标行车辅助信息的显示方式的一种实现方式。This step S1310 may be an implementation of determining a display manner of one or more target driving assistance information.

类似于实施例一的步骤S1301，不再赘述。It is similar to step S1301 inEmbodiment 1, and will not be repeated here.

对于本发明实施例，对于显示装置为单个头戴式显示装置的情况，设备可以将生成的AR消息显示在对应交通警察的旁边，AR消息可以面朝驾驶人视线，也可以和交通警察身体朝向保持一致。For the embodiment of the present invention, if the display device is a single head-mounted display device, the device can display the generated AR message next to the corresponding traffic policeman. The AR message can face the driver's line of sight, or face the traffic policeman's body be consistent.

对于本发明实施例，对于显示装置为单个车载显示装置的情况，设备可以将生成的AR消息面向驾驶人视线显示。For the embodiment of the present invention, if the display device is a single vehicle-mounted display device, the device may display the generated AR message facing the driver's line of sight.

对于本发明实施例，当存在多个交通警察的时候，设备可同时显示多个AR消息在对应交通警察的旁边，也可以对与当前面朝本车指挥的交通警察相关的AR消息进行优先显示。For the embodiment of the present invention, when there are multiple traffic policemen, the device can simultaneously display multiple AR messages next to the corresponding traffic policemen, and can also preferentially display the AR messages related to the traffic policeman currently facing the vehicle. .

步骤S1410(图中未标注)、显示生成的目标行车辅助信息对应的AR信息。Step S1410 (not marked in the figure), displaying AR information corresponding to the generated target driving assistance information.

该步骤S1410可以是显示一个或多个目标行车辅助信息对应的虚拟三维AR信息的一种实现方式。This step S1410 may be an implementation manner of displaying virtual 3D AR information corresponding to one or more target driving assistance information.

类似于实施例一的步骤S1401，不再赘述。It is similar to step S1401 inEmbodiment 1, and will not be repeated here.

实施例十一Embodiment Eleven

本发明实施例提供了一种行车辅助信息的显示方法，该方法用于对操作表盘按键的位置，功能和操作方式进行显示并显示相应的增强现实辅助驾驶显示信息。An embodiment of the present invention provides a method for displaying driving assistance information. The method is used for displaying the position, function and operation mode of the buttons on the operation dial and displaying corresponding augmented reality auxiliary driving display information.

本发明实施例的方法包括：The method of the embodiment of the present invention comprises:

步骤S1111(图中未标注)、设备确定是否需要对操作表盘进行显示。Step S1111 (not marked in the figure), the device determines whether the operation dial needs to be displayed.

该步骤S1111可以是设备确定需要显示的一个或多个目标行车辅助信息的一种实现方式。This step S1111 may be an implementation manner for the device to determine one or more target driving assistance information to be displayed.

对于本发明实施例，设备通过确定本车周围环境和/或本车内环境，判断驾驶人是否需要进行某种操作，例如前车窗有雾气，设备可以判断驾驶人需要使用雨刷，自适应开启显示功能；也可以根据用户的指令开始识别和/或显示的功能；也可以是以上方式的组合。其中，用户指令可以是通过手势、语音、物理按键、指纹等生物识别标识来承载的。For the embodiment of the present invention, the device judges whether the driver needs to perform certain operations by determining the surrounding environment of the vehicle and/or the environment inside the vehicle. Display function; it can also start to identify and/or display the function according to the user's instruction; it can also be a combination of the above methods. Among them, user instructions can be carried by gestures, voice, physical buttons, fingerprints and other biometric identifications.

步骤S1211(图中未标注)、设备生成需要显示的内容。Step S1211 (not marked in the figure), the device generates content to be displayed.

该步骤S1211可以是获取信息，处理信息，生成目标行车辅助信息内容的一种实现方式。This step S1211 may be an implementation manner of acquiring information, processing information, and generating target driving assistance information content.

类似于判断实施例八的步骤S1208，设备判断所需操作的按键是否在驾驶人的视野内。当所需操作的按键在驾驶人视野内时，设备通过区域高亮，或箭头指示，或边缘包围的方式突出此按键，并生成按键的功能名称，和/或操作指示。例如，在前照灯灯光旋钮周围标出前照灯的名称，和/或用箭头开灯的旋转方向和关灯的旋转方向，如图16所示。Similar to step S1208 in the eighth judging embodiment, the device judges whether the key to be operated is within the driver's field of vision. When the key to be operated is within the driver's field of vision, the device highlights the key by highlighting the area, or indicating the arrow, or enclosing the edge, and generates the function name and/or operation instruction of the key. For example, marking the name of the headlight around the headlight light knob, and/or arrows to turn the light on and turn it off, as shown in FIG. 16 .

步骤S1311(图中未标注)、确定目标行车辅助信息内容的显示方式。Step S1311 (not marked in the figure), determining the display mode of the content of the target driving assistance information.

该步骤S1311可以是确定一个或多个目标行车辅助信息的显示方式的一种实现方式。This step S1311 may be an implementation manner of determining a display manner of one or more target driving assistance information.

特别的，本实施例只能配合头戴式显示装置应用。In particular, this embodiment can only be applied in conjunction with a head-mounted display device.

对于显示装置为单个头戴式显示装置的情况，设备可以将生成的AR消息显示在按键的旁边，AR消息可以面朝驾驶人视线，也可以和按键朝向保持一致，操作指示和真实按键的操作动作保持一致。For the case where the display device is a single head-mounted display device, the device can display the generated AR message next to the button. The AR message can face the driver's line of sight, or it can be consistent with the direction of the button. The operation instruction and the operation of the real button Actions remain consistent.

步骤S1411(图中未标注)、显示生成的目标行车辅助信息对应的AR信息。Step S1411 (not marked in the figure), displaying AR information corresponding to the generated target driving assistance information.

该步骤S1411可以是显示一个或多个目标行车辅助信息对应的虚拟三维AR信息的一种实现方式。This step S1411 may be an implementation manner of displaying virtual 3D AR information corresponding to one or more target driving assistance information.

类似于实施例一的步骤S1401，不再赘述。It is similar to step S1401 inEmbodiment 1, and will not be repeated here.

实施例十二Embodiment 12

本发明实施例提供了一种行车辅助信息的显示方法，该方法用于对允许停车且适合停车区域、允许停车但不适合停车区域，以及不允许停车区域中至少之一进行显示并显示相应的增强现实辅助驾驶显示信息，如图17所示。An embodiment of the present invention provides a method for displaying driving assistance information. The method is used for displaying at least one of parking allowed and suitable for a parking area, parking allowed but not suitable for a parking area, and a parking area not allowed, and displaying the corresponding Augmented reality assisted driving display information, as shown in Figure 17.

本发明实施例的方法包括：The method of the embodiment of the present invention comprises:

步骤S1112(图中未标注)、设备确定是否需要对允许停车且适合停车区域、允许停车但不适合停车区域，以及不允许停车区域中至少之一进行显示。Step S1112 (not marked in the figure), the device determines whether it is necessary to display at least one of the areas where parking is allowed and suitable for parking, areas where parking is allowed but not suitable for parking, and areas where parking is not allowed.

该步骤S1112可以是设备确定需要显示的一个或多个目标行车辅助信息的一种实现方式。This step S1112 may be an implementation manner for the device to determine one or more target driving assistance information to be displayed.

对于本发明实施例，设备通过确定本车周围环境和/或本车内环境和/或驾驶人意图，判断驾驶人是否在寻找停车位准备停车，例如驾驶人到达指定目的地或驶入停车场，自适应开启显示功能；也可以根据用户的指令开始识别和/或显示的功能；也可以是以上方式的组合。其中，用户指令可以是通过手势、语音、物理按键、指纹等生物识别标识来承载的。For the embodiment of the present invention, the device judges whether the driver is looking for a parking space to park by determining the surrounding environment of the vehicle and/or the environment inside the vehicle and/or the driver's intention, for example, the driver arrives at the designated destination or enters the parking lot , to enable the display function adaptively; the recognition and/or display function can also be started according to the user's instruction; it can also be a combination of the above methods. Among them, user instructions can be carried by gestures, voice, physical buttons, fingerprints and other biometric identifications.

步骤S1212(图中未标注)、设备生成需要显示的内容。Step S1212 (not marked in the figure), the device generates content to be displayed.

该步骤S1212可以是获取信息，处理信息，生成目标行车辅助信息内容的一种实现方式。This step S1212 may be an implementation manner of acquiring information, processing information, and generating target driving assistance information content.

对于本发明实施例，设备通过车载的一个或多个相机检测本车周围的区域，检测识别是否存在禁止停车标识，分析出允许停车的区域和/或不允许停车的区域。通过图像处理技术，结合本车体积，判断允许停车区域是否平坦，空间是否足够，是否存在积水，泥泞等不利于停车的因素，对允许停车区域进行筛选排序，显示允许停车且适合停车区域和/或允许停车但不适合停车的区域。For the embodiment of the present invention, the device detects the area around the vehicle through one or more cameras on the vehicle, detects and identifies whether there is a no-parking sign, and analyzes the area where parking is allowed and/or the area where parking is not allowed. Through image processing technology, combined with the volume of the vehicle, it is judged whether the allowed parking area is flat, whether the space is sufficient, whether there are water, mud and other factors that are not conducive to parking, and the allowed parking area is screened and sorted, and the parking area is allowed and suitable for parking and / or areas where parking is permitted but not suitable for parking.

步骤S1312(图中未标注)、确定目标行车辅助信息内容的显示方式。Step S1312 (not marked in the figure), determining the display mode of the content of the target driving assistance information.

该步骤S1312可以是确定一个或多个目标行车辅助信息的显示方式的一种实现方式。This step S1312 may be an implementation of determining a display manner of one or more target driving assistance information.

类似于实施例一的步骤S1301，不再赘述。特别的，设备对允许停车且适合停车区域、允许停车但不适合停车区域，以及不允许停车区域，用不同的方式呈现。It is similar to step S1301 inEmbodiment 1, and will not be repeated here. In particular, the device presents differently for areas where parking is allowed and suitable for parking, areas where parking is allowed but not suitable for parking, and areas where parking is not allowed.

步骤S1412(图中未标注)、显示生成的目标行车辅助信息对应的AR信息。Step S1412 (not marked in the figure), displaying AR information corresponding to the generated target driving assistance information.

该步骤S1412可以是显示一个或多个目标行车辅助信息对应的虚拟三维AR信息的一种实现方式。This step S1412 may be an implementation manner of displaying virtual 3D AR information corresponding to one or more target driving assistance information.

类似于实施例一的步骤S1401，不再赘述。It is similar to step S1401 inEmbodiment 1, and will not be repeated here.

特别的，对于所有实施例，设备自适应的确定如下三条：开始显示一个或多个AR信息的时机，和/或停止显示一个或多个AR信息的时机，和/或一个或多个AR信息的显示时长。设备自适应的确定如上时机和/或时长的时候，设备综合考虑到车辆的状态、路况信息、设备的系统延迟中的至少之一，以达到更优的显示开始和/或显示停止时机，和更优的显示时长，以避免包括但不限于过早、过晚、过长，或过短的显示等问题，误导干扰驾驶人。In particular, for all embodiments, the device adaptively determines the following three items: when to start displaying one or more AR information, and/or when to stop displaying one or more AR information, and/or when to display one or more AR information display duration. When determining the timing and/or duration of device self-adaptation, the device comprehensively considers at least one of the state of the vehicle, road condition information, and system delay of the device to achieve a better display start and/or display stop timing, and Better display duration to avoid misleading and interfering drivers, including but not limited to too early, too late, too long, or too short display.

下面举例说明，对于警示安全距离不足AR警告消息，设备可以在本车与周围车辆的距离小于安全距离的时刻开始显示，在本车与周围车辆的距离大于安全距离的时刻停止显示。然而，由于安全距离随着情况不同而不同，当车辆行驶在冰雪路面时，本车和周围车辆的安全距离应当比行驶在普通路面要大。而且安全距离与本车和周围车辆的行驶速度相关，速度越快，所需的安全距离越大。也就是说，安全距离应当根据具体情况自适应的确定。因此，基于车辆状态(车速等)，路况信息(冰雪路面等)，以及设备的系统延迟，设备可以自适应的计算当前情况下的安全距离。因此，相应的，设备对于警示安全距离不足的AR警告消息的显示开始时间，显示停止时间，显示时长也随着安全距离自适应的调整，而非固定。The following example illustrates that for the AR warning message warning that the safety distance is insufficient, the device can start displaying when the distance between the vehicle and surrounding vehicles is less than the safety distance, and stop displaying when the distance between the vehicle and surrounding vehicles is greater than the safety distance. However, since the safety distance varies with different situations, when the vehicle is driving on an ice-snow road, the safety distance between the vehicle and surrounding vehicles should be larger than that when driving on an ordinary road. Moreover, the safety distance is related to the driving speed of the vehicle and surrounding vehicles, the faster the speed, the greater the required safety distance. That is to say, the safety distance should be determined adaptively according to the specific situation. Therefore, based on the vehicle status (vehicle speed, etc.), road condition information (ice and snow roads, etc.), and the system delay of the device, the device can adaptively calculate the safe distance under the current situation. Therefore, correspondingly, the display start time, display stop time, and display duration of the AR warning message warning that the safety distance is insufficient are adaptively adjusted according to the safety distance instead of being fixed.

特别的，对于所有实施例，设备自适应减少两种延迟：一种是注意力延迟，另一种是显示延迟。注意力延迟定义为从设备显示AR信息到驾驶人注意到AR信息的时间延迟；显示延迟定义为设备在生成，渲染和显示AR信息上花费的时间。In particular, for all embodiments, the device adaptively reduces two kinds of delays: one is attention delay and the other is display delay. Attention latency is defined as the time delay from when the device displays AR information to when the driver notices the AR information; display latency is defined as the time the device spends generating, rendering, and displaying the AR information.

其中，为减少注意力延迟，设备可以采取如下五种方式中至少之一：1)自适应的将高优先级的AR信息显示在驾驶人当前视野内的显著位置，并且根据驾驶人当前的注视深度确定显示方式；2)自适应的根据真实场景环境调整AR信息的显示形式，增加AR信息和真实场景环境的对比度，以突显AR信息；3)对于高优先级的AR信息，为突显高优先级的AR信息，停止和/或暂停显示其周围的低优先级的AR信息以突显高优先级AR信息；4)对于高优先级的AR信息，设备自适应的将其显示在驾驶人视野内的显著位置，并且在驾驶人视野变化(转头、转眼睛等)时自适应的调整高优先级AR信息的显示位置，使其保持在驾驶人视野内的显著位置；5)设备配合使用声音，动画，震动，灯光闪烁等其他方式中至少之一吸引驾驶人注意力。Among them, in order to reduce attention delay, the device can adopt at least one of the following five methods: 1) Adaptively display high-priority AR information at a prominent position in the driver's current field of vision, and Depth determines the display mode; 2) Adaptively adjust the display form of AR information according to the real scene environment, increase the contrast between AR information and real scene environment, so as to highlight AR information; 3) For high-priority AR information, the high priority 4) For high-priority AR information, the device adaptively displays it in the driver's field of view , and adaptively adjust the display position of high-priority AR information when the driver's field of view changes (turning head, turning eyes, etc.), so that it remains in a prominent position within the driver's field of vision; 5) The device cooperates with the use of sound , animation, vibration, flashing lights and other means to attract the driver's attention.

举例说明，当驾驶人看向本车左方时，本车右侧发生爆炸，设备立刻将生成的AR警报显示在驾驶人当前的视野内，即本车左方。For example, when the driver looks to the left of the car and an explosion occurs on the right side of the car, the device immediately displays the generated AR alarm in the driver's current field of vision, that is, the left side of the car.

其中，为减小显示延迟，设备可以采用如下方式至少之一：1)预先准备要显示的数据：设备采集本车以及周围较大范围的信息，生成相应的AR信息，并大致确定每个AR信息的显示方式；但设备只自适应的选择性显示在驾驶人当前视野范围内的AR信息；在计算量允许的情况下设备可以预先渲染好AR信息，当AR信息进入驾驶人视野范围时，设备可以直接调取预先渲染好的AR信息模型，根据当前情况调整后显示，如图18所示；2)改变要显示的AR信息的呈现方式：考虑到延迟的可接受程度与当前情况有关(比如当车速为10千米/小时的时候，5毫秒的延时可能是可以接受的，但当车速上升到50千米/小时的时候，3毫米的延时可能就已经无法接受)，设备可以自适应的根据当前情况估算可接受延迟的程度，并且自适应的改变AR内容的呈现方式。举例说明，如图19所示，当本车车速较低的时候，设备可以完整的显示行车区域；但当本车车速上升后，为降低延迟，设备可以自适应的将行车区域的呈现形式改为条纹状，即通过减少显示数据量来降低延迟。Among them, in order to reduce the display delay, the device can adopt at least one of the following methods: 1) Prepare the data to be displayed in advance: the device collects the information of the vehicle and the surrounding area, generates corresponding AR information, and roughly determines the The display method of information; however, the device only adaptively and selectively displays the AR information within the driver's current field of vision; the device can pre-render the AR information if the calculation amount allows, and when the AR information enters the driver's field of vision, The device can directly call the pre-rendered AR information model, and display it after adjustment according to the current situation, as shown in Figure 18; 2) Change the presentation mode of the AR information to be displayed: Considering that the acceptable degree of delay is related to the current situation ( For example, when the speed of the vehicle is 10 km/h, a delay of 5 milliseconds may be acceptable, but when the speed of the vehicle rises to 50 km/h, a delay of 3 mm may be unacceptable), the device can Adaptively estimate the degree of acceptable delay according to the current situation, and adaptively change the presentation mode of AR content. For example, as shown in Figure 19, when the speed of the vehicle is low, the device can completely display the driving area; but when the speed of the vehicle increases, the device can adaptively change the presentation form of the driving area to reduce the delay. It is striped, that is, the delay is reduced by reducing the amount of display data.

特别的，如图20所示，当同时存在多个AR信息需要显示的时候，如果设备只对每个AR信息分别考虑，为每个AR信息自适应的选择最优的显示方式，则在同时显示这些AR信息给驾驶人时，驾驶人可能会因为这些AR信息之间存在相互遮挡关系而感觉AR信息模糊不清晰，从而对驾驶人造成困扰。In particular, as shown in Figure 20, when multiple AR information needs to be displayed at the same time, if the device only considers each AR information separately, and adaptively selects the optimal display method for each AR information, then at the same time When the AR information is displayed to the driver, the driver may feel that the AR information is blurred and unclear due to the mutual occlusion relationship between the AR information, thereby causing trouble to the driver.

因此，对同时显示多个存在相互遮挡关系多个AR信息，设备可以使用以下方式至少之一：1)对必须同时显示，且无法合并和/或整合，且确实应该存在遮挡关系的AR信息情况，设备自适应的根据当前位置关系不显示被遮挡AR信息的被遮挡部分，以避免干扰前景AR信息；2)对可以合并和/或整合的多条AR信息，设备自适应的将多条AR信息合并和/或整合为一条或多条信息，如可以将四条AR信息精简为两条AR信息；设备在合并和/或整合多条AR信息时，即可以是简单的同类信息归并，也可以是更高层面的根据信息含义生成新的AR信息，如图20所示，设备可以根据“左边有驾校学员车”和“后方有货车”两条AR信息生成更为精简的新AR信息“小心左方和后方”；3)对于可以延迟显示的AR信息，设备可以推迟显示，例如，关于较远处的非紧急AR信息可以推迟到本车行驶到更近处再开始显示，以避免干扰关于较近处的AR信息的显示；同理，设备也可以延迟显示、暂停显示、停止显示，甚至放弃显示，不重要的AR信息以减少/消除遮挡；4)如图20所示，设备可以改变一条或多条AR信息的显示位置，和/或内容详细程度，和/或呈现方式，减弱甚至消除AR信息之间的相互遮挡关系。Therefore, the device can use at least one of the following methods to simultaneously display multiple AR information with mutual occlusion relationship: 1) For AR information that must be displayed at the same time, cannot be merged and/or integrated, and should indeed have an occlusion relationship , the device adaptively does not display the occluded part of the occluded AR information according to the current position relationship, so as to avoid interference with the foreground AR information; 2) For multiple pieces of AR information that can be merged and/or integrated The information is merged and/or integrated into one or more pieces of information. For example, four pieces of AR information can be reduced to two pieces of AR information; It is a higher level to generate new AR information based on the meaning of the information. As shown in Figure 20, the device can generate more streamlined new AR information based on the two AR information "there is a driving school car on the left" and "there is a truck behind" "Caution 3) For the AR information that can be delayed, the device can delay the display. For example, the non-emergency AR information about the distance can be postponed until the vehicle drives closer to avoid interference with the relevant information. Display of nearby AR information; similarly, the device can also delay display, pause display, stop display, or even give up displaying unimportant AR information to reduce/eliminate occlusion; 4) As shown in Figure 20, the device can change The display position of one or more pieces of AR information, and/or the level of detail of the content, and/or the presentation manner, weaken or even eliminate the mutual occlusion relationship between the AR information.

特别的，在确定AR信息的显示位置时，设备考虑到如下情况中至少之一：1)设备将AR信息显示在物理正确的位置，即和真实空间中相应的物体对齐；2)设备将AR信息显示在不干扰驾驶人驾驶的区域，比如天空区域；3)设备将重要的AR信息直接显示在驾驶人的当前视野的显著位置；4)设备将AR信息显示在驾驶人视野相对开阔的一侧；例如，对于驾驶人座位在左侧的情况，驾驶人右侧视野相对更加开阔；对于驾驶人座位在右侧的情况，驾驶人左侧视野相对更加开阔；5)设备将AR信息显示在驾驶人注意力相对不足的区域，在驾驶中，为确保安全，驾驶人需要对各个方向都进行充分的观察。因此，设备通过统计驾驶人视线在各个方向的停留时间，如果发现驾驶人的注意力在某个或多个区域显著不足，则将AR信息以较为显著的方式显示在这些区域，以吸引驾驶人的注意力，帮助驾驶人平衡注意力。其中吸引驾驶人的注意力的方式也可以结合语音/音效，动画，灯光等。如图21所示，当驾驶人的视线统计显示驾驶人的注意力偏向左侧时，设备可以将AR信息显示在右侧，将驾驶人的注意力吸引到右侧，从而帮助驾驶人平衡注意力。In particular, when determining the display position of the AR information, the device takes into account at least one of the following situations: 1) the device displays the AR information at a physically correct position, that is, aligns with the corresponding object in the real space; 2) the device displays the AR information at a physically correct position; The information is displayed in an area that does not interfere with the driver's driving, such as the sky area; 3) The device directly displays important AR information in a prominent position in the driver's current field of vision; 4) The device displays AR information in a relatively wide field of vision for the driver For example, if the driver's seat is on the left side, the driver's right side view is relatively wider; if the driver's seat is on the right side, the driver's left side view is relatively wider; 5) The device displays the AR information on the The area where the driver's attention is relatively insufficient. During driving, in order to ensure safety, the driver needs to fully observe all directions. Therefore, the device counts the dwell time of the driver's sight in various directions, and if the driver's attention is found to be significantly insufficient in one or more areas, the AR information will be displayed in these areas in a more prominent way to attract the driver. to help drivers balance their attention. The way to attract the driver's attention can also combine speech/sound effects, animation, lighting and so on. As shown in Figure 21, when the driver's sight statistics show that the driver's attention is biased to the left, the device can display the AR information on the right to attract the driver's attention to the right, thereby helping the driver to balance attention force.

本发明实施例提供了一种用于辅助驾驶的增强现实的装置，如图22所示，包括：确定模块2201、显示模块2202。An embodiment of the present invention provides an augmented reality device for assisting driving, as shown in FIG. 22 , including: a determination module 2201 and a display module 2202 .

确定模块2201，用于基于行车过程中获取的信息来确定行车辅助信息。A determining module 2201, configured to determine driving assistance information based on information acquired during driving.

显示模块2202，用于显示确定模块2201确定的行车辅助信息对应的虚拟三维显示信息。The display module 2202 is configured to display virtual three-dimensional display information corresponding to the driving assistance information determined by the determination module 2201 .

本发明实施例提供了一种用于辅助驾驶的增强现实的装置，与现有技术相比，本发明实施例基于行车过程中获取的信息来确定行车辅助信息，然后显示行车辅助信息对应的虚拟三维显示信息，即通过车辆行驶过程中获取到的信息，确定出行车过程中的行车辅助信息，并将行车过程中的行车辅助信息对应的虚拟三维显示信息，通过视觉和/或听觉的方式呈现给驾驶人，以通知或者警示驾驶人，从而可以实现在车辆行驶过程中应用增强现实技术帮助驾驶人更好的掌握车辆行驶过程中的行车信息，进而可以提升用户体验。The embodiment of the present invention provides an augmented reality device for assisting driving. Compared with the prior art, the embodiment of the present invention determines the driving assistance information based on the information obtained during driving, and then displays the virtual reality corresponding to the driving assistance information. Three-dimensional display information, that is, determine the driving assistance information during the driving process through the information obtained during the driving process of the vehicle, and present the virtual three-dimensional display information corresponding to the driving auxiliary information during the driving process through visual and/or auditory means To the driver, to notify or warn the driver, so that the application of augmented reality technology in the driving process of the vehicle can help the driver better grasp the driving information during the driving process of the vehicle, thereby improving the user experience.

本发明实施例提供的用于辅助驾驶的增强现实的装置，适用于上述方法实施例，在此不再赘述。The augmented reality device for assisting driving provided by the embodiment of the present invention is applicable to the above-mentioned method embodiment, and details are not repeated here.

本技术领域技术人员可以理解，本发明包括涉及用于执行本申请中所述操作中的一项或多项的设备。这些设备可以为所需的目的而专门设计和制造，或者也可以包括通用计算机中的已知设备。这些设备具有存储在其内的计算机程序，这些计算机程序选择性地激活或重构。这样的计算机程序可以被存储在设备(例如，计算机)可读介质中或者存储在适于存储电子指令并分别耦联到总线的任何类型的介质中，所述计算机可读介质包括但不限于任何类型的盘(包括软盘、硬盘、光盘、CD-ROM、和磁光盘)、ROM(Read-Only Memory，只读存储器)、RAM(Random Access Memory，随即存储器)、EPROM(Erasable ProgrammableRead-Only Memory，可擦写可编程只读存储器)、EEPROM(Electrically ErasableProgrammable Read-Only Memory，电可擦可编程只读存储器)、闪存、磁性卡片或光线卡片。也就是，可读介质包括由设备(例如，计算机)以能够读的形式存储或传输信息的任何介质。Those skilled in the art will appreciate that the present invention includes devices related to performing one or more of the operations described in this application. These devices may be specially designed and fabricated for the required purposes, or they may include known devices found in general purpose computers. These devices have computer programs stored therein that are selectively activated or reconfigured. Such a computer program can be stored in a device (e.g., computer) readable medium, including but not limited to any type of medium suitable for storing electronic instructions and respectively coupled to a bus. Types of disks (including floppy disks, hard disks, CDs, CD-ROMs, and magneto-optical disks), ROM (Read-Only Memory, read-only memory), RAM (Random Access Memory, random access memory), EPROM (Erasable Programmable Read-Only Memory, Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory, Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, a readable medium includes any medium that stores or transmits information in a form readable by a device (eg, a computer).

本技术领域技术人员可以理解，可以用计算机程序指令来实现这些结构图和/或框图和/或流图中的每个框以及这些结构图和/或框图和/或流图中的框的组合。本技术领域技术人员可以理解，可以将这些计算机程序指令提供给通用计算机、专业计算机或其他可编程数据处理方法的处理器来实现，从而通过计算机或其他可编程数据处理方法的处理器来执行本发明公开的结构图和/或框图和/或流图的框或多个框中指定的方案。Those skilled in the art will understand that computer program instructions can be used to implement each block in these structural diagrams and/or block diagrams and/or flow diagrams and combinations of blocks in these structural diagrams and/or block diagrams and/or flow diagrams . Those skilled in the art can understand that these computer program instructions can be provided to general-purpose computers, professional computers, or processors of other programmable data processing methods for implementation, so that the computer or processors of other programmable data processing methods can execute the present invention. A scheme specified in a block or blocks of a structure diagram and/or a block diagram and/or a flow diagram of the invention disclosure.

本技术领域技术人员可以理解，本发明中已经讨论过的各种操作、方法、流程中的步骤、措施、方案可以被交替、更改、组合或删除。进一步地，具有本发明中已经讨论过的各种操作、方法、流程中的其他步骤、措施、方案也可以被交替、更改、重排、分解、组合或删除。进一步地，现有技术中的具有与本发明中公开的各种操作、方法、流程中的步骤、措施、方案也可以被交替、更改、重排、分解、组合或删除。Those skilled in the art can understand that the various operations, methods, and steps, measures, and solutions in the processes discussed in the present invention can be replaced, changed, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes that have been discussed in the present invention may also be replaced, changed, rearranged, decomposed, combined, or deleted. Further, steps, measures, and schemes in the prior art that have operations, methods, and processes disclosed in the present invention can also be alternated, changed, rearranged, decomposed, combined, or deleted.

以上所述仅是本发明的部分实施方式，应当指出，对于本技术领域的普通技术人员来说，在不脱离本发明原理的前提下，还可以做出若干改进和润饰，这些改进和润饰也应视为本发明的保护范围。The above descriptions are only part of the embodiments of the present invention. It should be pointed out that those skilled in the art can make some improvements and modifications without departing from the principles of the present invention. It should be regarded as the protection scope of the present invention.

Claims (46)

1. A method for augmented reality for driving assistance, comprising:

determining driving auxiliary information based on information acquired in the driving process;

displaying virtual three-dimensional display information corresponding to the driving auxiliary information;

and when the display delay of the virtual three-dimensional display information exceeds the acceptable display delay estimated according to the vehicle speed in the driving process, adjusting the presentation mode of the virtual three-dimensional display information by reducing the display data volume.

2. The method of claim 1,

determining driving assistance information based on information acquired during driving, comprising: determining shielded driving auxiliary information based on information of a sensible area acquired in the driving process;

displaying virtual three-dimensional display information corresponding to the driving auxiliary information, comprising: and displaying the virtual three-dimensional display information corresponding to the shielded driving auxiliary information.

3. The method according to claim 2, wherein when the obstructed driving assistance information comprises: and when the road surface road information and/or the non-road surface traffic sign information are/is displayed, displaying virtual three-dimensional display information corresponding to the sheltered driving auxiliary information, wherein the virtual three-dimensional display information comprises:

and displaying virtual three-dimensional display information corresponding to the shielded driving auxiliary information at the position of the shielded driving auxiliary information.

4. The method of claim 3, wherein the obstructed driving assistance information is determined based on information of a perceptible area acquired in the driving process, and the method comprises at least one of the following steps:

if the shielded driving auxiliary information is only partially shielded, determining the shielded driving auxiliary information according to the sensible part of the driving auxiliary information;

determining the shielded driving auxiliary information based on the position of the current vehicle and the reference object information of the perceptible area in the current driving;

determining the shielded driving auxiliary information based on the multimedia information of the shielded driving auxiliary information acquired from other angles except the visual angle of the driver;

based on multimedia information of the sheltered driving auxiliary information in the perceptible area acquired in the driving process, the multimedia information is enhanced and/or restored, and the sheltered driving auxiliary information is determined;

when the shielded driving auxiliary information comprises road surface road information, determining the shielded driving auxiliary information according to a map by aligning the current road with the map of the current road;

and determining the current sheltered driving auxiliary information according to other driving auxiliary information.

5. The method of claim 4, wherein after determining the obstructed driving assistance information based on the information of the perceivable area acquired during driving, the method further comprises:

correcting the determined shielded driving auxiliary information;

displaying virtual three-dimensional display information corresponding to the shielded driving assistance information, comprising: and displaying virtual three-dimensional display information corresponding to the corrected driving auxiliary information at the corrected position.

6. The method according to claim 5, characterized in that correcting the determined obstructed driving assistance information comprises at least one of:

when the shielded driving auxiliary information comprises lane related information, correcting the position of the shielded driving auxiliary information based on the driving tracks and/or road rut information of other vehicles in the preset range of the current vehicle;

when the shielded driving assistance information includes road surface road information, correcting the position of the shielded driving assistance information according to a map of a current road by aligning the current road with the map of the current road.

7. The method according to claim 2, wherein when the obstructed driving assistance information comprises: when the information about the lane is provided,

the displayed lane width is less than the actual lane width.

8. The method according to claim 2, wherein when the obstructed driving assistance information comprises: during the blind area information, show the virtual three-dimensional display information that is sheltered from driving auxiliary information corresponds, include:

and displaying virtual three-dimensional display information corresponding to the blind area information in the expansion area of the rearview mirror.

9. The method according to claim 8, wherein when the side mirror is a side mirror, the virtual three-dimensional display information displayed in the extended area is generated by a real object corresponding to the virtual three-dimensional display information in accordance with the mirror surface property of the side mirror and the driver's field of view.

10. The method of claim 1,

determining driving assistance information based on information acquired in the driving process, comprising: acquiring traffic rules and/or traffic police action information of a current road section, and converting the determined traffic rules and/or traffic police action information of the current road section in a presentation mode;

displaying virtual three-dimensional display information corresponding to the driving auxiliary information, comprising: and displaying the converted virtual three-dimensional display information corresponding to the traffic rules and/or the traffic police action information of the current road section.

11. The method according to claim 1, wherein the step of displaying the virtual three-dimensional display information corresponding to the driving assistance information comprises at least one of:

when the abnormal rut information is sensed, displaying virtual three-dimensional display information corresponding to the determined abnormal rut area and/or virtual three-dimensional display information of warning information of the abnormal rut area in the area;

when the traffic sign of the road area which is driven by the current vehicle needs to be displayed, displaying the acquired virtual three-dimensional display information corresponding to the traffic sign of the road area which is driven by the vehicle; the acquired traffic signs which have traveled comprise distance indications; the displayed virtual three-dimensional display information comprises the acquired traveled traffic sign and a distance indication updated based on the distance traveled by the vehicle from the traveled traffic sign;

when the traffic sign and/or the traffic light at the intersection where the current vehicle is located are sensed to be present and the traffic sign and/or the traffic light meet the preset display condition, displaying virtual three-dimensional display information corresponding to the traffic sign and/or the traffic light at the intersection; the predetermined display condition includes at least one of: traffic signs and/or traffic light damage; the traffic signs and/or traffic lights are not clearly displayed; the traffic signs and/or traffic lights are not completely within the driver's current visual range;

when the key information in the operation dial plate needs to be displayed, displaying virtual three-dimensional display information corresponding to at least one of the following information: the position information of the key, the function name information of the key, the operation indication information of the key and the key;

when the parking area information needs to be displayed, virtual three-dimensional display information corresponding to at least one of a parking-permitted area suitable for parking, a parking-permitted area not suitable for parking and a parking-not-permitted area is displayed.

12. The method according to claim 11, wherein the step of determining driving assistance information based on information obtained during driving comprises at least one of:

determining whether the road surface rut information has abnormal rut information, and if the abnormal rut information exists, determining that an abnormal rut area exists;

when the traffic sign of the road area which is driven by the current vehicle needs to be displayed, the traffic sign of the road area which is driven by the current vehicle is determined from the acquired multimedia information and/or the traffic sign database;

when the parking area information needs to be displayed, at least one of a parking-permitted area which is suitable for parking, a parking-permitted area which is not suitable for parking and a parking-not-permitted area is determined according to at least one of whether a parking-prohibited mark exists in an area around the current vehicle, the volume of the current vehicle and the current road surface condition.

13. The method according to claim 1, wherein displaying the virtual three-dimensional display information corresponding to the driving assistance information comprises:

and enhancing and displaying the virtual three-dimensional display information corresponding to the track information.

14. The method of claim 11, wherein displaying the acquired virtual three-dimensional display information corresponding to the traffic sign of the road region on which the vehicle has currently traveled comprises:

and according to the current position of the vehicle and the virtual three-dimensional display information corresponding to the traffic sign of the road area which is driven by the vehicle, adjusting the virtual three-dimensional display information corresponding to the traffic sign of the road area which is driven by the vehicle, and displaying the virtual three-dimensional display information corresponding to the adjusted traffic sign.

15. The method according to claim 1, wherein the step of displaying the virtual three-dimensional display information corresponding to the driving assistance information comprises:

determining a display mode corresponding to the virtual three-dimensional display information;

displaying virtual three-dimensional display information corresponding to the driving auxiliary information based on the determined display mode;

wherein the display mode comprises at least one of the following:

a display position of the virtual three-dimensional display information; displaying the posture of the virtual three-dimensional display information; the display size of the virtual three-dimensional display information; a display start time of the virtual three-dimensional display information; display end time of the virtual three-dimensional display information; displaying duration of the virtual three-dimensional display information; the detail degree of the display content of the virtual three-dimensional display information; presenting the virtual three-dimensional display information; the interrelationship of display among a plurality of virtual three-dimensional display information;

the presentation mode comprises at least one of the following: a text mode; icon mode; an animation mode; a sound effect mode; a lighting mode; a vibration mode.

16. The method of claim 1, further comprising at least one of:

when a plurality of pieces of virtual three-dimensional display information to be displayed exist at the same time, combining the plurality of pieces of virtual three-dimensional display information to be displayed, and displaying the processed virtual three-dimensional display information;

when a plurality of pieces of virtual three-dimensional display information to be displayed are displayed simultaneously, the plurality of pieces of virtual three-dimensional display information to be displayed are integrated based on semantics, and the processed virtual three-dimensional display information is displayed.

17. The method of claim 1, further comprising at least one of:

displaying virtual three-dimensional display information corresponding to the driving auxiliary information with priority higher than the first preset priority at a remarkable position of the current visual field of the driver, and adjusting the position for displaying the virtual three-dimensional display information in real time according to the position of the current visual field of the driver;

displaying virtual three-dimensional display information corresponding to the driving auxiliary information with the priority higher than the first preset priority, and pausing and/or stopping displaying the virtual three-dimensional display information corresponding to the driving auxiliary information with the priority lower than the second preset priority.

18. The method according to claim 1, wherein the step of displaying the virtual three-dimensional display information corresponding to the driving assistance information comprises:

determining at least one of display start time, display end time and display duration of the virtual three-dimensional display information according to at least one of the current state of the vehicle, the current road condition information and the system delay condition of the equipment;

and displaying the virtual three-dimensional display information corresponding to the driving auxiliary information according to at least one of the determined display starting time, display ending time and display duration of the virtual three-dimensional display information.

19. The method of claim 1,

when virtual three-dimensional display information corresponding to a plurality of pieces of driving assistance information to be displayed simultaneously exists and an occlusion relationship exists among the plurality of pieces of virtual three-dimensional display information to be displayed, the method further includes at least one of:

according to the position relation among the plurality of pieces of virtual three-dimensional display information with the shielding relation, only displaying the parts which are not shielded in the virtual three-dimensional display information;

respectively displaying virtual three-dimensional display information in the plurality of virtual three-dimensional display information with the shielding relation at different display time;

and adjusting at least one of the display position, the content detail degree and the presentation mode of at least one piece of virtual three-dimensional display information in the plurality of pieces of virtual three-dimensional display information with the shielding relation, and displaying each piece of virtual three-dimensional display information in the plurality of pieces of virtual three-dimensional display information with the shielding relation according to the adjustment mode.

20. The method according to claim 1, wherein the step of displaying the virtual three-dimensional display information corresponding to the driving assistance information comprises:

displaying virtual three-dimensional display information corresponding to the driving auxiliary information to be displayed at a preset display position;

wherein the preset display position comprises at least one of the following items:

a display position aligned with the real driving assistance information; the area position where the driver drives is not interfered; a prominent position of the driver's current field of view; a relatively open driver view; the position of the driver with insufficient attention.

21. The method of claim 1, further comprising:

pre-rendering virtual three-dimensional display information to be displayed;

when a preset display triggering condition is met, acquiring virtual three-dimensional display information to be displayed from the pre-rendered virtual three-dimensional display information, adjusting the presentation mode of the virtual three-dimensional display information according to the current environment, and displaying the virtual three-dimensional display information according to the adjusted presentation mode;

and adjusting the display mode of the virtual three-dimensional display information in real time according to the current environment, and displaying the virtual three-dimensional display information according to the adjusted display mode.

22. The method according to claim 1, wherein the adjusting the manner of presenting the virtual three-dimensional display information comprises:

adjusting a portion of the virtual three-dimensional display information into a stripe pattern.

23. An apparatus for augmented reality for driving assistance, comprising:

the determining module is used for determining driving auxiliary information based on the information acquired in the driving process;

the display module is used for displaying the virtual three-dimensional display information corresponding to the driving auxiliary information determined by the determination module; and when the display delay of the virtual three-dimensional display information exceeds the acceptable display delay estimated according to the vehicle speed in the driving process, adjusting the presentation mode of the virtual three-dimensional display information by reducing the display data volume.

24. The apparatus of claim 23,

the determining module is used for determining the shielded driving auxiliary information based on the information of the perceptible area acquired in the driving process;

and the display module is used for displaying the virtual three-dimensional display information corresponding to the shielded driving auxiliary information.

25. The apparatus of claim 24, wherein when the obstructed driving assistance information comprises: the display module is used for displaying road surface road information and/or non-road surface traffic sign information:

and displaying virtual three-dimensional display information corresponding to the shielded driving auxiliary information at the position of the shielded driving auxiliary information.

26. The apparatus of claim 25, the means for determining is configured to perform at least one of:

if the shielded driving auxiliary information is only partially shielded, determining the shielded driving auxiliary information according to the sensible part of the driving auxiliary information;

determining the shielded driving auxiliary information based on the position of the current vehicle and the reference object information of the perceptible area in the current driving;

determining the shielded driving auxiliary information based on the multimedia information of the shielded driving auxiliary information acquired from other angles except the visual angle of the driver;

based on multimedia information of the sheltered driving auxiliary information in the perceptible area acquired in the driving process, the multimedia information is enhanced and/or restored, and the sheltered driving auxiliary information is determined;

when the shielded driving auxiliary information comprises road surface road information, determining the shielded driving auxiliary information according to a map by aligning the current road with the map of the current road;

and determining the current sheltered driving auxiliary information according to other driving auxiliary information.

27. The apparatus of claim 26, wherein the determining module, after performing the determination of the obstructed driving assistance information based on the information of the perceivable area acquired during driving, is further configured to perform:

correcting the determined shielded driving auxiliary information;

displaying virtual three-dimensional display information corresponding to the sheltered driving assistance information, including: and displaying virtual three-dimensional display information corresponding to the corrected driving auxiliary information at the corrected position.

28. The apparatus of claim 27, wherein the determining module is configured to perform the correction of the determined obstructed driving assistance information comprises at least one of:

when the shielded driving auxiliary information comprises lane related information, correcting the position of the shielded driving auxiliary information based on the driving tracks and/or road rut information of other vehicles in the preset range of the current vehicle;

when the shielded driving assistance information includes road surface road information, correcting the position of the shielded driving assistance information according to a map of a current road by aligning the current road with the map of the current road.

29. The apparatus according to claim 24, wherein when the obstructed driving assistance information comprises: when the information related to the lane is received,

the display module is also used for displaying that the lane width is smaller than the actual lane width.

30. The apparatus of claim 24, wherein when the obstructed driving assistance information comprises: during the blind area information, the display module is used for showing when the virtual three-dimensional display information that is sheltered from driving auxiliary information corresponds, still is used for:

and displaying virtual three-dimensional display information corresponding to the blind area information in the expansion area of the rearview mirror.

31. The apparatus according to claim 30, wherein when the side mirror is a side mirror, the virtual three-dimensional display information displayed in the extended area is generated by a real object corresponding to the virtual three-dimensional display information in accordance with the mirror surface property of the side mirror and the driver's field of view.

32. The apparatus of claim 23,

the determining module is used for acquiring the traffic rule and/or the traffic police action information of the current road section and converting the presentation mode of the determined traffic rule and/or the determined traffic police action information of the current road section;

and the display module is used for displaying the converted virtual three-dimensional display information corresponding to the traffic rules and/or the traffic police action information of the current road section.

33. The apparatus of claim 23, wherein the display module is configured to perform at least one of:

when the abnormal rut information is sensed, displaying virtual three-dimensional display information corresponding to the determined abnormal rut area and/or virtual three-dimensional display information of warning information of the abnormal rut area in the area;

when the traffic sign of the road area which is driven by the current vehicle needs to be displayed, displaying the acquired virtual three-dimensional display information corresponding to the traffic sign of the road area which is driven by the vehicle; the acquired traffic signs which have traveled comprise distance indications; the displayed virtual three-dimensional display information comprises the acquired traveled traffic sign and a distance indication updated based on the distance traveled by the vehicle from the traveled traffic sign;

when the traffic sign and/or the traffic signal lamp at the intersection where the current vehicle is located is sensed to be in existence, and the traffic sign and/or the traffic signal lamp meet the preset display condition, displaying virtual three-dimensional display information corresponding to the traffic sign and/or the traffic signal lamp at the intersection; the predetermined display condition includes at least one of: traffic signs and/or traffic lights damaged; the traffic signs and/or traffic lights are not clearly displayed; the traffic signs and/or traffic lights are not completely within the driver's current visual range;

when the key information in the operation dial needs to be displayed, displaying virtual three-dimensional display information corresponding to at least one of the following information: the position information of the key, the function name information of the key, the operation indication information of the key and the key;

when the information of the parking area needs to be displayed, virtual three-dimensional display information corresponding to at least one of the parking areas which are allowed to be parked and are suitable for the parking area, the parking areas which are allowed to be parked and are not suitable for the parking area and the parking area which is not allowed to be parked is displayed.

34. The apparatus of claim 33, wherein the determining module is configured to perform at least one of:

determining whether the road rut information has abnormal rut information, and if the abnormal rut information exists, determining that an abnormal rut area exists;

when the traffic sign of the road area which is driven by the current vehicle needs to be displayed, the traffic sign of the road area which is driven by the current vehicle is determined from the acquired multimedia information and/or the traffic sign database;

when the parking area information needs to be displayed, at least one of a parking-permitted area which is suitable for parking, a parking-permitted area which is not suitable for parking and a parking-not-permitted area is determined according to at least one of whether a parking-prohibited mark exists in the area around the current vehicle, the volume of the current vehicle and the current road surface condition.

35. The apparatus of claim 23, wherein the display module is configured to display virtual three-dimensional display information corresponding to the driving assistance information, and includes:

and enhancing virtual three-dimensional display information corresponding to the rut information.

36. The device of claim 33, wherein the display module is configured to display the acquired virtual three-dimensional display information corresponding to the traffic sign of the road region on which the current vehicle has traveled, and includes:

and according to the current position of the vehicle and the virtual three-dimensional display information corresponding to the traffic sign of the road area which has already traveled, adjusting the virtual three-dimensional display information corresponding to the traffic sign of the road area which has already traveled currently, and displaying the virtual three-dimensional display information corresponding to the adjusted traffic sign.

37. The apparatus of claim 23, wherein the display module is configured to:

determining a display mode corresponding to the virtual three-dimensional display information;

displaying virtual three-dimensional display information corresponding to the driving auxiliary information based on the determined display mode;

wherein the display mode comprises at least one of the following:

a display position of the virtual three-dimensional display information; displaying the posture of the virtual three-dimensional display information; the display size of the virtual three-dimensional display information; a display start time of the virtual three-dimensional display information; display end time of the virtual three-dimensional display information; displaying duration of the virtual three-dimensional display information; the detail degree of the display content of the virtual three-dimensional display information; presenting the virtual three-dimensional display information; the interrelationship of display among a plurality of virtual three-dimensional display information;

the presentation mode comprises at least one of the following: a text mode; icon mode; an animation mode; a sound effect mode; a lighting mode; a vibration mode.

38. The apparatus of claim 23, wherein the display module is further configured to perform at least one of:

when a plurality of pieces of virtual three-dimensional display information to be displayed exist at the same time, combining the plurality of pieces of virtual three-dimensional display information to be displayed, and displaying the processed virtual three-dimensional display information;

when a plurality of pieces of virtual three-dimensional display information to be displayed are displayed simultaneously, the plurality of pieces of virtual three-dimensional display information to be displayed are integrated based on semantics, and the processed virtual three-dimensional display information is displayed.

39. The apparatus of claim 23, wherein the display module is further configured to perform at least one of:

displaying virtual three-dimensional display information corresponding to the driving auxiliary information with priority higher than the first preset priority at a remarkable position of the current visual field of the driver, and adjusting the position for displaying the virtual three-dimensional display information in real time according to the position of the current visual field of the driver;

displaying virtual three-dimensional display information corresponding to the driving auxiliary information with the priority higher than the first preset priority, and pausing and/or stopping displaying the virtual three-dimensional display information corresponding to the driving auxiliary information with the priority lower than the second preset priority.

40. The apparatus of claim 23, wherein the display module is configured to:

determining at least one of display start time, display end time and display duration of the virtual three-dimensional display information according to at least one of the current state of the vehicle, the current road condition information and the system delay condition of the equipment;

and displaying the virtual three-dimensional display information corresponding to the driving auxiliary information according to at least one of the determined display starting time, display ending time and display duration of the virtual three-dimensional display information.

41. The apparatus of claim 23,

when virtual three-dimensional display information corresponding to a plurality of pieces of driving assistance information to be displayed simultaneously exists and a shielding relation exists between the plurality of pieces of virtual three-dimensional display information to be displayed, the display module is further configured to execute at least one of the following:

according to the position relation among a plurality of pieces of virtual three-dimensional display information with shielding relation, only the part which is not shielded in the virtual three-dimensional display information is displayed;

respectively displaying the virtual three-dimensional display information in the plurality of virtual three-dimensional display information with the shielding relation at different display time;

and adjusting at least one of the display position, the content detail degree and the presentation mode of at least one piece of virtual three-dimensional display information in the plurality of pieces of virtual three-dimensional display information with the shielding relation, and displaying each piece of virtual three-dimensional display information in the plurality of pieces of virtual three-dimensional display information with the shielding relation according to the adjustment mode.

42. The apparatus of claim 23, wherein the display module is configured to:

displaying virtual three-dimensional display information corresponding to the driving auxiliary information to be displayed at a preset display position;

wherein the preset display position comprises at least one of the following items:

a display position aligned with the real driving assistance information; the area position where the driver drives is not interfered; a prominent position of the driver's current field of view; a relatively open driver view; the position of the driver with insufficient attention.

43. The apparatus of claim 23, wherein the display module is further configured to:

rendering virtual three-dimensional display information to be displayed in advance;

when a preset display triggering condition is met, acquiring virtual three-dimensional display information to be displayed from the pre-rendered virtual three-dimensional display information, adjusting the presentation mode of the virtual three-dimensional display information according to the current environment, and displaying the virtual three-dimensional display information according to the adjusted presentation mode;

and adjusting the display mode of the virtual three-dimensional display information in real time according to the current environment, and displaying the virtual three-dimensional display information according to the adjusted display mode.

44. The apparatus of claim 23, wherein the adjusting the manner of presenting the virtual three-dimensional display information comprises:

adjusting a portion of the virtual three-dimensional display information into a stripe pattern.

45. An electronic device comprising a memory and a processor;

the memory has stored therein a computer program;

the processor, when executing the computer program, is configured to perform the method of any of claims 1 to 22.

46. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method of any one of claims 1 to 22.

Images