本申请为分案申请，其母案的发明名称为“虚拟现实系统和方法”，申请日为2019年3月13日，申请号为201980018658.6。This application is a divisional application, and the invention name of its parent case is "Virtual Reality System and Method", the filing date is March 13, 2019, and the application number is 201980018658.6.
相关申请的交叉引用Cross References to Related Applications
本申请要求于2018年3月13日提交的美国专利申请序列号62/642,488的优先权，其全部内容通过引用合并于此。This application claims priority to US Patent Application Serial No. 62/642,488, filed March 13, 2018, the entire contents of which are hereby incorporated by reference.
技术领域technical field
本说明书一般地涉及用于捕捉和处理二维(2D)和三维(3D)图像的方法和设备。The present description generally relates to methods and devices for capturing and processing two-dimensional (2D) and three-dimensional (3D) images.
背景技术Background technique
虚拟现实(VR)带来了非凡的沉浸式体验，提供了观看世界的新方式以及探索真实和虚构的新环境的能力。然而，与物理现实相比，与其他人共享这些体验可能很困难，因为VR头戴式视图器(headset)使得创建参与体验的人的完整图景成为挑战。Virtual reality (VR) enables extraordinary immersive experiences, offering new ways of seeing the world and the ability to explore new environments, both real and imaginary. However, sharing these experiences with others can be difficult compared to physical reality, as VR headsets make it challenging to create a complete picture of those participating in the experience.
通过混合现实(MR)减轻了该脱节现象(disconnect)中的一些，作为相关的媒体，其以二维视频格式共享VR用户的虚拟上下文(context)，允许其他观看者感受到该用户的虚拟体验。虽然MR促进了共享，但头戴式视图器继续阻挡面部表情和眼睛注视，这对于完全参与VR中的人的体验和完整视图构成了重大障碍。Some of this disconnect is mitigated by Mixed Reality (MR), as a related medium, which shares the VR user's virtual context in a 2D video format, allowing other viewers to experience the user's virtual experience . While MR facilitates sharing, headsets continue to block facial expressions and eye gaze, which poses a significant barrier to the experience and full view for those fully engaged in VR.
发明内容Contents of the invention
下文描述的所示实施例的目的和优势将在下面的描述中阐明并从中清楚。通过在撰写的说明书及其权利要求书中以及从附图特别指出的设备、系统和方法，将实现并获得所示实施例的附加优势。Purposes and advantages of the illustrated embodiments described hereinafter will be set forth in, and apparent from, the description that follows. Additional advantages of the illustrated embodiments will be realized and attained by the devices, systems and methods particularly pointed out in the written description and claims hereof as well as from the accompanying drawings.
该项目的一般目的是提供实况(live)VR聊天体验。当前，VR聊天应用使用动画化身。本发明的目的是用没有用户的头戴式视图器的图像的实况视频代替化身，用户的头戴式视图器的图像将被用户的头部的3D模型(例如，在用户没有穿戴头戴式视图器的情况下先前记录的)可视地代替。本发明的另一个目的是每个用户可以调整他们的现实水平(例如，从化身到实况3D视频图像)。The general purpose of this project is to provide a live (live) VR chat experience. Currently, VR chat apps use animated avatars. The purpose of the present invention is to replace the avatar with a live video without an image of the user's headset, which would be replaced by a 3D model of the user's head (e.g., when the user is not wearing the headset). previously documented in the case of the viewer) is visually replaced. Another object of the invention is that each user can adjust their reality level (eg, from avatar to live 3D video image).
附图说明Description of drawings
所附的附录和/或附图示出了根据本公开的各种非限制性的示例创造性方面：The appended appendices and/or figures illustrate various non-limiting example inventive aspects according to the present disclosure:
图1示出了可以用于实现本文描述的技术的计算机设备和移动设备的示例；Figure 1 shows an example of a computer device and a mobile device that can be used to implement the techniques described herein;
图2描绘了穿戴头戴式视图器的用户以及另一头戴式视图器穿戴用户的头戴式视图器穿戴用户感知的视图；2 depicts a view of the headset-wearing user's perception of a user wearing the headset and another headset-wearing user;
图3描绘了从用户移除头戴式视图器的图像以创建用户的VR就像头戴式视图器被移除一样的过程；以及Figure 3 depicts the process of removing an image of the headset from the user to create the user's VR as if the headset were removed; and
图4-7示出了根据所描述的示出实施例的用于在用户之间创建VR聊天的过程的部分。4-7 illustrate portions of a process for creating a VR chat between users in accordance with the described illustrated embodiments.
具体实施方式Detailed ways
现在参考附图更充分地描述所示的实施例，其中，相同的附图标记标识相似的结构/功能特征。所示的实施例不以任何方式被限制为所示的内容，因为以下描述的所示的实施例仅是示例性的，其可以以各种形式实施，如本领域的技术人员所理解的那样。因此，应当理解，本文公开的任何结构和功能细节都不解释为限制性的，而仅仅作为权利要求的基础以及作为教导本领域技术人员各种使用所讨论的实施例的表示。此外，本文所使用的术语和短语不旨在是限制性的，而是提供对所示实施例的可理解的描述。The illustrated embodiments are now described more fully with reference to the drawings, wherein like reference numerals identify like structural/functional features. The illustrated embodiments are not limited in any way to what is shown, as the illustrated embodiments described below are merely exemplary, which may be implemented in various forms, as would be understood by those skilled in the art . Therefore, it should be understood that any structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a basis for the claims and as representations for teaching one skilled in the art to various uses of the discussed embodiments. Furthermore, the terms and phrases used herein are not intended to be limiting, but to provide an understandable description of the illustrated embodiments.
在提供值的范围的情况下，应理解，该范围的上限和下限与该所述范围中的任何其他所述或居间值之间的、除非上下文另有明确规定否则达到下限的单位的十分之一(thetenth of the unit)的每个居间值(intervening value)被包括在所示实施例内。这些较小范围的上限和下限可以独立地被包括在较小范围中，其也被包括在所示实施例之内，受制于所述范围内的任何具体排除的限制。在所述范围包括一个或两个限制的情况下，排除那些包括的限制的任两个的范围也被包括在所示实施例中。Where a range of values is provided, it is understood that tenths of the unit between the upper and lower limits of that range and any other stated or intervening value in that stated range up to the lower limit unless the context clearly dictates otherwise Each intervening value of the tenth of the unit is included within the illustrated embodiment. The upper and lower limits of these smaller ranges may independently be included in the smaller ranges, which are also encompassed within the illustrated embodiments, subject to any specifically excluded limit in the stated range. Where the stated range includes one or both of the limits, ranges excluding either two of those included limits are also included in the illustrated embodiments.
除非另有定义，否则本文使用的所有技术和科学术语具有与由本发明所属领域的普通技术人员通常所理解的含义相同的含义。虽然类似于或等同于本文描述的那些方法和材料的任何方法和材料也可以用于实践或测试所示实施例，但是现在描述示例性方法和材料。本文提及的所有出版物通过引用并入本文，以公开和描述与引用出版物有关的方法和/或材料。Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although any methods and materials similar or equivalent to those described herein can also be used in the practice or testing of the illustrated embodiments, exemplary methods and materials are now described. All publications mentioned herein are incorporated herein by reference to disclose and describe the methods and/or materials in connection with which the publications are cited.
必须注意的是，如本文和所附权利要求书中所使用的，单数形式“一”、“一个”和“该”包括复数参照物(referent)，除非上下文另有明确规定。因此，例如，对“刺激”的提及包括多个这样的刺激并且对“该信号”的提及包括对一个或多个信号及本领域技术人员已知的其等同物的提及，以及诸如此类。It must be noted that, as used herein and in the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a stimulus" includes a plurality of such stimuli and reference to "the signal" includes reference to one or more signals and equivalents thereof known to those skilled in the art, and so on. .
提供本文讨论的出版物仅是为了本申请的提交日期之前的它们的公开。本文中没有内容被解释为承认所示实施例无权借助在先发明而早于此类出版物。此外，提供的出版的日期可能与实际出版的日期有所不同，这可能需要独立确认。The publications discussed herein are provided only for their disclosure prior to the filing date of the present application. Nothing herein is to be construed as an admission that the illustrated embodiments are not entitled to antedate such publication by virtue of prior invention. In addition, the dates of publication provided may differ from the actual publication dates, which may need to be independently confirmed.
应当理解，下面讨论的所示实施例优选地是驻留在具有控制逻辑的计算机可用介质上的软件算法、程序或代码，该控制逻辑用于使能在具有计算机处理器的机器上的执行。该机器通常包括存储器存储装置，其被配置为提供来自计算机算法或程序的执行的输出。It should be understood that the illustrated embodiments discussed below are preferably software algorithms, programs or codes residing on a computer usable medium with control logic for enabling execution on a machine with a computer processor. The machines typically include memory storage devices configured to provide output from the execution of computer algorithms or programs.
如本文所使用的，术语“软件”意味着是与可以在主机计算机的处理器中的任何代码或程序的同义词，而不论实现在硬件、固件中还是作为盘、存储器存储设备上可用的软件计算机产品或用于从远程机器下载的软件计算机产品。本文描述的实施例包括用于实现上述方程、关系和算法的此类软件。基于上述实施例，本领域技术人员将理解所示实施例的进一步的特征和优势。因此，除了由所附权利要求书所指示的以外，所示实施例不受已经特别示出和描述的内容的限制。本文引用的所有出版物和参考文献均通过引用全文明确地并入本文。As used herein, the term "software" is meant to be synonymous with any code or program that can reside in the processor of a host computer, whether implemented in hardware, firmware, or as software available on a disk, memory storage device Product or software computer product for downloading from a remote machine. Embodiments described herein include such software for implementing the equations, relationships, and algorithms described above. Based on the above-described embodiments, those skilled in the art will appreciate further features and advantages of the illustrated embodiments. Accordingly, the illustrated embodiments are not to be limited by what has been particularly shown and described, except as indicated by the appended claims. All publications and references cited herein are expressly incorporated by reference in their entirety.
现在描述性地转向附图，在附图中，相似的附图标记贯穿若干视图表示相似的元素，图1描绘了示例性通用计算系统600和通用移动计算机设备650，它们可以与这里描述的技术一起使用。应当理解，计算系统600和移动设备650应被理解为示例性实施例，其中其一个或多个部件可以与如本文所述的本发明的所示实施例一起使用。Turning now descriptively to the drawings, in which like numerals represent like elements throughout the several views, FIG. use together. It should be appreciated that computing system 600 and mobile device 650 are to be understood as exemplary embodiments wherein one or more components thereof may be used with the illustrated embodiments of the invention as described herein.
计算设备600包括处理器602、存储器604、存储设备606、连接到存储器604和高速扩展端口610的高速接口608以及连接到低速总线614和存储设备606的低速接口612。使用各种总线互连部件602、604、606、608、610和612中的每个，并且互连部件602、604、606、608、610和612中的每个可以安装在共同主板上或以其他适当方式安装。处理器602可以处理用于在计算设备600内执行的指令，包括存储在存储器604中或存储设备606上以在外部输入/输出设备上显示GUI的图形信息的指令，外部输入/输出设备诸如是耦合到高速接口608的显示器616。在其他实现中，可以适当地使用多个处理器和/或多个总线，连同多个存储器和存储器的类型。另外，可以连接多个计算设备600，其中每个设备提供必要操作的部分(例如，作为服务器组、刀片服务器的组或多处理器系统)。Computing device 600 includes processor 602 , memory 604 , storage device 606 , high-speed interface 608 connected to memory 604 and high-speed expansion port 610 , and low-speed interface 612 connected to low-speed bus 614 and storage device 606 . Each of the interconnect components 602, 604, 606, 608, 610, and 612 are interconnected using various buses, and each of the interconnect components 602, 604, 606, 608, 610, and 612 may be mounted on a common motherboard or in a other appropriate means of installation. Processor 602 may process instructions for execution within computing device 600, including instructions stored in memory 604 or on storage device 606, to display graphical information for a GUI on an external input/output device, such as Display 616 coupled to high speed interface 608 . In other implementations, multiple processors and/or multiple buses may be used as appropriate, along with multiple memories and types of memories. Additionally, multiple computing devices 600 may be connected, with each device providing portions of the necessary operations (eg, as a bank of servers, a bank of blade servers, or a multi-processor system).
存储器604将信息存储在计算设备600内。在一个实现中，存储器604是一个或多个易失性存储器单元。在另一实现中，存储器604是一个或多个非易失性存储器单元。存储器604也可以是另一种形式的计算机可读介质，诸如磁盘或光盘。Memory 604 stores information within computing device 600 . In one implementation, memory 604 is one or more volatile memory units. In another implementation, memory 604 is one or more non-volatile memory units. Memory 604 may also be another form of computer-readable medium, such as a magnetic or optical disk.
存储设备606能够为计算设备600提供大容量存储。在一个实现中，存储设备606可以是或包含计算机可读介质，诸如软盘设备、硬盘设备、光盘设备或带设备、闪存或其他类似的固态存储器设备，或设备的阵列，包括存储区域网络中或其他配置的设备。计算机程序产品可以有形地体现在信息载体中。计算机程序产品还可以包含指令，指令在被执行时执行一种或多种方法，诸如上述那些方法。信息载体是计算机或机器可读介质，诸如存储器604、存储设备606或处理器602上的存储器。The storage device 606 is capable of providing mass storage for the computing device 600 . In one implementation, the storage device 606 may be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device or tape device, a flash memory or other similar solid-state memory device, or an array of devices, including in a storage area network or other configured devices. A computer program product can be tangibly embodied in an information carrier. The computer program product may also contain instructions which, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as memory 604 , storage device 606 or memory on processor 602 .
高速控制器608管理计算设备600的带宽密集型操作，而低速控制器612管理较低带宽密集型操作。这种功能分配仅是示例性的。在一个实现中，高速控制器608(例如，通过图形处理器或加速器)耦合到存储器604、显示器616，并且耦合到可以接受各种扩展卡(未示出)的高速扩展端口610。在该实现中，低速控制器612耦合到存储设备606和低速扩展端口614。可以包括各种通信端口(例如，USB、蓝牙、以太网、无线以太网)的低速扩展端口可以例如通过网络适配器耦合到一个或多个输入/输出设备，诸如键盘、指点设备、扫描仪或联网设备，诸如交换机或路由器。High-speed controller 608 manages bandwidth-intensive operations of computing device 600 , while low-speed controller 612 manages less bandwidth-intensive operations. This allocation of functions is exemplary only. In one implementation, high-speed controller 608 is coupled (eg, through a graphics processor or accelerator) to memory 604, display 616, and to high-speed expansion ports 610, which can accept various expansion cards (not shown). In this implementation, low-speed controller 612 is coupled to storage device 606 and low-speed expansion port 614 . Low-speed expansion ports, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input/output devices, such as keyboards, pointing devices, scanners, or networked devices, such as through network adapters. A device, such as a switch or router.
如图中所示，计算设备600可以以多种不同的形式实现。例如，它可以被实现为标准服务器620，或者在这样的服务器的组中实现多次。它也可以被实现为机架服务器系统624的部分。此外，它还可以被实现在诸如膝上型计算机622之类的个人计算机中。替代地，来自计算设备600的部件可以与移动设备中的其他部件(未示出)组合，移动设备诸如是设备650。这样的设备中的每个可以包含计算设备600、650中的一个或多个，并且整个系统可以由彼此通信的多个计算设备600、650组成。As shown in the figure, computing device 600 may be implemented in a variety of different forms. For example, it may be implemented as a standard server 620, or multiple times in a group of such servers. It can also be implemented as part of rack server system 624 . Furthermore, it can also be implemented in a personal computer such as laptop computer 622 . Alternatively, components from computing device 600 may be combined with other components (not shown) in a mobile device, such as device 650 . Each of such devices may contain one or more of computing devices 600, 650, and the overall system may consist of multiple computing devices 600, 650 in communication with each other.
除其他部件之外，计算设备650包括处理器652、存储器664、诸如显示器654、通信接口666和收发器668之类的输入/输出设备。设备650还可以配备有存储设备，诸如微驱动器或其他设备，以提供附加的存储。使用各种总线互连部件650、652、664、654、666和668中的每个，并且部件中的若干部件可以安装在公共主板上或以适当的其他方式安装。Computing device 650 includes, among other components, a processor 652 , memory 664 , input/output devices such as a display 654 , a communication interface 666 , and a transceiver 668 . Device 650 may also be equipped with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 650, 652, 664, 654, 666, and 668 are interconnected using various buses, and several of the components may be mounted on a common motherboard or otherwise mounted as appropriate.
处理器652可以在计算设备650内执行指令，所述指令包括存储在存储器664中的指令。处理器可以被实现为包括分离的以及多个模拟和数字处理器的芯片的芯片组。处理器可以提供例如设备650的其他部件的协调，诸如对用户界面、由设备650运行的应用以及由设备650进行的无线通信的控制。Processor 652 may execute instructions within computing device 650 , including instructions stored in memory 664 . A processor may be implemented as a chipset including separate and multiple analog and digital processor chips. The processor may provide, for example, coordination of other components of device 650 , such as control of a user interface, applications run by device 650 , and wireless communications by device 650 .
处理器652可以通过耦合到显示器654的显示接口656和控制接口658与用户通信。显示器654可以是例如TFT LCD(薄膜晶体管液晶显示器)或OLED(有机发光二极管)显示器，或其他适当的显示技术。显示接口656可以包括用于驱动显示器654以向用户呈现图形和其他信息的适当电路。控制接口658可以从用户接收命令并且转换它们用于提交给处理器652。另外，可以提供与处理器652通信的外部接口662，以便使能设备650与其他设备的附近区域通信。外部接口662可以例如在一些实现中提供有线通信，或在其他实现中提供无线通信，并且也可以使用多个接口。Processor 652 can communicate with a user through display interface 656 and control interface 658 coupled to display 654 . Display 654 may be, for example, a TFT LCD (Thin Film Transistor Liquid Crystal Display) or OLED (Organic Light Emitting Diode) display, or other suitable display technology. Display interface 656 may include appropriate circuitry for driving display 654 to present graphical and other information to a user. Control interface 658 may receive commands from a user and convert them for submission to processor 652 . Additionally, an external interface 662 may be provided in communication with the processor 652 to enable the device 650 to communicate with other devices in the vicinity. External interface 662 may, for example, provide for wired communication in some implementations or wireless communication in other implementations, and multiple interfaces may also be used.
存储器664在计算设备650内存储信息。存储器664可以被实现为一个或多个计算机可读介质、一个或多个易失性存储器单元或一个或多个非易失性存储器单元中的一个或多个。还可以提供扩展存储器674，并通过扩展接口672将其连接到设备650，扩展接口672可以包括例如SIMM(单列存储器模块)卡接口。这样的扩展存储器674可以为设备650提供额外的存储空间，或者还可以为设备650存储应用或其他信息。具体地，扩展存储器674可以包括用于执行或补充上述过程的指令，并且还可以包括安全信息。因此，例如，扩展存储器674可以被提供为设备650的安全模块，并且可以用允许安全使用设备650的指令来编程。此外，可以经由SIMM卡连同附加信息来提供安全应用，诸如以非可入侵(non-hackable)方式在SIMM卡上放置标识信息。Memory 664 stores information within computing device 650 . Memory 664 may be implemented as one or more of one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. Expansion memory 674 may also be provided and connected to device 650 via expansion interface 672, which may include, for example, a SIMM (Single Inline Memory Module) card interface. Such expanded memory 674 may provide additional storage space for device 650 or may also store applications or other information for device 650 . Specifically, the expansion memory 674 may include instructions for performing or supplementing the above-described processes, and may also include security information. Thus, for example, expansion memory 674 may be provided as a security module of device 650 and may be programmed with instructions that allow device 650 to be used securely. Furthermore, security applications can be provided via the SIMM card along with additional information, such as placing identification information on the SIMM card in a non-hackable manner.
存储器可以包括例如闪存和/或NVRAM存储器，如下所述。在一个实现中，计算机程序产品有形地体现在信息载体中。计算机程序产品包含指令，指令在被执行时执行一种或多种方法，诸如上述那些方法。信息载体是计算机或机器可读介质，诸如存储器664、扩展存储器674或处理器652上的存储器，其可以例如通过收发器668或外部接口662被接收。Memory may include, for example, flash memory and/or NVRAM memory, as described below. In one implementation, a computer program product is tangibly embodied in an information carrier. A computer program product comprises instructions which, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as memory 664 , extended memory 674 or memory on processor 652 , which can be received eg via transceiver 668 or external interface 662 .
设备650可以通过通信接口666无线通信，通信接口666在必要时可以包括数字信号处理电路。通信接口666可以提供各种模式或协议下的通信，各种模式或协议除其他之外诸如是GSM语音呼叫、SMS、EMS或MMS消息收发、CDMA、TDMA、PDC、WCDMA、CDMA2000或GPRS。这样的通信可以例如通过射频收发器668发生。此外，可以诸如使用蓝牙、Wi-Fi或其他这样的收发器(未示出)发生短距离通信。此外，GPS(全球定位系统)接收器模块670可以向设备650提供附加导航以及位置相关无线数据，设备650上运行的应用可以适当使用这些数据。Device 650 may communicate wirelessly through communication interface 666, which may include digital signal processing circuitry, if necessary. The communication interface 666 may provide communication in various modes or protocols such as GSM voice calling, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, among others. Such communication may occur via radio frequency transceiver 668, for example. Additionally, short-range communications may occur, such as using Bluetooth, Wi-Fi, or other such transceivers (not shown). In addition, GPS (Global Positioning System) receiver module 670 can provide additional navigational and location-related wireless data to device 650, which can be used as appropriate by applications running on device 650.
设备650还可以使用音频编解码器660可听地进行通信，音频编解码器660可以从用户接收语音信息并将其转换为可用的数字信息。音频编解码器660可以同样地为用户生成可听的声音，诸如通过例如设备650的头戴式视图器中的扬声器。这种声音可以包括来自语音电话呼叫的声音，可以包括记录的声音(例如，语音消息、音乐文件、等等)，并且还可以包括由在设备650上操作的应用生成的声音。Device 650 can also communicate audibly using audio codec 660, which can receive spoken information from a user and convert it into usable digital information. Audio codec 660 may likewise generate audible sound for the user, such as through speakers in a headset of device 650, for example. Such sounds may include sounds from voice telephone calls, may include recorded sounds (eg, voice messages, music files, etc.), and may also include sounds generated by applications operating on device 650 .
如图中所示，可以以多种不同的形式来实现计算设备650。例如，它可以被实现为蜂窝电话680。它也可以被实现为智能电话682、个人数字助理或其他类似的移动设备的部分。As shown in the figure, computing device 650 may be implemented in a variety of different forms. It can be implemented as a cellular phone 680, for example. It may also be implemented as part of a smartphone 682, personal digital assistant, or other similar mobile device.
可以在数字电子电路、集成电路、专门设计的ASIC(专用集成电路)、计算机硬件、固件、软件和/或其组合中实现此处描述的系统和技术的各种实现。这些各种实现可以包括在一个或多个计算机程序中的实现，该一个或多个计算机程序在包括至少一个可编程处理器的可编程系统上可执行和/或可解释，该至少一个可编程处理器可以是专用的或通用的，耦合以从存储系统、至少一个输入设备和至少一个输出设备接收数据和指令并向存储系统、至少一个输入设备和至少一个输出设备传输数据和指令。Various implementations of the systems and techniques described herein can be implemented in digital electronic circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations may include implementation in one or more computer programs executable and/or interpretable on a programmable system comprising at least one programmable processor, the at least one programmable The processor, which may be special purpose or general purpose, is coupled to receive data and instructions from and transmit data and instructions to the storage system, at least one input device, and at least one output device.
这些计算机程序(也称为程序、软件、软件应用或代码)包括用于可编程处理器的机器指令，并且可以以高级过程和/或面向对象的编程语言和/或以汇编/机器语言实现。如本文所使用的，术语“机器可读介质”“计算机可读介质”是指用于向可编程处理器提供机器指令和/或数据的任何计算机程序产品、装置和/或设备(例如磁盘、光盘、存储器、可编程逻辑器件(PLD))，包括接收机器指令作为机器可读信号的机器可读介质。术语“机器可读信号”是指用于向可编程处理器提供机器指令和/或数据的任何信号。These computer programs (also known as programs, software, software applications or codes) comprise machine instructions for a programmable processor and may be implemented in a high-level procedural and/or object-oriented programming language and/or in assembly/machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and/or device (such as a disk, Optical discs, memories, programmable logic devices (PLDs), including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
为了提供与用户的交互，此处描述的系统和技术可以在具有用于向用户显示信息的显示设备(例如CRT(阴极射线管)或LCD(液晶显示器)监视器)(以及其他合适的显示技术)以及用户通过其可以向计算机提供输入的键盘和指点设备(例如鼠标或跟踪球)的计算机上实现。其他种类的设备也可以用于提供与用户的交互；例如，提供给用户的反馈可以是任何形式的感觉反馈(例如视觉反馈、听觉反馈或触觉反馈)；并且可以以任何形式接收来自用户的输入，包括声音、语音或触觉输入。To provide interaction with the user, the systems and techniques described herein may be implemented on a display device (such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) (and other suitable display technology) for displaying information to the user. ) and a keyboard and pointing device (such as a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (such as visual, auditory, or tactile feedback); and input from the user may be received in any form , including sound, speech, or tactile input.
此处描述的系统和技术可以实现在计算系统中，所述计算系统包括后端部件(例如，作为数据服务器)，或包括中间件部件(例如，应用服务器)或包括前端部件(例如，具有图形用户接口或Web浏览器的客户端计算机，用户可以通过图形用户接口或Web浏览器与此处描述的系统和技术的实现进行交互)，或实现在此类后端、中间件或前端部件的任何组合中。系统的部件可以通过数字数据通信的任何形式或介质(例如，通信网络)互连。通信网络的示例包括局域网(“LAN”)、广域网(“WAN”)和因特网。The systems and techniques described herein can be implemented in computing systems that include back-end components (e.g., as data servers), or include middleware components (e.g., application servers), or include front-end components (e.g., with graphics a client computer with a user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any implementation of such back-end, middleware, or front-end components in combination. The components of the system can be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), and the Internet.
计算系统可以包括客户端和服务器。客户端和服务器通常彼此远离，并且通常通过通信网络进行交互。客户端和服务器的关系是通过在相应计算机上运行并彼此具有客户端-服务器关系的计算机程序产生的。A computing system can include clients and servers. Clients and servers are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
图1中描绘的计算设备与虚拟现实(VR头戴式视图器690)对接。在所示实施例中，VR头戴式视图器中包括一个或多个传感器，以在一个用户与另一用户交互时虚拟移除头戴式视图器690的外观。例如，头戴式视图器传感器690可以包括在跟踪用户的瞳孔时照亮用户的眼睛的一个或多个LED 691和用于记录用户的眼睛/瞳孔的一个或多个微型照相机693。The computing device depicted in Figure 1 interfaces with virtual reality (VR headset 690). In the illustrated embodiment, one or more sensors are included in the VR headset to virtually remove the appearance of the headset 690 when one user interacts with another user. For example, the headset sensor 690 may include one or more LEDs 691 to illuminate the user's eyes while tracking the user's pupils and one or more miniature cameras 693 to record the user's eyes/pupils.
例如，本发明的实施例在穿戴头戴式视图器690时使用照相机设备400作为用户的隐藏的面部的代理(proxy)来创建没有头戴式视图器690的用户的面部704的模型。该代理用于在混合现实(MR)视频中合成面部，从而产生如下效果(impression)：当一个使用(oneusing)在VR领域中与另一个用户进行交互时，头戴式视图器690被移除。例如，该初始校准步骤可能要求用户704坐在彩色+深度(color+depth)照相机400和监视器前面，并且然后利用他们的眼睛跟踪监视器上的标记。该校准过程获取用户704的3D面部模型，并且创建数据库，该数据库将外观图像(或纹理)映射到不同的眼睛注视方向和眨眼(经由传感器691和693)。前述眼睛注视数据库(即，优选地具有通过眼睛注视索引的纹理的面部模型)使得用户的面部的外观能够被动态地改变以生成期望的(和实时的)眼睛注视，因此使合成的面部看起来自然且活泼。应当理解，也可以通过使用适当的线框软件技术来改变面部的构造。For example, embodiments of the invention use the camera device 400 as a proxy for the user's hidden face while wearing the headset 690 to create a model of the user's face 704 without the headset 690. This agent is used to synthesize faces in Mixed Reality (MR) video, resulting in the impression that the headset 690 is removed while oneusing interacts with another user in the VR realm . For example, this initial calibration step may require the user 704 to sit in front of the color+depth camera 400 and monitor, and then use their eyes to track markers on the monitor. This calibration process takes a 3D facial model of the user 704 and creates a database that maps appearance images (or textures) to different eye gaze directions and blinks (via sensors 691 and 693). The aforementioned eye gaze database (i.e., preferably a facial model with texture indexed by eye gaze) enables the appearance of the user's face to be dynamically changed to generate the desired (and real-time) eye gaze, thus making the synthesized face look Natural and lively. It should be understood that the configuration of the face may also be altered through the use of appropriate wireframing software techniques.
在一些实现中，除了计算设备之外的一个或多个输入设备(例如，鼠标、键盘)可以在计算机生成的3D环境中被呈现(render)。呈现的输入设备(例如，呈现的鼠标、呈现的键盘)可以在VR空间中如呈现的那样使用以控制VR空间中的对象。In some implementations, one or more input devices (eg, mouse, keyboard) other than the computing device may be rendered in the computer-generated 3D environment. A presented input device (eg, presented mouse, presented keyboard) can be used as presented in the VR space to control objects in the VR space.
计算设备600旨在表示各种形式的数字计算机，诸如膝上型计算机、台式计算机、工作站、个人数字助理、服务器、刀片服务器、大型机和其他适当的计算机。计算设备650旨在表示各种形式的移动设备，诸如个人数字助理、蜂窝电话、智能电话和其他类似的计算设备。此处所示的部件、它们的连接和关系以及它们的功能意味着仅是示例性的，并不意味着限制本文档中描述和/或要求保护的本发明的实现。Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device 650 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed in this document.
在上面一般地示出和讨论图1的示例性计算系统环境100的情况下，以及现在参考图2-7，在下面描述的是各种示例性实施例的实现。With the exemplary computing system environment 100 of FIG. 1 generally illustrated and discussed above, and with reference now to FIGS. 2-7 , described below are implementations of various exemplary embodiments.
应当理解，用于本发明的所示实施例的使用的环境涉及一种系统600，该系统600优选地利用实况视频使能VR中的在线聊天。在下面讨论的用于描述所示实施例的示例中，应当理解，其假设两个人与本地用户和远程用户实况聊天(本发明不应被理解为局限于此)。示例性实施例包括上述系统600的一个或多个部件，包括诸如Oculus RiftTM之类的VR系统、每个用户一个或多个网络摄像头400以及优选的绿屏。参考图2，在根据所示实施例的VR聊天会话期间，穿戴着头戴式视图器690的远程用户700例如在VR中将观看另一用户702(穿戴着头戴式视图器690)的实况视频，减去本地用户正穿戴着704的头戴式视图器690。同样，在相同的VR聊天会话期间，穿戴着头戴式视图器690的远程用户702在VR中将观看用户700(穿戴着头戴式视图器690)的实况视频，减去本地用户正穿戴着706的头戴式视图器690。It should be understood that the context for use of the illustrated embodiments of the present invention relates to a system 600 that preferably utilizes live video to enable online chat in VR. In the examples discussed below to describe the illustrated embodiments, it should be understood that it is assumed that two people are chatting live with a local user and a remote user (the invention should not be construed as so limited). The exemplary embodiment includes one or more components of the system 600 described above, including a VR system such as Oculus Rift™ , one or more webcams 400 per user, and preferably a green screen. Referring to FIG. 2, during a VR chat session according to the illustrated embodiment, a remote user 700 wearing a headset 690 will view a live feed of another user 702 (wearing the headset 690), for example in VR. Video, minus the headset 690 that the local user is wearing 704. Likewise, during the same VR chat session, remote user 702 wearing headset 690 will be watching live video of user 700 (wearing headset 690) in VR, minus the local user wearing headset 690. 706 of the headset 690 .
根据本发明的一个方面(并且参考图3)，远程用户的头戴式视图器视觉被本地用户的头部的预先记录的静态图像/视频的覆盖物代替。优选地，需要叠加用户头部的仅一部分，因为当穿戴着VR头戴式视图器时，嘴和面部的下部仍是大致可见的。优选地，优选地使用标准2D网络摄像头400(图4)通过抠像(chromakey)(“绿屏”)技术来创建实况视频效果。因此，根据所示实施例，每个用户704、706优选地被放置在便携式绿屏(402)的前面。应当理解，这可以通过某些软件技术来消除，或者替代地，通过使用3D照相机在没有绿屏的情况下确定用户轮廓可以是可能的。According to one aspect of the invention (and referring to FIG. 3 ), the remote user's headset vision is replaced by a pre-recorded still image/video overlay of the local user's head. Preferably, only a portion of the user's head needs to be superimposed, since the mouth and lower part of the face are still generally visible when the VR headset is worn. Preferably, the live video effect is created by chromakey ("green screen") techniques, preferably using a standard 2D webcam 400 (Fig. 4). Thus, according to the illustrated embodiment, each user 704, 706 is preferably placed in front of the portable green screen (402). It should be understood that this could be eliminated by some software techniques, or alternatively it might be possible to determine the user profile without a green screen by using a 3D camera.
如下所述，根据所示实施例，两个或更多用户(704和706)可以在每个用户位置处使用特殊的网络摄像头400来参与实况在线聊天。每个用户(704、706)优选地穿戴VR头戴式视图器690(700、702)，从而每个用户(704、706)观看另一个用户的头部(706、704)的叠加和预先记录的图像。广播图像优选地是实况视频和预先记录的图像的合成。实况视频优选地包括直到仅用户的嘴部区域上方的身体(图3)(例如，VR头戴式视图器690通常驻留的位置)。因此，记录的图像优选地是用户(704、706)的鼻子、眼睛和头部的顶部。应当理解，如下应该是可能的：实现与两个以上的人的实况聊天，诸如公司电话会议。As described below, according to the illustrated embodiment, two or more users (704 and 706) may participate in a live online chat using a special webcam 400 at each user location. Each user (704, 706) preferably wears a VR headset 690 (700, 702) such that each user (704, 706) views an overlay and pre-recorded of the other user's head (706, 704) Image. The broadcast image is preferably a composite of live video and pre-recorded images. The live video preferably includes up to only the user's body ( FIG. 3 ) above the mouth area (eg, where the VR headset 690 typically resides). Therefore, the recorded images are preferably the nose, eyes and top of the head of the user (704, 706). It should be appreciated that it should be possible to implement a live chat with more than two people, such as a corporate conference call.
该记录的图像也可以使用特殊形式的摄影测量法(photogrammetry)在之前创建。这是其中将多个2D图像缝合在一起并进行处理以创建3D模型和纹理的技术。当前，该过程是昂贵的、耗时的并且需要特殊的硬件和多个照相机。相反，根据本发明的实施例，可以利用廉价的立体照相机的实现，其由面对用户的两个照相机组成，每个照相机具有立体或单视场(monoscopic)视觉。例如，当照相机记录并捕捉场景时，用户将在他们的椅子中旋转。立体或单视场照相机输出的后处理将用于生成3D模型。This recorded image may also have been previously created using a special form of photogrammetry. This is the technique in which multiple 2D images are stitched together and processed to create 3D models and textures. Currently, the process is expensive, time-consuming and requires special hardware and multiple cameras. Instead, an inexpensive stereoscopic camera implementation consisting of two cameras facing the user, each with stereoscopic or monoscopic vision, may be utilized in accordance with embodiments of the present invention. For example, the user will spin in their chair while the camera records and captures the scene. Post-processing of stereo or monoscopic camera output will be used to generate the 3D model.
现在参考图4-7，描述了根据所示实施例的用于使能VR会话的示例性过程。首先，由网络摄像头400对没有头戴式视图器的每个用户704(图4)、706(图5)的面部拍照。如图6和7中所示，每个用户(704、706)然后穿戴头戴式视图器690，头戴式视图器690具有用于检测眼睛注视、移动和眨眼的前述LED传感器691和693。优选地，当微型摄像机传感器693记录用户的瞳孔时，传感器691照亮头戴式视图器690内的用户的一只眼睛以跟踪瞳孔移动。处理芯片602优选地被配置并且可操作以通过优选地经由软件“缝合”移动瞳孔来将移动瞳孔恢复到面部来完成头戴式视图器690当前遮挡的面部的图像。因此，用户704和706两者都可以实时地观看彼此的3D图像(经由他们各自的头戴式视图器690)，就好像另一个用户704、706的面部在他们的前面并且未被头戴式视图器690遮挡一样。Referring now to FIGS. 4-7 , exemplary processes for enabling a VR session in accordance with the illustrated embodiments are described. First, the face of each user 704 ( FIG. 4 ), 706 ( FIG. 5 ) without a headset is photographed by the webcam 400 . As shown in Figures 6 and 7, each user (704, 706) then wears a headset 690 with the aforementioned LED sensors 691 and 693 for detecting eye gaze, movement and blinking. Preferably, while the miniature camera sensor 693 records the user's pupil, the sensor 691 illuminates one eye of the user within the headset 690 to track pupil movement. The processing chip 602 is preferably configured and operable to complete the image of the face currently occluded by the headset 690 by restoring the moving pupil to the face, preferably via software "stitching" of the moving pupil. Thus, both users 704 and 706 can view each other's 3D images in real-time (via their respective headsets 690) as if the other user's 704, 706 face was in front of them and not being held by the headset. Viewer 690 occlusion is the same.
应当进一步理解的是，通过进一步使用优选地位于每个用户的额头和眉毛上的电敏设备，还可以经由处理芯片602上的软件处理来跟踪每个用户704、706的皱眉表情。It should be further understood that the frown expression of each user 704, 706 may also be tracked via software processing on the processing chip 602 by further use of electro-sensitive devices preferably located on each user's forehead and eyebrows.
总之，并且根据一个或多个所示实施例，本发明使能在视频聊天中的用户的网络摄像头视频流的虚拟投影，减去用户的头戴式视图器。这是通过使用诸如摄影测量法之类的建模技术以3D形式对用户头部进行建模来实现的。例如，该模型实时地替代用户的头部的顶部，因此其他参与用户仅观看到参与用户的身体、头部、面部和眼睛，减去头戴式视图器690。In summary, and in accordance with one or more of the illustrated embodiments, the present invention enables virtual projection of a user's webcam video stream, minus the user's headset, in a video chat. This is achieved by modeling the user's head in 3D using modeling techniques such as photogrammetry. For example, the model replaces the top of the user's head in real time, so other participating users only see the participating user's body, head, face, and eyes, minus the headset 690.
应当理解，将包括多个用户(例如，大于两个用户)，其中使用一个或多个固定的立体3D照相机(例如，StereoLabs ZED照相机、Microsoft Kinect、Intel RealSense系列3D照相机等)对每个用户进行建模。附加的3D照相机技术可以计算用户的位置，提供VR合作互动。例如，两个用户可以在火星上玩捉迷藏游戏，或者两个人可以一起在大峡谷(GrandCanyon)中进行虚拟漫步。It should be understood that multiple users (e.g., greater than two users) will be included, where each user is imaged using one or more fixed stereoscopic 3D cameras (e.g., StereoLabs ZED camera, Microsoft Kinect, Intel RealSense family of 3D cameras, etc.) modeling. Additional 3D camera technology can calculate the user's position, providing VR cooperative interaction. For example, two users can play a game of hide-and-seek on Mars, or two people can take a virtual walk through the Grand Canyon together.
应当理解，本发明计算系统600和头戴式视图器690可以被配置和操作用于在增强现实(AR)设置中使用。应将AR理解为一种如下的技术：将计算机生成的增强的层置于现有现实之上，以便通过与之交互的能力使其变得更有意义。所示实施例包括AR被开发优选地作为小应用(app)使用头戴式视图器690，以用数字部件和真实世界彼此增强的这样的方式将数字部件混合到现实世界中。例如，根据本文的教导，每个都在不同位置的穿戴本发明头戴式视图器690的两个或更多用户可以使用/操作头戴式视图器690，以便在所选择的设置(例如，跳伞)和/或位置(例如，巴黎)中出现在彼此旁边，由此，他们不仅实时地将相同的设置(例如，法国巴黎的街道)一起可视化，而且他们看到彼此，就好像头戴式视图器690被移除一样。It should be understood that the inventive computing system 600 and head-mounted viewer 690 may be configured and operative for use in an augmented reality (AR) setting. AR should be understood as a technology that puts a computer-generated enhanced layer on top of existing reality so that it becomes more meaningful through the ability to interact with it. The illustrated embodiment includes AR being developed preferably as an applet (app) using the headset 690 to mix digital parts into the real world in such a way that the digital parts and the real world augment each other. For example, in accordance with the teachings herein, two or more users wearing a headset 690 of the present invention, each in a different location, can use/operate the headset 690 to operate in a selected setting (e.g., skydive) and/or location (e.g., Paris), whereby they not only visualize the same setting (e.g., the streets of Paris, France) together in real time, but they see each other as if a headset Viewer 690 is removed.
还应当理解，计算系统600和/或头戴式视图器690的附加实施例可以被配置为优选地使用至少两个分离的视频馈送和/或单个视频馈送优选地结合预定的静止帧图像高速缓存(still-frame Image Cache)提供实时图像缝合。优选地，经由安装在VR显示设备内部的照相机来执行眼追踪，使得经高速缓存的图像和/或“混合的”图像的实时选择能够从将在图像缝合步骤中使用的初始“建立”阶段(图像高速缓存)导出，因为用户的面部的“眼睛区域”通常因为VR显示设备而被遮盖，因此使用最终(合成/缝合的)视频提供实时3D面部跟踪和重构。应当理解，通过采用实时数字创建的面部(代表用户)，面部/结果优选地被合成到用户的实时视频和/或深度提取上。应当理解，优选地使用绿屏技术和/或深度提取技术从其背景提取用户身体。应当进一步理解的是，完全重构的用户可以被合成为多种应用，诸如(但不限于)：360度视频、VR聊天室和其他适用的应用。应当进一步理解，当前正在使用和/或探索的以下各种各样的技术(例如，SDK、API等)包括(但不限于)：EmguCV、OpenCV和MATLAB；具有SDK的ZED照相机；ARKit；主要编程语言：C#和C++；Unity(游戏引擎)；和Oculus和/或Vive(VR显示设备)。It should also be understood that additional embodiments of the computing system 600 and/or the head-mounted viewer 690 may be configured to use preferably at least two separate video feeds and/or a single video feed, preferably in combination with a predetermined still frame image cache (still-frame Image Cache) provides real-time image stitching. Eye tracking is preferably performed via a camera mounted inside the VR display device, enabling real-time selection of cached and/or "blended" images from the initial "build" stage to be used in the image stitching step ( image cache) to provide real-time 3D face tracking and reconstruction using the final (synthesized/stitched) video since the "eye area" of the user's face is usually covered by the VR display device. It should be understood that the face/result is preferably composited onto the user's live video and/or depth extraction by employing a real-time digitally created face (representing the user). It should be understood that the user's body is preferably extracted from its background using green screen techniques and/or depth extraction techniques. It should be further understood that the fully reconstructed user can be composited into a variety of applications such as (but not limited to): 360 degree video, VR chat rooms, and other applicable applications. It should be further understood that the following various technologies (e.g., SDKs, APIs, etc.) are currently being used and/or explored including (but not limited to): EmguCV, OpenCV, and MATLAB; ZED camera with SDK; ARKit; primary programming Languages: C# and C++; Unity (game engine); and Oculus and/or Vive (VR display device).
对于上面描述的某些所示实施例，应当理解，本文描述的各种非限制性实施例可以单独地、组合地或选择性地组合用于特定应用。此外，可以使用上述非限制性实施例的各种特征中的一些，而不相应地使用其他所描述的特征。因此，本文中提供的描述应被认为仅是本发明的原理、教导和示例性实施例的说明，并且不是对其的限制。With respect to certain illustrated embodiments described above, it should be understood that the various non-limiting embodiments described herein may be used alone, in combination, or selectively in combination for a particular application. Furthermore, some of the various features of the above non-limiting embodiments may be used without the corresponding use of other described features. Accordingly, the description provided herein should be considered as illustrative of the principles, teachings and exemplary embodiments of this invention only, and not in limitation thereof.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US201862642488P | 2018-03-13 | 2018-03-13 | |
| US62/642488 | 2018-03-13 | ||
| CN201980018658.6ACN111936910A (en) | 2018-03-13 | 2019-03-13 | Virtual reality system and method |
| PCT/US2019/022121WO2019178276A1 (en) | 2018-03-13 | 2019-03-13 | Virtual reality system and method |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201980018658.6ADivisionCN111936910A (en) | 2018-03-13 | 2019-03-13 | Virtual reality system and method |
| Publication Number | Publication Date |
|---|---|
| CN116700489Atrue CN116700489A (en) | 2023-09-05 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201980018658.6APendingCN111936910A (en) | 2018-03-13 | 2019-03-13 | Virtual reality system and method |
| CN202310667894.9APendingCN116700489A (en) | 2018-03-13 | 2019-03-13 | Virtual reality system and method |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201980018658.6APendingCN111936910A (en) | 2018-03-13 | 2019-03-13 | Virtual reality system and method |
| Country | Link |
|---|---|
| EP (1) | EP3729177A4 (en) |
| JP (3) | JP7250809B2 (en) |
| CN (2) | CN111936910A (en) |
| CA (1) | CA3089885A1 (en) |
| WO (1) | WO2019178276A1 (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019178276A1 (en)* | 2018-03-13 | 2019-09-19 | Ronald Winston | Virtual reality system and method |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3464754B2 (en)* | 1997-09-19 | 2003-11-10 | 日本電信電話株式会社 | Method and apparatus for synthesizing a face image of a person wearing a head-mounted display |
| US6774869B2 (en)* | 2000-12-22 | 2004-08-10 | Board Of Trustees Operating Michigan State University | Teleportal face-to-face system |
| US20080165195A1 (en)* | 2007-01-06 | 2008-07-10 | Outland Research, Llc | Method, apparatus, and software for animated self-portraits |
| JP4794678B1 (en)* | 2010-05-24 | 2011-10-19 | 株式会社ソニー・コンピュータエンタテインメント | Video processing apparatus, video processing method, and video communication system |
| US9265415B1 (en)* | 2012-01-06 | 2016-02-23 | Google Inc. | Input detection |
| US9063330B2 (en)* | 2013-05-30 | 2015-06-23 | Oculus Vr, Llc | Perception based predictive tracking for head mounted displays |
| WO2015185537A1 (en)* | 2014-06-03 | 2015-12-10 | Thomson Licensing | Method and device for reconstruction the face of a user wearing a head mounted display |
| CA2956508C (en)* | 2014-08-04 | 2020-04-21 | Facebook, Inc. | Method and system for reconstructing obstructed face portions for virtual reality environment |
| GB2534580B (en) | 2015-01-28 | 2020-06-17 | Sony Interactive Entertainment Europe Ltd | Image processing |
| US9910275B2 (en) | 2015-05-18 | 2018-03-06 | Samsung Electronics Co., Ltd. | Image processing for head mounted display devices |
| EP3185176A1 (en)* | 2015-12-21 | 2017-06-28 | THOMSON Licensing | Method and device for synthesizing an image of a face partially occluded |
| US11163358B2 (en)* | 2016-03-17 | 2021-11-02 | Sony Interactive Entertainment Inc. | Spectating virtual (VR) environments associated with VR user interactivity |
| CN105867617B (en)* | 2016-03-25 | 2018-12-25 | 京东方科技集团股份有限公司 | Augmented reality equipment, system, image processing method and device |
| US20170287215A1 (en)* | 2016-03-29 | 2017-10-05 | Google Inc. | Pass-through camera user interface elements for virtual reality |
| WO2017173457A1 (en)* | 2016-04-01 | 2017-10-05 | Incontext Solutions, Inc. | Virtual reality platform for retail environment simulation |
| CN107305621A (en)* | 2016-04-17 | 2017-10-31 | 张翔宇 | The image capture device and image synthesis system of a kind of virtual reality glasses |
| US10722800B2 (en)* | 2016-05-16 | 2020-07-28 | Google Llc | Co-presence handling in virtual reality |
| GB2551323B (en)* | 2016-06-08 | 2021-02-10 | Companion Ltd | System providing a shared environment |
| CN107491165A (en)* | 2016-06-12 | 2017-12-19 | 张翔宇 | A kind of vr eyewear faces 3d images, plane picture capture and gesture capture systems |
| CN107680069B (en)* | 2017-08-30 | 2020-09-11 | 歌尔股份有限公司 | Image processing method and device and terminal equipment |
| WO2019178276A1 (en)* | 2018-03-13 | 2019-09-19 | Ronald Winston | Virtual reality system and method |
| Publication number | Publication date |
|---|---|
| WO2019178276A1 (en) | 2019-09-19 |
| JP2021517300A (en) | 2021-07-15 |
| CN111936910A (en) | 2020-11-13 |
| EP3729177A1 (en) | 2020-10-28 |
| CA3089885A1 (en) | 2019-09-19 |
| JP7250809B2 (en) | 2023-04-03 |
| JP2023078342A (en) | 2023-06-06 |
| JP2025063244A (en) | 2025-04-15 |
| EP3729177A4 (en) | 2021-10-06 |
| Publication | Publication Date | Title |
|---|---|---|
| JP7504968B2 (en) | Avatar display device, avatar generation device and program | |
| US11563779B2 (en) | Multiuser asymmetric immersive teleconferencing | |
| US10684467B2 (en) | Image processing for head mounted display devices | |
| US12067686B2 (en) | System and method for rendering virtual reality interactions | |
| US10602121B2 (en) | Method, system and apparatus for capture-based immersive telepresence in virtual environment | |
| US20130101164A1 (en) | Method of real-time cropping of a real entity recorded in a video sequence | |
| JP2019087226A (en) | Information processing device, information processing system, and method of outputting facial expression images | |
| US12020389B2 (en) | Systems and methods for providing real-time composite video from multiple source devices featuring augmented reality elements | |
| JP2025063244A (en) | Virtual Reality Systems | |
| US20230386147A1 (en) | Systems and Methods for Providing Real-Time Composite Video from Multiple Source Devices Featuring Augmented Reality Elements | |
| JP2022081271A (en) | Image generating apparatus, image generating method, and program | |
| JP5894505B2 (en) | Image communication system, image generation apparatus, and program | |
| CN115079826B (en) | Virtual reality implementation method, electronic equipment and storage medium |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |