相关申请Related Applications
本申请要求于2023年1月27日提交的、申请号为63/482,014且名称为“Techniquesfor Using Inward-Facing Eye-Tracking Cameras of a Head-Worn Device to MeasureHeart Rate,and Systems and Methods Using Those Techniques(使用头戴式设备的面向内的眼动追踪摄像头来测量心率的技术、以及使用这些技术的系统和方法)”的美国临时申请的优先权，以及于2023年10月26日提交的、申请号为18/495,388的美国专利申请的优先权；这两个申请的全部内容通过引用结合在本文中。This application claims priority to U.S. Provisional Application No. 63/482,014, filed on January 27, 2023, entitled “Techniques for Using Inward-Facing Eye-Tracking Cameras of a Head-Worn Device to Measure Heart Rate, and Systems and Methods Using Those Techniques,” and U.S. Patent Application No. 18/495,388, filed on October 26, 2023; the entire contents of both applications are incorporated herein by reference.
技术领域Technical Field
本公开总体上涉及用于眼动追踪和生物特征检测及报告的设备和方法，包括但不限于用于使用头部可穿戴设备的眼动追踪摄像头来确定用户的心率的设备和方法。The present disclosure generally relates to apparatus and methods for eye tracking and biometric detection and reporting, including but not limited to apparatus and methods for determining a user's heart rate using an eye tracking camera of a head wearable device.
背景技术Background technique
光学体积描记图(photoplethysmogram，PPG)可以用于检测(例如，组织的微血管床中的)血容量的变化。可以使用对皮肤进行照射并测量光吸收变化的血氧计(例如，光学传感器和光源)来获取PPG。例如，血氧计可以监测皮肤的真皮和皮下组织的血液灌注。然而，准确的PPG需要来自用户的同一区域的连续测量结果，以准确地监测血液灌注。为此，常规的PPG系统可能需要用户在PPG测量期间保持不动和/或专注于一特定点上。A photoplethysmogram (PPG) can be used to detect changes in blood volume (e.g., in the microvascular bed of a tissue). A PPG can be acquired using an oximeter (e.g., an optical sensor and a light source) that illuminates the skin and measures changes in light absorption. For example, an oximeter can monitor blood perfusion in the dermis and subcutaneous tissue of the skin. However, an accurate PPG requires continuous measurements from the same area of the user to accurately monitor blood perfusion. To do this, conventional PPG systems may require the user to remain still and/or focus on a specific point during the PPG measurement.
发明内容Summary of the invention
本公开描述了这样的系统和方法：所述系统和方法用于在用户运动(例如，与头部可穿戴设备的用户界面交互)时使用该头部可穿戴设备(例如，人工现实头戴式视图器(headset))来获取PPG和/或监测生物特征(例如心率)。本文所描述的各头部可穿戴设备中的一些头部可穿戴设备包括一个或多个面向内的摄像头，该一个或多个面向内的摄像头被配置为追踪用户的眼睛。如本文所描述的，该一个或多个面向内的摄像头还可以用于监测用户的眼睛和/或面部组织中的血容量变化。The present disclosure describes systems and methods for acquiring PPG and/or monitoring biometrics (e.g., heart rate) using a head wearable device (e.g., an artificial reality headset) while the user is in motion (e.g., interacting with a user interface of the head wearable device). Some of the head wearable devices described herein include one or more inward-facing cameras configured to track the user's eyes. As described herein, the one or more inward-facing cameras can also be used to monitor blood volume changes in the user's eyes and/or facial tissue.
作为一示例，佩戴着虚拟现实头戴式视图器的用户可能希望在参与虚拟现实活动(例如玩视频游戏或进行锻炼或健身计划)时监测其心率。在该示例中，虚拟现实头戴式视图器包括一个或多个面向内的摄像头，该一个或多个面向内的摄像头例如被配置为追踪佩戴者的眼睛。如本文所描述的，虚拟现实头戴式视图器的面向内的摄像头可以用于在用户参与虚拟现实活动时监测该用户的眼睛或面部组织中的血容量变化。可以使用血容量变化来获取用户的PPG，并且可以使用PPG来确定用户的心率。以此方式，用户可以在参与虚拟现实活动时(例如，在不需要保持静止和/或专注于一固定点的情况下)监测其心率。将一个或多个面向内的摄像头作为头部可穿戴设备的部件允许一个或多个摄像头在用户正移动并正与用户界面和虚拟现实对象交互时、保持相对于用户面部的固定位置。以此方式，可以获得用户面部的区域的连续测量结果并对这些测量结果进行比较以生成PPG。As an example, a user wearing a virtual reality headset may wish to monitor their heart rate while participating in a virtual reality activity (e.g., playing a video game or performing an exercise or fitness program). In this example, the virtual reality headset includes one or more inward-facing cameras, which are configured, for example, to track the wearer's eyes. As described herein, the inward-facing cameras of the virtual reality headset can be used to monitor changes in blood volume in the user's eyes or facial tissue while the user is participating in a virtual reality activity. The blood volume changes can be used to obtain a PPG for the user, and the PPG can be used to determine the user's heart rate. In this way, the user can monitor his or her heart rate while participating in a virtual reality activity (e.g., without having to remain still and/or focus on a fixed point). Having one or more inward-facing cameras as a component of a head wearable device allows the one or more cameras to maintain a fixed position relative to the user's face while the user is moving and interacting with a user interface and virtual reality objects. In this way, continuous measurements of areas of the user's face can be obtained and compared to generate a PPG.
根据一些实施例，提供了一种监测用户的心率的方法。该方法包括：经由人工现实头戴式视图器的面向内的摄像头采集用户的面部的区域的多幅图像，该区域包括该用户的眼睛。该多幅图像中的第一图像包括瞳孔在眼睛中的第一位置；并且该多幅图像中的第二图像包括瞳孔在眼睛中的第二位置，该第二位置与该第一位置不同。该方法还包括基于该多幅图像确定心率信息。According to some embodiments, a method of monitoring a user's heart rate is provided. The method includes: acquiring, via an inward-facing camera of an artificial reality head mounted view device, a plurality of images of an area of a user's face, the area including the user's eyes. A first image in the plurality of images includes a first position of a pupil in the eye; and a second image in the plurality of images includes a second position of the pupil in the eye, the second position being different from the first position. The method also includes determining heart rate information based on the plurality of images.
在一些实施例中，计算设备(例如，腕部可穿戴设备或头戴式设备、或诸如智能手机或台式计算机或膝上型计算机等中间设备)包括一个或多个处理器、存储器、显示器(在一些实施例中，例如对于以下的某些示例中间设备，该显示器可以是可选的：所述中间设备可以对腕部可穿戴设备和头戴式设备处的操作进行协调，并因此具有充足的处理资源和功率资源而不需要具有其自己的显示器)、以及存储在该存储器中的一个或多个程序。该一个或多个程序被配置用于由该一个或多个处理器执行。该一个或多个程序包括用于执行(或使得执行)本文所描述的任何方法的指令。In some embodiments, a computing device (e.g., a wrist wearable device or a head mounted device, or an intermediate device such as a smartphone or a desktop or laptop computer) includes one or more processors, memory, a display (in some embodiments, the display may be optional, such as for some of the following example intermediate devices: the intermediate device may coordinate operations at the wrist wearable device and the head mounted device and thus have sufficient processing resources and power resources without having its own display), and one or more programs stored in the memory. The one or more programs are configured for execution by the one or more processors. The one or more programs include instructions for performing (or causing the performance of) any of the methods described herein.
在一些实施例中，非暂态计算机可读存储介质存储有一个或多个程序，该一个或多个程序被配置用于由具有一个或多个处理器和存储器的计算设备(例如，可穿戴设备、或者诸如智能手机或台式计算机或膝上型计算机等中间设备，该中间设备可以被配置为对在腕部可穿戴设备和头戴式设备处的操作进行协调)执行。该一个或多个程序包括用于执行(或使得执行)本文所描述的任何方法的指令。In some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by a computing device (e.g., a wearable device, or an intermediate device such as a smartphone or a desktop or laptop computer, which can be configured to coordinate operations at a wrist wearable device and a head-mounted device) having one or more processors and a memory. The one or more programs include instructions for performing (or causing the performance of) any of the methods described herein.
因此，公开了用于眼动追踪和生物特征监测及报告的方法、系统和计算机可读存储介质。这种方法和系统可以补充或替代用于眼动追踪和生物特征监测及报告的常规方法。Thus, methods, systems, and computer-readable storage media for eye tracking and biometric monitoring and reporting are disclosed. Such methods and systems may supplement or replace conventional methods for eye tracking and biometric monitoring and reporting.
说明书中所描述的特征和优点不一定是全部包含的，特别地，鉴于本公开中所提供的附图、说明书和权利要求书，一些附加特征和优点对于本领域普通技术人员而言将是显而易见的。此外，应注意的是，本说明书中所使用的用语主要是出于可读性和指导目的而选择的，并不一定被选择来描绘或限制本文所描述的主题。The features and advantages described in the specification are not necessarily all inclusive, and in particular, some additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, descriptions, and claims provided in this disclosure. In addition, it should be noted that the terms used in this specification are primarily selected for readability and instructional purposes and are not necessarily selected to describe or limit the subject matter described herein.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
为了更详细地理解本公开，可以通过参考各个实施例的特征来获得更具体的描述，在附图中示出了各个实施例中的一些实施例。然而，附图仅示出了本公开的相关特征，并因此并不意味着必须被认为是限制性的，因为如本领域技术人员在阅读本公开时将理解的那样，本说明书可以承认其它有效的特征。In order to understand the present disclosure in more detail, a more specific description may be obtained by reference to the features of various embodiments, some of which are shown in the accompanying drawings. However, the accompanying drawings only illustrate the relevant features of the present disclosure and are therefore not meant to be considered limiting, as the specification may admit other effective features as will be understood by those skilled in the art upon reading the present disclosure.
图1A至图1C示出了根据一些实施例的经由头部可穿戴设备监测生物特征数据的示例。1A-1C illustrate examples of monitoring biometric data via a head wearable device according to some embodiments.
图2A至图2C示出了根据一些实施例的经由头部可穿戴设备监测生物特征数据的示例用户场景。2A-2C illustrate example user scenarios for monitoring biometric data via a head wearable device according to some embodiments.
图3A至图3D示出了根据一些实施例的经由头部可穿戴设备监测生物特征数据的另一示例用户场景。3A-3D illustrate another example user scenario for monitoring biometric data via a head wearable device according to some embodiments.
图3E至图3G示出了根据一些实施例的经由头部可穿戴设备监测生物特征数据的另一示例用户场景。3E-3G illustrate another example user scenario for monitoring biometric data via a head wearable device in accordance with some embodiments.
图4A和图4B是示出了根据一些实施例的用于监测心率的示例方法的流程图。4A and 4B are flow charts illustrating example methods for monitoring heart rate according to some embodiments.
图4C是示出了根据一些实施例的用于监测心率的另一示例方法的流程图。4C is a flow chart illustrating another example method for monitoring heart rate in accordance with some embodiments.
图5A、图5B、图5C-1、图5C-2、图5D-1和图5D-2示出了根据一些实施例的示例人工现实系统。5A, 5B, 5C-1, 5C-2, 5D-1, and 5D-2 illustrate example artificial reality systems according to some embodiments.
图6A和图6B示出了根据一些实施例的示例腕部可穿戴设备。6A and 6B illustrate example wrist wearable devices in accordance with some embodiments.
图7A、图7B-1、图7B-2和图7C示出了根据一些实施例的示例头部可穿戴设备。7A, 7B-1, 7B-2, and 7C illustrate example head wearable devices according to some embodiments.
图8A和图8B示出了根据一些实施例的示例手持中间处理设备。8A and 8B illustrate example handheld intermediate processing devices in accordance with some embodiments.
图9A至图9C示出了根据一些实施例的示例基于智能纺织品的服饰。9A-9C illustrate example smart textile-based apparel, according to some embodiments.
根据惯例，附图中所示出的各个特征不一定是按比例绘制的，并且在整个说明书和附图中可以使用相似的附图标记来表示相似的特征。According to common practice, the various features shown in the drawings are not necessarily drawn to scale, and like reference numerals may be used to denote like features throughout the specification and drawings.
具体实施方式Detailed ways
作为一示例，佩戴着虚拟现实头戴式视图器的用户可以使用该虚拟现实头戴式视图器的面向内的摄像头来监测其心率和/或其它生物特征数据。虚拟现实头戴式视图器可以用于监测用户眼睛和/或面部组织中的血容量水平，以获取该用户的PPG信息。可以从PPG信息中获取心率和其它生物特征数据并向用户提供该心率和其它生物特征数据。PPG测量结果还可以用于确定氧饱和度(脉搏血氧饱和度)、呼吸频率、血压和心输出量。PPG测量结果也可以用于评估自主神经功能和对周围血管疾病的检测。例如，由虚拟现实头戴式视图器呈现的用户界面可以包括对心率和/或其它生物特征数据的呈现。作为另一示例，可以从另一可穿戴设备(例如腕部可穿戴设备)中获取辅助生物特征数据，并将该辅助生物特征数据与来自虚拟现实头戴式视图器的生物特征数据进行结合。As an example, a user wearing a virtual reality head-mounted view device can use the inward-facing camera of the virtual reality head-mounted view device to monitor his heart rate and/or other biometric data. The virtual reality head-mounted view device can be used to monitor the blood volume level in the user's eyes and/or facial tissue to obtain PPG information for the user. Heart rate and other biometric data can be obtained from the PPG information and provided to the user. PPG measurements can also be used to determine oxygen saturation (pulse oximetry), respiratory rate, blood pressure, and cardiac output. PPG measurements can also be used to assess autonomic function and detect peripheral vascular disease. For example, a user interface presented by a virtual reality head-mounted view device can include a presentation of heart rate and/or other biometric data. As another example, auxiliary biometric data can be obtained from another wearable device (e.g., a wrist wearable device) and combined with the biometric data from the virtual reality head-mounted view device.
现在转到附图。图1A至图1C示出了根据一些实施例的经由头部可穿戴设备监测生物特征数据的示例。图1A示出了根据一些实施例的在第一时间t1佩戴着头部可穿戴设备102的用户101。在一些实施例中，头部可穿戴设备102是人工现实头戴式视图器(例如，虚拟现实头戴式视图器)。头部可穿戴设备102包括面向内的摄像头104(例如，被配置为感测红外波长和/或近红外波长的摄像头)，该面向内的摄像头被定位为采集用户101的眼睛110的图像、和用户101的面部组织112的图像。根据一些实施例，头部可穿戴设备102包括被定位为照射用户101的眼睛110的照明源106、和被定位为照射用户101的面部组织112的照明源108。在一些实施例中，照明源106和/或照明源108被配置为发射红外光和/或近红外光。图1A还显示了：基于眼睛的PPG信息114，该基于眼睛的PPG信息是基于眼动和/或用户的眼睛110中的血液流动来测量的；和基于脸红的PPG信息116，该基于脸红的PPG信息是基于用户的面部组织112中的血容量水平来测量的。图1A还显示了心率信息118，该心率信息是基于以下项来确定的：基于眼睛的PPG信息114和/或基于脸红的PPG信息116。Turning now to the drawings. Figures 1A-1C illustrate examples of monitoring biometric data via a head wearable device in accordance with some embodiments. Figure 1A illustrates a user 101 wearing a head wearable device 102 at a first time t1 in accordance with some embodiments. In some embodiments, the head wearable device 102 is an artificial reality head mounted viewer (e.g., a virtual reality head mounted viewer). The head wearable device 102 includes an inward facing camera 104 (e.g., a camera configured to sense infrared wavelengths and/or near infrared wavelengths) positioned to capture images of eyes 110 of the user 101, and images of facial tissue 112 of the user 101. In accordance with some embodiments, the head wearable device 102 includes an illumination source 106 positioned to illuminate the eyes 110 of the user 101, and an illumination source 108 positioned to illuminate the facial tissue 112 of the user 101. In some embodiments, the illumination source 106 and/or the illumination source 108 are configured to emit infrared light and/or near infrared light. 1A also shows eye-based PPG information 114, which is measured based on eye movement and/or blood flow in the user's eye 110, and blush-based PPG information 116, which is measured based on blood volume level in the user's facial tissue 112. FIG1A also shows heart rate information 118, which is determined based on the eye-based PPG information 114 and/or the blush-based PPG information 116.
图1B示出了根据一些实施例的在第一时间之后的第二时间t2佩戴着头部可穿戴设备102的用户101。图1B还显示了基于脸红的PPG信息116，与基于眼睛的PPG信息114相比，该基于脸红的PPG信息116在t1与t2之间的时间期间具有增大的噪声。在一些实施例中，t1与t2之间的心率信息118是基于基于眼睛的PPG信息114的，因为与该时间段的基于脸红的PPG信息116相比，该基于眼睛的PPG信息具有更少的噪声。在一些情况下，附加的噪声是由于用户相对于摄像头的运动和/或摄像头与用户之间的照明水平的变化。在一些情况下，附加的噪声是由于摄像头聚焦在用户的眼睛110上。在一些实施例中，根据确定基于脸红的PPG信息的噪声更多，摄像头104聚焦在眼睛110上，如聚焦区域120所指示的。FIG. 1B illustrates a user 101 wearing a head wearable device 102 at a second time t2 after the first time, in accordance with some embodiments. FIG. 1B also shows blush-based PPG information 116 having increased noise during the time between t1 and t2 compared to the eye-based PPG information 114. In some embodiments, the heart rate information 118 between t1 and t2 is based on the eye-based PPG information 114 because the eye-based PPG information has less noise than the blush-based PPG information 116 during this time period. In some cases, the additional noise is due to the user's motion relative to the camera and/or a change in lighting level between the camera and the user. In some cases, the additional noise is due to the camera focusing on the user's eyes 110. In some embodiments, based on determining that the blush-based PPG information is more noisy, the camera 104 focuses on the eyes 110, as indicated by the focus area 120.
图1C示出了根据一些实施例的在第二时间之后的第三时间t3佩戴着头部可穿戴设备102的用户101。图1C还显示了在t2与t3之间的时间期间基于眼睛的PPG信息114没有数据。在一些实施例中，在t2与t3之间的心率信息118是基于基于脸红的PPG信息116的，因为基于眼睛的PPG信息114在该时间段不可用。在一些情况下，基于眼睛的PPG信息114因为眼睛110的摄像头视野(例如，由于用户闭上了其眼睛)被遮挡而不可用。在一些情况下，基于眼睛的PPG信息114由于眼睛相对于摄像头的运动和/或摄像头与用户的眼睛110之间的照明水平的变化而不可用。在一些实施例中，根据确定基于眼睛的PPG信息不可用，摄像头104聚焦在面部组织112上，如聚焦区域122所指示的。FIG. 1C illustrates a user 101 wearing a head wearable device 102 at a third time t3 after the second time, according to some embodiments. FIG. 1C also shows that there is no data for the eye-based PPG information 114 during the time between t2 and t3. In some embodiments, the heart rate information 118 between t2 and t3 is based on the blushing-based PPG information 116 because the eye-based PPG information 114 is not available during this time period. In some cases, the eye-based PPG information 114 is not available because the camera field of view of the eye 110 is blocked (e.g., because the user has closed his eyes). In some cases, the eye-based PPG information 114 is not available due to movement of the eye relative to the camera and/or a change in the lighting level between the camera and the user's eye 110. In some embodiments, based on determining that the eye-based PPG information is not available, the camera 104 focuses on the facial tissue 112, as indicated by the focus area 122.
图2A至图2C示出了根据一些实施例的经由头部可穿戴设备102监测生物特征数据的示例用户场景。图2A显示了佩戴着头部可穿戴设备102和腕部可穿戴设备202的用户101。图2A还显示了经由头部可穿戴设备102向用户101呈现的用户界面204(例如，健身应用用户界面)。用户界面204包括锻炼计数器206、消息元素212和具有用户101的心率信息210的心率图208。在一些实施例中，心率信息210是基于来自头部可穿戴设备102的PPG测量结果(例如，基于眼睛的PPG信息114和/或基于脸红的PPG信息116)和/或来自腕部可穿戴设备202的PPG测量结果来确定的。2A-2C illustrate example user scenarios for monitoring biometric data via a head wearable device 102 according to some embodiments. FIG. 2A shows a user 101 wearing the head wearable device 102 and the wrist wearable device 202. FIG. 2A also shows a user interface 204 (e.g., a fitness application user interface) presented to the user 101 via the head wearable device 102. The user interface 204 includes a workout counter 206, a message element 212, and a heart rate graph 208 with heart rate information 210 of the user 101. In some embodiments, the heart rate information 210 is determined based on PPG measurements from the head wearable device 102 (e.g., eye-based PPG information 114 and/or blush-based PPG information 116) and/or PPG measurements from the wrist wearable device 202.
图2B显示了正进行锻炼(抬膝)的用户101以及相应地更新的用户界面204。图2B中的用户界面204显示了锻炼计数器206，该锻炼计数器206更新以指示用户已经进行了第一组抬膝的30个抬膝中的8个抬膝。图2B中的用户界面204还显示了利用附加心率信息214更新的心率图208。在一些实施例中，附加心率信息214是基于来自头部可穿戴设备102的PPG测量结果(例如，基于眼睛的PPG信息114和/或基于脸红的PPG信息116)和/或来自腕部可穿戴设备202的PPG测量结果来确定的。FIG2B shows the user 101 performing an exercise (knee raises) and the user interface 204 updated accordingly. The user interface 204 in FIG2B shows an exercise counter 206 that updates to indicate that the user has performed 8 of the 30 knee raises in the first set of knee raises. The user interface 204 in FIG2B also shows a heart rate graph 208 updated with additional heart rate information 214. In some embodiments, the additional heart rate information 214 is determined based on PPG measurements from the head wearable device 102 (e.g., eye-based PPG information 114 and/or blush-based PPG information 116) and/or PPG measurements from the wrist wearable device 202.
图2C显示了已经完成锻炼的用户101以及相应地更新的用户界面204。图2C中的用户界面204在消息元素212中显示了消息(“训练完成！”)。图2C中的锻炼计数器206显示了用户101已经完成该一组，并且心率图208显示了用户在锻炼过程期间的心率。图2C中的用户界面204还显示了指示用户(例如，在该一组期间)的平均心率和最大心率的心率信息230。在一些实施例中，心率信息的一部分是基于来自头部可穿戴设备102的PPG测量结果的，并且心率信息的一部分是基于来自腕部可穿戴设备202的PPG测量结果的。例如，心率信息224可以是基于来自头部可穿戴设备102的PPG测量结果(例如，PPG 220)的，心率信息226可以是基于来自腕部可穿戴设备202的PPG测量结果(例如，PPG 222)的，并且心率信息228可以是基于来自头部可穿戴设备102的PPG测量结果的。在一些实施例中，心率信息是基于被确定为最稳定(例如，具有最少的噪声量)的PPG数据的。FIG2C shows user 101 having completed a workout and user interface 204 updated accordingly. User interface 204 in FIG2C displays a message (“Workout Complete!”) in message element 212. Workout counter 206 in FIG2C displays that user 101 has completed a set, and heart rate graph 208 displays the user's heart rate during the workout. User interface 204 in FIG2C also displays heart rate information 230 indicating the user's average heart rate and maximum heart rate (e.g., during the set). In some embodiments, a portion of the heart rate information is based on PPG measurements from head wearable device 102, and a portion of the heart rate information is based on PPG measurements from wrist wearable device 202. For example, heart rate information 224 can be based on PPG measurements from head wearable device 102 (e.g., PPG 220), heart rate information 226 can be based on PPG measurements from wrist wearable device 202 (e.g., PPG 222), and heart rate information 228 can be based on PPG measurements from head wearable device 102. In some embodiments, the heart rate information is based on PPG data determined to be the most stable (e.g., having the least amount of noise).
图3A至图3D示出了根据一些实施例的经由头部可穿戴设备102监测生物特征数据的另一示例用户场景。图3A显示了佩戴着头部可穿戴设备102和腕部可穿戴设备202的用户101。图3A还显示了经由头部可穿戴设备102向用户101呈现的用户界面302(例如，拳击游戏用户界面)。用户界面302包括关于拳击游戏的信息和开始玩游戏的虚拟按钮304。3A to 3D illustrate another example user scenario for monitoring biometric data via a head wearable device 102 according to some embodiments. FIG3A shows a user 101 wearing a head wearable device 102 and a wrist wearable device 202. FIG3A also shows a user interface 302 (e.g., a boxing game user interface) presented to the user 101 via the head wearable device 102. The user interface 302 includes information about a boxing game and a virtual button 304 for starting to play the game.
图3B显示了如用户界面312所指示的、正在第一时间(t1)玩拳击游戏的用户101。用户界面302包括拳击部分313和信息部分314，该拳击部分313显示了拳击游戏玩法，该信息部分314包括关于回合剩余时间的信息和心率信息315。在一些实施例中，心率信息315是从经由头部可穿戴设备102获取的PPG测量结果中获取的。在一些实施例中，PPG测量结果是经由摄像头104根据用户的眼睛110中的血液流动来获取的。3B shows the user 101 playing a boxing game at a first time (t1) as indicated by the user interface 312. The user interface 302 includes a boxing portion 313 that displays the boxing game play and an information portion 314 that includes information about the remaining time of the round and heart rate information 315. In some embodiments, the heart rate information 315 is obtained from PPG measurements obtained via the head wearable device 102. In some embodiments, the PPG measurements are obtained via the camera 104 based on blood flow in the user's eyes 110.
图3C显示了如用户界面312所指示的、正在第一时间之后的第二时间(t2)玩拳击游戏的用户101。将图3C中的信息部分314更新为包括心率信息334以及心率信息315。在一些实施例中，心率信息334是从经由头部可穿戴设备102获取的PPG测量结果中获取的。在一些实施例中，PPG测量结果是经由摄像头104根据用户的面部组织112中的血容量水平(例如，脸红水平)变化来获取的。在一些实施例中，面部组织112被照明源108照射(例如，以协助摄像头104检测血容量水平)。如图3B和图3C所示，可以根据眼睛和/或面部组织的血容量水平来获取PPG测量结果。例如，头部可穿戴设备102可以从眼睛110和面部组织112这二者中获取测量结果，并选择具有最小噪声/变化的测量结果(例如，选择给定时间段(例如1秒、10秒、30秒或1分钟)的PPG测量结果源)以用于确定用户的心率信息。3C shows the user 101 playing a boxing game at a second time (t2) after the first time, as indicated by the user interface 312. The information portion 314 in FIG. 3C is updated to include heart rate information 334 and heart rate information 315. In some embodiments, the heart rate information 334 is obtained from a PPG measurement obtained via the head wearable device 102. In some embodiments, the PPG measurement is obtained via the camera 104 based on changes in blood volume levels (e.g., blushing levels) in the user's facial tissue 112. In some embodiments, the facial tissue 112 is illuminated by the illumination source 108 (e.g., to assist the camera 104 in detecting blood volume levels). As shown in FIGS. 3B and 3C, the PPG measurement can be obtained based on the blood volume levels of the eyes and/or facial tissue. For example, the head wearable device 102 can obtain measurements from both the eyes 110 and the facial tissue 112, and select the measurement with the least noise/variation (e.g., selecting a PPG measurement source for a given time period (e.g., 1 second, 10 seconds, 30 seconds, or 1 minute)) for determining the user's heart rate information.
图3D显示了如用户界面340所指示的、已经完成了拳击游戏的用户101。根据一些实施例，用户界面340包括心率图342和心率信息346，该心率图342指示用户在拳击比赛期间的心率，该心率信息346指示在拳击比赛期间的最大心率和平均心率。3D shows user 101 having completed the boxing game as indicated by user interface 340. According to some embodiments, user interface 340 includes a heart rate graph 342 indicating the user's heart rate during the boxing match and heart rate information 346 indicating the maximum heart rate and the average heart rate during the boxing match.
图3E至图3G示出了根据一些实施例的经由头部可穿戴设备102监测生物特征数据的另一示例用户场景。图3E显示了佩戴着头部可穿戴设备102和腕部可穿戴设备202的用户101。图3E还显示了经由头部可穿戴设备102向用户101呈现的用户界面350(例如，恐怖游戏用户界面)。用户界面350包括关于恐怖游戏的信息和开始玩游戏的虚拟按钮352。3E-3G illustrate another example user scenario for monitoring biometric data via a head wearable device 102 according to some embodiments. FIG3E shows a user 101 wearing a head wearable device 102 and a wrist wearable device 202. FIG3E also shows a user interface 350 (e.g., a horror game user interface) presented to the user 101 via the head wearable device 102. The user interface 350 includes information about the horror game and a virtual button 352 for starting to play the game.
图3F显示了如用户界面354所指示的、正在玩恐怖游戏的用户101。用户界面354包括游戏玩法部分356和信息部分358，该信息部分358包括已经检测到惊恐时刻的通知和心率信息360。心率信息360指示用户在玩恐怖游戏时的心率。在一些实施例中，心率信息360是从经由头部可穿戴设备102和/或腕部可穿戴设备202获取到的PPG测量结果中获取的。在一些实施例中，惊恐时刻是基于用户的心率达到或超过预设阈值来检测的。在一些实施例中，惊恐时刻是基于识别到心率信息360中的局部最大值来检测的。图3G显示了正在观看时刻用户界面370(例如，已经停止玩恐怖游戏)的用户101。用户界面370包括多个惊恐时刻片段372，每个惊恐时刻片段具有相关联的心率测量结果。FIG3F shows a user 101 playing a horror game as indicated by a user interface 354. The user interface 354 includes a gameplay portion 356 and an information portion 358, which includes a notification that a scary moment has been detected and heart rate information 360. The heart rate information 360 indicates the user's heart rate while playing the horror game. In some embodiments, the heart rate information 360 is obtained from PPG measurements obtained via the head wearable device 102 and/or the wrist wearable device 202. In some embodiments, the scary moment is detected based on the user's heart rate reaching or exceeding a preset threshold. In some embodiments, the scary moment is detected based on identifying a local maximum in the heart rate information 360. FIG3G shows a user 101 viewing a moment user interface 370 (e.g., having stopped playing the horror game). The user interface 370 includes a plurality of scary moment segments 372, each having an associated heart rate measurement.
另外，尽管参考图2A至图2C和图3A至图3G而描述的用户场景被描述为单独的序列，但是在一些实施例中，各用户场景彼此组合。例如，参考图2A至图2C描述的序列可以在参考图3A至图3D和图3E至图3G描述的序列之前(或之后)发生(例如，这三个序列都可以在用户101正与头部可穿戴设备102交互时发生)。In addition, although the user scenarios described with reference to Figures 2A to 2C and Figures 3A to 3G are described as separate sequences, in some embodiments, the user scenarios are combined with each other. For example, the sequence described with reference to Figures 2A to 2C may occur before (or after) the sequence described with reference to Figures 3A to 3D and Figures 3E to 3G (for example, all three sequences may occur when the user 101 is interacting with the head wearable device 102).
参考图2A至图2C和图3A至图3G而描述的用户场景涉及特定的用户界面和应用，例如用户界面204、302、312、340、350、354和370。然而，可以结合其它类型的用户界面、菜单和应用使用这些序列、手势、动作和操作，所述用户界面、菜单和应用例如为网页浏览、随手记、社交媒体、文字处理、数据输入和编程等。The user scenarios described with reference to FIGS. 2A-2C and 3A-3G relate to specific user interfaces and applications, such as user interfaces 204, 302, 312, 340, 350, 354, and 370. However, these sequences, gestures, actions, and operations may be used in conjunction with other types of user interfaces, menus, and applications, such as web browsing, note taking, social media, word processing, data entry, and programming.
图4A和图4B是示出了根据一些实施例的用于监测心率的方法400的流程图。方法400在具有一个或多个处理器和存储器的计算系统(例如，可穿戴设备和/或中间设备)处执行。在一些实施例中，该存储器存储有一个或多个程序，该一个或多个程序被配置用于由该一个或多个处理器执行。图4A和图4B中所示出的多个操作中的至少一些操作与存储在计算机存储器或计算机可读存储介质(例如，可穿戴带610的存储器650和/或计算系统720的存储器750A)中的指令相对应。在一些实施例中，计算系统是诸如头部可穿戴设备102等可穿戴设备。在一些实施例中，计算系统是或包括诸如智能手机、个人计算机或视频游戏控制台等中间设备。4A and 4B are flowcharts showing a method 400 for monitoring heart rate according to some embodiments. Method 400 is performed at a computing system (e.g., a wearable device and/or an intermediate device) having one or more processors and a memory. In some embodiments, the memory stores one or more programs configured to be executed by the one or more processors. At least some of the multiple operations shown in FIG. 4A and FIG. 4B correspond to instructions stored in a computer memory or a computer-readable storage medium (e.g., a memory 650 of a wearable band 610 and/or a memory 750A of a computing system 720). In some embodiments, the computing system is a wearable device such as a head wearable device 102. In some embodiments, the computing system is or includes an intermediate device such as a smart phone, a personal computer, or a video game console.
在一些实施例中，系统使用人工现实头戴式视图器(例如，头部可穿戴设备102)的照明源(例如，照明源106)照射(402)用户的眼睛(例如，眼睛110)。在一些实施例中，照明源发射红外光和/或近红外光。In some embodiments, the system illuminates (402) a user's eye (e.g., eye 110) using an illumination source (e.g., illumination source 106) of an artificial reality head-mounted viewer (e.g., head wearable device 102). In some embodiments, the illumination source emits infrared light and/or near infrared light.
该系统经由人工现实头戴式视图器的面向内的摄像头(例如，摄像头104)采集(404)用户的面部的区域的多幅图像，该区域包括用户的眼睛。例如，用户的面部的区域可以包括眼睛110和面部组织112。在一些实施例中，第一摄像头采集用户的眼睛的多幅图像，并且第二摄像头采集用户的面部组织的多幅图像(例如，系统基于检测到的噪声和/或变化性来选择将哪个用于PPG和/或心率)。The system captures (404) via an inward-facing camera of the artificial reality headset (e.g., camera 104) multiple images of an area of the user's face, the area including the user's eyes. For example, the area of the user's face may include eyes 110 and facial tissue 112. In some embodiments, a first camera captures multiple images of the user's eyes and a second camera captures multiple images of the user's facial tissue (e.g., the system selects which to use for PPG and/or heart rate based on detected noise and/or variability).
该多幅图像中的第一图像包括(406)瞳孔在眼睛中的第一位置。该多幅图像中的第二图像包括(408)该瞳孔在该眼睛中的第二位置，该第二位置与第一位置不同。例如，PPG测量结果是基于视网膜、脉络膜、结膜和/或虹膜的血管中的血液流动来获取的。A first image in the plurality of images includes (406) a first position of a pupil in an eye. A second image in the plurality of images includes (408) a second position of the pupil in the eye, the second position being different from the first position. For example, the PPG measurement is obtained based on blood flow in blood vessels of the retina, choroid, conjunctiva, and/or iris.
在一些实施例中，系统向用户呈现(410)用户界面(例如，用户界面204)，其中该多幅图像是在用户正观看该用户界面时采集的。例如，该系统呈现与应用或视频游戏相关的用户界面，并且在用户与该用户界面交互时采集用户的眼睛的图像。In some embodiments, the system presents (410) a user interface (e.g., user interface 204) to a user, wherein the plurality of images are captured while the user is viewing the user interface. For example, the system presents a user interface associated with an application or video game and captures images of the user's eyes while the user interacts with the user interface.
该系统基于该多幅图像确定(412)心率信息(例如，心率信息118)。例如，该系统从用户的一只或两只眼睛和/或面部组织测量PPG，并根据该PPG来推断心率信息。The system determines (412) heart rate information (e.g., heart rate information 118) based on the plurality of images. For example, the system measures PPG from one or both eyes and/or facial tissue of the user and infers heart rate information based on the PPG.
在一些实施例中，该系统从该多幅图像中确定(414)眼动追踪信息，并根据该眼动追踪信息更新用户界面。例如，该系统允许用户基于眼动来与用户界面交互(例如改变用户的视场或选择用户界面元素)。In some embodiments, the system determines (414) eye tracking information from the plurality of images and updates the user interface based on the eye tracking information. For example, the system allows the user to interact with the user interface based on eye movements (e.g., to change the user's field of view or to select a user interface element).
在一些实施例中，该系统经由用户界面向用户呈现(416)心率信息。例如，该系统呈现心率图(例如，心率图342)、最大心率和/或平均心率(例如，心率信息346)。In some embodiments, the system presents (416) heart rate information to the user via the user interface. For example, the system presents a heart rate graph (e.g., heart rate graph 342), a maximum heart rate, and/or an average heart rate (e.g., heart rate information 346).
在一些实施例中，系统从用户所佩戴的腕部可穿戴设备(例如，腕部可穿戴设备202)中(例如，从PPG 222)获取(418)辅助心率信息。例如，腕部可穿戴设备包括聚焦于用户的腕部的图像传感器，并且该腕部可穿戴设备采集用户腕部的血容量水平。以此方式，该系统可以获取用户的PPG测量结果，并基于获取到的PPG测量结果确定用户的心率。在一些实施例中，腕部可穿戴设备包括用于确定用户心率的一个或多个非图像传感器。In some embodiments, the system obtains (418) auxiliary heart rate information from a wrist wearable device (e.g., wrist wearable device 202) worn by the user (e.g., from PPG 222). For example, the wrist wearable device includes an image sensor focused on the user's wrist, and the wrist wearable device collects blood volume levels at the user's wrist. In this way, the system can obtain PPG measurements of the user and determine the user's heart rate based on the obtained PPG measurements. In some embodiments, the wrist wearable device includes one or more non-image sensors for determining the user's heart rate.
在一些实施例中，该系统根据辅助心率信息满足一个或多个标准，选择(420)该辅助心率信息作为选定心率信息；根据辅助心率信息不满足一个或多个标准，选择心率信息作为选定心率信息；以及向用户呈现选定心率信息。例如，如果辅助心率信息噪声太大和/或获取得不一致，则选择主要心率信息。在一些情况下，由于腕部可穿戴设备(例如，因用户的运动)没有被正确定位，因此辅助心率信息有噪声和/或不一致。In some embodiments, the system selects (420) the secondary heart rate information as the selected heart rate information based on the secondary heart rate information satisfying one or more criteria; selects the heart rate information as the selected heart rate information based on the secondary heart rate information not satisfying one or more criteria; and presents the selected heart rate information to the user. For example, if the secondary heart rate information is too noisy and/or obtained inconsistently, the primary heart rate information is selected. In some cases, the secondary heart rate information is noisy and/or inconsistent because the wrist wearable device is not properly positioned (e.g., due to the user's movement).
在一些实施例中，该系统根据心率信息满足一个或多个标准，选择(422)该心率信息作为选定心率信息；根据心率信息不满足一个或多个标准，选择辅助心率信息作为选定心率信息；以及向用户呈现选定心率信息。例如，如果主要心率信息噪声太大和/或获取得不一致，则选择辅助心率信息。In some embodiments, the system selects (422) the heart rate information as the selected heart rate information based on the heart rate information satisfying one or more criteria; selects the auxiliary heart rate information as the selected heart rate information based on the heart rate information not satisfying one or more criteria; and presents the selected heart rate information to the user. For example, if the primary heart rate information is too noisy and/or obtained inconsistently, the auxiliary heart rate information is selected.
在一些实施例中，该系统对心率信息和辅助心率信息进行汇总(424)，以获得汇总后的心率信息；以及向用户呈现该汇总后的心率信息。例如，系统根据心率信息和辅助心率信息获得平均心率信息，并且该系统向用户呈现该平均心率信息。In some embodiments, the system aggregates the heart rate information and the auxiliary heart rate information (424) to obtain aggregated heart rate information; and presents the aggregated heart rate information to the user. For example, the system obtains average heart rate information based on the heart rate information and the auxiliary heart rate information, and the system presents the average heart rate information to the user.
图4C是示出了根据一些实施例的用于监测心率的方法450的流程图。方法450在具有一个或多个处理器和存储器的计算系统(例如，可穿戴设备或中间设备)处执行。在一些实施例中，该存储器存储有一个或多个程序，该一个或多个程序被配置用于由该一个或多个处理器执行。图4C中所示出的多个操作中的至少一些操作与存储在计算机存储器或计算机可读存储介质(例如，可穿戴带610的存储器650和/或计算系统720的存储器750A)中的指令相对应。在一些实施例中，计算系统是诸如头部可穿戴设备102等可穿戴设备。在一些实施例中，计算系统是或包括诸如智能手机、个人计算机或视频游戏控制台等中间设备。FIG4C is a flowchart showing a method 450 for monitoring heart rate according to some embodiments. Method 450 is performed at a computing system (e.g., a wearable device or an intermediate device) having one or more processors and a memory. In some embodiments, the memory stores one or more programs configured to be executed by the one or more processors. At least some of the multiple operations shown in FIG4C correspond to instructions stored in a computer memory or a computer-readable storage medium (e.g., a memory 650 of a wearable band 610 and/or a memory 750A of a computing system 720). In some embodiments, the computing system is a wearable device such as a head wearable device 102. In some embodiments, the computing system is or includes an intermediate device such as a smartphone, a personal computer, or a video game console.
系统经由人工现实头戴式视图器(例如，头部可穿戴设备102)的面向内的摄像头(例如，摄像头104)采集(454)用户的面部的区域的多幅图像，该区域包括用户的眼睛(例如，眼睛110)。The system captures (454) multiple images of an area of a user's face, including the user's eyes (e.g., eye 110), via an inward-facing camera (e.g., camera 104) of an artificial reality head-mounted viewer (e.g., head wearable device 102).
该多幅图像中的第一图像包括(456)瞳孔在眼睛中的第一位置。该多幅图像中的第二图像包括(458)该瞳孔在该眼睛中的第二位置，该第二位置与第一位置不同。例如，在采集该多幅图像时，用户在阅读和/或环顾四周。A first image in the plurality of images includes (456) a first position of a pupil in an eye. A second image in the plurality of images includes (458) a second position of the pupil in the eye, the second position being different from the first position. For example, while the plurality of images are being acquired, the user is reading and/or looking around.
该系统确定(460)该多幅图像是否满足一个或多个质量标准。例如，该系统确定该多幅图像中是否采集到了用户的面部的特定区域的血容量水平。如果摄像头相对于用户的面部偏移或者被遮挡或未聚焦，则该多幅图像可能不满足该一个或多个质量标准。The system determines (460) whether the plurality of images meets one or more quality criteria. For example, the system determines whether the blood volume level of a particular region of the user's face is captured in the plurality of images. The plurality of images may not meet the one or more quality criteria if the camera is offset relative to the user's face or is obscured or out of focus.
根据该多幅图像满足该一个或多个质量标准，该系统基于该多幅图像中用户的眼睛确定(462)心率信息。例如，心率信息是基于以下项确定的：眼睛的运动、眼睛内的血液流动和/或眼睛中的血容量水平。例如，虹膜/瞳孔的不自主运动与用户的心跳相对应，并因此与用户的心率相对应。Based on the plurality of images satisfying the one or more quality criteria, the system determines (462) heart rate information based on the user's eyes in the plurality of images. For example, the heart rate information is determined based on: movement of the eyes, blood flow within the eyes, and/or blood volume level in the eyes. For example, involuntary movement of the iris/pupil corresponds to the user's heartbeat, and therefore to the user's heart rate.
根据该多幅图像不满足该一个或多个质量标准，该系统基于该多幅图像中的肤色变化确定(464)心率信息。例如，该系统确定在该多幅图像中用户的眼睛(例如，眼睛110)是否可见且聚焦。在该示例中，如果用户的眼睛在该多幅图像的至少一个子集中被遮挡或失焦，则系统根据该多幅图像中采集到的面部组织(例如，面部组织112)来确定心率信息。In accordance with the plurality of images not satisfying the one or more quality criteria, the system determines (464) heart rate information based on skin color variations in the plurality of images. For example, the system determines whether eyes of the user (e.g., eyes 110) are visible and in focus in the plurality of images. In this example, if the eyes of the user are obscured or out of focus in at least a subset of the plurality of images, the system determines heart rate information based on facial tissue (e.g., facial tissue 112) captured in the plurality of images.
应理解的是，已经描述的图4A至图4C中的各操作的特定顺序仅是一示例，且不旨在指示所描述的顺序是可执行各操作的唯一顺序。本领域普通技术人员将认识到对本文所描述的各操作进行重新排序的各种方式。另外，应注意的是，本文中关于本文所描述的其它方法(例如，方法400)而描述的其它过程的细节也可以以类似于上述方法450的方式来应用。It should be understood that the specific order of the operations in Figures 4A to 4C that have been described is only an example and is not intended to indicate that the described order is the only order in which the operations can be performed. Those of ordinary skill in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., method 400) can also be applied in a manner similar to the above-described method 450.
以上参考图4A至图4C所描述的各操作可选地由图1A至图1C、图5A和图5B、图5C-1、图5C-2、图5D-1和图5D-2、图6A和图6B以及图7A、图7B-1、图7B-2、图7C、图8A和图8B以及图9A至图9C中所描绘的部件实现。例如，照射操作402、采集操作404和确定操作412可选地由照明源106、计算系统720的成像传感器726和一个或多个处理器748A实现。类似地，本领域普通技术人员将清楚如何基于图1A至图1C、图5A和图5B、图5C-1、图5C-2、图5D-1和图5D-2、图6A和图6B以及图7A、图7B-1、图7B-2、图7C、图8A和图8B以及图9A至图9C中所描绘的部件来实现其它过程。The operations described above with reference to Figures 4A to 4C may optionally be implemented by the components depicted in Figures 1A to 1C, Figures 5A and 5B, Figures 5C-1, 5C-2, Figures 5D-1 and 5D-2, Figures 6A and 6B, and Figures 7A, 7B-1, 7B-2, Figure 7C, Figures 8A and 8B, and Figures 9A to 9C. For example, the illumination operation 402, the acquisition operation 404, and the determination operation 412 may optionally be implemented by the illumination source 106, the imaging sensor 726 of the computing system 720, and one or more processors 748A. Similarly, it will be clear to a person of ordinary skill in the art how to implement other processes based on the components depicted in Figures 1A to 1C, Figures 5A and 5B, Figures 5C-1, 5C-2, Figures 5D-1 and 5D-2, Figures 6A and 6B, and Figures 7A, 7B-1, 7B-2, Figure 7C, Figures 8A and 8B, and Figures 9A to 9C.
另外，在本文所描述的方法中，在一个或多个步骤取决于已满足一个或多个条件的情况下，应理解的是，所描述的方法可以在多次重复中被重复，从而在进行这些重复的过程中，在方法的不同重复中已经满足了该方法中的各步骤所取决的所有条件。例如，如果方法需要在满足条件的情况下执行第一步骤，并且在不满足条件的情况下执行第二步骤，则本领域普通技术人员将理解的是，以不特定的顺序重复所要求保护的步骤，直到条件均已满足和不满足。因此，用取决于已满足一个或多个条件的一个或多个步骤描述的方法可以被重写为这样的方法：该方法被重复，直到该方法中所描述的多个条件中的每个条件已满足。然而，这对于系统或计算机可读介质权利要求不是必需的，其中该系统或该计算机可读介质包含这样的指令：所述指令用于基于满足了相应的一个或多个条件来执行视情况而定的操作，并且因此能够在没有对方法的各步骤进行明确地重复的情况下确定该取决项是否已经被满足，直到该方法中的各步骤所取决的所有条件都得到满足。本领域普通技术人员还将理解的是，与具有视情况而定的步骤的方法类似地，系统或计算机可读存储介质可以根据需要将方法的各步骤重复多次，以确保所有视情况而定的步骤都已执行。In addition, in the methods described herein, where one or more steps depend on one or more conditions being satisfied, it is understood that the described method can be repeated in multiple repetitions, so that in the process of performing these repetitions, all conditions on which the steps in the method depend have been satisfied in different repetitions of the method. For example, if the method requires the execution of a first step when a condition is satisfied, and the execution of a second step when the condition is not satisfied, it will be understood by a person of ordinary skill in the art that the steps claimed are repeated in an unspecified order until the conditions are satisfied and unsatisfied. Therefore, a method described with one or more steps depending on one or more conditions being satisfied can be rewritten as a method in which the method is repeated until each of the multiple conditions described in the method is satisfied. However, this is not necessary for a system or computer-readable medium claim, wherein the system or computer-readable medium contains instructions for performing a contingent operation based on the satisfaction of the corresponding one or more conditions, and thus being able to determine whether the dependent item has been satisfied without explicitly repeating the steps of the method until all the conditions on which the steps in the method depend are satisfied. Those skilled in the art will also understand that, similar to a method having dependent steps, a system or computer-readable storage medium may repeat the steps of the method as many times as necessary to ensure that all dependent steps have been performed.
已经以此方式描述了示例顺序以及使用所述示例顺序的操作的方法，现在将注意力转向可在其上(利用其)实现所述方法的硬件和软件的系统级描述。Having described in this manner an example sequence and method of operation using the example sequence, attention is now turned to a system level description of the hardware and software upon which (with which) the method may be implemented.
以下更详细地描述了上述各设备，这些设备包括系统、腕部可穿戴设备、头戴式视图器设备和基于智能纺织品的服饰。上述特定操作可以作为特定硬件的结果而发生，这种硬件在以下进行了更详细地描述。下文所描述的各设备不是限制性的，并且可以移除这些设备上的特征，也可以向这些设备增加附加的特征。不同的设备可以包括一个或多个类似的硬件部件。为了简洁起见，以下描述了类似的设备和部件。各设备和各部件间的任何差异在下面其各自的部分中进行了描述。The above-mentioned devices, including systems, wrist wearable devices, head-mounted viewer devices, and smart textile-based apparel, are described in more detail below. The above-mentioned specific operations may occur as a result of specific hardware, which is described in more detail below. The devices described below are not restrictive, and features on these devices may be removed or additional features may be added to these devices. Different devices may include one or more similar hardware components. For the sake of brevity, similar devices and components are described below. Any differences between the devices and components are described in their respective sections below.
如本文所述，处理器(例如，中央处理单元(central processing unit，CPU)或微控制器单元(microcontroller unit，MCU))是负责执行指令并控制电子设备(例如，腕部可穿戴设备600、头部可穿戴设备、手持式中间处理设备(HIPD)800、基于智能纺织品的服饰900或其它计算机系统)的操作的电子部件。存在各种类型的处理器，该各种类型的处理器可以互换地使用或者是由本文所描述的各实施例专门要求的。例如，处理器可以是：(i)通用处理器，该通用处理器被设计为执行较广范围的任务，例如运行软件应用、管理操作系统、以及执行算术运算和逻辑运算；(ii)微控制器，该微控制器被设计用于特定任务，例如控制电子设备、传感器和电机；(iii)图形处理单元(graphics processing unit，GPU)，该GPU被设计为加速对图像、视频和动画(例如，虚拟现实动画，例如三维建模)的创建和渲染；(iv)现场可编程门阵列(field-programmable gate array，FPGA)，该FPGA可以在制造后被编程和重配置，和/或被定制为执行特定任务，例如信号处理、加密和机器学习；(v)数字信号处理器(digital signal processor，DSP)，该DSP被设计为对信号(例如音频、视频和无线电波)执行数学运算。本领域技术人员将理解的是，可以在本文所描述的各个实施例中使用一个或多个电子设备的一个或多个处理器。As described herein, a processor (e.g., a central processing unit (CPU) or a microcontroller unit (MCU)) is an electronic component responsible for executing instructions and controlling the operation of an electronic device (e.g., a wrist wearable device 600, a head wearable device, a handheld intermediate processing device (HIPD) 800, a smart textile-based garment 900, or other computer system). There are various types of processors that may be used interchangeably or may be specifically required by the various embodiments described herein. For example, the processor may be: (i) a general-purpose processor designed to perform a wide range of tasks, such as running software applications, managing operating systems, and performing arithmetic and logical operations; (ii) a microcontroller designed for specific tasks, such as controlling electronic devices, sensors, and motors; (iii) a graphics processing unit (GPU) designed to accelerate the creation and rendering of images, videos, and animations (e.g., virtual reality animations, such as three-dimensional modeling); (iv) a field-programmable gate array (FPGA) that can be programmed and reconfigured after manufacturing, and/or customized to perform specific tasks, such as signal processing, encryption, and machine learning; (v) a digital signal processor (DSP) designed to perform mathematical operations on signals (e.g., audio, video, and radio waves). It will be appreciated by those skilled in the art that one or more processors of one or more electronic devices may be used in the various embodiments described herein.
如本文所述，控制器是管理和协调电子设备内其它部件的操作(例如，控制输入、处理数据和/或生成输出)的电子部件。控制器的示例可以包括：(i)微控制器，该微控制器包括通常用于嵌入式系统和物联网(Internet of Things，IoT)设备中的小型低功率控制器；(ii)可编程逻辑控制器(programmable logic controller，PLC)，该PLC可以被配置为用于工业自动化系统中以控制和监测制造过程；(iii)片上系统(system-on-a-chip，SoC)控制器，该SoC控制器将诸如处理器、存储器、输入/输出(I/O)接口和其它外围设备等多个部件集成为单个芯片；和/或DSP。如本文所述，图形模块是被设计为处理图形运算和/或图形化过程的部件或软件模块，并且该图形模块可以包括硬件模块和/或软件模块。As described herein, a controller is an electronic component that manages and coordinates the operation of other components within an electronic device (e.g., controlling inputs, processing data, and/or generating outputs). Examples of controllers may include: (i) a microcontroller, which includes a small, low-power controller commonly used in embedded systems and Internet of Things (IoT) devices; (ii) a programmable logic controller (PLC), which may be configured for use in industrial automation systems to control and monitor manufacturing processes; (iii) a system-on-a-chip (SoC) controller, which integrates multiple components such as a processor, memory, input/output (I/O) interfaces, and other peripherals into a single chip; and/or a DSP. As described herein, a graphics module is a component or software module designed to handle graphics operations and/or graphics processes, and the graphics module may include a hardware module and/or a software module.
如本文所述，存储器是指计算机或电子设备中、存储数据和指令以供处理器访问和操作的电子部件。本文所描述的设备可以包括易失性存储器和非易失性存储器。存储器的示例可以包括：(i)被配置为临时地存储数据和指令的随机存取存储器(random accessmemory，RAM)，例如动态随机存取存储器(DRAM)、静态随机存取存储器(SRAM)、双倍速率随机存取存储器(DDR RAM)或其它随机存取固态存储设备；(ii)被配置为永久性地存储数据和指令的只读存储器(read-only memory，ROM)(例如，系统固件中和/或启动加载器中的一个或多个部分)；(iii)可被配置为存储电子设备中的数据的闪存、磁盘存储设备、光盘存储设备、其它非易失性固态存储设备(例如，通用串行总线(universal serial bus，USB)驱动器、存储卡和/或固态驱动器(solid-state drive，SSD))；以及(iv)被配置为临时存储频繁访问的数据和指令的高速缓冲存储器。如本文所述，存储器可以包括结构化数据(例如，结构化查询语言(SQL)数据库、MongoDB数据库、GraphQL数据或JSON数据)。存储器的其它示例可以包括：(i)资料数据，该资料数据包括用户账号数据、用户设置和/或由用户存储的其它用户数据；(ii)由一个或多个传感器检测到和/或以其它方式获取到的传感器数据；(iii)媒体内容数据，该媒体内容数据包括所存储的图像数据、音频数据、和文档等；(iv)应用数据，该应用数据可以包括在应用的使用期间收集到和/或以其它方式获取到并存储的数据；和/或本文所描述的任何其它类型的数据。As described herein, memory refers to an electronic component in a computer or electronic device that stores data and instructions for access and operation by a processor. The devices described herein may include volatile memory and non-volatile memory. Examples of memory may include: (i) random access memory (RAM) configured to temporarily store data and instructions, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM) or other random access solid-state storage devices; (ii) read-only memory (ROM) configured to permanently store data and instructions (e.g., one or more parts in system firmware and/or boot loader); (iii) flash memory, magnetic disk storage devices, optical disk storage devices, other non-volatile solid-state storage devices (e.g., universal serial bus (USB) drives, memory cards and/or solid-state drives (SSD)) that can be configured to store data in electronic devices; and (iv) a cache memory configured to temporarily store frequently accessed data and instructions. As described herein, the memory may include structured data (e.g., a structured query language (SQL) database, a MongoDB database, GraphQL data, or JSON data). Other examples of memory may include: (i) profile data, which includes user account data, user settings, and/or other user data stored by the user; (ii) sensor data detected by one or more sensors and/or otherwise acquired; (iii) media content data, which includes stored image data, audio data, and documents, etc.; (iv) application data, which may include data collected and/or otherwise acquired and stored during use of the application; and/or any other type of data described herein.
如本文所述，电子设备的电力系统被配置为将输入电力转换为可被用于运行设备的形式。电力系统可以包括各种部件，这些部件包括：(i)电源，该电源可以是交流(alternating current，AC)适配器电源或直流(direct current，DC)适配器电源；(ii)充电器输入，该充电器输入可以被配置为使用有线连接和/或无线连接(该充电器输入可以是外围接口的一部分，该外围接口例如为USB、微USB接口、近场磁耦合、磁感应和磁共振充电、和/或射频(radio frequency，RF)充电)；(iii)电源管理集成电路，该电源管理集成电路被配置为将电力分配给设备的各个部件并确保设备在安全限制内运行(例如，调节电压、控制电流和/或管理散热)；和/或(iv)电池，该电池被配置为存储电力以向一个或多个电子设备的部件提供可用电力。As described herein, a power system of an electronic device is configured to convert input power into a form that can be used to operate the device. The power system can include various components, including: (i) a power source, which can be an alternating current (AC) adapter power source or a direct current (DC) adapter power source; (ii) a charger input, which can be configured to use a wired connection and/or a wireless connection (the charger input can be part of a peripheral interface, such as a USB, a micro USB interface, near-field magnetic coupling, magnetic induction and magnetic resonance charging, and/or radio frequency (RF) charging); (iii) a power management integrated circuit, which is configured to distribute power to various components of the device and ensure that the device operates within safe limits (e.g., regulating voltage, controlling current and/or managing heat dissipation); and/or (iv) a battery, which is configured to store power to provide usable power to one or more components of the electronic device.
如本文所述，外围接口是这样的电子部件(例如，电子设备的电子部件)：所述电子部件允许电子设备与其它设备或外围设备通信，并且可以提供用于对数据和信号进行输入和输出的途径。外围接口的示例可以包括：(i)USB接口和/或微USB接口，该USB接口和/或微USB接口被配置用于将设备连接到电子设备；(ii)蓝牙接口，该蓝牙接口被配置为允许多个设备彼此通信，该蓝牙接口包括低功耗蓝牙(Bluetooth Low Energy，BLE)；(iii)近场通信(near-field communication，NFC)接口，该NFC接口被配置作为用于诸如接入控制等操作的短距离无线接口；(iv)弹簧针(POGO pin)，该弹簧针可以是被配置为提供充电接口的小型、加载有弹簧的管脚；(v)无线充电接口；(vi)全球定位系统(global-position system，GPS)接口；(vii)Wi-Fi接口，该Wi-Fi接口用于提供设备与无线网络之间的连接；以及(viii)传感器接口。As described herein, a peripheral interface is an electronic component (e.g., an electronic component of an electronic device) that allows the electronic device to communicate with other devices or peripherals and can provide a means for inputting and outputting data and signals. Examples of peripheral interfaces may include: (i) a USB interface and/or a micro USB interface configured to connect a device to an electronic device; (ii) a Bluetooth interface configured to allow multiple devices to communicate with each other, including Bluetooth Low Energy (BLE); (iii) a near-field communication (NFC) interface configured as a short-range wireless interface for operations such as access control; (iv) a pogo pin, which can be a small, spring-loaded pin configured to provide a charging interface; (v) a wireless charging interface; (vi) a global-position system (GPS) interface; (vii) a Wi-Fi interface used to provide a connection between a device and a wireless network; and (viii) a sensor interface.
如本文所述，传感器是被配置为检测物理变化和环境变化并生成电信号的电子部件(例如，电子设备(例如可穿戴设备)中的电子部件和/或以其它方式与电子设备进行电子通信的电子部件)。传感器的示例可以包括：(i)用于收集成像数据的成像传感器(例如，包括设置在相应电子设备上的一个或多个摄像头)；(ii)生物电势信号传感器；(iii)惯性测量单元(例如，IMU)，该惯性测量单元用于检测例如角速度、力、磁场和/或加速度变化；(iv)心率传感器，该心率传感器用于测量用户的心率；(v)血氧饱和度(SpO2)传感器，该SpO2传感器用于测量用户的血氧饱和度和/或其它生物特征数据；(vi)电容式传感器，该电容式传感器用于检测用户身体的一部位(例如，传感器-皮肤接口)处的电势变化和/或其它设备或对象的接近度；以及(vii)光传感器(例如，飞行时间(ToF)传感器、红外光传感器、或可见光传感器)，和/或用于感测来自用户或用户的环境的数据的传感器。如本文所述，生物电势信号感测部件是用于测量身体内的电活动的设备(例如，生物电势信号传感器)。一些类型的生物电势信号传感器包括：(i)脑电图(electroencephalography，EEG)传感器，该EEG传感器被配置为测量大脑中的电活动以诊断神经疾病；(ii)心电图(electrocardiography，ECG或EKG)传感器，该ECG或EKG传感器被配置为测量心脏的电活动以诊断心脏问题；(iii)肌电图(electromyography，EMG)传感器，该EMG传感器被配置为测量肌肉的电活动并诊断神经肌肉紊乱；(iv)眼电图(electrooculography，EOG)传感器，该EOG传感器被配置为测量眼部肌肉的电活动以检测眼动并诊断眼部紊乱。As described herein, a sensor is an electronic component (e.g., an electronic component in an electronic device (e.g., a wearable device) and/or otherwise in electronic communication with an electronic device) that is configured to detect physical and environmental changes and generate an electrical signal. Examples of sensors may include: (i) an imaging sensor for collecting imaging data (e.g., including one or more cameras disposed on a corresponding electronic device); (ii) a biopotential signal sensor; (iii) an inertial measurement unit (e.g., an IMU) for detecting, for example, angular velocity, force, magnetic field, and/or acceleration changes; (iv) a heart rate sensor for measuring a user's heart rate; (v) a blood oxygen saturation (SpO2) sensor for measuring a user's blood oxygen saturation and/or other biometric data; (vi) a capacitive sensor for detecting a change in electrical potential at a portion of a user's body (e.g., a sensor-skin interface) and/or proximity of other devices or objects; and (vii) an optical sensor (e.g., a time-of-flight (ToF) sensor, an infrared light sensor, or a visible light sensor), and/or a sensor for sensing data from a user or the user's environment. As described herein, a biopotential signal sensing component is a device (e.g., a biopotential signal sensor) for measuring electrical activity within a body. Some types of biopotential signal sensors include: (i) an electroencephalography (EEG) sensor, which is configured to measure electrical activity in the brain to diagnose neurological diseases; (ii) an electrocardiography (ECG or EKG) sensor, which is configured to measure the electrical activity of the heart to diagnose heart problems; (iii) an electromyography (EMG) sensor, which is configured to measure the electrical activity of muscles and diagnose neuromuscular disorders; (iv) an electrooculography (EOG) sensor, which is configured to measure the electrical activity of eye muscles to detect eye movements and diagnose eye disorders.
如本文所述，存储在电子设备的存储器中的应用(例如，软件)包括存储在存储器中的指令。这种应用的示例包括：(i)游戏；(ii)文字处理器；(iii)消息传递应用；(iv)媒体流应用；(v)金融应用；(vi)日历；(vii)时钟；(viii)网页浏览器；(ix)社交媒体应用；(x)相机应用；(xi)基于网络的应用；(xii)健康应用；(xiii)人工现实(artificial-reality，AR)应用，和/或可存储在存储器中的任何其它应用。应用可以结合数据和/或一设备或通信耦接的多个设备的一个或多个部件运行，以执行一个或多个操作和/或功能。As described herein, an application (e.g., software) stored in a memory of an electronic device includes instructions stored in the memory. Examples of such applications include: (i) games; (ii) word processors; (iii) messaging applications; (iv) media streaming applications; (v) financial applications; (vi) calendars; (vii) clocks; (viii) web browsers; (ix) social media applications; (x) camera applications; (xi) web-based applications; (xii) health applications; (xiii) artificial reality (AR) applications, and/or any other application that may be stored in the memory. An application may operate in conjunction with data and/or one or more components of a device or multiple devices coupled in communication to perform one or more operations and/or functions.
如本文所述，通信接口模块可以包括能够使用以下各种协议中的任何一种进行数据通信的硬件和/或软件：定制无线协议或标准无线协议(例如，IEEE 802.15.4、Wi-Fi、紫蜂(ZigBee)、6LoWPAN、Thread、Z-Wave、低功耗蓝牙(Bluetooth Smart)、ISA100.11a、无线可寻址远程传感器高速通道(WirelessHART)或MiWi)；定制有线协议或标准有线协议(例如，以太网或HomePlug)；和/或任何其它合适的通信协议(包括截止本文件的提交日期尚未开发的通信协议)。通信接口是使不同的系统或设备能够彼此交换信息和数据的机构，其包括硬件、软件或者硬件和软件两者的组合。例如，通信接口可以指一设备上的、实现与其它设备的通信的物理连接器和/或端口(例如，USB、以太网、高清多媒体接口(HDMI)或蓝牙)。在一些实施例中，通信接口可以指使不同软件程序能够彼此通信的软件层(例如，应用编程接口(application programming interface，API)和诸如超文本传输协议(HTTP)和传输控制协议/网际协议(TCP/IP)等协议)。As described herein, the communication interface module may include hardware and/or software capable of communicating data using any of the following protocols: a custom wireless protocol or a standard wireless protocol (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a, WirelessHART, or MiWi); a custom wired protocol or a standard wired protocol (e.g., Ethernet or HomePlug); and/or any other suitable communication protocol (including communication protocols that have not been developed as of the filing date of this document). A communication interface is a mechanism that enables different systems or devices to exchange information and data with each other, including hardware, software, or a combination of both hardware and software. For example, a communication interface may refer to a physical connector and/or port on a device that enables communication with other devices (e.g., USB, Ethernet, High Definition Multimedia Interface (HDMI), or Bluetooth). In some embodiments, a communication interface may refer to a software layer (e.g., an application programming interface (API) and protocols such as Hypertext Transfer Protocol (HTTP) and Transmission Control Protocol/Internet Protocol (TCP/IP)) that enables different software programs to communicate with each other.
如本文所述，图形模块是被设计为处理图形运算和/或图形化过程的部件或软件模块，并且该图形模块可以包括硬件模块和/或软件模块。As described herein, a graphics module is a component or software module designed to process graphics operations and/or graphics processes, and the graphics module may include a hardware module and/or a software module.
如本文所述，非暂态计算机可读存储介质是可用于以非暂态形式存储电子数据(例如，使得数据被永久性地存储直到该数据被有意删除或修改为止)的物理设备或存储介质。As described herein, a non-transitory computer-readable storage medium is a physical device or storage medium that can be used to store electronic data in a non-transitory form (eg, such that the data is permanently stored until the data is intentionally deleted or modified).
示例AR系统500a至500dExample AR Systems 500a-500d
图5A、图5B、图5C-1、图5C-2、图5D-1和图5D-2示出了根据一些实施例的示例AR系统。图5A显示了第一AR系统500a以及第一示例用户交互，该第一示例用户交互使用腕部可穿戴设备600、头部可穿戴设备(例如，AR设备700)和/或手持式中间处理设备(handheldintermediary processing device，HIPD)800。图5B显示了第二AR系统500b以及第二示例用户交互，该第二示例用户交互使用腕部可穿戴设备600、AR设备700和/或HIPD 800。图5C-1和图5C-2显示了第三AR系统500c以及第三示例用户交互，该第三示例用户交互使用腕部可穿戴设备600、头部可穿戴设备(例如，虚拟现实(virtual-reality，VR)设备710)和/或HIPD 800。图5D-1和图5D-2显示了第四AR系统500d以及第四示例用户交互，该第四示例用户交互使用腕部可穿戴设备600、VR设备710和/或基于智能纺织品的服饰900(例如，可穿戴手套、触觉手套)。如本领域技术人员在阅读本文所提供的描述时将理解的，以上示例AR系统(在下文进行了详细描述)可以执行以上参考图1A至图4C所描述的各种功能和/或操作。Figures 5A, 5B, 5C-1, 5C-2, 5D-1, and 5D-2 illustrate example AR systems according to some embodiments. Figure 5A shows a first AR system 500a and a first example user interaction, which uses a wrist wearable device 600, a head wearable device (e.g., an AR device 700), and/or a handheld intermediary processing device (HIPD) 800. Figure 5B shows a second AR system 500b and a second example user interaction, which uses a wrist wearable device 600, an AR device 700, and/or a HIPD 800. Figures 5C-1 and 5C-2 show a third AR system 500c and a third example user interaction, which uses a wrist wearable device 600, a head wearable device (e.g., a virtual reality (VR) device 710), and/or a HIPD 800. 5D-1 and 5D-2 show a fourth AR system 500d and a fourth example user interaction using a wrist wearable device 600, a VR device 710, and/or a smart textile-based apparel 900 (e.g., a wearable glove, a haptic glove). As will be understood by those skilled in the art upon reading the description provided herein, the above example AR systems (described in detail below) can perform the various functions and/or operations described above with reference to FIGS. 1A to 4C.
以下参考图6A和图6B描述了腕部可穿戴设备600及其组成部件；以下参考图7A至图7C描述了头部可穿戴设备及其组成部件；并且以下参考图8A和图8B描述了HIPD 800及其组成部件。以下参考图9A至图9C描述了基于智能纺织品的服饰900及其一个或多个部件。腕部可穿戴设备600、头部可穿戴设备和/或HIPD 800可以经由网络525(例如，蜂窝、近场、Wi-Fi、个人局域网或无线局域网(LAN))通信耦接。另外，腕部可穿戴设备600、头部可穿戴设备和/或HIPD 800还可以经由网络525(例如，蜂窝、近场、Wi-Fi、个人局域网或无线LAN)与一个或多个服务器530、计算机540(例如，膝上型计算机或计算机)、移动设备550(例如，智能手机或平板电脑)和/或其它电子设备通信耦接。类似地，基于智能纺织品的服饰900在被使用时，也可以经由网络525与腕部可穿戴设备600、头部可穿戴设备、HIPD 800、一个或多个服务器530、计算机540、移动设备550和/或其它电子设备通信耦接。The wrist wearable device 600 and its components are described below with reference to Figures 6A and 6B; the head wearable device and its components are described below with reference to Figures 7A to 7C; and the HIPD 800 and its components are described below with reference to Figures 8A and 8B. The smart textile-based clothing 900 and one or more components thereof are described below with reference to Figures 9A to 9C. The wrist wearable device 600, the head wearable device, and/or the HIPD 800 can be communicatively coupled via a network 525 (e.g., cellular, near-field, Wi-Fi, a personal area network, or a wireless local area network (LAN)). In addition, the wrist wearable device 600, the head wearable device, and/or the HIPD 800 can also be communicatively coupled to one or more servers 530, computers 540 (e.g., laptops or computers), mobile devices 550 (e.g., smartphones or tablets), and/or other electronic devices via a network 525 (e.g., cellular, near-field, Wi-Fi, a personal area network, or a wireless LAN). Similarly, when being used, the smart textile-based clothing 900 can also be communicatively coupled to the wrist wearable device 600, the head wearable device, the HIPD 800, one or more servers 530, the computer 540, the mobile device 550 and/or other electronic devices via the network 525.
转到图5A，显示了佩戴着腕部可穿戴设备600和AR设备700的用户502，并且该用户将HIPD 800放在其桌子上。腕部可穿戴设备600、AR设备700和HIPD 800促进用户与AR环境的交互。具体地，如第一AR系统500a所示，腕部可穿戴设备600、AR设备700和/或HIPD 800使得呈现一个或多个化身(avatar)504、联系人的数字表示506和虚拟对象508。如以下所论述的，用户502可以经由腕部可穿戴设备600、AR设备700和/或HIPD 800与一个或多个化身504、联系人的数字表示506和虚拟对象508交互。Turning to FIG. 5A , a user 502 wearing a wrist wearable device 600 and an AR device 700 is shown, and the user places a HIPD 800 on his desk. The wrist wearable device 600, the AR device 700, and the HIPD 800 facilitate the user's interaction with the AR environment. Specifically, as shown in the first AR system 500a, the wrist wearable device 600, the AR device 700, and/or the HIPD 800 cause one or more avatars 504, digital representations 506 of contacts, and virtual objects 508 to be presented. As discussed below, the user 502 can interact with one or more avatars 504, digital representations 506 of contacts, and virtual objects 508 via the wrist wearable device 600, the AR device 700, and/or the HIPD 800.
用户502可以使用腕部可穿戴设备600、AR设备700和/或HIPD 800中的任一者来提供用户输入。例如，用户502可以执行一个或多个手势，以提供用户输入，该一个或多个手势由腕部可穿戴设备600(例如，使用以下参考图6A和图6B进行描述的一个或多个EMG传感器和/或IMU)检测和/或由AR设备700(例如，使用以下参考图7A和图7B-1以及图7B-2进行描述的一个或多个图像传感器或摄像头)检测。替代地或附加地，用户502可以经由以下项来提供用户输入：腕部可穿戴设备600、AR设备700和/或HIPD 800的一个或多个触摸表面，和/或由腕部可穿戴设备600、AR设备700和/或HIPD 800的传声器采集到的语音命令。在一些实施例中，腕部可穿戴设备600、AR设备700和/或HIPD 800包括用于帮助用户提供用户输入(例如，完成一系列操作、推荐不同的操作或命令、提供提醒或确认命令)的数字助理。在一些实施例中，用户502可以经由一个或多个面部姿态和/或面部表情提供用户输入。例如，腕部可穿戴设备600、AR设备700和/或HIPD 800的摄像头可以追踪用户502的眼睛以用于导航用户界面。The user 502 may provide user input using any of the wrist wearable device 600, the AR device 700, and/or the HIPD 800. For example, the user 502 may perform one or more gestures to provide user input, which are detected by the wrist wearable device 600 (e.g., one or more EMG sensors and/or IMUs described below with reference to FIG. 6A and FIG. 6B ) and/or by the AR device 700 (e.g., one or more image sensors or cameras described below with reference to FIG. 7A and FIG. 7B-1 and FIG. 7B-2 ). Alternatively or additionally, the user 502 may provide user input via one or more touch surfaces of the wrist wearable device 600, the AR device 700, and/or the HIPD 800, and/or voice commands collected by microphones of the wrist wearable device 600, the AR device 700, and/or the HIPD 800. In some embodiments, the wrist wearable device 600, the AR device 700, and/or the HIPD 800 include a digital assistant for helping the user provide user input (e.g., completing a series of operations, recommending different operations or commands, providing reminders or confirming commands). In some embodiments, the user 502 can provide user input via one or more facial gestures and/or facial expressions. For example, a camera of the wrist wearable device 600, the AR device 700, and/or the HIPD 800 can track the eyes of the user 502 for navigating the user interface.
腕部可穿戴设备600、AR设备700和/或HIPD 800可以单独运行或结合运行，以允许用户502与AR环境交互。在一些实施例中，HIPD 800被配置为作为如下设备的中央枢纽或控制中心来运行：腕部可穿戴设备600、AR设备700和/或另一通信耦接的设备。例如，用户502可以在腕部可穿戴设备600、AR设备700和/或HIPD 800中的任一者处提供与AR环境交互的输入，并且HIPD 800可以识别一个或多个后端任务和前端任务以使得执行所请求的交互，并分发指令以使得在腕部可穿戴设备600、AR设备700和/或HIPD 800处执行该一个或多个后端任务和前端任务。在一些实施例中，后端任务是用户不可感知的后台处理任务(例如，渲染内容、解压缩或压缩)，而前端任务是用户可感知的面向用户的任务(例如，向用户呈现信息或向用户提供反馈)。如以下参考图8A和图8B所描述的，HIPD 800可以执行后端任务，并向腕部可穿戴设备600和/或AR设备700提供与所执行的后端任务相对应的操作数据，使得腕部可穿戴设备600和/或AR设备700可以执行前端任务。以此方式，HIPD 800(其具有比腕部可穿戴设备600和/或AR设备700更多的计算资源和更大的热余量(thermalheadroom))执行计算密集型任务，并降低腕部可穿戴设备600和/或AR设备700的计算机资源利用和/或电力使用。The wrist wearable device 600, the AR device 700, and/or the HIPD 800 can operate alone or in combination to allow the user 502 to interact with the AR environment. In some embodiments, the HIPD 800 is configured to operate as a central hub or control center for the following devices: the wrist wearable device 600, the AR device 700, and/or another communicatively coupled device. For example, the user 502 can provide input for interacting with the AR environment at any one of the wrist wearable device 600, the AR device 700, and/or the HIPD 800, and the HIPD 800 can identify one or more back-end tasks and front-end tasks to perform the requested interaction, and distribute instructions to perform the one or more back-end tasks and front-end tasks at the wrist wearable device 600, the AR device 700, and/or the HIPD 800. In some embodiments, the back-end task is a background processing task that is not perceptible to the user (e.g., rendering content, decompression, or compression), and the front-end task is a user-oriented task that is perceptible to the user (e.g., presenting information to the user or providing feedback to the user). As described below with reference to FIG8A and FIG8B, HIPD 800 can perform backend tasks and provide operation data corresponding to the performed backend tasks to wrist wearable device 600 and/or AR device 700, so that wrist wearable device 600 and/or AR device 700 can perform frontend tasks. In this way, HIPD 800 (which has more computing resources and greater thermal headroom than wrist wearable device 600 and/or AR device 700) performs computationally intensive tasks and reduces computer resource utilization and/or power usage of wrist wearable device 600 and/or AR device 700.
在第一AR系统500a所示的示例中，HIPD 800识别与发起与一个或多个其他用户(由化身504和联系人的数字表示506来表示)的AR视频通话的用户请求相关联的一个或多个后端任务和前端任务，并分发指令以使得执行该一个或多个后端任务和前端任务。具体地，HIPD 800执行用于处理和/或渲染与AR视频通话相关联的图像数据(和其它数据)的后端任务，并向AR设备700提供与所执行的后端任务相关联的操作数据，使得AR设备700执行用于呈现AR视频通话(例如，呈现化身504和联系人的数字表示506)的前端任务。In the example shown in the first AR system 500a, the HIPD 800 identifies one or more back-end tasks and front-end tasks associated with a user request to initiate an AR video call with one or more other users (represented by avatars 504 and digital representations 506 of contacts), and distributes instructions to cause the one or more back-end tasks and front-end tasks to be performed. Specifically, the HIPD 800 performs back-end tasks for processing and/or rendering image data (and other data) associated with the AR video call, and provides the AR device 700 with operation data associated with the performed back-end tasks, so that the AR device 700 performs front-end tasks for presenting the AR video call (e.g., presenting the avatars 504 and digital representations 506 of contacts).
在一些实施例中，HIPD 800可以作为使得信息呈现的焦点或锚点来运行。这允许用户502大体上知道信息呈现在哪里。例如，如第一AR系统500a所示，化身504和联系人的数字表示506被呈现在HIPD 800上方。具体地，HIPD 800和AR设备700结合运行以确定用于呈现化身504和联系人的数字表示506的位置。在一些实施例中，可以在距HIPD 800的预定距离内(例如，五米内)呈现信息。例如，如第一AR系统500a所示，虚拟对象508被呈现在桌子上、距HIPD 800一定距离。类似于以上示例，HIPD 800和AR设备700可以结合运行以确定用于呈现虚拟对象508的位置。替代地，在一些实施例中，信息的呈现不受HIPD 800的约束。更具体地，化身504、联系人的数字表示506和虚拟对象508不必被呈现在HIPD 800的预定距离内。In some embodiments, the HIPD 800 can be operated as a focus or anchor point for information presentation. This allows the user 502 to know generally where the information is presented. For example, as shown in the first AR system 500a, the avatar 504 and the digital representation 506 of the contact are presented above the HIPD 800. Specifically, the HIPD 800 and the AR device 700 operate in combination to determine the location for presenting the avatar 504 and the digital representation 506 of the contact. In some embodiments, information can be presented within a predetermined distance (e.g., within five meters) from the HIPD 800. For example, as shown in the first AR system 500a, the virtual object 508 is presented on the table at a distance from the HIPD 800. Similar to the above example, the HIPD 800 and the AR device 700 can be combined to operate to determine the location for presenting the virtual object 508. Alternatively, in some embodiments, the presentation of information is not constrained by the HIPD 800. More specifically, the avatar 504 , the digital representation of the contact 506 , and the virtual object 508 do not have to be presented within a predetermined distance of the HIPD 800 .
对在腕部可穿戴设备600、AR设备700和/或HIPD 800处提供的用户输入进行协调，使得用户可以使用任何设备来发起、继续和/或完成一操作。例如，用户502可以向AR设备700提供用户输入以使AR设备700呈现虚拟对象508，并且在AR设备700呈现虚拟对象508时，用户502可以经由腕部可穿戴设备600提供一个或多个手势以与虚拟对象508交互和/或操纵虚拟对象508。User input provided at the wrist wearable device 600, the AR device 700, and/or the HIPD 800 is coordinated so that the user can use any device to initiate, continue, and/or complete an operation. For example, the user 502 can provide user input to the AR device 700 to cause the AR device 700 to present the virtual object 508, and when the AR device 700 presents the virtual object 508, the user 502 can provide one or more gestures via the wrist wearable device 600 to interact with and/or manipulate the virtual object 508.
图5B显示了佩戴着腕部可穿戴设备600和AR设备700、并握持着HIPD 800的用户502。在第二AR系统500b中，腕部可穿戴设备600、AR设备700和/或HIPD 800用于接收一个或多个消息和/或向用户502的联系人提供一个或多个消息。具体地，腕部可穿戴设备600、AR设备700和/或HIPD 800检测并协调一个或多个用户输入以启动消息传递应用，并经由该消息传递应用准备对接收到的消息的响应。5B shows a user 502 wearing a wrist wearable device 600 and an AR device 700, and holding a HIPD 800. In the second AR system 500b, the wrist wearable device 600, the AR device 700, and/or the HIPD 800 are used to receive one or more messages and/or provide one or more messages to contacts of the user 502. Specifically, the wrist wearable device 600, the AR device 700, and/or the HIPD 800 detect and coordinate one or more user inputs to launch a messaging application, and prepare a response to the received message via the messaging application.
在一些实施例中，用户502经由用户输入来启动腕部可穿戴设备600、AR设备700和/或HIPD 800上的应用，这使得该应用在至少一个设备上启动。例如，在第二AR系统500b中，用户502执行与用于启动消息传递应用(由消息传递用户界面512表示)的命令相关联的手势；腕部可穿戴设备600检测该手势，并基于确定用户502正佩戴着AR设备700，使AR设备700呈现消息传递应用的消息传递用户界面512。AR设备700可以经由其显示器(例如，如用户502的视场510所示)向用户502呈现消息传递用户界面512。在一些实施例中，应用被启动且可以在检测到用于启动应用的用户输入的设备(例如，腕部可穿戴设备600、AR设备700和/或HIPD 800)上运行，并且该设备向另一设备提供操作数据以使得呈现消息传递应用。例如，腕部可穿戴设备600可以检测用于启动消息传递应用的用户输入，启动并运行消息传递应用，并且向AR设备700和/或HIPD 800提供操作数据以使得呈现消息传递应用。替代地，应用可以被启动且在与检测到用户输入的设备不同的设备上运行。例如，腕部可穿戴设备600可以检测与启动消息传递应用相关联的手势，并且可以使HIPD 800运行消息传递应用并协调对消息传递应用的呈现。In some embodiments, the user 502 launches an application on the wrist wearable device 600, the AR device 700, and/or the HIPD 800 via user input, which causes the application to be launched on at least one device. For example, in the second AR system 500b, the user 502 performs a gesture associated with a command for launching a messaging application (represented by the messaging user interface 512); the wrist wearable device 600 detects the gesture and, based on determining that the user 502 is wearing the AR device 700, causes the AR device 700 to present the messaging user interface 512 of the messaging application. The AR device 700 can present the messaging user interface 512 to the user 502 via its display (e.g., as shown in the field of view 510 of the user 502). In some embodiments, the application is launched and can be run on the device (e.g., the wrist wearable device 600, the AR device 700, and/or the HIPD 800) that detects the user input for launching the application, and the device provides operational data to another device to cause the messaging application to be presented. For example, the wrist wearable device 600 can detect user input for launching a messaging application, launch and run the messaging application, and provide operation data to the AR device 700 and/or the HIPD 800 so that the messaging application is presented. Alternatively, the application can be launched and run on a device different from the device that detects the user input. For example, the wrist wearable device 600 can detect a gesture associated with launching a messaging application, and can cause the HIPD 800 to run the messaging application and coordinate the presentation of the messaging application.
此外，用户502可以提供在腕部可穿戴设备600、AR设备700和/或HIPD 800处提供的用户输入，以继续和/或完成在另一设备处发起的操作。例如，在经由腕部可穿戴设备600启动了消息传递应用之后、且在AR设备700呈现消息传递用户界面512时，用户502可以在HIPD 800处提供输入以准备响应(例如，由在HIPD 800上执行的滑动手势所示)。用户502在HIPD 800上执行的手势可以被提供和/或被显示在另一设备上。例如，用户502在HIPD 800上执行的滑动手势被显示在由AR设备700显示的消息传递用户界面512的虚拟键盘上。In addition, the user 502 can provide user input provided at the wrist wearable device 600, the AR device 700, and/or the HIPD 800 to continue and/or complete an operation initiated at another device. For example, after launching a messaging application via the wrist wearable device 600 and when the AR device 700 presents the messaging user interface 512, the user 502 can provide input at the HIPD 800 to prepare a response (e.g., as shown by a sliding gesture performed on the HIPD 800). The gesture performed by the user 502 on the HIPD 800 can be provided and/or displayed on another device. For example, the sliding gesture performed by the user 502 on the HIPD 800 is displayed on the virtual keyboard of the messaging user interface 512 displayed by the AR device 700.
在一些实施例中，腕部可穿戴设备600、AR设备700、HIPD 800和/或其它通信耦接的设备可以向用户502呈现一个或多个通知。该通知可以是对新消息、来电、应用更新、状态更新等的指示。用户502可以经由腕部可穿戴设备600、AR设备700或HIPD 800来对通知进行选择，并使得在至少一个设备上呈现与该通知相关联的应用或操作。例如，用户502可以在腕部可穿戴设备600、AR设备700、HIPD 800和/或其它通信耦接的设备处接收接收到一消息的通知，并在腕部可穿戴设备600、AR设备700和/或HIPD 800处提供用户输入以查看该通知，并且检测到该用户输入的设备可以使得启动与该通知相关联的应用和/或在腕部可穿戴设备600、AR设备700和/或HIPD 800处呈现与该通知相关联的应用。In some embodiments, the wrist wearable device 600, the AR device 700, the HIPD 800, and/or other communicatively coupled devices may present one or more notifications to the user 502. The notification may be an indication of a new message, an incoming call, an application update, a status update, etc. The user 502 may select a notification via the wrist wearable device 600, the AR device 700, or the HIPD 800, and cause an application or operation associated with the notification to be presented on at least one device. For example, the user 502 may receive a notification of receiving a message at the wrist wearable device 600, the AR device 700, the HIPD 800, and/or other communicatively coupled devices, and provide user input at the wrist wearable device 600, the AR device 700, and/or the HIPD 800 to view the notification, and the device detecting the user input may cause the application associated with the notification to be launched and/or the application associated with the notification to be presented at the wrist wearable device 600, the AR device 700, and/or the HIPD 800.
尽管以上示例描述了用于与消息传递应用交互的、协调后的输入，但是本领域技术人员在阅读本描述时将理解的是，用户输入可以被协调以与任何数量的应用交互，这些应用包括但不限于游戏应用、社交媒体应用、相机应用、基于网页的应用和金融应用等。例如，AR设备700可以向用户502呈现游戏应用数据，而HIPD 800可以使用控制器来向游戏提供输入。类似地，用户502可以使用腕部可穿戴设备600来启动AR设备700的摄像头，并且用户可以使用腕部可穿戴设备600、AR设备700和/或HIPD 800来操纵图像采集(例如，放大或缩小、或应用滤镜)并采集图像数据。Although the above example describes coordinated input for interacting with a messaging application, it will be understood by those skilled in the art upon reading this description that user input can be coordinated to interact with any number of applications, including but not limited to gaming applications, social media applications, camera applications, web-based applications, and financial applications, etc. For example, the AR device 700 can present gaming application data to the user 502, and the HIPD 800 can use a controller to provide input to the game. Similarly, the user 502 can use the wrist wearable device 600 to activate the camera of the AR device 700, and the user can use the wrist wearable device 600, the AR device 700, and/or the HIPD 800 to manipulate image acquisition (e.g., zoom in or out, or apply filters) and acquire image data.
转到图5C-1和图5C-2，显示了佩戴着腕部可穿戴设备600和VR设备710、并握持着HIPD 800的用户502。在第三AR系统500c中，腕部可穿戴设备600、VR设备710和/或HIPD 800用于在AR环境(例如VR游戏或其它AR应用)内交互。尽管VR设备710向用户502呈现VR游戏的表示(例如，第一AR游戏环境520)，但是腕部可穿戴设备600、VR设备710和/或HIPD 800对一个或多个用户输入进行检测和协调，以允许用户502与VR游戏交互。Turning to Figures 5C-1 and 5C-2, a user 502 wearing a wrist wearable device 600 and a VR device 710 and holding a HIPD 800 is shown. In the third AR system 500c, the wrist wearable device 600, the VR device 710, and/or the HIPD 800 are used to interact within an AR environment (e.g., a VR game or other AR application). Although the VR device 710 presents a representation of a VR game (e.g., a first AR game environment 520) to the user 502, the wrist wearable device 600, the VR device 710, and/or the HIPD 800 detect and coordinate one or more user inputs to allow the user 502 to interact with the VR game.
在一些实施例中，用户502可以经由腕部可穿戴设备600、VR设备710和/或HIPD800提供在相应的AR环境中引起动作的用户输入。例如，第三AR系统500c中的用户502(如图5C-1所示)举起HIPD 800，以准备进行第一AR游戏环境520中的挥动。VR设备710响应于用户502举起HIPD 800，使用户的AR表示522执行类似的动作(例如，举起虚拟对象，例如虚拟剑524)。在一些实施例中，每个设备使用相应的传感器数据和/或图像数据来检测用户输入并提供用户502的运动的准确表示。例如，HIPD 800的成像传感器854(例如，以下在图8A和图8B中所论述的即时定位与地图构建(SLAM)摄像头或其它摄像头)可以用于检测HIPD 800相对于用户502的身体的位置，使得虚拟对象可以被适当地定位在第一AR游戏环境520内；来自腕部可穿戴设备600的传感器数据可以用于检测用户502举起HIPD 800的速度，使得用户的AR表示522和虚拟剑524与用户502的移动同步；并且VR设备710的图像传感器726(图7A至图7C)可以用于表示用户502的身体、边界条件或第一AR游戏环境520内的真实世界对象。In some embodiments, the user 502 can provide user input that causes an action in the corresponding AR environment via the wrist wearable device 600, the VR device 710, and/or the HIPD 800. For example, the user 502 in the third AR system 500c (as shown in FIG. 5C-1) raises the HIPD 800 to prepare for a swing in the first AR game environment 520. The VR device 710 responds to the user 502 raising the HIPD 800, causing the user's AR representation 522 to perform a similar action (e.g., raising a virtual object, such as a virtual sword 524). In some embodiments, each device uses corresponding sensor data and/or image data to detect user input and provide an accurate representation of the user's 502 movement. For example, the imaging sensor 854 of the HIPD 800 (e.g., a simultaneous localization and mapping (SLAM) camera or other camera discussed below in FIGS. 8A and 8B ) can be used to detect the position of the HIPD 800 relative to the body of the user 502 so that virtual objects can be appropriately positioned within the first AR gaming environment 520; sensor data from the wrist wearable device 600 can be used to detect the speed at which the user 502 raises the HIPD 800 so that the user's AR representation 522 and virtual sword 524 are synchronized with the movement of the user 502; and the image sensor 726 ( FIGS. 7A to 7C ) of the VR device 710 can be used to represent the body of the user 502, boundary conditions, or real-world objects within the first AR gaming environment 520.
在图5C-2中，用户502在握持着HIPD 800时执行向下挥动。腕部可穿戴设备600、VR设备710和/或HIPD 800检测到用户502的向下挥动，并且在第一AR游戏环境520中执行相应的动作。在一些实施例中，由每个设备采集到的数据用于提升用户在AR环境内的体验。例如，腕部可穿戴设备600的传感器数据可以用于确定执行向下挥动的速度和/或力，并且HIPD 800和/或VR设备710的图像传感器可以用于确定挥动的位置以及应该如何在第一AR游戏环境520中表示该挥动，这继而可以用作对AR环境的输入(例如，游戏机制，该游戏机制可以使用检测到的速度、力、位置和/或用户502的动作的各方面来对用户的输入进行分类(例如，用户执行轻击、重击、暴击、斜击(glancing strike)、未击中(miss))或者可以计算输出(例如，损坏量))。In FIG. 5C-2 , user 502 performs a downward swing while holding HIPD 800. The wrist wearable device 600, VR device 710, and/or HIPD 800 detects the downward swing of user 502 and performs a corresponding action in the first AR game environment 520. In some embodiments, the data collected by each device is used to enhance the user's experience in the AR environment. For example, sensor data from the wrist wearable device 600 can be used to determine the speed and/or force of performing a downward swing, and the image sensor of the HIPD 800 and/or VR device 710 can be used to determine the location of the swing and how the swing should be represented in the first AR game environment 520, which in turn can be used as an input to the AR environment (e.g., a game mechanism that can use the detected speed, force, position, and/or aspects of the user's 502 action to classify the user's input (e.g., the user performs a light strike, a heavy strike, a critical strike, a glancing strike, a miss) or can calculate an output (e.g., an amount of damage)).
尽管腕部可穿戴设备600、VR设备710和/或HIPD 800被描述为检测用户输入，但是在一些实施例中，在单个设备处检测检测用户输入(其中该单个设备负责将信号分发到用于执行用户输入的其它设备)。例如，HIPD 800可以运行用于生成第一AR游戏环境520的应用，并向VR设备710提供用于使第一AR游戏环境520呈现的相应数据，以及检测用户502(在握持着HIPD 800时)的移动，以使得在第一AR游戏环境520内执行相应的动作。附加地或替代地，在一些实施例中，向单个设备(例如，HIPD 800)提供一个或多个设备的操作数据(例如，传感器数据、图像数据、应用数据、设备数据和/或其它数据)，以对该操作数据进行处理，并使得相应设备执行与经处理的操作数据相关联的动作。Although the wrist wearable device 600, VR device 710, and/or HIPD 800 are described as detecting user input, in some embodiments, the user input is detected at a single device (where the single device is responsible for distributing the signal to other devices for performing the user input). For example, the HIPD 800 can run an application for generating the first AR game environment 520, and provide the VR device 710 with corresponding data for rendering the first AR game environment 520, and detect the movement of the user 502 (when holding the HIPD 800) so that corresponding actions are performed within the first AR game environment 520. Additionally or alternatively, in some embodiments, operational data (e.g., sensor data, image data, application data, device data, and/or other data) of one or more devices is provided to a single device (e.g., HIPD 800) to process the operational data and cause the corresponding device to perform an action associated with the processed operational data.
在图5D-1和图5D-2中，显示了佩戴着腕部可穿戴设备600、VR设备710和基于智能纺织品的服饰900的用户502。在第四AR系统500d中，腕部可穿戴设备600、VR设备710和/或基于智能纺织品的服饰900用于在AR环境(例如，以上参考图5A至图5C-2以及[包括任何相关的附图]所描述的任何AR系统)内交互。尽管VR设备710向用户502呈现VR游戏(例如，第二AR游戏环境535)的表示，但是腕部可穿戴设备600、VR设备710和/或基于智能纺织品的服饰900对一个或多个用户输入进行检测和协调，以允许用户502与AR环境交互。In FIGS. 5D-1 and 5D-2 , a user 502 wearing a wrist wearable device 600, a VR device 710, and a smart textile-based apparel 900 is shown. In a fourth AR system 500d, the wrist wearable device 600, the VR device 710, and/or the smart textile-based apparel 900 are used to interact within an AR environment (e.g., any AR system described above with reference to FIGS. 5A to 5C-2 and [including any related figures]). Although the VR device 710 presents a representation of a VR game (e.g., a second AR game environment 535) to the user 502, the wrist wearable device 600, the VR device 710, and/or the smart textile-based apparel 900 detect and coordinate one or more user inputs to allow the user 502 to interact with the AR environment.
在一些实施例中，用户502可以经由腕部可穿戴设备600、VR设备710和/或基于智能纺织品的服饰900提供在相应的AR环境中引起动作的用户输入。例如，第四AR系统500d中的用户502(如图5D-1所示)举起穿着基于智能纺织品的服饰900的手，以准备在第二AR游戏环境535内施展魔法或投掷对象。VR设备710响应于用户502举起其(穿着基于智能纺织品的服饰900的)手，使得用户的AR表示522执行类似的动作(例如，握持虚拟对象或投掷火球534)。在一些实施例中，每个设备使用相应的传感器数据和/或图像数据来检测用户输入并提供用户502的运动的准确表示。In some embodiments, the user 502 can provide user input that causes an action in the corresponding AR environment via the wrist wearable device 600, the VR device 710, and/or the smart textile-based clothing 900. For example, the user 502 in the fourth AR system 500d (as shown in FIG. 5D-1) raises his hand wearing the smart textile-based clothing 900 to prepare to cast a magic or throw an object in the second AR game environment 535. The VR device 710 responds to the user 502 raising his hand (wearing the smart textile-based clothing 900) so that the user's AR representation 522 performs a similar action (e.g., holding a virtual object or throwing a fireball 534). In some embodiments, each device uses corresponding sensor data and/or image data to detect user input and provide an accurate representation of the user's 502 movement.
在图5D-2中，用户502在穿着基于智能纺织品的服饰900时执行投掷动作。腕部可穿戴设备600、VR设备710和/或基于智能纺织品的服饰900检测到用户502的投掷动作，并且在第二AR游戏环境535中执行相应的动作。如上所述，由每个设备采集到的数据用于提升用户在AR环境内的体验。尽管未示出，但是基于智能纺织品的服饰900可以与VR设备710和/或HIPD 800结合使用。In FIG. 5D-2 , user 502 performs a throwing action while wearing smart textile-based apparel 900. The wrist wearable device 600, the VR device 710, and/or the smart textile-based apparel 900 detect the throwing action of user 502 and perform a corresponding action in the second AR game environment 535. As described above, the data collected by each device is used to enhance the user's experience within the AR environment. Although not shown, the smart textile-based apparel 900 can be used in conjunction with the VR device 710 and/or the HIPD 800.
已经较概括地论述了示例AR系统、用于与这种AR系统交互的设备以及其它计算系统，现在将在下文更详细地论述各设备和各部件。为了便于参考，此处限定了如下设备和部件的一些定义：这些设备和部件可以被包括在以下所论述的多个示例设备中的一些或全部示例设备中。本领域技术人员将理解的是，以下所描述的某些类型的部件可能更适合于特定的一组设备，而不太适合于不同的一组设备。但是，后续对此处所限定的部件的引用应视为由所提供的定义涵盖。Having discussed example AR systems, devices for interacting with such AR systems, and other computing systems in a relatively general manner, each device and each component will now be discussed in more detail below. For ease of reference, some definitions of the following devices and components are defined herein: these devices and components may be included in some or all of the multiple example devices discussed below. It will be understood by those skilled in the art that certain types of components described below may be more suitable for a particular set of devices and less suitable for a different set of devices. However, subsequent references to the components defined herein should be deemed to be covered by the definitions provided.
在以下所论述的一些实施例中，将论述包括电子设备和系统的多个示例设备和系统。这种示例设备和系统不旨在进行限制，并且本领域技术人员将理解的是，可以使用本文所描述的示例设备和系统的替代设备和系统来执行本文所描述的操作并构造本文所描述的系统和设备。In some embodiments discussed below, multiple example devices and systems including electronic devices and systems will be discussed. Such example devices and systems are not intended to be limiting, and those skilled in the art will appreciate that alternative devices and systems to the example devices and systems described herein may be used to perform the operations described herein and to construct the systems and devices described herein.
如本文所述，电子设备是使用电能来执行特定功能的设备。该电子设备可以是包含电子部件(例如晶体管、电阻器、电容器、二极管和集成电路)的任何物理对象。电子设备的示例包括智能手机、膝上型计算机、数码相机、电视、游戏控制台、和音乐播放器、以及本文所论述的各示例电子设备。如本文所述，中间电子设备是这样的设备：该设备位于两个其它电子设备之间和/或一个或多个电子设备的多个部件的子集之间，该设备促进相应的电子设备和/或电子部件之间的通信、和/或数据处理、和/或数据传输。As described herein, an electronic device is a device that uses electrical energy to perform a specific function. The electronic device can be any physical object containing electronic components (e.g., transistors, resistors, capacitors, diodes, and integrated circuits). Examples of electronic devices include smart phones, laptop computers, digital cameras, televisions, game consoles, and music players, as well as each of the example electronic devices discussed herein. As described herein, an intermediate electronic device is a device that is located between two other electronic devices and/or between a subset of multiple components of one or more electronic devices, and that facilitates communication, and/or data processing, and/or data transmission between corresponding electronic devices and/or electronic components.
示例腕部可穿戴设备Example wrist wearable device
图6A和图6B示出了根据一些实施例的示例腕部可穿戴设备600。腕部可穿戴设备600是参考本文中图2A至图3G描述的腕部可穿戴设备202的实例，使得该腕部可穿戴设备应被理解为具有腕部可穿戴设备600的特征，并且反之亦然。图6A示出了腕部可穿戴设备600的多个部件，这些部件可以单独使用或组合使用，其包括这样的组合：所述组合包括其它电子设备和/或电子部件。6A and 6B illustrate an example wrist wearable device 600 according to some embodiments. Wrist wearable device 600 is an example of wrist wearable device 202 described with reference to FIGS. 2A to 3G herein, such that the wrist wearable device should be understood to have features of wrist wearable device 600, and vice versa. FIG. 6A illustrates multiple components of wrist wearable device 600, which can be used alone or in combination, including combinations that include other electronic devices and/or electronic components.
如以下所论述的，图6A显示了可穿戴带610和手表主体620(或舱(capsule))耦接，以形成腕部可穿戴设备600。腕部可穿戴设备600可以执行与在用户界面中导航和选择性地开启应用相关联的各种功能和/或操作、以及以上参考图2A至图3G描述的功能和/或操作。As discussed below, FIG6A shows a wearable band 610 and a watch body 620 (or capsule) coupled to form a wrist wearable device 600. The wrist wearable device 600 can perform various functions and/or operations associated with navigating in a user interface and selectively launching applications, as well as the functions and/or operations described above with reference to FIGS. 2A to 3G .
如以下将更详细地描述的，由腕部可穿戴设备600执行的操作可以包括：(i)向用户呈现内容(例如，经由显示器605显示视觉内容)；(ii)检测(例如，感测)用户输入(例如，感测在外围按钮623上的触摸和/或在显示器605的触摸屏处的触摸、感测由传感器(例如，生物电势传感器)检测到的手势)；(iii)经由一个或多个传感器613感测生物特征数据(例如，神经肌肉信号、心率、温度或睡眠)；消息传递(例如，文本、语音或视频)；经由一个或多个成像设备或摄像头625的图像采集；无线通信(例如，蜂窝、近场、Wi-Fi或个人局域网)；位置确定；金融交易；提供触觉反馈；告警；通知；生物特征认证；健康监测；和/或睡眠监测。As will be described in more detail below, operations performed by wrist wearable device 600 may include: (i) presenting content to a user (e.g., displaying visual content via display 605); (ii) detecting (e.g., sensing) user input (e.g., sensing a touch on peripheral button 623 and/or a touch at a touch screen of display 605, sensing a gesture detected by a sensor (e.g., a biopotential sensor)); (iii) sensing biometric data (e.g., neuromuscular signals, heart rate, temperature, or sleep) via one or more sensors 613; messaging (e.g., text, voice, or video); image acquisition via one or more imaging devices or cameras 625; wireless communications (e.g., cellular, near-field, Wi-Fi, or personal area network); location determination; financial transactions; providing tactile feedback; alarms; notifications; biometric authentication; health monitoring; and/or sleep monitoring.
以上示例功能可以在手表主体620中独立地执行、在可穿戴带610中独立地执行和/或经由手表主体620与可穿戴带610之间的电子通信来执行。在一些实施例中，在(例如，经由AR系统500a至500d中的一个AR系统)呈现AR环境时，可以在腕部可穿戴设备600上执行各功能。如本领域技术人员在阅读本文所提供的描述时将理解的，本文所描述的新型可穿戴设备可以与其它类型的AR环境一起使用。The above example functions may be performed independently in the watch body 620, independently in the wearable band 610, and/or via electronic communication between the watch body 620 and the wearable band 610. In some embodiments, when an AR environment is presented (e.g., via one of the AR systems 500a to 500d), the functions may be performed on the wrist wearable device 600. As will be understood by those skilled in the art upon reading the description provided herein, the novel wearable device described herein may be used with other types of AR environments.
可穿戴带610可以被配置为由用户佩戴，使得可穿戴带610的可穿戴结构611的内(或内侧)表面与用户的皮肤接触。传感器613在被用户佩戴着时，接触用户的皮肤。传感器613可以感测生物特征数据，例如用户的心率、饱和氧水平、温度、汗液水平、神经肌肉信号或它们的组合。传感器613还可以感测关于用户的环境的数据，包括用户的运动、高度、地点、方位、步态、加速度、位置或它们的组合。在一些实施例中，传感器613被配置为追踪可穿戴带610的位置和/或运动。一个或多个传感器613可以包括以上所限定的和/或以下关于图6B所论述的多个传感器中的任何传感器。The wearable band 610 can be configured to be worn by the user so that the inner (or inner side) surface of the wearable structure 611 of the wearable band 610 contacts the user's skin. The sensor 613 contacts the user's skin when worn by the user. The sensor 613 can sense biometric data, such as the user's heart rate, saturated oxygen level, temperature, sweat level, neuromuscular signal, or a combination thereof. The sensor 613 can also sense data about the user's environment, including the user's movement, height, location, orientation, gait, acceleration, position, or a combination thereof. In some embodiments, the sensor 613 is configured to track the position and/or movement of the wearable band 610. One or more sensors 613 may include any of the multiple sensors defined above and/or discussed below with respect to FIG. 6B.
一个或多个传感器613可以分布在可穿戴带610的内表面和/或外表面上。在一些实施例中，一个或多个传感器613沿着可穿戴带610均匀间隔开。替代地，在一些实施例中，一个或多个传感器613位于沿可穿戴带610的不同点处。如图6A所示，一个或多个传感器613可以是相同的或不同的。例如，在一些实施例中，一个或多个传感器613可以被成形为药丸状(例如，传感器613a)、卵形、圆形、正方形、椭圆形(例如，传感器613c)和/或维持与用户的皮肤接触的任何其它形状(例如，使得可以在用户的皮肤处准确地测量神经肌肉信号和/或其它生物特征数据)。在一些实施例中，一个或多个传感器613被对齐以形成传感器对(例如，用于基于每个相应传感器内的差分感测来感测神经肌肉信号)。例如，传感器613b与相邻传感器对齐以形成传感器对614a，并且传感器613d与相邻传感器对齐以形成传感器对614b。在一些实施例中，可穿戴带610不具有传感器对。替代地，在一些实施例中，可穿戴带610具有预定数量的传感器对(一个传感器对、三个传感器对、四个传感器对、六个传感器对或十六个传感器对)。One or more sensors 613 may be distributed on the inner and/or outer surfaces of the wearable band 610. In some embodiments, one or more sensors 613 are evenly spaced along the wearable band 610. Alternatively, in some embodiments, one or more sensors 613 are located at different points along the wearable band 610. As shown in FIG. 6A, one or more sensors 613 may be the same or different. For example, in some embodiments, one or more sensors 613 may be shaped as a pill (e.g., sensor 613a), an oval, a circle, a square, an ellipse (e.g., sensor 613c) and/or any other shape that maintains contact with the user's skin (e.g., so that neuromuscular signals and/or other biometric data can be accurately measured at the user's skin). In some embodiments, one or more sensors 613 are aligned to form a sensor pair (e.g., for sensing neuromuscular signals based on differential sensing within each corresponding sensor). For example, sensor 613b is aligned with an adjacent sensor to form a sensor pair 614a, and sensor 613d is aligned with an adjacent sensor to form a sensor pair 614b. In some embodiments, the wearable band 610 does not have a sensor pair. Alternatively, in some embodiments, the wearable band 610 has a predetermined number of sensor pairs (one sensor pair, three sensor pairs, four sensor pairs, six sensor pairs, or sixteen sensor pairs).
可穿戴带610可以包括任何合适数量的传感器613。在一些实施例中，传感器613的数量和布置取决于使用可穿戴带610的特定应用。例如，被配置作为臂带、腕带或胸带的可穿戴带610可以包括多个传感器613，与游戏用例或一般日常用例相比，每个用例(例如医疗用例)具有不同数量的传感器613和不同的布置。Wearable band 610 may include any suitable number of sensors 613. In some embodiments, the number and arrangement of sensors 613 depends on the specific application in which wearable band 610 is used. For example, a wearable band 610 configured as an armband, wristband, or chest band may include multiple sensors 613, with each use case (e.g., a medical use case) having a different number of sensors 613 and a different arrangement than a gaming use case or a general everyday use case.
根据一些实施例，可穿戴带610还包括电接地电极和屏蔽电极。与传感器613类似，电接地电极和屏蔽电极可以分布在可穿戴带610的内表面上，使得它们接触用户的皮肤的一部分。例如，电接地电极和屏蔽电极可以位于耦接机构616的内表面处或可穿戴结构611的内表面处。电接地电极和屏蔽电极可以形成和/或使用与传感器613相同的部件。在一些实施例中，可穿戴带610包括多于一个电接地电极和多于一个屏蔽电极。According to some embodiments, the wearable band 610 also includes an electrical ground electrode and a shield electrode. Similar to the sensor 613, the electrical ground electrode and the shield electrode can be distributed on the inner surface of the wearable band 610 so that they contact a portion of the user's skin. For example, the electrical ground electrode and the shield electrode can be located at the inner surface of the coupling mechanism 616 or at the inner surface of the wearable structure 611. The electrical ground electrode and the shield electrode can form and/or use the same components as the sensor 613. In some embodiments, the wearable band 610 includes more than one electrical ground electrode and more than one shield electrode.
传感器613可以形成为可穿戴带610的可穿戴结构611的一部分。在一些实施例中，传感器613与可穿戴结构611平齐或大致平齐，使得这些传感器不会延伸超出可穿戴结构611的表面。尽管与可穿戴结构611平齐，但传感器613仍被配置为(例如，经由皮肤接触表面)接触用户的皮肤。替代地，在一些实施例中，传感器613延伸超出可穿戴结构611预定距离(例如，0.1毫米(mm)至2mm)，以接触并压入用户的皮肤。在一些实施例中，传感器613耦接到致动器(未示出)，该致动器被配置为调整传感器613的延伸高度(例如，距可穿戴结构611的表面的距离)，使得传感器613接触并压入用户的皮肤。在一些实施例中，致动器在0.01mm至1.2mm之间调节延伸高度。这允许用户定制传感器613的位置，以提高可穿戴带610在被佩戴时的整体舒适性，同时仍允许传感器613接触用户的皮肤。在一些实施例中，传感器613在被用户佩戴着时不与可穿戴结构611区分。The sensor 613 can be formed as part of the wearable structure 611 of the wearable band 610. In some embodiments, the sensor 613 is flush or substantially flush with the wearable structure 611 so that the sensors do not extend beyond the surface of the wearable structure 611. Despite being flush with the wearable structure 611, the sensor 613 is still configured to contact the user's skin (e.g., via a skin contact surface). Alternatively, in some embodiments, the sensor 613 extends beyond the wearable structure 611 by a predetermined distance (e.g., 0.1 millimeters (mm) to 2 mm) to contact and press into the user's skin. In some embodiments, the sensor 613 is coupled to an actuator (not shown) that is configured to adjust the extension height of the sensor 613 (e.g., the distance from the surface of the wearable structure 611) so that the sensor 613 contacts and presses into the user's skin. In some embodiments, the actuator adjusts the extension height between 0.01 mm and 1.2 mm. This allows the user to customize the position of the sensor 613 to improve the overall comfort of the wearable band 610 when being worn, while still allowing the sensor 613 to contact the user's skin. In some embodiments, the sensor 613 is not distinguishable from the wearable structure 611 when being worn by the user.
可穿戴结构611可以由弹性材料、弹性体等形成，该弹性材料、该弹性体等被配置为被拉伸和适于被用户佩戴。在一些实施例中，可穿戴结构611是纺织品或机织布。如上所述，传感器613可以形成为可穿戴结构611的一部分。例如，传感器613可以被模制到可穿戴结构611中或集成到机织布中(例如，传感器613可以被缝进织物中并模拟织物的柔韧性(例如，传感器613可以由一系列编织的织物线构成))。The wearable structure 611 can be formed of an elastic material, an elastomer, etc., which is configured to be stretched and suitable for being worn by a user. In some embodiments, the wearable structure 611 is a textile or a woven fabric. As described above, the sensor 613 can be formed as a part of the wearable structure 611. For example, the sensor 613 can be molded into the wearable structure 611 or integrated into the woven fabric (for example, the sensor 613 can be sewn into the fabric and simulate the flexibility of the fabric (for example, the sensor 613 can be composed of a series of woven fabric threads)).
可穿戴结构611可以包括将被包括在可穿戴带610中的传感器613、电子电路和/或其它电子部件(以下参考图6B进行了描述)互连的柔性电子连接器。在一些实施例中，柔性电子连接器被配置为将可穿戴带610的传感器613、电子电路和/或其它电子部件与另一电子设备(例如，手表主体620)的相应传感器和/或其它电子部件互连。柔性电子连接器被配置为与可穿戴结构611一起移动，使得用户对可穿戴结构611的调整(例如，调整大小、拉动或折叠)不会对可穿戴带610的各部件的电耦接产生应力或应变。The wearable structure 611 may include a flexible electronic connector that interconnects the sensors 613, electronic circuits, and/or other electronic components (described below with reference to FIG. 6B ) included in the wearable band 610. In some embodiments, the flexible electronic connector is configured to interconnect the sensors 613, electronic circuits, and/or other electronic components of the wearable band 610 with corresponding sensors and/or other electronic components of another electronic device (e.g., a watch body 620). The flexible electronic connector is configured to move with the wearable structure 611 so that adjustments (e.g., resizing, pulling, or folding) of the wearable structure 611 by the user do not cause stress or strain on the electrical coupling of the components of the wearable band 610.
如上所述，可穿戴带610被配置为由用户佩戴。具体地，可穿戴带610可以被成形为或被以其它方式操纵为由用户佩戴。例如，可穿戴带610可以被成形为具有大致圆形的形状，使得该可穿戴带可以被配置为被佩戴在用户的下臂或腕部上。替代地，可穿戴带610可以被成形为被佩戴在用户的另一身体部位(例如用户的上臂(例如，二头肌周围)、前臂、胸部、腿部等)上。可穿戴带610可以包括保持机构612(例如，搭扣或钩环紧固件)，该保持机构用于将可穿戴带610固定到用户的腕部或其它身体部位。在可穿戴带610被用户佩戴着时，传感器613感测来自用户的皮肤的数据(被称为传感器数据)。特别地，可穿戴带610的传感器613获取(例如，感测和记录)神经肌肉信号。As described above, the wearable band 610 is configured to be worn by a user. Specifically, the wearable band 610 can be shaped or otherwise manipulated to be worn by a user. For example, the wearable band 610 can be shaped to have a generally circular shape so that the wearable band can be configured to be worn on the user's lower arm or wrist. Alternatively, the wearable band 610 can be shaped to be worn on another body part of the user (e.g., the user's upper arm (e.g., around the biceps), forearm, chest, leg, etc.). The wearable band 610 may include a retaining mechanism 612 (e.g., a buckle or hook-and-loop fastener) for securing the wearable band 610 to the user's wrist or other body part. When the wearable band 610 is worn by the user, the sensor 613 senses data from the user's skin (referred to as sensor data). In particular, the sensor 613 of the wearable band 610 acquires (e.g., senses and records) neuromuscular signals.
所感测到的数据(例如，所感测到的神经肌肉信号)可以用于检测和/或确定用户执行某些运动动作的意图。特别地，在用户执行肌肉激活(例如，运动或手势)时，传感器613感测并记录来自用户的神经肌肉信号。所检测到和/或所确定的运动动作(例如，指骨(或手指)运动、腕部运动、手部运动和/或其它肌肉意图)可以用于确定用于使计算设备执行一个或多个输入命令的控制命令或控制信息(在感测到数据之后执行某些命令的指令)。例如，所感测到的神经肌肉信号可以用于控制显示在腕部可穿戴设备600的显示器605上的某些用户界面，和/或可被发送到负责渲染AR环境的设备(例如，头戴式显示器)以在相关联的AR环境中执行动作(例如以控制向用户显示的虚拟设备的运动)。用户所执行的肌肉激活可以包括：静态手势，例如将用户的手掌朝下放在桌子上；动态手势，例如抓住物理对象或虚拟对象；以及另一个人察觉不到的隐蔽手势，例如通过联合收缩相对的肌肉或使用肌下激活来轻微地绷紧关节。用户所执行的肌肉激活可以包括符号手势(例如，例如基于指定了手势到命令的映射的手势词汇表而映射到其它手势、交互或命令的手势)。The sensed data (e.g., the sensed neuromuscular signals) can be used to detect and/or determine the user's intention to perform certain motor actions. In particular, when the user performs muscle activation (e.g., movement or gesture), the sensor 613 senses and records the neuromuscular signals from the user. The detected and/or determined motor actions (e.g., phalanges (or fingers) movements, wrist movements, hand movements and/or other muscle intentions) can be used to determine control commands or control information (instructions for executing certain commands after sensing the data) for causing the computing device to execute one or more input commands. For example, the sensed neuromuscular signals can be used to control certain user interfaces displayed on the display 605 of the wrist wearable device 600, and/or can be sent to a device (e.g., head-mounted display) responsible for rendering the AR environment to perform actions in the associated AR environment (e.g., to control the movement of a virtual device displayed to the user). The muscle activations performed by the user may include static gestures, such as placing the user's palm down on a table; dynamic gestures, such as grasping a physical or virtual object; and covert gestures that are imperceptible to another person, such as slightly tensing a joint by co-contracting opposing muscles or using submuscular activation. The muscle activations performed by the user may include symbolic gestures (e.g., gestures that are mapped to other gestures, interactions, or commands, such as based on a gesture vocabulary that specifies a mapping of gestures to commands).
传感器613所感测到的传感器数据可以用于向用户提供与物理对象(例如，与可穿戴带610通信耦接的设备)和/或由AR系统生成的、AR应用中的虚拟对象(例如，在显示器605或另一计算设备(例如，智能手机)上呈现的用户界面对象)的增强交互。The sensor data sensed by sensor 613 can be used to provide the user with enhanced interaction with physical objects (e.g., devices communicatively coupled to wearable band 610) and/or virtual objects in an AR application generated by the AR system (e.g., user interface objects presented on display 605 or another computing device (e.g., a smartphone)).
在一些实施例中，可穿戴带610包括一个或多个触觉设备646(图6B；例如，振动触觉致动器)，该一个或多个触觉设备被配置为向用户的皮肤提供触觉反馈(例如，皮肤感觉和/或动觉)。传感器613和/或触觉设备646可以被配置为结合多个应用来运行，该多个应用包括但不限于健康监测、社交媒体、游戏和AR(例如，与AR相关联的应用)。In some embodiments, the wearable band 610 includes one or more haptic devices 646 (FIG. 6B; e.g., vibrotactile actuators) configured to provide tactile feedback (e.g., skin sensation and/or kinesthetic sensation) to the user's skin. The sensor 613 and/or the haptic device 646 may be configured to operate in conjunction with a plurality of applications, including but not limited to health monitoring, social media, gaming, and AR (e.g., applications associated with AR).
可穿戴带610还可以包括耦接机构616(例如，耦接机构的托架或形状可以与腕部可穿戴设备600的手表主体620的形状相对应)，该耦接机构用于(经由手表主体620的耦接表面)将舱(例如，计算单元)或手表主体620可拆卸地耦接到可穿戴带610。特别地，耦接机构616可以被配置为接收手表主体620的靠近底侧(例如，与手表主体620的显示器605所在的前侧相对的一侧)的耦接表面，使得用户可以将手表主体620向下推入耦接机构616以将手表主体620附接到耦接机构616。在一些实施例中，耦接机构616可以被配置为接收手表主体620的顶侧(例如，靠近手表主体620的显示器605所在的前侧的一侧)，该手表主体被向上推入托架中，而不是被向下推入耦接机构616中。在一些实施例中，耦接机构616是可穿戴带610的集成部件，使得可穿戴带610和耦接机构616是单个统一结构。在一些实施例中，耦接机构616是一类框架或外壳，该类框架或外壳允许手表主体620的耦接表面保持在可穿戴带610的耦接机构616(例如，托架、追踪带、支撑底座或扣环)内、或保持在该耦接机构上。The wearable band 610 may also include a coupling mechanism 616 (e.g., a bracket or shape of the coupling mechanism may correspond to the shape of the watch body 620 of the wrist wearable device 600) for removably coupling the capsule (e.g., computing unit) or the watch body 620 to the wearable band 610 (via a coupling surface of the watch body 620). In particular, the coupling mechanism 616 may be configured to receive a coupling surface of the watch body 620 near a bottom side (e.g., a side opposite to the front side where the display 605 of the watch body 620 is located) so that a user can push the watch body 620 downward into the coupling mechanism 616 to attach the watch body 620 to the coupling mechanism 616. In some embodiments, the coupling mechanism 616 may be configured to receive a top side (e.g., a side near the front side where the display 605 of the watch body 620 is located) of the watch body 620, which is pushed upward into the bracket instead of being pushed downward into the coupling mechanism 616. In some embodiments, coupling mechanism 616 is an integrated component of wearable band 610, such that wearable band 610 and coupling mechanism 616 are a single unified structure. In some embodiments, coupling mechanism 616 is a type of frame or housing that allows the coupling surface of watch body 620 to be retained within or on coupling mechanism 616 (e.g., a cradle, tracking band, support base, or buckle) of wearable band 610.
耦接机构616可以允许手表主体620通过以下项可拆卸地耦接到可穿戴带610：摩擦配合、磁耦接、基于旋转的连接器、剪刀销联结器、保持弹簧、一个或多个磁体、夹子、销轴、钩环紧固件、或它们的组合。用户可以执行任何类型的动作，以将手表主体620耦接到可穿戴带610以及将手表主体620与可穿戴带610断开耦接。例如，用户可以相对于可穿戴带610扭转、滑动、转动、推、拉或旋转(或者它们的组合)手表主体620，以将手表主体620附接到可穿戴带610以及从可穿戴带610拆卸手表主体620。替代地，如以下所论述的，在一些实施例中，可以通过释放机构629的致动来将手表主体620与可穿戴带610断开耦接。The coupling mechanism 616 can allow the watch body 620 to be removably coupled to the wearable band 610 by: friction fit, magnetic coupling, a rotation-based connector, a scissor pin coupling, a retaining spring, one or more magnets, a clip, a pin, a hook and loop fastener, or a combination thereof. The user can perform any type of action to couple the watch body 620 to the wearable band 610 and decouple the watch body 620 from the wearable band 610. For example, the user can twist, slide, turn, push, pull, or rotate (or a combination thereof) the watch body 620 relative to the wearable band 610 to attach the watch body 620 to the wearable band 610 and detach the watch body 620 from the wearable band 610. Alternatively, as discussed below, in some embodiments, the watch body 620 can be decoupled from the wearable band 610 by actuation of the release mechanism 629.
可穿戴带610可以与手表主体620耦接，以增加可穿戴带610的功能(例如，将可穿戴带610转换为腕部可穿戴设备600、添加附加计算单元和/或电池以增加可穿戴带610的计算资源和/或电池寿命、或添加附加传感器以改进感测到的数据)。如上所述，可穿戴带610(和耦接机构616)被配置为独立于手表主体620运行(例如，独立于该手表主体执行功能)。例如，耦接机构616可以包括一个或多个传感器613，该一个或多个传感器在用户佩戴着可穿戴带610时接触用户的皮肤，并提供用于确定控制命令的传感器数据。The wearable band 610 can be coupled to the watch body 620 to increase the functionality of the wearable band 610 (e.g., converting the wearable band 610 into a wrist wearable device 600, adding an additional computing unit and/or battery to increase the computing resources and/or battery life of the wearable band 610, or adding additional sensors to improve the sensed data). As described above, the wearable band 610 (and the coupling mechanism 616) are configured to operate independently of the watch body 620 (e.g., perform functions independently of the watch body). For example, the coupling mechanism 616 may include one or more sensors 613 that contact the user's skin when the user wears the wearable band 610 and provide sensor data for determining control commands.
用户可以从可穿戴带610拆卸手表主体620(或舱)，以便减少腕部可穿戴设备600对用户的负担。对于手表主体620可拆卸的实施例，手表主体620可以被称为可拆卸结构，使得在这些实施例中，腕部可穿戴设备600包括可穿戴部分(例如，可穿戴带610)和可拆卸结构(手表主体620)。The user can remove the watch body 620 (or the capsule) from the wearable band 610 to reduce the burden on the user of the wrist wearable device 600. For embodiments in which the watch body 620 is detachable, the watch body 620 can be referred to as a detachable structure, so that in these embodiments, the wrist wearable device 600 includes a wearable portion (e.g., the wearable band 610) and a detachable structure (the watch body 620).
转到手表主体620，手表主体620可以具有大致矩形或圆形的形状。手表主体620被配置为被用户佩戴在其腕部或另一身体部位上。更具体地，手表主体620被设置尺寸为容易被用户携带、容易附接在用户服饰的一部分上和/或容易耦接到可穿戴带610(从而形成腕部可穿戴设备600)。如上所述，手表主体620可以具有与可穿戴带610的耦接机构616相对应的形状。在一些实施例中，手表主体620包括单个释放机构629或多个释放机构(例如，被定位在手表主体620的相对两侧上的两个释放机构629，例如弹簧支承的按钮)，以将手表主体620与可穿戴带610断开耦接。释放机构629可以包括但不限于，按钮、旋钮、塞子、手柄、操纵杆、紧固件、扣环、拨盘、闩锁、或它们的组合。Turning to the watch body 620, the watch body 620 can have a generally rectangular or circular shape. The watch body 620 is configured to be worn by the user on his wrist or another body part. More specifically, the watch body 620 is sized to be easily carried by the user, easily attached to a portion of the user's clothing, and/or easily coupled to the wearable band 610 (thereby forming a wrist wearable device 600). As described above, the watch body 620 can have a shape corresponding to the coupling mechanism 616 of the wearable band 610. In some embodiments, the watch body 620 includes a single release mechanism 629 or multiple release mechanisms (e.g., two release mechanisms 629 positioned on opposite sides of the watch body 620, such as spring-loaded buttons) to disconnect the watch body 620 from the wearable band 610. The release mechanism 629 may include, but is not limited to, a button, a knob, a plug, a handle, a lever, a fastener, a buckle, a dial, a latch, or a combination thereof.
用户可以通过推动、转动、抬起、按下、移动释放机构629或者对该释放机构629执行其它动作，来致动释放机构629。对释放机构629的致动可以将手表主体620从可穿戴带610的耦接机构616释放(例如，断开耦接)，从而允许用户独立于可穿戴带610使用手表主体620，反之亦然。例如，将手表主体620从可穿戴带610断开耦接可以允许用户使用后置摄像头625b来采集图像。尽管耦接机构616被显示为定位在手表主体620的一角，但是释放机构629可以定位在手表主体620上便于用户致动的任何位置。另外，在一些实施例中，可穿戴带610还可以包括用于将手表主体620从耦接机构616断开耦接的相应释放机构。在一些实施例中，释放机构629是可选的，并且如上所述，手表主体620可以(例如，经由扭转或旋转)从耦接机构616断开耦接。The user can actuate the release mechanism 629 by pushing, turning, lifting, pressing, moving the release mechanism 629, or performing other actions on the release mechanism 629. The actuation of the release mechanism 629 can release the watch body 620 from the coupling mechanism 616 of the wearable band 610 (e.g., disconnect the coupling), thereby allowing the user to use the watch body 620 independently of the wearable band 610, and vice versa. For example, disconnecting the watch body 620 from the wearable band 610 can allow the user to use the rear camera 625b to capture images. Although the coupling mechanism 616 is shown as being positioned at a corner of the watch body 620, the release mechanism 629 can be positioned at any position on the watch body 620 that is convenient for the user to actuate. In addition, in some embodiments, the wearable band 610 may also include a corresponding release mechanism for disconnecting the watch body 620 from the coupling mechanism 616. In some embodiments, the release mechanism 629 is optional, and as described above, the watch body 620 can be disconnected from the coupling mechanism 616 (e.g., via twisting or rotation).
手表主体620可以包括用于在手表主体620处执行各种操作的一个或多个外围按钮623和627。例如，外围按钮623和627可以用于开启或唤醒显示器605(例如，使该显示器605从睡眠状态转换到活动状态)、解锁手表主体620、增大或减小音量、提高或降低亮度、与一个或多个应用交互、与一个或多个用户界面交互。附加地或替代地，在一些实施例中，显示器605充当触摸屏，并且允许用户提供用于与手表主体620交互的一个或多个输入。The watch body 620 may include one or more peripheral buttons 623 and 627 for performing various operations at the watch body 620. For example, the peripheral buttons 623 and 627 may be used to turn on or wake up the display 605 (e.g., to transition the display 605 from a sleep state to an active state), unlock the watch body 620, increase or decrease the volume, increase or decrease the brightness, interact with one or more applications, interact with one or more user interfaces. Additionally or alternatively, in some embodiments, the display 605 acts as a touch screen and allows a user to provide one or more inputs for interacting with the watch body 620.
在一些实施例中，手表主体620包括一个或多个传感器621。手表主体620的传感器621可以与可穿戴带610的传感器613相同或不同。手表主体620的传感器621可以分布在手表主体620的内表面和/或外表面上。在一些实施例中，传感器621被配置为在用户佩戴着手表主体620时接触用户的皮肤。例如，传感器621可以被放置在手表主体620的底侧上，并且耦接机构616可以是具有开口的托架，该开口允许手表主体620的底侧直接接触用户的皮肤。替代地，在一些实施例中，手表主体620不包括被配置为接触用户的皮肤的传感器(例如，包括手表主体620内部和/或外部的传感器，这些传感器被配置为感测手表主体620的数据和手表主体620的周围环境的数据)。在一些实施例中，传感器621被配置为追踪手表主体620的位置和/或运动。In some embodiments, the watch body 620 includes one or more sensors 621. The sensors 621 of the watch body 620 may be the same as or different from the sensors 613 of the wearable band 610. The sensors 621 of the watch body 620 may be distributed on the inner surface and/or outer surface of the watch body 620. In some embodiments, the sensor 621 is configured to contact the user's skin when the user wears the watch body 620. For example, the sensor 621 may be placed on the bottom side of the watch body 620, and the coupling mechanism 616 may be a bracket with an opening that allows the bottom side of the watch body 620 to directly contact the user's skin. Alternatively, in some embodiments, the watch body 620 does not include a sensor configured to contact the user's skin (e.g., including sensors inside and/or outside the watch body 620 that are configured to sense data of the watch body 620 and data of the surrounding environment of the watch body 620). In some embodiments, the sensor 621 is configured to track the position and/or movement of the watch body 620.
手表主体620和可穿戴带610可以使用有线通信方法(例如，通用异步接收发送设备(Universal Asynchronous Receiver/Transmitter，UART)或USB收发器)和/或无线通信方法(例如，近场通信或蓝牙)来共享数据。例如，手表主体620和可穿戴带610可以共享由传感器613和621感测到的数据、以及特定于应用和特定于设备(例如，活动的和/或可用的应用、输出设备(例如，显示器或扬声器)和/或输入设备(例如，触摸屏、传声器或成像传感器))的信息。The watch body 620 and the wearable band 610 can share data using a wired communication method (e.g., a Universal Asynchronous Receiver/Transmitter (UART) or a USB transceiver) and/or a wireless communication method (e.g., near field communication or Bluetooth). For example, the watch body 620 and the wearable band 610 can share data sensed by sensors 613 and 621, as well as application-specific and device-specific information (e.g., active and/or available applications, output devices (e.g., displays or speakers), and/or input devices (e.g., touch screens, microphones, or imaging sensors).
在一些实施例中，手表主体620可以包括但不限于，前置摄像头625a和/或后置摄像头625b、传感器621(例如，生物特征传感器、IMU传感器、心率传感器、饱和氧传感器、神经肌肉信号传感器、高度计传感器、温度传感器、生物阻抗传感器、计步器传感器、光学传感器(例如，图6B；成像传感器663)、触摸传感器、汗液传感器)。在一些实施例中，手表主体620可以包括一个或多个触觉设备676(图6B；振动触觉致动器)，该一个或多个触觉设备被配置为向用户提供触觉反馈(例如，皮肤感觉和/或动觉)。传感器621和/或触觉设备676还可以被配置为结合多个应用运行，该多个应用包括但不限于健康监测应用、社交媒体应用、游戏应用和AR应用(例如，与AR相关联的应用)。In some embodiments, the watch body 620 may include, but is not limited to, a front camera 625a and/or a rear camera 625b, a sensor 621 (e.g., a biometric sensor, an IMU sensor, a heart rate sensor, a saturated oxygen sensor, a neuromuscular signal sensor, an altimeter sensor, a temperature sensor, a bioimpedance sensor, a pedometer sensor, an optical sensor (e.g., FIG. 6B ; imaging sensor 663 ), a touch sensor, a sweat sensor). In some embodiments, the watch body 620 may include one or more tactile devices 676 ( FIG. 6B ; a vibration tactile actuator) configured to provide tactile feedback (e.g., skin sensation and/or kinesthetic sensation) to the user. The sensor 621 and/or the tactile device 676 may also be configured to run in conjunction with multiple applications, including but not limited to health monitoring applications, social media applications, gaming applications, and AR applications (e.g., applications associated with AR).
如上所述，手表主体620和可穿戴带610在耦接时可以形成腕部可穿戴设备600。手表主体620和可穿戴带610在耦接时充当单个设备以执行本文所描述的各功能(例如，操作、检测或通信)。在一些实施例中，每个设备被提供有用于执行腕部可穿戴设备600的一个或多个操作的特定指令。例如，根据确定手表主体620不包括神经肌肉信号传感器，可穿戴带610可以包括用于执行相关联指令的替代指令(例如，经由不同的电子设备向手表主体620提供感测到的神经肌肉信号数据)。腕部可穿戴设备600的各操作可以由手表主体620单独执行或由该手表主体与可穿戴带610(例如，经由相应处理器和/或硬件部件)结合执行，反之亦然。在一些实施例中，腕部可穿戴设备600、手表主体620和/或可穿戴带610的各操作可以与另一通信耦接的设备(例如，图8A和图8B；HIPD 800)的一个或多个处理器和/或硬件部件结合执行。As described above, the watch body 620 and the wearable band 610 can form a wrist wearable device 600 when coupled. The watch body 620 and the wearable band 610 act as a single device when coupled to perform the functions described herein (e.g., operation, detection, or communication). In some embodiments, each device is provided with specific instructions for performing one or more operations of the wrist wearable device 600. For example, based on determining that the watch body 620 does not include a neuromuscular signal sensor, the wearable band 610 may include an alternative instruction for executing the associated instruction (e.g., providing the sensed neuromuscular signal data to the watch body 620 via a different electronic device). The various operations of the wrist wearable device 600 can be performed by the watch body 620 alone or by the watch body in combination with the wearable band 610 (e.g., via corresponding processors and/or hardware components), and vice versa. In some embodiments, various operations of the wrist wearable device 600, the watch body 620 and/or the wearable band 610 may be performed in conjunction with one or more processors and/or hardware components of another communicatively coupled device (e.g., FIGS. 8A and 8B; HIPD 800).
如以下参考图6B的框图所描述的，可穿戴带610和/或手表主体620可以各自包括独立地执行功能所需的独立资源。例如，可穿戴带610和/或手表主体620可以各自包括电源(例如，电池)、存储器、数据存储器、处理器(例如，CPU)、通信、光源、和/或输入/输出设备。As described below with reference to the block diagram of Figure 6B, the wearable band 610 and/or the watch body 620 may each include independent resources required to independently perform functions. For example, the wearable band 610 and/or the watch body 620 may each include a power source (e.g., a battery), a memory, a data storage, a processor (e.g., a CPU), communications, a light source, and/or an input/output device.
图6B示出了根据一些实施例的与可穿戴带610相对应的计算系统630、以及与手表主体620相对应的计算系统660的框图。根据一些实施例，腕部可穿戴设备600的计算系统包括可穿戴带计算系统630的部件与手表主体计算系统660的部件的组合。6B shows a block diagram of a computing system 630 corresponding to the wearable band 610 and a computing system 660 corresponding to the watch body 620, according to some embodiments. According to some embodiments, the computing system of the wrist wearable device 600 includes a combination of components of the wearable band computing system 630 and components of the watch body computing system 660.
手表主体620和/或可穿戴带610可以包括手表主体计算系统660中所示出的一个或多个部件。在一些实施例中，单个集成电路包括手表主体计算系统660的所有部件或大部分部件，其包括在单个集成电路中。替代地，在一些实施例中，手表主体计算系统660的各部件被包括在通信耦接的多个集成电路中。在一些实施例中，手表主体计算系统660被配置为(例如，经由有线连接或无线连接)与可穿戴带计算系统630耦接，这允许这两个计算系统共享部件、分发任务和/或(单独地或作为单个设备)执行本文所描述的其它操作。The watch body 620 and/or the wearable band 610 may include one or more components shown in the watch body computing system 660. In some embodiments, a single integrated circuit includes all or most of the components of the watch body computing system 660, which are included in a single integrated circuit. Alternatively, in some embodiments, the components of the watch body computing system 660 are included in multiple integrated circuits that are communicatively coupled. In some embodiments, the watch body computing system 660 is configured to be coupled to the wearable band computing system 630 (e.g., via a wired connection or a wireless connection), which allows the two computing systems to share components, distribute tasks, and/or perform other operations described herein (separately or as a single device).
手表主体的计算系统660可以包括一个或多个处理器679、控制器677、外围接口661、电力系统695和存储器(例如，存储器680)，以上中的每一者都在上文进行了限定并且在以下进行了更详细地描述。The computing system 660 of the watch body may include one or more processors 679, a controller 677, a peripheral interface 661, a power system 695, and a memory (e.g., memory 680), each of which is defined above and described in more detail below.
电力系统695可以包括充电器输入696、电力管理集成电路(power-managementintegrated circuit，PMIC)697和电池698，以上中的每一者都在上文进行了限定。在一些实施例中，手表主体620和可穿戴带610可以具有各自的充电器输入(例如，充电器输入696和657)、各自的电池(例如，电池698和659)，并且可以彼此共享电力(例如，手表主体620可以给可穿戴带610供电和/或充电，反之亦然)。尽管手表主体620和/或可穿戴带610可以包括各自的充电器输入，但是在这两个设备耦接时，单个充电器输入可以对这两个设备充电。手表主体620和可穿戴带610可以使用各种技术来接收电荷。在一些实施例中，手表主体620和可穿戴带610可以使用有线充电组件(例如，电源线)来接收电荷。替代地或附加地，手表主体620和/或可穿戴带610可以被配置用于无线充电。例如，便携式充电设备可以被设计为与手表主体620的一部分和/或可穿戴带610的一部分匹配，并向手表主体620的电池和/或可穿戴带610的电池无线地传送可用电力。手表主体620和可穿戴带610可以具有独立的电力系统(例如，电力系统695和656)，以使各自能够独立运行。手表主体620和可穿戴带610还可以经由各自的PMIC(例如，PMIC 697和658)共享电力(例如，一者可以为另一者充电)，这些PMIC可以通过电力导体和接地导体和/或通过无线充电天线来共享电力。The power system 695 may include a charger input 696, a power management integrated circuit (PMIC) 697, and a battery 698, each of which is defined above. In some embodiments, the watch body 620 and the wearable band 610 may have respective charger inputs (e.g., charger inputs 696 and 657), respective batteries (e.g., batteries 698 and 659), and may share power with each other (e.g., the watch body 620 may power and/or charge the wearable band 610, and vice versa). Although the watch body 620 and/or the wearable band 610 may include respective charger inputs, a single charger input may charge the two devices when the two devices are coupled. The watch body 620 and the wearable band 610 may receive a charge using various techniques. In some embodiments, the watch body 620 and the wearable band 610 may receive a charge using a wired charging component (e.g., a power cord). Alternatively or additionally, the watch body 620 and/or the wearable band 610 may be configured for wireless charging. For example, the portable charging device can be designed to match a portion of the watch body 620 and/or a portion of the wearable band 610 and wirelessly transmit available power to the battery of the watch body 620 and/or the battery of the wearable band 610. The watch body 620 and the wearable band 610 can have independent power systems (e.g., power systems 695 and 656) to enable each to operate independently. The watch body 620 and the wearable band 610 can also share power (e.g., one can charge the other) via their respective PMICs (e.g., PMIC 697 and 658), which can share power through power conductors and ground conductors and/or through wireless charging antennas.
在一些实施例中，外围接口661可以包括一个或多个传感器621，以下列出的该一个或多个传感器中的许多传感器都在上文进行了限定。传感器621可以包括一个或多个耦接传感器662，该一个或多个耦接传感器用于检测手表主体620何时与另一电子设备(例如，可穿戴带610)耦接。传感器621可以包括成像传感器663(摄像头625和/或单独的成像传感器663(例如，热成像传感器)中的一者或多者)。在一些实施例中，传感器621包括一个或多个SpO2传感器664。在一些实施例中，传感器621包括一个或多个生物电势信号传感器(例如，EMG传感器665，该EMG传感器可以被设置在手表主体620和/或可穿戴带610的面向用户的部分上)。在一些实施例中，传感器621包括一个或多个电容式传感器666。在一些实施例中，传感器621包括一个或多个心率传感器667。在一些实施例中，传感器621包括一个或多个IMU 668。在一些实施例中，一个或多个IMU 668可以被配置为检测用户的手部的移动、或者手表主体620被放置或被握持的其它位置的移动)。In some embodiments, the peripheral interface 661 may include one or more sensors 621, many of which are defined above. The sensor 621 may include one or more coupling sensors 662, which are used to detect when the watch body 620 is coupled to another electronic device (e.g., the wearable band 610). The sensor 621 may include an imaging sensor 663 (one or more of the camera 625 and/or a separate imaging sensor 663 (e.g., a thermal imaging sensor)). In some embodiments, the sensor 621 includes one or more SpO2 sensors 664. In some embodiments, the sensor 621 includes one or more biopotential signal sensors (e.g., an EMG sensor 665, which may be disposed on a user-facing portion of the watch body 620 and/or the wearable band 610). In some embodiments, the sensor 621 includes one or more capacitive sensors 666. In some embodiments, the sensor 621 includes one or more heart rate sensors 667. In some embodiments, the sensor 621 includes one or more IMUs 668. In some embodiments, one or more IMUs 668 can be configured to detect movement of a user's hand, or movement of other locations where the watch body 620 is placed or held).
在一些实施例中，外围接口661包括NFC部件669、GPS部件670、长期演进(long-term evolution，LTE)部件671、和/或Wi-Fi和/或蓝牙通信部件672。在一些实施例中，外围接口661包括一个或多个按钮673(例如，图6A中的外围按钮623和627)，该一个或多个按钮在被用户选择时使得在手表主体620处执行操作。在一些实施例中，外围接口661包括一个或多个指示器(例如发光二极管(light-emitting diode，LED))，以向用户提供视觉指示器(例如，接收到消息、低电池电量、活动的传声器和/或摄像头)。In some embodiments, peripheral interface 661 includes NFC component 669, GPS component 670, long-term evolution (LTE) component 671, and/or Wi-Fi and/or Bluetooth communication component 672. In some embodiments, peripheral interface 661 includes one or more buttons 673 (e.g., peripheral buttons 623 and 627 in FIG. 6A ) that, when selected by the user, cause an operation to be performed at watch body 620. In some embodiments, peripheral interface 661 includes one or more indicators (e.g., light-emitting diodes (LEDs)) to provide visual indicators to the user (e.g., received messages, low battery, active microphone and/or camera).
手表主体620可以包括至少一个显示器605，以用于向用户显示对信息或数据的视觉表示，该视觉表示包括用户界面元素和/或三维(three-dimensional，3D)虚拟对象。显示器还可以包括用于输入用户输入(例如触摸手势和滑动手势等)的触摸屏。手表主体620可以包括至少一个扬声器674和至少一个传声器675，以用于向用户提供音频信号并接收来自用户的音频输入。用户可以通过传声器675提供用户输入，并且还可以接收来自扬声器674的音频输出，作为由触觉控制器678提供的触觉事件的一部分。手表主体620可以包括至少一个摄像头625，该至少一个摄像头625包括前置摄像头625a和后置摄像头625b。摄像头625可以包括超广角摄像头、广角摄像头、鱼眼摄像头、球面摄像头、长焦摄像头、深度感测摄像头或其它类型的摄像头。The watch body 620 may include at least one display 605 for displaying a visual representation of information or data to the user, the visual representation including user interface elements and/or three-dimensional (3D) virtual objects. The display may also include a touch screen for inputting user input (e.g., touch gestures and sliding gestures, etc.). The watch body 620 may include at least one speaker 674 and at least one microphone 675 for providing audio signals to the user and receiving audio input from the user. The user may provide user input through the microphone 675, and may also receive audio output from the speaker 674 as part of the tactile event provided by the tactile controller 678. The watch body 620 may include at least one camera 625, the at least one camera 625 including a front camera 625a and a rear camera 625b. The camera 625 may include an ultra-wide-angle camera, a wide-angle camera, a fisheye camera, a spherical camera, a telephoto camera, a depth sensing camera, or other types of cameras.
手表主体计算系统660可以包括一个或多个触觉控制器678和相关联的部件(例如，触觉设备676)，该一个或多个触觉控制器和该相关联的部件用于在手表主体620处提供触觉事件(例如，对手表主体620处的事件作出响应的振动感觉或音频输出)。触觉控制器678可以与一个或多个触觉设备676(例如电声设备)通信，该一个或多个触觉设备包括一个或多个扬声器674中的扬声器和/或其它音频部件和/或将能量转换为线性运动的机电设备(例如电机、电磁线圈、电活性聚合物、压电致动器、静电致动器或其它触觉输出生成部件(例如，将电信号转换为设备上的触觉输出的部件))。触觉控制器678可以向相应的触觉致动器提供手表主体620的用户能够感觉到的触觉事件。在一些实施例中，一个或多个触觉控制器678可以接收来自应用682中的一应用的输入信号。The watch body computing system 660 may include one or more haptic controllers 678 and associated components (e.g., haptic devices 676) for providing haptic events at the watch body 620 (e.g., a vibration sensation or audio output in response to an event at the watch body 620). The haptic controller 678 may communicate with one or more haptic devices 676 (e.g., an electroacoustic device) including a speaker in one or more speakers 674 and/or other audio components and/or an electromechanical device that converts energy into linear motion (e.g., a motor, an electromagnetic coil, an electroactive polymer, a piezoelectric actuator, an electrostatic actuator, or other haptic output generating component (e.g., a component that converts an electrical signal into a haptic output on the device)). The haptic controller 678 may provide a haptic event that can be felt by a user of the watch body 620 to a corresponding haptic actuator. In some embodiments, the one or more haptic controllers 678 may receive an input signal from an application in the applications 682.
在一些实施例中，计算系统660可以包括存储器680，该存储器可以由一个或多个控制器677和/或一个或多个处理器679的存储器控制器来控制。在一些实施例中，存储在存储器680中的软件组件包括被配置为在手表主体620处执行操作的一个或多个应用682。在一些实施例中，该一个或多个应用682包括游戏、文字处理器、消息传递应用、通话应用、网页浏览器、社交媒体应用、媒体流应用、金融应用、日历、时钟等。在一些实施例中，存储在存储器680中的软件组件包括如上限定的一个或多个通信接口模块683。在一些实施例中，存储在存储器680中的软件组件包括：用于对音频数据和/或视频数据进行渲染、编码和/或解码的一个或多个图形模块684；以及用于收集、组织和/或提供对存储在存储器680中的数据687的访问的一个或多个数据管理模块685。在一些实施例中，应用682中的一个或多个应用和/或一个或多个模块可以彼此结合工作，以在手表主体620处执行各种任务。In some embodiments, the computing system 660 may include a memory 680, which may be controlled by one or more controllers 677 and/or a memory controller of one or more processors 679. In some embodiments, the software components stored in the memory 680 include one or more applications 682 configured to perform operations at the watch body 620. In some embodiments, the one or more applications 682 include games, word processors, messaging applications, call applications, web browsers, social media applications, media streaming applications, financial applications, calendars, clocks, etc. In some embodiments, the software components stored in the memory 680 include one or more communication interface modules 683 as defined above. In some embodiments, the software components stored in the memory 680 include: one or more graphics modules 684 for rendering, encoding and/or decoding audio data and/or video data; and one or more data management modules 685 for collecting, organizing and/or providing access to data 687 stored in the memory 680. In some embodiments, one or more applications and/or one or more modules in the application 682 may work in conjunction with each other to perform various tasks at the watch body 620.
在一些实施例中，存储在存储器680中的软件组件可以包括一个或多个操作系统681(例如，基于Linux的操作系统、安卓(Android)操作系统等)。存储器680还可以包括数据687。数据687可以包括简档数据688A、传感器数据689A、媒体内容数据690和应用数据691。In some embodiments, the software components stored in the memory 680 may include one or more operating systems 681 (e.g., Linux-based operating systems, Android operating systems, etc.). The memory 680 may also include data 687. The data 687 may include profile data 688A, sensor data 689A, media content data 690, and application data 691.
应理解的是，手表主体计算系统660是手表主体620内的计算系统的示例，并且手表主体620可以具有比手表主体计算系统660中所示的部件更多的部件或更少的部件、可以组合两个或更多个部件、和/或可以具有这些部件的不同配置和/或布置。手表主体计算系统660中所示的各个部件以硬件、软件、固件或它们的组合(包括一个或多个信号处理电路和/或专用集成电路)来实现。It should be understood that watch body computing system 660 is an example of a computing system within watch body 620, and that watch body 620 may have more components or fewer components, may combine two or more components, and/or may have a different configuration and/or arrangement of the components than those shown in watch body computing system 660. The various components shown in watch body computing system 660 are implemented in hardware, software, firmware, or a combination thereof (including one or more signal processing circuits and/or application specific integrated circuits).
转到可穿戴带计算系统630，显示了可被包括在可穿戴带610中的一个或多个部件。可穿戴带计算系统630可以包括比手表主体计算系统660中所示的部件更多的部件或更少的部件，可以组合两个或更多个部件，和/或可以具有这些部件中的一些或所有部件的不同配置和/或布置。在一些实施例中，可穿戴带计算系统630的多个部件中的所有部件或大部分部件被包括在单个集成电路中。替代地，在一些实施例中，可穿戴带计算系统630的多个部件被包括在通信耦接的多个集成电路中。如上所述，在一些实施例中，可穿戴带计算系统630被配置为(例如，经由有线连接或无线连接)与手表主体计算系统660耦接，这允许这两个计算系统共享部件、分配任务和/或(单独地或作为单个设备)执行本文所描述的其它操作。Turning to the wearable band computing system 630, one or more components that may be included in the wearable band 610 are shown. The wearable band computing system 630 may include more components or fewer components than those shown in the watch body computing system 660, may combine two or more components, and/or may have different configurations and/or arrangements of some or all of these components. In some embodiments, all or most of the multiple components of the wearable band computing system 630 are included in a single integrated circuit. Alternatively, in some embodiments, the multiple components of the wearable band computing system 630 are included in multiple integrated circuits that are communicatively coupled. As described above, in some embodiments, the wearable band computing system 630 is configured to be coupled to the watch body computing system 660 (e.g., via a wired connection or a wireless connection), which allows the two computing systems to share components, distribute tasks, and/or perform other operations described herein (separately or as a single device).
类似于手表主体计算系统660，可穿戴带计算系统630可以包括：一个或多个处理器649；一个或多个控制器647(包括一个或多个触觉控制器648)；外围接口631，该外围接口631可以包括一个或多个传感器613和其它外围设备；电源(例如，电力系统656)；以及存储器(例如，存储器650)，该存储器包括操作系统(例如，操作系统651)、数据(例如，数据654，其包括资料数据688B、传感器数据689B等)和一个或多个模块(例如，通信接口模块652、数据管理模块653等)。Similar to the watch body computing system 660, the wearable band computing system 630 may include: one or more processors 649; one or more controllers 647 (including one or more tactile controllers 648); a peripheral interface 631, which may include one or more sensors 613 and other peripheral devices; a power source (e.g., a power system 656); and a memory (e.g., a memory 650), which includes an operating system (e.g., an operating system 651), data (e.g., data 654, which includes profile data 688B, sensor data 689B, etc.) and one or more modules (e.g., a communication interface module 652, a data management module 653, etc.).
鉴于以上限定，一个或多个传感器613可以类似于计算系统660的传感器621。例如，传感器613可以包括一个或多个耦接传感器632、一个或多个SpO2传感器634、一个或多个EMG传感器635、一个或多个电容式传感器636、一个或多个心率传感器637、以及一个或多个IMU传感器638。In view of the above definition, the one or more sensors 613 can be similar to the sensor 621 of the computing system 660. For example, the sensor 613 can include one or more coupling sensors 632, one or more SpO2 sensors 634, one or more EMG sensors 635, one or more capacitive sensors 636, one or more heart rate sensors 637, and one or more IMU sensors 638.
外围接口631还可以包括与计算系统660的外围接口661中所包括的那些部件类似的其它部件，如以上参考外围接口661所描述的，这些其它部件包括NFC部件639、GPS部件640、LTE部件641、Wi-Fi和/或蓝牙通信部件642和/或一个或多个触觉设备646。在一些实施例中，外围接口631包括一个或多个按钮643、显示器633、扬声器644、传声器645和摄像头655。在一些实施例中，外围接口631包括一个或多个指示器，例如LED。Peripheral interface 631 may also include other components similar to those included in peripheral interface 661 of computing system 660, as described above with reference to peripheral interface 661, including NFC component 639, GPS component 640, LTE component 641, Wi-Fi and/or Bluetooth communication component 642, and/or one or more haptic devices 646. In some embodiments, peripheral interface 631 includes one or more buttons 643, display 633, speaker 644, microphone 645, and camera 655. In some embodiments, peripheral interface 631 includes one or more indicators, such as LEDs.
应理解的是，可穿戴带计算系统630是可穿戴带610内的计算系统的示例，并且可穿戴带610可以具有比可穿戴带计算系统630中所示的部件更多的部件或更少的部件、可以组合两个或更多个部件、和/或可以具有这些部件的不同配置和/或布置。可穿戴带计算系统630中所示的各个部件可以以硬件、软件和固件中的一者或硬件、软件与固件的组合(包括一个或多个信号处理电路和/或专用集成电路)来实现。It should be understood that the wearable band computing system 630 is an example of a computing system within the wearable band 610, and the wearable band 610 may have more components or fewer components than those shown in the wearable band computing system 630, may combine two or more components, and/or may have a different configuration and/or arrangement of the components. The various components shown in the wearable band computing system 630 may be implemented in one of hardware, software, and firmware, or a combination of hardware, software, and firmware, including one or more signal processing circuits and/or application specific integrated circuits.
参考图6A的腕部可穿戴设备600是可穿戴带610和手表主体620耦接的示例，因此腕部可穿戴设备600将被理解为包括针对可穿戴带计算系统630和手表主体计算系统660所示和所描述的各部件。在一些实施例中，腕部可穿戴设备600在手表主体620与可穿戴带610之间具有分体式架构(例如，分体式机械架构或分体式电子架构)。换言之，在可穿戴带计算系统630和手表主体计算系统660中所示的所有部件可以被容纳或被以其它方式布置在组合后的腕部可穿戴设备600中，或者被容纳或被以其它方式布置在手表主体620、可穿戴带610和/或它们的部分(例如，可穿戴带610的耦接机构616)中的单个部件内。The wrist wearable device 600 of reference FIG6A is an example of a wearable band 610 and a watch body 620 coupled, and thus the wrist wearable device 600 will be understood to include the components shown and described for the wearable band computing system 630 and the watch body computing system 660. In some embodiments, the wrist wearable device 600 has a split architecture (e.g., a split mechanical architecture or a split electronic architecture) between the watch body 620 and the wearable band 610. In other words, all of the components shown in the wearable band computing system 630 and the watch body computing system 660 can be housed or otherwise arranged in the combined wrist wearable device 600, or can be housed or otherwise arranged in a single component in the watch body 620, the wearable band 610, and/or portions thereof (e.g., the coupling mechanism 616 of the wearable band 610).
上述技术可以与用于感测神经肌肉信号的任何设备(包括图6A和图6B中的手臂可穿戴设备)一起使用，但是也可以与用于感测神经肌肉信号的其它类型的可穿戴设备(例如可具有更靠近大脑或脊柱的神经肌肉传感器的身体可穿戴设备或头部可穿戴设备)一起使用。The above techniques can be used with any device for sensing neuromuscular signals (including the arm-wearable devices in Figures 6A and 6B), but can also be used with other types of wearable devices for sensing neuromuscular signals (such as body-wearable devices or head-wearable devices that may have neuromuscular sensors closer to the brain or spine).
在一些实施例中，腕部可穿戴设备600可以与以下所描述的头部可穿戴设备(例如，AR设备700和VR设备710)和/或HIPD 800结合使用，并且腕部可穿戴设备600还可以被配置为用于允许用户控制人工现实的方面(例如，通过使用基于EMG的手势来控制人工现实中的用户界面对象，和/或通过允许用户与腕部可穿戴设备上的触摸屏交互，以便也控制人工现实的各方面)。在一些实施例中，腕部可穿戴设备600还可以与可穿戴服饰(例如以下参考图9A至图9C描述的基于智能纺织品的服饰900)结合使用。已经以此方式描述了示例腕部可穿戴设备，现在将注意力转向示例头部可穿戴设备，例如AR设备700和VR设备710。In some embodiments, the wrist wearable device 600 can be used in conjunction with the head wearable devices described below (e.g., AR device 700 and VR device 710) and/or HIPD 800, and the wrist wearable device 600 can also be configured to allow the user to control aspects of the artificial reality (e.g., by using EMG-based gestures to control user interface objects in the artificial reality, and/or by allowing the user to interact with the touch screen on the wrist wearable device to also control aspects of the artificial reality). In some embodiments, the wrist wearable device 600 can also be used in conjunction with wearable clothing (e.g., smart textile-based clothing 900 described below with reference to Figures 9A to 9C). Having described example wrist wearable devices in this manner, attention is now turned to example head wearable devices, such as AR device 700 and VR device 710.
示例头部可穿戴设备Example head wearable device
图7A、图7B-1、图7B-2和图7C显示了根据一些实施例的示例头部可穿戴设备。头部可穿戴设备可以包括但不限于AR设备700(例如，AR眼镜设备或智能眼镜设备，例如智能眼镜、智能单片眼镜、智能隐形眼镜等)、VR设备710(例如，VR头戴式视图器或头戴式显示器(head-mounted display，HMD))或其它视觉地耦接的设备。AR设备700和VR设备710是本文参考图1A至图3G所描述的头部可穿戴设备102的实例，使得头部可穿戴设备应该被理解为具有AR设备700和/或VR设备710的特征，反之亦然。AR设备700和VR设备710可以执行与在用户界面中导航和选择性地开启应用相关联的各种功能和/或操作、以及以上参考图1A至图3G描述的功能和/或操作。Figures 7A, 7B-1, 7B-2, and 7C show example head wearable devices according to some embodiments. The head wearable device may include, but is not limited to, an AR device 700 (e.g., an AR glasses device or a smart glasses device, such as smart glasses, smart monocles, smart contact lenses, etc.), a VR device 710 (e.g., a VR head-mounted viewer or a head-mounted display (HMD)), or other visually coupled devices. The AR device 700 and the VR device 710 are examples of the head wearable device 102 described herein with reference to Figures 1A to 3G, so that the head wearable device should be understood as having the features of the AR device 700 and/or the VR device 710, and vice versa. The AR device 700 and the VR device 710 can perform various functions and/or operations associated with navigating in a user interface and selectively opening an application, as well as the functions and/or operations described above with reference to Figures 1A to 3G.
在一些实施例中，AR系统(例如，图5A至图5D-2；AR系统500a至500d)包括AR设备700(如图7A所示)和/或VR设备710(如图7B-1和图7B-2所示)。在一些实施例中，AR设备700和VR设备710可以包括一个或多个类似的部件(例如，用于呈现交互式AR环境的部件，例如处理器、存储器和/或呈现设备，所述呈现设备包括一个或多个显示器和/或一个或多个波导)，参考图7C更详细地描述了这些部件中的一些部件。头部可穿戴设备可以使用显示投影仪(例如，显示投影仪组件707A和707B)和/或波导，以用于向用户投影数据的表示。头部可穿戴设备的一些实施例不包括显示器。In some embodiments, an AR system (e.g., FIGS. 5A to 5D-2 ; AR systems 500a to 500d ) includes an AR device 700 (as shown in FIG. 7A ) and/or a VR device 710 (as shown in FIGS. 7B-1 and 7B-2 ). In some embodiments, the AR device 700 and the VR device 710 may include one or more similar components (e.g., components for presenting an interactive AR environment, such as a processor, a memory, and/or a presentation device, the presentation device including one or more displays and/or one or more waveguides), some of which are described in more detail with reference to FIG. 7C . The head wearable device may use a display projector (e.g., display projector components 707A and 707B) and/or a waveguide for projecting a representation of data to a user. Some embodiments of the head wearable device do not include a display.
图7A显示了AR设备700(例如，在本文中该AR设备700还可以被描述为增强现实眼镜和/或智能眼镜)的示例视觉描绘。AR设备700可以与图7A中未示出的附加电子部件(例如可穿戴附件设备和/或中间处理设备)结合工作，这些附加电子部件与AR设备700电子通信或以其它方式被配置为与AR设备700结合使用。在一些实施例中，可穿戴附件设备和/或中间处理设备可以被配置为经由与耦接传感器724(如图7C所示)电子通信的耦接机构而与AR设备700耦接，其中耦接传感器724可以检测电子设备何时与AR设备700物理耦接或电子耦接。在一些实施例中，AR设备700可以被配置为耦接到外壳(例如，框架704的一部分或镜腿705的一部分)，该外壳可以包括被配置为与附加附件设备耦接的一个或多个附加耦接机构。图7A中所示的各部件可以以硬件、软件、固件或它们的组合(包括一个或多个信号处理部件和/或专用集成电路(application-specific integrated circuit，ASIC))来实现。FIG. 7A shows an example visual depiction of an AR device 700 (e.g., the AR device 700 may also be described herein as augmented reality glasses and/or smart glasses). The AR device 700 may work in conjunction with additional electronic components (e.g., wearable accessory devices and/or intermediate processing devices) not shown in FIG. 7A that are in electronic communication with the AR device 700 or otherwise configured for use in conjunction with the AR device 700. In some embodiments, the wearable accessory device and/or intermediate processing device may be configured to couple with the AR device 700 via a coupling mechanism that is in electronic communication with a coupling sensor 724 (shown in FIG. 7C ), where the coupling sensor 724 may detect when the electronic device is physically or electronically coupled to the AR device 700. In some embodiments, the AR device 700 may be configured to couple to a housing (e.g., a portion of the frame 704 or a portion of the temple 705), which may include one or more additional coupling mechanisms configured to couple with additional accessory devices. The components shown in FIG. 7A may be implemented in hardware, software, firmware, or a combination thereof (including one or more signal processing components and/or application-specific integrated circuits (ASICs)).
AR设备700包括机械眼镜部件，所述机械眼镜部件包括框架704，该框架704被配置为保持一个或多个镜片(例如，镜片706-1和706-2中的一者或两者)。本领域普通技术人员将理解的是，AR设备700可以包括附加的机械部件，例如被配置为允许AR设备700的框架704的部分折叠和展开的铰链、被配置为跨越镜片706-1与镜片706-2之间的间隙并依靠在用户的鼻子上的镜梁、被配置为依靠在鼻梁上并为AR设备700提供支撑的鼻托、被配置为依靠在用户的耳朵上并为AR设备700提供附加支撑的耳承(earpiece)、以及被配置为从铰链延伸到AR设备700的耳承的镜腿705等。本领域普通技术人员还将理解的是，AR设备700的一些示例可以不包括本文所描述的机械部件。例如，被配置为向用户呈现AR的智能隐形眼镜可以不包括AR设备700的任何部件。The AR device 700 includes a mechanical eyeglass component, which includes a frame 704 configured to hold one or more lenses (e.g., one or both of the lenses 706-1 and 706-2). It will be appreciated by those of ordinary skill in the art that the AR device 700 may include additional mechanical components, such as a hinge configured to allow a portion of the frame 704 of the AR device 700 to be folded and unfolded, a bridge configured to span the gap between the lenses 706-1 and 706-2 and rest on the user's nose, a nose pad configured to rest on the bridge of the nose and provide support for the AR device 700, an earpiece configured to rest on the user's ear and provide additional support for the AR device 700, and a temple 705 configured to extend from the hinge to the earpiece of the AR device 700, etc. It will also be appreciated by those of ordinary skill in the art that some examples of the AR device 700 may not include the mechanical components described herein. For example, a smart contact lens configured to present AR to a user may not include any components of the AR device 700.
镜片706-1和706-2可以是单独的显示器或显示设备(例如，用于所投影的表示的波导)。镜片706-1和706-2可以协同工作或独立工作，以向用户呈现一幅图像或一系列图像。在一些实施例中，镜片706-1和706-2可以与一个或多个显示投影仪组件707A和707B结合运行，以向用户呈现图像数据。尽管AR设备700包括两个显示器，但是本公开的各实施例可以在具有单个近眼显示器(near-eye display，NED)或多于两个NED的AR设备中实现。Lenses 706-1 and 706-2 can be separate displays or display devices (e.g., waveguides for projected representations). Lenses 706-1 and 706-2 can work together or independently to present an image or a series of images to a user. In some embodiments, lenses 706-1 and 706-2 can operate in conjunction with one or more display projector components 707A and 707B to present image data to a user. Although AR device 700 includes two displays, embodiments of the present disclosure can be implemented in an AR device with a single near-eye display (NED) or more than two NEDs.
AR设备700包括多个电子部件，这些电子部件中的许多电子部件将在以下参考图7C进行更详细地描述。在图7A中示出了一些示例电子部件，这些电子部件包括传感器723-1、723-2、723-3、723-4、723-5和723-6，这些传感器可以沿着AR设备700的框架704的大部分分布。以下参考图7C描述了不同类型的传感器。AR设备700还包括位于框架704的不同侧的左摄像头739A和右摄像头739B。并且眼镜设备包括嵌入到框架704的一部分中的一个或多个处理器748A-1和748A-2(例如，集成微处理器，例如ASIC)。The AR device 700 includes multiple electronic components, many of which will be described in more detail below with reference to FIG. 7C. Some example electronic components are shown in FIG. 7A, including sensors 723-1, 723-2, 723-3, 723-4, 723-5, and 723-6, which can be distributed along most of the frame 704 of the AR device 700. Different types of sensors are described below with reference to FIG. 7C. The AR device 700 also includes a left camera 739A and a right camera 739B located on different sides of the frame 704. And the eyewear device includes one or more processors 748A-1 and 748A-2 (e.g., integrated microprocessors, such as ASICs) embedded in a portion of the frame 704.
图7B-1和图7B-2显示了VR设备710(例如，头戴式显示器(HMD)712，本文中也称为AR头戴式视图器、头部可穿戴设备或VR头戴式视图器)的示例视觉描绘。HMD 712包括前部体714和被成形为适配用户头部的框架716(例如，条或带)。在一些实施例中，前部体714和/或框架716包括一个或多个电子元件(例如，显示器、处理器(例如，处理器748A-1)、IMU、追踪发射器或检测器、或传感器)，该一个或多个电子元件用于促进AR系统和/或VR系统的呈现、和/或与AR系统和/或VR系统的交互。在一些实施例中，如图7B-2所示，HMD 712包括输出音频转换器(例如，音频转换器718)。在一些实施例中，如图7B-2所示，一个或多个部件(例如一个或多个输出音频转换器718和框架716)(例如，框架716和/或输出音频转换器718的一部分或全部)可以被配置为附接到HMD 712和从该HMD712拆卸(例如，其被可拆卸地附接到HMD712)。在一些实施例中，将可拆卸部件耦接到HMD 712使得该可拆卸部件进入与HMD712的电子通信。VR设备710包括电子部件，这些电子部件中的许多电子部件将在以下参考图7C进行更详细地描述。7B-1 and 7B-2 show example visual depictions of a VR device 710 (e.g., a head-mounted display (HMD) 712, also referred to herein as an AR head-mounted viewer, a head wearable device, or a VR head-mounted viewer). HMD 712 includes a front body 714 and a frame 716 (e.g., a strip or band) shaped to fit a user's head. In some embodiments, front body 714 and/or frame 716 include one or more electronic components (e.g., a display, a processor (e.g., processor 748A-1), an IMU, a tracking transmitter or detector, or a sensor) that facilitates presentation of an AR system and/or a VR system, and/or interaction with an AR system and/or a VR system. In some embodiments, as shown in FIG. 7B-2, HMD 712 includes an output audio transducer (e.g., an audio transducer 718). In some embodiments, as shown in FIG. 7B-2 , one or more components (e.g., one or more output audio transducers 718 and frame 716) (e.g., a portion or all of frame 716 and/or output audio transducer 718) can be configured to be attached to and detached from HMD 712 (e.g., it is detachably attached to HMD 712). In some embodiments, coupling a detachable component to HMD 712 causes the detachable component to enter electronic communication with HMD 712. VR device 710 includes electronic components, many of which will be described in more detail below with reference to FIG. 7C .
图7B-1和图7B-2还显示了VR设备710具有一个或多个摄像头，例如左摄像头739A和右摄像头739B，该左摄像头739A和右摄像头739B可以类似于AR设备700的框架704上的左摄像头和右摄像头。在一些实施例中，VR设备710包括一个或多个附加摄像头(例如，摄像头739C和739D)，该一个或多个附加摄像头可以被配置为通过提供更多信息来增强由摄像头739A和739B获取的图像数据。例如，摄像头739C可以用于提供摄像头739A和739B未识别的颜色信息。在一些实施例中，摄像头739A至739D中的一个或多个摄像头可以包括可选的红外(infrared，IR)截止滤光器，该IR截止滤光器被配置为去除在相应摄像头传感器处接收到的IR光。7B-1 and 7B-2 also show that the VR device 710 has one or more cameras, such as a left camera 739A and a right camera 739B, which can be similar to the left camera and the right camera on the frame 704 of the AR device 700. In some embodiments, the VR device 710 includes one or more additional cameras (e.g., cameras 739C and 739D), which can be configured to enhance the image data acquired by the cameras 739A and 739B by providing more information. For example, camera 739C can be used to provide color information that is not recognized by cameras 739A and 739B. In some embodiments, one or more of the cameras 739A to 739D can include an optional infrared (IR) cut filter that is configured to remove IR light received at the corresponding camera sensor.
VR设备710可以包括外壳790，该外壳存储VR设备710的一个或多个部件、和/或VR设备710的附加部件。外壳790可以是这样的模块化电子设备：该模块化电子设备被配置为与VR设备710(或AR设备700)耦接，并补充和/或扩展VR设备710(或AR设备700)的能力。例如，外壳790可以包括附加的传感器、摄像头、电源和处理器(例如，处理器748A-2)，以改进和/或增加VR设备710的功能。以下参考图7C描述了包括在外壳790中的不同部件的示例。The VR device 710 may include a housing 790 that stores one or more components of the VR device 710 and/or additional components of the VR device 710. The housing 790 may be a modular electronic device that is configured to couple with the VR device 710 (or the AR device 700) and to supplement and/or expand the capabilities of the VR device 710 (or the AR device 700). For example, the housing 790 may include additional sensors, cameras, power supplies, and processors (e.g., processor 748A-2) to improve and/or increase the functionality of the VR device 710. Examples of different components included in the housing 790 are described below with reference to FIG. 7C.
替代地或附加地，在一些实施例中，头部可穿戴设备(例如VR设备710和/或AR设备700)包括另一外部设备(例如，配对设备)或通信耦接到另一外部设备，该另一外部设备例如为(以下参考图8A和图8B论述的)HIPD 800和/或可选的颈带。可选的颈带可以经由一个或多个连接器(例如，有线连接器或无线连接器)耦接到头部可穿戴设备。头部可穿戴设备和颈带可以在它们之间没有任何有线连接或无线连接的情况下独立地运行。在一些实施例中，头部可穿戴设备的各部件和颈带的各部件位于与头部可穿戴设备、颈带或它们的某种组合配对的一个或多个附加外围设备上。此外，颈带旨在代表任何合适的类型或形式的配对设备。因此，对颈带的以下论述也可以适用于各种其它配对设备，例如智能手表、智能手机、腕带、其它可穿戴设备、手持控制器、平板电脑或膝上型计算机。Alternatively or additionally, in some embodiments, the head wearable device (e.g., VR device 710 and/or AR device 700) includes another external device (e.g., a pairing device) or is communicatively coupled to another external device, such as HIPD 800 (discussed below with reference to FIGS. 8A and 8B ) and/or an optional neckband. The optional neckband can be coupled to the head wearable device via one or more connectors (e.g., a wired connector or a wireless connector). The head wearable device and the neckband can operate independently without any wired or wireless connection between them. In some embodiments, the components of the head wearable device and the components of the neckband are located on one or more additional peripheral devices that are paired with the head wearable device, the neckband, or some combination thereof. In addition, the neckband is intended to represent any suitable type or form of paired device. Therefore, the following discussion of the neckband may also apply to various other paired devices, such as smart watches, smart phones, wristbands, other wearable devices, handheld controllers, tablets, or laptop computers.
在一些情况下，将外部设备(例如中间处理设备(例如，HIPD 800、可选的颈带和/或可穿戴附件设备))与头部可穿戴设备(例如，AR设备700和/或VR设备710)配对使头部可穿戴设备能够实现类似一副眼镜的形状要素，同时仍为所扩展的能力提供足够的电池电量和计算能力。头部可穿戴设备的电池电量、计算资源和/或附加特征中的一些或全部可以由配对设备来提供，或者在配对设备与该头部可穿戴设备之间共享，从而在总体上降低了头部可穿戴设备的重量、热分布(heat profile)和形状要素，同时允许头部可穿戴设备保持其所期望的功能。例如，中间处理设备(例如，HIPD 800)可以允许将以其它方式包括在头部可穿戴设备中的部件包括在中间处理设备(和/或可穿戴设备或附件设备)中，从而将重量负荷从用户的头部和颈部转移到用户身体的一个或多个其它部位。在一些实施例中，中间处理设备具有在其上向周围环境扩散和分散热量的更大的表面积区域。因此，与单独的头部可穿戴设备上以其它方式可能具有的电池容量和计算能力相比，中间处理设备可以允许更大的电池容量和计算能力。由于中间处理设备中所承载的重量对用户的侵入性可以比头部可穿戴设备中所承载的重量对用户的侵入性更小，因此与用户忍受佩戴更重的单独眼镜设备相比，用户可以忍受佩戴更轻的眼镜设备并携带或佩戴配对设备达更长的时间，从而使AR环境能够更充分地融入到用户的日常活动中。In some cases, pairing an external device, such as an intermediate processing device (e.g., HIPD 800, an optional neckband, and/or a wearable accessory device) with a head wearable device (e.g., AR device 700 and/or VR device 710) enables the head wearable device to achieve a form factor similar to a pair of glasses while still providing sufficient battery power and computing power for the expanded capabilities. Some or all of the battery power, computing resources, and/or additional features of the head wearable device can be provided by the paired device, or shared between the paired device and the head wearable device, thereby reducing the weight, heat profile, and form factor of the head wearable device overall while allowing the head wearable device to maintain its desired functionality. For example, an intermediate processing device (e.g., HIPD 800) can allow components that are otherwise included in the head wearable device to be included in the intermediate processing device (and/or wearable device or accessory device), thereby shifting the weight load from the user's head and neck to one or more other parts of the user's body. In some embodiments, the intermediate processing device has a larger surface area area over which to diffuse and disperse heat to the surrounding environment. Thus, the intermediate processing device may allow for greater battery capacity and computing power than would otherwise be available on a separate head wearable device. Because the weight carried in the intermediate processing device may be less intrusive to the user than the weight carried in the head wearable device, the user may tolerate wearing a lighter eyewear device and carrying or wearing a paired device for longer periods of time than the user would tolerate wearing a heavier separate eyewear device, thereby enabling the AR environment to be more fully integrated into the user's daily activities.
在一些实施例中，中间处理设备与头部可穿戴设备通信耦接和/或通信耦接到其它设备。这些其它设备可以向头部可穿戴设备提供某些功能(例如，追踪、定位、深度图构建、处理和/或存储)。在一些实施例中，中间处理设备包括控制器和电源。在一些实施例中，中间处理设备的传感器被配置为感测附加数据，该附加数据可以以电子格式(模拟或数字)与头部可穿戴设备共享。In some embodiments, the intermediate processing device is communicatively coupled to the head wearable device and/or communicatively coupled to other devices. These other devices can provide certain functions (e.g., tracking, positioning, depth map construction, processing and/or storage) to the head wearable device. In some embodiments, the intermediate processing device includes a controller and a power supply. In some embodiments, the sensor of the intermediate processing device is configured to sense additional data, which can be shared with the head wearable device in an electronic format (analog or digital).
中间处理设备的控制器处理由中间处理设备上和/或头部可穿戴设备上的传感器生成的信息。中间处理设备(例如HIPD 800)可以处理由其多个传感器中的一个或多个传感器生成的信息和/或由其它通信耦接的设备提供的信息。例如，头部可穿戴设备可以包括IMU，并且中间处理设备(颈带和/或HIPD 800)可以根据位于头部可穿戴设备上的IMU来计算所有惯性计算和空间计算。以下参考图8A和图8B提供了由通信耦接的设备(例如HIPD800)执行的处理的附加示例。The controller of the intermediate processing device processes information generated by sensors on the intermediate processing device and/or on the head wearable device. The intermediate processing device (e.g., HIPD 800) can process information generated by one or more of its multiple sensors and/or information provided by other communicatively coupled devices. For example, the head wearable device may include an IMU, and the intermediate processing device (neckband and/or HIPD 800) can calculate all inertial calculations and spatial calculations based on the IMU located on the head wearable device. Additional examples of processing performed by a communicatively coupled device (e.g., HIPD 800) are provided below with reference to Figures 8A and 8B.
AR系统可以包括各种类型的视觉反馈机制。例如，AR设备700和/或VR设备710中的显示设备可以包括一个或多个液晶显示器(liquid-crystal display，LCD)、发光二极管(light emitting diode，LED)显示器、有机LED(organic LED，OLED)显示器和/或任何其它合适类型的显示屏。AR系统可以包括用于双眼的单个显示屏或者可以为每只眼睛提供一个显示屏，这可以允许用于变焦调节或用于校正与用户的视力相关联的屈光不正的额外灵活性。一些AR系统还包括具有一个或多个镜片(例如，常规的凹透镜或凸透镜、菲涅尔(Fresnel)透镜或可调节液体透镜)的光学子系统，用户可以通过这些镜片观看显示屏。除了使用显示屏或代替使用显示屏，一些AR系统包括一个或多个投影系统。例如，AR设备700和/或VR设备710中的显示设备可以包括(例如，使用波导)将光投射到显示设备的微型LED投影仪，例如允许环境光穿透的透明组合透镜。显示设备可以将所投射的光折射到用户的瞳孔，并且可以使用户能够同时观看AR内容和真实世界这两者。AR系统还可以被配置有任何其它合适的类型或形式的图像投影系统。如所指出的，一些AR系统可以不是将人工现实与真实现实混合，而是用虚拟体验大体上代替用户对真实世界的一个或多个感知知觉。The AR system may include various types of visual feedback mechanisms. For example, the display device in the AR device 700 and/or the VR device 710 may include one or more liquid crystal displays (LCD), light emitting diode (LED) displays, organic LED (OLED) displays, and/or any other suitable type of display screen. The AR system may include a single display screen for both eyes or may provide a display screen for each eye, which may allow additional flexibility for zoom adjustment or for correcting refractive errors associated with the user's vision. Some AR systems also include an optical subsystem having one or more lenses (e.g., conventional concave or convex lenses, Fresnel lenses, or adjustable liquid lenses) through which the user can view the display screen. In addition to or instead of using a display screen, some AR systems include one or more projection systems. For example, the display device in the AR device 700 and/or the VR device 710 may include a micro LED projector (e.g., using a waveguide) that projects light to the display device, such as a transparent combination lens that allows ambient light to penetrate. The display device may refract the projected light to the user's pupil, and may enable the user to view both the AR content and the real world simultaneously. The AR system may also be configured with any other suitable type or form of image projection system. As noted, some AR systems may not mix artificial reality with real reality, but rather substantially replace one or more sensory perceptions of the user's real world with a virtual experience.
尽管在本文中将多个示例头部可穿戴设备分别描述为AR设备700和VR设备710，但是本文所描述的多个示例头部可穿戴设备中的任一者或两者可以被配置为呈现在用户的基本上全部的视场中所呈现的完全沉浸式VR场景，作为对在用户的一部分视场(小于全部视场)内所呈现的更微小的增强现实场景的补充或替代。Although multiple example head wearable devices are described herein as AR device 700 and VR device 710, respectively, any or both of the multiple example head wearable devices described herein can be configured to present a fully immersive VR scene presented in substantially all of the user's field of view, in addition to or in lieu of a more subtle augmented reality scene presented within a portion of the user's field of view (less than the full field of view).
在一些实施例中，AR设备700和/或VR设备710可以包括触觉反馈系统。触觉反馈系统可以提供各种类型的皮肤反馈，包括振动、力、牵拉、剪应力、纹理和/或温度。触觉反馈系统还可以提供各种类型的动觉反馈，例如运动和顺应性。触觉反馈可以使用电机、压电致动器、流体系统和/或各种其它类型的反馈机构来实现。触觉反馈系统可以独立于其它AR设备实现、在其它AR设备内实现、和/或结合其它AR设备(例如，可结合到头饰、手套、连体衣中的腕部可穿戴设备；手持控制器；环境设备(例如，椅子或地毯)；和/或任何其它类型的设备或系统，例如腕部可穿戴设备600、HIPD 800、基于智能纺织品的服饰900)和/或本文所描述的其它设备来实现。In some embodiments, the AR device 700 and/or the VR device 710 may include a tactile feedback system. The tactile feedback system can provide various types of skin feedback, including vibration, force, traction, shear stress, texture and/or temperature. The tactile feedback system can also provide various types of kinesthetic feedback, such as motion and compliance. Tactile feedback can be implemented using motors, piezoelectric actuators, fluid systems and/or various other types of feedback mechanisms. The tactile feedback system can be implemented independently of other AR devices, implemented in other AR devices, and/or in combination with other AR devices (e.g., wrist wearable devices that can be incorporated into headwear, gloves, jumpsuits; handheld controllers; environmental equipment (e.g., chairs or carpets); and/or any other type of device or system, such as wrist wearable devices 600, HIPD 800, smart textile-based clothing 900) and/or other devices described herein.
图7C示出了计算系统720和可选的外壳790，以上中的每一者均显示了可被包括在头部可穿戴设备(例如，AR设备700和/或VR设备710)中的部件。在一些实施例中，根据所描述的相应头部可穿戴设备的实际约束，可以在可选的外壳790中包括更多的部件或更少的部件。附加地或替代地，可选的外壳790可以包括附加的部件以扩展和/或增强头部可穿戴设备的功能。7C shows a computing system 720 and an optional housing 790, each of which shows components that may be included in a head wearable device (e.g., AR device 700 and/or VR device 710). In some embodiments, more components or fewer components may be included in the optional housing 790, depending on the actual constraints of the respective head wearable device described. Additionally or alternatively, the optional housing 790 may include additional components to expand and/or enhance the functionality of the head wearable device.
在一些实施例中，计算系统720和/或可选的外壳790可以包括一个或多个外围接口722A和722B、一个或多个电力系统742A和742B(包括充电器输入743、PMIC 744和电池745)、一个或多个控制器746A和746B(包括一个或多个触觉控制器747)、一个或多个处理器748A和748B(如以上所限定的，包括所提供的多个示例中的任何示例)以及存储器750A和750B，以上部件可以彼此电子通信。例如，一个或多个处理器748A和/或748B可以被配置为执行存储在存储器750A和/或750B中的指令，所述指令可以使得一个或多个控制器746A和/或746B中的一控制器来使在外围接口722A和/或722B的一个或多个外围设备处执行多个操作。在一些实施例中，所描述的每个操作可以基于由电力系统742A和/或742B提供的电力而进行。In some embodiments, computing system 720 and/or optional housing 790 may include one or more peripheral interfaces 722A and 722B, one or more power systems 742A and 742B (including charger input 743, PMIC 744, and battery 745), one or more controllers 746A and 746B (including one or more haptic controllers 747), one or more processors 748A and 748B (as defined above, including any of the multiple examples provided), and memory 750A and 750B, which may be in electronic communication with each other. For example, one or more processors 748A and/or 748B may be configured to execute instructions stored in memory 750A and/or 750B, which may cause one of the one or more controllers 746A and/or 746B to cause multiple operations to be performed at one or more peripheral devices of peripheral interfaces 722A and/or 722B. In some embodiments, each of the operations described may be performed based on power provided by power systems 742A and/or 742B.
在一些实施例中，外围接口722A可以包括被配置作为计算系统720的一部分的一个或多个设备，该一个或多个设备中的许多设备已经在上文进行了限定和/或参考图6A和图6B中所示出的腕部可穿戴设备进行了描述。例如，外围接口可以包括一个或多个传感器723A。一些示例传感器包括一个或多个耦接传感器724、一个或多个声学传感器725、一个或多个成像传感器726、一个或多个EMG传感器727、一个或多个电容式传感器728和/或一个或多个IMU 729。在一些实施例中，传感器723A还包括：深度传感器767；光传感器768；和/或以上所限定的或参考本文所论述的任何其它实施例而描述的任何其它类型的传感器。In some embodiments, the peripheral interface 722A may include one or more devices configured as part of the computing system 720, many of which have been defined above and/or described with reference to the wrist wearable device shown in Figures 6A and 6B. For example, the peripheral interface may include one or more sensors 723A. Some example sensors include one or more coupling sensors 724, one or more acoustic sensors 725, one or more imaging sensors 726, one or more EMG sensors 727, one or more capacitive sensors 728, and/or one or more IMUs 729. In some embodiments, the sensor 723A also includes: a depth sensor 767; an optical sensor 768; and/or any other type of sensor defined above or described with reference to any other embodiment discussed herein.
在一些实施例中，外围接口可以包括一个或多个附加外围设备，该一个或多个附加外围设备包括：一个或多个NFC设备730；一个或多个GPS设备731；一个或多个LTE设备732；一个或多个Wi-Fi和/或蓝牙设备733；一个或多个按钮734(例如，包括可滑动的或以其它方式可调整的按钮)；一个或多个显示器735A；一个或多个扬声器736A；一个或多个传声器737A；一个或多个摄像头738A(例如，包括第一摄像头739-1至第n摄像头739-n，该第一摄像头739-1至第n摄像头739-n类似于左摄像头739A和/或右摄像头739B)；一个或多个触觉设备740；和/或以上所限定的或参考本文所论述的任何其它实施例而描述的任何其它类型的外围设备。In some embodiments, the peripheral interface may include one or more additional peripheral devices, including: one or more NFC devices 730; one or more GPS devices 731; one or more LTE devices 732; one or more Wi-Fi and/or Bluetooth devices 733; one or more buttons 734 (e.g., including slidable or otherwise adjustable buttons); one or more displays 735A; one or more speakers 736A; one or more microphones 737A; one or more cameras 738A (e.g., including a first camera 739-1 to an nth camera 739-n, which are similar to the left camera 739A and/or the right camera 739B); one or more tactile devices 740; and/or any other type of peripheral device defined above or described with reference to any other embodiments discussed herein.
头部可穿戴设备可以包括各种类型的视觉反馈机构(例如，呈现设备)。例如，AR设备700和/或VR设备710中的显示设备可以包括一个或多个液晶显示器(LCD)、发光二极管(LED)显示器、有机LED(OLED)显示器、微型LED和/或任何其它合适类型的显示屏。头部可穿戴设备可以包括单个显示屏(例如，被配置为被两只眼睛观看)，和/或可以为每只眼睛提供单独的显示屏，这可以允许用于变焦调节和/或用于校正与用户的视力相关联的屈光不正的额外灵活性。头部可穿戴设备的一些实施例还包括具有一个或多个镜片(例如，常规的凹透镜或凸透镜、菲涅尔透镜或可调节液体透镜)的光学子系统，用户可以通过这些镜片观看显示屏。例如，相应的显示器735A可以被耦接到AR设备700的镜片706-1和镜片706-2中的每一者。耦接到镜片706-1和镜片706-2中的每一者的显示器735A可以协同工作或独立工作，以向用户呈现一幅图像或一系列图像。在一些实施例中，AR设备700和/或VR设备710包括单个显示器735A(例如，近眼显示器)或多于两个显示器735A。The head wearable device may include various types of visual feedback mechanisms (e.g., presentation devices). For example, the display device in the AR device 700 and/or the VR device 710 may include one or more liquid crystal displays (LCDs), light emitting diode (LED) displays, organic LED (OLED) displays, micro-LEDs, and/or any other suitable type of display screen. The head wearable device may include a single display screen (e.g., configured to be viewed by both eyes), and/or a separate display screen may be provided for each eye, which may allow additional flexibility for zoom adjustment and/or for correcting refractive errors associated with the user's vision. Some embodiments of the head wearable device also include an optical subsystem having one or more lenses (e.g., conventional concave or convex lenses, Fresnel lenses, or adjustable liquid lenses) through which the user can view the display screen. For example, a corresponding display 735A may be coupled to each of the lenses 706-1 and 706-2 of the AR device 700. The display 735A coupled to each of the lenses 706-1 and 706-2 may work in conjunction or independently to present an image or a series of images to the user. In some embodiments, the AR device 700 and/or VR device 710 includes a single display 735A (e.g., a near-eye display) or more than two displays 735A.
在一些实施例中，可以使用第一组的一个或多个显示器735A来呈现增强现实环境，并且可以使用第二组的一个或多个显示设备735A来呈现VR环境。在一些实施例中，与向AR设备700和/或VR设备710的用户呈现AR内容相结合地使用一个或多个波导(例如，作为将来自显示投影仪组件和/或一个或多个显示器735A的光传送到用户的眼睛的手段)。在一些实施例中，将一个或多个波导完全地或部分地集成到AR设备700和/或VR设备710中。作为对显示屏的补充或替代，一些AR系统包括一个或多个投影系统。例如，AR设备700和/或VR设备710中的显示设备可以包括(例如，使用波导)将光投射到显示设备的微型LED投影仪，例如允许环境光穿透的透明组合透镜。显示设备可以将所投射的光折射到用户的瞳孔，并且可以使用户能够同时观看AR内容和真实世界这两者。头部可穿戴设备还可以被配置有任何其它合适的类型或形式的图像投影系统。在一些实施例中，作为对一个或多个显示器735A的补充或替代，提供了一个或多个波导。In some embodiments, one or more displays 735A of the first group may be used to present an augmented reality environment, and one or more display devices 735A of the second group may be used to present a VR environment. In some embodiments, one or more waveguides are used in conjunction with presenting AR content to a user of the AR device 700 and/or VR device 710 (e.g., as a means of transmitting light from a display projector assembly and/or one or more displays 735A to the user's eyes). In some embodiments, one or more waveguides are fully or partially integrated into the AR device 700 and/or VR device 710. In addition to or in lieu of a display screen, some AR systems include one or more projection systems. For example, a display device in an AR device 700 and/or VR device 710 may include a micro LED projector that projects light to a display device (e.g., using a waveguide), such as a transparent combination lens that allows ambient light to penetrate. The display device may refract the projected light to the user's pupil, and may enable the user to simultaneously view both AR content and the real world. The head wearable device may also be configured with any other suitable type or form of image projection system. In some embodiments, in addition to or in lieu of one or more displays 735A, one or more waveguides are provided.
在头部可穿戴设备的一些实施例中，可以使环境光和/或真实世界实时视图(例如，用户将正常看到的周围环境的实时馈送)穿透正在呈现AR系统的各方面的相应头部可穿戴设备的显示元件。在一些实施例中，可以使环境光和/或真实世界实时视图穿透在用户的视场内呈现的AR环境的一部分(少于全部AR环境)(例如，AR环境的与用户的真实世界环境中位于指定边界(例如，守护边界)内的物理对象位于同一位置的一部分，该指定边界被配置为由用户在其正与AR环境交互时使用)。例如，可以将视觉用户界面元素(例如，通知用户界面元素)呈现在头部可穿戴设备处，并且可以使一定量的环境光和/或真实世界实时视图(例如，环境光和/或真实世界实时视图的15％至50％)穿透该用户界面元素，使得用户可以区分物理环境的正在其上显示该用户界面元素的至少一部分。In some embodiments of the head wearable device, ambient light and/or a real-world real-time view (e.g., a real-time feed of the surrounding environment that a user would normally see) can be made to penetrate through display elements of the corresponding head wearable device that is presenting aspects of the AR system. In some embodiments, ambient light and/or a real-world real-time view can be made to penetrate a portion (less than the entire AR environment) of the AR environment presented within the user's field of view (e.g., a portion of the AR environment that is co-located with physical objects in the user's real-world environment that are within a specified boundary (e.g., a guard boundary) that is configured to be used by the user when they are interacting with the AR environment). For example, a visual user interface element (e.g., a notification user interface element) can be presented at the head wearable device, and a certain amount of ambient light and/or a real-world real-time view (e.g., 15% to 50% of the ambient light and/or real-world real-time view) can be made to penetrate the user interface element so that the user can distinguish at least a portion of the physical environment on which the user interface element is displayed.
头部可穿戴设备可以包括用于向用户呈现信息的一个或多个外部显示器735A。例如，外部显示器735A可以用于显示当前电池电量、网络活动(例如，连接、断开)、当前活动(例如，玩游戏、通话、开会或看电影)和/或其它相关信息。在一些实施例中，外部显示器735A可以用于与其他人通信。例如，头部可穿戴设备的用户可以使外部显示器735A呈现“请勿打扰”通知。用户还可以使用外部显示器735A来(例如，在一个或多个应用的运行和/或执行期间)共享由外围接口722A的一个或多个部件采集的和/或由头部可穿戴设备生成的任何信息。The head wearable device may include one or more external displays 735A for presenting information to the user. For example, the external display 735A may be used to display the current battery level, network activity (e.g., connected, disconnected), current activity (e.g., playing games, talking, meeting, or watching movies), and/or other relevant information. In some embodiments, the external display 735A may be used to communicate with other people. For example, a user of the head wearable device may cause the external display 735A to present a "Do Not Disturb" notification. The user may also use the external display 735A to share any information collected by one or more components of the peripheral interface 722A and/or generated by the head wearable device (e.g., during the operation and/or execution of one or more applications).
存储器750A可以包括可由一个或多个处理器748A(和/或外壳790的处理器748B)和/或一个或多个控制器746A(和/或外壳790的控制器746B)的存储器控制器执行的指令和/或数据。存储器750A可以包括：一个或多个操作系统751；一个或多个应用752，一个或多个通信接口模块753A，一个或多个图形模块754A，一个或多个AR处理模块755A；和/或以上所限定的或参考本文中所论述的任何其它实施例而描述的任何其它类型的模块或部件。The memory 750A may include instructions and/or data executable by one or more processors 748A (and/or processor 748B of housing 790) and/or a memory controller of one or more controllers 746A (and/or controller 746B of housing 790). The memory 750A may include: one or more operating systems 751; one or more applications 752, one or more communication interface modules 753A, one or more graphics modules 754A, one or more AR processing modules 755A; and/or any other type of module or component as defined above or described with reference to any other embodiment discussed herein.
存储在存储器750A中的数据760可以与以上所论述的应用和/或程序中的一者或多者结合使用。数据760可以包括：资料数据761；传感器数据762；媒体内容数据763；AR应用数据764；和/或以上所限定的或参考本文中所论述的任何其它实施例而描述的任何其它类型的数据。The data 760 stored in the memory 750A may be used in conjunction with one or more of the applications and/or programs discussed above. The data 760 may include: profile data 761; sensor data 762; media content data 763; AR application data 764; and/or any other type of data defined above or described with reference to any other embodiment discussed herein.
在一些实施例中，头部可穿戴设备的控制器746A处理由头部可穿戴设备上的传感器723A、和/或头部可穿戴设备的另一部件、和/或与头部可穿戴设备通信耦接的另一部件(例如，外壳790的部件，例如外围接口722B的部件)生成的信息。例如，控制器746A可以对来自声学传感器725和/或图像传感器726的信息进行处理。对于每个检测到的声音，控制器746A可以执行波达方向(direction of arrival，DOA)估计，以估计检测到的声音到达头部可穿戴设备的方向。在多个声学传感器725中的一个或多个声学传感器检测到声音时，控制器746A可以用(例如，由传感器数据762表示的)信息填充音频数据集合。In some embodiments, the controller 746A of the head wearable device processes information generated by the sensor 723A on the head wearable device, and/or another component of the head wearable device, and/or another component communicatively coupled to the head wearable device (e.g., a component of the housing 790, such as a component of the peripheral interface 722B). For example, the controller 746A can process information from the acoustic sensor 725 and/or the image sensor 726. For each detected sound, the controller 746A can perform a direction of arrival (DOA) estimation to estimate the direction from which the detected sound arrived at the head wearable device. When one or more of the plurality of acoustic sensors 725 detects a sound, the controller 746A can populate the audio data set with information (e.g., represented by the sensor data 762).
在一些实施例中，物理电子连接器可以在头部可穿戴设备与另一电子设备之间传送信息，和/或在头部可穿戴设备的一个或多个处理器748A与控制器746A之间传送信息。该信息可以处于以下形式：光学数据；电数据；无线数据；或任何其它可传输的数据形式。将对头部可穿戴设备生成的信息的处理移到中间处理设备，可以减少眼镜设备的重量和热量，从而使其对于用户更舒适和更安全。在一些实施例中，可选的附件设备(例如，电子颈带或HIPD 800)经由一个或多个连接器耦接到头部可穿戴设备。各连接器可以是有线连接器或无线连接器，并且可以包括电子部件和/或非电子(例如，结构)部件。在一些实施例中，头部可穿戴设备和附件设备可以在它们之间没有任何有线连接或无线连接的情况下独立地运行。In some embodiments, a physical electronic connector can transmit information between a head wearable device and another electronic device, and/or transmit information between one or more processors 748A and a controller 746A of a head wearable device. The information can be in the form of: optical data; electrical data; wireless data; or any other transmittable form of data. Moving the processing of information generated by the head wearable device to an intermediate processing device can reduce the weight and heat of the eyewear device, making it more comfortable and safer for the user. In some embodiments, an optional accessory device (e.g., an electronic neckband or HIPD 800) is coupled to the head wearable device via one or more connectors. Each connector can be a wired connector or a wireless connector, and can include electronic components and/or non-electronic (e.g., structural) components. In some embodiments, the head wearable device and the accessory device can operate independently without any wired or wireless connection between them.
头部可穿戴设备可以包括各种类型的计算机视觉部件和子系统。例如，AR设备700和/或VR设备710可以包括一个或多个光学传感器，例如二维(two-dimensional，2D)摄像头或三维(3D)摄像头、ToF深度传感器、单波束测距仪或扫描激光测距仪、3D激光雷达(LiDAR)传感器、和/或任何其它合适的类型或形式的光学传感器。头部可穿戴设备可以处理来自这些传感器中的一个或多个传感器的数据，以识别用户的位置和/或用户的真实世界物理环境的多个方面(包括真实世界对象在真实世界物理环境中的位置)。在一些实施例中，本文所描述的方法用于映射真实世界，以向用户提供关于真实世界环境的上下文、和/或以生成可交互的虚拟对象(该虚拟对象可以是可与AR环境交互的真实世界对象的副本或数字孪生体)等各种其它功能。例如，图7B-1和图7B-2显示了具有摄像头739A至739D的VR设备710，这些摄像头可以用于提供用于创建体素场和2D网格的深度信息，以向用户提供对象信息从而避免碰撞。The head wearable device may include various types of computer vision components and subsystems. For example, the AR device 700 and/or the VR device 710 may include one or more optical sensors, such as a two-dimensional (2D) camera or a three-dimensional (3D) camera, a ToF depth sensor, a single beam rangefinder or a scanning laser rangefinder, a 3D laser radar (LiDAR) sensor, and/or any other suitable type or form of optical sensor. The head wearable device may process data from one or more of these sensors to identify the user's position and/or multiple aspects of the user's real-world physical environment (including the position of real-world objects in the real-world physical environment). In some embodiments, the methods described herein are used to map the real world to provide the user with context about the real-world environment, and/or to generate various other functions such as interactive virtual objects (which may be copies or digital twins of real-world objects that can interact with the AR environment). For example, Figures 7B-1 and 7B-2 show a VR device 710 with cameras 739A to 739D that can be used to provide depth information for creating a voxel field and a 2D grid to provide object information to the user for collision avoidance.
可选的外壳790可以包括与以上参考计算系统720描述的那些部件类似的部件。例如，可选的外壳790可以包括相应的外围接口722B，该外围接口722B包括以上参考外围接口722A描述的部件更多的部件或更少的部件。如上所述，可选的外壳790的各部件可以用于增强和/或扩展头部可穿戴设备的功能。例如，可选的外壳790可以包括相应的传感器723B、扬声器736B、显示器735B、传声器737B、摄像头738B和/或用于采集和/或呈现数据的其它部件。类似地，可选的外壳790可以包括一个或多个处理器748B、控制器746B和/或存储器750B(包括相应的通信接口模块753B、一个或多个图形模块754B、一个或多个AR处理模块755B)，以上部件可以单独使用和/或与计算系统720的部件结合使用。The optional housing 790 may include components similar to those described above with reference to the computing system 720. For example, the optional housing 790 may include a corresponding peripheral interface 722B, which includes more components or fewer components than those described above with reference to the peripheral interface 722A. As described above, the components of the optional housing 790 can be used to enhance and/or expand the functionality of the head wearable device. For example, the optional housing 790 may include corresponding sensors 723B, speakers 736B, displays 735B, microphones 737B, cameras 738B, and/or other components for collecting and/or presenting data. Similarly, the optional housing 790 may include one or more processors 748B, controllers 746B, and/or memory 750B (including corresponding communication interface modules 753B, one or more graphics modules 754B, and one or more AR processing modules 755B), which may be used alone and/or in combination with components of the computing system 720.
以上在图7A至图7C中描述的技术可以与不同的头部可穿戴设备一起使用。在一些实施例中，头部可穿戴设备(例如，AR设备700和/或VR设备710)可以与一个或多个可穿戴设备结合使用，该一个或多个可穿戴设备例如为腕部可穿戴设备600(或其部件)和/或基于智能纺织品的服饰900(图9A至图9C)以及HIPD 800。已经以此方式描述了示例头部可穿戴设备，现在将注意力转向示例手持中间处理设备，例如HIPD 800。The techniques described above in FIGS. 7A to 7C can be used with different head wearable devices. In some embodiments, a head wearable device (e.g., an AR device 700 and/or a VR device 710) can be used in conjunction with one or more wearable devices, such as a wrist wearable device 600 (or a component thereof) and/or a smart textile-based garment 900 (FIGS. 9A to 9C) and a HIPD 800. Having described an example head wearable device in this manner, attention is now turned to an example handheld intermediate processing device, such as a HIPD 800.
示例手持中间处理设备Example Handheld Intermediate Processing Device
图8A和图8B示出了根据一些实施例的示例手持中间处理设备(HIPD)800。HIPD800可以执行与在用户界面中导航和选择性地开启应用相关联的各种功能和/或操作、以及以上参考图1A至图4C描述的功能和/或操作。8A and 8B illustrate an example handheld intermediate processing device (HIPD) 800 according to some embodiments. HIPD 800 may perform various functions and/or operations associated with navigating in a user interface and selectively launching applications, as well as the functions and/or operations described above with reference to FIGS. 1A to 4C .
图8A显示了HIPD 800的俯视图805和侧视图825。HIPD 800被配置为与关联于用户的一个或多个可穿戴设备(或其它电子设备)通信耦接。例如，HIPD 800被配置为与用户的腕部可穿戴设备600(或其部件，例如手表主体620和可穿戴带610)、AR设备700和/或VR设备710通信耦接。HIPD 800可以被配置为由用户手持(例如，作为手持控制器)、用户(例如，在其口袋中或其包中)随身携带、放在用户附近(例如，当用户坐在桌子旁时放在其桌子上，或放在充电底座上)和/或放在距可穿戴设备或其它电子设备的预定距离处或预定距离内(例如，其中，在一些实施例中，该预定距离是HIPD 800可以成功地与电子设备(例如可穿戴设备)通信耦接的最大距离(例如，10米))。FIG8A shows a top view 805 and a side view 825 of the HIPD 800. The HIPD 800 is configured to be communicatively coupled to one or more wearable devices (or other electronic devices) associated with the user. For example, the HIPD 800 is configured to be communicatively coupled to the user's wrist wearable device 600 (or its components, such as the watch body 620 and the wearable band 610), the AR device 700, and/or the VR device 710. The HIPD 800 can be configured to be held by the user (e.g., as a handheld controller), carried by the user (e.g., in his pocket or in his bag), placed near the user (e.g., on his desk when the user is sitting at a desk, or placed on a charging base), and/or placed at or within a predetermined distance from the wearable device or other electronic device (e.g., where, in some embodiments, the predetermined distance is the maximum distance (e.g., 10 meters) at which the HIPD 800 can successfully communicate with the electronic device (e.g., wearable device)).
HIPD 800可以独立地执行各种功能、和/或与一个或多个可穿戴设备(例如，腕部可穿戴设备600、AR设备700和/或VR设备710)结合地执行各种功能。HIPD 800被配置为增加和/或改进通信耦接的设备(例如可穿戴设备)的功能。HIPD 800被配置为执行与以下项相关联的一个或多个功能或操作：与通信耦接的设备的用户界面和应用交互、与AR环境交互、与VR环境交互和/或充当人机界面控制器。另外，如以下将更详细地描述的，HIPD 800的功能和/或操作可以包括但不限于：任务转移和/或传送；热量转移和/或传送；6自由度(6degrees of freedom，6DoF)射线投射和/或游戏(例如，使用成像设备或摄像头814A和814B，该成像设备或摄像头814A和814B可以用于即时定位与地图构建(simultaneouslocalization and mapping，SLAM)和/或与其它图像处理技术一起使用)；便携式充电；消息传递；经由一个或多个成像设备或摄像头(例如，摄像头822A和822B)进行图像采集；感测用户输入(例如，感测多触摸输入表面802上的触摸)；无线通信和/或互连(例如，蜂窝、近场、Wi-Fi或个人局域网)；位置确定；金融交易；提供触觉反馈；告警；通知；生物特征认证；健康监测；睡眠监测。上述示例功能可以在HIPD 800中独立地执行和/或在HIPD 800与本文所描述的另一可穿戴设备之间的通信中执行。在一些实施例中，可以在HIPD 800上结合AR环境来执行各功能。如本领域技术人员在阅读本文所提供的各描述时将理解的，本文所描述的HIPD 800可以与任何类型的合适的AR环境一起使用。HIPD 800 can perform various functions independently and/or in combination with one or more wearable devices (e.g., wrist wearable device 600, AR device 700, and/or VR device 710). HIPD 800 is configured to increase and/or improve the functionality of a communication-coupled device (e.g., a wearable device). HIPD 800 is configured to perform one or more functions or operations associated with: interacting with a user interface and application of a communication-coupled device, interacting with an AR environment, interacting with a VR environment, and/or acting as a human-machine interface controller. In addition, as will be described in more detail below, the functions and/or operations of the HIPD 800 may include, but are not limited to: task transfer and/or transmission; heat transfer and/or transmission; 6 degrees of freedom (6DoF) ray casting and/or gaming (e.g., using imaging devices or cameras 814A and 814B, which can be used for simultaneous localization and mapping (SLAM) and/or used with other image processing techniques); portable charging; messaging; image acquisition via one or more imaging devices or cameras (e.g., cameras 822A and 822B); sensing user input (e.g., sensing touch on the multi-touch input surface 802); wireless communication and/or interconnection (e.g., cellular, near field, Wi-Fi, or personal area network); location determination; financial transactions; providing tactile feedback; alarms; notifications; biometric authentication; health monitoring; sleep monitoring. The above example functions may be performed independently in the HIPD 800 and/or in communication between the HIPD 800 and another wearable device described herein. In some embodiments, the functions may be performed in conjunction with an AR environment on the HIPD 800. As will be understood by those skilled in the art upon reading the various descriptions provided herein, the HIPD 800 described herein may be used with any type of suitable AR environment.
在HIPD 800与可穿戴设备和/或其它电子设备通信耦接时，HIPD 800被配置为执行在该可穿戴设备和/或该其它电子设备处发起的一个或多个操作。特别地，可以将可穿戴设备和/或其它电子设备的一个或多个操作转移到HIPD 800执行。HIPD 800执行可穿戴设备和/或其它电子设备的一个或多个操作，并向该可穿戴设备和/或该其它电子设备提供与所完成的操作相对应的数据。例如，用户可以使用AR设备700发起视频流，并且与执行该视频流相关联的后端任务(例如，视频渲染)可以被转移到HIPD 800，HIPD 800执行所述后端任务并向AR设备700提供相应数据，以执行与该视频流相关联的剩余前端任务(例如，经由AR设备700的显示器呈现所渲染的视频数据)。以此方式，具有比可穿戴设备更多的计算资源和更大的热余量的HIPD 800可以为可穿戴设备执行计算密集型任务，从而提高由可穿戴设备执行的操作的性能。When the HIPD 800 is communicatively coupled with the wearable device and/or other electronic device, the HIPD 800 is configured to perform one or more operations initiated at the wearable device and/or the other electronic device. In particular, one or more operations of the wearable device and/or other electronic device can be transferred to the HIPD 800 for execution. The HIPD 800 performs one or more operations of the wearable device and/or other electronic device, and provides data corresponding to the completed operations to the wearable device and/or the other electronic device. For example, a user can initiate a video stream using an AR device 700, and a back-end task (e.g., video rendering) associated with executing the video stream can be transferred to the HIPD 800, which performs the back-end task and provides corresponding data to the AR device 700 to perform the remaining front-end tasks associated with the video stream (e.g., presenting the rendered video data via the display of the AR device 700). In this way, the HIPD 800, which has more computing resources and greater thermal margin than the wearable device, can perform computationally intensive tasks for the wearable device, thereby improving the performance of the operations performed by the wearable device.
HIPD 800包括在第一侧(例如，前表面)上的多触摸输入表面802，该多触摸输入表面被配置为检测一个或多个用户输入。特别地，多触摸输入表面802可以检测单击输入、多击输入(multi-tap input)、滑动手势和/或输入、基于力和/或基于压力的触摸输入和按住点击等。多触摸输入表面802被配置为检测电容式触摸输入和/或力(和/或压力)触摸输入。多触摸输入表面802包括由表面凹陷限定的第一触摸输入表面804和由基本上平坦的部分限定的第二触摸输入表面806。第一触摸输入表面804可以被设置为与第二触摸输入表面806相邻。在一些实施例中，第一触摸输入表面804和第二触摸输入表面806可以是不同的尺寸、形状和/或可以覆盖多触摸输入表面802的不同部分。例如，第一触摸输入表面804可以是大致圆形的，并且第二触摸输入表面806是大致矩形的。在一些实施例中，多触摸输入表面802的表面凹陷被配置为引导用户对HIPD 800的操纵。特别地，表面凹陷被配置为使得用户在单手握持HIPD 800时直立地握住该HIPD 800(例如，使得所使用的成像设备或摄像头814A和814B指向天花板或天空)。另外，表面凹陷被配置为使得用户的拇指位于第一触摸输入表面804内。HIPD 800 includes a multi-touch input surface 802 on a first side (e.g., a front surface), which is configured to detect one or more user inputs. In particular, the multi-touch input surface 802 can detect single-click input, multi-tap input, sliding gestures and/or inputs, force-based and/or pressure-based touch inputs and press-and-hold clicks, etc. The multi-touch input surface 802 is configured to detect capacitive touch inputs and/or force (and/or pressure) touch inputs. The multi-touch input surface 802 includes a first touch input surface 804 defined by a surface depression and a second touch input surface 806 defined by a substantially flat portion. The first touch input surface 804 can be arranged adjacent to the second touch input surface 806. In some embodiments, the first touch input surface 804 and the second touch input surface 806 can be different sizes, shapes and/or can cover different parts of the multi-touch input surface 802. For example, the first touch input surface 804 can be substantially circular, and the second touch input surface 806 is substantially rectangular. In some embodiments, the surface depression of the multi-touch input surface 802 is configured to guide the user's manipulation of the HIPD 800. In particular, the surface depression is configured to enable the user to hold the HIPD 800 upright when holding the HIPD 800 in one hand (e.g., such that the imaging device or camera 814A and 814B used is pointed toward the ceiling or the sky). In addition, the surface depression is configured to enable the user's thumb to be located within the first touch input surface 804.
在一些实施例中，不同的触摸输入表面包括多个触摸输入区。例如，第二触摸输入表面806至少包括处于该第二触摸输入区806内的第一触摸输入区808、以及处于该第一触摸输入区808内的第三触摸输入区810。在一些实施例中，多个触摸输入区中的一个或多个触摸输入区是可选的和/或用户限定的(例如，用户可以基于其偏好来指定触摸输入区)。在一些实施例中，每个触摸输入表面和/或触摸输入区与预定的一组命令相关联。例如，在第一触摸输入区808内检测到的用户输入使HIPD 800执行第一命令，并且在第二触摸输入区806内检测到的用户输入使HIPD 800执行与第一命令不同的第二命令。在一些实施例中，不同的触摸输入表面和/或触摸输入区被配置为检测一种或多种类型的用户输入。不同的触摸输入表面和/或触摸输入区可以被配置为检测同一类型或不同类型的用户输入。例如，第一触摸输入区808可以被配置为检测力触摸输入(例如，用户按下的幅度)和电容式触摸输入，并且第二触摸输入区806可以被配置为检测电容式触摸输入。In some embodiments, different touch input surfaces include multiple touch input areas. For example, the second touch input surface 806 includes at least a first touch input area 808 in the second touch input area 806 and a third touch input area 810 in the first touch input area 808. In some embodiments, one or more touch input areas in the multiple touch input areas are optional and/or user-defined (for example, the user can specify the touch input area based on his preference). In some embodiments, each touch input surface and/or touch input area is associated with a predetermined set of commands. For example, the user input detected in the first touch input area 808 causes the HIPD 800 to execute a first command, and the user input detected in the second touch input area 806 causes the HIPD 800 to execute a second command different from the first command. In some embodiments, different touch input surfaces and/or touch input areas are configured to detect one or more types of user input. Different touch input surfaces and/or touch input areas can be configured to detect the same type or different types of user input. For example, the first touch input region 808 may be configured to detect force touch input (eg, the magnitude of a user's press) and capacitive touch input, and the second touch input region 806 may be configured to detect capacitive touch input.
HIPD 800包括一个或多个传感器851(如图8B所示)，该一个或多个传感器用于感测在执行一个或多个操作和/或功能时使用的数据。例如，HIPD 800可以包括IMU，该IMU与摄像头814结合使用以在AR环境或VR环境中进行3维对象操纵(例如，对对象进行放大、移动、销毁等)。HIPD 800中所包括的传感器851的非限制性示例包括光传感器、磁力计、深度传感器、压力传感器和力传感器。以下参考图8B提供了传感器851的附加示例。HIPD 800 includes one or more sensors 851 (as shown in FIG. 8B ) for sensing data used when performing one or more operations and/or functions. For example, HIPD 800 may include an IMU that is used in conjunction with camera 814 to perform 3D object manipulation (e.g., enlarging, moving, destroying, etc. an object) in an AR environment or a VR environment. Non-limiting examples of sensors 851 included in HIPD 800 include light sensors, magnetometers, depth sensors, pressure sensors, and force sensors. Additional examples of sensors 851 are provided below with reference to FIG. 8B .
HIPD 800可以包括一个或多个光指示器812，以向用户提供一个或多个通知。在一些实施例中，光指示器是LED或其它类型的照明设备。光指示器812可以充当隐私灯来通知用户和/或用户附近的其他人：成像设备和/或传声器是活动的。在一些实施例中，光指示器被定位在一个或多个触摸输入表面附近。例如，光指示器可以被定位在第一触摸输入表面804周围。光指示器可以以不同的颜色和/或图案来进行照射，以向用户提供关于设备的一个或多个通知和/或信息。例如，被定位在第一触摸输入表面804周围的光指示器可以在用户接收通知(例如，消息)时闪烁、在HIPD 800失去电力时变为红色、充当进度条(例如，光环，该光环在任务完成时闭合(例如，0％到100％))、充当音量指示器等。HIPD 800 may include one or more light indicators 812 to provide one or more notifications to the user. In some embodiments, the light indicator is an LED or other type of lighting device. The light indicator 812 can act as a privacy light to notify the user and/or other people near the user that the imaging device and/or microphone are active. In some embodiments, the light indicator is positioned near one or more touch input surfaces. For example, the light indicator can be positioned around the first touch input surface 804. The light indicator can be illuminated in different colors and/or patterns to provide the user with one or more notifications and/or information about the device. For example, the light indicator positioned around the first touch input surface 804 can flash when the user receives a notification (e.g., a message), turn red when the HIPD 800 loses power, act as a progress bar (e.g., a halo that closes when the task is completed (e.g., 0% to 100%)), act as a volume indicator, etc.
在一些实施例中，HIPD 800包括另一表面上的一个或多个附加传感器。例如，如图8A所示，HIPD 800包括HIPD 800的边缘上的一组一个或多个传感器(例如，传感器组820)。传感器组820在被定位在HIPD 800的边缘上时，可以按预定倾斜角(例如，26度)进行定位，这允许传感器组820在被放在桌子或其它平面上时朝向用户倾斜。替代地，在一些实施例中，传感器组820被定位在与多触摸输入表面802相对的表面(例如，背面)上。以下详细地论述了传感器组820中的一个或多个传感器。In some embodiments, the HIPD 800 includes one or more additional sensors on another surface. For example, as shown in FIG. 8A , the HIPD 800 includes a group of one or more sensors (e.g., sensor group 820) on the edge of the HIPD 800. When the sensor group 820 is positioned on the edge of the HIPD 800, it can be positioned at a predetermined tilt angle (e.g., 26 degrees), which allows the sensor group 820 to be tilted toward the user when placed on a table or other flat surface. Alternatively, in some embodiments, the sensor group 820 is positioned on a surface (e.g., the back) opposite to the multi-touch input surface 802. One or more sensors in the sensor group 820 are discussed in detail below.
HIPD 800的侧视图825显示了传感器组820和摄像头814B。传感器组820包括一个或多个摄像头822A和822B、深度投影仪824、环境光传感器828和深度接收器830。在一些实施例中，传感器组820包括光指示器826。光指示器826可以充当隐私指示器，以让用户和/或用户周围的人知道摄像头和/或传声器是活动的。传感器组820被配置为采集用户的面部表情，使得用户可以操纵定制化身(例如，在用户的化身或数字表示上显示情绪(例如微笑、大笑等))。传感器组820可以被配置作为侧面的立体红-绿-蓝(red-green-blue，RGB)系统、后面的间接飞行时间(indirect time-of-flight，iToF)系统或后面的立体RGB系统。如本领域技术人员在阅读本文所提供的描述时将理解的，本文所描述的HIPD 800可以使用不同的传感器组820配置和/或传感器组820布置。The side view 825 of HIPD 800 shows sensor group 820 and camera 814B. Sensor group 820 includes one or more cameras 822A and 822B, depth projector 824, ambient light sensor 828 and depth receiver 830. In some embodiments, sensor group 820 includes light indicator 826. Light indicator 826 can act as a privacy indicator to let the user and/or people around the user know that the camera and/or microphone are active. Sensor group 820 is configured to collect the user's facial expressions so that the user can manipulate a customized avatar (e.g., display emotions (e.g., smile, laugh, etc.) on the user's avatar or digital representation). Sensor group 820 can be configured as a stereo red-green-blue (red-green-blue, RGB) system on the side, an indirect time-of-flight (indirect time-of-flight, iToF) system behind, or a stereo RGB system behind. As will be appreciated by those skilled in the art upon reading the description provided herein, the HIPD 800 described herein may use different sensor group 820 configurations and/or sensor group 820 arrangements.
在一些实施例中，HIPD 800包括一个或多个触觉设备871(图8B；例如，振动触觉致动器)，该一个或多个触觉设备被配置为提供触觉反馈(例如，动觉)。传感器851和/或触觉设备871可以被配置为与多个应用和/或通信耦接的设备结合运行，所述多个应用和/或所述通信耦接的设备包括但不限于可穿戴设备、健康监测应用、社交媒体应用、游戏应用和人工现实应用(例如，与人工现实相关联的应用)。In some embodiments, HIPD 800 includes one or more haptic devices 871 ( FIG. 8B ; e.g., vibrotactile actuators) configured to provide tactile feedback (e.g., kinesthetic). Sensor 851 and/or haptic device 871 may be configured to operate in conjunction with a plurality of applications and/or communicatively coupled devices, including but not limited to wearable devices, health monitoring applications, social media applications, gaming applications, and artificial reality applications (e.g., applications associated with artificial reality).
HIPD 800被配置为在没有显示器的情况下运行。然而，在可选的实施例中，HIPD800可以包括显示器868(图8B)。HIPD 800还可以获得一个或多个可选的外围按钮867(图8B)。例如，外围按钮867可以用于开启或关闭HIPD 800。此外，HIPD 800的外壳可以由聚合物和/或弹性体形成。HIPD 800可以被配置为具有防滑表面，以允许在不需要用户看护HIPD800的情况下将HIPD 800放在一表面上。换言之，HIPD 800被设计为使得不会轻易地从一表面滑落。在一些实施例中，HIPD 800包括一个或多个磁体，该一个或多个磁体用于将HIPD800耦接到另一表面。这允许用户将HIPD 800放置到不同的表面，并向用户提供了使用HIPD800的更大的灵活性。HIPD 800 is configured to operate without a display. However, in an optional embodiment, HIPD 800 may include a display 868 (FIG. 8B). HIPD 800 may also obtain one or more optional peripheral buttons 867 (FIG. 8B). For example, peripheral buttons 867 may be used to turn HIPD 800 on or off. In addition, the housing of HIPD 800 may be formed by polymers and/or elastomers. HIPD 800 may be configured to have a non-slip surface to allow HIPD 800 to be placed on a surface without the need for a user to look after HIPD 800. In other words, HIPD 800 is designed so as not to easily slide off a surface. In some embodiments, HIPD 800 includes one or more magnets, which are used to couple HIPD 800 to another surface. This allows the user to place HIPD 800 on different surfaces, and provides greater flexibility in using HIPD 800 to the user.
如上所述，HIPD 800可以分发和/或提供用于在HIPD 800和/或通信耦接的设备处执行一个或多个任务的指令。例如，HIPD 800可以识别要由HIPD 800执行的一个或多个后端任务以及要由通信耦接的设备执行的一个或多个前端任务。尽管HIPD 800被配置为转移和/或传送通信耦接的设备的任务，但是HIPD 800可以(例如，经由一个或多个处理器，例如CPU 877；图8B)执行后端任务和前端任务这两者。HIPD 800可以但不限于可用于执行：增强通话(例如，接收和/或发送3D或2.5D实时立体通话(volumetric call)、实时数字人表示通话和/或化身通话)、谨慎的消息传递、6DoF纵向/横向(portrait/landscape)游戏、AR/VR对象操纵、AR/VR内容显示(例如，经由虚拟显示器呈现内容)和/或其它AR/VR交互。HIPD 800可以单独地或与可穿戴设备(或其它通信耦接的电子设备)结合地执行以上操作。As described above, the HIPD 800 may distribute and/or provide instructions for performing one or more tasks at the HIPD 800 and/or the communication coupled device. For example, the HIPD 800 may identify one or more back-end tasks to be performed by the HIPD 800 and one or more front-end tasks to be performed by the communication coupled device. Although the HIPD 800 is configured to transfer and/or transmit tasks of the communication coupled device, the HIPD 800 may (e.g., via one or more processors, such as CPU 877; FIG. 8B) perform both back-end tasks and front-end tasks. The HIPD 800 may be, but is not limited to, used to perform: enhanced calls (e.g., receiving and/or sending 3D or 2.5D real-time volumetric calls, real-time digital human representation calls and/or avatar calls), discreet messaging, 6DoF portrait/landscape games, AR/VR object manipulation, AR/VR content display (e.g., presenting content via a virtual display) and/or other AR/VR interactions. HIPD 800 may perform the above operations alone or in conjunction with a wearable device (or other communicatively coupled electronic device).
图8B显示了根据一些实施例的HIPD 800的计算系统840(本文中描述为HIPD计算系统)的框图。以上详细描述的HIPD 800可以包括HIPD计算系统840中所示的一个或多个部件。HIPD 800将被理解为包括以下针对HIPD计算系统840所示和所描述的各部件。在一些实施例中，HIPD计算系统840的多个部件中的所有部件或大部分部件被包括在单个集成电路中。替代地，在一些实施例中，HIPD计算系统840的多个部件被包括在通信耦接的多个集成电路中。8B shows a block diagram of a computing system 840 (described herein as a HIPD computing system) of the HIPD 800 according to some embodiments. The HIPD 800 described in detail above may include one or more components shown in the HIPD computing system 840. The HIPD 800 will be understood to include the components shown and described below for the HIPD computing system 840. In some embodiments, all or most of the multiple components of the HIPD computing system 840 are included in a single integrated circuit. Alternatively, in some embodiments, the multiple components of the HIPD computing system 840 are included in multiple integrated circuits that are communicatively coupled.
HIPD计算系统840可以包括：处理器(例如，CPU 877、GPU、和/或具有集成图形的CPU)；控制器875；外围接口850，该外围接口850包括一个或多个传感器851和其它外围设备；电源(例如，电力系统895)；以及存储器(例如，存储器878)，该存储器包括操作系统(例如，操作系统879)、数据(例如，数据888)、一个或多个应用(例如，应用880)以及一个或多个模块(例如，通信接口模块881、图形模块882、任务和处理管理模块883、互操作性模块884、AR处理模块885、数据管理模块886等)。HIPD计算系统840还包括电力系统895，该电力系统895包括充电器输入和输出896、PMIC 897和电池898，以上中的全部部件都在上文进行了限定。The HIPD computing system 840 may include: a processor (e.g., a CPU 877, a GPU, and/or a CPU with integrated graphics); a controller 875; a peripheral interface 850, which includes one or more sensors 851 and other peripherals; a power source (e.g., a power system 895); and a memory (e.g., a memory 878), which includes an operating system (e.g., an operating system 879), data (e.g., data 888), one or more applications (e.g., applications 880), and one or more modules (e.g., a communication interface module 881, a graphics module 882, a task and processing management module 883, an interoperability module 884, an AR processing module 885, a data management module 886, etc.). The HIPD computing system 840 also includes a power system 895, which includes a charger input and output 896, a PMIC 897, and a battery 898, all of which are defined above.
在一些实施例中，外围接口850可以包括一个或多个传感器851。传感器851可以包括与以上参考图6B描述的那些传感器类似的传感器。例如，传感器851可以包括成像传感器854、(可选的)EMG传感器856、IMU 858和电容式传感器860。在一些实施例中，传感器851可以包括：用于感测压力数据的一个或多个压力传感器852；用于感测HIPD 800的高度的高度计853；用于感测磁场的磁力计855；用于确定摄像头与图像中的对象之间的差的深度传感器857(或飞行时间传感器)；用于感测HIPD 800的一部分的相对位移或位置变化的位置传感器859(例如，柔性位置传感器)；用于感测施加到HIPD 800的一部分的力的力传感器861；以及用于检测光的量的光传感器862(例如，环境光传感器)。传感器851可以包括图8B中未示出的一个或多个传感器。In some embodiments, the peripheral interface 850 may include one or more sensors 851. The sensor 851 may include sensors similar to those described above with reference to FIG. 6B. For example, the sensor 851 may include an imaging sensor 854, an (optional) EMG sensor 856, an IMU 858, and a capacitive sensor 860. In some embodiments, the sensor 851 may include: one or more pressure sensors 852 for sensing pressure data; an altimeter 853 for sensing the height of the HIPD 800; a magnetometer 855 for sensing a magnetic field; a depth sensor 857 (or a time-of-flight sensor) for determining the difference between the camera and the object in the image; a position sensor 859 (e.g., a flexible position sensor) for sensing a relative displacement or position change of a portion of the HIPD 800; a force sensor 861 for sensing a force applied to a portion of the HIPD 800; and a light sensor 862 (e.g., an ambient light sensor) for detecting the amount of light. The sensor 851 may include one or more sensors not shown in FIG. 8B.
与以上参考图6B描述的外围设备类似，外围接口850也可以包括NFC部件863、GPS部件864、LTE部件865、Wi-Fi和/或蓝牙通信部件866、扬声器869、触觉设备871以及传声器873。如以上参考图8A所描述的，HIPD 800可以可选地包括显示器868和/或一个或多个按钮867。外围接口850还可以包括一个或多个摄像头870、触摸表面872和/或一个或多个光发射器874。以上参考图8A描述的多触摸输入表面802是触摸表面872的示例。光发射器874可以是一个或多个LED、激光器等，并且可以用于向用户投射或呈现信息。例如，光发射器874可以包括以上参考图8A描述的光指示器812和826。摄像头870(例如，以上在图8A中描述的摄像头814A、814B和822)可以包括一个或多个广角摄像头、鱼眼摄像头、球面摄像头、复眼摄像头(例如，立体和多摄像头)、深度摄像头、RGB摄像头、ToF摄像头、RGB-D摄像头(深度和ToF摄像头)和/或其它可用的摄像头。摄像头870可以用于：SLAM；6DoF射线投射，游戏，对象操纵和/或其它渲染；面部识别和面部表情识别等。Similar to the peripheral devices described above with reference to FIG. 6B, the peripheral interface 850 may also include an NFC component 863, a GPS component 864, an LTE component 865, a Wi-Fi and/or Bluetooth communication component 866, a speaker 869, a tactile device 871, and a microphone 873. As described above with reference to FIG. 8A, the HIPD 800 may optionally include a display 868 and/or one or more buttons 867. The peripheral interface 850 may also include one or more cameras 870, a touch surface 872, and/or one or more light emitters 874. The multi-touch input surface 802 described above with reference to FIG. 8A is an example of a touch surface 872. The light emitter 874 may be one or more LEDs, lasers, etc., and may be used to project or present information to a user. For example, the light emitter 874 may include the light indicators 812 and 826 described above with reference to FIG. 8A. Camera 870 (e.g., cameras 814A, 814B, and 822 described above in FIG. 8A ) may include one or more wide-angle cameras, fisheye cameras, spherical cameras, compound eye cameras (e.g., stereo and multi-cameras), depth cameras, RGB cameras, ToF cameras, RGB-D cameras (depth and ToF cameras), and/or other available cameras. Camera 870 may be used for: SLAM; 6DoF ray casting, gaming, object manipulation and/or other rendering; facial recognition and facial expression recognition, etc.
与以上参考图6B描述的手表主体计算系统660和可穿戴带计算系统630类似，HIPD计算系统840可以包括一个或多个触觉控制器876和相关联的部件(例如，触觉设备871)，该一个或多个触觉控制器和该相关联的部件用于在HIPD 800处提供触觉事件。Similar to the watch body computing system 660 and the wearable band computing system 630 described above with reference to FIG. 6B , the HIPD computing system 840 may include one or more tactile controllers 876 and associated components (e.g., a tactile device 871 ) for providing tactile events at the HIPD 800 .
存储器878可以包括高速随机存取存储器和/或非易失性存储器，例如一个或多个磁盘存储设备、闪存设备或其它非易失性固态存储设备。HIPD 800的其它部件(例如一个或多个处理器和外围接口850)对存储器878的访问可以由控制器875中的存储器控制器来控制。The memory 878 may include high-speed random access memory and/or non-volatile memory, such as one or more disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The access of other components of the HIPD 800 (such as one or more processors and the peripheral interface 850) to the memory 878 may be controlled by a memory controller in the controller 875.
在一些实施例中，存储在存储器878中的软件组件包括一个或多个操作系统879、一个或多个应用880、一个或多个通信接口模块881、一个或多个图形模块882、一个或多个数据管理模块886，这些软件组件与以上参考图6B描述的软件组件类似。In some embodiments, the software components stored in the memory 878 include one or more operating systems 879, one or more applications 880, one or more communication interface modules 881, one or more graphics modules 882, and one or more data management modules 886, which are similar to the software components described above with reference to FIG6B.
在一些实施例中，存储在存储器878中的软件组件包括任务和处理管理模块883，该任务和处理管理模块883用于：识别与用户所执行的操作相关联的一个或多个前端任务和后端任务；执行一个或多个前端任务和/或后端任务；和/或向一个或多个通信耦接的设备提供使得执行该一个或多个前端任务和/或后端任务的指令。在一些实施例中，任务和处理管理模块883使用数据888(例如，设备数据890)来基于以下项分发一个或多个前端任务和/或后端任务：通信耦接的设备的计算资源、可用电力、热余量、正在进行的操作、和/或其它要素。例如，任务和处理管理模块883可以根据确定操作正利用AR设备700处可用的预定量(例如，至少70％)计算资源，来使得在HIPD 800处执行(在通信耦接的AR设备700处执行的操作的)一个或多个后端任务。In some embodiments, the software components stored in the memory 878 include a task and processing management module 883, which is used to: identify one or more front-end tasks and back-end tasks associated with the operation performed by the user; execute one or more front-end tasks and/or back-end tasks; and/or provide instructions to one or more communication-coupled devices to execute the one or more front-end tasks and/or back-end tasks. In some embodiments, the task and processing management module 883 uses data 888 (e.g., device data 890) to distribute one or more front-end tasks and/or back-end tasks based on the following items: computing resources, available power, thermal headroom, ongoing operations, and/or other factors of the communication-coupled devices. For example, the task and processing management module 883 can cause one or more back-end tasks (of the operation performed at the communication-coupled AR device 700) to be performed at the HIPD 800 based on determining that the operation is utilizing a predetermined amount (e.g., at least 70%) of computing resources available at the AR device 700.
在一些实施例中，存储在存储器878中的软件组件包括互操作性模块884，该互操作性模块884用于交换和利用接收到和/或向不同的通信耦接的设备提供的信息。互操作性模块884允许不同的系统、设备和/或应用以协调的方式连接和通信，而无需用户输入。在一些实施例中，存储在存储器878中的软件组件包括AR模块885，该AR模块885被配置为至少基于传感器数据来处理信号，以供在AR和/或VR环境中使用。例如，AR处理模块885可以用于3D对象操纵、手势识别、面部识别和面部表情识别等。In some embodiments, the software components stored in the memory 878 include an interoperability module 884 for exchanging and utilizing information received and/or provided to different communication coupled devices. The interoperability module 884 allows different systems, devices, and/or applications to connect and communicate in a coordinated manner without user input. In some embodiments, the software components stored in the memory 878 include an AR module 885, which is configured to process signals based at least on sensor data for use in an AR and/or VR environment. For example, the AR processing module 885 can be used for 3D object manipulation, gesture recognition, facial recognition, and facial expression recognition, etc.
存储器878还可以包括数据888，该数据888包括结构化数据。在一些实施例中，数据888可以包括资料数据889、设备数据890(包括与HIPD 800通信耦接的一个或多个设备的设备数据，例如设备类型、硬件、软件、配置等)、传感器数据891、媒体内容数据892和应用数据893。The memory 878 may also include data 888, which includes structured data. In some embodiments, the data 888 may include profile data 889, device data 890 (including device data of one or more devices communicatively coupled to the HIPD 800, such as device type, hardware, software, configuration, etc.), sensor data 891, media content data 892, and application data 893.
应理解的是，HIPD计算系统840是HIPD 800内的计算系统的示例，并且HIPD 800可以具有比HIPD计算系统840中所示的部件更多的部件或更少的部件、可以组合两个或更多个部件、和/或可以具有这些部件的不同配置和/或布置。HIPD计算系统840中所示的各个部件以硬件、软件、固件或它们的组合(包括一个或多个信号处理电路和/或专用集成电路)来实现。It should be understood that HIPD computing system 840 is an example of a computing system within HIPD 800, and HIPD 800 may have more components or fewer components than those shown in HIPD computing system 840, may combine two or more components, and/or may have a different configuration and/or arrangement of these components. The various components shown in HIPD computing system 840 are implemented in hardware, software, firmware, or a combination thereof (including one or more signal processing circuits and/or application specific integrated circuits).
以上在图8A和图8B中描述的各技术可以与用作人机界面控制器的任何设备一起使用。在一些实施例中，HIPD 800可以与一个或多个可穿戴设备(例如头部可穿戴设备(例如，AR设备700和VR设备710)和/或腕部可穿戴设备600(或其部件))结合使用。在一些实施例中，HIPD 800还可以与可穿戴服饰(例如基于智能纺织品的服饰900(图9A至图9C))结合使用。已经以此方式描述了HIPD 800，现在将注意力转向示例反馈设备(例如基于智能纺织品的服饰900)。The various techniques described above in Figures 8A and 8B can be used with any device used as a human-machine interface controller. In some embodiments, the HIPD 800 can be used in conjunction with one or more wearable devices, such as head wearable devices (e.g., AR devices 700 and VR devices 710) and/or wrist wearable devices 600 (or parts thereof). In some embodiments, the HIPD 800 can also be used in conjunction with wearable clothing, such as clothing 900 based on smart textiles (Figures 9A to 9C). Having described the HIPD 800 in this manner, attention is now turned to an example feedback device, such as clothing 900 based on smart textiles.
示例基于智能纺织品的服饰Example of clothing based on smart textiles
图9A和图9B示出了根据一些实施例的示例基于智能纺织品的服饰。基于智能纺织品的服饰900(例如，可穿戴手套、衬衫、头带、腕带、袜子等)被配置为与一个或多个电子设备(例如腕部可穿戴设备600、头部可穿戴设备、HIPD 800、膝上型计算机、平板电脑和/或其它计算设备)通信耦接。基于智能纺织品的服饰900可以执行与在用户界面中导航和选择性地开启应用相关联的各种功能和/或操作、以及以上参考图2A至图2C描述的功能和/或操作。9A and 9B illustrate example smart textile-based apparel according to some embodiments. Smart textile-based apparel 900 (e.g., wearable gloves, shirts, headbands, wristbands, socks, etc.) is configured to be communicatively coupled with one or more electronic devices (e.g., wrist wearable device 600, head wearable device, HIPD 800, laptop computer, tablet computer, and/or other computing device). Smart textile-based apparel 900 can perform various functions and/or operations associated with navigating in a user interface and selectively opening applications, as well as the functions and/or operations described above with reference to FIGS. 2A to 2C.
基于智能纺织品的服饰900可以是AR系统(例如以上参考图5D-1和图5D-2描述的AR系统500d)的一部分。基于智能纺织品的服饰900还被配置为基于用户与计算系统的交互(例如，对用户界面的导航、对应用程序的操作(例如，游戏振动、媒体响应性触觉)、设备通知等)和/或用户在AR环境内的交互，来向用户提供反馈(例如，触感反馈或其它触觉反馈)。在一些实施例中，基于智能纺织品的服饰900接收来自通信耦接的设备(例如，腕部可穿戴设备600、头部可穿戴设备和HIPD 800等)的指令，以使得执行反馈响应。替代地或附加地，在一些实施例中，基于智能纺织品的服饰900确定向用户提供的一个或多个反馈响应。基于智能纺织品的服饰900可以基于由其多个传感器中的一个或多个传感器(例如，传感器951；图9C)或通信耦接的传感器(例如，腕部可穿戴设备600的传感器、头部可穿戴设备的传感器、HIPD 800的传感器和/或其它计算设备的传感器)所采集的传感器数据，来确定一个或多个反馈响应。The smart textile-based apparel 900 can be part of an AR system (e.g., the AR system 500d described above with reference to Figures 5D-1 and 5D-2). The smart textile-based apparel 900 is also configured to provide feedback (e.g., tactile feedback or other tactile feedback) to the user based on the user's interaction with the computing system (e.g., navigation of the user interface, operation of the application (e.g., game vibration, media responsive haptics), device notifications, etc.) and/or the user's interaction within the AR environment. In some embodiments, the smart textile-based apparel 900 receives instructions from a communicatively coupled device (e.g., a wrist wearable device 600, a head wearable device, and a HIPD 800, etc.) to execute a feedback response. Alternatively or additionally, in some embodiments, the smart textile-based apparel 900 determines one or more feedback responses to be provided to the user. The smart textile-based garment 900 can determine one or more feedback responses based on sensor data collected by one or more of its multiple sensors (e.g., sensor 951; FIG. 9C ) or communicatively coupled sensors (e.g., sensors of a wrist wearable device 600, sensors of a head wearable device, sensors of a HIPD 800, and/or sensors of other computing devices).
由基于智能纺织品的服饰900和/或通信耦接的设备确定的反馈的非限制性示例包括：视觉反馈、音频反馈、触觉(例如，触感、动觉等)反馈、热或温度反馈、和/或其它感官可感知的反馈。基于智能纺织品的服饰900可以包括用于向用户提供反馈响应的相应的反馈设备(例如，触觉设备或组件962或者其它反馈设备或组件)。类似地，基于智能纺织品的服饰900可以与另一设备(和/或另一设备的反馈设备)通信耦接，以协调向用户提供的反馈。例如，VR设备710可以向用户呈现AR环境，并且在用户与AR环境内的对象(例如虚拟杯子)交互时，基于智能纺织品的服饰900向用户提供相应的响应。特别地，基于智能纺织品的服饰900可以提供防止用户的一个或多个手指弯曲超过某个点(或至少阻碍/防止用户的一个或多个手指移动)来模拟触摸到固体杯子的感觉的触觉反馈，和/或提供模拟对冷饮料或热饮料的感觉的热反馈。Non-limiting examples of feedback determined by the smart textile-based apparel 900 and/or a communicatively coupled device include: visual feedback, audio feedback, tactile (e.g., touch, kinesthetic, etc.) feedback, thermal or temperature feedback, and/or other sensory perceptible feedback. The smart textile-based apparel 900 may include a corresponding feedback device (e.g., a tactile device or component 962 or other feedback device or component) for providing feedback responses to the user. Similarly, the smart textile-based apparel 900 may be communicatively coupled to another device (and/or a feedback device of another device) to coordinate the feedback provided to the user. For example, the VR device 710 may present an AR environment to the user, and when the user interacts with an object (e.g., a virtual cup) within the AR environment, the smart textile-based apparel 900 provides a corresponding response to the user. In particular, the smart textile-based apparel 900 may provide tactile feedback that prevents one or more fingers of the user from bending beyond a certain point (or at least hinders/prevents one or more fingers of the user from moving) to simulate the feeling of touching a solid cup, and/or provide thermal feedback that simulates the feeling of a cold or hot drink.
附加地或替代地，在一些实施例中，基于智能纺织品的服饰900被配置为充当这样的控制器：该控制器被配置为执行与以下项相关联的一个或多个功能或操作、以及以上参考图1A至图4C描述的功能和/或操作：与通信耦接的设备的用户界面和应用交互、与AR环境交互、与VR环境交互和/或充当人机界面控制器。Additionally or alternatively, in some embodiments, the smart textile-based garment 900 is configured to act as a controller that is configured to perform one or more functions or operations associated with the following items, as well as the functions and/or operations described above with reference to Figures 1A to 4C: interacting with a user interface and application of a communicatively coupled device, interacting with an AR environment, interacting with a VR environment and/or acting as a human-machine interface controller.
图9A显示了基于智能纺织品的服饰900的邻近用户手部的手掌一侧的部分上的一个或多个触觉组件962(例如，第一触觉组件962-1至第四触觉组件962-4)，并且图9B示出了基于智能纺织品的服饰900的邻近用户手部的手背一侧的部分上的附加触觉组件(例如，第五触觉组件962-5)。在一些实施例中，触觉组件962包括这样的机构：该机构至少在相应的触觉组件962从第一状态(例如，第一加压状态(例如，处于大气压力或泄气的))转换到第二状态(例如，第二加压状态(例如，膨胀到阈值压力))时提供阻力。换言之，所描述的触觉组件962可以在第一加压状态和第二加压状态之间进行转换，以向用户提供触觉反馈。触觉组件962的结构可以被集成到被配置为接触或接近用户的皮肤的各种设备中，这些设备包括但不限于诸如手套穿戴设备、身体穿戴服饰设备、头戴式视图器设备等设备。各触觉组件962中的每个可以被包括在基于智能纺织品的服饰900的服饰部件904中、或物理耦接到该基于智能纺织品的服饰的服饰部件。例如，触觉组件962-1、962-2、962-3、……、962-N中的每个触觉组件物理耦接到服饰部件904，并且被配置为接触用户的拇指和手指的相应指骨。FIG. 9A shows one or more haptic components 962 (e.g., first haptic components 962-1 to fourth haptic components 962-4) on a portion of a smart textile-based garment 900 adjacent to a palm side of a user's hand, and FIG. 9B shows an additional haptic component (e.g., fifth haptic component 962-5) on a portion of a smart textile-based garment 900 adjacent to a back side of a user's hand. In some embodiments, the haptic component 962 includes a mechanism that provides resistance at least when the corresponding haptic component 962 is converted from a first state (e.g., a first pressurized state (e.g., at atmospheric pressure or deflated)) to a second state (e.g., a second pressurized state (e.g., expanded to a threshold pressure)). In other words, the described haptic component 962 can be converted between the first pressurized state and the second pressurized state to provide tactile feedback to the user. The structure of the haptic component 962 can be integrated into various devices configured to contact or approach the user's skin, including but not limited to devices such as glove-worn devices, body-worn garment devices, head-mounted viewer devices, and the like. Each of haptic components 962 may be included in, or physically coupled to, apparel component 904 of smart textile-based apparel 900. For example, each of haptic components 962-1, 962-2, 962-3, ..., 962-N is physically coupled to apparel component 904 and is configured to contact corresponding phalanges of a user's thumb and fingers.
由于人工现实的千变万化的性质，因此触觉组件962在单次使用期间可能需要在多个状态之间转换数百次或可能数千次。因此，本文所描述的触觉组件962是耐用的，并且被设计为快速地在状态与状态间进行转换。为了提供某种背景，在第一加压状态下，触觉组件962不会阻碍穿戴者的身体的一部分的自由运动。例如，结合到手套中的一个或多个触觉组件962由不阻碍穿戴者的手部和手指的自由运动的柔性材料(例如，静电拉链致动器)制成。触觉组件962被配置为在处于第一加压状态时顺应穿戴者的身体的一部分的形状。然而，一旦处于第二加压状态，触觉组件962可以被配置为限制和/或阻碍穿戴者的身体的一部分(例如，用户手部的附肢(appendage))的自由运动。例如，相应的触觉组件962(或多个相应的触觉组件)可以在该触觉组件962处于第二加压状态时，限制穿戴者的手指的运动(例如，防止手指弯曲或伸展)。此外，一旦处于第二加压状态，触觉组件962可以采用不同的形状，其中一些触觉组件962被配置为采用平面、刚性(例如，平坦的且刚性的)形状，而一些其它触觉组件962被配置为至少部分地弯曲或弯折。Due to the ever-changing nature of artificial reality, the haptic component 962 may need to switch between multiple states hundreds or possibly thousands of times during a single use. Therefore, the haptic component 962 described herein is durable and is designed to quickly switch between states. To provide some background, in the first pressurized state, the haptic component 962 does not hinder the free movement of a part of the wearer's body. For example, one or more haptic components 962 incorporated into the glove are made of a flexible material (e.g., an electrostatic zipper actuator) that does not hinder the free movement of the wearer's hand and fingers. The haptic component 962 is configured to conform to the shape of a part of the wearer's body when in the first pressurized state. However, once in the second pressurized state, the haptic component 962 can be configured to limit and/or hinder the free movement of a part of the wearer's body (e.g., an appendage of the user's hand). For example, a corresponding haptic component 962 (or multiple corresponding haptic components) can limit the movement of the wearer's fingers (e.g., prevent the fingers from bending or extending) when the haptic component 962 is in the second pressurized state. Additionally, once in the second pressurized state, the haptic component 962 can adopt different shapes, with some haptic components 962 being configured to adopt a planar, rigid (e.g., flat and rigid) shape and some other haptic components 962 being configured to at least partially bend or flex.
基于智能纺织品的服饰900可以是AR系统(例如，图5A至图5D-2的AR系统)中的多个设备中的一个设备。例如，用户可以穿戴一副手套(例如，第一类型的基于智能纺织品的服饰900)、穿戴腕部可穿戴设备600(图6A和图6B)的触觉部件、穿戴头带(例如，第二类型的基于智能纺织品的服饰900)、握持HIPD 800等。如以上所解释的，触觉组件962被配置为向基于智能纺织品的服饰900的穿戴者提供触觉模拟。每个基于智能纺织品的服饰900的服饰部件904可以是各种服饰物品(例如，手套、袜子、衬衫、裤子等)中的一种。因此，用户可以穿戴多个基于智能纺织品的服饰900，每个基于智能纺织品的服饰被配置为向身体的穿戴着基于智能纺织品的服饰900的相应部位提供触觉刺激。尽管基于智能纺织品的服饰900被描述为单独的设备，但是在一些实施例中，基于智能纺织品的服饰900可以与本文所描述的其它可穿戴设备结合。例如，基于智能纺织品的服饰900可以形成VR设备710的一部分(例如，头带部分)。The smart textile-based apparel 900 can be one of the multiple devices in the AR system (e.g., the AR system of Figures 5A to 5D-2). For example, the user can wear a pair of gloves (e.g., the first type of smart textile-based apparel 900), wear the tactile components of the wrist wearable device 600 (Figures 6A and 6B), wear a headband (e.g., the second type of smart textile-based apparel 900), hold the HIPD 800, etc. As explained above, the tactile component 962 is configured to provide tactile simulation to the wearer of the smart textile-based apparel 900. The apparel component 904 of each smart textile-based apparel 900 can be one of various apparel items (e.g., gloves, socks, shirts, pants, etc.). Therefore, the user can wear multiple smart textile-based apparel 900, each of which is configured to provide tactile stimulation to the corresponding part of the body of the wearer of the smart textile-based apparel 900. Although smart textile-based apparel 900 is described as a separate device, in some embodiments, smart textile-based apparel 900 can be combined with other wearable devices described herein. For example, smart textile-based apparel 900 can form a portion of VR device 710 (e.g., a headband portion).
图9C显示了根据一些实施例的触觉组件962的计算系统940的框图。计算系统940可以包括一个或多个外围接口950、一个或多个电力系统995(包括充电器输入996、PMIC997和电池998)、一个或多个控制器975(包括一个或多个触觉控制器976)、一个或多个处理器977(如以上所限定的，包括所提供的多个示例中的任何示例)以及存储器978，这些部件都可以彼此电子通信。例如，一个或多个处理器977可以被配置为执行存储在存储器978中的指令，所述指令可以使得一个或多个控制器975中的一控制器来使在外围接口950的一个或多个外围设备处执行多个操作。在一些实施例中，所描述的每个操作可以基于由电力系统995提供的电力而进行。9C shows a block diagram of a computing system 940 of a haptic assembly 962 according to some embodiments. The computing system 940 may include one or more peripheral interfaces 950, one or more power systems 995 (including a charger input 996, a PMIC 997, and a battery 998), one or more controllers 975 (including one or more haptic controllers 976), one or more processors 977 (as defined above, including any of the multiple examples provided), and a memory 978, all of which may be in electronic communication with each other. For example, one or more processors 977 may be configured to execute instructions stored in the memory 978, which may cause one of the one or more controllers 975 to cause multiple operations to be performed at one or more peripheral devices of the peripheral interface 950. In some embodiments, each of the operations described may be performed based on power provided by the power system 995.
在一些实施例中，外围接口950可以包括被配置作为计算系统940的一部分的一个或多个设备，该一个或多个设备中的许多设备已经在上文进行了限定和/或参考图6A至图8B中所示出的腕部可穿戴设备进行了描述。例如，外围接口950可以包括一个或多个传感器951，例如一个或多个压力传感器952、一个或多个EMG传感器956、一个或多个IMU 958、一个或多个位置传感器959、一个或多个电容式传感器960、一个或多个力传感器961；和/或以上所限定的或参考本文中所论述的任何其它实施例而描述的任何其它类型的传感器。在一些实施例中，外围接口可以包括一个或多个附加外围设备，该一个或多个附加外围设备包括：一个或多个Wi-Fi和/或蓝牙设备968；LTE部件969；GPS部件970；传声器971；一个或多个触觉组件962；一个或多个支承结构963，该一个或多个支承结构963可以包括一个或多个囊状物964；一个或多个歧管(manifold)965；一个或多个压力改变设备967；一个或多个显示器972；一个或多个按钮973；一个或多个扬声器974；和/或以上所限定的或参考本文中所论述的任何其它实施例而描述的任何其它类型的外围设备。在一些实施例中，计算系统940包括比图9C中所示的部件更多的部件或更少的部件。In some embodiments, the peripheral interface 950 may include one or more devices configured as part of the computing system 940, many of which have been defined above and/or described with reference to the wrist wearable device shown in Figures 6A to 8B. For example, the peripheral interface 950 may include one or more sensors 951, such as one or more pressure sensors 952, one or more EMG sensors 956, one or more IMUs 958, one or more position sensors 959, one or more capacitive sensors 960, one or more force sensors 961; and/or any other type of sensor defined above or described with reference to any other embodiment discussed herein. In some embodiments, the peripheral interface may include one or more additional peripherals including: one or more Wi-Fi and/or Bluetooth devices 968; LTE component 969; GPS component 970; microphone 971; one or more haptic components 962; one or more support structures 963, which may include one or more bladders 964; one or more manifolds 965; one or more pressure changing devices 967; one or more displays 972; one or more buttons 973; one or more speakers 974; and/or any other type of peripherals defined above or described with reference to any other embodiment discussed herein. In some embodiments, computing system 940 includes more components or fewer components than those shown in FIG. 9C .
在一些实施例中，每个触觉组件962包括支承结构963和至少一个囊状物964。囊状物964(例如，膜状物(membrane))是由耐用且抗穿刺的材料(例如热塑性聚氨酯(thermoplastic polyurethane，TPU)或柔性聚合物等)制成的、密封的可膨胀口袋。囊状物964包含可添加到囊状物964或从该囊状物移除以改变囊状物964内部的压力(例如，流体压力)的介质(例如，诸如空气、惰性气体或甚至液体等流体)。支承结构963由比囊状物964的材料更坚固和更坚硬的材料制成。耦接到相应囊状物964的相应支承结构963被配置为在相应囊状物因囊状物内部压力(例如，流体压力)的改变而改变形状和大小时，加强相应囊状物964。以上示例触觉组件962是非限制性的。触觉组件962可以包括偏心旋转质量(eccentric rotating mass，ERM)、线性谐振致动器(linear resonant actuator，LRA)、音圈电机(voice coil motor，VCM)、压电触觉致动器、热电设备、电磁线圈致动器、超声波转换器、热阻式加热器、帕尔帖(Peltier)设备、和/或被配置为生成可感知响应的其它设备。In some embodiments, each haptic component 962 includes a support structure 963 and at least one bladder 964. The bladder 964 (e.g., a membrane) is a sealed, expandable pocket made of a durable and puncture-resistant material (e.g., thermoplastic polyurethane (TPU) or a flexible polymer, etc.). The bladder 964 contains a medium (e.g., a fluid such as air, an inert gas, or even a liquid) that can be added to or removed from the bladder 964 to change the pressure (e.g., fluid pressure) inside the bladder 964. The support structure 963 is made of a material that is stronger and harder than the material of the bladder 964. The corresponding support structure 963 coupled to the corresponding bladder 964 is configured to strengthen the corresponding bladder 964 when the corresponding bladder changes shape and size due to changes in the pressure (e.g., fluid pressure) inside the bladder. The above example haptic components 962 are non-limiting. The tactile component 962 may include an eccentric rotating mass (ERM), a linear resonant actuator (LRA), a voice coil motor (VCM), a piezoelectric tactile actuator, a thermoelectric device, an electromagnetic coil actuator, an ultrasonic transducer, a thermal resistive heater, a Peltier device, and/or other devices configured to generate a perceptible response.
基于智能纺织品的服饰900还包括触觉控制器976和压力改变设备967。替代地，在一些实施例中，计算系统940与触觉控制器976和/或压力改变设备967通信耦接(例如，与计算系统940的一个或多个处理器977进行电子通信)。触觉控制器976被配置为控制压力改变设备967的运行，并进而控制基于智能纺织品的服饰900的运行。例如，触觉控制器976向压力改变设备967发送一个或多个信号，以激活压力改变设备967(例如，开启和关闭该压力改变设备)。该一个或多个信号可以对要由压力改变设备967输出的期望压力(例如，磅每平方英寸)进行指定。对该一个或多个信号的生成以及进而由压力改变设备967输出的压力可以基于由基于智能纺织品的服饰900的传感器951和/或其它通信耦接的设备收集到的信息。例如，触觉控制器976可以基于收集到的传感器数据提供一个或多个信号，以使压力改变设备967在第一时间增大第一触觉组件962内部的压力(例如，流体压力)；并且该触觉控制器976可以基于附加传感器数据向压力改变设备967提供一个或多个附加信号，以使压力改变设备967进一步在第一时间之后的第二时间增大第二触觉组件962内部的压力。此外，触觉控制器976可以提供一个或多个信号，以使压力改变设备967使位于基于智能纺织品的服饰900的第一部分(例如，第一手指)中的一个或多个囊状物964膨胀，而位于基于智能纺织品的服饰900的第二部分(例如，第二手指)中的一个或多个囊状物964保持不变。另外，触觉控制器976可以提供一个或多个信号，以使压力改变设备967使位于第一基于智能纺织品的服饰900中的一个或多个囊状物964膨胀到第一压力，并使位于第一基于智能纺织品的服饰900中的一个或多个其它囊状物964膨胀到不同于第一压力的第二压力。根据由压力改变设备967服务的基于智能纺织品的服饰900的数量、以及这些基于智能纺织品的服饰中的囊状物的数量，可以通过一个或多个信号实现许多不同的膨胀配置，并且以上多个示例并不意味着是限制性的。The smart textile-based garment 900 also includes a haptic controller 976 and a pressure changing device 967. Alternatively, in some embodiments, the computing system 940 is communicatively coupled to the haptic controller 976 and/or the pressure changing device 967 (e.g., in electronic communication with one or more processors 977 of the computing system 940). The haptic controller 976 is configured to control the operation of the pressure changing device 967 and, in turn, the operation of the smart textile-based garment 900. For example, the haptic controller 976 sends one or more signals to the pressure changing device 967 to activate the pressure changing device 967 (e.g., to turn the pressure changing device on and off). The one or more signals may specify a desired pressure (e.g., pounds per square inch) to be output by the pressure changing device 967. The generation of the one or more signals and, in turn, the pressure output by the pressure changing device 967 may be based on information collected by the sensor 951 and/or other communicatively coupled devices of the smart textile-based garment 900. For example, haptic controller 976 may provide one or more signals based on the collected sensor data to cause pressure changing device 967 to increase the pressure (e.g., fluid pressure) inside first haptic component 962 at a first time; and haptic controller 976 may provide one or more additional signals to pressure changing device 967 based on the additional sensor data to cause pressure changing device 967 to further increase the pressure inside second haptic component 962 at a second time after the first time. In addition, haptic controller 976 may provide one or more signals to cause pressure changing device 967 to expand one or more bladders 964 located in a first portion (e.g., a first finger) of smart textile-based apparel 900, while one or more bladders 964 located in a second portion (e.g., a second finger) of smart textile-based apparel 900 remain unchanged. In addition, haptic controller 976 may provide one or more signals to cause pressure changing device 967 to expand one or more bladders 964 located in a first smart textile-based apparel 900 to a first pressure, and to expand one or more other bladders 964 located in the first smart textile-based apparel 900 to a second pressure different from the first pressure. Depending on the number of smart textile-based garments 900 served by pressure changing device 967, and the number of bladders in those smart textile-based garments, many different inflation configurations may be achieved via one or more signals, and the above multiple examples are not meant to be limiting.
基于智能纺织品的服饰900可以包括可选的歧管965，该可选的歧管位于压力改变设备967、触觉组件962和/或基于智能纺织品的服饰900的其它部分之间。歧管965可以包括一个或多个阀门(未示出)，该一个或多个阀门经由管道将多个触觉组件962中的每个触觉组件与压力改变设备967气动耦接。在一些实施例中，歧管965与控制器975通信，并且控制器975控制歧管965的一个或多个阀门(例如，控制器生成一个或多个控制信号)。歧管965被配置为将压力改变设备967与基于智能纺织品的服饰900的一个或多个触觉组件962可切换地耦接。在一些实施例中，一个或多个基于智能纺织品的服饰900或其它触觉设备可以耦接在触觉设备的网络中，并且歧管965可以在经耦接的多个基于智能纺织品的服饰900之间分配流体。The smart textile-based garment 900 may include an optional manifold 965 located between the pressure changing device 967, the haptic component 962, and/or other parts of the smart textile-based garment 900. The manifold 965 may include one or more valves (not shown) that pneumatically couple each of the multiple haptic components 962 to the pressure changing device 967 via a conduit. In some embodiments, the manifold 965 communicates with a controller 975, and the controller 975 controls one or more valves of the manifold 965 (e.g., the controller generates one or more control signals). The manifold 965 is configured to switchably couple the pressure changing device 967 to one or more haptic components 962 of the smart textile-based garment 900. In some embodiments, one or more smart textile-based garments 900 or other haptic devices may be coupled in a network of haptic devices, and the manifold 965 may distribute fluid between the coupled multiple smart textile-based garments 900.
在一些实施例中，代替使用歧管965来将压力改变设备967与触觉组件962气动耦接，基于智能纺织品的服饰900可以包括多个压力改变设备967，其中每个压力改变设备967直接与单个(或多个)触觉组件962气动耦接。在一些实施例中，压力改变设备967和可选的歧管965可以被配置作为一个或多个基于智能纺织品的服饰900的一部分(未示出)，而在其它实施例中，压力改变设备967和可选的歧管965可以被配置为处于基于智能纺织品的服饰900的外部。在一些实施例中，单个压力改变设备967可以由多个基于智能纺织品的服饰900或其它触觉设备共享。在一些实施例中，压力改变设备967是气动设备、液压设备、气动液压设备、或能够向一个或多个触觉组件962添加介质(例如，流体、液体或气体)以及从该一个或多个触觉组件移除介质的一些其它设备。In some embodiments, instead of using a manifold 965 to pneumatically couple a pressure changing device 967 to a haptic component 962, a smart textile-based garment 900 may include multiple pressure changing devices 967, each of which is pneumatically coupled directly to a single (or multiple) haptic component 962. In some embodiments, the pressure changing device 967 and the optional manifold 965 may be configured as part of one or more smart textile-based garments 900 (not shown), while in other embodiments, the pressure changing device 967 and the optional manifold 965 may be configured to be external to the smart textile-based garment 900. In some embodiments, a single pressure changing device 967 may be shared by multiple smart textile-based garments 900 or other haptic devices. In some embodiments, the pressure changing device 967 is a pneumatic device, a hydraulic device, a pneumatic-hydraulic device, or some other device capable of adding a medium (e.g., a fluid, liquid, or gas) to one or more haptic components 962 and removing a medium from the one or more haptic components.
存储器978包括指令和数据，所述指令和数据中的一些或全部可以作为非暂态计算机可读存储介质而存储在存储器978内。例如，存储器978可以包括：一个或多个操作系统979；一个或多个通信接口模块981；一个或多个互操作性模块984；一个或多个AR处理模块985；一个或多个数据管理模块986；和/或以上所限定的或参考图6A至图8B而描述的任何其它类型的数据。The memory 978 includes instructions and data, some or all of which may be stored as non-transitory computer-readable storage media within the memory 978. For example, the memory 978 may include: one or more operating systems 979; one or more communication interface modules 981; one or more interoperability modules 984; one or more AR processing modules 985; one or more data management modules 986; and/or any other type of data defined above or described with reference to Figures 6A to 8B.
存储器978还包括数据988，该数据可以结合以上所论述的多个应用中的一个或多个应用使用。数据988可以包括：设备数据990；传感器数据991；和/或以上所限定的或参考图6A至图8B而描述的任何其它类型的数据。Memory 978 also includes data 988, which can be used in conjunction with one or more of the multiple applications discussed above. Data 988 can include: device data 990; sensor data 991; and/or any other type of data defined above or described with reference to Figures 6A to 8B.
图9A至图9C中所示的计算系统940(以及基于智能纺织品的服饰900)的不同部件可以经由有线连接(例如，经由总线)耦接。替代地，图9A至图9C中所示的多个设备中的一个或多个设备可以(例如，经由短距离通信信号)无线连接。The different components of computing system 940 (and smart textile-based apparel 900) shown in Figures 9A to 9C can be coupled via wired connections (e.g., via a bus). Alternatively, one or more of the multiple devices shown in Figures 9A to 9C can be wirelessly connected (e.g., via short-range communication signals).
示例实施例Example Embodiments
现在转向前文所描述的方法、设备、系统和计算机可读存储介质的一些示例实施例。Turning now to some example embodiments of the aforementioned methods, apparatus, systems, and computer-readable storage media.
(A1)在一个方面，一些实施例包括一种监测用户的心率的方法(例如，方法400)。在一些实施例中，该方法在具有存储器(例如，可穿戴带610的存储器650、和/或计算系统720的存储器750A)和一个或多个处理器(例如，可穿戴带610的一个或多个处理器649、和/或计算系统720的一个或多个处理器748A)的可穿戴设备(例如，头部可穿戴设备102和/或腕部可穿戴设备202)处执行。该方法包括：(i)经由人工现实头戴式视图器(例如，头部可穿戴设备102)的面向内的摄像头(例如，摄像头104)采集该用户的面部的区域的多幅图像，该区域包括该用户的眼睛(例如，眼睛110)，其中(a)该多幅图像中的第一图像包括瞳孔在该眼睛中的第一位置；以及(b)该多幅图像中的第二图像包括该瞳孔在该眼睛中的第二位置，该第二位置与第一位置不同；以及(ii)基于该多幅图像确定心率信息(例如，心率信息118)。在一些实施例中，心率信息是基于来自该多幅图像的PPG测量结果而确定的。在一些实施例中，该摄像头被配置为具有足够大的用于采集用户表情的视场(例如，足够大的用于采集用户的眼睛和周围面部组织的视场)。(A1) In one aspect, some embodiments include a method for monitoring a user's heart rate (e.g., method 400). In some embodiments, the method is performed at a wearable device (e.g., head wearable device 102 and/or wrist wearable device 202) having a memory (e.g., memory 650 of wearable band 610 and/or memory 750A of computing system 720) and one or more processors (e.g., one or more processors 649 of wearable band 610 and/or one or more processors 748A of computing system 720). The method includes: (i) capturing, via an inward-facing camera (e.g., camera 104) of an artificial reality head-mounted view device (e.g., head wearable device 102), multiple images of an area of the user's face, the area including the user's eyes (e.g., eyes 110), wherein (a) a first image in the multiple images includes a first position of a pupil in the eye; and (b) a second image in the multiple images includes a second position of the pupil in the eye, the second position being different from the first position; and (ii) determining heart rate information (e.g., heart rate information 118) based on the multiple images. In some embodiments, the heart rate information is determined based on PPG measurements from the multiple images. In some embodiments, the camera is configured to have a field of view large enough to capture the user's expression (e.g., a field of view large enough to capture the user's eyes and surrounding facial tissue).
(A2)在A1的一些实施例中，该方法还包括：(i)从该多幅图像中确定眼动追踪信息；以及(ii)根据该眼动追踪信息更新用户界面(例如，用户界面204)。在一些实施例中，摄像头104用于追踪用户的眼睛，并且来自该摄像头104的图像用于确定PPG信息和心率信息。(A2) In some embodiments of A1, the method further includes: (i) determining eye tracking information from the plurality of images; and (ii) updating a user interface (e.g., user interface 204) based on the eye tracking information. In some embodiments, camera 104 is used to track the user's eyes, and images from camera 104 are used to determine PPG information and heart rate information.
(A3)在A2的一些实施例中，根据该眼动追踪信息更新该用户界面包括：基于该用户的凝视方向来更新该用户界面中的焦点。例如，可以基于检测到用户101具有与其在之前的点不同的凝视方向，来使用户界面204在用户的视场内移动。在一些实施例中，更新用户界面包括根据用户的眼动改变视角和/或视场。在一些实施例中，更新用户界面包括根据用户的眼动选择用户界面元素。(A3) In some embodiments of A2, updating the user interface based on the eye tracking information includes: updating the focus in the user interface based on the user's gaze direction. For example, the user interface 204 can be moved within the user's field of view based on detecting that the user 101 has a gaze direction different from that at a previous point. In some embodiments, updating the user interface includes changing the perspective and/or field of view based on the user's eye movement. In some embodiments, updating the user interface includes selecting a user interface element based on the user's eye movement.
(A4)在A1至A3中任一项的一些实施例中，该方法还包括：使用该人工现实头戴式视图器的照明源(例如，照明源106和/或108)来照射该眼睛。例如，头戴式视图器是具有有限外部光(例如，头戴式视图器与用户之间的受控光)的VR头戴式视图器。在一些实施例中，人工现实头戴式视图器被配置为限制环境光进入头戴式视图器与用户的眼睛之间的空间。(A4) In some embodiments of any one of A1 to A3, the method further includes: illuminating the eye using an illumination source of the artificial reality head-mounted view (e.g., illumination source 106 and/or 108). For example, the head-mounted view is a VR head-mounted view with limited external light (e.g., controlled light between the head-mounted view and the user). In some embodiments, the artificial reality head-mounted view is configured to limit ambient light from entering the space between the head-mounted view and the user's eyes.
(A5)在A4的一些实施例中，该照明源被配置为在红外波段或近红外波段中的至少一者中工作。在一些实施例中，照明源被配置为发射红光、近红外光和/或红外光。在一些实施例中，摄像头被配置为采集红光、近红外光和/或红外光。例如，摄像头104可以被配置为感测红外波长和/或近红外波长。(A5) In some embodiments of A4, the illumination source is configured to operate in at least one of an infrared band or a near infrared band. In some embodiments, the illumination source is configured to emit red light, near infrared light, and/or infrared light. In some embodiments, the camera is configured to collect red light, near infrared light, and/or infrared light. For example, the camera 104 can be configured to sense infrared wavelengths and/or near infrared wavelengths.
(A6)在A1至A5中任一项的一些实施例中，该方法还包括：向该用户呈现用户界面，其中，该多幅图像是在该用户正观看该用户界面(例如，用户界面312和/或354)时采集的。在一些实施例中，该多幅图像是在用户正在导航或以其它方式与用户界面交互时采集的。(A6) In some embodiments of any one of A1 to A5, the method further includes: presenting a user interface to the user, wherein the plurality of images are captured while the user is viewing the user interface (e.g., user interface 312 and/or 354). In some embodiments, the plurality of images are captured while the user is navigating or otherwise interacting with the user interface.
(A7)在A6的一些实施例中，该方法还包括：经由该用户界面向该用户呈现心率信息(例如，心率图342)。例如，心率信息是根据以下项来呈现的：用户完成活动、观看特定用户界面和/或输入请求。例如，心率信息是在用户已完成视频游戏的游戏玩耍之后呈现的，以免在游戏玩耍期间分散用户的注意力(和/或在游戏玩耍期间模糊用户的视觉)。(A7) In some embodiments of A6, the method further includes: presenting heart rate information (e.g., heart rate graph 342) to the user via the user interface. For example, the heart rate information is presented based on: the user completing an activity, viewing a particular user interface, and/or inputting a request. For example, the heart rate information is presented after the user has completed gameplay of a video game so as not to distract the user during gameplay (and/or obscure the user's vision during gameplay).
(A8)在A6或A7的一些实施例中，用户界面与在人工现实头戴式视图器上执行的应用相对应。例如，该应用是虚拟现实应用或增强现实应用。例如，用户界面是经由头部可穿戴设备102的显示器呈现的。在一些实施例中，用户界面被显示在与人工现实头戴式视图器不同的显示器上。例如，该应用可以是健身应用或视频游戏。(A8) In some embodiments of A6 or A7, the user interface corresponds to an application executed on the artificial reality head-mounted view device. For example, the application is a virtual reality application or an augmented reality application. For example, the user interface is presented via a display of the head wearable device 102. In some embodiments, the user interface is displayed on a display different from the artificial reality head-mounted view device. For example, the application can be a fitness application or a video game.
(A9)在A1至A8中任一项的一些实施例中，该面向内的摄像头被配置为在近红外范围内工作。在一些实施例中，该面向内的摄像头(例如，摄像头104)被配置为在近红外范围和可见光范围内工作。在一些实施例中，该面向内的摄像头被配置为在从红外光到可见光的范围内工作。红外波段可能是有益的，因为与可见光波段相比，血液在红外波段中具有增大的不透明度。另外，眼动追踪摄像头得益于在红外波段中工作，因为用于眼动追踪的红外(IR)照明源将不会干扰用户的视觉。(A9) In some embodiments of any of A1 to A8, the inward-facing camera is configured to operate in a near infrared range. In some embodiments, the inward-facing camera (e.g., camera 104) is configured to operate in a near infrared range and a visible light range. In some embodiments, the inward-facing camera is configured to operate in a range from infrared light to visible light. The infrared band may be beneficial because blood has increased opacity in the infrared band compared to the visible light band. Additionally, the eye tracking camera benefits from operating in the infrared band because an infrared (IR) illumination source used for eye tracking will not interfere with the user's vision.
(A10)在A1至A9中任一项的一些实施例中，该心率信息是基于该眼睛的扩张变化的，该眼睛的扩张变化是基于该多幅图像计算的。在一些实施例中，心率信息是基于眼睛(例如，眼睛110)内的血液流动的。在一些实施例中，心率信息是基于眼睛内的血容量水平的。(A10) In some embodiments of any of A1 to A9, the heart rate information is based on a change in dilation of the eye, the change in dilation of the eye calculated based on the plurality of images. In some embodiments, the heart rate information is based on blood flow within an eye (e.g., eye 110). In some embodiments, the heart rate information is based on a blood volume level within the eye.
(A11)在A1至A10中任一项的一些实施例中，该心率信息是基于该用户的肤色变化的，该用户的肤色变化是基于该多幅图像计算的。在一些实施例中，心率信息是基于用户的组织中的血容量水平的。(A11) In some embodiments of any one of A1 to A10, the heart rate information is based on changes in the user's skin color, and the changes in the user's skin color are calculated based on the multiple images. In some embodiments, the heart rate information is based on the blood volume level in the user's tissue.
(A12)在A11的一些实施例中，该用户的肤色变化与该用户的面部的血液流动(例如，脸颊中的红晕的量)相对应。例如，在一些实施例中，摄像头104的视场被配置为采集用户101的面部表情的一部分，包括用户101的眼睛110周围的皮肤的部分。(A12) In some embodiments of A11, the change in skin color of the user corresponds to blood flow to the user's face (e.g., the amount of blush in the cheeks). For example, in some embodiments, the field of view of camera 104 is configured to capture a portion of the facial expression of user 101, including a portion of the skin around eyes 110 of user 101.
(A13)在A1至A12中任一项的一些实施例中，该方法还包括：获取来自用户所佩戴的腕部可穿戴设备(例如，腕部可穿戴设备202)的辅助心率信息(例如，PPG 222)。在一些实施例中，腕部可穿戴设备包括一个或多个图像传感器，该一个或多个图像传感器用于基于用户的组织中的血容量水平确定PPG。在一些实施例中，腕部可穿戴设备包括用于确定心率信息的一个或多个非图像传感器。(A13) In some embodiments of any one of A1 to A12, the method further includes: obtaining auxiliary heart rate information (e.g., PPG 222) from a wrist wearable device (e.g., wrist wearable device 202) worn by the user. In some embodiments, the wrist wearable device includes one or more image sensors, which are used to determine the PPG based on the blood volume level in the user's tissue. In some embodiments, the wrist wearable device includes one or more non-image sensors for determining heart rate information.
(A14)在A13的一些实施例中，该方法还包括：(i)对心率信息和辅助心率信息汇总，以获得汇总后的心率信息；以及(ii)向用户呈现该汇总后的心率信息。在一些实施例中，将心率信息与辅助心率信息结合，以获得中位数心率和/或平均心率。在一些实施例中，基于心率信息和/或辅助心率信息的质量来选择该心率信息和该辅助心率信息中的一者。(A14) In some embodiments of A13, the method further includes: (i) aggregating the heart rate information and the auxiliary heart rate information to obtain aggregated heart rate information; and (ii) presenting the aggregated heart rate information to the user. In some embodiments, the heart rate information is combined with the auxiliary heart rate information to obtain a median heart rate and/or an average heart rate. In some embodiments, one of the heart rate information and the auxiliary heart rate information is selected based on the quality of the heart rate information and/or the auxiliary heart rate information.
(A15)在A13或A14的一些实施例中，该方法还包括：(i)根据辅助心率信息满足一个或多个标准，选择该辅助心率信息作为选定心率信息；(ii)根据该辅助心率信息不满足一个或多个标准，选择心率信息作为选定心率信息；以及(iii)向用户呈现该选定心率信息。例如，该一个或多个标准被选择为评估腕部可穿戴设备与用户的腕部之间的耦接。在一些实施例中，该一个或多个标准是基于用户的腕部的运动量(例如，基于IMU的运动检测)的。在一些实施例中，该一个或多个标准包括一个或多个质量标准(例如，噪声量)。在一些实施例中，该一个或多个标准包括关于来自摄像头的图像的质量的一个或多个标准。例如，用户(例如，在参与体能活动和/或玩视频游戏时)的运动可以使腕部可穿戴设备的传感器相对于用户的身体移动/断开耦接。另外，用户的手部/腕部中的血容量水平通常显著低于用户的面部/眼睛中的血容量水平，因此，从面部/眼睛获取PPG信息可能是有益的。(A15) In some embodiments of A13 or A14, the method further includes: (i) selecting the auxiliary heart rate information as the selected heart rate information based on the auxiliary heart rate information satisfying one or more criteria; (ii) selecting the heart rate information as the selected heart rate information based on the auxiliary heart rate information not satisfying one or more criteria; and (iii) presenting the selected heart rate information to the user. For example, the one or more criteria are selected to evaluate the coupling between the wrist wearable device and the user's wrist. In some embodiments, the one or more criteria are based on the amount of movement of the user's wrist (e.g., based on IMU-based motion detection). In some embodiments, the one or more criteria include one or more quality criteria (e.g., amount of noise). In some embodiments, the one or more criteria include one or more criteria regarding the quality of the image from the camera. For example, the movement of the user (e.g., while participating in physical activities and/or playing video games) may cause the sensor of the wrist wearable device to move/disconnect relative to the user's body. In addition, the blood volume level in the user's hand/wrist is typically significantly lower than the blood volume level in the user's face/eyes, and therefore, obtaining PPG information from the face/eyes may be beneficial.
(A16)在A13至A15中任一项的一些实施例中，该方法还包括：(i)根据心率信息满足一个或多个标准，选择该心率信息作为选定心率信息；(ii)根据该心率信息不满足一个或多个标准，选择辅助心率信息作为选定心率信息；以及(iii)向用户呈现该选定心率信息。例如，该一个或多个标准被选择为评估该多幅图像的图像质量。在一些实施例中，该一个或多个标准包括一个或多个质量标准(例如，噪声量)。(A16) In some embodiments of any one of A13 to A15, the method further includes: (i) selecting the heart rate information as the selected heart rate information based on the heart rate information satisfying one or more criteria; (ii) selecting the auxiliary heart rate information as the selected heart rate information based on the heart rate information not satisfying one or more criteria; and (iii) presenting the selected heart rate information to the user. For example, the one or more criteria are selected to evaluate the image quality of the multiple images. In some embodiments, the one or more criteria include one or more quality criteria (e.g., amount of noise).
(A17)在A1至A16中任一项的一些实施例中，心率信息包括关于该用户在一时间段(例如，图3C中从t0到t2)内的心率的信息。例如，心率信息包括该时间段的中位数心率和/或平均心率。在一些实施例中，向用户呈现该心率信息(例如，包括呈现曲线图、平均值、最大值和/或最小值)。(A17) In some embodiments of any one of A1 to A16, the heart rate information includes information about the user's heart rate over a time period (e.g., from t0 to t2 in FIG. 3C ). For example, the heart rate information includes a median heart rate and/or an average heart rate for the time period. In some embodiments, the heart rate information is presented to the user (e.g., including presenting a graph, an average, a maximum value, and/or a minimum value).
(B1)在另一方面，一些实施例包括一种监测用户的心率的方法。在一些实施例中，该方法在具有存储器(例如，可穿戴带610的存储器650、和/或计算系统720的存储器750A)和一个或多个处理器(例如，可穿戴带610的一个或多个处理器649、和/或计算系统720的一个或多个处理器748A)的可穿戴设备(例如，头部可穿戴设备102和/或腕部可穿戴设备202)处执行。该方法包括：(i)经由该人工现实头戴式视图器(例如，头部可穿戴设备102)的面向内的摄像头(例如，摄像头104)采集用户的面部的区域的多幅图像，该区域包括该用户的眼睛(例如，眼睛110)，其中(a)该多幅图像中的第一图像包括瞳孔在该眼睛中的第一位置；以及(b)该多幅图像中的第二图像包括该瞳孔在该眼睛中的第二位置，该第二位置与第一位置不同；(ii)确定该多幅图像是否满足一个或多个质量标准(例如，确定眼睛的一部分在该多幅图像的至少一个子集中是否被遮挡和/或失焦)；(iii)根据该多幅图像满足该一个或多个质量标准，基于该多幅图像中的眼睛确定心率信息；以及(iv)根据该多幅图像不满足该一个或多个质量标准，基于该多幅图像中的肤色变化确定心率信息。例如，如果眼睛被遮挡或失焦，则根据用户的面部组织中的血容量水平来获取PPG信息。作为另一个示例，如果眼睛在图像中聚焦且未被遮挡，则根据眼动、眼睛中的血液流动和/或血容量水平来获取PPG信息。在一些实施例中，从该多幅图像中识别用户的皮肤的多个区域，并且对每个区域执行PPG。在一些实施例中，选择具有最高质量PPG(例如，最小噪声)的区域用于确定心率信息。在一些实施例中，根据头戴式视图器和/或摄像头相对于用户面部的定位来选择用户的皮肤的区域以进行PPG分析。(B1) In another aspect, some embodiments include a method for monitoring a user's heart rate. In some embodiments, the method is performed at a wearable device (e.g., head wearable device 102 and/or wrist wearable device 202) having a memory (e.g., memory 650 of wearable band 610 and/or memory 750A of computing system 720) and one or more processors (e.g., one or more processors 649 of wearable band 610 and/or one or more processors 748A of computing system 720). The method includes: (i) capturing, via an inward-facing camera (e.g., camera 104) of the artificial reality head-mounted view device (e.g., head wearable device 102), a plurality of images of an area of a user's face, the area including the user's eyes (e.g., eyes 110), wherein (a) a first image in the plurality of images includes a first position of a pupil in the eye; and (b) a second image in the plurality of images includes a second position of the pupil in the eye, the second position being different from the first position; (ii) determining whether the plurality of images meet one or more quality criteria (e.g., determining whether a portion of the eye is occluded and/or out of focus in at least one subset of the plurality of images); (iii) determining heart rate information based on the eyes in the plurality of images based on the plurality of images meeting the one or more quality criteria; and (iv) determining heart rate information based on skin color changes in the plurality of images based on the plurality of images not meeting the one or more quality criteria. For example, if the eyes are occluded or out of focus, PPG information is obtained based on blood volume levels in facial tissue of the user. As another example, if the eye is in focus in the image and is not obstructed, PPG information is obtained based on eye movement, blood flow in the eye, and/or blood volume level. In some embodiments, multiple areas of the user's skin are identified from the multiple images, and PPG is performed on each area. In some embodiments, the area with the highest quality PPG (e.g., the least noise) is selected for determining heart rate information. In some embodiments, the area of the user's skin is selected for PPG analysis based on the positioning of the head mounted view device and/or camera relative to the user's face.
(B2)在B1的一些实施例中，该一个或多个质量标准包括与该多幅图像中所示的瞳孔的一部分相对应的标准。例如，如果瞳孔的一部分在该多幅图像的至少一个子集中被遮挡或模糊，则不使用眼睛来确定心率信息(例如，从面部组织或用户身体的其它部位来获取PPG信息和/或心率信息)。(B2) In some embodiments of B1, the one or more quality criteria include a criterion corresponding to a portion of a pupil shown in the plurality of images. For example, if a portion of the pupil is occluded or blurred in at least a subset of the plurality of images, the eye is not used to determine the heart rate information (e.g., PPG information and/or heart rate information is obtained from facial tissue or other parts of the user's body).
(B3)在B1或B2的一些实施例中，该一个或多个质量标准包括与该多幅图像中所示的瞳孔的照射相对应的标准。例如，如果瞳孔的至少一部分在该多幅图像的至少一个子集中太暗或太亮而无法确定血液流动/容量信息，则不使用眼睛来确定相应时间段的心率信息。例如，如参考图1C所论述的，在一些情况下，基于眼睛的PPG信息114因为眼睛110的摄像头104视野被遮挡而不可用。(B3) In some embodiments of B1 or B2, the one or more quality criteria include a criterion corresponding to an illumination of the pupil shown in the plurality of images. For example, if at least a portion of the pupil is too dark or too bright in at least a subset of the plurality of images to determine blood flow/volume information, the eye is not used to determine heart rate information for the corresponding time period. For example, as discussed with reference to FIG. 1C , in some cases, the eye-based PPG information 114 is not available because the field of view of the camera 104 of the eye 110 is blocked.
(B4)在B1至B3中任一项的一些实施例中，该一个或多个质量标准包括与该多幅图像中用户的皮肤的照射相对应的标准。例如，如果皮肤在该多幅图像的至少一个子集中太暗或太亮而无法确定血液流动/容量信息，则不使用皮肤来确定相应时间段的心率信息。(B4) In some embodiments of any one of B1 to B3, the one or more quality criteria include a criterion corresponding to illumination of the user's skin in the plurality of images. For example, if the skin is too dark or too light in at least a subset of the plurality of images to determine blood flow/volume information, the skin is not used to determine heart rate information for the corresponding time period.
(B5)在B1至B4中任一项的一些实施例中，该一个或多个质量标准包括与该多幅图像中用户的皮肤的量相对应的标准。例如，如果该多幅图像中所示的皮肤的量不足以确定血液流动/容积信息，则不使用皮肤来确定相应时间段的心率信息。(B5) In some embodiments of any one of B1 to B4, the one or more quality criteria include a criterion corresponding to an amount of skin of the user in the plurality of images. For example, if the amount of skin shown in the plurality of images is insufficient to determine blood flow/volume information, the skin is not used to determine heart rate information for the corresponding time period.
(B6)在B1至B5中任一项的一些实施例中，该一个或多个质量标准包括与该多幅图像中的焦点相对应的标准。例如，如果在该多幅图像中的一些图像中用户的面部组织聚焦且用户的眼睛未聚焦(例如，模糊)，则可以使用面部组织来确定相应时间段的心率信息。(B6) In some embodiments of any one of B1 to B5, the one or more quality criteria include a criterion corresponding to a focus in the plurality of images. For example, if facial tissue of the user is in focus and the user's eyes are out of focus (e.g., blurred) in some of the plurality of images, the facial tissue may be used to determine the heart rate information for the corresponding time period.
(B7)在B1至B6中任一项的一些实施例中，该一个或多个质量标准包括与该多幅图像中所示的瞳孔的相对质量和该多幅图像中用户的皮肤的相对质量相对应的标准。例如，如果眼睛在该多幅图像中聚焦且未被遮挡，则系统(例如，头戴式视图器)选择眼睛用于PPG测量；并且如果眼睛在该多幅图像中失焦或被遮挡，则系统选择皮肤用于PPG测量。(B7) In some embodiments of any one of B1 to B6, the one or more quality criteria include criteria corresponding to a relative quality of a pupil shown in the plurality of images and a relative quality of the user's skin in the plurality of images. For example, if the eye is in focus and unobstructed in the plurality of images, the system (e.g., a head mounted view device) selects the eye for PPG measurement; and if the eye is out of focus or obstructed in the plurality of images, the system selects the skin for PPG measurement.
(B8)在B1至B7中任一项的一些实施例中，该方法还包括：(i)从该多幅图像确定眼动追踪信息；以及(ii)根据该眼动追踪信息更新用户界面。在一些实施例中，摄像头104用于追踪用户的眼睛，并且来自该摄像头104的图像用于确定PPG信息和心率信息。(B8) In some embodiments of any one of B1 to B7, the method further includes: (i) determining eye tracking information from the plurality of images; and (ii) updating the user interface based on the eye tracking information. In some embodiments, camera 104 is used to track the user's eyes, and images from camera 104 are used to determine PPG information and heart rate information.
(B9)在B8的一些实施例中，根据该眼动追踪信息更新该用户界面包括：基于用户的凝视方向来更新该用户界面中的焦点。在一些实施例中，更新用户界面包括根据用户的眼动改变视角和/或视场。在一些实施例中，更新用户界面包括根据用户的眼动选择用户界面元素。例如，根据用户的凝视更新用户界面312中所示的场景(例如，视场)。(B9) In some embodiments of B8, updating the user interface based on the eye tracking information includes: updating the focus in the user interface based on the user's gaze direction. In some embodiments, updating the user interface includes changing the perspective and/or field of view based on the user's eye movement. In some embodiments, updating the user interface includes selecting user interface elements based on the user's eye movement. For example, the scene (e.g., field of view) shown in user interface 312 is updated based on the user's gaze.
(B10)在B1至B9中任一项的一些实施例中，该方法还包括：使用人工现实头戴式视图器的照明源(例如，照明源106和/或108)照射眼睛。例如，头戴式视图器是被设计为限制/防止外部光线进入头戴式视图器与用户的眼睛之间的空间的虚拟现实头戴式视图器。(B10) In some embodiments of any one of B1 to B9, the method further includes: illuminating the eye using an illumination source of the artificial reality head mounted view (e.g., illumination source 106 and/or 108). For example, the head mounted view is a virtual reality head mounted view designed to limit/prevent external light from entering the space between the head mounted view and the user's eyes.
(B11)在B10的一些实施例中，该照明源被配置为在红外波段或近红外波段中的至少一者中工作。在一些实施例中，照明源被配置为发射红光、近红外光和/或红外光。红外光和/或近红外光对大多数用户是不可见的，因此不会干扰用户的观看体验。(B11) In some embodiments of B10, the illumination source is configured to operate in at least one of an infrared band or a near infrared band. In some embodiments, the illumination source is configured to emit red light, near infrared light, and/or infrared light. Infrared light and/or near infrared light are invisible to most users and therefore do not interfere with the user's viewing experience.
(B12)在B1至B11中任一项的一些实施例中，该方法还包括：向用户呈现用户界面，其中，该多幅图像是在该用户正观看该用户界面时采集的。(B12) In some embodiments of any one of B1 to B11, the method further includes: presenting a user interface to a user, wherein the plurality of images are captured while the user is viewing the user interface.
(B13)在B12的一些实施例中，该方法还包括：经由用户界面(例如，用户界面204)向用户呈现心率信息。(B13) In some embodiments of B12, the method further includes: presenting the heart rate information to the user via a user interface (eg, user interface 204).
(B14)在B12或B13的一些实施例中，该用户界面与在该人工现实头戴式视图器上执行的应用(例如，计算系统720的多个应用752中的一应用)相对应。(B14) In some embodiments of B12 or B13, the user interface corresponds to an application (e.g., an application among the plurality of applications 752 of the computing system 720) executed on the artificial reality head-mounted view device.
(B15)在B1至B14中任一项的一些实施例中，面向内的摄像头被配置为在近红外范围内工作。在一些实施例中，面向内的摄像头被配置为在近红外范围和可见光范围内工作。在一些实施例中，面向内的摄像头被配置为在从红外光到可见光的范围内工作。(B15) In some embodiments of any one of B1 to B14, the inward-facing camera is configured to operate in a near infrared range. In some embodiments, the inward-facing camera is configured to operate in a near infrared range and a visible light range. In some embodiments, the inward-facing camera is configured to operate in a range from infrared light to visible light.
(B16)在B1至B15中任一项的一些实施例中，该方法还包括：获取来自用户所佩戴的腕部可穿戴设备(例如，腕部可穿戴设备600和/或可穿戴带610)的辅助心率信息。在一些实施例中，主要PPG信息(例如，PPG 220)是从头戴式视图器获取的，并且辅助PPG信息(例如，PPG 222)是从腕部可穿戴设备获取的。(B16) In some embodiments of any one of B1 to B15, the method further includes: obtaining auxiliary heart rate information from a wrist wearable device worn by the user (e.g., wrist wearable device 600 and/or wearable band 610). In some embodiments, primary PPG information (e.g., PPG 220) is obtained from a head-mounted viewer, and auxiliary PPG information (e.g., PPG 222) is obtained from the wrist wearable device.
(B17)在B16的一些实施例中，该方法还包括：(i)对心率信息和辅助心率信息进行汇总，以获得汇总后的心率信息；以及(ii)向用户呈现该汇总后的心率信息。在一些实施例中，对心率信息和辅助心率信息进行汇总包括：使用心率信息选择第一时间段的心率信息，以及使用辅助心率信息选择第二时间段的心率信息。(B17) In some embodiments of B16, the method further includes: (i) aggregating the heart rate information and the auxiliary heart rate information to obtain aggregated heart rate information; and (ii) presenting the aggregated heart rate information to the user. In some embodiments, aggregating the heart rate information and the auxiliary heart rate information includes: using the heart rate information to select heart rate information for a first time period, and using the auxiliary heart rate information to select heart rate information for a second time period.
(B18)在B16或B17的一些实施例中，该方法还包括：(i)根据辅助心率信息满足一个或多个标准，选择该辅助心率信息作为选定心率信息；(ii)根据该辅助心率信息不满足一个或多个标准，选择心率信息作为选定心率信息；以及(iii)向用户呈现该选定心率信息。(B18) In some embodiments of B16 or B17, the method also includes: (i) selecting the auxiliary heart rate information as the selected heart rate information based on the auxiliary heart rate information satisfying one or more criteria; (ii) selecting the heart rate information as the selected heart rate information based on the auxiliary heart rate information not satisfying one or more criteria; and (iii) presenting the selected heart rate information to the user.
(B19)在B16至B18中任一项的一些实施例中，该方法还包括：(i)根据心率信息满足一个或多个标准，选择该心率信息作为选定心率信息；(ii)根据该心率信息不满足一个或多个标准，选择辅助心率信息作为选定心率信息；以及(iii)向用户呈现该选定心率信息。在一些实施例中，心率信息和辅助心率信息各自被评估质量，并且具有更高的所评估质量的一者成为选定心率信息。例如，在时刻用户界面370内所呈现的心率测量结果可以对应于选定心率信息，该选定心率信息可以是心率信息或辅助心率信息。(B19) In some embodiments of any one of B16 to B18, the method further includes: (i) selecting the heart rate information as the selected heart rate information based on the heart rate information satisfying one or more criteria; (ii) selecting the auxiliary heart rate information as the selected heart rate information based on the heart rate information not satisfying one or more criteria; and (iii) presenting the selected heart rate information to the user. In some embodiments, the heart rate information and the auxiliary heart rate information are each evaluated for quality, and the one with a higher evaluated quality becomes the selected heart rate information. For example, the heart rate measurement result presented in the moment user interface 370 may correspond to the selected heart rate information, which may be the heart rate information or the auxiliary heart rate information.
(B20)在B1至B19中任一项的一些实施例中，心率信息包括关于用户在一时间段内的心率的信息(例如，图2C中所示的从t0到t2的心率信息)。(B20) In some embodiments of any one of B1 to B19, the heart rate information includes information about the user's heart rate over a period of time (eg, the heart rate information from t0 to t2 shown in FIG. 2C ).
在另一方面，一些实施例包括计算系统，该计算系统包括一个或多个处理器和存储器，该存储器耦接到该一个或多个处理器，该存储器存储有一个或多个程序，该一个或多个程序被配置为由该一个或多个处理器执行，并且该一个或多个程序包括用于执行本文所描述的多个方法(例如，以上方法400、450，A1至A17，或B1至B20)中的任何方法的指令。On the other hand, some embodiments include a computing system comprising one or more processors and a memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, and the one or more programs including instructions for performing any of the multiple methods described herein (e.g., methods 400, 450, A1 to A17, or B1 to B20 above).
在又一方面，一些实施例包括非暂态计算机可读存储介质，该非暂态计算机可读存储介质存储有一个或多个程序，该一个或多个程序用于由计算系统的一个或多个处理器执行，该一个或多个程序包括用于执行本文所描述的多个方法(例如，以上方法400、450，A1至A17，或B1至B20)中的任何方法的指令。On the other hand, some embodiments include a non-transitory computer-readable storage medium storing one or more programs for execution by one or more processors of a computing system, the one or more programs including instructions for performing any of the multiple methods described herein (e.g., methods 400, 450, A1 to A17, or B1 to B20 above).
由本文所描述的各设备和/或被配置为执行或使得执行以上参考任何附图所描述的不同实施例的任何设备(下文中称为“设备”)执行的任何数据收集是在用户同意的情况下并且以符合所有适用的隐私法的方式进行的。向用户提供了用于允许设备收集数据的选项以及用于限制或拒绝设备收集数据的选项。用户能够随时选择启用或选择禁用任何数据收集。此外，向用户提供了用于请求删除任何收集到的数据的选项。Any data collection performed by the various devices described herein and/or any device configured to perform or cause the performance of the various embodiments described above with reference to any of the figures (hereinafter referred to as a "device") is performed with the user's consent and in a manner that complies with all applicable privacy laws. The user is provided with the option to allow the device to collect data and the option to limit or deny the device from collecting data. The user can choose to enable or disable any data collection at any time. In addition, the user is provided with the option to request deletion of any collected data.
将理解的是，尽管术语“第一”、“第二”等在本文中可以被用于描述各种元素，但是这些元素不应受这些术语的限制。这些术语仅被用于将一个元素与另一个元素区分开。It will be understood that, although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.
本文中所使用的术语仅出于描述特定实施例的目的，且不旨在对权利要求进行限制。如在对各实施例和所附权利要求书的描述中所使用的，除非上下文另有明确指示，否则单数形式“一”、“一个”和“该”旨在也包括复数形式。还将理解的是，本文中所使用的术语“和/或”是指并且涵盖相关联的所列项中的一个或多个项的任何和所有可能组合。还将理解的是，术语“包括”和/或“包含”在本说明书中使用时，指定存在所陈述的特征、整数、步骤、操作、元素和/或部件，但不排除存在或添加一个或多个其它特征、整数、步骤、操作、元素、部件和/或它们的组合。The terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the claims. As used in the description of each embodiment and the appended claims, unless the context clearly indicates otherwise, the singular forms "one", "an" and "the" are intended to also include plural forms. It will also be understood that the term "and/or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "including" and/or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, elements and/or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and/or their combinations.
如在本文中所使用的，取决于上下文，术语“如果”可以被解释为意味着：“当所陈述的条件先例为真时”；或“一经”所陈述的条件先例为真；或“响应于确定”所陈述的条件先例为真；或“根据确定”所陈述的条件先例为真；或“响应于检测到”所陈述的条件先例为真。类似地，取决于上下文，短语“如果确定[所陈述的条件先例为真]”或“如果[所陈述的条件先例为真]”或“当[所陈述的条件先例为真]时”可以被解释为意味着：“一经确定”所陈述的条件先例为真；或“响应于确定”所陈述的条件先例为真；或“根据确定”所陈述的条件先例为真；或“一经检测到”所陈述的条件先例为真；或“响应于在检测到”所陈述的条件先例为真。As used herein, the term “if” may be interpreted to mean, depending on the context: “when the stated conditional precedent is true”; or “upon” the stated conditional precedent is true; or “in response to determining” the stated conditional precedent is true; or “upon determining” the stated conditional precedent is true; or “in response to detecting” the stated conditional precedent is true. Similarly, depending on the context, the phrase “if it is determined that [the stated conditional precedent is true]” or “if [the stated conditional precedent is true]” or “when [the stated conditional precedent is true]” may be interpreted to mean: “upon determining” the stated conditional precedent is true; or “in response to determining” the stated conditional precedent is true; or “upon determining” the stated conditional precedent is true; or “upon detecting” the stated conditional precedent is true; or “in response to detecting” the stated conditional precedent is true.
出于解释的目的，前面的描述已经参考特定实施例进行了描述。然而，以上的说明性论述并不旨在是穷尽的或将权利要求限定到所公开的精确形式。鉴于上述教导，许多修改和变型是可能的。选择和描述各实施例是为了最好地解释操作的原理和实际应用，从而使本领域其它技术人员能够实现。For purposes of explanation, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the claims to the precise forms disclosed. In view of the above teachings, many modifications and variations are possible. The various embodiments are selected and described in order to best explain the principles of operation and practical application, thereby enabling others skilled in the art to realize.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US63/482,014 | 2023-01-27 | ||
| US18/495,388US20240252050A1 (en) | 2023-01-27 | 2023-10-26 | Techniques for Using Inward-Facing Eye-Tracking Cameras of a Head-Worn Device To Measure Heart Rate, and Systems and Methods Using Those Techniques |
| US18/495,388 | 2023-10-26 |
| Publication Number | Publication Date |
|---|---|
| CN118402771Atrue CN118402771A (en) | 2024-07-30 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410124128.2APendingCN118402771A (en) | 2023-01-27 | 2024-01-29 | Techniques for measuring heart rate using an inwardly facing eye-tracking camera of a head-mounted device, and systems and methods using these techniques |
| Country | Link |
|---|---|
| CN (1) | CN118402771A (en) |
| Publication | Publication Date | Title |
|---|---|---|
| CN120358982A (en) | Detecting neuromuscular signals at a wearable device to facilitate performance of physical activity and methods and systems thereof | |
| US20250103195A1 (en) | Facilitating extended reality target selection by blending gaze and controller raycasting, and systems and methods of use thereof | |
| EP4530700A1 (en) | Head-wearable device configured to accommodate multiple facial profiles by adjusting a depth between a lens and a wearer`s face, and methods of use thereof | |
| CN119847323A (en) | Multimodal input for wearable devices, systems and methods of use thereof | |
| US12437489B2 (en) | Presenting meshed representations of physical objects within defined boundaries for interacting with artificial-reality content, and systems and methods of use thereof | |
| EP4410190A1 (en) | Techniques for using inward-facing eye-tracking cameras of a head-worn device to measure heart rate, and systems and methods using those techniques | |
| CN118402771A (en) | Techniques for measuring heart rate using an inwardly facing eye-tracking camera of a head-mounted device, and systems and methods using these techniques | |
| US12314463B2 (en) | Activation force detected via neuromuscular-signal sensors of a wearable device, and systems and methods of use thereof | |
| US20240329738A1 (en) | Techniques for determining that impedance changes detected at sensor-skin interfaces by biopotential-signal sensors correspond to user commands, and systems and methods using those techniques | |
| US12333096B2 (en) | Coprocessor for biopotential signal pipeline, and systems and methods of use thereof | |
| US12299196B2 (en) | Sensors for accurately interacting with objects in an artificial-reality environment, and systems and methods of use thereof | |
| EP4625025A1 (en) | Facial interface structures for virtual reality headsets, and systems thereof | |
| US20240148331A1 (en) | Systems for detecting fit of a wearable device on a user by measuring the current draw to amplify a biopotential signal sensor and method of use thereof | |
| US20250321639A1 (en) | Sensors for accurately interacting with objects in an artificial-reality environment, and systems and methods of use thereof | |
| US20250237878A1 (en) | Force-offsetting mechanisms for interpupillary distance adjustments in artificial-reality devices | |
| US20240169681A1 (en) | Arrangements of illumination sources within and outside of a digit-occluded region of a top cover of a handheld controller to assist with positional tracking of the controller by an artificial-reality system, and systems and methods of use thereof | |
| US20240386678A1 (en) | Techniques for binocular disparity measurement and correction using selected times and positions for presenting realignment patterns at a head-wearable device | |
| US20250322628A1 (en) | Techniques for interactive visualization for workspace awareness in collaborative authoring of metaverse environments, and systems and methods of use thereof | |
| US20250218110A1 (en) | Systems and methods for optimizing for virtual content occlusion in mixed reality | |
| EP4455829A1 (en) | Strain-locking knit band structures with embedded electronics for wearable devices | |
| US20250314895A1 (en) | Extended reality headset with user-removable temple tips that include a battery and systems thereof | |
| US20250130646A1 (en) | Systems and methods for guiding users to perform detectable biopotential-based gestures tied to specific motor units, and actionable feedback associated therewith | |
| US20250279497A1 (en) | Systems and methods for improved spatial computing systems | |
| CN120712544A (en) | Techniques for determining whether an impedance change detected by a biopotential signal sensor at a sensor-skin interface corresponds to a user command, and systems and methods using these techniques | |
| CN118866429A (en) | Strain-locked knitted band structures with embedded electronics for wearable devices |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |