相关申请的交叉引用CROSS-REFERENCE TO RELATED APPLICATIONS
本申请要求2022年3月30日提交的且标题为“POSE DETECTION USING THERMALDATA(使用热数据进行的姿势检测)”的美国序列号17/708,493的优先权和权益。美国序列号17/708,493是2021年11月2日提交的且标题为“USER INTERFACE FOR DETERMININGLOCATION,TRAJECTORY AND BEHAVIOR(用于对位置、轨迹和行为进行确定的用户界面)”的美国序列号17/516,954的部分延续，并且要求其优先权和利益。美国序列号17/516,954是2021年4月16日提交的且标题为“THERMAL DATAANALYSIS FOR DETERMINING LOCATION,TRAJECTORY AND BEHAVIOR(用于对位置、轨迹和行为进行确定的热数据分析)”的美国序列号17/232,551的部分延续。美国序列号17/232,551是2021年2月18日提交的且标题为“MONITORING HUMAN LOCATION,TRAJECTORY AND BEHAVIOR USING THERMAL DATA(使用热数据对人类的位置、轨迹和行为进行监测)”的美国序列号17/178,784(又名2021年6月1日公布的美国专利号11,022,495)的延续。美国序列号17/178,784要求于2020年3月6日提交的且标题为“MULTI-WIRELESS-SENSOR SYSTEM,DEVICE,AND METHOD FOR MONITORINGHUMAN LOCATION AND BEHAVIOR(用于对人类位置和行为进行监测的多无线传感器系统、设备和方法)”的美国临时序列号62/986,442的优先权。上述的所有公开内容在此通过引用整体并入、用于所有目的。This application claims priority to and the benefit of U.S. Serial No. 17/708,493, filed on March 30, 2022, and entitled “POSE DETECTION USING THERMALDATA.” U.S. Serial No. 17/708,493 is a continuation-in-part of, and claims priority to, U.S. Serial No. 17/516,954, filed on November 2, 2021, and entitled “USER INTERFACE FOR DETERMINING LOCATION, TRAJECTORY AND BEHAVIOR.” U.S. Serial No. 17/516,954 is a continuation-in-part of U.S. Serial No. 17/232,551, filed on April 16, 2021, and entitled “THERMAL DATA ANALYSIS FOR DETERMINING LOCATION, TRAJECTORY AND BEHAVIOR.” U.S. Serial No. 17/232,551 is a continuation of U.S. Serial No. 17/178,784, filed on February 18, 2021, and entitled “MONITORING HUMAN LOCATION, TRAJECTORY AND BEHAVIOR USING THERMAL DATA” (also known as U.S. Patent No. 11,022,495, issued on June 1, 2021). U.S. Serial No. 17/178,784 claims priority to U.S. Provisional Serial No. 62/986,442, filed on March 6, 2020, and entitled “MULTI-WIRELESS-SENSOR SYSTEM, DEVICE, AND METHOD FOR MONITORING HUMAN LOCATION AND BEHAVIOR”. All of the above disclosures are hereby incorporated by reference in their entirety for all purposes.
技术领域Technical Field
本公开总体上涉及使用热数据对姿势进行检测，并且更具体地涉及使用姿势信息进行跌倒检测和其他活动分析。The present disclosure relates generally to gesture detection using thermal data, and more particularly to fall detection and other activity analysis using gesture information.
背景技术Background Art
虽然一些商家可能会尝试使用基本的机器来对进入和离开通向商店的特定门的人的数目进行计数，但这种信息对于对商店内这些人的动作进行分析非常有限。商家可能对更好地了解顾客在商家的商店内的移动、轨迹和活动非常有兴趣。例如，商家可能有兴趣了解商店中特定过道中的某个展示是否吸引了更多顾客到该过道。此外，商家可能有兴趣了解有多少沿着过道#4走的顾客也沿着过道#5走以及有多少沿着过道#4走的顾客略过了过道#5而是接着沿着过道#6走。这些数据可以帮助商家对其运营进行优化并且使利润最大化。While some merchants may attempt to use basic machines to count the number of people entering and leaving a specific door leading to a store, this information is very limited for analyzing the movements of these people within the store. Merchants may be very interested in better understanding the movement, trajectory, and activities of customers within the merchant's store. For example, a merchant may be interested in understanding whether a certain display in a specific aisle in the store attracts more customers to that aisle. In addition, a merchant may be interested in understanding how many customers who walked along aisle #4 also walked along aisle #5 and how many customers who walked along aisle #4 skipped aisle #5 and continued to walk along aisle #6. This data can help merchants optimize their operations and maximize profits.
商家也可能有兴趣更全面地了解商家的商店中与时间相关的一般流量模式。为了帮助商家更好地分配其自己的资源并且对其与合作第三方的业务关系进行优化，商家可能想要了解一天中最繁忙的时间期间、在一周内最繁忙的日子期间、在特定月份期间和/或在特定年份期间的流量模式。此外，为了帮助对异常行为进行识别以及/或者对事故进行实时检测，商家可能想要获得有关流量的空间模式和/或时间模式和占用水平的更多信息。Merchants may also be interested in gaining a more complete understanding of general traffic patterns in their stores as they relate to time. To help merchants better allocate their own resources and optimize their business relationships with partner third parties, merchants may want to understand traffic patterns during the busiest times of the day, during the busiest days of the week, during specific months, and/or during specific years. In addition, merchants may want to obtain more information about spatial and/or temporal patterns of traffic and occupancy levels to help identify abnormal behavior and/or detect incidents in real time.
此外，为了对居民的幸福感进行分析并且确定居民是否有资格独立生活，辅助生活提供者通常希望获得租户的空间和时间移动数据。例如，提供者可能想要基于租户在整个时间内的室内位置来对租户的移动速度进行分析、基于租户的移动来对消耗的总卡路里进行计算、以及/或者对租户的体温进行监测。In addition, assisted living providers often want to obtain spatial and temporal movement data of tenants in order to analyze the well-being of residents and determine whether residents are eligible for independent living. For example, the provider may want to analyze the movement speed of tenants based on their indoor positions over time, calculate the total calories consumed based on the movement of the tenants, and/or monitor the body temperature of the tenants.
对人类姿势进行检测也可能有助于进行跌倒检测和其他活动分析。然而，在私人住宅或涉及隐私的其他私人环境中，通常需要进行姿势检测。在这方面，用户典型地倾向于任何数据收集或扫描仪不获取或存储人员身份信息。如此，需要一种非侵入式技术来在住宅中实现这种跌倒检测分析。在这方面，高分辨率相机或其他技术可能并不可取，因为这种相机可能会使用关键点来获取和/或存储面部数据或其他人员身份信息。关键点可能包括人类骨骼上的点的子集，这需要获取有关人类的详细信息。在低分辨率图像中对关键点进行识别非常困难，并且通常是不可能的，因此典型地需要高分辨率技术来对关键点进行识别。相比之下，住宅环境则倾向于使用低分辨率数据。低分辨率数据的示例可能是热数据。如此，存在一种需要将热数据与可以对热数据进行处理以提供姿势检测的算法结合使用。Detecting human gestures may also be helpful for fall detection and other activity analysis. However, gesture detection is often required in private homes or other private environments where privacy is a concern. In this regard, users typically prefer that any data collection or scanner does not obtain or store personal identity information. Thus, a non-invasive technology is needed to implement such fall detection analysis in homes. In this regard, high-resolution cameras or other technologies may not be desirable because such cameras may use key points to obtain and/or store facial data or other personal identity information. Key points may include a subset of points on a human skeleton, which requires detailed information about humans. It is very difficult and often impossible to identify key points in low-resolution images, so high-resolution technology is typically required to identify key points. In contrast, residential environments tend to use low-resolution data. An example of low-resolution data may be thermal data. Thus, there is a need to use thermal data in combination with an algorithm that can process the thermal data to provide gesture detection.
除了相机之外，其他解决方案也已用于跌倒检测，诸如手表(例如，使用加速度计和陀螺仪)、雷达或激光雷达。然而，手表技术需要人类主动穿戴该技术。此外，雷达的使用可能导致误报(例如，雷达可能由宠物触发)。此外，雷达技术作为检测过程的部分通常无法有效对静止的人进行区别。此外，现有的系统可能需要对密集聚合的数据点进行分析，这通常会导致准确性较低。对这种密集点的分析可能更加昂贵，因为可能需要附加的计算能力以对密集点云中的点进行区分。In addition to cameras, other solutions have been used for fall detection, such as watches (e.g., using accelerometers and gyroscopes), radar, or lidar. However, watch technology requires a human to actively wear the technology. In addition, the use of radar may lead to false alarms (e.g., the radar may be triggered by a pet). In addition, radar technology as part of the detection process is generally not effective in distinguishing stationary people. In addition, existing systems may require analysis of densely aggregated data points, which generally results in lower accuracy. Analysis of such dense points may be more expensive because additional computing power may be required to distinguish points in a dense point cloud.
发明内容Summary of the invention
在各种实施方式中，系统可以实现一种方法，该方法包括：由处理器从传感器接收人类的图像；由处理器接收图像上的边界框的位置(placement，放置)，其中，边界框包含图像中人类的像素数据；由处理器从边界框内获取边界框数据；以及由处理器基于边界框数据来对人类的姿势进行确定。In various embodiments, a system may implement a method comprising: receiving, by a processor, an image of a human being from a sensor; receiving, by the processor, a placement of a bounding box on the image, wherein the bounding box contains pixel data of the human being in the image; obtaining, by the processor, bounding box data from within the bounding box; and determining, by the processor, a posture of the human being based on the bounding box data.
在各种实施方式中，该方法还可以包括由处理器对神经网络进行训练以对图像上的边界框的位置进行预测。该方法还可以包括由处理器使用人类的像素数据、人类的热数据、和环境数据来对神经网络进行训练。该方法还可以包括由处理器基于环境数据来对神经网络的算法进行调整，其中，环境数据包括以下各者中的至少一者：环境温度、室内温度、平面图、非人类热对象、人类的性别、人类的年龄、传感器的高度、人类的衣着或人类的身体体重。In various embodiments, the method may further include training, by the processor, a neural network to predict the location of a bounding box on an image. The method may further include training, by the processor, the neural network using pixel data of a human, thermal data of a human, and environmental data. The method may further include adjusting, by the processor, an algorithm of the neural network based on the environmental data, wherein the environmental data includes at least one of: ambient temperature, room temperature, floor plan, non-human thermal objects, sex of the human, age of the human, height of the sensor, clothing of the human, or body weight of the human.
在各种实施方式中，传感器可以获取有关人类的热数据。用户可以指示图像上的边界框的位置。对姿势进行确定可以是针对由传感器捕获的图像的帧而进行的。对姿势进行确定还可以包括对一段时间内横跨多个帧的聚合姿势进行确定。该图像可以是人类的视频片段的部分。获取边界框数据可以包括：在随着时间的推移或在初始校准会话期间中的至少一者的情况下，获取边界框数据。姿势可以包括以下各者中的至少一者：坐下、站立、躺卧、锻炼、跳舞、跑步或吃饭。In various embodiments, the sensor may acquire thermal data about the human being. A user may indicate a location of a bounding box on an image. Determining a posture may be performed for a frame of an image captured by the sensor. Determining a posture may also include determining an aggregated posture across multiple frames over a period of time. The image may be part of a video clip of the human being. Acquiring bounding box data may include acquiring bounding box data at least one of over time or during an initial calibration session. The posture may include at least one of: sitting, standing, lying, exercising, dancing, running, or eating.
在各种实施方式中，该方法还可以包括：由处理器，基于经聚合的姿势从站立姿势或坐下姿势中的至少一者变化到躺卧姿势并且所述躺卧姿势持续一定时间量，来对跌倒进行确定。该方法可以包括由处理器使用模式识别从图像的多个帧中提取区分特征。该方法可以包括由处理器基于以下各者中的至少一者来对图像的分辨率进行限制：隐私问题、传感器的功耗、像素数据的成本、像素数据的带宽、计算成本或计算带宽。该方法可以包括由处理器对图像中的人类的姿势进行标记。In various embodiments, the method may also include determining, by the processor, a fall based on the aggregated posture changing from at least one of a standing posture or a sitting posture to a lying posture and the lying posture persisting for a certain amount of time. The method may include extracting, by the processor, distinguishing features from multiple frames of the image using pattern recognition. The method may include limiting, by the processor, the resolution of the image based on at least one of the following: privacy issues, power consumption of the sensor, cost of pixel data, bandwidth of pixel data, computational cost, or computational bandwidth. The method may include labeling, by the processor, the posture of humans in the image.
在各种实施方式中，该方法还可以包括：由处理器基于关于来自人类的红外(IR)能量的IR能量数据来对空间中人类的温度进行确定；由处理器对空间中人类的位置坐标进行确定；由传感器系统将人类的位置坐标与固定物的位置坐标进行比较；以及响应于对象的温度在一范围内并且响应于人类的位置坐标不同于固定物的位置坐标，由传感器系统确定该人类(human)是人(human being)。该方法可以包括由处理器对来自人类的头顶热特征的模式的区分特征进行分析，以确定人类的姿势。该方法可以包括由处理器基于像素数据中的温度的变化来对人类的轨迹进行确定，其中，温度被投射到像素网格上。In various embodiments, the method may also include: determining, by a processor, a temperature of a human being in space based on infrared (IR) energy data regarding infrared (IR) energy from the human being; determining, by a processor, position coordinates of the human being in space; comparing, by a sensor system, the position coordinates of the human being with position coordinates of a fixture; and determining, by a sensor system, that the human is a human being in response to the temperature of the object being within a range and in response to the position coordinates of the human being being different from the position coordinates of the fixture. The method may include analyzing, by a processor, distinguishing features of a pattern of thermal signatures from the top of the human being's head to determine a posture of the human being. The method may include determining, by a processor, a trajectory of the human being based on changes in temperature in pixel data, wherein the temperature is projected onto a pixel grid.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
本公开的主题在说明书的结论部分中被特别指出和清楚地要求保护。然而，通过在结合附图考虑时参考详细描述和权利要求，可以最佳地获得对本公开的更完整的理解。The subject matter of the present disclosure is particularly pointed out and distinctly claimed in the concluding portion of the specification. However, a more complete understanding of the disclosure may be best obtained by referring to the detailed description and claims when considered in conjunction with the accompanying drawings.
图1A是根据各种实施方式的作为整个系统的部分的传感器节点的主要部件的示例性示意图。FIG. 1A is an exemplary schematic diagram of the main components of a sensor node as part of an overall system according to various embodiments.
图1B是根据各种实施方式的作为整个系统的部分的网关和微处理器的示例性示意图。1B is an exemplary schematic diagram of a gateway and a microprocessor as part of an overall system according to various embodiments.
图2是根据各种实施方式的示例性数据流图。FIG. 2 is an exemplary data flow diagram according to various embodiments.
图3是根据各种实施方式的示例性系统架构。FIG. 3 is an exemplary system architecture in accordance with various implementations.
图4A和图4B是根据各种实施方式的示例性用户界面。4A and 4B are exemplary user interfaces according to various implementations.
图5是根据各种实施方式的示例性建筑物布局。FIG. 5 is an exemplary building layout according to various embodiments.
图6是根据各种实施方式的示例性建筑物布局，其示出了某些传感器节点和针对每个传感器节点的覆盖区域。6 is an exemplary building layout showing certain sensor nodes and the coverage area for each sensor node, in accordance with various embodiments.
图7是根据各种实施方式的示例性用户界面，其示出了在用户成功登录传感器节点之后，应用程序对检测到物理空间中的传感器节点的响应。7 is an exemplary user interface illustrating an application's response to detecting a sensor node in a physical space after a user successfully logs into the sensor node, in accordance with various embodiments.
图8A和图8B是根据各种实施方式的示例性用户界面，提示用户将可以作为其物理空间的表示的补充视觉信息的任何特定文件上传至应用程序。8A and 8B are exemplary user interfaces that prompt a user to upload any specific files to an application that may serve as supplemental visual information for a representation of their physical space, in accordance with various embodiments.
图9是根据各种实施方式的示例性用户界面，其示出了位于可以用名称标记使得用户理解每个传感器节点在它们的等效空间中的背景信息的空间中的某些传感器节点。9 is an exemplary user interface showing certain sensor nodes located in a space that may be labeled with names so that a user understands the context of each sensor node in their equivalent space, in accordance with various embodiments.
图10是根据各种实施方式的示例性用户界面，其示出了对传感器节点进行设置和校准的能力。10 is an exemplary user interface illustrating the ability to set up and calibrate a sensor node in accordance with various embodiments.
图11A至图11C示出了根据各种实施方式的示例性热特征模式，其包括站立姿势、坐下姿势和躺卧姿势。11A-11C illustrate exemplary thermal signature patterns including a standing posture, a sitting posture, and a lying posture, according to various embodiments.
图12示出了根据各种实施方式的示例性姿势推断过程。FIG. 12 illustrates an exemplary posture inference process in accordance with various implementations.
图13示出了根据各种实施方式的示例性跌倒检测过程。FIG. 13 illustrates an exemplary fall detection process in accordance with various embodiments.
具体实施方式DETAILED DESCRIPTION
在各种实施方式中，系统被配置成对生物在环境中的活动进行定位、跟踪和/或分析。系统不需要输入人员生物识别数据。尽管本公开可以讨论人类活动，但本公开设想对可以提供红外(IR)能量的任何项目诸如例如动物或任何对象进行跟踪。尽管本公开可以讨论室内环境，但系统也可以在室外环境(例如，室外音乐会场地、室外娱乐公园等)或者室外环境和室内环境的混合环境中进行跟踪。In various embodiments, the system is configured to locate, track and/or analyze the activities of organisms in the environment. The system does not require the input of biometric data of the person. Although the present disclosure may discuss human activities, the present disclosure contemplates tracking any item such as, for example, an animal or any object that can provide infrared (IR) energy. Although the present disclosure may discuss indoor environments, the system may also track in outdoor environments (e.g., outdoor concert venues, outdoor amusement parks, etc.) or a mixture of outdoor and indoor environments.
如图1A、图1B和图3中更详细地阐述的，在各种实施方式中，该系统可以包括多个传感器节点102、网关135、微处理器140、计算模块350(例如，云计算模块)、数据库360和/或用户界面370(例如，图4A、图4B、图7、图8、图9和图10)。每个传感器节点102可以包括封围件105、天线110、传感器模块115、开关120、发光二极管(LED)125和电力源130。As described in more detail in Figures 1A, 1B, and 3, in various embodiments, the system may include multiple sensor nodes 102, a gateway 135, a microprocessor 140, a computing module 350 (e.g., a cloud computing module), a database 360, and/or a user interface 370 (e.g., Figures 4A, 4B, 7, 8, 9, and 10). Each sensor node 102 may include an enclosure 105, an antenna 110, a sensor module 115, a switch 120, a light emitting diode (LED) 125, and a power source 130.
在各种实施方式中，传感器模块115可以是任何类型的传感器，诸如热电堆传感器模块。热电堆传感器模块115可以包括，例如，Heimann GmbH传感器模块或PanasonicAMG8833。每个传感器模块115可以被容纳在封围件105中。传感器模块115被配置成通过对来自对象(例如，生物)的IR能量进行检测而从一定距离处对温度进行测量。如果生物有较高的温度，则生物将会发出更多的IR能量。热电堆传感器模块115中的热电堆感测元件可以包括硅芯片上的热电偶。热电偶吸收IR能量并且生成指示IR能量的量的输出信号。如此，较高的温度会导致更多的IR能量被热电偶吸收，从而导致更高的信号输出。In various embodiments, the sensor module 115 can be any type of sensor, such as a thermopile sensor module. The thermopile sensor module 115 can include, for example, a Heimann GmbH sensor module or a Panasonic AMG8833. Each sensor module 115 can be housed in an enclosure 105. The sensor module 115 is configured to measure temperature from a distance by detecting IR energy from an object (e.g., a living being). If the living being has a higher temperature, the living being will emit more IR energy. The thermopile sensing element in the thermopile sensor module 115 can include a thermocouple on a silicon chip. The thermocouple absorbs IR energy and generates an output signal indicating the amount of IR energy. In this way, a higher temperature causes more IR energy to be absorbed by the thermocouple, resulting in a higher signal output.
在各种实施方式中，传感器节点102接口(interface)可以是无线的，以帮助降低与安装相关联的劳动力成本和材料成本。在各种实施方式中，传感器节点102中的每个传感器节点可以从任何电力源130获得电力。电力源130可以为一个或更多个传感器节点102供电。传感器节点102中的每个传感器节点可以单独由电池供电。电池可以是足够低的功率的以使用单个电池(例如，19wh电池)工作超过2年。电池130可以包括来自任何制造商的电池和/或PKCELL电池、D-cell电池或任何其他电池类型。电池130可以被容纳在电池保持器(例如，Bulgin电池保持器)内。该系统还可以对电池130(例如，D-cell电池)的电池电压进行测量。可以使用与天线110(例如Midatronics Dusty PCB天线)一起位于板载上的模拟数字转换器来对电池电压进行测量。当获取到电池电压测量时，系统也可以将时间戳添加至电池电压数据。In various embodiments, the sensor node 102 interface can be wireless to help reduce the labor and material costs associated with installation. In various embodiments, each sensor node in the sensor node 102 can obtain power from any power source 130. The power source 130 can power one or more sensor nodes 102. Each sensor node in the sensor node 102 can be powered by a battery alone. The battery can be low enough power to work for more than 2 years using a single battery (e.g., a 19wh battery). The battery 130 can include batteries from any manufacturer and/or PKCELL batteries, D-cell batteries, or any other battery type. The battery 130 can be housed in a battery holder (e.g., a Bulgin battery holder). The system can also measure the battery voltage of the battery 130 (e.g., a D-cell battery). The battery voltage can be measured using an analog-to-digital converter located on-board with an antenna 110 (e.g., a Midatronics Dusty PCB antenna). When the battery voltage measurement is obtained, the system can also add a timestamp to the battery voltage data.
通过将更多传感器节点102添加至传感器节点102阵列，该系统可以扩展至更大的占用空间。在各种实施方式中，传感器节点102可以被动态地添加，其中在图7中阐述了用于添加传感器的示例性用户界面。特别地，用户可以在任何时间或任何地点向已建立的传感器节点102的网络添加传感器节点102或从已建立的传感器节点102的网络中去除传感器节点102。传感器节点102可以位于任何地方，只要传感器节点102位置在网关135的带宽内即可。如此，大量传感器节点102可以形成网状网络以与网关135通信。传感器节点102可以大约同时与网关135通信。连接至网关135的每个新传感器节点102可以使网状网络的边界进一步延伸，提高系统稳定性并且提高系统性能。如此，大量传感器节点102可以形成网状网络以与网关135通信。传感器节点102可以大约同时与网关135通信。连接至网关135的每个新传感器节点102使网状网络的边界进一步延伸，提高系统稳定性并且提高系统性能。传感器节点102可以被安置或安装在建筑物的任何部分上或任何对象上。例如，可以使用本领域中已知的任何紧固件将传感器节点102安装至期望空间的天花板、侧壁或地板中。对于2.5米高的天花板，传感器节点102之间的距离可以是间隔开例如4米。By adding more sensor nodes 102 to the array of sensor nodes 102, the system can be expanded to a larger footprint. In various embodiments, sensor nodes 102 can be added dynamically, wherein an exemplary user interface for adding sensors is set forth in FIG. 7. In particular, a user can add sensor nodes 102 to an established network of sensor nodes 102 or remove sensor nodes 102 from an established network of sensor nodes 102 at any time or any place. The sensor nodes 102 can be located anywhere, as long as the sensor node 102 location is within the bandwidth of the gateway 135. In this way, a large number of sensor nodes 102 can form a mesh network to communicate with the gateway 135. The sensor nodes 102 can communicate with the gateway 135 at about the same time. Each new sensor node 102 connected to the gateway 135 can further extend the boundaries of the mesh network, improve system stability and improve system performance. In this way, a large number of sensor nodes 102 can form a mesh network to communicate with the gateway 135. The sensor nodes 102 can communicate with the gateway 135 at about the same time. Each new sensor node 102 connected to the gateway 135 further extends the boundaries of the mesh network, improves system stability and improves system performance. The sensor nodes 102 can be placed or installed on any part of the building or on any object. For example, the sensor nodes 102 can be installed in the ceiling, side walls or floor of the desired space using any fasteners known in the art. For a 2.5 meter high ceiling, the distance between the sensor nodes 102 can be spaced apart, for example, 4 meters.
在各种实施方式中，被安装在任何给定空间中的每个传感器节点102可以具有分配给传感器节点102的唯一编号(例如，MAC地址)。该系统使用唯一编号来创建与已编号的传感器节点102的结构化网络。如图6中所示的，每个传感器节点可以覆盖不同的区域并且携带由用户在数字环境中以及也在物理环境中两者能够访问的该唯一编号。特别地，该唯一编号被清楚地印刷在该传感器节点的封围件105上，并且当用户正在该安装传感器节点时该唯一编号公布在用户的屏幕上。这样，用户可以对他们的物理传感器节点的位置是否与他们已经在安装应用程序中创建的电子空间中的数字表示相匹配进行识别。In various embodiments, each sensor node 102 installed in any given space can have a unique number (e.g., a MAC address) assigned to the sensor node 102. The system uses the unique number to create a structured network with numbered sensor nodes 102. As shown in FIG. 6, each sensor node can cover a different area and carry the unique number that can be accessed by the user both in the digital environment and in the physical environment. In particular, the unique number is clearly printed on the enclosure 105 of the sensor node and is posted on the user's screen when the user is installing the sensor node. In this way, users can identify whether the location of their physical sensor nodes matches the digital representation in the electronic space they have created in the installation application.
如图7和图8B中阐述的，在系统中添加并设置传感器节点102之后，传感器节点102创建简档(profile)。用户被提示要么扫描位于传感器上的QR码，从而自动地注册物理传感器的数字表示，要么手动地输入所讨论的传感器的MAC地址。传感器节点102简档可以包括传感器节点102的高度、mac地址、空间中的相对位置、周围对象和/或背景信息。系统基于用户输入作为其简档中的信息的传感器节点102的高度来确定每个传感器节点的覆盖范围的程度。背景信息的一些示例可以包括传感器所在房间的名称、传感器本身的名称(如果用户希望分配一个名称)以及分配给传感器的编号。用户可以上传有关周围环境的文件，如图8A中所示的。这种文件可以包括例如PDF、JPG、PNG、3DM、OBJ、FBX、STL和/或SKP。周围信息可以包括传感器节点102场内的家具、传感器节点102场周围的建筑布局等。用户可以自行决定添加空间内的周围对象。传感器节点可能不会注册周围对象；然而，周围对象可以为用户自己的人员使用提供更丰富的视觉环境。传感器节点102简档可以被存储在数据库360中的用户的简档中。As illustrated in FIGS. 7 and 8B , after adding and setting up a sensor node 102 in the system, the sensor node 102 creates a profile. The user is prompted to either scan a QR code located on the sensor, thereby automatically registering a digital representation of the physical sensor, or manually enter the MAC address of the sensor in question. The sensor node 102 profile may include the height, mac address, relative position in space, surrounding objects, and/or contextual information of the sensor node 102. The system determines the extent of coverage of each sensor node based on the height of the sensor node 102 that the user enters as information in its profile. Some examples of contextual information may include the name of the room in which the sensor is located, the name of the sensor itself (if the user wishes to assign a name), and the number assigned to the sensor. The user may upload a file about the surrounding environment, as shown in FIG. 8A . Such a file may include, for example, PDF, JPG, PNG, 3DM, OBJ, FBX, STL, and/or SKP. Surrounding information may include furniture within the sensor node 102 field, building layout around the sensor node 102 field, and the like. The user may decide to add surrounding objects within the space. The sensor node 102 profile may not register surrounding objects; however, the surrounding objects may provide a richer visual environment for the user's own personnel use. The sensor node 102 profile may be stored in the user's profile in the database 360.
热电堆传感器模块115可以将对象的温度投射到网格上。该网格可以是8像素×8像素网格、16像素×16像素网格、或32像素×32像素网格(分别为64像素、256像素、或1024像素)。热电堆传感器模块115可以被调谐以对特定的热谱进行检测，从而允许对具有标准温度的对象(例如人类身体)进行检测。平均正常体温通常被公认为98.6°F(37℃)。然而，正常体温可以具有很广的范围，从97°F(36.1℃)至99°F(37.2℃)。较高的温度通常表明感染或疾病。该系统可以检测到这种温度差异，因为传感器模块115可以具有0.5℃的精度。在多个人类身体处于同一区域中的情况下，热电堆传感器模块115将每个身体捕获并处理为不同的热量源。特别地，该系统通过包括内置到3D前端部中的校准过程(示例性用户界面在图10中示出)来避免来自不同身体的体温读数重叠。The thermopile sensor module 115 can project the temperature of the object onto a grid. The grid can be an 8 pixel × 8 pixel grid, a 16 pixel × 16 pixel grid, or a 32 pixel × 32 pixel grid (64 pixels, 256 pixels, or 1024 pixels, respectively). The thermopile sensor module 115 can be tuned to detect a specific thermal spectrum, allowing detection of objects with standard temperatures (such as human bodies). The average normal body temperature is generally recognized as 98.6°F (37°C). However, normal body temperature can have a wide range, from 97°F (36.1°C) to 99°F (37.2°C). Higher temperatures usually indicate infection or disease. The system can detect such temperature differences because the sensor module 115 can have an accuracy of 0.5°C. In the case of multiple human bodies in the same area, the thermopile sensor module 115 captures and processes each body as a different heat source. In particular, the system avoids overlap of temperature readings from different bodies by including a calibration process built into the 3D front end (an exemplary user interface is shown in FIG. 10 ).
作为校准过程(示例性用户界面在图10中示出)的一部分，通过应用程序的界面要求用户退出所有传感器模块115的覆盖范围，使得系统可以自动地对传感器模块115的灵敏度进行调整。从最大灵敏度开始，系统逐渐降低其灵敏度，直到不再检测到高频噪声。噪声的绝对消除允许将身体检测为不同的热量源，以及随后，将诸如人类身体等的两个热量源检测为不同且独立的实体。对于两个传感器模块115的“视野”之间的空间重叠读数，在校准过程期间，系统对两个传感器模块115之间的重叠区域进行识别并且对两个传感器模块115之间的共同检测进行平均。在超过两个传感器模块115之间检测到重叠的情况下，系统将按顺序对每对的重叠进行平均。例如，对于传感器模块115A、B和C之间的重叠，系统先对A和B进行平均，然后继续对AB的结果与C进行平均。As part of the calibration process (an exemplary user interface is shown in FIG. 10 ), the user is asked through the application's interface to exit coverage of all sensor modules 115 so that the system can automatically adjust the sensitivity of the sensor modules 115. Starting from maximum sensitivity, the system gradually reduces its sensitivity until high frequency noise is no longer detected. The absolute elimination of noise allows the body to be detected as a distinct heat source, and subsequently, two heat sources such as a human body to be detected as distinct and separate entities. For spatially overlapping readings between the "field of view" of two sensor modules 115, during the calibration process, the system identifies the overlap region between the two sensor modules 115 and averages the common detections between the two sensor modules 115. In the event that an overlap is detected between more than two sensor modules 115, the system will average the overlap of each pair in sequence. For example, for overlap between sensor modules 115 A, B, and C, the system averages A and B first, and then proceeds to average the results of AB with C.
如果在校准过程(示例性用户界面在图10中示出)期间自动校准失败，系统会自动地生成已安装的传感器节点之间的数字路径，从而提示用户物理上站在每个传感器节点下方。当这样做时，传感器节点102对用户的移动进行检测并且该检测在应用程序内变得可见。如果成功，则提示用户遵循路径并且因此完成对网络的每个传感器节点102的校准。如果任何传感器没有如上面所提及的进行响应，则会提示用户通过滑动数字条以及相应地对所讨论的传感器节点的灵敏度水平进行调整来对传感器模块115的灵敏度进行数字地操纵。更具体地，在校准过程的故障排除期间，提示用户站在物理传感器下方，在其视野的每个四个角落处——一次一个角落。在应用程序中默认命名为A、B、C和D的每个角落处，只要系统检测到他们的存在并且成功地将其打印在传感器的数字替身上，用户就会被呼叫停留。一旦检测到位置，应用程序就会询问用户他们试图标记的四个可能的角落中的哪一个。If the automatic calibration fails during the calibration process (an exemplary user interface is shown in FIG. 10 ), the system automatically generates a digital path between the installed sensor nodes, prompting the user to physically stand under each sensor node. When doing so, the sensor node 102 detects the user's movement and the detection becomes visible within the application. If successful, the user is prompted to follow the path and thus complete the calibration of each sensor node 102 of the network. If any sensor does not respond as mentioned above, the user is prompted to digitally manipulate the sensitivity of the sensor module 115 by sliding a digital bar and adjusting the sensitivity level of the sensor node in question accordingly. More specifically, during troubleshooting of the calibration process, the user is prompted to stand under the physical sensor at each of the four corners of its field of view - one corner at a time. At each corner, which is named A, B, C and D by default in the application, the user is called to stop as long as the system detects their presence and successfully prints it on the digital double of the sensor. Once the location is detected, the application asks the user which of the four possible corners they are trying to mark.
在各种实施方式中，热电堆传感器模块115可以对温度读数的阵列进行检测。除了基于像素值来对生物的温度进行检测之外，热电堆传感器模块115也可以获得传感器模块115所处环境的温度。该系统使用局部信息和全局信息——单独地每个像素及整个形成网络的所有传感器模块115的像素两者)来确定背景温度场是什么。热电堆传感器模块115可以获得传感器节点102本身的独立温度测量。传感器模块115的温度可以使用板载热电偶而获得。该系统可以使用传感器节点102的温度和/或传感器模块115的温度以给出对被监测的空间的温度概况(profile)的评估。板载热电偶测量本身对传感器所在位置处的空间的温度进行测量。该系统使用双线性插值来对空间中传感器节点102和/或传感器模块115之间的温度进行估计，以近似温度分布。此外，在各种实施方式中，系统可以全天多次对环境的温度进行测量和捕获，以便减少为阈值计算保持固定背景温度场的不利影响，从而增加了在环境温度是动态的现实世界场景中整体检测的准确性。In various embodiments, the thermopile sensor module 115 can detect an array of temperature readings. In addition to detecting the temperature of a living being based on pixel values, the thermopile sensor module 115 can also obtain the temperature of the environment in which the sensor module 115 is located. The system uses local information and global information (both individually for each pixel and across all the pixels of the sensor modules 115 that form the network) to determine what the background temperature field is. The thermopile sensor module 115 can obtain independent temperature measurements of the sensor node 102 itself. The temperature of the sensor module 115 can be obtained using an onboard thermocouple. The system can use the temperature of the sensor node 102 and/or the temperature of the sensor module 115 to give an assessment of the temperature profile of the monitored space. The onboard thermocouple measurement itself measures the temperature of the space where the sensor is located. The system uses bilinear interpolation to estimate the temperature between the sensor nodes 102 and/or sensor modules 115 in the space to approximate the temperature distribution. Additionally, in various embodiments, the system may measure and capture the temperature of the environment multiple times throughout the day to reduce the adverse effects of maintaining a fixed background temperature field for threshold calculations, thereby increasing overall detection accuracy in real-world scenarios where the ambient temperature is dynamic.
在各种实施方式中，多个传感器节点102可以实时提供信息以帮助产生人类活动的实时位置、轨迹和/或行为分析。通过采用多个传感器节点102，并且基于网络的密度，系统可以推断出由传感器节点102检测到的任何移动对象的轨迹。如上面所提及的，在传感器节点内部的热电堆传感器模块115被设计成通过检测对象的红外(IR)能量而从一定距离处测量温度。温度越高，发出的IR能量越多。由硅芯片上的小热电偶组成的热电堆传感器模块115吸收能量并且生成输出信号。输出信号是小电压，该小电压与传感器前面的发出IR的对象的表面温度成比例。每个热电堆传感器模块115具有64个热电堆，热电堆中的每个热电堆都对由对象发出的IR能量是灵敏的。为了确定轨迹，在各种实施方式中，每个传感器模块115将由传感器模块115捕获的区域划分为多个像素，这些像素在与64个热电堆对齐的方向上以矩形网格组织，64个热电堆中的每个热电堆与上述网格的一个8×8部分相关联。该系统对连续像素的温度的序列变化进行监测。系统确定这种序列变化指示生物的运动。该系统将这种移动记录为空间中的轨迹的形成。网络中的节点越多，关于轨迹所进行的推断就越准确，因为检测到的轨迹不会被“盲点”中断。In various embodiments, multiple sensor nodes 102 can provide information in real time to help generate real-time location, trajectory and/or behavior analysis of human activities. By using multiple sensor nodes 102, and based on the density of the network, the system can infer the trajectory of any moving object detected by the sensor node 102. As mentioned above, the thermopile sensor module 115 inside the sensor node is designed to measure temperature from a certain distance by detecting the infrared (IR) energy of the object. The higher the temperature, the more IR energy is emitted. The thermopile sensor module 115, which consists of small thermocouples on a silicon chip, absorbs energy and generates an output signal. The output signal is a small voltage that is proportional to the surface temperature of the object emitting IR in front of the sensor. Each thermopile sensor module 115 has 64 thermopiles, each of which is sensitive to the IR energy emitted by the object. To determine the trajectory, in various embodiments, each sensor module 115 divides the area captured by the sensor module 115 into a plurality of pixels organized in a rectangular grid in a direction aligned with the 64 thermopiles, each of the 64 thermopiles being associated with an 8×8 portion of the aforementioned grid. The system monitors the sequential changes in the temperature of consecutive pixels. The system determines that such sequential changes indicate the movement of the organism. The system records such movement as the formation of a trajectory in space. The more nodes in the network, the more accurate the inferences made about the trajectory, because the detected trajectory is not interrupted by "blind spots".
计算引擎对人类行为和轨迹进行分析。例如，关于占用控制，系统可以对空间内的总人数进行计算以与已建立的占用要求进行比较。系统对由传感器模块115监测的空间中的所有热量源进行识别，并且将由人产生的所有热量源的数目相加。The calculation engine analyzes human behavior and trajectory. For example, with respect to occupancy control, the system can calculate the total number of people in a space to compare with established occupancy requirements. The system identifies all heat sources in the space monitored by the sensor module 115 and adds up the number of all heat sources generated by people.
关于占用者温度筛查，系统可以通过捕获人员的身体热量来对人员的存在进行检测。一旦检测到这种检测，温度筛查可以包括对传感器模块115的灵敏度的自动调整。应当注意的是，体温筛查可能不同于身体位置检测。体温筛查意指对被检测的人员的升高的体温进行检测，使得灵敏度要求高于仅身体位置检测。With respect to occupant temperature screening, the system may detect the presence of a person by capturing the person's body heat. Once such detection is detected, temperature screening may include automatic adjustment of the sensitivity of the sensor module 115. It should be noted that body temperature screening may be different from body position detection. Body temperature screening means detecting an elevated body temperature of the detected person, resulting in higher sensitivity requirements than body position detection alone.
关于对占用者的体温进行监测，通过使用传感器模块115中更详细的32×32网格来读取眼窝附近区域的温度，系统100可以能够在距离传感器节点102一(1)米的近场中获得用户的体温。为了使系统对眼窝进行定位，可以要求用户直接凝视传感器节点102，从而允许传感器模块115对温度最高的像素进行检测。With respect to monitoring the body temperature of an occupant, the system 100 may be able to obtain the user's body temperature within a near field of one (1) meter from the sensor node 102 by using a more detailed 32×32 grid in the sensor module 115 to read the temperature of the area near the eye sockets. In order for the system to locate the eye sockets, the user may be required to gaze directly at the sensor node 102, thereby allowing the sensor module 115 to detect the pixel with the highest temperature.
关于对占用者移动速度进行分析，系统可以记录人员在传感器节点102的网络下的移动。根据时间产生一系列的“路径点”。该系统使用基于路径点信息行进的距离和行进所述距离所花费的时间，以对所讨论的用户的移动速度进行计算。With regard to analyzing the speed of movement of occupants, the system can record the movement of people under the network of sensor nodes 102. A series of "way points" are generated over time. The system uses the distance traveled based on the way point information and the time taken to travel the distance to calculate the movement speed of the user in question.
关于对基于占用者的移动的燃烧的总卡路里进行计算，用户通过界面370向系统输入诸如占用者的体重、性别和年龄等的信息。系统可以使用所捕获的移动速度和距离(如上面所提及的)，以对所捕获的移动的时间期间燃烧的粗略卡路里进行计算。With respect to calculating total calories burned based on the occupant's movement, the user inputs information such as the occupant's weight, gender, and age to the system via interface 370. The system may use the captured movement speed and distance (as mentioned above) to calculate a rough calculation of calories burned during the time of the captured movement.
行为分析源于如下的事实：通过将结构化网络与实际空间重叠，所捕获的数据变得情境化。例如，系统可以通过将由传感器节点102捕获的实际轨迹和逗留时间与携带有关特定产品和过道的位置的信息的建筑平面图进行交叉引用来了解移动身体的购物行为，如图5和图6中阐述的。特别地，在各种实施方式中，用户界面370允许用户创建所讨论的空间的三维表示，如图5、图6和图9中所示的。例如，杂货商店的所有者可以通过使用位于传感器节点102的场内的特定产品对每个传感器节点102进行命名或“标记”来记录有关冰淇淋冰箱的位置或产品的信息。如此，如果传感器节点#1检测到IR能量持续30秒，然后系统确定人员在传感器节点#1的场内逗留持续30秒。如果传感器节点#1被标记为在冰淇淋冰箱的前面，系统会提供人员在冰淇淋冰箱前面逗留持续30秒的数据。系统输出的示例如图4A和图4B所示。Behavioral analysis stems from the fact that by overlaying the structured network with the actual space, the captured data becomes contextualized. For example, the system can understand the shopping behavior of the mobile body by cross-referencing the actual trajectory and dwell time captured by the sensor node 102 with the building floor plan carrying information about the location of specific products and aisles, as illustrated in Figures 5 and 6. In particular, in various embodiments, the user interface 370 allows the user to create a three-dimensional representation of the space in question, as shown in Figures 5, 6, and 9. For example, the owner of a grocery store can record information about the location or product of an ice cream refrigerator by naming or "tagging" each sensor node 102 with a specific product located in the field of the sensor node 102. Thus, if sensor node #1 detects IR energy for 30 seconds, then the system determines that the person stayed in the field of sensor node #1 for 30 seconds. If sensor node #1 is marked as being in front of the ice cream refrigerator, the system will provide data that the person stayed in front of the ice cream refrigerator for 30 seconds. Examples of system outputs are shown in Figures 4A and 4B.
在各种实施方式中，并且如图1A和图1B所示，多个传感器节点102中的每个传感器节点都可以与模块相互交流(interface)。该模块可以包括例如HTPA32D模块或HTPA16D模块。该模块可以是无线电模块。该模块可能是无线的。该模块可以是硬件模块。In various embodiments, and as shown in FIG. 1A and FIG. 1B , each of the plurality of sensor nodes 102 can interface with a module. The module can include, for example, an HTPA32D module or an HTPA16D module. The module can be a radio module. The module can be wireless. The module can be a hardware module.
传感器节点102可以包括对传感器节点102的供电进行控制的开关120(例如，ALPS)。开关120可以允许系统的制造商关闭传感器节点102的供电，以在从制造商至客户的整个转移或运输过程中节省模块的电池130。在传感器节点102被输送至客户之后，可以安装系统，其中针对传感器节点102的开关120被接通和保持开通。如果客户关闭商店或系统一段时间，则客户可以使用开关120来关掉传感器节点102以节省电池寿命。传感器节点102上的LED 125指示系统状态，诸如，例如开启模式和关闭模式。The sensor node 102 may include a switch 120 (e.g., ALPS) that controls the power supply to the sensor node 102. The switch 120 may allow the manufacturer of the system to turn off the power supply to the sensor node 102 to save the battery 130 of the module during the entire transfer or transportation process from the manufacturer to the customer. After the sensor node 102 is delivered to the customer, the system can be installed with the switch 120 for the sensor node 102 turned on and left on. If the customer closes the store or system for a period of time, the customer can use the switch 120 to turn off the sensor node 102 to save battery life. The LED 125 on the sensor node 102 indicates the system status, such as, for example, an on mode and an off mode.
图2中阐述了根据各种实施方式的通用数据流。传感器节点102可以接收来自环境的原始热数据(步骤205)。对原始热数据进行压缩以创建压缩后的热数据(步骤210)。网关135接收来自传感器节点102的压缩后的热数据。网关135对压缩后的热数据进行解压缩以创建解压缩后的热数据(步骤215)。服务器上的云计算模块350接收解压缩后的热数据并且创建检测数据(步骤220)。服务器上的后处理计算模块接收来自云计算模块350的检测数据。后处理计算模块对检测数据进行处理以创建经后处理的检测数据(步骤225)。后处理计算模块将经后处理的检测数据发送至数据库360。数据库360使用经后处理的检测数据以创建时间序列检测结果数据(步骤230)。系统将背景分析算法应用于时间序列检测结果数据以创建分析结果(步骤235)。系统将API服务380和客户端APP应用于分析结果以获得数据的3D/2D可视化(步骤240)。FIG. 2 illustrates a general data flow according to various embodiments. The sensor node 102 may receive raw thermal data from the environment (step 205). The raw thermal data is compressed to create compressed thermal data (step 210). The gateway 135 receives compressed thermal data from the sensor node 102. The gateway 135 decompresses the compressed thermal data to create decompressed thermal data (step 215). The cloud computing module 350 on the server receives the decompressed thermal data and creates detection data (step 220). The post-processing computing module on the server receives the detection data from the cloud computing module 350. The post-processing computing module processes the detection data to create post-processed detection data (step 225). The post-processing computing module sends the post-processed detection data to the database 360. The database 360 uses the post-processed detection data to create time series detection result data (step 230). The system applies a background analysis algorithm to the time series detection result data to create an analysis result (step 235). The system applies the API service 380 and the client APP to the analysis results to obtain 3D/2D visualization of the data (step 240).
根据各种实施方式，在图3中阐述了包括关于数据流的更多细节的通用系统架构。在硬件中，传感器节点102获取原始数据，然后执行边缘压缩(步骤305)和/或边缘计算(步骤310)以创建消息队列遥测传输(MQTT，Message Queuing Telemetry Transport)原始数据。服务器上的云计算模块350经由网关135接收来自传感器模块115的MQTT原始数据主题1。云计算模块350对MQTT原始数据主题1应用数据拼接和解压缩(步骤315)以创建MQTT原始数据主题2。云计算模块350对MQTT原始数据主题2应用核心算法(步骤320)以创建MQTT结果主题1。云计算模块350对MQTT结果主题1应用世界坐标重映射(步骤325)以创建MQTT结果主题2。云计算模块350将MQTT原始数据主题1、MQTT原始数据主题2、MQTT结果主题1、以及MQTT结果主题2发送至数据库360。influxDB接收数据。数据库360经由背景分析算法对数据应用背景分析(步骤330)以创建背景结果(分析结果)。背景结果被存储在DynamoDB中。DynamoDB还存储传感器节点102简档。数据库360可以响应于对传感器模块115简档的更新或附加设置而应用附加背景分析。API 380从数据库360获得背景结果。API将实时原始数据、实时检测、历史原始数据、历史检测、历史占用、历史流量和/或历史持续时间应用于数据(步骤335)。API将结果发送至用户界面370(例如，在客户端设备上)。用户界面370提供可视化(步骤340)(例如，图4A和图4B)。用户界面370还提供设置界面(步骤345)。设置界面可以提供对传感器模块115简档的更新。用户界面370还提供登录功能(步骤350)。登录功能可以包括AuthO/Firebase。According to various embodiments, a general system architecture including more details about the data flow is set forth in FIG. 3 . In hardware, the sensor node 102 acquires raw data and then performs edge compression (step 305) and/or edge computing (step 310) to create Message Queuing Telemetry Transport (MQTT) raw data. The cloud computing module 350 on the server receives the MQTT raw data topic 1 from the sensor module 115 via the gateway 135. The cloud computing module 350 applies data concatenation and decompression (step 315) to the MQTT raw data topic 1 to create the MQTT raw data topic 2. The cloud computing module 350 applies the core algorithm (step 320) to the MQTT raw data topic 2 to create the MQTT result topic 1. The cloud computing module 350 applies world coordinate remapping (step 325) to the MQTT result topic 1 to create the MQTT result topic 2. The cloud computing module 350 sends the MQTT raw data topic 1, the MQTT raw data topic 2, the MQTT result topic 1, and the MQTT result topic 2 to the database 360. influxDB receives the data. Database 360 applies background analysis (step 330) to the data via a background analysis algorithm to create background results (analysis results). The background results are stored in DynamoDB. DynamoDB also stores sensor node 102 profiles. Database 360 can apply additional background analysis in response to updates or additional settings to the sensor module 115 profile. API 380 obtains background results from database 360. API applies real-time raw data, real-time detection, historical raw data, historical detection, historical occupancy, historical traffic, and/or historical duration to the data (step 335). API sends the results to user interface 370 (e.g., on a client device). User interface 370 provides visualization (step 340) (e.g., Figures 4A and 4B). User interface 370 also provides a settings interface (step 345). The settings interface can provide updates to the sensor module 115 profile. User interface 370 also provides a login function (step 350). The login function can include AuthO/Firebase.
更特别地，传感器模块115可以对传感器模块115数据进行收集、对数据进行预处理以及/或者将所收集的传感器模块115数据发送至网关135。该模块可以包括板载微处理器140。来自传感器模块115的原始数据可以被保存在微处理器140的RAM中。RAM用作系统的临时存储器。微处理器140被配置成通过消除原始数据中的异常值来对原始数据进行预处理。More specifically, the sensor module 115 may collect sensor module 115 data, pre-process the data, and/or send the collected sensor module 115 data to the gateway 135. The module may include an onboard microprocessor 140. The raw data from the sensor module 115 may be stored in the RAM of the microprocessor 140. The RAM serves as temporary storage for the system. The microprocessor 140 is configured to pre-process the raw data by eliminating outliers in the raw data.
具体地，微处理器140将限定的统计程序应用于原始数据以获得经处理的数据。在各种实施方式中，该模块使用固件软件进行预处理。固件软件以统计方式确定温度读数的异常值。例如，可以通过从帧的平均值中减去每个像素值来使数据归一化来限定异常值。结果除以帧的标准差。使用双线性插值技术，即，用相邻值的插值的乘积，去除并替换高于标准偏差的三倍或低于标准偏差的三分之一的像素值。像素值被替换而不是被去除，使得输入检测在程序之前和之后是相似的。该技术有助于修复可能由于传感器模块115数据质量的潜在缺陷而导致的小轻微数据问题。固件软件、电路设计和驱动程序的组合使系统能够运行算法，以确定数据帧中的每个数据帧上的表示传感器模块115视野下的人类活动的“感兴趣区域”。感兴趣的区域不是具有特定温度的像素，而是相对于其周围像素具有不同温度(更高或有时更低)的像素。然后使用感兴趣的区域来对经处理的数据进行压缩并且为无线传输准备经压缩的数据。Specifically, the microprocessor 140 applies a defined statistical procedure to the raw data to obtain processed data. In various embodiments, the module uses firmware software for preprocessing. The firmware software statistically determines the outliers of the temperature readings. For example, the outliers can be defined by normalizing the data by subtracting each pixel value from the average value of the frame. The result is divided by the standard deviation of the frame. Use a bilinear interpolation technique, that is, remove and replace pixel values that are higher than three times the standard deviation or lower than one-third of the standard deviation with the product of the interpolation of adjacent values. Pixel values are replaced instead of removed so that input detection is similar before and after the procedure. This technology helps to repair small minor data problems that may be caused by potential defects in the quality of the sensor module 115 data. The combination of firmware software, circuit design, and drivers enables the system to run an algorithm to determine the "region of interest" on each data frame in the data frame that represents human activity under the field of view of the sensor module 115. The region of interest is not a pixel with a specific temperature, but a pixel with a different temperature (higher or sometimes lower) relative to its surrounding pixels. The region of interest is then used to compress the processed data and prepare the compressed data for wireless transmission.
该系统可以包括十个数据帧的滚动高速缓存以进行预处理。更具体地，微处理器140的固件可以使用所捕获的数据的最新十个数据帧来进行预处理程序和后处理程序。由于板载上的RAM内存(memory)的量有限(例如，应用程序需要8kb)，系统可能仅对数据的子集进行处理。The system may include a rolling cache of ten frames of data for pre-processing. More specifically, the firmware of the microprocessor 140 may use the most recent ten frames of data captured to perform pre-processing and post-processing routines. Due to the limited amount of RAM memory on board (e.g., 8kb required for the application), the system may only process a subset of the data.
通过网关135的数据可以上传至服务器上的云计算模块350。网关135可以由任何电力源供电。在各种实施方式中，网关135由110V插座(outlet)供电。网关135包括经由以太网、wifi和/或蜂窝连接而连接至网络(例如，互联网)的模块。如此，网关135可以将数据上传至任何数据库360、服务器和/或云。网关135将经预处理的数据和经压缩的数据发送至云计算模块350中的计算引擎，该计算引擎又将结果输出至数据库360s。网关135从服务器拉取操作命令以执行管理功能，诸如软件更新、命令模块打开和关闭传感器模块115、改变采样频率等。The data passed through the gateway 135 can be uploaded to the cloud computing module 350 on the server. The gateway 135 can be powered by any power source. In various embodiments, the gateway 135 is powered by a 110V outlet. The gateway 135 includes a module connected to a network (e.g., the Internet) via Ethernet, wifi, and/or cellular connections. In this way, the gateway 135 can upload data to any database 360, server, and/or cloud. The gateway 135 sends the pre-processed data and the compressed data to the computing engine in the cloud computing module 350, which in turn outputs the results to the database 360s. The gateway 135 pulls operating commands from the server to perform management functions, such as software updates, command modules to turn on and off sensor modules 115, change sampling frequencies, etc.
在各种实施方式中，网关135捕获传输中的经压缩的原始数据并且将该经压缩的原始数据发送至在处理器(例如，Raspberry Pi 4，型号BCM2711)上运行的算法，然后将信息转发至服务器(例如，云计算)用于进一步处理。服务器上的对数据的处理包括对经压缩的原始数据进行解码、根据每个传感器模块115的固件和环境设置使传感器模块115的温度数据归一化、对对象进行检测、对对象进行分类、进行空间变换以用于世界坐标系统定位和融合、多传感器模块115数据融合、对象跟踪和轨迹生成、对异常值像素级读数的清理、以及其他后处理。In various embodiments, the gateway 135 captures the compressed raw data in transit and sends the compressed raw data to an algorithm running on a processor (e.g., Raspberry Pi 4, model BCM2711), which then forwards the information to a server (e.g., cloud computing) for further processing. The processing of the data on the server includes decoding the compressed raw data, normalizing the temperature data of the sensor modules 115 according to the firmware and environmental settings of each sensor module 115, detecting objects, classifying objects, performing spatial transformations for world coordinate system positioning and fusion, multi-sensor module 115 data fusion, object tracking and trajectory generation, cleaning of outlier pixel-level readings, and other post-processing.
在各种实施方式中，处理步骤使用经解压缩的原始数据工作。对经压缩的原始数据进行解码优化了数据传输以及电池130的消耗水平。此外，传感器节点102的温度至适当的温度范围的归一化使得处理步骤适应各种质量差异和环境差异(这些差异是从位于空间中不同位置(spot)的传感器节点102期望的)。In various embodiments, the processing steps work using decompressed raw data. Decoding the compressed raw data optimizes data transmission and the consumption level of the battery 130. In addition, normalization of the temperature of the sensor node 102 to an appropriate temperature range enables the processing steps to adapt to various quality differences and environmental differences (these differences are expected from sensor nodes 102 located at different locations (spots) in space).
云计算模块350的计算引擎的核心处理步骤之一是对象检测和分类。该处理步骤对帧中感兴趣的对象的位置(position)进行检测，并且将对象分类为人员或不同类别的对象(例如，膝上型电脑、咖啡杯等)。从局部坐标系至世界坐标系的空间变换使分析“背景感知”。利用空间变换，该系统可以将空间传感器模块115的覆盖范围与所讨论的空间的实际平面图和3D模型进行比较和交叉引用。多传感器模块115数据融合在多个传感器模块115之间的缺失信息或重叠覆盖范围的情况下统一数据。如上面所提及的，利用各种算法，对象跟踪和轨迹生成能够通过时间将多个人员彼此区分开。对象跟踪和轨迹生成提供了一组轨迹，源于检测到的对象和人员。系统使用这样的轨迹来确定行为分析(例如，逗留的位置和持续时间)、移动速度和方向。后处理步骤解决了检测和跟踪算法中的任何较小的不一致。例如，当轨迹中存在缺失检测或间隙时，后处理步骤有助于将信息拼接在一起并且修补任何损坏的轨迹。One of the core processing steps of the computing engine of the cloud computing module 350 is object detection and classification. This processing step detects the position of the object of interest in the frame and classifies the object into a person or an object of a different category (e.g., a laptop, a coffee cup, etc.). The spatial transformation from the local coordinate system to the world coordinate system makes the analysis "context aware". Using the spatial transformation, the system can compare and cross-reference the coverage of the spatial sensor module 115 with the actual floor plan and 3D model of the space in question. Multi-sensor module 115 data fusion unifies the data in the case of missing information or overlapping coverage between multiple sensor modules 115. As mentioned above, using various algorithms, object tracking and trajectory generation can distinguish multiple people from each other through time. Object tracking and trajectory generation provide a set of trajectories, derived from detected objects and personnel. The system uses such trajectories to determine behavioral analysis (e.g., the location and duration of stay), movement speed and direction. The post-processing step solves any minor inconsistencies in the detection and tracking algorithms. For example, when there are missing detections or gaps in the trajectory, the post-processing step helps to stitch the information together and repair any damaged trajectories.
该系统可以使用热应用协议接口(API)，该热应用协议接口可以位于系统架构中的API层中，如图3中阐述的。该API托管利用传感器模块115解决方案增强的空间的实时和历史人数统计数据。基于REST构建，该API返回JSON响应，并且支持跨域资源共享。该解决方案采用标准HTTP动词以执行CRUD操作，而为了错误指示的目的，API返回标准HTTP响应代码。此外，命名空间用于实现API版本，同时使用令牌身份验证对每个API请求进行身份验证。仪表板上的API令牌被用于对所有API端点进行身份验证。The system may use a hot application protocol interface (API) that may be located in an API layer in the system architecture, as illustrated in FIG3 . The API hosts real-time and historical population statistics for a space enhanced with the sensor module 115 solution. Built on REST, the API returns a JSON response and supports cross-domain resource sharing. The solution uses standard HTTP verbs to perform CRUD operations, and the API returns standard HTTP response codes for error indication purposes. In addition, namespaces are used to implement API versions, while token authentication is used to authenticate each API request. The API token on the dashboard is used to authenticate all API endpoints.
该令牌可能包括在授权HTTP标头中，以字符串文字“令牌”为前缀，两个字符串之间用单个空格分隔。如果API调用中未涉及正确的授权标头，则会生成403错误消息。HTTP授权标头授权：令牌YOUR_API_TOKEN。端点使用标准HTTP错误代码。响应包括有关错误的任何附加信息。The token may be included in an Authorization HTTP header, prefixed with the string literal "token", with the two strings separated by a single space. If the correct Authorization header is not involved in the API call, a 403 error message is generated. HTTP Authorization header Authorization: Token YOUR_API_TOKEN. The endpoint uses standard HTTP error codes. The response includes any additional information about the error.
API列出针对传感器模块115和一段时间的低级“传感器模块115事件”。在每个传感器模块115事件中包括相对于所讨论的传感器模块115的时间戳和轨迹。轨迹不必等于相对于空间(例如，入口或出口)的任何方向。该调用应仅被用于对传感器模块115的性能进行测试。The API lists low-level "sensor module 115 events" for a sensor module 115 and a period of time. Included in each sensor module 115 event is a timestamp and a trajectory relative to the sensor module 115 in question. The trajectory does not have to equal any direction relative to space (e.g., entrance or exit). This call should only be used to test the performance of the sensor module 115.
该API提供关于一周的持续时间内每天进入特定空间的总数的信息。伴随着该间隔的数据和入口总数的分析对象被嵌套在每个结果的区间对象中。该调用可以被用于了解有多少人在一周中的不同日子访问空间。This API provides information about the total number of entries into a specific space each day for the duration of a week. An analytics object is nested in the interval object for each result along with the data for that interval and the total number of entries. This call can be used to understand how many people visit a space on different days of the week.
API记录、统计并列出一整天(或任何24小时时段)从感兴趣空间的所有个人退出。每个结果都带有时间戳和方向(例如，-1)。该调用被用于查明人们何时离开空间。The API records, counts, and lists all individual exits from a space of interest for a whole day (or any 24-hour period). Each result is accompanied by a timestamp and direction (e.g., -1). This call is used to find out when people leave a space.
API提供有关在一天期间的任何给定时间在特定空间的入口处的当前等待时间和历史等待时间的信息。伴随该间隔的数据和所估计的总等待持续时间的分析对象被嵌套在每个结果的间隔对象中。该调用被用于查明有多少人在排队等候以进入不同时间跨度的空间。The API provides information about the current and historical wait times at the entrance of a specific space at any given time during a day. An analytics object along with the data for that interval and the estimated total wait duration is nested in the interval object for each result. This call is used to find out how many people are waiting in line to enter a space at different time spans.
Webhook订阅允许接收服务器上对指定端点的回调。对于事件发生在其中的每个空间，可以在从传感器模块115之一接收到每个事件之后触发webhook。系统可以创建webhook、获取webhook、对webhook进行更新或删除webhook。当接收到webhook时，JSON数据将类似于前一部分中的空间和传感器模块115事件。它将具有附加信息：关联空间的当前计数和空间本身的ID。方向场将是1，表示入口，而-1，表示出口。如果为webhook配置了任何附加标头，则附加标头将被包括在POST请求中。接收到的Webhook数据的示例可以是在连接至两个空间的路径处发生的单个事件。Webhook subscriptions allow callbacks to specified endpoints to be received on the server. For each space in which the event occurs, a webhook can be triggered after each event is received from one of the sensor modules 115. The system can create a webhook, get a webhook, update a webhook, or delete a webhook. When a webhook is received, the JSON data will be similar to the space and sensor module 115 events in the previous section. It will have additional information: the current count of the associated space and the ID of the space itself. The direction field will be 1, indicating an entrance, and -1, indicating an exit. If any additional headers are configured for the webhook, the additional headers will be included in the POST request. An example of received Webhook data would be a single event that occurred at a path connecting two spaces.
在各种实施方式中，该系统可以包括一个或更多个工具，以帮助促进安装、帮助对软件和硬件进行设置、提供更准确的检测、创建虚拟表示、使人类移动可视化、对设备进行测试以及/或者对设备进行故障排除。该系统可以包括任何类型的软件和/或硬件，诸如，例如一个或更多个应用程序、GUI、仪表板、API、平台、工具、基于网络的工具和/或算法。该系统可以是可下载软件的形式。例如，该软件可以是从网站下载的应用程序的形式，该应用程序可以在桌上型电脑或膝上型电脑上使用。该软件还可以包括网络应用程序，该网络应用程序可以经由浏览器被访问。这种网络应用程序可以是设备无关的且自适应的，使得网络应用程序可以是在桌上型电脑、膝上型电脑或移动设备上可访问的。该系统可以通过许可或订阅来获得。一个或更多个登录凭证可以用于部分地或全部地访问系统。In various embodiments, the system may include one or more tools to help facilitate installation, help set up software and hardware, provide more accurate detection, create virtual representations, visualize human movement, test equipment, and/or troubleshoot equipment. The system may include any type of software and/or hardware, such as, for example, one or more applications, GUIs, dashboards, APIs, platforms, tools, network-based tools, and/or algorithms. The system may be in the form of downloadable software. For example, the software may be in the form of an application downloaded from a website that can be used on a desktop or laptop. The software may also include a web application that can be accessed via a browser. Such a web application may be device-independent and adaptive, so that the web application may be accessible on a desktop, laptop, or mobile device. The system may be obtained by licensing or subscription. One or more login credentials may be used to access the system partially or in full.
如本文所使用的，空间可以包括可以由一个或更多个房间组成的区域的整体布局。系统功能可能会分别影响不同的空间。系统可能会关联空间内的多个房间。房间可以包括空间的任何有墙壁的部分(例如，会议室)或空间内的开放区域(例如，热点办公桌区域或走廊)。人头统计可以包括在给定时间范围内进出某个空间或房间的人数。占用可以包括特定时间内房间或空间内部的人数。固定物可以包括家具(例如，椅子、桌子等)或设备(例如，洗衣机、炉灶等)。As used herein, a space may include the overall layout of an area that may be composed of one or more rooms. System functions may affect different spaces separately. The system may associate multiple rooms within a space. A room may include any walled part of a space (e.g., a conference room) or an open area within a space (e.g., a hot desk area or a corridor). Head count may include the number of people entering and exiting a space or room within a given time frame. Occupancy may include the number of people inside a room or space at a specific time. Fixtures may include furniture (e.g., chairs, tables, etc.) or equipment (e.g., washing machines, stoves, etc.).
一般来说，在各种实施方式中，系统可以通过例如使传感器位置可视化、使用3D拖放界面使空间中的覆盖范围可视化、以及/或者了解可能适合于房间或空间的传感器和/或箱体(hives)的数目来规划安装。该系统可以通过例如对空间背景进行分析以区分人类和无生命对象(和其他混杂因素)，从而实现更准确的检测。该系统可以通过接收有关空间例如3D家具、房间和标记/标签的布局的输入来获取空间背景。对空间背景的分析可能涉及人工智能和/或机器学习。AI用于利用空间背景来了解空间，使得人工智能可以对人类的存在、行为、姿势和其他特定活动进行准确地识别。系统可以创建真实位置的虚拟表示(例如，横跨任何仪表板、设置使用算法和API使空间数据可视化的应用程序和任何其他应用程序)。虚拟表示可以基于工具接收用于传感器、房间和空间的标记、标签和/或名称。虚拟表示可以被示出为仪表板和/或API中的唯一标识符。Generally speaking, in various embodiments, the system can plan installation by, for example, visualizing sensor locations, visualizing coverage in a space using a 3D drag-and-drop interface, and/or understanding the number of sensors and/or hives that may be suitable for a room or space. The system can achieve more accurate detection by, for example, analyzing the spatial context to distinguish between humans and inanimate objects (and other confounding factors). The system can obtain the spatial context by receiving input about the layout of the space, such as 3D furniture, rooms, and markers/labels. Analysis of the spatial context may involve artificial intelligence and/or machine learning. AI is used to understand the space using the spatial context so that artificial intelligence can accurately identify human presence, behavior, posture, and other specific activities. The system can create a virtual representation of a real location (e.g., across any dashboard, setting an application that uses algorithms and APIs to visualize spatial data, and any other application). The virtual representation can receive tags, labels, and/or names for sensors, rooms, and spaces based on the tool. The virtual representation can be shown as a unique identifier in the dashboard and/or API.
在各种实施方式中，系统可以通过例如显示当前帧和前一帧(例如，以诸如点的图像形式)、显示和列出传感器、虚拟空间布局和虚拟固定物的背景内的位置坐标、显示人员的轨迹、以及/或者显示人员的姿势(例如，站立、坐下、躺卧)来使人类移动可视化。该系统可以通过例如向用户示出传感器正在检测的内容来对设备进行测试和故障排除。然后，用户可以确认实际对象的类型和位置与视觉表示相关。该系统可以以空间内的人类存在和移动的实时和/或逐帧表示的方式显示传感器正在检测的内容。系统还可以示出传感器何时在线、离线、连接和/或断开连接。In various embodiments, the system can visualize human movement by, for example, displaying the current frame and the previous frame (e.g., in the form of images such as dots), displaying and listing the location coordinates of sensors, virtual space layouts, and virtual fixtures within the background, displaying the trajectory of people, and/or displaying the posture of people (e.g., standing, sitting, lying down). The system can test and troubleshoot the device by, for example, showing the user what the sensors are detecting. The user can then confirm that the type and location of the actual object is related to the visual representation. The system can display what the sensors are detecting in a real-time and/or frame-by-frame representation of human presence and movement within the space. The system can also show when the sensors are online, offline, connected, and/or disconnected.
在各种实施方式中，系统可以包括创建空间布局的功能。创建空间布局可能涉及添加空间和添加空间名称。然后系统会将该空间与其名称一起存储。该系统为用户提供了创建和管理多个空间的功能。在对一栋建筑物中的多个层(例如，第一层、第二层)、具有多个单独房间的设施(例如，老年公寓)、或位于不同物理位置的多个设施(例如，波士顿实验室、旧金山实验室)进行监测时，具有几个分开的空间可能是有用的。在各种实施方式中，作为设置的部分，系统可以提供以下各者中的一项或更多项的能力：对空间进行重命名、添加自动对齐、添加可视化平滑、添加显示局部检测、添加工具栏(例如，主工具栏、侧边工具栏等)、添加来自库中的固定物(例如，来自家具库的成件家具)或者返回到项目库。主工具栏可以包括与房间、传感器、箱体、语言和/或保存相关的功能。示例性侧边工具栏功能可以包括2D或3D、显示或隐藏传感器、显示或隐藏房间、显示或隐藏固定物等。工具栏和功能可以被描述为位于主工具栏或侧边工具栏上，但任何功能都可以与任何工具栏相关联。In various embodiments, the system may include the ability to create a spatial layout. Creating a spatial layout may involve adding a space and adding a space name. The system will then store the space with its name. The system provides users with the ability to create and manage multiple spaces. It may be useful to have several separate spaces when monitoring multiple floors in a building (e.g., first floor, second floor), facilities with multiple separate rooms (e.g., senior citizen apartments), or multiple facilities located in different physical locations (e.g., Boston laboratory, San Francisco laboratory). In various embodiments, as part of the settings, the system may provide the ability to rename the space, add automatic alignment, add visual smoothing, add display local detection, add toolbars (e.g., main toolbar, side toolbar, etc.), add fixtures from the library (e.g., pieces of furniture from the furniture library), or return to the project library. The main toolbar may include functions related to rooms, sensors, boxes, languages, and/or saves. Exemplary side toolbar functions may include 2D or 3D, display or hide sensors, display or hide rooms, display or hide fixtures, etc. Toolbars and features may be described as being located on the main toolbar or on the side toolbar, but any feature can be associated with any toolbar.
在各种实施方式中，系统可以包括用于添加房间以匹配或类似于层布局的功能。系统可能会显示控制面板(例如，响应于选择房间)。控制面板可以允许改变尺寸、对某些位置或特征进行标记、和/或为每个房间选择边框颜色。响应于对房间图标的选择，系统可以向空间添加一个或更多个房间。响应于自动对齐被激活，被移动的对象(例如，固定物，传感器或房间)可以自动地捕捉到附近同类对象的边缘。例如，椅子可以自动地移动到另一对象旁边，以使用户更轻松且更快地以整齐且有序的方式对对象进行排列以匹配平面图。系统可以基于与对象中的每个对象相关联的相似标识符或标记来确定被移动的对象与现有对象是同类的对象。响应于自动对齐被停用，用户可以自由地、增量地移动可能与类似对象没有对齐的对象。In various embodiments, the system may include functionality for adding rooms to match or resemble a floor layout. The system may display a control panel (e.g., in response to selecting a room). The control panel may allow for changing the size, marking certain locations or features, and/or selecting a border color for each room. In response to selecting a room icon, the system may add one or more rooms to the space. In response to automatic alignment being activated, objects being moved (e.g., fixtures, sensors, or rooms) may automatically snap to the edges of nearby similar objects. For example, a chair may be automatically moved next to another object to make it easier and faster for a user to arrange objects in a neat and orderly manner to match a floor plan. The system may determine that the object being moved is of the same type as an existing object based on similar identifiers or tags associated with each of the objects. In response to automatic alignment being deactivated, a user may freely and incrementally move objects that may not be aligned with similar objects.
在各种实施方式中，系统可以包括用于在GUI中向虚拟空间添加固定物的功能(其可以与物理空间中存在的固定物相关)。固定物可以包括，例如，家具或设备，用户可以将该家具和设备添加至空间。固定物可以允许用户将房间区别开并且允许用户将屏幕上看到的移动置于背景中考虑。该系统可以允许用户通过选择家具或设备图标中的任何来添加固定物，然后将固定物拖放到不同房间中的特定位置处。该系统还为用户提供用于虚拟地选择固定物并且然后使用面板控件对固定物进行删除或旋转的功能。该系统还为用户提供用于对固定物的大小或位置进行虚拟地调整的功能。在各种实施方式中，用户可以输入家具的具体坐标，使得系统了解家具的尺寸和位置。此外，当用户移动固定物时，系统可以显示固定物的中央点与放置该固定物的房间的四面墙壁中的每面墙壁之间的距离。家具相对于整个空间的坐标可以通过API存储在系统中。这样，用户可以根据需要从后端提取该信息。用户可以根据需要在空间或房间中添加任意数目的固定物。用户可以将固定物或家具层叠在其他固定物或家具之上。另外，如果物理表特别大，用户可以使用多个虚拟表以匹配其大小。该系统对固定物的存在和坐标进行编码、识别、存储和考虑。当确定检测是否为人类并且确定该检测是否应该计入房间或空间的占用或人头数时，系统会使用这些数据。该系统包括使用固定物类型对每个固定物(例如，桌子、门等)进行编码的功能(例如，使用API和算法)。用户可以从图标中选择固定物和固定物类型。API可以将固定物及其坐标保存在系统上，使得算法可以使用该信息以对检测和行为进行识别。响应于由用户选择的位置，系统对固定物中央点x-y坐标、固定物类型以及从中央点的旋转(以度为单位(例如，0、90、180、270)度)进行记录。旋转可以包括将对象(房间、传感器、固定物)从其中央点旋转0、90、180、270度的动作。可以通过选择圆形箭头以使对象旋转来实现旋转。系统中的固定物可以被设置为处于默认指向方向，该默认指向方向可以与房间中的真实家具的方向不同。用户可以沿着模型的中央点将虚拟家具模型旋转至4个不同的方向(即，从其默认方向旋转0度、90度、180度、270度)。In various embodiments, the system may include functionality for adding fixtures to a virtual space in the GUI (which may be related to fixtures present in the physical space). Fixtures may include, for example, furniture or equipment that a user may add to a space. Fixtures may allow a user to distinguish rooms and allow a user to contextualize the movement seen on the screen. The system may allow a user to add fixtures by selecting any of the furniture or equipment icons and then dragging and dropping the fixtures to specific locations in different rooms. The system also provides a user with functionality for virtually selecting a fixture and then deleting or rotating the fixture using panel controls. The system also provides a user with functionality for virtually adjusting the size or position of a fixture. In various embodiments, a user may enter specific coordinates of a piece of furniture so that the system understands the size and position of the furniture. In addition, when a user moves a fixture, the system may display the distance between the center point of the fixture and each of the four walls of the room in which the fixture is placed. The coordinates of the furniture relative to the entire space may be stored in the system via an API. In this way, a user may extract this information from the backend as needed. A user may add any number of fixtures to a space or room as needed. A user may stack fixtures or furniture on top of other fixtures or furniture. Additionally, if the physical table is particularly large, the user can use multiple virtual tables to match its size. The system encodes, identifies, stores, and takes into account the presence and coordinates of fixtures. The system uses this data when determining whether a detection is human and whether the detection should be counted in the occupancy or headcount of a room or space. The system includes functionality (e.g., using an API and an algorithm) to encode each fixture (e.g., table, door, etc.) using a fixture type. The user can select a fixture and fixture type from an icon. The API can save the fixture and its coordinates on the system so that the algorithm can use this information to identify detections and behaviors. In response to a location selected by the user, the system records the x-y coordinates of the center point of the fixture, the fixture type, and the rotation from the center point in degrees (e.g., 0, 90, 180, 270 degrees). Rotation can include the action of rotating an object (room, sensor, fixture) by 0, 90, 180, 270 degrees from its center point. Rotation can be achieved by selecting a circular arrow to rotate the object. Fixtures in the system can be set to be in a default pointing direction that may be different from the orientation of the real furniture in the room. The user can rotate the virtual furniture model to 4 different orientations (i.e., 0, 90, 180, 270 degrees from its default orientation) along the center point of the model.
系统可以显示人们何时位于成件的家具上、何时在成件的家具周围或何时经过成件的家具。系统存储有与不同固定物相关联的名称或图标。这些名称或图标包括系统在对固定物进行分析时考虑的因素或规则。例如，床图标可以包括人类可以位于床上的规则，而桌子图标可以包括人类不会位于桌子上的规则。其他固定物(具有人类将不在固定物上的规则)的示例包括桌子、柜台、炉灶、冰箱、洗碗机、水槽、散热器、计数器和/或洗涤机。其他固定物(具有人类可以位于固定物上或固定物中的规则)的示例包括床、沙发、椅子、马桶和/或淋浴间。该系统可以基于人员位于家具旁边持续特定的时间量来推断其活动。例如，如果系统检测到人员在电视附近持续一个小时，系统可以推断出该人员喜欢电视上播放的节目。该位置和推断信息还为算法提供了有价值的背景信息，以允许系统推断出日常活动并且能够实现更准确的人类检测。例如，对于日常活动，如果检测到的(例如，以紫色球体表示的)对象位于床固定物的轮廓上或轮廓内，则系统可以推断出人类正在睡觉。然后，系统可以通过确定人员位于床固定物上多长时间来推断出该人员的睡眠时间。如另一示例，为了更准确的人类检测，系统可以对人类热量源和非人类热量源(诸如炉灶和膝上型电脑)两者进行检测。如果发现检测到的热量源位于诸如桌子等固定物的中间，则系统可以识别出这不是人类。因此，系统可以不将此检测计入占用数据的部分(例如，用户可以经由API或仪表板接收该占用数据)。该系统可以包括用于激活或停用“显示局部检测”选项的功能以查看或隐藏检测(例如，紫色球体)。The system can show when people are on, around or passing through pieces of furniture. The system stores names or icons associated with different fixtures. These names or icons include factors or rules that the system considers when analyzing fixtures. For example, a bed icon can include a rule that humans can be on the bed, while a table icon can include a rule that humans will not be on the table. Examples of other fixtures (with rules that humans will not be on fixtures) include tables, counters, stoves, refrigerators, dishwashers, sinks, radiators, counters and/or washing machines. Examples of other fixtures (with rules that humans can be on or in fixtures) include beds, sofas, chairs, toilets and/or showers. The system can infer the activities of people based on the amount of time they are next to furniture. For example, if the system detects that a person is near a TV for an hour, the system can infer that the person likes the shows played on TV. The location and inference information also provides valuable context for the algorithm to allow the system to infer daily activities and enable more accurate human detection. For example, for daily activities, if a detected object (e.g., represented by a purple sphere) is located on or within the outline of a bed fixture, the system can infer that a human is sleeping. The system can then infer the person's sleep time by determining how long the person is on the bed fixture. As another example, for more accurate human detection, the system can detect both human heat sources and non-human heat sources (such as stoves and laptops). If the detected heat source is found to be located in the middle of a fixture such as a table, the system can identify that this is not a human. Therefore, the system may not count this detection as part of the occupancy data (e.g., the user may receive the occupancy data via an API or dashboard). The system may include functionality for activating or deactivating a "show local detection" option to view or hide the detection (e.g., purple sphere).
在各种实施方式中，系统可以包括“可视化平滑”的功能。停用“可视化平滑”使检测球体按照检测到的方式逐帧、准确地显示在特定坐标上。激活“可视化平滑”使紫色球体的逐帧移动以平滑、连续的动画被显示。更特别地，当传感器在物理空间中检测到对象时，系统可以创建出现在虚拟空间中相应位置的“代理”(例如，紫色球体)。系统还可以为该代理分配“生命周期”。“生命周期”可以是紫色球体出现在虚拟空间中以显示检测的时间的长度。在各种实施方式中，这种生命周期可以设置为持续300毫秒，这与传感器发送新的检测的间隔相匹配(或相似)。系统使用物理世界中新的位置的新的检测以对虚拟空间中紫色球体的位置进行更新。如所提及的，系统可以逐帧或通过可视化平滑的方式显示紫色球体。紫色球体可能看起来像是在闪烁，但球体实际上可能正在显示由传感器每300毫秒捕获的实时检测。闪烁效果是由于紫色球体在其300毫秒的生命周期内从不透明变为透明的事实而产生的。如果球体更不透明，则检测可能是较新的。如果人员在实际传感器下站立不动，则紫色球体看起来就像在原地闪烁。如果人员在实际传感器下移动，并且未激活可视化平滑，则系统可能会显示一串紫色球体。例如，紫色球体中的每个紫色球体可能每300毫秒在每个检测坐标(或像素)上连续从不透明到透明。随着可视化平滑的激活，屏幕上出现仅一个紫色球体，并且似乎在线性移动。在300毫秒的生命周期期间，系统会在1英尺半径范围内搜索下一个检测并且将其生命周期“再填充”至另一300毫秒。这意指球体不断出现，按照检测坐标(像素)从一个点移动至另一点。In various embodiments, the system may include a "visual smoothing" feature. Disabling "visual smoothing" causes the detection sphere to be displayed at specific coordinates, frame by frame, exactly as it is detected. Activating "visual smoothing" causes the frame-by-frame movement of the purple sphere to be displayed as a smooth, continuous animation. More specifically, when a sensor detects an object in physical space, the system may create an "agent" (e.g., a purple sphere) that appears at the corresponding location in virtual space. The system may also assign a "lifetime" to the agent. The "lifetime" may be the length of time that the purple sphere appears in virtual space to show the detection. In various embodiments, this lifecycle may be set to last 300 milliseconds, which matches (or is similar to) the interval at which the sensor sends a new detection. The system uses new detections of new locations in the physical world to update the position of the purple sphere in virtual space. As mentioned, the system may display the purple sphere frame by frame or with visual smoothing. The purple sphere may appear to be flickering, but the sphere may actually be displaying real-time detections captured by the sensor every 300 milliseconds. The flickering effect is caused by the fact that the purple sphere changes from opaque to transparent during its 300 millisecond lifecycle. If the sphere is more opaque, the detection is likely newer. If a person is standing still under the actual sensor, the purple sphere will appear to flicker in place. If a person is moving under the actual sensor, and visualization smoothing is not activated, the system may display a string of purple spheres. For example, each of the purple spheres may continuously go from opaque to transparent at each detection coordinate (or pixel) every 300 milliseconds. With visualization smoothing activated, only one purple sphere appears on the screen, and it appears to move linearly. During the 300 millisecond lifecycle, the system searches for the next detection within a 1-foot radius and "refills" its lifecycle to another 300 milliseconds. This means that the spheres continue to appear, moving from one point to another according to the detection coordinates (pixels).
在各种实施方式中，该系统可以包括用于添加一个或更多个箱体和/或热传感器的功能。来自房间外部的数据可能仍经由API进行存储，但房间外部的数据可能不会显示在仪表板上。仪表板显示对空间内的房间特定的活动和占用。因此，系统指示用户将传感器和对象放置在房间内。箱体可以提供网关功能以连接热传感器并且将数据从热传感器传输至存储位置(例如，云)。例如，箱体可以连接至一定数目的传感器(例如，最多12个传感器)，然后可能需要附加的箱体用于附加的传感器。为了便于安装，每个箱体可以预配置有一组传感器(例如，将传感器ID作为传感器数据加载到箱体的数据库中)。预配置过程可以包括两个部分。传感器可以设置为具有与箱体相同的NetID。这使得传感器能够连接至彼此。箱体可以获得编程到配置文件中的传感器MAC地址和传感器模式。这使得箱体对传感器帧速率进行正确地管理。该系统可以允许用户向空间添加一个或更多个箱体。该系统可以包括通过扫描代码(例如，QR码)或通过输入箱体ID(例如，在箱体下面的贴纸上找到的输入箱体ID)将箱体添加至空间的功能。在箱体数据被添加至系统的过程期间，系统还接收预配置的传感器数据(因为箱体数据已预配置为包括传感器数据)。系统使用传感器数据以显示每个箱体的“传感器”图标。系统可以显示来自不同箱体的传感器，因为传感器可以是颜色编码的，以显示传感器属于特定的组和箱体。具体的实现方式可能会发生变化，但通常可以通过系统以视觉方式识别。响应于接收用于箱体的传感器图标的选择，将显示与该箱体相关联的所有传感器，以便可以将每个传感器添加至空间中。用户可以将每个虚拟传感器拖放到房间的某个地方以记录房间占用，或者拖放在门口上以记录人进出该门口的人头数。每个传感器可以是唯一的，并且可以用唯一的地址(例如，MAC地址)来识别。如此，用户应该将虚拟传感器放置在与相应物理传感器的位置相同的位置、与相应物理传感器的取向相同的取向和与相应物理传感器的房间相同的房间。In various embodiments, the system may include functionality for adding one or more boxes and/or thermal sensors. Data from outside the room may still be stored via the API, but data from outside the room may not be displayed on the dashboard. The dashboard displays room-specific activity and occupancy within the space. Therefore, the system instructs the user to place sensors and objects within the room. The box may provide a gateway function to connect to the thermal sensors and transfer data from the thermal sensors to a storage location (e.g., the cloud). For example, a box may be connected to a certain number of sensors (e.g., up to 12 sensors), and then additional boxes may be required for additional sensors. For ease of installation, each box may be pre-configured with a set of sensors (e.g., the sensor ID is loaded as sensor data into the box's database). The pre-configuration process may include two parts. The sensor can be set to have the same NetID as the box. This enables the sensors to connect to each other. The box can obtain the sensor MAC address and sensor mode programmed into the configuration file. This enables the box to properly manage the sensor frame rate. The system may allow the user to add one or more boxes to the space. The system may include functionality to add a box to a space by scanning a code (e.g., a QR code) or by entering a box ID (e.g., an input box ID found on a sticker under the box). During the process of adding the box data to the system, the system also receives preconfigured sensor data (because the box data is preconfigured to include sensor data). The system uses the sensor data to display a "sensor" icon for each box. The system can display sensors from different boxes because the sensors can be color-coded to show that the sensor belongs to a specific group and box. The specific implementation may vary, but it can generally be visually identified by the system. In response to receiving a selection of a sensor icon for a box, all sensors associated with the box will be displayed so that each sensor can be added to the space. The user can drag and drop each virtual sensor somewhere in the room to record room occupancy, or drag and drop it on a doorway to record the number of people entering and exiting the doorway. Each sensor can be unique and can be identified by a unique address (e.g., a MAC address). As such, the user should place the virtual sensor at the same location as the location of the corresponding physical sensor, at the same orientation as the orientation of the corresponding physical sensor, and in the same room as the room of the corresponding physical sensor.
在各种实施方式中，该系统可以包括对虚拟热传感器进行设置和/或校准的功能。每个虚拟热传感器都可以出现在显示器上(例如，作为方块)。响应于虚拟传感器被关闭，虚拟传感器可以以某种方式(例如，黑色方块)出现以指示实际传感器未检测到任何东西。响应于虚拟传感器被打开，虚拟传感器显示为网格(例如，由64个方块组成的8×8网格)。64个方块中的每个方块可以表示每个像素，并且每个像素的颜色可以表示实际传感器在网格中的该点处检测到的温度。每个像素的颜色可以包括一系列色度以指示温度水平。例如，所述一系列色度可以是从黄色(较低温度)至红色(较高温度)。In various embodiments, the system may include functionality for setting up and/or calibrating virtual thermal sensors. Each virtual thermal sensor may appear on the display (e.g., as a square). In response to the virtual sensor being turned off, the virtual sensor may appear in some manner (e.g., a black square) to indicate that the actual sensor is not detecting anything. In response to the virtual sensor being turned on, the virtual sensor is displayed as a grid (e.g., an 8×8 grid consisting of 64 squares). Each of the 64 squares may represent each pixel, and the color of each pixel may represent the temperature detected by the actual sensor at that point in the grid. The color of each pixel may include a range of chromaticities to indicate the temperature level. For example, the range of chromaticities may be from yellow (lower temperature) to red (higher temperature).
响应于接收虚拟传感器的选择，系统显示控制面板(例如，位于显示器的左侧边上)。使用控制面板，系统可以允许用户设置虚拟传感器高度。例如，杂货店中的过道位于视野范围内可能是重要的，但看门人的壁橱可以位于视野范围外部。实际传感器的最佳(和优选的最大)高度为约3.2米。这样的高度可以提供人类检测所需的最大覆盖范围和最佳分辨率。最佳分辨率可以是基于算法从来自传感器的热图图像中对人类存在进行检测的能力。最佳分辨率指系统可以对人类存在进行准确地和可靠地检测的分辨率。虚拟传感器的高度对应于天花板的高度或实际传感器将附接至的墙壁上的高度。传感器在地板之上越高，传感器对地板的覆盖范围越广。传感器距离地板越接低，传感器对地板的覆盖范围越窄。基于高度，系统能够确定传感器正在监测(覆盖)的地板空间有多大。In response to receiving the selection of the virtual sensor, the system displays a control panel (e.g., located on the left side of the display). Using the control panel, the system can allow the user to set the virtual sensor height. For example, it may be important that the aisles in the grocery store are within the field of view, but the janitor's closet may be located outside the field of view. The optimal (and preferred maximum) height of the actual sensor is about 3.2 meters. Such a height can provide the maximum coverage and optimal resolution required for human detection. The optimal resolution can be based on the ability of the algorithm to detect human presence from the thermal image from the sensor. The optimal resolution refers to the resolution at which the system can accurately and reliably detect human presence. The height of the virtual sensor corresponds to the height of the ceiling or the height on the wall to which the actual sensor will be attached. The higher the sensor is above the floor, the wider the coverage of the floor by the sensor. The lower the sensor is from the floor, the narrower the coverage of the floor by the sensor. Based on the height, the system can determine how big the floor space that the sensor is monitoring (covering) is.
系统可以使用公式(90％*2*tan(30°)*高度)^2。例如，3米的天花板高度可以得出地板上3.03m×3.03m的覆盖范围。标准超市天花板高度为5.78m，标准办公室天花板高度为3.12m，并且标准门高度为2.43m。该系统还允许用户对不同的高度进行测试，以确认某些区域位于传感器的视野范围内(或视野范围外部)。110英寸(2.8m)的传感器高度可以提供106英寸(2.7m)的有效覆盖宽度，102英寸(2.6m)的传感器高度可以提供78英寸(2.0m)的有效覆盖宽度，95英寸(2.4m)的传感器高度可以提供63英寸(1.6m)的有效覆盖宽度，以及87英寸(2.2m)的传感器高度可以提供56英寸(1.4m)的有效覆盖宽度。The system can use the formula (90%*2*tan(30°)*height)^2. For example, a ceiling height of 3 meters can result in a coverage area of 3.03m×3.03m on the floor. The standard supermarket ceiling height is 5.78m, the standard office ceiling height is 3.12m, and the standard door height is 2.43m. The system also allows the user to test different heights to confirm that certain areas are within the sensor's field of view (or outside the field of view). A sensor height of 110 inches (2.8m) can provide an effective coverage width of 106 inches (2.7m), a sensor height of 102 inches (2.6m) can provide an effective coverage width of 78 inches (2.0m), a sensor height of 95 inches (2.4m) can provide an effective coverage width of 63 inches (1.6m), and a sensor height of 87 inches (2.2m) can provide an effective coverage width of 56 inches (1.4m).
在各种实施方式中，系统还可以允许用户设置虚拟传感器方向以符合实际传感器进行物理安装的方式，以确保虚拟世界(例如，在设置应用程序中)与物理世界之间的准确表示。现实世界和虚拟世界两者中的传感器方向必须相似，使得可见的检测也匹配。例如，站在房间的东北角的人员必须出现在设置应用程序中相应虚拟传感器像素的东北角。为了帮助匹配传感器方向，物理传感器可以包括箭头(例如，在其安装板上)。当用户将物理传感器添加至设置应用程序作为虚拟传感器时，用户通过使虚拟传感器旋转来将虚拟传感器箭头的方向与物理传感器箭头的方向相匹配。In various embodiments, the system may also allow the user to set the virtual sensor orientation to match the way the actual sensor is physically mounted to ensure accurate representation between the virtual world (e.g., in the settings application) and the physical world. The sensor orientation in both the real world and the virtual world must be similar so that visible detections also match. For example, a person standing in the northeast corner of a room must appear in the northeast corner of the corresponding virtual sensor pixel in the settings application. To help match sensor orientation, the physical sensor can include an arrow (e.g., on its mounting plate). When the user adds the physical sensor to the settings application as a virtual sensor, the user matches the direction of the virtual sensor arrow to the direction of the physical sensor arrow by rotating the virtual sensor.
在各种实施方式中，系统可以包括查看检测的功能。“检测”主要指对人的存在的检测。实际的传感器可以以例如每秒3帧至5帧捕获某个区域的热图。该系统可以通过识别热图的与人类的身体温度(或“热特征”)有关的区域来对人类存在进行检测。系统可以在显示器上表示对人的检测(例如，紫色球体)。人的平均正常体温通常接受为98.6°F(37℃)。然而，正常体温可以具有广泛范围，从97°F(36.1℃)至99°F(37.2℃)。100.4°F(38℃)之上的温度可以意味着该人员可能因感染或疾病而发烧。In various embodiments, the system may include a function to view detections. "Detection" refers primarily to the detection of human presence. The actual sensor may capture a thermal map of a region at, for example, 3 to 5 frames per second. The system may detect human presence by identifying areas of the thermal map that are related to human body temperature (or "thermal signature"). The system may indicate detection of a person on a display (e.g., a purple sphere). The average normal body temperature of a person is generally accepted to be 98.6°F (37°C). However, normal body temperature may have a wide range, from 97°F (36.1°C) to 99°F (37.2°C). Temperatures above 100.4°F (38°C) may mean that the person may have a fever due to an infection or disease.
在各种实施方式中，检测过程可以包括灵敏度调整。该系统可以接收基于位于物理空间或房间内的温度计的有关空间或房间中温度的数据。灵敏度调整可以涉及提高系统在其温度接近人类身体的温度的环境中对人类存在进行检测的能力。通过改变涉及检测(人类温度)与周围环境之间的温差的参数来改善检测。提高灵敏度可以涉及使这种温差最小化，使得系统可以更容易地检测到人类在例如非常温暖的气候中的存在。换言之，在65度的较冷气候下，系统可以更容易地确定比一般室温高30度或更高的任何对象是人类。然而，在96度的较暖气候下，系统会需要检测到具有温度比室温高2度至3度的对象，才能将该对象视为人类。In various embodiments, the detection process may include sensitivity adjustment. The system may receive data about the temperature in a space or room based on a thermometer located in the physical space or room. Sensitivity adjustment may involve improving the system's ability to detect human presence in an environment where its temperature is close to that of the human body. Detection is improved by changing parameters involving the temperature difference between the detection (human temperature) and the surrounding environment. Improving sensitivity may involve minimizing this temperature difference so that the system can more easily detect the presence of humans in, for example, very warm climates. In other words, in a colder climate of 65 degrees, the system may more easily determine that any object that is 30 degrees or more above the general room temperature is a human. However, in a warmer climate of 96 degrees, the system may need to detect an object with a temperature 2 to 3 degrees higher than the room temperature before the object can be considered a human.
该检测信息可以通过系统算法被进一步处理，以再次检查并确保最终发送至API和仪表板的数据是准确的。这种进一步的处理可以包括附加的标准，以过滤掉那些不像人类一样进行行为的检测。系统可以过滤掉具有温度低于人类身体温度范围(例如，从97°F(36.1℃)至99°F(37.2℃))的任何对象。过滤可以包括根本不移动的检测(例如，像炉灶这样的设备)或似乎位于人类不期望位于的固定物(例如，在桌子中间)上的静止检测。例如，系统可以确定桌子上显示的热图更有可能指示膝上型电脑。此外，系统可以存储每个固定物周围的坐标使得，不对具有与任何固定物的坐标重叠的坐标的对象的检测进行计数。换言之，固定物周围的坐标被列入黑名单，以便不对这些坐标内的任何检测进行计数。This detection information can be further processed by the system algorithm to double-check and ensure that the data ultimately sent to the API and dashboard is accurate. This further processing can include additional criteria to filter out detections that do not behave like humans. The system can filter out any objects with a temperature below the human body temperature range (e.g., from 97°F (36.1°C) to 99°F (37.2°C)). Filtering can include detections that do not move at all (e.g., appliances like stoves) or stationary detections that appear to be located on fixtures where humans are not expected to be located (e.g., in the middle of a table). For example, the system may determine that a heat map displayed on a table is more likely to indicate a laptop. In addition, the system can store coordinates around each fixture so that detections of objects with coordinates that overlap with the coordinates of any fixture are not counted. In other words, the coordinates around the fixture are blacklisted so that any detections within these coordinates are not counted.
在各种实施方式中，检测可以包括人头数。根据进入和离开房间的人来确定人头数的传感器(人头数传感器)可以安装在房间的入口门口上方(例如，在面向房间内部的墙壁上)。人头数传感器可以使用与虚拟阈值(或门套线)相关联的数据，该虚拟阈值可以是距门的一定距离。人头数传感器可以仅对穿过门套线的人员的检测进行计数。例如，“In”是在人员从左向右穿过门套线时，而“Out”是在人员从右向左穿过门套线时。门套线减少了对例如可能会把他们的头部伸进门内部看房间内部有什么但人员永远不会完全进入房间的人的错误读数。In various embodiments, detection may include head counting. A sensor that determines head count based on people entering and leaving a room (a head count sensor) may be mounted above an entry doorway to the room (e.g., on a wall facing the interior of the room). The head count sensor may use data associated with a virtual threshold (or door casing), which may be a certain distance from the door. The head count sensor may only count detections of people crossing the door casing. For example, an "In" is when a person crosses the door casing from left to right, while an "Out" is when a person crosses the door casing from right to left. The door casing reduces false readings of people who, for example, may stick their head inside a door to see what is inside the room, but who never fully enter the room.
在各种实施方式中，系统可以包括用于2D或3D视图设置的功能。响应于接收到2D/3D按钮的选择，系统可以以2D或3D形式显示整个空间的全部(或任何部分)。3D视图可以为不熟悉平面图的用户提供更“真实”的空间背景。这种视图还可以提供一种更直观的方式以理解空间、检测和数据。在各种实施方式中，系统可以包括用于显示或隐藏各种特征(诸如，例如，传感器、房间或固定物)的功能。该系统可以包括用于对空间进行编辑、对空间进行管理、以视觉方式传达数据的不同查看模式等的功能。In various embodiments, the system may include functionality for 2D or 3D view settings. In response to receiving a selection of a 2D/3D button, the system may display all (or any portion) of the entire space in 2D or 3D. A 3D view may provide a more "real" spatial context for users who are not familiar with floor plans. Such a view may also provide a more intuitive way to understand spaces, detections, and data. In various embodiments, the system may include functionality for showing or hiding various features (such as, for example, sensors, rooms, or fixtures). The system may include functionality for editing spaces, managing spaces, different viewing modes for visually communicating data, and the like.
在各种实施方式中，系统可以以视觉方式传达有关一定时间段内的人流和/或停留时间的数据。人流可以基于“进”和“出”给定门口(或横跨门套线)并且进入房间或空间的人的数目来传达。人流也可以通过空间内人们移动的路径或轨迹线来传达。该系统可以包括人头数视图，其中可以在虚拟布局上显示进出数目。该系统还可以包括显示一段时间内的多个检测的轨迹视图，从而在显示器中形成检测流。检测流可以用于确定人流的路径。该系统还可以显示直线状的移动轨迹，其中，系统通过多个检测创建线，从而创建表示人流路径的线。停留时间可以是人员在房间或空间的一组坐标中花费的时间量。可以通过对在一定区域内检测到人员在的时间量进行测量来确定停留时间。系统可以通过位置和轨迹对唯一的检测(同一人员)进行推断或识别，即使系统无法确定它是否是先前检测到的“同一”人员。停留时间可以显示为热图，其中颜色越深可以表示在特定区域花费的时间越多，而颜色越浅可以表示在某个区域花费的时间越少。In various embodiments, the system can visually convey data about the flow of people and/or dwell time over a certain period of time. The flow of people can be conveyed based on the number of people who "enter" and "exit" a given doorway (or across the door frame) and enter a room or space. The flow of people can also be conveyed by the path or trajectory line that people move in the space. The system can include a head count view, where the number of in and out can be displayed on a virtual layout. The system can also include a trajectory view that displays multiple detections over a period of time, thereby forming a detection stream in the display. The detection stream can be used to determine the path of the flow of people. The system can also display a linear movement trajectory, where the system creates lines through multiple detections to create lines representing the path of the flow of people. The dwell time can be the amount of time a person spends in a set of coordinates in a room or space. The dwell time can be determined by measuring the amount of time a person is detected in a certain area. The system can infer or identify a unique detection (the same person) by location and trajectory, even if the system cannot determine whether it is the "same" person previously detected. Dwell time can be displayed as a heat map, where darker colors can indicate more time spent in a particular area, and lighter colors can indicate less time spent in an area.
如上面所讨论的，传统计算机视觉使用高分辨率图像来开发人类姿势检测算法。高分辨率图像可以包括例如所记录的高分辨率视频片段。现有的算法中的许多算法都是对关键点(关键点可以是人类骨骼上的点)进行检测以从图像中对人类姿势进行检测，而对关键点的检测需要高分辨率的相机系统。如此，使用降低分辨率的系统对关键点的检测可能是不可行的。然而，可以使用降低分辨率图像的系统可能对促进更多隐私是重要的。As discussed above, traditional computer vision uses high-resolution images to develop human gesture detection algorithms. High-resolution images can include, for example, recorded high-resolution video clips. Many of the existing algorithms detect key points (key points can be points on the human skeleton) to detect human gestures from images, and the detection of key points requires a high-resolution camera system. As such, the detection of key points using a reduced-resolution system may not be feasible. However, a system that can use reduced-resolution images may be important for promoting more privacy.
在各种实施方式中，系统可以通过在降低分辨率的图像上使用姿势检测算法来提供用于姿势检测的功能。例如，系统可以显示人员是站立、坐下、躺卧还是已经跌倒。一般来说，该算法可以包括用于提取包含人类像素的边界框的子模块。该算法还可以包括用于对边界框内部的人类的姿势进行检测的子模块。这些算法可以是基于深度神经网络学习的算法，该基于深度神经网络学习的算法可以是数据驱动的。神经网络可以是CNN(卷积神经网络)，该CNN是用于图像识别和处理中的被特定地设计成对像素数据进行处理的一种人工神经网络。In various embodiments, the system can provide functionality for gesture detection by using a gesture detection algorithm on a reduced resolution image. For example, the system can display whether a person is standing, sitting, lying down, or has fallen. In general, the algorithm can include a submodule for extracting a bounding box containing human pixels. The algorithm can also include a submodule for detecting the gesture of a human inside the bounding box. These algorithms can be algorithms based on deep neural network learning, which can be data-driven. The neural network can be a CNN (convolutional neural network), which is an artificial neural network specifically designed to process pixel data for image recognition and processing.
一般来说，对边界框的边界进行限定是由神经网络学习的行为之一。神经网络可以在具有边界框的图像上进行训练，该边界框可以包含与图像中的人类相对应的所有像素(或像素的子集)。神经网络可以使用该信息来对(多个)边界框应该在图像上的哪里进行预测。更具体地，在各种实施方式中，系统可以接收人类的图像。输入图像/帧可以是低分辨率的，诸如，例如8×8、32×32或64×64像素图像。对于每一帧，人类注释者可以首先对图像中人类(例如，测试主体)的姿势进行观察，对姿势进行标记并且围绕图像中的人绘制边界框。人类注释者可以通过查看所记录的高分辨率视频/图像来确认人类存在于图像中。然而，这种高分辨率图像可能无法用于对算法进行训练。Generally speaking, defining the boundaries of a bounding box is one of the behaviors learned by a neural network. The neural network can be trained on an image with a bounding box that can contain all pixels (or a subset of pixels) corresponding to a human in the image. The neural network can use this information to predict where (multiple) bounding boxes should be on the image. More specifically, in various embodiments, the system can receive an image of a human. The input image/frame can be low resolution, such as, for example, an 8×8, 32×32, or 64×64 pixel image. For each frame, a human annotator can first observe the posture of a human (e.g., a test subject) in the image, mark the posture and draw a bounding box around the person in the image. The human annotator can confirm that a human is present in the image by viewing the recorded high-resolution video/image. However, such high-resolution images may not be used to train the algorithm.
在各种实施方式中，用户可以用整数对姿势进行标记。用户可以在屏幕上输入针对特定姿势的整数。GUI可以包括具有字段的文本框，其接受针对一个或更多个姿势的整数。系统可以将该整数与数据库中的姿势相关联。例如，标记0可以表示坐下姿势，标记1可以表示站立姿势，标记2可以表示躺卧姿势等。在各种实施方式中，用户可以在屏幕上(例如，使用在GUI上接受输入的任何类型的设备)围绕图像中的人类绘制边界框，使得边界框存储为系统可以识别的(x，y)坐标。系统可以使用这种标签和手工注释的数据(例如边界框)以对学习算法进行训练。可以使用例如基于梯度下降的训练来对学习算法进行训练。基于训练，学习算法可以学习如何自动创建边界框、从边界框内收集数据以及从所收集的数据中确定人类姿势。由算法使用的所收集的数据可以是随着时间的推移或在初始校准会话期间在系统运行时收集的。系统可以从部署有传感器的特定环境获取环境数据，因此系统可以使用环境数据来基于特定环境对其算法进行调整。来自环境的数据可以包括来自该环境的参数和/或变量，诸如，例如环境温度、室内温度、平面图、非人类热对象、人类的性别、人类的年龄、所安装的传感器的高度、人类的衣着、人类的身体体重等。像素数据、热数据、和/或从环境收集的环境数据可以用于通过机器学习或人工智能来对该算法进行训练。In various embodiments, a user may label a posture with an integer. A user may enter an integer for a particular posture on the screen. The GUI may include a text box with a field that accepts an integer for one or more postures. The system may associate the integer with a posture in the database. For example, label 0 may represent a sitting posture, label 1 may represent a standing posture, label 2 may represent a lying posture, and the like. In various embodiments, a user may draw a bounding box around a human in an image on the screen (e.g., using any type of device that accepts input on the GUI) so that the bounding box is stored as an (x, y) coordinate that the system can recognize. The system may use such labels and manually annotated data (e.g., bounding boxes) to train a learning algorithm. The learning algorithm may be trained using, for example, training based on gradient descent. Based on the training, the learning algorithm may learn how to automatically create bounding boxes, collect data from within the bounding boxes, and determine human postures from the collected data. The collected data used by the algorithm may be collected over time or during an initial calibration session while the system is running. The system may obtain environmental data from a specific environment in which sensors are deployed, so the system may use environmental data to adjust its algorithm based on the specific environment. Data from the environment may include parameters and/or variables from the environment, such as, for example, ambient temperature, room temperature, floor plan, non-human thermal objects, gender of the human, age of the human, height of the installed sensor, clothing of the human, body weight of the human, etc. Pixel data, thermal data, and/or environmental data collected from the environment may be used to train the algorithm through machine learning or artificial intelligence.
由于低的分辨率，对不同像素之间的图像强度的差异进行确定可能是困难的，因此该算法可能不会尝试基于图像强度来对特征进行确定。相反，该算法可以关注来自人类存在的头顶热特征模式的区分特征。CNN可能会隐式地限定什么构成了“区分特征”。网络的每个层都限定了通过随机梯度下降学习到的“特征图”。用户可能没有意识到网络认为重要的特征。系统可以发现边缘、曲线、相邻像素值的鲜明对比等等。然而，由于神经网络的本质，系统可以没有任何(或很少具有)关于哪些特征对于区分姿势重要的信息。Due to the low resolution, it may be difficult to determine the difference in image intensity between different pixels, so the algorithm may not attempt to determine features based on image intensity. Instead, the algorithm may focus on distinguishing features from the characteristic patterns of overhead thermal signatures of human presence. CNNs may implicitly define what constitutes a "distinguishing feature." Each layer of the network defines a "feature map" learned through stochastic gradient descent. The user may not be aware of the features that the network considers important. The system can discover edges, curves, sharp contrasts in adjacent pixel values, and so on. However, due to the nature of neural networks, the system may not have any (or little) information about which features are important for distinguishing postures.
如上面所讨论的，在各种实施方式中，系统可以提取包括人类的区分特征的区域(例如，该区域被表示为表示边界框的矩形)。边界框可以对由系统分析的数据的量进行限制，并且该框会强制算法仅关注人类轮廓。在算法的这一步中，系统可以关注所提取的边界框，以试图对人类姿势进行分类。系统可以提取每一帧的人类姿势信息。帧可以是由热相机中的一个热相机捕获的单幅图像。As discussed above, in various embodiments, the system can extract a region that includes distinguishing features of a human (e.g., the region is represented as a rectangle representing a bounding box). The bounding box can limit the amount of data analyzed by the system, and the box forces the algorithm to focus only on human outlines. In this step of the algorithm, the system can focus on the extracted bounding box in an attempt to classify the human pose. The system can extract human pose information for each frame. A frame can be a single image captured by one of the thermal cameras.
在各种实施方式中，系统可以对“经聚合的姿势”进行计算以帮助使在一段时间内横跨多个帧的姿势变化平滑。例如，可以基于在给定时间段内收集的姿势集合(例如，集合中出现最频繁的姿势)的模式来对经聚合的姿势进行确定。集合中的所有姿势可能都不相同，但姿势聚合方法会创建共识。该共识可以被称为“经聚合的姿势”。例如，系统可以每5秒获得一次姿势共识。经聚合的姿势可以包括例如坐下、站立或躺卧。系统可以基于姿势的某些变化或姿势持续一定时间量来确定事件为跌倒。例如，如果事件由以下各者组成：(i)经聚合的姿势从站立/坐下变化到躺卧，以及(ii)躺卧姿势持续一定时间量(例如，至少30秒)，则确定该事件为跌倒。In various embodiments, the system may calculate "aggregated postures" to help smooth posture changes across multiple frames over a period of time. For example, aggregated postures may be determined based on a pattern of a collection of postures collected over a given period of time (e.g., the most frequent postures in the collection). All postures in the collection may be different, but the posture aggregation method creates a consensus. This consensus may be referred to as an "aggregated posture." For example, the system may obtain a posture consensus every 5 seconds. Aggregated postures may include, for example, sitting down, standing up, or lying down. The system may determine that an event is a fall based on certain changes in posture or that the posture lasts for a certain amount of time. For example, if an event consists of the following: (i) the aggregated posture changes from standing/sitting down to lying down, and (ii) the lying posture lasts for a certain amount of time (e.g., at least 30 seconds), the event is determined to be a fall.
利用模式识别、模糊逻辑、人工智能和/或机器学习，系统可以基于相似的热特征对人类姿势进行确定。在各种实施方式中，关于模式识别，系统可以从帧中提取区分特征和/或模式，以帮助对帧包含的姿势进行识别。在各种实施方式中，关于人工智能，并且基于所收集的人类注释数据和/或帧，系统可以通过使用CNN来自动地学习这些模式。如图11所阐述的，来自人类热特征的模式可以是不同的，并且不同的模式可以使系统对人类姿势进行不同的分类。例如，图11A示出了指示站立位置的热特征，图11B示出了指示坐下位置的热特征，以及图11C示出了指示躺卧位置的热特征。Using pattern recognition, fuzzy logic, artificial intelligence and/or machine learning, the system can determine human postures based on similar thermal features. In various embodiments, with respect to pattern recognition, the system can extract distinguishing features and/or patterns from frames to help identify the postures contained in the frames. In various embodiments, with respect to artificial intelligence, and based on the collected human annotation data and/or frames, the system can automatically learn these patterns by using CNN. As illustrated in Figure 11, the patterns from human thermal features can be different, and different patterns can cause the system to classify human postures differently. For example, Figure 11A shows thermal features indicating a standing position, Figure 11B shows thermal features indicating a sitting position, and Figure 11C shows thermal features indicating a lying position.
如图12所阐述的，在各种实施方式中，系统可以进行姿势推断过程。系统可以从红外传感器获取输入图像(例如，8×8、32×32或64×64像素图像)(步骤1205)。分辨率的上限和/或下限可以是基于隐私问题、传感器功耗、数据成本、数据带宽、计算成本和/或计算带宽的。系统可以将CNN应用于输入图像(步骤1210)。CNN可以通过随机梯度下降在网络的每个后续层处学习不同的过滤器，从而从图像中隐式地提取重要信息。As illustrated in FIG. 12 , in various embodiments, the system may perform a pose inference process. The system may obtain an input image (e.g., an 8×8, 32×32, or 64×64 pixel image) from an infrared sensor (step 1205). The upper and/or lower limits of the resolution may be based on privacy issues, sensor power consumption, data cost, data bandwidth, computational cost, and/or computational bandwidth. The system may apply a CNN to the input image (step 1210). The CNN may learn different filters at each subsequent layer of the network through stochastic gradient descent, thereby implicitly extracting important information from the image.
系统可以创建图像特征的向量(步骤1215)。该向量可以是由CNN输出的由学习到的图像特征组成的中间结果。可以使用transformer编码器/解码器(步骤1220)以将图像特征的向量转换成框预测(步骤1225)。DETR(DEtectionTRansformer)可以使用传统的CNN主干来对输入图像的2D表示进行学习。在将图像传递至Transformer编码器之前，模型可能会将该图像展平，并且用位置编码对图像进行补充。然后，Transformer解码器可以将少量固定数量的学习到的位置嵌入(例如，对象查询)作为输入，并且附加地参加到编码器输出。系统可以将解码器的每个输出嵌入传递至共享前馈网络(FFN)，该共享前馈网络可以对检测(例如，类和边界框)或“无对象”类进行预测。The system can create a vector of image features (step 1215). The vector can be an intermediate result composed of learned image features output by a CNN. A transformer encoder/decoder (step 1220) can be used to convert the vector of image features into box predictions (step 1225). DETR (DEtectionTRansformer) can use a traditional CNN backbone to learn a 2D representation of the input image. Before passing the image to the Transformer encoder, the model may flatten the image and supplement the image with a positional encoding. The Transformer decoder can then take a small fixed number of learned positional embeddings (e.g., object queries) as input and additionally join the encoder output. The system can pass each output embedding of the decoder to a shared feedforward network (FFN), which can predict a detection (e.g., class and bounding box) or a "no object" class.
在各种实施方式中，系统可以包括框预测，该框预测可以对包括人类的区分特征的边界框的大小和位置进行预测。系统可以获得边界框的内部(步骤1230)。系统获得边界框的内部，因为落在边界框外部的任何东西可能都不是系统感兴趣的，因为系统感兴趣的是对人类的人类姿势进行识别。在各种实施方式中，系统将CNN应用于边界框的内部(步骤1235)。更具体地，应用于边界框的内部的CNN可能会对作为训练给出的图像的特征进行隐式地学习。这些特征对于人类用户来说可以是不可见的，并且可能永远不会被神经网络明确表达出来。基于CNN，系统创建针对图像的姿势预测(步骤1240)。更具体地，在模型的训练阶段期间，系统可以提取特征以对人类姿势进行区分。这是在推断/测试阶段期间使用训练模型以对各种姿势进行区分时使用的先验知识。姿势预测可以包括例如坐下、站立、躺卧或人类的任何其他姿势或配置。系统还可以使用更高的分辨率(32×32像素或更高像素)对其他姿势进行检测。例如，系统可以使用更高的分辨率来获得更好的性能，使得系统可以能够区分站立姿势和坐下姿势。该系统还可以对锻炼、跳舞、跑步、吃饭等活动进行区分。In various embodiments, the system may include a box prediction that may predict the size and position of a bounding box that includes distinguishing features of a human. The system may obtain the interior of the bounding box (step 1230). The system obtains the interior of the bounding box because anything that falls outside the bounding box may not be of interest to the system because the system is interested in recognizing human postures of humans. In various embodiments, the system applies a CNN to the interior of the bounding box (step 1235). More specifically, the CNN applied to the interior of the bounding box may implicitly learn features of the image given as training. These features may be invisible to human users and may never be explicitly expressed by the neural network. Based on the CNN, the system creates a posture prediction for the image (step 1240). More specifically, during the training phase of the model, the system may extract features to distinguish between human postures. This is the prior knowledge used during the inference/testing phase when using the training model to distinguish between various postures. Posture predictions may include, for example, sitting, standing, lying, or any other posture or configuration of a human. The system can also detect other postures using higher resolutions (32×32 pixels or higher). For example, the system can use higher resolution to achieve better performance, allowing the system to distinguish between standing and sitting postures. The system can also distinguish between activities such as exercising, dancing, running, eating, etc.
如图13所阐述的，在各种实施方式中，系统可以执行跌倒检测过程。系统可以获取图像(步骤1305)。例如，所有图像可以在5秒内被聚合。该系统可以对用于聚合图像的时间的量进行基准测试。如果用于聚合图像的时间的量太短，则系统可能会丢失图像细节。如果用于聚合图像的时间的量太长，系统可能会在图像中获取更嘈杂的数据。系统可以不对有关帧中人员的位置做出任何假设。该人员可以是静止的或者正在移动中。系统获取有关每一帧中该人员的姿势的数据。对图像执行上面的姿势推断过程，如图12所阐述的(步骤1310)。对每一帧执行姿势推断算法的结果是该特定帧的姿势预测。在经过姿势推断之后，系统可以得到许多所预测的“姿势”。图12示出其为“5秒内收集的姿势”。在经过“姿势聚合”模块之后，系统可以对一个“经聚合的姿势”进行确定，是站立/坐下还是躺卧。然后系统可以使用该信息以进一步判断是否发生了跌倒。As illustrated in FIG. 13 , in various embodiments, the system may perform a fall detection process. The system may acquire images (step 1305). For example, all images may be aggregated within 5 seconds. The system may benchmark the amount of time used to aggregate the images. If the amount of time used to aggregate the images is too short, the system may lose image details. If the amount of time used to aggregate the images is too long, the system may acquire noisier data in the images. The system may not make any assumptions about the position of the person in the frame. The person may be stationary or moving. The system acquires data about the posture of the person in each frame. The above posture inference process is performed on the image, as illustrated in FIG. 12 (step 1310). The result of performing the posture inference algorithm on each frame is a posture prediction for that particular frame. After posture inference, the system may obtain many predicted "postures". FIG. 12 shows that it is "postures collected within 5 seconds". After passing through the "posture aggregation" module, the system can determine whether a "aggregated posture" is standing/sitting or lying down. The system can then use this information to further determine whether a fall has occurred.
在各种实施方式中，系统可以执行姿势聚合(步骤1315)。更具体地，系统可以基于在给定时间段内收集的姿势的集合的模式来对经聚合的姿势进行确定。例如，集合中出现频率最高的姿势。虽然集合中的所有姿势可能不相同，但姿势聚合方法会创建一个共识，即经聚合的姿势。系统可以对经聚合的姿势进行确定(步骤1320)。经聚合的姿势可以被确定为站立或坐下(步骤1325)。系统可以确定经聚合的姿势是躺卧(步骤1330)。如果经聚合的姿势是躺卧，系统会查看先前经聚合的姿势数据的数据库，并且确定先前时间戳处经聚合的姿势是否是站立或坐下。系统可以将该姿势与前一帧中已识别出的姿势进行比较。帧可以按时间进行排序。系统可以确定经聚合的姿势从站立或坐下变为躺卧，然后该姿势保持在躺卧位置持续至少30秒(步骤1335)。In various embodiments, the system may perform posture aggregation (step 1315). More specifically, the system may determine the aggregated posture based on the pattern of the collection of postures collected within a given time period. For example, the posture that appears most frequently in the collection. Although all postures in the collection may not be the same, the posture aggregation method creates a consensus, that is, the aggregated posture. The system may determine the aggregated posture (step 1320). The aggregated posture may be determined to be standing or sitting (step 1325). The system may determine that the aggregated posture is lying (step 1330). If the aggregated posture is lying, the system will check the database of previously aggregated posture data and determine whether the aggregated posture at the previous timestamp is standing or sitting. The system can compare the posture with the posture recognized in the previous frame. The frames can be sorted by time. The system can determine that the aggregated posture changes from standing or sitting to lying, and then the posture remains in the lying position for at least 30 seconds (step 1335).
系统可以存储与各种动作中的每个动作相关联的时间戳。系统可以对开始时间戳与当前时间戳之间的差异进行检查以确定时间差异是否超过阈值。该阈值可以是预先指定的、预先确定的、动态调整的、基于算法的等。在各种实施方式中，如果跌倒超过阈值时间量，则系统发出跌倒警报。跌倒警报可以经由任何通信方式发送至任何其他设备。例如，该系统可以通过互联网向亲属智能手机上的应用程序发送信号，以通知亲属可能发生了跌倒。除了警报之外，系统还可以提供有关人员、位置、设施、健康信息、人口统计简档、跌倒历史等的数据。The system can store timestamps associated with each of the various actions. The system can check the difference between the start timestamp and the current timestamp to determine if the time difference exceeds a threshold. The threshold can be pre-specified, predetermined, dynamically adjusted, algorithm-based, etc. In various embodiments, if the fall exceeds a threshold amount of time, the system issues a fall alarm. The fall alarm can be sent to any other device via any communication method. For example, the system can send a signal to an application on a relative's smartphone via the Internet to notify the relative that a fall may have occurred. In addition to the alarm, the system can also provide data about people, locations, facilities, health information, demographic profiles, fall history, etc.
由于CNN的本质，CNN可以分配某种类型的姿势。但是，如果系统对姿势或帧是不确定的，则系统可以忽略一些帧。例如，系统可能使用0.7的置信度分数。0.7的置信度分数可以表明，如果系统以70％的置信度进行预测，那么该姿势在一定程度上是跌倒。该系统还可以以超过0.7或低于0.7置信度来对跌倒进行预测。Due to the nature of CNNs, CNNs can assign a certain type of pose. However, the system can ignore some frames if it is unsure about the pose or the frame. For example, the system might use a confidence score of 0.7. A confidence score of 0.7 can indicate that if the system predicts with 70% confidence, then the pose is a fall to a certain extent. The system can also predict a fall with more than 0.7 or less than 0.7 confidence.
系统可以确定潜在跌倒(步骤1340)。例如，如果人员躺在床上或沙发上，系统可以将这种动作确定为“潜在跌倒”。然而，躺在床上或者沙发上可以是一种正常的动作，并且这种动作可能不是真正的跌倒。如此，系统可以对潜在跌倒执行后处理(步骤1345)。在各种实施方式中，后处理可以包括过滤器，诸如，例如遮挡一些不可能合理地发生跌倒的空间位置(例如，床)。另一过滤器可以包括使用算法对潜在跌倒的检测的置信度阈值。The system can determine a potential fall (step 1340). For example, if a person lies on a bed or a sofa, the system can determine such action as a "potential fall." However, lying on a bed or a sofa can be a normal action, and such action may not be a true fall. As such, the system can perform post-processing on the potential fall (step 1345). In various embodiments, the post-processing can include filters, such as, for example, blocking some spatial locations where a fall could not reasonably occur (e.g., a bed). Another filter can include a confidence threshold for the detection of a potential fall using an algorithm.
基于后处理，系统可以将潜在跌倒确认为已确认跌倒(步骤1350)。特别地，在使用一个或更多个过滤器之后，系统可以确认跌倒并且提供已确认跌倒的通知。例如，在用户界面/软件产品中，用户可以在设置应用程序中创建和放置虚拟家具(例如，床、椅子、壁橱等)或遮挡区域。即使根据机器学习算法检测到“潜在跌倒”，系统也可以自动地将这些区域列入黑名单，以防止触发跌倒警报。例如，如上面所讨论的，如果人员躺卧或睡在床上，机器学习算法可以检测到“可能跌倒”。因为躺卧或睡在床上是正常动作，所以这种动作不会触发面向用户的跌倒警报。然而，如果人员实际跌倒在地板上，机器学习算法可以检测到“潜在跌倒”。然后，系统确定潜在跌倒是在黑名单区域内还是在黑名单区域的外部。如果潜在跌倒位于黑名单区域中，则系统将不会发出警报。如果潜在跌倒位于黑名单区域外部，则系统可以确定该跌倒是“已确认跌倒”并且触发(例如，发送)面向用户的跌倒警报。Based on post-processing, the system can confirm the potential fall as a confirmed fall (step 1350). In particular, after using one or more filters, the system can confirm the fall and provide a notification of the confirmed fall. For example, in the user interface/software product, the user can create and place virtual furniture (e.g., bed, chair, closet, etc.) or occluded areas in the settings application. Even if a "potential fall" is detected according to the machine learning algorithm, the system can automatically blacklist these areas to prevent the triggering of a fall alarm. For example, as discussed above, if a person lies down or sleeps in bed, the machine learning algorithm can detect a "possible fall". Because lying down or sleeping in bed is a normal action, such an action will not trigger a fall alarm for the user. However, if the person actually falls on the floor, the machine learning algorithm can detect a "potential fall". The system then determines whether the potential fall is within the blacklist area or outside the blacklist area. If the potential fall is located in the blacklist area, the system will not issue an alarm. If the potential fall is outside the blacklist area, the system can determine that the fall is a "confirmed fall" and trigger (e.g., send) a fall alarm for the user.
姿势检测系统可以包括许多商业应用和商业利益。例如，对于老年生活社区，该系统可以提供预测见解和处方。在这方面，该系统可以向护理人员提供标记或通知，以便尽早进行干预。具体地，该系统可以基于例如对基线移动模式和虚弱中的变化进行分析来对虚弱进行测量和跟踪。该系统还可以对异常活动(例如，在床上或浴室中待的时间过长)进行检测和/或标记。该系统还可以对夜间前往厕所的出行进行检测和/或标记。The posture detection system can include many commercial applications and business benefits. For example, for senior living communities, the system can provide predictive insights and prescriptions. In this regard, the system can provide flags or notifications to caregivers so that intervention can be made as early as possible. Specifically, the system can measure and track frailty based on, for example, analysis of baseline movement patterns and changes in frailty. The system can also detect and/or flag abnormal activities (e.g., staying in bed or bathroom for too long). The system can also detect and/or flag trips to the bathroom at night.
之前的系统典型地需要主动步骤和使用可穿戴设备或完成调查。然而，当前的系统可以提供私密和非侵入的价值主张。系统还可以被动地对实际行为进行感知，而不考虑行为的变化。Previous systems typically required active steps and the use of wearable devices or the completion of surveys. However, current systems can offer a value proposition of privacy and non-intrusion. The system can also passively sense actual behavior without considering changes in behavior.
目前，虚弱是在诊所或医生办公室进行测量的。医生可以执行对患者的测试，其包括一系列可能需要5分钟至10分钟才能完成的活动。然而，如此快速的测试不能全面了解患者，并且测试是在人为情境中进行的，患者可能会更加专注、更加努力等。在这方面，由于不同情景和患者所做的努力不同，虚弱评分可能会随时间的推移而不同。当前系统通过使用“纵向”跟踪(包括对人员随时间的移动进行跟踪)改进了现有技术的虚弱测试。如此，当前系统能够提供对随时间推移的虚弱的更全面地了解。Currently, frailty is measured in a clinic or doctor's office. The doctor can perform a test on the patient that includes a series of activities that may take 5 to 10 minutes to complete. However, such a quick test does not provide a comprehensive understanding of the patient, and the test is performed in an artificial context, where the patient may be more focused, more diligent, etc. In this regard, frailty scores may vary over time due to different scenarios and different efforts made by the patient. The current system improves on the prior art frailty testing by using "longitudinal" tracking, which includes tracking the movement of a person over time. In this way, the current system is able to provide a more comprehensive understanding of frailty over time.
使用姿势检测功能，系统可以向例如护理人员、建筑管理系统、警报系统、通知系统和/或紧急响应系统提供有关事件(例如跌倒、异常行为等)的通知或报告。报告可以包括与该事件(例如，跌倒)相关联的数据，诸如，例如位置、一天中的时间、跌倒之前和/或之后的动作、附近的物品(例如，家具)、人员持有的物品(例如，杂货、助行器、手杖、另外的人员)等。与雷达或其他传感器设备相比，该系统可以更便宜、安装更快并且安装更容易。该系统还可以对图像进行分析以促进审计和/或合规。例如，系统可以使用一段时间内的图像以对所提供的护理进行监测或审计(例如，前一天晚上11点完成床位检查)。该系统还可以使用一段时间内的图像以对提供护理所花费的时间进行测量(例如，每天与患者一起在浴室的平均分钟数)。该系统还可以与其他系统诸如调度系统和报告系统集成。例如，系统可以获得有关员工何时开始轮班、员工何时结束轮班、员工姓名和/或标识符、员工帮助某个患者所声称的时间等的数据。系统可以将这些提交的数据与从居民活动和/或居民护理的图像中获得的实际数据进行比较，以确定这些提交的数据的准确性。Using the gesture detection function, the system can provide notifications or reports about events (e.g., falls, abnormal behavior, etc.) to, for example, nursing staff, building management systems, alarm systems, notification systems, and/or emergency response systems. The report can include data associated with the event (e.g., fall), such as, for example, location, time of day, actions before and/or after the fall, nearby items (e.g., furniture), items held by people (e.g., groceries, walkers, canes, other people), etc. Compared with radar or other sensor devices, the system can be cheaper, faster to install, and easier to install. The system can also analyze images to facilitate audits and/or compliance. For example, the system can use images over a period of time to monitor or audit the care provided (e.g., bed checks were completed at 11 pm the previous night). The system can also use images over a period of time to measure the time spent providing care (e.g., the average number of minutes in the bathroom with the patient every day). The system can also be integrated with other systems such as scheduling systems and reporting systems. For example, the system can obtain data about when an employee starts a shift, when an employee ends a shift, employee names and/or identifiers, and the time claimed by an employee to help a patient. The system may compare such submitted data with actual data obtained from images of resident activity and/or resident care to determine the accuracy of such submitted data.
本文对各个实施方式的详细描述参考了附图和图片，这些附图和图片通过图示的方式示出了各种实施方式。尽管对这些各种实施方式进行了足够详细的描述以使本领域的技术人员能够实践本公开，但应当理解的是，可以实现其他实施方式并且可以在不脱离本公开的精神和范围的情况下进行逻辑和机械改变。因此，呈现本文的详细描述仅出于说明的目的而不是出于限制的目的。例如，在任何方法或过程描述中列举的步骤可以以任何顺序执行并且不限于所呈现的顺序。此外，任何功能或步骤可以外包给一个或更多个第三方或者可以由一个或更多个第三方执行。在不脱离本公开的范围的情况下，可以对本文所描述的系统、装置和方法进行修改、添加或省略。例如，系统和装置的部件可以被集成或分离。此外，本文所公开的系统和装置的操作可以由更多、更少或其他部件执行，并且所描述的方法可以包括更多、更少或其他步骤。此外，可以以任何合适的顺序执行这些步骤。如本文档中所使用的，“每个”指集合的每个成员或集合的子集的每个成员。此外，对单数的任何提及包括多个实施方式，并且对多于一个部件的任何提及可以包括单数实施方式。尽管本文已经列举了具体的优点，但是各种实施方式可以包括一些列举的优点、不包括列举的优点或包括所有列举的优点。提供了系统和方法。The detailed description of various embodiments herein refers to the drawings and pictures, which illustrate various embodiments by way of illustration. Although these various embodiments are described in sufficient detail to enable those skilled in the art to practice the present disclosure, it should be understood that other embodiments can be implemented and logical and mechanical changes can be made without departing from the spirit and scope of the present disclosure. Therefore, the detailed description herein is presented for the purpose of illustration only and not for the purpose of limitation. For example, the steps listed in any method or process description can be performed in any order and are not limited to the order presented. In addition, any function or step can be outsourced to one or more third parties or can be performed by one or more third parties. Without departing from the scope of the present disclosure, the systems, devices and methods described herein can be modified, added or omitted. For example, the components of the systems and devices can be integrated or separated. In addition, the operation of the systems and devices disclosed herein can be performed by more, less or other components, and the described methods can include more, less or other steps. In addition, these steps can be performed in any suitable order. As used in this document, "each" refers to each member of a set or each member of a subset of a set. In addition, any reference to the singular includes multiple embodiments, and any reference to more than one component can include a singular embodiment. Although specific advantages have been enumerated herein, various implementations may include some, none, or all of the enumerated advantages.Systems and methods are provided.
在本文详细的描述中，对“各种实施方式”、“一个实施方式”、“实施方式”、“示例实施方式”等的提及表示所描述的实施方式可以包括特定的特征、结构或特性，但每个实施方式不一定包括特定的特征、结构或特性。此外，这些短语不一定指相同的实施方式。此外，当结合实施方式对特定特征、结构或特性进行描述时，无论是否明确描述，都认为在本领域技术人员的知识范围内影响与其他实施方式结合的这种特征、结构或特性。在阅读了描述之后，相关领域的技术人员将清楚如何在替代实施方式中实施本公开。In the detailed description herein, references to "various embodiments," "one embodiment," "embodiment," "example embodiment," etc. indicate that the described embodiments may include specific features, structures, or characteristics, but each embodiment does not necessarily include the specific features, structures, or characteristics. In addition, these phrases do not necessarily refer to the same embodiment. In addition, when a specific feature, structure, or characteristic is described in conjunction with an embodiment, whether or not explicitly described, it is considered to be within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in conjunction with other embodiments. After reading the description, it will be clear to those skilled in the relevant art how to implement the present disclosure in alternative embodiments.
本文已经关于特定实施方式描述了益处、其他优点和问题的解决方案。然而，益处、优点、问题的解决方案、以及可能导致任何益处、优点或解决方案出现或变得更加明显的任何元素不应被解释为本发明的关键、必需或基本特征或元素。因此，本发明的范围仅受所附权利要求书的限制，其中除非明确说明如此，否则以单数形式对元素的提及并非意指“一个且只有一个”，而是“一个或更多个”。此外，当在权利要求中使用类似于“A、B或C中的至少一个”的短语时，旨在将该短语解释为意指可以在实施方式中单独存在A、可以在实施方式中单独存在B、可以在实施方式中单独存在C、或者可以在单个实施方式中存在元素A、B和C的任意组合；例如，A和B、A和C、B和C、或者A和B和C。此外，无论权利要求中是否明确记载了元素、部件或方法步骤，本公开中的任何元素、部件或方法步骤均不旨在专用于公众。本文的任何权利要求元素均不得根据35U.S.C.112(f)的规定进行解释，除非该元素使用短语“用于……的装置”明确记载。如本文所使用的，术语“包括”、“包含”或其任何其他变体旨在涵盖非排他性包含，使得包括元素列表的过程、方法、制品或装置不包括仅这些元素，而可能包括未明确列出的其他元素或这种过程、方法、制品或装置所固有的其他元素。Benefits, other advantages and solutions to problems have been described herein with respect to specific embodiments. However, benefits, advantages, solutions to problems, and any elements that may cause any benefits, advantages or solutions to appear or become more apparent should not be interpreted as key, essential or basic features or elements of the present invention. Therefore, the scope of the present invention is limited only by the appended claims, wherein unless explicitly stated otherwise, reference to an element in the singular does not mean "one and only one", but "one or more". In addition, when a phrase similar to "at least one of A, B or C" is used in a claim, it is intended that the phrase be interpreted as meaning that A can exist alone in an embodiment, B can exist alone in an embodiment, C can exist alone in an embodiment, or any combination of elements A, B and C can exist in a single embodiment; for example, A and B, A and C, B and C, or A and B and C. In addition, regardless of whether an element, component or method step is explicitly recorded in a claim, any element, component or method step in the present disclosure is not intended to be dedicated to the public. Any claim element herein shall not be interpreted according to the provisions of 35 U.S.C.112 (f) unless the element is explicitly recorded using the phrase "device for..." As used herein, the terms "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
计算机程序(也被称为计算机控制逻辑)被存储在主存储器和/或辅助存储器中。计算机程序也可以经由通信接口来接收。这种计算机程序在被执行时使计算机系统能够执行如本文所讨论的特征。特别地，计算机程序在被执行时使处理器能够执行各种实施方式的特征。因此，这样的计算机程序代表计算机系统的控制器。A computer program (also referred to as computer control logic) is stored in a main memory and/or a secondary memory. The computer program may also be received via a communication interface. Such a computer program, when executed, enables a computer system to perform features as discussed herein. In particular, the computer program, when executed, enables a processor to perform features of various embodiments. Therefore, such a computer program represents a controller of a computer system.
这些计算机程序指令可以被加载到通用计算机、专用计算机或其他可编程数据处理装置上以生产机器，使得在计算机或其他可编程数据处理装置上执行的指令创建用于实现在一个或更多个流程图块中指定的功能的装置。这些计算机程序指令还可以被存储在计算机可读存储器中，该计算机可读存储器可以指导计算机或其他可编程数据处理装置以特定方式运行，使得被存储在计算机可读存储器中的指令产生包括实现一个或更多个流程图块中指定的功能的指令装置的制品。计算机程序指令也可以被加载到计算机或其他可编程数据处理装置上，以使一系列操作步骤在计算机或其他可编程装置上被执行，以产生计算机实现的处理，使得在计算机或其他可编程装置上执行的指令提供用于实现一个或更多个流程图块中指定的功能的步骤。These computer program instructions may be loaded onto a general purpose computer, a special purpose computer, or other programmable data processing device to produce a machine, such that the instructions executed on the computer or other programmable data processing device create a device for implementing the functions specified in one or more flowchart blocks. These computer program instructions may also be stored in a computer readable memory, which may direct the computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the functions specified in one or more flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing device to cause a series of operating steps to be executed on the computer or other programmable device to produce a computer-implemented process, such that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flowchart blocks.
在各种实施方式中，软件可以被存储在计算机程序产品中并且使用可移动存储驱动器、硬盘驱动器或通信接口被加载到计算机系统中。当由处理器执行时，控制逻辑(软件)使处理器执行如本文所描述的各种实施方式的功能。在各种实施方式中，硬件部件可以采取专用集成电路(ASIC)的形式。硬件的实现方式以执行本文中所描述的功能对于相关领域的技术人员来说将是显而易见的。In various embodiments, the software can be stored in a computer program product and loaded into a computer system using a removable storage drive, a hard disk drive, or a communication interface. When executed by a processor, the control logic (software) causes the processor to perform the functions of the various embodiments as described herein. In various embodiments, the hardware components can take the form of an application specific integrated circuit (ASIC). The implementation of hardware to perform the functions described herein will be apparent to those skilled in the relevant art.
如本领域普通技术人员将理解的，该系统可以被体现为现有系统的定制、附加产品、执行升级软件的处理装置、独立系统、分布式系统、方法、数据处理系统、用于数据处理的设备和/或计算机程序产品。因此，系统或模块的任何部分可以采取执行代码的处理装置、基于互联网的实施方式、完全硬件实施方式、或结合互联网、软件和硬件方面的实施方式的形式。此外，该系统可以采取计算机可读存储介质上的计算机程序产品的形式，该计算机可读存储介质具有在存储介质中体现的计算机可读程序代码装置。可以利用任何合适的计算机可读存储介质，包括硬盘、CD-ROM、BLU-RAY光学存储设备、磁存储设备等。As will be appreciated by one of ordinary skill in the art, the system may be embodied as a customization of an existing system, an add-on product, a processing device that executes upgraded software, a stand-alone system, a distributed system, a method, a data processing system, an apparatus for data processing, and/or a computer program product. Thus, any portion of the system or module may take the form of a processing device that executes code, an Internet-based implementation, a fully hardware implementation, or an implementation that combines Internet, software, and hardware aspects. Additionally, the system may take the form of a computer program product on a computer-readable storage medium having a computer-readable program code device embodied in the storage medium. Any suitable computer-readable storage medium may be utilized, including hard disks, CD-ROMs, BLU-RAY Optical storage devices, magnetic storage devices, etc.
在各种实施方式中，系统的部件、模块和/或引擎可以被实现为微应用(micro-application)或微应用程序(micro-app)。微应用程序典型地被部署在移动操作系统的背景中，包括例如移动操作系统、操作系统、操作系统、公司的操作系统等。微应用程序可以被配置成经由管理各种操作系统和硬件资源的操作的一组预定规则来利用更大操作系统和相关硬件的资源。例如，在微应用程序期望与移动设备或移动操作系统以外的设备或网络通信的情况下，微应用程序可以在移动操作系统的预定规则下利用操作系统和相关设备硬件的通信协议。此外，在微应用程序期望来自用户的输入的情况下，微应用程序可以被配置成请求来自操作系统的响应，该操作系统对各种硬件部件进行监测，以及然后将检测到的输入从硬件传送至微应用程序。In various embodiments, the components, modules and/or engines of the system may be implemented as micro-applications or micro-apps. Micro-applications are typically deployed in the context of a mobile operating system, including, for example, Mobile operating systems, operating system, operating system, The micro-application can be configured to utilize the resources of the larger operating system and related hardware via a set of predetermined rules for managing the operation of various operating systems and hardware resources. For example, in the case where the micro-application expects to communicate with a device or network other than a mobile device or a mobile operating system, the micro-application can utilize the communication protocol of the operating system and related device hardware under the predetermined rules of the mobile operating system. In addition, in the case where the micro-application expects input from the user, the micro-application can be configured to request a response from the operating system, which monitors various hardware components, and then transmits the detected input from the hardware to the micro-application.
该系统和方法可以在本文中根据功能块部件、屏幕截图、可选的选择和各种处理步骤来描述。应当理解的是，这种功能块可以通过被配置成执行指定功能的任何数目的硬件和/或软件部件来实现。例如，该系统可以采用各种集成电路部件，例如存储器元件、处理元件、逻辑元件、查找表等，它们可以在一个或更多个微处理器或其他控制设备的控制下执行各种功能。类似地，系统的软件元素可以用任何编程或诸如以下各者的脚本语言来实现：C、C++、C#、Object Notation(JSON)、VBScript、Macromedia COLD FUSION、COBOL、公司的动态伺服页、汇编、PHP、awk、Visual Basic、SQL存储规程、PL/SQL、任何shell脚本、以及可扩展标记语言(XML)，其中各种算法利用数据结构、对象、进程、例程或其他编程元素的任意组合来实现。此外，应当注意的是，该系统可以采用任何数目的常规技术用于数据传输、信号传输、数据处理、网络控制等。更进一步，该系统可以被用于对客户端脚本语言(诸如VBScript等)进行检测或防止客户端脚本语言(诸如VBScript等)的安全问题。The systems and methods may be described herein in terms of functional block components, screen shots, optional selections, and various processing steps. It should be understood that such functional blocks may be implemented by any number of hardware and/or software components configured to perform the specified functions. For example, the system may employ various integrated circuit components, such as memory elements, processing elements, logic elements, lookup tables, etc., which may perform various functions under the control of one or more microprocessors or other control devices. Similarly, the software elements of the system may be implemented in any programming or scripting language such as: C, C++, C#, Object Notation (JSON), VBScript, Macromedia COLD FUSION, COBOL, The company's dynamic server pages, compilations, PHP, awk, Visual Basic, SQL stored procedures, PL/SQL, any shell scripts, and extensible markup language (XML), where various algorithms are implemented using any combination of data structures, objects, processes, routines, or other programming elements. In addition, it should be noted that the system can employ any number of conventional techniques for data transmission, signal transmission, data processing, network control, etc. Further, the system can be used to implement client-side scripting languages such as VBScript, etc.) or prevent client-side scripting languages such as VBScript, etc.) security issues.
本文参考根据各种实施方式的方法、装置和计算机程序产品的屏幕截图、框图和流程图图示来描述系统和方法。应当理解的是，框图和流程图图示中的每个功能块、以及框图和流程图图示中的功能块的组合可以分别通过计算机程序指令来实现。The present invention describes systems and methods with reference to screenshots, block diagrams and flowchart illustrations of methods, apparatuses and computer program products according to various embodiments. It should be understood that each functional block in the block diagrams and flowchart illustrations, and the combination of functional blocks in the block diagrams and flowchart illustrations can be implemented by computer program instructions, respectively.
因此，框图和流程图图示的功能块支持用于执行指定功能的装置的组合、用于执行指定功能的步骤的组合、以及用于执行指定功能的程序指令装置。还应当理解的是，框图和流程图图示的每个功能块、以及框图和流程图图示中的功能块的组合可以由执行指定功能或步骤的、基于硬件的专用计算机系统或者专用硬件和计算机指令的合适组合来实现。此外，过程流的图示及其描述可以参考用户应用程序、网页、网站、网络表单、提示等。从业者将理解的是，本文中所描述的图示步骤可以在任何数目的配置中包括：包括应用程序、网页、网络表单、弹出应用程序、提示等的使用。还应理解的是，所图示和描述的多个步骤可以被组合成单个网页和/或应用程序，但为了简单起见已被扩展。在其他情况下，作为单个处理步骤图示和描述的步骤可以被分成多个网页和/或应用程序，但为了简单起见已被组合。Therefore, the functional blocks of the block diagrams and flowcharts support a combination of means for performing the specified functions, a combination of steps for performing the specified functions, and program instruction means for performing the specified functions. It should also be understood that each functional block of the block diagrams and flowcharts, and the combination of functional blocks in the block diagrams and flowcharts can be implemented by a hardware-based special-purpose computer system that performs the specified functions or steps, or a suitable combination of special-purpose hardware and computer instructions. In addition, the diagrams and descriptions of the process flows can refer to the user Applications, web pages, websites, web forms, prompts, etc. Practitioners will appreciate that the illustrated steps described herein may be included in any number of configurations: including Applications, web pages, web forms, popups Use of applications, prompts, etc. It should also be understood that the multiple steps illustrated and described may be combined into a single web page and/or application, but has been expanded for simplicity. In other cases, steps illustrated and described as a single processing step may be separated into multiple pages and/or applications, but have been combined for simplicity.
中间件可以包括任何硬件和/或软件，该硬件和/或软件被适当地配置成促进全异的计算系统之间的通信和/或处理事务。中间件部件是可商购的并且在本领域中是已知的。中间件可以通过可商购的硬件和/或软件、通过定制的硬件和/或软件部件、或通过它们的组合来实现。中间件可以驻留在各种配置中，并且可以作为独立系统存在，或者可以是驻留在互联网服务器上的软件部件。中间件可以被配置成对应用服务器的各种部件与任何数目的内部系统或外部系统之间的事务进行处理，以用于本文所公开的任何目的。Inc.(Armonk,NY)的MQTM(以前的MQSeries)是可商购的中间件产品的示例。企业服务总线(“ESB”)应用程序是中间件的另一示例。Middleware may include any hardware and/or software that is appropriately configured to facilitate communication and/or processing transactions between disparate computing systems. Middleware components are commercially available and known in the art. Middleware may be implemented by commercially available hardware and/or software, by customized hardware and/or software components, or by a combination thereof. Middleware may reside in various configurations and may exist as an independent system, or may be a software component that resides on an internet server. Middleware may be configured to process transactions between the various components of an application server and any number of internal systems or external systems for any purpose disclosed herein. Inc. (Armonk, NY) MQTM (formerly MQSeries) is an example of a commercially available middleware product.An Enterprise Service Bus ("ESB") application is another example of middleware.
本文所讨论的计算机可以提供由用户可访问的合适网站或其他基于互联网的图形用户界面。在一个实施方式中，公司的互联网信息服务(IIS)、事务服务器(MTS)服务、以及数据库与操作系统、网络服务器软件、SQL数据库、以及商业服务器结合使用。此外，诸如软件、数据库、软件、软件、软件、软件、软件等的部件可以被用于提供符合动态数据对象(ADO)兼容的数据库管理系统。在一个实施方式中，网络服务器与操作系统、数据库和PHP、Ruby和/或编程语言结合使用。The computer discussed herein may provide a suitable website or other Internet-based graphical user interface accessible by a user. In one embodiment, The company's Internet Information Services (IIS), Transaction Server (MTS) services, and Database and operating system, Web server software, SQL Databases, and In addition, such as software, database, software, software, software, software, Components of software and the like can be used to provide a database management system that is compatible with Dynamic Data Objects (ADO). In one embodiment, Network Server and operating system, Database and PHP, Ruby, and/or Used in conjunction with programming languages.
为简洁起见，本文可能对传统的数据联网、应用程序开发和系统的其他功能方面(以及系统的各种操作部件的部件)没有进行详细描述。此外，本文所包含的各种附图中所示的连接线旨在表示各种元件之间的示例性功能关系和/或物理耦合。应该注意的是，在实际系统中可能存在许多替代或附加的功能关系或物理连接。For the sake of brevity, conventional data networking, application development, and other functional aspects of the system (as well as components of the various operating components of the system) may not be described in detail herein. In addition, the connecting lines shown in the various figures included herein are intended to represent exemplary functional relationships and/or physical couplings between the various elements. It should be noted that many alternative or additional functional relationships or physical connections may exist in an actual system.
在各种实施方式中，本文所描述的方法是使用本文所描述的各种特定机器来实现的。如本领域技术人员将立即理解的，本文所描述的方法可以使用以下特定机器以及下文中开发的那些机器以任何合适的组合来实现。此外，正如从本公开中明确的，本文所描述的方法可以导致某些制品的各种变换。In various embodiments, the methods described herein are implemented using various specific machines described herein. As will be immediately understood by those skilled in the art, the methods described herein can be implemented using the following specific machines and those machines developed hereinafter in any suitable combination. In addition, as will be apparent from this disclosure, the methods described herein can result in various transformations of certain articles of manufacture.
在各种实施方式中，系统和各种部件可以与一种或更多种智能数字助理技术集成。例如，示例性智能数字助理技术可以包括由公司开发的系统、由Alphabet,Inc.开发的GOOGLE系统、公司的系统和/或类似的数字辅助技术。系统、GOOGLE系统和系统可以各自提供基于云的语音激活服务，这些语音激活服务可以协助完成任务、娱乐、一般信息甚至更多。所有设备，诸如AMAZONAMAZON ECHOAMAZON和AMAZONTV，都可以访问系统。系统、GOOGLE系统和系统可以经由其语音激活技术接收语音命令、激活其他功能、对智能设备进行控制以及/或者对信息进行收集。例如，智能数字助理技术可以被用于与以下各者进行交互：音乐、电子邮件、文本、电话、问答、住宅装修(improvement)信息、智能住宅通信/激活、游戏、购物、制作待办事项、设置闹钟、流媒体交互播客、播放有声读物、以及提供天气、交通和其他实时信息，诸如新闻。GOOGLE和系统还可以允许用户在所有支持数字助理的设备上访问有关与在线账户有联系的合格交易账户的信息。In various embodiments, the system and various components may be integrated with one or more intelligent digital assistant technologies. For example, exemplary intelligent digital assistant technologies may include Developed by the company system, GOOGLE developed by Alphabet, Inc. system, Company systems and/or similar digital assistive technologies. System, GOOGLE System and The systems can each offer cloud-based voice-activated services that can assist with tasks, entertainment, general information and more. Devices such as AMAZON AMAZON ECHO AMAZON and AMAZON TV, you can access system. System, GOOGLE System and The system can receive voice commands, activate other functions, control smart devices, and/or collect information via its voice activation technology. For example, the intelligent digital assistant technology can be used to interact with the following: music, email, text, phone, Q&A, home improvement information, smart home communication/activation, games, shopping, making to-do lists, setting alarms, streaming interactive podcasts, playing audiobooks, and providing weather, traffic and other real-time information such as news. GOOGLE and The system may also allow the user to access information about eligible transaction accounts linked to the online account on all digital assistant-enabled devices.
本文所讨论的各种系统部件可以包括以下各者中的一项或更多项：主机服务器或其他计算系统，包括用于对数字数据进行处理的处理器；存储器，该存储器耦合至处理器，用于存储数字数据；输入数字化器，该输入数字化器联接至处理器，用于输入数字数据；应用程序，该应用程序被存储在存储器中并且可由处理器访问，用于指导处理器对数字数据的处理；显示设备，该显示设备联接至处理器和存储器，用于显示源自由处理器处理的数字数据的信息；以及多个数据库。本文所使用的各种数据库可以包括：客户端数据；商家数据；金融机构数据；和/或对系统的操作有用的数据。如本领域技术人员将理解的，用户计算机可以包括操作系统(例如，等)以及典型地与计算机相关联的各种常规支持软件和驱动程序。The various system components discussed herein may include one or more of the following: a host server or other computing system including a processor for processing digital data; a memory coupled to the processor for storing digital data; an input digitizer coupled to the processor for inputting digital data; an application stored in the memory and accessible by the processor for directing the processing of the digital data by the processor; a display device coupled to the processor and the memory for displaying information derived from the digital data processed by the processor; and a plurality of databases. The various databases used herein may include: client data; merchant data; financial institution data; and/or data useful for the operation of the system. As will be appreciated by those skilled in the art, a user computer may include an operating system (e.g., etc.) and various conventional supporting software and drivers typically associated with a computer.
本系统或其任何(一个或多个)部分或(一个或多个)功能可以使用硬件、软件或它们的组合来实现，并且可以在一个或更多个计算机系统或其他处理系统中实现。然而，由实施方式执行的操作可以用诸如匹配或选择的术语来指代，这些术语通常与由人类操作者执行的智力操作相关联。在大多数情况下，在本文所描述的任何操作中，人类操作员的这种能力不是必需的或是不期望的。相反，操作可以是机器操作，或者任何操作可以通过人工智能(AI)或机器学习来进行或增强。AI通常可以指对感知周围世界、形成计划并做出决策以实现其目标的代理(例如，机器、基于计算机的系统等)的研究。AI的基础包括数学、逻辑、哲学、概率、语言学、神经科学和决策理论。许多领域都属于AI的范畴，诸如计算机视觉、机器人学、机器学习和自然语言处理。用于执行各个实施方式的有用机器包括通用数字计算机或类似设备。The present system or any (one or more) part or (one or more) function thereof can be implemented using hardware, software or a combination thereof, and can be implemented in one or more computer systems or other processing systems. However, the operation performed by the embodiment can be referred to by terms such as matching or selecting, which are usually associated with the intellectual operation performed by a human operator. In most cases, in any operation described herein, this ability of a human operator is not necessary or undesirable. On the contrary, the operation can be machine operation, or any operation can be performed or enhanced by artificial intelligence (AI) or machine learning. AI can generally refer to the study of agents (e.g., machines, computer-based systems, etc.) that perceive the surrounding world, form plans, and make decisions to achieve their goals. The foundations of AI include mathematics, logic, philosophy, probability, linguistics, neuroscience, and decision theory. Many fields fall within the scope of AI, such as computer vision, robotics, machine learning, and natural language processing. Useful machines for performing each embodiment include general-purpose digital computers or similar devices.
在各个实施方式中，实施方式针对能够执行本文所描述的功能的一个或更多个计算机系统。计算机系统包括一个或更多个处理器。处理器连接至通信基础设施(例如，通信总线、分频杆(crossover bar)、网络等)。根据该示例性计算机系统描述了各种软件实施方式。在阅读了该说明之后，相关领域的技术人员将清楚如何使用其他计算机系统和/或架构来实现各种实施方式。计算机系统可以包括显示界面，该显示界面从通信基础设施(或从未示出的帧缓冲器)转发图形、文本和其他数据，以在显示单元上显示。In various embodiments, the embodiments are directed to one or more computer systems capable of performing the functions described herein. The computer system includes one or more processors. The processor is connected to a communication infrastructure (e.g., a communication bus, a crossover bar, a network, etc.). Various software implementations are described based on this exemplary computer system. After reading this description, it will be clear to those skilled in the relevant art how to use other computer systems and/or architectures to implement various implementations. The computer system may include a display interface that forwards graphics, text, and other data from the communication infrastructure (or from a frame buffer not shown) to display on a display unit.
计算机系统还包括主存储器，诸如随机存取存储器(RAM)，并且还可以包括辅助存储器。辅助存储器可以包括例如硬盘驱动器、固态驱动器和/或可移动存储驱动器。可移动存储驱动器以众所周知的方式从可移动存储单元读取和/或写入至可移动存储单元。如将理解的，可移动存储单元包括其中存储有计算机软件和/或数据的计算机可用存储介质。The computer system also includes a main memory, such as a random access memory (RAM), and may also include a secondary memory. The secondary memory may include, for example, a hard disk drive, a solid state drive, and/or a removable storage drive. The removable storage drive reads from and/or writes to a removable storage unit in a well-known manner. As will be appreciated, a removable storage unit includes a computer-usable storage medium having computer software and/or data stored therein.
在各种实施方式中，辅助存储器可以包括用于允许将计算机程序或其他指令加载到计算机系统中的其他类似设备。这种设备可以包括例如可移动存储单元和接口。这种示例可以包括程序盒和盒接口(诸如在视频游戏设备中发现的)、可移动存储器芯片(诸如可擦除可编程只读存储器(EPROM)、可编程只读存储器(PROM))和相关的插座、或者其他可移动存储单元和接口，它们允许将软件和数据从可移动存储单元传送至计算机系统。In various embodiments, auxiliary storage may include other similar devices for allowing computer programs or other instructions to be loaded into a computer system. Such devices may include, for example, removable storage units and interfaces. Such examples may include program cartridges and cartridge interfaces (such as those found in video game devices), removable memory chips (such as erasable programmable read-only memories (EPROMs), programmable read-only memories (PROMs)) and associated sockets, or other removable storage units and interfaces that allow software and data to be transferred from the removable storage unit to the computer system.
术语“计算机程序介质”、“计算机可用介质”和“计算机可读介质”被用于通常指诸如可移动存储驱动器和安装在硬盘驱动器中的硬盘等的介质。这些计算机程序产品向计算机系统提供软件。The terms "computer program medium", "computer usable medium" and "computer readable medium" are used to generally refer to media such as removable storage drives and hard disks installed in hard drives, etc. These computer program products provide software to computer systems.
计算机系统还可以包括通信接口。通信接口允许在计算机系统与外部设备之间传送软件和数据。这种通信接口的示例可以包括调制解调器、网络接口(诸如以太网卡)、通信端口等。经由通信接口传送的软件和数据是信号的形式，该信号可以是电子的、电磁的、光学的，或其他能够被通信接口接收的信号。这些信号经由通信路径(例如，信道)提供至通信接口。该信道承载信号并且可以使用电线、电缆、光纤、电话线、蜂窝链路、射频(RF)链路、无线和其他通信信道来实现。The computer system may also include a communication interface. The communication interface allows software and data to be transmitted between the computer system and an external device. Examples of such communication interfaces may include a modem, a network interface (such as an Ethernet card), a communication port, etc. The software and data transmitted via the communication interface are in the form of a signal, which may be electronic, electromagnetic, optical, or other signals that can be received by the communication interface. These signals are provided to the communication interface via a communication path (e.g., a channel). The channel carries signals and can be implemented using wires, cables, optical fibers, telephone lines, cellular links, radio frequency (RF) links, wireless and other communication channels.
如本文所使用的，“标识符”可以是唯一标识项目的任何合适的标识符。例如，标识符可以是全局唯一标识符(“GUID”)。GUID可以是在通用唯一标识符标准下创建和/或实现的标识符。此外，GUID可以被存储为128位值，该128位值可以被显示为32个十六进制数字。标识符还可以包括主要号码和次要号码。主要号码和次要号码均可以是16位整数。As used herein, an "identifier" may be any suitable identifier that uniquely identifies an item. For example, an identifier may be a globally unique identifier ("GUID"). A GUID may be an identifier created and/or implemented under the Universal Unique Identifier standard. In addition, a GUID may be stored as a 128-bit value that may be displayed as 32 hexadecimal digits. An identifier may also include a major number and a minor number. Both the major number and the minor number may be 16-bit integers.
在各种实施方式中，服务器可以包括应用服务器(例如，POSTGRES PLUS ADVANCED等)。在各种实施方式中，服务器可以包括网络服务器(例如，Apache、IIS、网络服务器、系统网络服务器、在或操作系统上运行的虚拟机)。In various embodiments, the server may include an application server (e.g., POSTGRES PLUS ADVANCED In various embodiments, the server may include a web server (e.g., Apache, IIS, Network servers, System network server, or Running on the operating system virtual machines).
网络客户端包括经由任何网络进行通信的任何设备或软件，诸如，例如本文中所讨论的任何设备或软件。网络客户端可以包括安装在计算单元或系统内的互联网浏览软件，以进行在线交易和/或通信。这些计算单元或系统可以采取计算机或计算机组的形式，尽管可以使用其他类型的计算单元或系统，包括个人计算机、膝上型计算机、笔记本电脑、平板电脑、智能手机、蜂窝电话、个人数字助理、服务器、池服务器、大型主机计算机、分布式计算集群、信息亭、终端、销售点(POS)设备或终端、电视、或任何其他能够通过网络接收数据的设备。网络客户端可以包括操作系统(例如，WINDOW操作系统、操作系统、操作系统、操作系统等)以及各种传统的支持软件和驱动程序，这些典型地与计算机相关联。网络客户端还可以运行INTERNET软件、软件、GOOGLE CHROMETM软件、软件、或任何其他可用于浏览互联网的无数软件包。A network client includes any device or software that communicates via any network, such as, for example, any device or software discussed herein. A network client may include Internet browsing software installed in a computing unit or system to conduct online transactions and/or communications. These computing units or systems may take the form of a computer or group of computers, although other types of computing units or systems may be used, including personal computers, laptops, notebook computers, tablet computers, smart phones, cellular phones, personal digital assistants, servers, pool servers, mainframe computers, distributed computing clusters, kiosks, terminals, point of sale (POS) devices or terminals, televisions, or any other device capable of receiving data over a network. A network client may include an operating system (e.g., WINDOW operating system, operating system, operating system, operating system, etc.) and various conventional supporting software and drivers that are typically associated with a computer. The network client may also run INTERNET software, software, GOOGLE CHROMETM software, software, or any of the countless software packages available for browsing the Internet.
如本领域技术人员将理解的，网络客户端可以与或可以不与服务器(例如，应用服务器、网络服务器等，如本文所讨论的)直接接触。例如，网络客户端可以通过另一服务器和/或硬件部件访问服务器的服务，该另一服务器和/或硬件部件可以直接或间接连接至互联网服务器。例如，网络客户端可以经由负载平衡器与服务器通信。在各个实施方式中，网络客户端访问是通过网络或通过可商购的网络浏览器软件包的互联网进行的。在这方面，网络客户端可以是在能够访问网络或互联网的住宅或商业环境中。网络客户端可以实现安全协议，诸如安全套接字层(SSL)和传输层安全(TLS)。网络客户端可以实现若干应用层协议，包括HTTP、HTTPS、FTP和SFTP。As will be appreciated by those skilled in the art, a network client may or may not be in direct contact with a server (e.g., an application server, a network server, etc., as discussed herein). For example, a network client may access the services of a server through another server and/or hardware component, which may be directly or indirectly connected to an Internet server. For example, a network client may communicate with a server via a load balancer. In various embodiments, network client access is performed over a network or over the Internet through a commercially available web browser software package. In this regard, a network client may be in a residential or commercial environment that is capable of accessing a network or the Internet. A network client may implement security protocols, such as secure socket layer (SSL) and transport layer security (TLS). A network client may implement several application layer protocols, including HTTP, HTTPS, FTP, and SFTP.
各种系统部件可以独立地、单独地或共同地经由数据链路适当地耦合至网络，这些数据链路包括例如通过本地环路与互联网服务提供商(ISP)的连接，该本地环路典型地与标准调制解调器通信、电缆调制解调器、DISHISDN、数字订户线(DSL)或各种无线通信方法结合使用。值得注意的是，该网络可以被实现为其他类型的网络，诸如交互式电视(ITV)网络。此外，该系统考虑通过具有本文所描述的类似功能的任何网络进行的使用、销售或分发任何商品、服务或信息。The various system components may be coupled to the network independently, individually or collectively as appropriate via data links including, for example, connection to an Internet Service Provider (ISP) through a local loop that typically communicates with a standard modem, cable modem, DISH ISDN, digital subscriber line (DSL) or various wireless communication methods are used in combination. It is worth noting that the network can be implemented as other types of networks, such as interactive television (ITV) networks. In addition, the system contemplates the use, sale or distribution of any goods, services or information through any network with similar functions as described herein.
该系统考虑与网络服务、公用事业计算、普及和个性化计算、安全和身份解决方案、自主计算、云计算、商品计算、移动和无线解决方案、开源、生物识别、网格计算和/或网状结构计算相关联的使用。The system contemplates uses associated with network services, utility computing, pervasive and personalized computing, security and identity solutions, autonomic computing, cloud computing, commodity computing, mobile and wireless solutions, open source, biometrics, grid computing, and/or mesh computing.
本文所讨论的任何通信、输入、存储、数据库或显示都可以通过具有网页的网站来促进。如本文所使用的术语“网页”并不意味着限制可用于与用户交互的文档和应用程序的类型。例如，除了标准HTML文档之外，典型的网站还可能包括各种表单、小程序、程序、活动服务器页面(ASP)、通用网关接口脚本(CGI)、可扩展标记语言(XML)、动态HTML、级联样式表(CSS)、AJAX(异步JAVASCRIPT和XML)程序、助手应用程序、插件程序等。服务器可以包括从网络服务器接收请求的网络服务，该请求包括URL和IP地址(192.168.1.1)。网络服务器检索适当的网页并将网页的数据或应用程序发送至IP地址。网络服务是能够通过诸如互联网等的通信方式与其他应用程序交互的应用程序。网络服务典型地基于标准或协议，诸如XML、SOAP、AJAX、WSDL和UDDI。网络服务方法在本领域中是众所周知的，并且在许多标准文本中都有涉及。例如，代表性状态传送(REST)或RESTful、网络服务可以提供一种实现应用程序之间互操作性的方式。Any communication, input, storage, database, or display discussed herein may be facilitated by a website having web pages. The term "web page" as used herein is not meant to limit the types of documents and applications that may be used to interact with a user. For example, in addition to standard HTML documents, a typical website may include various forms, Mini Programs, Programs, Active Server Pages (ASP), Common Gateway Interface scripts (CGI), Extensible Markup Language (XML), Dynamic HTML, Cascading Style Sheets (CSS), AJAX (Asynchronous JAVASCRIPT and XML) programs, helper applications, plug-ins, etc. The server may include a network service that receives a request from a network server, the request including a URL and an IP address (192.168.1.1). The network server retrieves the appropriate web page and sends the data of the web page or application to the IP address. A network service is an application that can interact with other applications through a communication method such as the Internet. Network services are typically based on standards or protocols such as XML, SOAP, AJAX, WSDL, and UDDI. Network service methods are well known in the art and are covered in many standard texts. For example, Representational State Transfer (REST) or RESTful, network services can provide a way to achieve interoperability between applications.
网络客户端的计算单元还可以配备有使用标准拨号、电缆、DSL或本领域已知的任何其他互联网协议连接至互联网或内联网的互联网浏览器。源自网络客户端的事务可能会通过防火墙，以防止来自其他网络的用户的未经授权的访问。此外，可以在CMS的不同部件之间部署附加的防火墙，以进一步增强安全性。The computing unit of the network client may also be equipped with an Internet browser that connects to the Internet or an intranet using standard dial-up, cable, DSL, or any other Internet protocol known in the art. Transactions originating from the network client may pass through a firewall to prevent unauthorized access by users from other networks. In addition, additional firewalls may be deployed between different components of the CMS to further enhance security.
可以通过本领域现在可用或可能变得可用的任何技术来执行加密——例如，Twofish、RSA、El Gamal、Schorr签名、DSA、PGP、PKI、GPG(GnuPG)、HPE格式保留加密(FPE)、电压、三重DES、Blowfish、AES、MD5、HMAC、IDEA、RC6、以及对称和非对称密码系统。该系统和方法还可以结合SHA系列密码方法、椭圆曲线密码(例如，ECC、ECDH、ECDSA等)和/或正在开发的其他后量子密码算法。Encryption can be performed by any technology that is now available or that may become available in the art - for example, Twofish, RSA, El Gamal, Schorr signatures, DSA, PGP, PKI, GPG (GnuPG), HPE format-preserving encryption (FPE), voltage, triple DES, Blowfish, AES, MD5, HMAC, IDEA, RC6, and symmetric and asymmetric cryptographic systems. The system and method can also be combined with the SHA family of cryptographic methods, elliptic curve cryptography (e.g., ECC, ECDH, ECDSA, etc.), and/or other post-quantum cryptographic algorithms being developed.
防火墙可以包括适当地被配置成保护CMS部件和/或企业计算资源不受其他网络的用户影响的任何硬件和/或软件。此外，防火墙可以被配置成限制或约束对通过网络服务器连接的网络客户端的防火墙后面的各种系统和部件的访问。防火墙可以驻留在不同的配置中，包括状态检查、基于代理、访问控制列表和数据包过滤等。防火墙可以被集成在网络服务器或任何其他CMS部件中，或者可以另外作为单独的实体驻留。防火墙可以实现网络地址转换(“NAT”)和/或网络地址端口转换(“NAPT”)。防火墙可以适应各种隧道协议以促进安全通信，诸如在虚拟专用网络中使用的那些。防火墙可以实现非军事化区(“DMZ”)以促进与公共网络(诸如互联网)的通信。防火墙可以作为软件被集成在互联网服务器或任何其他应用服务器部件中，驻留在另一计算设备中，或采取独立硬件部件的形式。The firewall may include any hardware and/or software that is appropriately configured to protect CMS components and/or enterprise computing resources from users of other networks. In addition, the firewall may be configured to limit or restrict access to various systems and components behind the firewall of network clients connected through a network server. The firewall may reside in different configurations, including stateful inspection, proxy-based, access control lists, and packet filtering. The firewall may be integrated into a network server or any other CMS component, or may reside as a separate entity in addition. The firewall may implement network address translation ("NAT") and/or network address port translation ("NAPT"). The firewall may adapt to various tunneling protocols to facilitate secure communications, such as those used in virtual private networks. The firewall may implement a demilitarized zone ("DMZ") to facilitate communication with a public network (such as the Internet). The firewall may be integrated into an Internet server or any other application server component as software, reside in another computing device, or take the form of an independent hardware component.
本文所讨论的任何数据库可以包括关系、分层、图形、区块链、面向对象结构和/或任何其他数据库配置。任何数据库还可以包括平面文件结构，其中数据可以以行和列的形式被存储在单个文件中，没有用于索引的结构，并且没有记录之间的结构关系。例如，平面文件结构可以包括界定的文本文件、CSV(逗号分隔值)文件和/或任何其他合适的平面文件结构。可以被用于实现数据库的常见数据库产品包括(Armonk,NY)的可购自Corporation(Redwood Shores,CA)的各种数据库产品、Corporation(Redmond,Washington)的MICROSOFT或MICROSOFT SQLMySQL AB(瑞典，乌普萨拉)的Redis、Apache公司的MapR-DB、或任何其他合适的数据库产品。此外，可以以任何合适的方式组织任何数据库，例如，作为数据表或查找表。每个记录可以是单个文件、一系列文件、链接的一系列数据字段、或任何其他数据结构。Any database discussed herein may include relational, hierarchical, graph, blockchain, object-oriented structures, and/or any other database configuration. Any database may also include a flat file structure, where data may be stored in a single file in the form of rows and columns, with no structure for indexing, and without structural relationships between records. For example, a flat file structure may include a delimited text file, a CSV (comma separated value) file, and/or any other suitable flat file structure. Common database products that may be used to implement a database include (Armonk, NY) Available from Corporation (Redwood Shores, CA) various database products, MICROSOFT from Corporation(Redmond,Washington) or MICROSOFT SQL MySQL AB (Uppsala, Sweden) Redis, Apache The company's MapR-DB, or any other suitable database product. In addition, any database can be organized in any suitable manner, for example, as a data table or a lookup table. Each record can be a single file, a series of files, a linked series of data fields, or any other data structure.
如本文所使用的，大数据可以指部分或完全结构化、半结构化或非结构化数据集，其包括数百万行和数十万列。例如，大数据集可以从一段时间内的购买交易历史、从网络注册、从社交媒体、从收费的记录(ROC)、从收费的摘要(SOC)、从内部数据、或从其他合适的来源进行编译。大数据集可以在没有描述性元数据(诸如列类型、计数、百分位数或其他解释性数据点)的情况下进行编译。As used herein, big data may refer to partially or fully structured, semi-structured, or unstructured data sets that include millions of rows and hundreds of thousands of columns. For example, a big data set may be compiled from purchase transaction history over a period of time, from web registrations, from social media, from records of charges (ROC), from summaries of charges (SOC), from internal data, or from other suitable sources. A big data set may be compiled without descriptive metadata such as column types, counts, percentiles, or other explanatory data points.
某些数据的关联可以通过任何期望的数据关联技术(诸如本领域已知或实践的那些)来实现。例如，可以手动或自动完成关联。自动关联技术可以包括，例如，数据库搜索、数据库合并、GREP、AGREP、SQL，使用表中的关键字段以加速搜索，通过所有表和文件的顺序搜索，根据已知顺序对文件中的记录进行排序以简化查找等。关联步骤可以通过数据库合并功能来完成，例如，使用预先选择的数据库或数据扇区中的“关键字段”来完成。设想了各种数据库调谐步骤以优化数据库性能。例如，经常使用的文件(诸如索引)可以被放置在单独的文件系统上，以减少输入/输出(“I/O”)瓶颈。The association of certain data can be achieved by any desired data association technique (such as those known or practiced in the art). For example, the association can be done manually or automatically. Automatic association techniques can include, for example, database searches, database merges, GREP, AGREP, SQL, using key fields in tables to speed up searches, sequential searches through all tables and files, sorting records in files according to a known order to simplify searches, etc. The association step can be accomplished by a database merge function, for example, using "key fields" in a pre-selected database or data sector. Various database tuning steps are envisioned to optimize database performance. For example, frequently used files (such as indexes) can be placed on a separate file system to reduce input/output ("I/O") bottlenecks.
更特别地，“关键字段”根据由关键字段限定的对象的高级类别来对数据库进行划分。例如，可以将某些类型的数据指定为多个相关数据表中的关键字段，以及然后可以基于关键字段中的数据类型来链接数据表。与链接数据表中的每个链接数据表中的关键字段相对应的数据优选地是相同的或相同类型的。然而，在关键字段中具有相似但不相同的数据的数据表也可以通过使用例如AGREP来链接。根据一个实施方式，可以利用任何合适的数据存储技术来存储没有标准格式的数据。可以使用任何合适的技术来存储数据集，包括例如使用ISO/IEC 7816-4文件结构存储单个文件；实现域，从而选择专用文件，该专用文件暴露一个或更多个包含一个或更多个数据集的基本文件；利用使用分层归档系统存储在单个文件中的数据集；作为记录存储在单个文件中的数据集(包括压缩、SQL可访问、经由一个或更多个键进行散列、数字、由第一元组按字母顺序排列等)；被存储为二进制大对象(BLOB)的数据；被存储为使用ISO/IEC7816-6数据元素编码的未分组数据元素的数据；被存储为使用ISO/IEC抽象语法符号(ASN.1)编码的未分组数据元素的数据，如在ISO/IEC 8824和8825中；其他专有技术，该专有技术可以包括分形压缩方法、图像压缩方法等。More specifically, "key fields" divide the database according to the high-level categories of objects defined by the key fields. For example, certain types of data can be designated as key fields in multiple related data tables, and the data tables can then be linked based on the type of data in the key fields. The data corresponding to the key fields in each of the linked data tables is preferably the same or of the same type. However, data tables with similar but not identical data in the key fields can also be linked by using, for example, AGREP. According to one embodiment, any suitable data storage technology can be used to store data that does not have a standard format. Any suitable technique may be used to store the data sets, including, for example, storing a single file using the ISO/IEC 7816-4 file structure; implementing a domain, thereby selecting a dedicated file that exposes one or more elementary files containing one or more data sets; utilizing data sets stored in a single file using a hierarchical archiving system; data sets stored as records in a single file (including compressed, SQL accessible, hashed via one or more keys, numeric, alphabetically arranged by first tuple, etc.); data stored as binary large objects (BLOBs); data stored as ungrouped data elements encoded using ISO/IEC7816-6 data elements; data stored as ungrouped data elements encoded using ISO/IEC Abstract Syntax Notation (ASN.1), such as in ISO/IEC 8824 and 8825; other proprietary techniques, which may include fractal compression methods, image compression methods, etc.
在各种实施方式中，通过将信息存储为BLOB来促进以不同格式存储多种信息的能力。因此，任何二进制信息都可以被存储在与数据集相关联的存储空间中。如上面所讨论的，二进制信息可以与系统相关联地被存储或在系统外部但附属于系统。BLOB方法可以使用固定存储分配、循环队列技术或关于存储器管理的最佳实践(例如，分页存储器、最近最少使用等)，经由固定存储器偏移量将数据集存储为被格式化为二进制块的未分组数据元素。通过使用BLOB方法，存储具有不同格式的各种数据集的能力促进了通过数据集的多个且不相关的所有者将数据存储在数据库中或存储与系统相关联的数据。例如，可以被存储的第一数据集可以由第一方提供，可以被存储的第二数据集可以由不相关的第二方提供，以及可以被存储的第三数据集可以由与第一方和第二方不相关的第三方提供。这三个示例性数据集中的每个示例性数据集都可以包含使用不同数据存储格式和/或技术存储的不同信息。此外，每个数据集可以包含也可能与其他子集不同的数据子集。In various embodiments, the ability to store multiple information in different formats is facilitated by storing information as BLOBs. Therefore, any binary information can be stored in a storage space associated with a data set. As discussed above, binary information can be stored in association with a system or outside the system but attached to the system. The BLOB method can use fixed storage allocation, circular queue technology, or best practices for memory management (e.g., paging memory, least recently used, etc.) to store data sets as ungrouped data elements formatted as binary blocks via fixed memory offsets. By using the BLOB method, the ability to store various data sets with different formats facilitates storing data in a database or storing data associated with a system by multiple and unrelated owners of the data set. For example, a first data set that can be stored can be provided by a first party, a second data set that can be stored can be provided by an unrelated second party, and a third data set that can be stored can be provided by a third party that is unrelated to the first party and the second party. Each of these three exemplary data sets can contain different information stored using different data storage formats and/or technologies. In addition, each data set can contain a subset of data that may also be different from other subsets.
如上面所陈述的，在各种实施方式中，可以在不考虑通用格式的情况下存储数据。然而，当提供用于操纵数据库或系统中的数据时，可以以标准方式注释数据集(例如，BLOB)。注释可以包括短标头、标尾、或与每个数据集相关联的其他适当指示符，每个数据集被配置成传达对管理各种数据集有用的信息。例如，注释在本文中可以被称为“条件标头”、“标头”、“标尾”或“状态”，并且可以包括数据集状态的指示或者可以包括与数据的特定发布者或所有者相关的标识符。在一个示例中，每个数据集BLOB的前三个字节可以被配置成或能够配置成指示该特定数据集的状态；例如，已加载、已初始化、已准备好、已阻止、可去除或已删除。数据的后续字节可以被用于指示例如发行者、用户、交易/会员账户标识符等的身份。本文进一步讨论这些条件注释中的每个条件注释。As stated above, in various embodiments, data can be stored without considering the common format. However, when providing for manipulating data in a database or system, a data set (e.g., BLOB) can be annotated in a standard manner. The annotation can include a short header, a footer, or other appropriate indicators associated with each data set, and each data set is configured to convey information useful for managing various data sets. For example, the annotation may be referred to as a "conditional header," "header," "footer," or "status" in this article, and may include an indication of the state of the data set or may include an identifier associated with a specific publisher or owner of the data. In one example, the first three bytes of each data set BLOB may be configured or can be configured to indicate the state of the specific data set; for example, loaded, initialized, ready, blocked, removable, or deleted. The subsequent bytes of the data may be used to indicate the identity of, for example, the publisher, user, transaction/member account identifier, etc. Each of these conditional annotations is further discussed herein.
数据集注释也可以被用于其他类型的状态信息以及各种其他目的。例如，数据集注释可以包括建立访问级别的安全信息。例如，访问级别可以被配置成仅允许某些个人、雇员的级别、公司或其他实体访问数据集，或者允许访问基于交易、商家、发行者、用户等的特定数据集。此外，安全信息可以仅限制/允许某些动作，诸如访问、修改和/或删除数据集。在一个示例中，数据集注释指示仅允许数据集所有者或用户删除数据集，可以允许各种已识别出的用户访问数据集以进行阅读，并且完全排除其他用户访问数据集。然而，也可以使用其他访问限制参数，从而允许各种实体访问具有各种适当权限级别的数据集。Dataset annotations may also be used for other types of state information and for various other purposes. For example, dataset annotations may include security information that establishes access levels. For example, access levels may be configured to allow only certain individuals, levels of employees, companies, or other entities to access a dataset, or to allow access to specific datasets based on transactions, merchants, publishers, users, etc. In addition, security information may only restrict/allow certain actions, such as accessing, modifying, and/or deleting a dataset. In one example, a dataset annotation indicates that only the dataset owner or user is allowed to delete the dataset, various identified users may be allowed to access the dataset for reading, and other users may be completely excluded from accessing the dataset. However, other access restriction parameters may also be used, allowing various entities to access datasets with various appropriate permission levels.
包括标头或标尾的数据可以由被配置成根据标头或标尾而添加、删除、修改或补充数据的独立交互设备接收。因此，在一个实施方式中，标头或标尾不与相关联的发行者拥有的数据一起被存储在交易设备上，而是可以通过在独立设备处向用户提供用于要采取的行动的适当的选项来采取适当的行动。该系统可以设想数据存储布置，其中数据的标头或标尾、或者标头历史或标尾历史与适当的数据相关地被存储在系统、设备或交易工具上。The data including the header or footer may be received by a separate interactive device configured to add, delete, modify or supplement data based on the header or footer. Thus, in one embodiment, the header or footer is not stored on the transaction device along with the associated issuer owned data, but appropriate action may be taken by providing the user at the separate device with appropriate options for the action to be taken. The system may contemplate a data storage arrangement in which a header or footer of data, or a history of headers or footers, is stored on the system, device or transaction tool in association with appropriate data.
本领域技术人员还将理解的是，出于安全原因，任何数据库、系统、设备、服务器或系统的其他部件可以包括其在单个位置或多个位置处的任何组合，其中每个数据库或系统包括各种合适的安全特征(诸如防火墙、访问代码、加密、解密、压缩、解压缩等)中的任何一者。Those skilled in the art will also understand that for security reasons, any database, system, device, server, or other component of the system may include any combination thereof at a single location or multiple locations, with each database or system including any of a variety of suitable security features (such as firewalls, access codes, encryption, decryption, compression, decompression, etc.).
从业者也将意识到有许多方法用于在基于浏览器的文档内显示数据。数据可以被表示为标准文本或在固定列表、可滚动列表、下拉列表、可编辑文本字段、固定文本字段、弹出窗口等中表示。同样，有许多方法可用于修改网页中的数据，诸如，例如，使用键盘输入自由文本、选择菜单项、复选框、选项框等。Practitioners will also appreciate that there are many methods for displaying data within a browser-based document. Data may be presented as standard text or in a fixed list, scrollable list, drop-down list, editable text field, fixed text field, pop-up window, etc. Likewise, there are many methods for modifying data in a web page, such as, for example, entering free text using a keyboard, selecting menu items, check boxes, option boxes, etc.
数据可以是由分布式计算集群处理的大数据。分布式计算集群可以是例如软件集群，其被配置成对大数据集进行处理和存储，其中一些节点包括分布式存储系统并且一些节点包括分布式处理系统。在这方面，分布式计算集群可以被配置成支持如由Apache软件基金会在www.hadoop.apache.org/docs指定的软件分布式文件系统(HDFS)。The data may be big data processed by a distributed computing cluster. The distributed computing cluster may be, for example, A software cluster configured to process and store large data sets, where some nodes include a distributed storage system and some nodes include a distributed processing system. In this regard, the distributed computing cluster can be configured to support the Apache Software Foundation as specified at www.hadoop.apache.org/docs. Software Distributed File System (HDFS).
如本文所使用的，术语“网络”包括结合硬件和/或软件部件的任何云、云计算系统、或者电子通信系统或方法。各方之间的通信可以通过任何合适的通信信道来完成，诸如，例如电话网络、外联网、内联网、互联网、交互点设备(销售点设备、个人数字助理(例如，设备、设备)、蜂窝电话、信息亭等)、在线通信、卫星通信、离线通信、无线通信、转发器通信、局域网(LAN)、广域网(WAN)、虚拟私有网络(VPN)、联网或链接设备、键盘、鼠标和/或任何合适的通信或数据输入形式。此外，尽管系统在本文经常被描述为使用TCP/IP通信协议实现，但系统也可以使用IPX、程序、IP-6、NetBIOS、OSI、任何隧道协议(例如IPsec、SSH等)、或任何数目的现有或未来协议来实现。如果网络具有公共网络的性质，例如互联网，则假定该网络不安全并且对窃听者开放可能是有利的。与结合互联网使用的协议、标准和应用软件有关的特定信息对于本领域技术人员来说通常是已知的，因此这里不需要详述。As used herein, the term "network" includes any cloud, cloud computing system, or electronic communication system or method that combines hardware and/or software components. Communications between parties may be accomplished through any suitable communication channel, such as, for example, a telephone network, an extranet, an intranet, the Internet, a point of interaction device (point of sale device, a personal digital assistant (e.g., equipment, devices), cellular phones, kiosks, etc.), online communications, satellite communications, offline communications, wireless communications, repeater communications, local area networks (LANs), wide area networks (WANs), virtual private networks (VPNs), networking or linking devices, keyboards, mice, and/or any suitable form of communication or data entry. In addition, although the system is often described herein as being implemented using the TCP/IP communication protocol, the system may also use IPX, The invention can be implemented using a protocol such as TCP, IP-6, NetBIOS, OSI, any tunneling protocol (e.g., IPsec, SSH, etc.), or any number of existing or future protocols. If the network is in the nature of a public network, such as the Internet, it may be advantageous to assume that the network is insecure and open to eavesdroppers. Specific information regarding protocols, standards, and application software used in conjunction with the Internet is generally known to those skilled in the art and does not need to be described in detail here.
“云”或“云计算”包括用于实现对可配置计算资源(例如，网络、服务器、存储、应用程序和服务)的共享池的方便、按需网络访问的模型，这些可配置计算资源可以以最少的管理工作或服务提供商交互来快速供应和发布。云计算可以包括与位置无关的计算，通过共享服务器按需向计算机和其他设备提供资源、软件和数据。"Cloud" or "cloud computing" includes models for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. Cloud computing can include location-independent computing that delivers resources, software, and data on demand to computers and other devices through shared servers.
如本文所使用的，“传输”可以包括通过网络连接将电子数据从一个系统部件发送至另一系统部件。此外，如本文所使用的，“数据”可以包括以数字或任何其他形式涵盖诸如命令、查询、文件、用于存储的数据等的信息。As used herein, "transmission" may include sending electronic data from one system component to another system component via a network connection. In addition, as used herein, "data" may include information in digital or any other form including commands, queries, files, data for storage, etc.
本文所讨论的任何数据库可以包括由多个计算设备(例如，节点)在对等网络上维护的分布式账本。每个计算设备维护分布式账本的副本和/或部分副本，并且与网络中的一个或更多个其他计算设备通信以验证数据并将数据写入分布式账本。分布式账本可以使用区块链技术的特征和功能，包括例如基于共识的验证、不变性和加密链接的数据块。区块链可以包括包含数据的互连块的账本。区块链可以提供增强的安全性，因为每个块都可以保存单独的交易和任何区块链可执行文件的结果。每个块可以链接到前一块并且可以包括时间戳。块可以链接，因为每个块都可以包括区块链中之前块的散列。链接的块形成一条链，其中只有一个后继块允许链接到单个链的另一个前驱块。在从先前统一的区块链建立发散链的情况下，可能会出现分叉，尽管典型地只有一个发散链将被维护为共识链。在各种实施方式中，区块链可以实现以分散方式实施数据工作流的智能合约。该系统还可以包括被部署在用户设备上的应用程序，诸如，例如计算机、平板电脑、智能手机、物联网设备(“IoT”设备)等。应用程序可以(例如，直接或经由区块链节点)与区块链通信以对数据进行传输和检索。在各种实施方式中，管理组织或联盟可以控制对存储在区块链上的数据的访问。利用(一个或更多个)管理组织的注册可以参与区块链网络。Any database discussed herein may include a distributed ledger maintained by multiple computing devices (e.g., nodes) on a peer-to-peer network. Each computing device maintains a copy and/or partial copy of the distributed ledger, and communicates with one or more other computing devices in the network to verify data and write data to the distributed ledger. The distributed ledger may use features and functions of blockchain technology, including, for example, consensus-based verification, immutability, and encrypted linked data blocks. The blockchain may include a ledger containing interconnected blocks of data. The blockchain may provide enhanced security because each block can save separate transactions and the results of any blockchain executable file. Each block may be linked to the previous block and may include a timestamp. Blocks may be linked because each block may include a hash of the previous block in the blockchain. The linked blocks form a chain in which only one successor block is allowed to link to another predecessor block of a single chain. In the case of establishing a divergent chain from a previously unified blockchain, a fork may occur, although typically only one divergent chain will be maintained as a consensus chain. In various embodiments, the blockchain may implement a smart contract that implements a data workflow in a decentralized manner. The system may also include applications deployed on user devices, such as, for example, computers, tablets, smartphones, Internet of Things devices ("IoT" devices), etc. The applications may communicate with the blockchain (e.g., directly or via a blockchain node) to transfer and retrieve data. In various embodiments, a management organization or consortium may control access to data stored on the blockchain. Registration with (one or more) management organizations may allow participation in the blockchain network.
通过基于区块链的系统执行的数据传送可以在可以由所实施的特定区块链技术的块创建时间确定的持续时间内传播至区块链网络内的连接对等点。例如，在基于的网络上，新的数据条目可以在写入完成后的约13秒至20秒内变得可用。在基于Fabric 1.0的平台上，持续时间由所选的特定共识算法驱动，并且可以在几秒钟内执行。在这方面，与现有系统相比，系统中的传播时间可以得到改善，并且实施成本和上市时间也可以大大减少。该系统还提供了更高的安全性，至少部分地是由于存储在区块链中的数据的不可变性，降低了篡改各种数据输入和输出的可能性。此外，该系统还可以通过在将数据存储在区块链上之前对数据执行加密处理来提供更高的数据安全性。因此，通过使用本文所描述的系统对数据进行传输、存储和访问，提高了数据的安全性，这降低了计算机或网络受到危害的风险。Data transmissions performed through a blockchain-based system may be propagated to connected peers within the blockchain network for a duration that may be determined by the block creation time of the particular blockchain technology implemented. On a network based on On the Fabric 1.0 platform, the duration is driven by the specific consensus algorithm selected and can be executed in seconds. In this regard, the propagation time in the system can be improved compared to existing systems, and the implementation cost and time to market can also be greatly reduced. The system also provides higher security, at least in part due to the immutability of the data stored in the blockchain, reducing the possibility of tampering with various data inputs and outputs. In addition, the system can also provide higher data security by performing cryptographic processing on the data before storing it on the blockchain. Therefore, by transmitting, storing, and accessing data using the system described in this article, the security of the data is improved, which reduces the risk of computers or networks being compromised.
在各种实施方式中，系统还可以通过提供通用数据结构来减少数据库同步错误，因此至少部分地提高存储数据的完整性。该系统还提供了比传统数据库(例如，关系数据库、分布式数据库等)更高的可靠性和容错性，因为每个节点都利用存储数据的完整副本进行操作，因此至少部分地减少了由于局部网络中断和硬件故障导致的停机时间。该系统还可以提高在具有可靠和不可靠对等点的网络环境中数据传送的可靠性，因为每个节点都向所有连接的对等点广播消息，并且由于每个块都包括到前一块的链接，因此节点可以快速检测到丢失的块并且将对丢失的块的请求传播至区块链网络中的其他节点。In various embodiments, the system can also reduce database synchronization errors by providing a common data structure, thereby at least partially improving the integrity of stored data. The system also provides higher reliability and fault tolerance than traditional databases (e.g., relational databases, distributed databases, etc.) because each node operates with a complete copy of the stored data, thereby at least partially reducing downtime due to local network outages and hardware failures. The system can also improve the reliability of data transmission in a network environment with reliable and unreliable peers because each node broadcasts messages to all connected peers, and because each block includes a link to the previous block, the node can quickly detect the missing block and propagate the request for the missing block to other nodes in the blockchain network.
本文所描述的特定区块链实现方式通过使用分散式数据库和改进的处理环境提供了对传统技术的改进。特别地，区块链实现方式通过例如利用分散的资源(例如，较低的延时)来提高计算机性能。分布式计算资源通过例如减少处理时间来提高计算机性能。此外，分布式计算资源通过使用例如密码协议提高安全性来提高计算机性能。The specific blockchain implementations described herein provide improvements over conventional techniques by using a decentralized database and an improved processing environment. In particular, the blockchain implementations improve computer performance by, for example, utilizing decentralized resources (e.g., lower latency). Distributed computing resources improve computer performance by, for example, reducing processing time. In addition, distributed computing resources improve computer performance by improving security using, for example, cryptographic protocols.
本文所讨论的任何通信、传输和/或信道可以包括用于传递内容(例如，数据、信息、元数据等)和/或内容本身的任何系统或方法。内容可以以任何形式或介质呈现，并且在各种实施方式中，内容可以以电子方式传递和/或能够以电子方式呈现。例如，渠道可以包括网站、移动应用程序或设备(例如，GOOGLECHROMECASTTM、等)统一资源定位器(“URL”)、文档(例如，Word或EXCELTM、可移植文档格式(PDF)文档等)、“电子书”、“电子杂志”、应用程序或微应用程序(如本文所描述的)、短消息服务(SMS)或其他类型的文本消息、电子邮件、消息、推文、多媒体消息服务(MMS)，和/或其他类型的通信技术。在各种实施方式中，渠道可以由数据伙伴托管或提供。在各种实施方式中，分销渠道可以包括商家网站、社交媒体网站、附属或合作伙伴网站、外部供应商、移动设备通信、社交媒体网络和/或基于位置的服务中的至少一者。分销渠道可以包括商家网站、社交媒体网站、附属或合作伙伴网站、外部供应商和移动设备通信中的至少一者。社交媒体网站的示例包括Any communication, transmission, and/or channel discussed herein may include any system or method for delivering content (e.g., data, information, metadata, etc.) and/or the content itself. The content may be presented in any form or medium, and in various embodiments, the content may be delivered electronically and/or may be capable of being presented electronically. For example, a channel may include a website, a mobile application, or a device (e.g., GOOGLECHROMECASTTM , etc.) Uniform Resource Locators ("URLs"), documents (e.g., Word or EXCELTM , Portable Document Format (PDF) documents, etc.), "electronic books", "electronic magazines", applications or micro-applications (as described herein), short message service (SMS) or other types of text messages, emails, information, In various embodiments, the distribution channel may include at least one of a merchant website, a social media website, an affiliated or partner website, an external supplier, a mobile device communication, a social media network, and/or a location-based service. The distribution channel may include at least one of a merchant website, a social media website, an affiliated or partner website, an external supplier, and a mobile device communication. Examples of social media websites include
等。附属或合作伙伴网站的示例包括AMERICAN等。 etc. Examples of affiliated or partner websites include AMERICAN wait.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US17/708,493 | 2022-03-30 | ||
| US17/708,493US12050133B2 (en) | 2020-03-06 | 2022-03-30 | Pose detection using thermal data |
| PCT/US2023/013980WO2023191987A1 (en) | 2022-03-30 | 2023-02-27 | Pose detection using thermal data |
| Publication Number | Publication Date |
|---|---|
| CN119301428Atrue CN119301428A (en) | 2025-01-10 |
| CN119301428B CN119301428B (en) | 2025-06-17 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202380043635.7AActiveCN119301428B (en) | 2022-03-30 | 2023-02-27 | Posture detection using thermal data |
| Country | Link |
|---|---|
| EP (1) | EP4500127A4 (en) |
| JP (1) | JP7667990B1 (en) |
| CN (1) | CN119301428B (en) |
| AU (1) | AU2023241553B2 (en) |
| WO (1) | WO2023191987A1 (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN120216523B (en)* | 2025-05-27 | 2025-10-03 | 浙江省测绘科学技术研究院 | Method, system, device, terminal and medium for logically checking and quality checking vector data |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102184020A (en)* | 2010-05-18 | 2011-09-14 | 微软公司 | Method for manipulating posture of user interface and posture correction |
| CN102262438A (en)* | 2010-05-18 | 2011-11-30 | 微软公司 | Gestures and gesture recognition for manipulating a user-interface |
| CN102541256A (en)* | 2010-10-28 | 2012-07-04 | 微软公司 | Position aware gestures with visual feedback as input method |
| CA2781511A1 (en)* | 2011-06-24 | 2012-12-24 | American Express Travel Related Services Company, Inc. | Systems and methods for gesture-based interaction with computer systems |
| CN112651291A (en)* | 2020-10-01 | 2021-04-13 | 新加坡依图有限责任公司(私有) | Video-based posture estimation method, device, medium and electronic equipment |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2978374B2 (en)* | 1992-08-21 | 1999-11-15 | 松下電器産業株式会社 | Image processing device, image processing method, and control device for air conditioner |
| WO2004110248A2 (en)* | 2003-05-27 | 2004-12-23 | Cardiowave, Inc. | Remote technique to detect core body temperature in a subject using thermal imaging |
| US8718748B2 (en)* | 2011-03-29 | 2014-05-06 | Kaliber Imaging Inc. | System and methods for monitoring and assessing mobility |
| CA2773507C (en) | 2011-04-04 | 2020-10-13 | Mark Andrew Hanson | Fall detection and reporting technology |
| US8509495B2 (en)* | 2011-04-15 | 2013-08-13 | Xerox Corporation | Subcutaneous vein pattern detection via multi-spectral IR imaging in an identity verification system |
| EP3143931B1 (en) | 2014-05-13 | 2020-12-09 | Omron Corporation | Posture estimation device and posture estimation method |
| US9989965B2 (en)* | 2015-08-20 | 2018-06-05 | Motionloft, Inc. | Object detection and analysis via unmanned aerial vehicle |
| KR102013935B1 (en)* | 2017-05-25 | 2019-08-23 | 삼성전자주식회사 | Method and system for detecting a dangerous situation |
| WO2020102988A1 (en) | 2018-11-20 | 2020-05-28 | 西安电子科技大学 | Feature fusion and dense connection based infrared plane target detection method |
| JP7196645B2 (en) | 2019-01-31 | 2022-12-27 | コニカミノルタ株式会社 | Posture Estimation Device, Action Estimation Device, Posture Estimation Program, and Posture Estimation Method |
| KR20220132617A (en)* | 2020-01-29 | 2022-09-30 | 인트린식 이노베이션 엘엘씨 | Systems and methods for pose detection and measurement |
| US11320312B2 (en)* | 2020-03-06 | 2022-05-03 | Butlr Technologies, Inc. | User interface for determining location, trajectory and behavior |
| US20210279967A1 (en)* | 2020-03-06 | 2021-09-09 | Apple Inc. | Object centric scanning |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102184020A (en)* | 2010-05-18 | 2011-09-14 | 微软公司 | Method for manipulating posture of user interface and posture correction |
| CN102262438A (en)* | 2010-05-18 | 2011-11-30 | 微软公司 | Gestures and gesture recognition for manipulating a user-interface |
| CN102541256A (en)* | 2010-10-28 | 2012-07-04 | 微软公司 | Position aware gestures with visual feedback as input method |
| CA2781511A1 (en)* | 2011-06-24 | 2012-12-24 | American Express Travel Related Services Company, Inc. | Systems and methods for gesture-based interaction with computer systems |
| CN112651291A (en)* | 2020-10-01 | 2021-04-13 | 新加坡依图有限责任公司(私有) | Video-based posture estimation method, device, medium and electronic equipment |
| Publication number | Publication date |
|---|---|
| AU2023241553A1 (en) | 2024-10-17 |
| JP7667990B1 (en) | 2025-04-24 |
| AU2023241553B2 (en) | 2024-12-12 |
| CN119301428B (en) | 2025-06-17 |
| EP4500127A1 (en) | 2025-02-05 |
| JP2025514572A (en) | 2025-05-07 |
| EP4500127A4 (en) | 2025-07-09 |
| WO2023191987A1 (en) | 2023-10-05 |
| Publication | Publication Date | Title |
|---|---|---|
| US12050133B2 (en) | Pose detection using thermal data | |
| Taiwo et al. | Enhanced intelligent smart home control and security system based on deep learning model | |
| US11320312B2 (en) | User interface for determining location, trajectory and behavior | |
| US20240169764A1 (en) | Computer vision system | |
| US11959805B2 (en) | Thermal data analysis for determining location, trajectory and behavior | |
| US11182622B2 (en) | Systems and methods for locating image data for selected regions of interest | |
| US10586433B2 (en) | Automatic detection of zones of interest in a video | |
| US20180018681A1 (en) | Holographic Technology Implemented Retail Solutions | |
| CN107683491A (en) | Start physical object using Internet of Things to perform the specific action that enhancing user interacts with physical object | |
| US11257000B2 (en) | Systems, apparatus, and methods for generating prediction sets based on a known set of features | |
| CN119301428B (en) | Posture detection using thermal data | |
| Khan | A Smart Fire Detector IoT System with Extinguisher Class Recommendation Using Deep Learning | |
| US20220093277A1 (en) | Path analytics of disease vectors in a physical space using smart floor tiles | |
| Devare | Analysis and design of IoT based physical location monitoring system |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |