技术领域technical field
本发明涉及康复训练技术领域，尤其是涉及上肢康复机器人主动训练方法、装置及上肢康复机器人。The invention relates to the technical field of rehabilitation training, in particular to an active training method and device for an upper limb rehabilitation robot and an upper limb rehabilitation robot.
背景技术Background technique
脑卒中俗称中风，是由于脑血管堵塞或破裂而造成的急性脑血管循环障碍疾病，具有高发病率、高死亡率、高致残率和高复发率等特点。在中风患者中只有少数轻微患者可以自然恢复，大部分脑卒中患者会残留不同程度的运动、感觉、认知、言语等功能障碍，严重影响其日常活动能力和生活质量。康复是降低卒中致残率最有效的方法，及时、科学、有效的康复训练，尤其是在疾病早期的“黄金期”，在治疗疾病、功能恢复、预防卒中复发、减少并发症等方面发挥着弥足轻重的作用。在传统的中风康复治疗中，主要通过医师以徒手方式对患者进行一对一的康复治疗，医师个人的治疗手段、经验差异、主观意识以及疲劳程度会直接影响治疗效果，治疗过程劳动强度大，护理成本高昂；以及医师和患者的数量比例严重失衡，难以满足日益增长的医疗需求，因此，引入医疗康复类机器人设备是帮助有效缓解康复供需矛盾的可行方案。Stroke, commonly known as apoplexy, is an acute cerebrovascular circulation disorder caused by blockage or rupture of cerebral blood vessels. It has the characteristics of high morbidity, high mortality, high disability and high recurrence rate. Among stroke patients, only a small number of mild patients can recover naturally, and most stroke patients will have residual motor, sensory, cognitive, speech and other functional impairments to varying degrees, seriously affecting their daily activities and quality of life. Rehabilitation is the most effective way to reduce the disability rate of stroke. Timely, scientific and effective rehabilitation training, especially in the early "golden period" of the disease, plays an important role in the treatment of diseases, functional recovery, prevention of stroke recurrence, and reduction of complications. Insignificant role. In the traditional stroke rehabilitation treatment, doctors mainly perform one-on-one rehabilitation treatment on patients with bare hands. Personal treatment methods, experience differences, subjective consciousness and fatigue degree of doctors will directly affect the treatment effect, and the treatment process is labor-intensive. The high cost of nursing care and the serious imbalance between the number of doctors and patients make it difficult to meet the growing medical needs. Therefore, the introduction of medical rehabilitation robotic equipment is a feasible solution to help effectively alleviate the contradiction between supply and demand for rehabilitation.
康复机器人可以辅助甚至替代医师为患者提供更加持续、有效以及更具针对性的康复训练治疗，缓解康复医疗人力资源紧缺问题，而且可以实时记录患者的治疗数据，为病情评估和方案改进提供客观依据。在实际应用中，康复机器人主要提供主动和被动两种康复训练模式，随着神经可塑性和功能重组理论及实践的深入研究，主动性康复的作用远大于被动性运动。Rehabilitation robots can assist or even replace physicians to provide patients with more continuous, effective and targeted rehabilitation training, alleviate the shortage of rehabilitation medical human resources, and can record patients' treatment data in real time, providing objective basis for disease assessment and program improvement . In practical applications, rehabilitation robots mainly provide active and passive rehabilitation training modes. With the in-depth research on the theory and practice of neural plasticity and functional reorganization, the role of active rehabilitation is far greater than that of passive exercise.
康复机器人的主动康复训练是指患者主动发起运动，但由于患者的肢体受损无法独立产生完成运动所需的全部力/力矩，因此，需要康复机器人提供部分辅助力/力矩，康复机器人在识别到患者的运动意图后提供所需的辅助力/力矩以协助患者完成运动。其中，主动康复训练的难点在于如何识别患者的运动意图，现有方法主要通过脑电或肌电信号识别患者的运动意图来实现主动训练，但是采集脑电或肌电信号的传感器价格比较昂贵，实用性不足，极大的限制了康复机器人主动训练模式的推广。The active rehabilitation training of the rehabilitation robot refers to the patient's initiative to initiate movement, but the patient's limbs are damaged and cannot independently generate all the force/torque required to complete the movement. Therefore, the rehabilitation robot is required to provide some auxiliary force/torque. After the patient's motion intention, the required assisting force/torque is provided to assist the patient to complete the motion. Among them, the difficulty of active rehabilitation training lies in how to identify the patient's movement intention. The existing methods mainly recognize the patient's movement intention through EEG or EMG signals to achieve active training, but the sensors for collecting EEG or EMG signals are relatively expensive. Insufficient practicability greatly limits the promotion of the active training mode of rehabilitation robots.
发明内容Contents of the invention
有鉴于此，本发明的目的在于提供上肢康复机器人主动训练方法、装置及上肢康复机器人，以缓解上述问题，降低了上肢康复机器人主动训练成本，提高了主动训练模式的实用性，便于在实际应用中推广实施。In view of this, the object of the present invention is to provide an active training method, device and upper limb rehabilitation robot for the upper limb rehabilitation robot, to alleviate the above problems, reduce the active training cost of the upper limb rehabilitation robot, improve the practicability of the active training mode, and facilitate practical application promotion and implementation.
第一方面，本发明实施例提供了一种上肢康复机器人主动训练方法，该方法包括：获取包含待训练手臂的RGB图像，并基于正运动学模型从RGB图像中提取感兴趣区域ROI，得到ROI图像；其中，ROI图像覆盖待训练手臂的所有运动区域；基于Farneback光流算法计算ROI图像的稠密光流；获取待训练手臂末端在ROI图像中的当前位置，并根据当前位置确定待训练手臂末端在ROI图像中的邻域；根据邻域内的光流确定待训练手臂末端的实际速度向量；获取待训练手臂末端在ROI图像中的目标位置，并根据目标位置和当前位置确定待训练手臂末端的目标速度向量；根据实际速度向量和目标速度向量确定待训练手臂的运动意图，并根据运动意图辅助待训练手臂进行康复训练，直至运动至目标位置。In the first aspect, an embodiment of the present invention provides an active training method for an upper limb rehabilitation robot. The method includes: acquiring an RGB image containing an arm to be trained, and extracting a region of interest ROI from the RGB image based on a positive kinematics model to obtain the ROI Image; wherein, the ROI image covers all the motion areas of the arm to be trained; the dense optical flow of the ROI image is calculated based on the Farneback optical flow algorithm; the current position of the end of the arm to be trained in the ROI image is obtained, and the end of the arm to be trained is determined according to the current position Neighborhood in the ROI image; determine the actual velocity vector of the end of the arm to be trained according to the optical flow in the neighborhood; obtain the target position of the end of the arm to be trained in the ROI image, and determine the end of the arm to be trained according to the target position and current position Target velocity vector: Determine the movement intention of the arm to be trained according to the actual velocity vector and the target velocity vector, and assist the arm to be trained to carry out rehabilitation training according to the movement intention until it moves to the target position.
优选地，基于Farneback光流算法计算ROI图像的稠密光流的步骤，包括：对ROI图像进行灰度转换处理，得到第一灰度图像；对第一灰度图像进行中值滤波处理，得到滤波后的第二灰度图像；基于Farneback光流算法，根据当前帧的第二灰度图像和上一帧的第二灰度图像，计算得到当前帧的第二灰度图像的光流。Preferably, the step of calculating the dense optical flow of the ROI image based on the Farneback optical flow algorithm includes: performing grayscale conversion processing on the ROI image to obtain a first grayscale image; performing median filtering processing on the first grayscale image to obtain a filtered The final second grayscale image; based on the Farneback optical flow algorithm, calculate the optical flow of the second grayscale image of the current frame according to the second grayscale image of the current frame and the second grayscale image of the previous frame.
优选地，根据当前位置确定待训练手臂末端在ROI图像中的邻域的步骤，包括：根据待训练手臂形状确定邻域宽度和邻域高度；以当前位置为中心，建立宽度为邻域宽度、高度为邻域高度的矩形区域，作为待训练手臂末端在ROI图像中的邻域。Preferably, the step of determining the neighborhood of the end of the arm to be trained in the ROI image according to the current position includes: determining the neighborhood width and neighborhood height according to the shape of the arm to be trained; taking the current position as the center, establishing the width as the neighborhood width, A rectangular area whose height is the height of the neighborhood is used as the neighborhood of the end of the arm to be trained in the ROI image.
优选地，光流包括X方向的第一光流和Y方向的第二光流；根据邻域内的光流确定待训练手臂末端的实际速度向量的步骤，包括：对邻域内的全部第一光流进行平均计算，得到X方向的第一速度；以及，对邻域内的全部第二光流进行平均计算，得到Y方向的第二速度；根据第一速度和第二速度得到实际速度向量。Preferably, the optical flow includes the first optical flow in the X direction and the second optical flow in the Y direction; the step of determining the actual velocity vector of the end of the arm to be trained according to the optical flow in the neighborhood includes: for all the first optical flow in the neighborhood The flow is averaged to obtain the first velocity in the X direction; and, all the second optical flows in the neighborhood are averaged to obtain the second velocity in the Y direction; an actual velocity vector is obtained according to the first velocity and the second velocity.
优选地，根据实际速度向量和目标速度向量确定待训练手臂的运动意图的步骤，包括：计算实际速度向量在目标速度向量上的投影；判断投影是否满足预设运动条件；如果是，确定待训练手臂产生正确的运动意图，并辅助待训练手臂进行康复训练，直至运动至目标位置。Preferably, the step of determining the movement intention of the arm to be trained according to the actual velocity vector and the target velocity vector includes: calculating the projection of the actual velocity vector on the target velocity vector; judging whether the projection meets the preset movement conditions; The arm produces the correct movement intention, and assists the arm to be trained to perform rehabilitation training until it moves to the target position.
优选地，判断投影是否满足预设运动条件的步骤，包括：判断投影的方向是否为正，且，投影的幅值是否达到预设阈值；如果均是，判定投影满足预设运动条件。Preferably, the step of judging whether the projection satisfies the preset motion condition includes: judging whether the direction of the projection is positive, and whether the amplitude of the projection reaches a preset threshold; if both are, judging that the projection satisfies the preset motion condition.
优选地，该方法还包括：如果投影不满足预设运动条件，确定待训练手臂产生错误的运动意图，并停止辅助待训练手臂进行康复训练。Preferably, the method further includes: if the projection does not satisfy the preset motion condition, determining that the arm to be trained produces a wrong motion intention, and stopping assisting the arm to be trained for rehabilitation training.
第二方面，本发明实施例还提供一种上肢康复机器人主动训练装置，该装置包括：图像获取模块，用于获取包含待训练手臂的RGB图像，并基于正运动学模型从RGB图像中提取感兴趣区域ROI，得到ROI图像；其中，ROI图像覆盖待训练手臂的所有运动区域；光流计算模块，用于基于Farneback光流算法计算ROI图像的稠密光流；邻域确定模块，用于获取待训练手臂末端在ROI图像中的当前位置，并根据当前位置确定待训练手臂末端在ROI图像中的邻域；实际速度确定模块，用于根据邻域内的光流确定待训练手臂末端的实际速度向量；目标速度确定模块，用于获取待训练手臂末端在ROI图像中的目标位置，并根据目标位置和当前位置确定待训练手臂末端的目标速度向量；运动意图确定模块，用于根据实际速度向量和目标速度向量确定待训练手臂的运动意图，并根据运动意图辅助待训练手臂进行康复训练，直至运动至目标位置。In the second aspect, the embodiment of the present invention also provides an active training device for an upper limb rehabilitation robot, which includes: an image acquisition module, configured to acquire an RGB image containing the arm to be trained, and extract sensory information from the RGB image based on a positive kinematics model. The region of interest ROI obtains the ROI image; wherein, the ROI image covers all motion areas of the arm to be trained; the optical flow calculation module is used to calculate the dense optical flow of the ROI image based on the Farneback optical flow algorithm; the neighborhood determination module is used to obtain the Train the current position of the end of the arm in the ROI image, and determine the neighborhood of the end of the arm to be trained in the ROI image according to the current position; the actual speed determination module is used to determine the actual speed vector of the end of the arm to be trained according to the optical flow in the neighborhood The target speed determination module is used to obtain the target position of the arm end to be trained in the ROI image, and determines the target speed vector of the arm end to be trained according to the target position and current position; the motion intention determination module is used to determine the target according to the actual speed vector and The target velocity vector determines the movement intention of the arm to be trained, and assists the arm to be trained to carry out rehabilitation training according to the movement intention until it moves to the target position.
第三方面，本发明实施例还提供一种上肢康复机器人，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，处理器执行计算机程序时实现上述第一方面的方法的步骤。In the third aspect, the embodiment of the present invention also provides an upper limb rehabilitation robot, including a memory, a processor, and a computer program stored in the memory and operable on the processor. When the processor executes the computer program, the method of the above-mentioned first aspect is realized. A step of.
第四方面，本发明实施例还提供一种计算机可读存储介质，计算机可读存储介质上存储有计算机程序，计算机程序被处理器运行时执行上述第一方面的方法的步骤。In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, the steps of the above-mentioned method in the first aspect are executed.
本发明实施例带来了以下有益效果：Embodiments of the present invention bring the following beneficial effects:
本发明实施例提供了上肢康复机器人主动训练方法、装置及上肢康复机器人，首先基于正运动学模型从RGB图像中提取感兴趣区域ROI，得到ROI图像；基于Farneback光流算法计算ROI图像的稠密光流，并确定待训练手臂末端在ROI图像中的邻域；根据邻域内的光流确定待训练手臂末端的实际速度向量；最后确定待训练手臂末端的目标速度向量，并根据实际速度向量和目标速度向量确定待训练手臂的运动意图，并根据运动意图辅助待训练手臂进行康复训练，直至运动至目标位置；上述训练方法在患者发起主动运动后肢体会产生微小位移，通过RGB图像捕获位移的大小和方向，以识别出患者的运动意图实现主动训练，从而降低了上肢康复机器人主动训练成本，提高了主动训练模式的实用性，便于在实际应用中推广实施。The embodiment of the present invention provides an active training method and device for an upper limb rehabilitation robot, and first extracts the region of interest ROI from the RGB image based on the forward kinematics model to obtain the ROI image; calculates the dense optical density of the ROI image based on the Farneback optical flow algorithm. Flow, and determine the neighborhood of the end of the arm to be trained in the ROI image; determine the actual velocity vector of the end of the arm to be trained according to the optical flow in the neighborhood; finally determine the target velocity vector of the end of the arm to be trained, and according to the actual velocity vector and the target The velocity vector determines the movement intention of the arm to be trained, and assists the arm to perform rehabilitation training according to the movement intention until it moves to the target position; the above training method will produce a small displacement of the limb after the patient initiates an active movement, and the size of the displacement is captured through the RGB image and direction, to realize the active training by identifying the patient's movement intention, thereby reducing the cost of active training of the upper limb rehabilitation robot, improving the practicability of the active training mode, and facilitating the promotion and implementation in practical applications.
本发明的其他特征和优点将在随后的说明书中阐述，并且，部分地从说明书中变得显而易见，或者通过实施本发明而了解。本发明的目的和其他优点在说明书以及附图中所特别指出的结构来实现和获得。Additional features and advantages of the invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and appended drawings.
为使本发明的上述目的、特征和优点能更明显易懂，下文特举较佳实施例，并配合所附附图，作详细说明如下。In order to make the above-mentioned objects, features and advantages of the present invention more comprehensible, preferred embodiments will be described in detail below together with the accompanying drawings.
附图说明Description of drawings
为了更清楚地说明本发明具体实施方式或现有技术中的技术方案，下面将对具体实施方式或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图是本发明的一些实施方式，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the following will briefly introduce the accompanying drawings that need to be used in the specific implementation or description of the prior art. Obviously, the accompanying drawings in the following description The drawings show some implementations of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.
图1为本发明实施例提供的一种上肢康复机器人的结构示意图；Fig. 1 is a schematic structural diagram of an upper limb rehabilitation robot provided by an embodiment of the present invention;
图2为本发明实施例提供的一种上肢康复机器人主动训练方法的流程图；2 is a flow chart of an active training method for an upper limb rehabilitation robot provided by an embodiment of the present invention;
图3为本发明实施例提供的一种正运动学模型的示意图；Fig. 3 is a schematic diagram of a forward kinematics model provided by an embodiment of the present invention;
图4为本发明实施例提供的一种感兴趣区域示意图；FIG. 4 is a schematic diagram of a region of interest provided by an embodiment of the present invention;
图5为本发明实施例提供的一种待训练手臂末端邻域示意图；Fig. 5 is a schematic diagram of the neighborhood of the end of the arm to be trained provided by the embodiment of the present invention;
图6为本发明实施例提供的一种待训练手臂末端运动状态下的光流示意图；Fig. 6 is a schematic diagram of optical flow in the state of movement of the end of the arm to be trained provided by the embodiment of the present invention;
图7为本发明实施例提供的一种上肢康复机器人主动训练装置的示意图；7 is a schematic diagram of an active training device for an upper limb rehabilitation robot provided by an embodiment of the present invention;
图8为本发明实施例提供的另一种上肢康复机器人的结构示意图。Fig. 8 is a schematic structural diagram of another upper limb rehabilitation robot provided by an embodiment of the present invention.
具体实施方式Detailed ways
为使本发明实施例的目的、技术方案和优点更加清楚，下面将结合附图对本发明的技术方案进行清楚、完整地描述，显然，所描述的实施例是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. the embodiment. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.
为便于对本实施例进行理解，下面首先对本发明实施例提供的上肢康复机器人主动训练方法进行详细介绍。其中，如图1所示，上肢康复机器人包括显示端10和主机端20；具体地，显示端10包括显示装置101和视觉传感器102，显示装置101用于实时显示ROI(RegionOf Interest，感兴趣区域)图像、待训练手臂末端邻域内的光流以及末端目标位置等，视觉传感器102固定安装在显示端10，用于获取包含待训练手臂的RGB图像；主机端20则包括上肢外骨骼201和控制主机202，上肢外骨骼201用于提供辅助力/力矩协助患者完成康复训练，控制主机202则用于调节上肢外骨骼201输出的辅助力/力矩。In order to facilitate the understanding of this embodiment, the active training method for the upper limb rehabilitation robot provided by the embodiment of the present invention will firstly be introduced in detail below. Wherein, as shown in Figure 1, the upper limb rehabilitation robot includes a display terminal 10 and a host terminal 20; specifically, the display terminal 10 includes a display device 101 and a visual sensor 102, and the display device 101 is used for real-time display of ROI (RegionOfInterest, region of interest) ) image, the optical flow in the neighborhood of the end of the arm to be trained, and the position of the end target, etc., the visual sensor 102 is fixedly installed on the display end 10 to obtain an RGB image containing the arm to be trained; the host end 20 includes an upper limb exoskeleton 201 and a control The host 202 and the upper extremity exoskeleton 201 are used to provide auxiliary force/torque to assist the patient to complete rehabilitation training, and the control host 202 is used to adjust the auxiliary force/torque output by the upper extremity exoskeleton 201 .
需要说明的是，在实际应用中，上肢外骨骼201具有左右手切换功能，以及大臂长度和小臂长度调节功能，并且至少包含以下4个自由度：肩关节外摆/内收自由度、前屈/后伸自由度，旋内/旋外自由度，肘关节屈曲/伸展自由度，具体可以根据实际情况进行设置。It should be noted that in practical applications, the upper limb exoskeleton 201 has the function of switching between left and right hands, as well as the function of adjusting the length of the upper arm and the length of the forearm, and includes at least the following 4 degrees of freedom: shoulder joint swing/adduction degree of freedom, front Flexion/extension degree of freedom, internal rotation/external rotation degree of freedom, elbow flexion/extension degree of freedom, which can be set according to the actual situation.
基于上述上肢康复机器人，本发明实施例提供了一种上肢康复机器人主动训练方法，如图2所示，该方法包括以下步骤：Based on the above-mentioned upper limb rehabilitation robot, an embodiment of the present invention provides an active training method for an upper limb rehabilitation robot, as shown in Figure 2, the method includes the following steps:
步骤S202，获取包含待训练手臂的RGB图像，并基于正运动学模型从RGB图像中提取感兴趣区域ROI，得到ROI图像；Step S202, acquiring the RGB image containing the arm to be trained, and extracting the region of interest ROI from the RGB image based on the forward kinematics model to obtain the ROI image;
具体地，当患者(非完全瘫痪的)发起主动运动后，待训练手臂会产生微小位移，此时，视觉传感器可以捕获位移的大小和方向，生成包含待训练手臂的RGB图像，控制主机获取到RGB图像后，基于正运动学模型对RGB图像进行裁剪处理，以提取感兴趣区域ROI图像，这里ROI图像覆盖待训练手臂的所有运动区域。Specifically, when the patient (not completely paralyzed) initiates an active movement, the arm to be trained will produce a small displacement. At this time, the visual sensor can capture the magnitude and direction of the displacement, generate an RGB image containing the arm to be trained, and the control host can obtain After the RGB image, the RGB image is cropped based on the positive kinematics model to extract the ROI image of the region of interest, where the ROI image covers all the motion areas of the arm to be trained.
对于正运动学模型，如图3所示，将上肢建模成四自由度的二连杆结构(大臂+小臂)，其中，坐标系XYZ表示相机坐标系；SH[x0,y0,z0]是肩关节中心在相机坐标系中的位置坐标(由安装位置确定)；EL表示肘关节中心；WR表示腕关节中心，即待训练手臂末端；Lu是大臂长度(从SH到EL的距离)；Ll是小臂长度(从EL到WR的距离)；q1、q2、q3、q4分别表示肩关节外摆/内收、肩关节前屈/后伸、肩关节旋内/旋外、肘关节屈曲/伸展的角度。For the forward kinematics model, as shown in Figure 3, the upper limb is modeled as a four-degree-of-freedom two-link structure (big arm + small arm), where the coordinate system XYZ represents the camera coordinate system; SH[x0 ,y0 , z0 ] is the position coordinate of the shoulder joint center in the camera coordinate system (determined by the installation position); EL indicates the elbow joint center; WR indicates the wrist joint center, which is the end of the arm to be trained; Lu is the length of the arm (from SH distance to EL); Ll is the length of the forearm (the distance from EL to WR); q1, q2, q3, and q4 respectively represent the shoulder joint swing/adduction, shoulder flexion/extension, and shoulder joint rotation /external rotation, angle of elbow flexion/extension.
基于上述正运动学模型可以确定待训练手臂末端(即腕关节中心WR)在相机坐标系中的位置，即：Based on the above forward kinematics model, the position of the end of the arm to be trained (that is, the center of the wrist joint WR) in the camera coordinate system can be determined, namely:
再通过相机内参矩阵K变换到像素坐标系，可以得到待训练手臂末端在RGB图像中的像素坐标：Then transform to the pixel coordinate system through the camera internal reference matrix K, the pixel coordinates of the end of the arm to be trained in the RGB image can be obtained:
其中，[x0,y0,z0]表示肩关节中心在相机坐标系中的位置坐标(由安装位置确定)，Lu表示大臂长度，Ll表示小臂长度，q1、q2、q3、q4分别表示肩关节外摆/内收、肩关节前屈/后伸、肩关节旋内/旋外、肘关节屈曲/伸展的角度，[xe,ye]表示腕关节中心在RGB图像像素坐标系中的像素坐标，K∈R3×3表示相机的内参数矩阵(由厂家给定)，相机坐标系与像素坐标系通过内参数矩阵进行变换。Among them, [x0 , y0 , z0 ] represent the position coordinates of the shoulder joint center in the camera coordinate system (determined by the installation position),Lu represents the length of the arm, Ll represents the length of the forearm, q1, q2, q3 , q4 represent the angles of shoulder joint swing/adduction, shoulder joint flexion/extension, shoulder joint internal/external rotation, elbow flexion/extension angle, [xe , ye ] represent the angle of the wrist joint center in the RGB image The pixel coordinates in the pixel coordinate system, K∈R3×3 represent the internal parameter matrix of the camera (given by the manufacturer), and the camera coordinate system and the pixel coordinate system are transformed through the internal parameter matrix.
因此，以q1、q2、q3、q4、Lu和Ll的取值范围作为约束，建立最优化问题可求出xe的最小值xe,min和最大值xe,max，以及ye的最小值ye,min和最大值ye,max，在RGB图像上截取[xe,min-we:xe,max+we,ye,min-he:ye,max+he]的矩形区域即为ROI区域，如图4所示的灰色阴影区域。图中采用标准像素坐标系，原点在图像左上角，X轴朝右，Y轴朝下；xe,min是腕关节中心x坐标的最小值，xe,max是腕关节中心x坐标的最大值；ye,min是腕关节中心y坐标的最小值；ye,max是腕关节中心y坐标的最大值；we表示宽度裕量，he表示高度裕量。需要说明的是，宽度裕量we和高度裕量he可以根据历史经验值或实验值设定。Therefore, with the value range of q1, q2, q3, q4,Lu and Ll as constraints, the establishment of an optimization problem can find the minimum value xe,min and the maximum value xe,max of x e, and ye The minimum value ye,min and the maximum value ye,max are intercepted on the RGB image [xe,min -we :xe,max +we ,ye,min -he :ye,max + he ] is the ROI area, as shown in the gray shaded area in Figure 4. The standard pixel coordinate system is used in the figure, the origin is in the upper left corner of the image, the X axis is facing right, and the Y axis is facing down; xe, min is the minimum value of the x coordinate of the center of the wrist joint, and xe, max is the maximum value of the x coordinate of the center of the wrist joint value; ye, min is the minimum value of the y coordinate of the wrist joint center; ye, max is the maximum value of the wrist joint center y coordinate; we represents the width margin, and hee represents the height margin. It should be noted that the width margin we and the height marginhe may be set according to historical experience values or experimental values.
综上，通过从RGB图像中提取ROI图像，可以减少计算量从而提高算法实时性。由于光流法的缺点之一是运算速度较慢，提取ROI图像相当于对视觉传感器输出的原始RGB图像进行裁剪，裁剪的范围是待训练手臂所有可能的运动区域，后续算法均在裁剪后的图像即ROI图像上进行，从而能够减小计算量，提高光流法的运算速度。In summary, by extracting the ROI image from the RGB image, the amount of calculation can be reduced to improve the real-time performance of the algorithm. One of the disadvantages of the optical flow method is the slow calculation speed. Extracting the ROI image is equivalent to cropping the original RGB image output by the visual sensor. The range of cropping is all possible motion areas of the arm to be trained. Subsequent algorithms are all after cropping. The image, that is, the ROI image, can reduce the amount of calculation and improve the calculation speed of the optical flow method.
步骤S204，基于Farneback光流算法计算ROI图像的稠密光流；Step S204, calculating the dense optical flow of the ROI image based on the Farneback optical flow algorithm;
目前，常用的视觉运动检测算法主要有帧间差分法、背景差分法和光流法等；光流法由于不需要预先知道场景的任何信息就能够检测出独立的运动目标，从而能够获得运动目标的完整信息，故适用于动态背景中。此外，与神经网络相比，由于神经网络更适合识别大位移的动作，在小位移的识别上效果一般，故针对患者发起主动运动后的肢体产生的微小位移，本发明实施例采用光流法进行视觉运动检测。At present, the commonly used visual motion detection algorithms mainly include inter-frame difference method, background difference method and optical flow method, etc.; the optical flow method can detect independent moving objects without knowing any information of the scene in advance, so that it can obtain the information of moving objects. Complete information, so suitable for dynamic background. In addition, compared with the neural network, since the neural network is more suitable for recognizing large-displacement actions, it is generally effective in the recognition of small displacements. Therefore, the embodiment of the present invention uses the optical flow method for the small displacement of the limbs after the patient initiates active movement. Perform visual motion detection.
其中，传统光流法主要包括稠密光流与稀疏光流，由于稠密光流是一种针对图像或指定的某一片区域进行逐点匹配的图像配准方法，它计算图像上所有的点的偏移量，从而形成一个稠密的光流场，通过这个稠密的光流场，可以进行像素级别的图像配准，故本发明实施例计算ROI图像的稠密光流。常用计算稠密光流的方法主要包括Brox算法、Farneback算法和TVL1算法等，由于Farneback算法实时性较好，且，在中低端GPU(GraphicsProcessing Unit，图形处理器)上的帧率能够达到30帧，能够满足实时使用的需求，因此，本发明实施例采用Farneback光流算法计算ROI图像的稠密光流。Among them, the traditional optical flow method mainly includes dense optical flow and sparse optical flow. Since dense optical flow is an image registration method that performs point-by-point matching on an image or a specified area, it calculates the deviation of all points on the image. shift, thereby forming a dense optical flow field, through which pixel-level image registration can be performed, so the embodiment of the present invention calculates the dense optical flow of the ROI image. The commonly used methods for calculating dense optical flow mainly include Brox algorithm, Farneback algorithm and TVL1 algorithm, etc., because the Farneback algorithm has good real-time performance, and the frame rate on the middle and low-end GPU (Graphics Processing Unit, graphics processing unit) can reach 30 frames , can meet the requirement of real-time use, therefore, the embodiment of the present invention uses the Farneback optical flow algorithm to calculate the dense optical flow of the ROI image.
具体地，对于上述ROI图像，计算光流的过程如下：对ROI图像进行灰度转换处理，得到第一灰度图像ROI_gray；对第一灰度图像ROI_gray进行中值滤波处理，得到滤波后的第二灰度图像ROI_gray_filted；基于Farneback光流算法，根据当前帧的第二灰度图像ROI_gray_filted和上一帧的第二灰度图像ROI_gray_filted，计算得到当前帧的第二灰度图像的光流flow。其中，光流flow由X和Y两个方向的光流组成，进一步将其拆分为X方向的光流flowx和Y方向的光流flowy，令W＝xe,max-xe,min+2we，H＝ye,max-ye,min+2he，则flowx和flowy都是维度为W×H的矩阵。Specifically, for the above ROI image, the process of calculating the optical flow is as follows: perform grayscale conversion processing on the ROI image to obtain the first grayscale image ROI_gray; perform median filter processing on the first grayscale image ROI_gray to obtain the filtered first grayscale image ROI_gray Two-grayscale image ROI_gray_filted; based on the Farneback optical flow algorithm, calculate the optical flow flow of the second grayscale image of the current frame based on the second grayscale image ROI_gray_filted of the current frame and the second grayscale image ROI_gray_filted of the previous frame. Among them, the optical flow flow is composed of the optical flow in the X and Y directions, which is further split into the optical flow flowx in the X direction and the optical flow flowy in the Y direction, so that W=xe,max -xe,min + 2we , H=ye,max -ye,min +2he , then both flowx and flowy are matrices with dimension W×H.
步骤S206，获取待训练手臂末端在ROI图像中的当前位置，并根据当前位置确定待训练手臂末端在ROI图像中的邻域；Step S206, obtaining the current position of the end of the arm to be trained in the ROI image, and determining the neighborhood of the end of the arm to be trained in the ROI image according to the current position;
由于ROI图像是在原始RGB图像上裁剪后得到的，因此，待训练手臂末端在ROI图像中的像素坐标与RGB图像中像素坐标发生了偏移，可以按照如下公式确定待训练手臂末端(即腕关节中心)在ROI图像中的像素坐标：Since the ROI image is obtained after cropping the original RGB image, the pixel coordinates of the end of the arm to be trained in the ROI image are offset from the pixel coordinates in the RGB image, and the end of the arm to be trained (that is, the wrist Joint center) pixel coordinates in the ROI image:
其中，[xe,ye]表示腕关节中心在RGB图像像素坐标系中的像素坐标，we表示宽度裕量，he表示高度裕量，表示待训练手臂末端在ROI图像中的像素坐标即当前位置。Among them, [xe , ye ] represents the pixel coordinates of the wrist joint center in the RGB image pixel coordinate system, we represents the width margin,he represents the height margin, Indicates the pixel coordinates of the end of the arm to be trained in the ROI image, that is, the current position.
如图5所示，图中采用标准像素坐标系，原点在图像左上角，X轴朝右，Y轴朝下；表示待训练手臂末端在ROI图像像素坐标系中的x坐标，/>表示待训练手臂末端在ROI图像像素坐标系中的y坐标；/>表示邻域宽度，/>表示邻域高度，/>和/>可根据手臂形状确定，基于上述待训练手臂末端在ROI图像中的当前位置/>确定邻域的过程如下：根据待训练手臂形状确定邻域宽度/>和邻域高度/>以当前位置/>为中心，建立宽度为邻域宽度/>高度为邻域高度/>的矩形区域，作为待训练手臂末端在ROI图像中的邻域即图5中矩形区域。需要说明的是，当处于图示静止状态时，邻域中每个像素的光流约为零，看上去像一个点。As shown in Figure 5, the standard pixel coordinate system is used in the figure, the origin is in the upper left corner of the image, the X axis faces right, and the Y axis faces down; Indicates the x-coordinate of the end of the arm to be trained in the pixel coordinate system of the ROI image, /> Indicates the y coordinate of the end of the arm to be trained in the pixel coordinate system of the ROI image; /> Indicates the neighborhood width, /> Indicates the neighborhood height, /> and /> It can be determined according to the shape of the arm, based on the current position of the end of the arm to be trained in the ROI image /> The process of determining the neighborhood is as follows: Determine the width of the neighborhood according to the shape of the arm to be trained /> and neighborhood height /> at current location /> As the center, the establishment width is the neighborhood width /> Height is neighborhood height /> The rectangular area of is used as the neighborhood of the end of the arm to be trained in the ROI image, that is, the rectangular area in Figure 5. It should be noted that, when the image is in a static state, the optical flow of each pixel in the neighborhood is about zero, which looks like a point.
步骤S208，根据邻域内的光流确定待训练手臂末端的实际速度向量；Step S208, determine the actual velocity vector of the end of the arm to be trained according to the optical flow in the neighborhood;
对于待训练手臂末端运动状态下的光流，如图6所示，[Ve,x,Ve,y]表示待训练手臂末端的实际速度向量，表示上肢末端目标位置，[Vd,x,Vd,y]表示待训练手臂末端的目标运动速度即目标速度向量。当待训练手臂产生了向左的微小运动时，邻域中部分像素点产生了X正方向的分量，看上去像一个向右的箭头(由于镜像原因，方向与实际运动相反)。由于光流包括X方向的第一光流即flowx和Y方向的第二光流即flowy；因此，可以通过平均待训练手臂末端邻域内的光流获得待训练手臂末端的实际速度向量[Ve,x,Ve,y]。For the optical flow of the end of the arm to be trained in motion, as shown in Figure 6, [Ve,x ,Ve,y ] represents the actual velocity vector of the end of the arm to be trained, Indicates the target position of the upper limb end, [Vd,x ,Vd,y ] indicates the target movement velocity of the arm end to be trained, that is, the target velocity vector. When the arm to be trained produces a small movement to the left, some pixels in the neighborhood produce a component in the positive direction of X, which looks like a rightward arrow (due to the mirror image, the direction is opposite to the actual movement). Since the optical flow includes the first optical flow in the X direction, i.e. flowx, and the second optical flow in the Y direction, i.e. flowy; therefore, the actual velocity vector [Ve ,x ,Ve,y ].
具体地，根据邻域内的光流确定待训练手臂末端的实际速度向量的过程如下：对邻域内的全部第一光流进行平均计算，得到X方向的第一速度即Ve,x；以及，对邻域内的全部第二光流进行平均计算，得到Y方向的第二速度即Ve,y；根据第一速度和第二速度得到实际速度向量。Specifically, the process of determining the actual velocity vector of the end of the arm to be trained according to the optical flow in the neighborhood is as follows: average all the first optical flows in the neighborhood to obtain the first velocity in the X direction, namely Ve,x ; and, The average calculation is performed on all the second optical flows in the neighborhood to obtain the second velocity in the Y direction, that is, Ve,y ; and the actual velocity vector is obtained according to the first velocity and the second velocity.
其中，平均计算公式如下：Among them, the average calculation formula is as follows:
其中，[i,j]∈S表示领域S中的所有像素坐标，flowx[i,j]表示矩阵flowx中第i行第j列的元素，flowy[i,j]表示矩阵flowy中第i行第j列的元素，表示邻域宽度、/>表示邻域高度。Among them, [i,j]∈S represents all pixel coordinates in the field S, flowx[i,j] represents the element in row i and column j in matrix flowx, and flowy[i,j] represents row i in matrix flowy the element of column j, Indicates the neighborhood width, /> Indicates the neighborhood height.
上述通过平均待训练手臂末端邻域内的光流，而不是待训练手臂末端单个像素的光流，以获得待训练手臂末端的实际速度向量，避免了由于单个像素的光流受到噪声影响，导致得到的速度向量无法反映真实的运动意图。因此，通过平均待训练手臂末端邻域内的光流可以减小噪声的影响，提高了实际速度向量的计算精确度。The above averages the optical flow in the neighborhood of the end of the arm to be trained instead of the optical flow of a single pixel at the end of the arm to be trained to obtain the actual velocity vector of the end of the arm to be trained, avoiding the fact that the optical flow of a single pixel is affected by noise, resulting in The velocity vector of can not reflect the real motion intention. Therefore, the influence of noise can be reduced by averaging the optical flow in the neighborhood of the end of the arm to be trained, and the calculation accuracy of the actual velocity vector is improved.
步骤S210，获取待训练手臂末端在ROI图像中的目标位置，并根据目标位置和当前位置确定待训练手臂末端的目标速度向量；Step S210, obtaining the target position of the end of the arm to be trained in the ROI image, and determining the target velocity vector of the end of the arm to be trained according to the target position and the current position;
具体地，按照如下公式计算目标速度向量[Vd,x,Vd,y]：Specifically, the target velocity vector [Vd,x ,Vd,y ] is calculated according to the following formula:
其中，表示待训练手臂末端在ROI图像中的目标位置即上肢末端目标位置，如图5或6所示，/>表示当前位置，T表示预先设定的运动持续时间。需要说明的是，目标位置/>为上肢康复机器人提供的目标导向康复训练中当前训练任务对应的位置，当当前训练任务完成后，控制主机生成下一个训练任务对应的目标位置，直至全部任务完成或训练时间结束。in, Indicates the target position of the end of the arm to be trained in the ROI image, that is, the target position of the upper limb end, as shown in Figure 5 or 6, /> Indicates the current position, and T indicates the preset exercise duration. It should be noted that the target location /> The position corresponding to the current training task in the goal-oriented rehabilitation training provided for the upper limb rehabilitation robot. After the current training task is completed, the control host generates the target position corresponding to the next training task until all tasks are completed or the training time is over.
步骤S212，根据实际速度向量和目标速度向量确定待训练手臂的运动意图，并根据运动意图辅助待训练手臂进行康复训练，直至运动至目标位置。Step S212: Determine the movement intention of the arm to be trained according to the actual velocity vector and the target velocity vector, and assist the arm to be trained to carry out rehabilitation training according to the movement intention until it moves to the target position.
具体地，根据实际速度向量和目标速度向量确定待训练手臂的运动意图的过程包括：计算实际速度向量在目标速度向量上的投影；判断投影是否满足预设运动条件；如果是，确定待训练手臂产生正确的运动意图，并辅助待训练手臂进行康复训练，直至运动至目标位置。Specifically, the process of determining the movement intention of the arm to be trained according to the actual velocity vector and the target velocity vector includes: calculating the projection of the actual velocity vector on the target velocity vector; judging whether the projection meets the preset movement conditions; Generate the correct movement intention, and assist the arm to be trained to perform rehabilitation training until it moves to the target position.
其中，按照如下公式计算实际速度向量在目标速度向量上的投影V：Among them, the projection V of the actual velocity vector on the target velocity vector is calculated according to the following formula:
其中，[Vd,x,Vd,y]表示目标速度向量，[Ve,x,Ve,y]表示实际速度向量。Wherein, [Vd,x ,Vd,y ] represents a target velocity vector, and [Ve,x ,Ve,y ] represents an actual velocity vector.
对于上述投影V，判断投影是否满足预设运动条件的过程包括：判断投影的方向是否为正，且，投影的幅值是否达到预设阈值；如果均是，判定投影满足预设运动条件。即当投影的方向为正且幅值达到预设阈值时，判定待训练手臂产生了与目标方向一致的运动意图，即正确的运动意图，此时，控制主机控制外骨骼提供辅助力/力矩使待训练手臂按目标速度向量[Vd,x,Vd,y]运动到目标位置当到达目标位置时，则视为完成当前任务，控制主机生成下一个目标位置，直至全部任务完成或训练时间结束。For the above projection V, the process of judging whether the projection satisfies the preset motion condition includes: judging whether the direction of the projection is positive, and whether the amplitude of the projection reaches a preset threshold; if both are, judging that the projection satisfies the preset motion condition. That is, when the projected direction is positive and the amplitude reaches the preset threshold, it is determined that the arm to be trained has produced a movement intention consistent with the target direction, that is, the correct movement intention. At this time, the control host controls the exoskeleton to provide auxiliary force/torque to make The arm to be trained moves to the target position according to the target velocity vector [Vd,x ,Vd,y ] When the target position is reached, the current task is considered to be completed, and the control host generates the next target position until all tasks are completed or the training time is over.
进一步地，该方法还包括：如果投影不满足预设运动条件，则确定待训练手臂产生错误的运动意图，此时，控制主机控制外骨骼保持静止，即停止辅助待训练手臂进行康复训练，直至待训练手臂产生正确的运动意图。Further, the method also includes: if the projection does not meet the preset motion conditions, then determine that the arm to be trained produces a wrong motion intention, at this time, the control host controls the exoskeleton to keep still, that is, stops assisting the arm to be trained to carry out rehabilitation training, until The arm to be trained produces the correct movement intention.
本发明实施例提供的上肢康复机器人主动训练方法，在患者发起主动运动后肢体会产生微小位移，通过RGB图像捕获位移的大小和方向，以识别出患者的运动意图实现主动训练，从而降低了上肢康复机器人主动训练成本，提高了主动训练模式的实用性，便于在实际应用中推广实施。In the active training method of the upper limb rehabilitation robot provided by the embodiment of the present invention, after the patient initiates an active movement, the limb will produce a small displacement, and the magnitude and direction of the displacement are captured through the RGB image, so as to realize the active training by identifying the patient's movement intention, thereby reducing the burden on the upper limb. The active training cost of the rehabilitation robot improves the practicability of the active training mode and facilitates the promotion and implementation in practical applications.
综上，上述上肢康复机器人主动训练方法，具有以下优点：①基于普通的RGB相机即可识别患者上肢的主动运动意图，与使用脑电或肌电信号的方法相比，能够大幅降低所需传感器的价格，有助于降低上肢康复机器人主动训练成本，提高主动训练模式的实用性；②通过对视觉传感器输出的RGB图像进行直通滤波，并基于正运动学模型从视觉传感器输出的RGB图像中裁剪出感兴趣区域(ROI)，后续算法均在ROI图像上进行，能够显著减小计算量，提高光流法的实用性；③通过平均待训练手臂末端邻域内的光流可以减小噪声的影响，提高了上肢主动运动意图识别的准确性，便于在实际应用中推广实施，In summary, the above-mentioned active training method for upper limb rehabilitation robots has the following advantages: ①The active movement intention of the patient's upper limbs can be identified based on a common RGB camera, and compared with the method using EEG or EMG signals, it can greatly reduce the number of sensors required. The price helps reduce the active training cost of the upper limb rehabilitation robot and improves the practicability of the active training mode; ②Through through-filtering the RGB image output by the visual sensor, and cropping from the RGB image output by the visual sensor based on the positive kinematics model Out of the region of interest (ROI), the subsequent algorithms are all performed on the ROI image, which can significantly reduce the amount of calculation and improve the practicability of the optical flow method; ③The influence of noise can be reduced by averaging the optical flow in the neighborhood of the end of the arm to be trained , which improves the accuracy of upper limb active movement intention recognition and facilitates the promotion and implementation in practical applications.
对应于上述方法实施例，本发明实施例还提供了一种上肢康复机器人主动训练装置，如图7所示，该装置包括：图像获取模块71、光流计算模块72、邻域确定模块73、实际速度确定模块74、目标速度确定模块75和运动意图确定模块76；其中，各个模块的功能如下：Corresponding to the above method embodiment, the embodiment of the present invention also provides an active training device for an upper limb rehabilitation robot, as shown in FIG. 7 , the device includes: an image acquisition module 71, an optical flow calculation module 72, a neighborhood determination module 73, Actual speed determination module 74, target speed determination module 75 and motion intention determination module 76; Wherein, the function of each module is as follows:
图像获取模块71，用于获取包含待训练手臂的RGB图像，并基于正运动学模型从RGB图像中提取感兴趣区域ROI，得到ROI图像；其中，ROI图像覆盖待训练手臂的所有运动区域；Image acquisition module 71, is used for obtaining the RGB image that comprises arm to be trained, and extracts region of interest ROI from RGB image based on positive kinematics model, obtains ROI image; Wherein, ROI image covers all motion areas of arm to be trained;
光流计算模块72，用于基于Farneback光流算法计算ROI图像的稠密光流；The optical flow calculation module 72 is used to calculate the dense optical flow of the ROI image based on the Farneback optical flow algorithm;
邻域确定模块73，用于获取待训练手臂末端在ROI图像中的当前位置，并根据当前位置确定待训练手臂末端在ROI图像中的邻域；The neighborhood determination module 73 is used to obtain the current position of the arm end to be trained in the ROI image, and determine the neighborhood of the arm end to be trained in the ROI image according to the current position;
实际速度确定模块74，用于根据邻域内的光流确定待训练手臂末端的实际速度向量；Actual velocity determination module 74, is used for determining the actual velocity vector of the end of the arm to be trained according to the optical flow in the neighborhood;
目标速度确定模块75，用于获取待训练手臂末端在ROI图像中的目标位置，并根据目标位置和当前位置确定待训练手臂末端的目标速度向量；The target speed determination module 75 is used to obtain the target position of the arm end to be trained in the ROI image, and determine the target speed vector of the arm end to be trained according to the target position and the current position;
运动意图确定模块76，用于根据实际速度向量和目标速度向量确定待训练手臂的运动意图，并根据运动意图辅助待训练手臂进行康复训练，直至运动至目标位置。The movement intention determination module 76 is used to determine the movement intention of the arm to be trained according to the actual velocity vector and the target velocity vector, and assist the arm to be trained to carry out rehabilitation training according to the movement intention until it moves to the target position.
本发明实施例提供的上肢康复机器人主动训练装置，在患者发起主动运动后肢体会产生微小位移，通过RGB图像捕获位移的大小和方向，以识别出患者的运动意图实现主动训练，从而降低了上肢康复机器人主动训练成本，提高了主动训练模式的实用性，便于在实际应用中推广实施。The upper limb rehabilitation robot active training device provided by the embodiment of the present invention, after the patient initiates an active movement, the limb will produce a small displacement, and the size and direction of the displacement are captured through the RGB image, so as to realize the active training by identifying the patient's movement intention, thereby reducing the burden on the upper limb. The active training cost of the rehabilitation robot improves the practicability of the active training mode and facilitates the promotion and implementation in practical applications.
优选地，上述光流计算模块72还用于：对ROI图像进行灰度转换处理，得到第一灰度图像；对第一灰度图像进行中值滤波处理，得到滤波后的第二灰度图像；基于Farneback光流算法，根据当前帧的第二灰度图像和上一帧的第二灰度图像，计算得到当前帧的第二灰度图像的光流。Preferably, the above-mentioned optical flow calculation module 72 is also used for: performing grayscale conversion processing on the ROI image to obtain a first grayscale image; performing median filtering processing on the first grayscale image to obtain a filtered second grayscale image ; Based on the Farneback optical flow algorithm, calculate the optical flow of the second grayscale image of the current frame according to the second grayscale image of the current frame and the second grayscale image of the previous frame.
优选地，上述邻域确定模块73还用于：根据待训练手臂形状确定邻域宽度和邻域高度；以当前位置为中心，建立宽度为邻域宽度、高度为邻域高度的矩形区域，作为待训练手臂末端在ROI图像中的邻域。Preferably, the above-mentioned neighborhood determination module 73 is also used to: determine the neighborhood width and neighborhood height according to the shape of the arm to be trained; take the current position as the center, establish a rectangular area whose width is the neighborhood width and height is the neighborhood height, as The neighborhood of the end of the arm to be trained in the ROI image.
优选地，光流包括X方向的第一光流和Y方向的第二光流；上述实际速度确定模块74还用于：对邻域内的全部第一光流进行平均计算，得到X方向的第一速度；以及，对邻域内的全部第二光流进行平均计算，得到Y方向的第二速度；根据第一速度和第二速度得到实际速度向量。Preferably, the optical flow includes the first optical flow in the X direction and the second optical flow in the Y direction; the above-mentioned actual speed determination module 74 is also used for: performing an average calculation on all the first optical flows in the neighborhood to obtain the second optical flow in the X direction. A speed; and, performing an average calculation on all second optical flows in the neighborhood to obtain a second speed in the Y direction; and obtaining an actual speed vector according to the first speed and the second speed.
优选地，上述运动意图确定模块76还用于：计算实际速度向量在目标速度向量上的投影；判断投影是否满足预设运动条件；如果是，确定待训练手臂产生正确的运动意图，并辅助待训练手臂进行康复训练，直至运动至目标位置。Preferably, the above-mentioned movement intention determining module 76 is also used for: calculating the projection of the actual velocity vector on the target velocity vector; judging whether the projection meets the preset movement condition; if so, determining that the arm to be trained produces a correct movement intention, and assisting Train the arm for rehab until it moves to the target position.
优选地，判断投影是否满足预设运动条件包括：判断投影的方向是否为正，且，投影的幅值是否达到预设阈值；如果均是，判定投影满足预设运动条件。Preferably, judging whether the projection satisfies the preset motion condition includes: judging whether the direction of the projection is positive, and whether the magnitude of the projection reaches a preset threshold; if both are, judging that the projection satisfies the preset motion condition.
优选地，该装置还包括：如果投影不满足预设运动条件，确定待训练手臂产生错误的运动意图，并停止辅助待训练手臂进行康复训练。Preferably, the device further includes: if the projection does not satisfy the preset motion condition, determining that the arm to be trained produces a wrong motion intention, and stopping assisting the arm to be trained for rehabilitation training.
本发明实施例提供的上肢康复机器人主动训练装置，与上述实施例提供的上肢康复机器人主动训练方法具有相同的技术特征，所以也能解决相同的技术问题，达到相同的技术效果。The active training device for the upper limb rehabilitation robot provided by the embodiment of the present invention has the same technical features as the active training method for the upper limb rehabilitation robot provided by the above embodiment, so it can also solve the same technical problems and achieve the same technical effect.
本发明实施例还提供一种上肢康复机器人，包括处理器和存储器，存储器存储有能够被处理器执行的机器可执行指令，处理器执行机器可执行指令以实现上述上肢康复机器人主动训练方法。An embodiment of the present invention also provides an upper limb rehabilitation robot, including a processor and a memory, the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the above active training method for the upper limb rehabilitation robot.
参见图8所示，该上肢康复机器人包括处理器100和存储器101，该存储器101存储有能够被处理器100执行的机器可执行指令，该处理器100执行机器可执行指令以实现上述上肢康复机器人主动训练方法。8, the upper limb rehabilitation robot includes a processor 100 and a memory 101, the memory 101 stores machine-executable instructions that can be executed by the processor 100, and the processor 100 executes the machine-executable instructions to realize the above-mentioned upper limb rehabilitation robot Active training methods.
进一步地，图8所示的上肢康复机器人还包括总线102和通信接口103，处理器100、通信接口103和存储器101通过总线102连接。Further, the upper limb rehabilitation robot shown in FIG. 8 also includes a bus 102 and a communication interface 103 , and the processor 100 , the communication interface 103 and the memory 101 are connected through the bus 102 .
其中，存储器101可能包含高速随机存取存储器(RAM，Random Access Memory)，也可能还包括非不稳定的存储器(non-volatile memory)，例如至少一个磁盘存储器。通过至少一个通信接口103(可以是有线或者无线)实现该系统网元与至少一个其他网元之间的通信连接，可以使用互联网，广域网，本地网，城域网等。总线102可以是ISA(IndustrialStandard Architecture，工业标准结构总线)总线、PCI(Peripheral ComponentInterconnect，外设部件互连标准)总线或EISA(Enhanced Industry StandardArchitecture，扩展工业标准结构)总线等。上述总线可以分为地址总线、数据总线、控制总线等。为便于表示，图8中仅用一个双向箭头表示，但并不表示仅有一根总线或一种类型的总线。Wherein, the memory 101 may include a high-speed random access memory (RAM, Random Access Memory), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local network, metropolitan area network, etc. can be used. The bus 102 may be an ISA (Industrial Standard Architecture, industry standard architecture bus) bus, a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus, or an EISA (Enhanced Industry Standard Architecture, extended industry standard architecture) bus, etc. The above bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one double-headed arrow is used in FIG. 8 , but it does not mean that there is only one bus or one type of bus.
处理器100可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过处理器100中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器100可以是通用处理器，包括中央处理器(Central Processing Unit，简称CPU)、网络处理器(Network Processor，简称NP)等；还可以是数字信号处理器(DigitalSignal Processor，简称DSP)、专用集成电路(Application Specific IntegratedCircuit，简称ASIC)、现场可编程门阵列(Field-Programmable Gate Array，简称FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本发明实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本发明实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器101，处理器100读取存储器101中的信息，结合其硬件完成前述实施例的方法的步骤。The processor 100 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method may be implemented by an integrated logic circuit of hardware in the processor 100 or instructions in the form of software. The above-mentioned processor 100 can be a general-purpose processor, including a central processing unit (Central Processing Unit, referred to as CPU), a network processor (Network Processor, referred to as NP), etc.; it can also be a digital signal processor (Digital Signal Processor, referred to as DSP) , Application Specific Integrated Circuit (ASIC for short), Field Programmable Gate Array (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Various methods, steps and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like. The steps of the methods disclosed in the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register. The storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101, and completes the steps of the method in the foregoing embodiments in combination with its hardware.
本实施例还提供一种机器可读存储介质，机器可读存储介质存储有机器可执行指令，机器可执行指令在被处理器调用和执行时，机器可执行指令促使处理器实现上述上肢康复机器人主动训练方法。This embodiment also provides a machine-readable storage medium. The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by the processor, the machine-executable instructions prompt the processor to realize the above-mentioned upper limb rehabilitation robot. Active training methods.
本发明实施例所提供的上肢康复机器人主动训练方法、装置和上肢康复机器人的计算机程序产品，包括存储了程序代码的计算机可读存储介质，所述程序代码包括的指令可用于执行前面方法实施例中所述的方法，具体实现可参见方法实施例，在此不再赘述。The active training method and device for the upper limb rehabilitation robot and the computer program product of the upper limb rehabilitation robot provided by the embodiments of the present invention include a computer-readable storage medium storing program codes, and the instructions included in the program codes can be used to execute the foregoing method embodiments The specific implementation of the method described in may refer to the method embodiments, and details are not repeated here.
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统和装置的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the above-described system and device can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.
另外，在本发明实施例的描述中，除非另有明确的规定和限定，术语“安装”、“相连”、“连接”应做广义理解，例如，可以是固定连接，也可以是可拆卸连接，或一体地连接；可以是机械连接，也可以是电连接；可以是直接相连，也可以通过中间媒介间接相连，可以是两个元件内部的连通。对于本领域的普通技术人员而言，可以具体情况理解上述术语在本发明中的具体含义。In addition, in the description of the embodiments of the present invention, unless otherwise specified and limited, the terms "installation", "connection" and "connection" should be interpreted in a broad sense, for example, it can be a fixed connection or a detachable connection , or integrally connected; it may be mechanically connected or electrically connected; it may be directly connected or indirectly connected through an intermediary, and it may be the internal communication of two components. Those of ordinary skill in the art can understand the specific meanings of the above terms in the present invention in specific situations.
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个处理器可执行的非易失的计算机可读取存储介质中。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本发明各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes. .
在本发明的描述中，需要说明的是，术语“中心”、“上”、“下”、“左”、“右”、“竖直”、“水平”、“内”、“外”等指示的方位或位置关系为基于附图所示的方位或位置关系，仅是为了便于描述本发明和简化描述，而不是指示或暗示所指的装置或元件必须具有特定的方位、以特定的方位构造和操作，因此不能理解为对本发明的限制。此外，术语“第一”、“第二”、“第三”仅用于描述目的，而不能理解为指示或暗示相对重要性。In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" etc. The indicated orientation or positional relationship is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred device or element must have a specific orientation, or in a specific orientation. construction and operation, therefore, should not be construed as limiting the invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and should not be construed as indicating or implying relative importance.
最后应说明的是：以上所述实施例，仅为本发明的具体实施方式，用以说明本发明的技术方案，而非对其限制，本发明的保护范围并不局限于此，尽管参照前述实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：任何熟悉本技术领域的技术人员在本发明揭露的技术范围内，其依然可以对前述实施例所记载的技术方案进行修改或可轻易想到变化，或者对其中部分技术特征进行等同替换；而这些修改、变化或者替换，并不使相应技术方案的本质脱离本发明实施例技术方案的精神和范围，都应涵盖在本发明的保护范围之内。因此，本发明的保护范围应所述以权利要求的保护范围为准。Finally, it should be noted that: the above-described embodiments are only specific implementations of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting them, and the scope of protection of the present invention is not limited thereto, although referring to the foregoing The embodiment has described the present invention in detail, and those of ordinary skill in the art should understand that any person familiar with the technical field can still modify the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention Changes can be easily thought of, or equivalent replacements are made to some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of the present invention within the scope of protection. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310420013.3ACN116492644B (en) | 2023-04-14 | 2023-04-14 | Active training method and device for upper limb rehabilitation robot and upper limb rehabilitation robot |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310420013.3ACN116492644B (en) | 2023-04-14 | 2023-04-14 | Active training method and device for upper limb rehabilitation robot and upper limb rehabilitation robot |
| Publication Number | Publication Date |
|---|---|
| CN116492644Atrue CN116492644A (en) | 2023-07-28 |
| CN116492644B CN116492644B (en) | 2025-05-06 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310420013.3AActiveCN116492644B (en) | 2023-04-14 | 2023-04-14 | Active training method and device for upper limb rehabilitation robot and upper limb rehabilitation robot |
| Country | Link |
|---|---|
| CN (1) | CN116492644B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130335543A1 (en)* | 2012-06-13 | 2013-12-19 | Esight Corp. | Apparatus and Method for Enhancing Human Visual Performance in a Head Worn Video System |
| CN105536205A (en)* | 2015-12-08 | 2016-05-04 | 天津大学 | Upper limb training system based on monocular video human body action sensing |
| US20160128642A1 (en)* | 2014-11-06 | 2016-05-12 | Fundacion Tecnalia Research & Innovation | Method and System for Functional Balance Assessment |
| CN111631726A (en)* | 2020-06-01 | 2020-09-08 | 深圳华鹊景医疗科技有限公司 | Upper limb function evaluation device and method and upper limb rehabilitation training system and method |
| CN112184767A (en)* | 2020-09-22 | 2021-01-05 | 深研人工智能技术(深圳)有限公司 | Method, device, equipment and storage medium for tracking moving object track |
| CN112891137A (en)* | 2021-01-21 | 2021-06-04 | 深圳华鹊景医疗科技有限公司 | Upper limb rehabilitation robot system, robot control method and device |
| CN114005073A (en)* | 2021-12-24 | 2022-02-01 | 东莞理工学院 | Upper limb mirror rehabilitation training, identification method and device |
| CN114028156A (en)* | 2021-10-28 | 2022-02-11 | 深圳华鹊景医疗科技有限公司 | Rehabilitation training method and device and rehabilitation robot |
| CN114964233A (en)* | 2022-05-16 | 2022-08-30 | 中国科学技术大学 | Meta-universe system and method for underwater augmented reality wearable robot |
| CN115937895A (en)* | 2022-11-11 | 2023-04-07 | 南通大学 | A Velocity and Force Feedback System Based on Depth Camera |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130335543A1 (en)* | 2012-06-13 | 2013-12-19 | Esight Corp. | Apparatus and Method for Enhancing Human Visual Performance in a Head Worn Video System |
| US20160128642A1 (en)* | 2014-11-06 | 2016-05-12 | Fundacion Tecnalia Research & Innovation | Method and System for Functional Balance Assessment |
| CN105536205A (en)* | 2015-12-08 | 2016-05-04 | 天津大学 | Upper limb training system based on monocular video human body action sensing |
| CN111631726A (en)* | 2020-06-01 | 2020-09-08 | 深圳华鹊景医疗科技有限公司 | Upper limb function evaluation device and method and upper limb rehabilitation training system and method |
| CN112184767A (en)* | 2020-09-22 | 2021-01-05 | 深研人工智能技术(深圳)有限公司 | Method, device, equipment and storage medium for tracking moving object track |
| CN112891137A (en)* | 2021-01-21 | 2021-06-04 | 深圳华鹊景医疗科技有限公司 | Upper limb rehabilitation robot system, robot control method and device |
| CN114028156A (en)* | 2021-10-28 | 2022-02-11 | 深圳华鹊景医疗科技有限公司 | Rehabilitation training method and device and rehabilitation robot |
| CN114005073A (en)* | 2021-12-24 | 2022-02-01 | 东莞理工学院 | Upper limb mirror rehabilitation training, identification method and device |
| CN114964233A (en)* | 2022-05-16 | 2022-08-30 | 中国科学技术大学 | Meta-universe system and method for underwater augmented reality wearable robot |
| CN115937895A (en)* | 2022-11-11 | 2023-04-07 | 南通大学 | A Velocity and Force Feedback System Based on Depth Camera |
| Title |
|---|
| 郭烈;赵宗艳;聂倩;张广西;: "利用腿部区域HOG特征的行人检测方法", 计算机工程与应用, no. 01, 24 October 2011 (2011-10-24)* |
| Publication number | Publication date |
|---|---|
| CN116492644B (en) | 2025-05-06 |
| Publication | Publication Date | Title |
|---|---|---|
| CN110480634B (en) | An arm-guided motion control method for robotic arm motion control | |
| CN110728241A (en) | A driver fatigue detection method based on deep learning multi-feature fusion | |
| WO2019033569A8 (en) | Eyeball movement analysis method, device and storage medium | |
| JP2021064367A (en) | Motion recognition method, motion recognition device, and electronic device | |
| CN105117681B (en) | Multiple features fatigue real-time detection method based on Android | |
| CN106066696A (en) | The sight tracing compensated based on projection mapping correction and point of fixation under natural light | |
| CN108803874A (en) | A kind of human-computer behavior exchange method based on machine vision | |
| CN108268858B (en) | High-robustness real-time sight line detection method | |
| CN109711239B (en) | Visual attention detection method based on improved mixed increment dynamic Bayesian network | |
| CN115147462A (en) | A gaze feature tracking method based on 3D eyeball model and Kalman filter | |
| CN105912126A (en) | Method for adaptively adjusting gain, mapped to interface, of gesture movement | |
| CN111144174A (en) | A system for identifying the fall behavior of the elderly in videos using neural networks and traditional algorithms | |
| CN115240224A (en) | Gesture feature extraction method based on three-dimensional hand key point and image feature fusion | |
| CN116492644A (en) | Active training method and device for upper limb rehabilitation robot and upper limb rehabilitation robot | |
| CN115089150B (en) | A pulse wave detection method, device, electronic device and storage medium based on drone | |
| CN113117295B (en) | Brain cognitive rehabilitation robot system based on hand recognition and training method | |
| Ganokratanaa et al. | Hand Gesture Recognition Algorithm for Smart Cities based on Wireless Sensor. | |
| CN110287823A (en) | Face Recognition Method Based on Improved LBP Operator and Support Vector Machine Classification | |
| KR100515798B1 (en) | Robot driving method using facial gestures | |
| Horng et al. | Improvements of driver fatigue detection system based on eye tracking and dynamic template matching | |
| CN117612251A (en) | Human body posture recognition method and system | |
| CN113095297B (en) | Fatigue detection method based on one-dimensional projection tracking eye movement rate | |
| CN112454358B (en) | A robotic arm motion planning method and system combining psychological safety and motion prediction | |
| CN107729823B (en) | Online human-computer interaction method based on E-SOINN network | |
| CN115841864A (en) | Rehabilitation exercise quality assessment method and system |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |