技术领域technical field
本发明涉及教学辅助领域，尤其涉及一种教学辅助方法及采用该方法的教学辅助系统。The invention relates to the field of teaching assistance, in particular to a teaching assistance method and a teaching assistance system adopting the method.
背景技术Background technique
在一般的教学活动中，由于上课的学生和授课老师的数量比例悬殊，授课老师在授课时没有太多的时间和精力通过观察每个学生的上课行为和表情来判断学生的学习状态。这就使得授课老师无法精确的了解每个学生的上课状态和对本次教授内容的被接受程度。很容易导致课堂上老师讲老师的，学生聊学生的，进而让整个教学活动被撕裂开来，也使得授课老师无法有的放矢的进行教学，严重的影响了教学质量和效率。所以，能够在学生上课时使用的教学辅助系统历来是教育界所关注的重点问题。设计教学辅助系统辅助授课教师顺利开展教学活动。目前的教学辅助系统研究强调功能性，主要从为学生提供自主的学习环境，为学生提供充分的学习资源，减轻教师的工作量几个方面展开的。运用不同技术手段，设计智能化辅助系统来提高教师的授课效果和学生的学习效率。In general teaching activities, due to the disparity in the number of students and teachers in the class, the teacher does not have much time and energy to judge the learning status of the students by observing the behavior and expression of each student in class. This makes it impossible for the teacher to accurately understand the class status of each student and the degree of acceptance of the teaching content. It is easy to cause the teacher to talk about the teacher and the students to talk about the students in the classroom, and then the whole teaching activity will be torn apart, and it will also make it impossible for the teacher to teach in a targeted manner, seriously affecting the quality and efficiency of teaching. Therefore, the teaching assistance system that can be used by students in class has always been the focus of attention in the education field. The teaching assistant system is designed to assist teachers to carry out teaching activities smoothly. The current research on teaching assistant system emphasizes functionality, mainly from the aspects of providing students with an independent learning environment, providing students with sufficient learning resources, and reducing the workload of teachers. Using different technical means, design an intelligent auxiliary system to improve the teaching effect of teachers and the learning efficiency of students.
在现有技术中，公开号为CN106097790A的中国发明专利公开了一种教学辅助装置，通过图像识别技术识别教学活动中的图像，进而来判断学生上课是否做与上课无关的事情，并根据识别结果通知老师做相应处理。In the prior art, the Chinese invention patent with the publication number CN106097790A discloses a teaching aid device, which uses image recognition technology to identify images in teaching activities, and then judges whether students are doing things that have nothing to do with class, and according to the recognition results Notify the teacher to deal with it accordingly.
由于该现有技术并未公开其图像识别模块识别图像的方法和过程，也没有公开其图像识别模块如何实现将现有图像与预存图像进行比对，并判断比对结果。技术人员根据该现有技术方案无法具体实现为教学过程进行辅助的技术效果。因此，现有的教学辅助方法存在不足。Because this prior art does not disclose the method and process of its image recognition module for recognizing images, nor does it disclose how its image recognition module compares existing images with pre-stored images and judges the comparison result. According to this prior art solution, technical personnel cannot specifically realize the technical effect of assisting the teaching process. Therefore, there are deficiencies in the existing teaching aid methods.
发明内容Contents of the invention
为了解决现有技术中存在的上述技术问题，本发明的目的在于提供一种具有较高的图像识别精度的教学辅助方法和采用该方法的教学辅助系统。In order to solve the above-mentioned technical problems in the prior art, the object of the present invention is to provide a teaching assistance method with high image recognition accuracy and a teaching assistance system using the method.
为了解决上述技术问题，本发明所采用的技术方案为：一种教学辅助方法，包括以下顺序步骤：In order to solve the above-mentioned technical problems, the technical solution adopted in the present invention is: a teaching assistance method, comprising the following sequential steps:
s1.采集模块实时采集现场的课堂图像，并传输给识别模块；s1. The acquisition module collects the on-site classroom images in real time and transmits them to the recognition module;
s2.所述识别模块对所述课堂图像进行分析，并判断所述课堂图像中行为异常的学生；s2. The recognition module analyzes the classroom image, and judges students with abnormal behavior in the classroom image;
s3.提示模块将所述识别模块的识别结果通知授课教师；s3. The prompt module notifies the teacher of the recognition result of the recognition module;
所述步骤s2中包括以下步骤：The step s2 includes the following steps:
s21.所述识别模块使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行行为检测。s21. The recognition module uses the trained deep tensor network model to detect the behavior of the students in the classroom image.
优选的，所述步骤s2还包括以下步骤：Preferably, said step s2 also includes the following steps:
s22.所述识别模块使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行表情识别。s22. The recognition module uses the trained deep tensor network model to recognize the expressions of the students in the classroom image.
优选的，所述步骤s22具体包括以下步骤：Preferably, said step s22 specifically includes the following steps:
s221.通过人脸检测子单元从所述采集模块采集到的所述课堂图像中识别出各学生的人脸区域；s221. Recognize the face area of each student from the classroom image collected by the acquisition module through the face detection subunit;
s222.通过卷积神经网络分类器对检测到的所述人脸区域做表情识别。s222. Using a convolutional neural network classifier to perform expression recognition on the detected face area.
优选的，步骤s1中包括以下步骤：Preferably, step s1 includes the following steps:
s11.所述采集模块在教室前方的左、中、右区域分别安装图像采集装置；s11. The acquisition module installs image acquisition devices in the left, middle and right areas in front of the classroom respectively;
s12.所述图像采集模块以班级中所有学生上半身图像为采集目标。s12. The image acquisition module takes the upper body images of all students in the class as the acquisition target.
优选的，还包括以下步骤：s4.存储模块同步存档所述识别结果。Preferably, the following step is also included: s4. The storage module archives the identification results synchronously.
优选的，所述步骤s4中包括以下步骤：Preferably, the step s4 includes the following steps:
s41.将每个学生对应的所述识别结果按班级制定成学生电子档案；s41. Make the identification result corresponding to each student into a student electronic file according to the class;
s42.根据所述学生电子档案绘出学生上课状态曲线，用以便于授课教师结合当时教授的内容以及考试成绩对学生进行有针对性的辅导。s42. Draw the class status curve of the students according to the electronic files of the students, so that the teachers can provide targeted guidance to the students in combination with the content taught at that time and the test results.
优选的，步骤s1之前还包括以下步骤：Preferably, the following steps are also included before step s1:
q1.构建数据集；q1. Build a data set;
q2.训练所述深度张量列网络模型。q2. Train the deep tensor column network model.
优选的，所述步骤q1包括以下步骤：Preferably, said step q1 includes the following steps:
q11.所述采集模块在教室长时间拍摄所述课堂图像并存储；q11. The acquisition module takes and stores the classroom image for a long time in the classroom;
q12.选取存在异常的学生图片进行标注。q12. Select the pictures of students with abnormalities for labeling.
优选的，所述步骤q2包括以下步骤：Preferably, said step q2 includes the following steps:
q21.通过神经网络模型的多层卷积层提取已标注的所述学生图片中的异常特征，所述异常特征与分解后的全连接层权重矩阵运算得到输出预测值；q21. Extracting the abnormal features in the marked student picture through the multi-layer convolutional layer of the neural network model, the abnormal features and the decomposed fully connected layer weight matrix operation to obtain the output prediction value;
q22.所述输出预测值与所述学生图片中的异常行为学生真实标注值的误差构成的损失函数；q22. The loss function formed by the error between the output predicted value and the real labeled value of the abnormal behavior student in the student picture;
q23.根据所述损失函数调整网络参数，得到训练好的深度张量列网络模型。q23. Adjust the network parameters according to the loss function to obtain a trained deep tensor sequence network model.
为了解决上述技术问题，本发明还提供一种教学辅助系统，设置有：采集模块、与所述采集模块连接的识别模块、与所述识别模块连接的提示模块；In order to solve the above technical problems, the present invention also provides a teaching assistance system, which is provided with: an acquisition module, an identification module connected to the acquisition module, and a prompt module connected to the identification module;
所述采集模块，用于实时采集现场的课堂图像并传输给识别模块；The collection module is used to collect classroom images on the spot in real time and transmit them to the identification module;
所述识别模块，用于对所述课堂图像进行分析，并判断所述课堂图像中行为异常的学生；所述识别模块包括：The identification module is used to analyze the classroom image and judge the students with abnormal behavior in the classroom image; the identification module includes:
行为检测单元，用于使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行行为检测；Behavior detection unit, for using the trained depth tensor column network model to carry out behavior detection to the students in the classroom image;
所述提示模块，用于将所述识别模块的识别结果通知授课教师。The prompting module is used to notify the teacher of the recognition result of the recognition module.
优选的，还设置有：与所述识别模块连接的存储模块；所述存储模块，用于同步存档所述识别结果并进行编辑分析；Preferably, it is also provided with: a storage module connected to the identification module; the storage module is used for synchronously archiving the identification results and performing editing and analysis;
所述识别模块还包括：The identification module also includes:
表情识别单元，用于使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行表情识别；The expression recognition unit is used to use the trained depth tensor network model to carry out expression recognition to the students in the classroom image;
所述表情识别单元包括人脸检测子单元和卷积神经网络分类器。The expression recognition unit includes a face detection subunit and a convolutional neural network classifier.
与现有技术相比，本发明的教学辅助方法，通过采用训练好的深度张量列网络模型来对所述课堂图像中的学生进行行为检测可以提供较高的图像识别精度和降低算法对硬件的要求，且能够在嵌入式设备上使用，降低了教学辅助方法的使用成本。Compared with the prior art, the teaching assistant method of the present invention can provide higher image recognition accuracy and reduce the burden on the hardware by using the trained deep tensor network model to detect the behavior of the students in the classroom image. requirements, and can be used on embedded devices, reducing the cost of using teaching aids.
进一步的，本发明还采用训练好的深度张量列网络模型来对所述课堂图像中的学生进行表情识别，使得教学辅助系统对学生上课时的异常行为识别精度更高。Further, the present invention also adopts the trained deep tensor sequence network model to recognize the expressions of the students in the classroom image, so that the teaching assistant system can recognize the abnormal behavior of the students in class with higher accuracy.
采用该方法的教学辅助系统，也同样具有上述优点。The teaching assistant system adopting this method also has the above-mentioned advantages.
附图说明Description of drawings
图1为一种教学辅助方法的基本流程图；Fig. 1 is a basic flowchart of a teaching aid method;
图2为一种教学辅助方法的详细流程图；Fig. 2 is a detailed flow chart of a teaching assistance method;
图3为采用图1教学辅助方法的教学辅助系统架构示意图；FIG. 3 is a schematic diagram of the architecture of the teaching assistance system adopting the teaching assistance method in FIG. 1;
图4为图3教学辅助系统的完整架构示意图；FIG. 4 is a schematic diagram of the complete architecture of the teaching assistance system in FIG. 3;
图5为全链接权值矩阵折叠和融合为三阶张量示意图；Figure 5 is a schematic diagram of folding and fusing the full link weight matrix into a third-order tensor;
图6为三阶张量进行张量列分解示意图；Fig. 6 is a schematic diagram of decomposing the tensor column of the third-order tensor;
图7为张量列分解示意图；Figure 7 is a schematic diagram of tensor column decomposition;
图8为矩阵的张量列分解示意图；Fig. 8 is the tensor column decomposition diagram of matrix;
图9为采集模块布设方式示意图；Fig. 9 is a schematic diagram of the layout of the acquisition module;
图10为行为检测所采用的深度张量列网络模型结构示意图；Figure 10 is a schematic diagram of the structure of the deep tensor column network model used in behavior detection;
图11为表情识别所采用的深度张量列网络模型结构采示意图。Figure 11 is a schematic diagram of the structure of the deep tensor sequence network model used in expression recognition.
具体实施方式Detailed ways
以下参考附图1至附图11，对本发明的各实施例予以进一步地详尽阐述。Various embodiments of the present invention will be further described in detail below with reference to accompanying drawings 1 to 11 .
如附图1所示一种教学辅助方法，包括以下顺序步骤：A kind of teaching aid method as shown in accompanying drawing 1, comprises the following sequential steps:
s1.采集模块实时采集现场的课堂图像，并传输给识别模块。s1. The acquisition module collects the on-site classroom images in real time and transmits them to the recognition module.
s2.所述识别模块对所述课堂图像进行分析，并判断所述课堂图像中行为异常的学生。s2. The identification module analyzes the classroom image, and judges students with abnormal behavior in the classroom image.
s3.提示模块将所述识别模块的识别结果通知授课教师。s3. The prompt module notifies the teacher of the recognition result of the recognition module.
所述步骤s2中包括以下步骤：The step s2 includes the following steps:
s21.所述识别模块使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行行为检测。s21. The recognition module uses the trained deep tensor network model to detect the behavior of the students in the classroom image.
具体的，步骤s21中采用的深度张量列网络模型通过对传统全连接层矩阵做张量列分解得来，极大压缩了全连接层矩阵张量的参数量，提高算法效率，降低了算法对硬件的要求，方便系统以嵌入式设备形式部署，使用更加方便简单且能够降低成本，利于本教学辅助系统的大规模推广。Specifically, the deep tensor column network model used in step s21 is obtained by decomposing the traditional fully connected layer matrix into tensor columns, which greatly compresses the parameter quantity of the fully connected layer matrix tensor, improves the efficiency of the algorithm, and reduces the The requirements for hardware facilitate the deployment of the system in the form of embedded devices, which are more convenient and simple to use and can reduce costs, which is conducive to the large-scale promotion of this teaching assistance system.
如附图2所示，所述步骤s2还包括以下步骤：As shown in accompanying drawing 2, described step s2 also comprises the following steps:
s22.所述识别模块使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行表情识别。s22. The recognition module uses the trained deep tensor network model to recognize the expressions of the students in the classroom image.
步骤s22重设了图像识别这一核心算法，通过联合对学生图片进行行为检测和表情识别，使深度张量列网络模型获得更好的识别精度和效率。在深度张量列网络模型降低了模型参数量，提升了系统的鲁棒性的基础上有效的提高了本教学辅助系统实时检测课堂上学生异常行为与表情的速度。Step s22 resets the core algorithm of image recognition. By jointly performing behavior detection and expression recognition on student pictures, the deep tensor network model can achieve better recognition accuracy and efficiency. On the basis of reducing the amount of model parameters and improving the robustness of the system, the deep tensor network model effectively improves the speed of real-time detection of abnormal behaviors and expressions of students in the classroom.
在本实施例中，所述步骤s22具体包括以下步骤：In this embodiment, the step s22 specifically includes the following steps:
s221.通过人脸检测子单元从所述采集模块采集到的所述课堂图像中识别出各学生的人脸区域。s221. Recognize the face area of each student from the classroom image collected by the collection module through the face detection subunit.
s222.通过卷积神经网络分类器对检测到的所述人脸区域做表情识别。s222. Using a convolutional neural network classifier to perform expression recognition on the detected face area.
在具体操作中，由于人脸表情特征相对较细，识别模块不方便直接提取表情特征，因此，本发明通过步骤s221和步骤s222来实现表情的识别。首先通过人脸检测子单元从图像采集模块采集到的课堂图片中检测出各学生人脸区域，再通过卷积神经网络分类器对检测到的各人脸区域图像块做表情识别。In the specific operation, since the facial expression features are relatively thin, it is inconvenient for the recognition module to directly extract the expression features. Therefore, the present invention realizes expression recognition through steps s221 and s222. First, through the face detection sub-unit, detect the face area of each student from the classroom pictures collected by the image acquisition module, and then perform expression recognition on the detected image blocks of each face area through the convolutional neural network classifier.
如附图9所示，步骤s1中包括以下步骤：As shown in accompanying drawing 9, step s1 comprises the following steps:
s11.所述采集模块在教室前方的左、中、右区域分别安装图像采集装置。s11. The acquisition module installs image acquisition devices in the left, middle and right areas in front of the classroom respectively.
在其他实施例中，也可以采用在教室前方左、右两个区域或者多个区域安装图像采集装置，以防止单个方向拍摄容易有学生被遮挡。In other embodiments, it is also possible to install image acquisition devices in the left and right areas or multiple areas in front of the classroom, so as to prevent students from being easily blocked when shooting in a single direction.
同时，在优选的实施例中，正常授课状态下大部分学生基本不会出现困惑、发呆、厌烦等异常行为，故可以为每个图像采集装置设置拍摄的时间间隔，以降低图像的采样率，节省相应的处理和存储资源。At the same time, in a preferred embodiment, under normal teaching conditions, most students will not appear in abnormal behaviors such as confusion, daze, and boredom, so the time interval for shooting can be set for each image acquisition device to reduce the sampling rate of the image. Corresponding processing and storage resources are saved.
s12.所述图像采集模块以班级中所有学生上半身图像为采集目标。s12. The image acquisition module takes the upper body images of all students in the class as the acquisition target.
在具体实施时，学生在上课时的行为和表情特征基本用上半身图像就可以提取和进行识别，以上半身为目标可以有针对性的拍摄特征比较富集的图像区域。In the specific implementation, the behavior and expression characteristics of students in class can be extracted and recognized basically by using the upper body image, and the upper body can be used as the target to shoot targeted image areas with richer features.
在本实施例中，还包括以下步骤：s4.存储模块同步存档所述识别结果。In this embodiment, the following steps are further included: s4. The storage module archives the identification results synchronously.
在本实施例中，通过对识别结果进行同步存储，可以进一步从整体方向对识别结果进行分析利用。比如：根据识别结果分析和评估教学效果和分析学生的学习曲线，可以更有针对性地开展教学活动，让接下来的教学工作更能有的放矢，整体提高教学水平和质量。In this embodiment, by synchronously storing the recognition results, the recognition results can be further analyzed and utilized from an overall perspective. For example, according to the analysis and evaluation of the teaching effect and the analysis of the students' learning curve based on the recognition results, the teaching activities can be carried out in a more targeted manner, so that the next teaching work can be more targeted, and the overall teaching level and quality can be improved.
优选的，所述步骤s4中包括以下步骤：Preferably, the step s4 includes the following steps:
s41.将每个学生对应的所述识别结果按班级制定成学生电子档案。s41. Making the identification results corresponding to each student into student electronic files by class.
有助于对每个学生上课状态的检测识别结果进行统计分析，以及主动的对学生在校的听课状态进行跟踪，避免仅靠学生的成绩来判断学生听课状态这种被动的方式迟滞性的弊端。It is helpful to conduct statistical analysis on the detection and identification results of each student's class status, and actively track the student's class status at school, avoiding the disadvantages of the passive method of hysteresis that only depends on the student's grades to judge the student's class status. .
s42.根据所述学生电子档案绘出学生上课状态曲线，用以便于授课教师结合当时教授的内容以及考试成绩对学生进行有针对性的辅导。s42. Draw the class status curve of the students according to the electronic files of the students, so that the teachers can provide targeted guidance to the students in combination with the content taught at that time and the test results.
同时，也可以将学生电子档案与教师的教学评估相结合，改进目前以学生考试成绩为课堂教学质量评估的主要参照过于片面的弊端。At the same time, students' electronic files can also be combined with teachers' teaching evaluations to improve the one-sided drawbacks of currently taking students' test scores as the main reference for classroom teaching quality evaluations.
在本实施例中，步骤s1之前还包括以下步骤：In this embodiment, the following steps are also included before step s1:
q1.构建数据集。q1. Build a dataset.
在具体实施中，可以分为为行为检测构建数据集和为表情识别构建数据集。In the specific implementation, it can be divided into building a data set for behavior detection and building a data set for expression recognition.
具体的，在为行为检测构建数据集时，构建一个合适的数据集是能否正确检测出异常行为学生的基础，直接关系到系统识别性能的高低。我们使用采集模块在多个教室长时间拍摄课堂上课情况，然后从中选取存在异常行为学生的图片进行标注，其中异常行为指一切表现为未认真听课的行为，如睡觉、说话、做小动作、发呆等。由于可能存在遮挡问题以及单视角的局限性，我们使用左、中、右三个视角的图像采集装置采集图片，分别标注。并对图片做简单处理，输入的固定尺寸以调整成适合模型，以方便在训练网络模型时使用。Specifically, when constructing a data set for behavior detection, constructing a suitable data set is the basis for the correct detection of students with abnormal behaviors, which is directly related to the recognition performance of the system. We use the acquisition module to take pictures of the class in multiple classrooms for a long time, and then select the pictures of students with abnormal behaviors to mark them. Abnormal behaviors refer to all behaviors that show that they do not listen to the class carefully, such as sleeping, talking, doing small movements, dazed, etc. . Due to the possible occlusion problem and the limitation of single viewing angle, we use image acquisition devices with three viewing angles of left, middle, and right to collect pictures and mark them separately. And do simple processing on the picture, and adjust the fixed size of the input to fit the model, so as to facilitate the use when training the network model.
在其他实施例中，也可以将专注、兴致盎然、思考等表情特征进行提取，并对深度张量列网络模型进行训练，使得认真听课的行为也可以通过该模型识别出来。In other embodiments, expression features such as focus, interest, and thinking can also be extracted, and the deep tensor network model can be trained, so that the behavior of listening carefully to the class can also be identified through the model.
在为表情识别构建数据集时，由于此处我们做人脸表情识别分两步进行，首先做人脸检测，再做表情识别，我们需构建两个数据集，一个为课堂人脸检测数据集，另一个为课堂学生上课表情数据集。When constructing a data set for expression recognition, since we do face expression recognition here in two steps, first do face detection, and then do expression recognition, we need to build two data sets, one is the classroom face detection data set, and the other A dataset of facial expressions for classroom students.
人脸检测数据集Face Detection Dataset
为能准确的从图像采集模块采集到的课堂图像中实时准确的检测出学生脸部，我们构建一个小型课堂人脸检测数据集。我们使用图像采集模块在多个教室长时间拍摄课堂上课情况采集到的课堂图像，对图片进行标注，给出图片中人脸位置，并对图片做简单处理，调整成适合模型输入的固定尺寸，以方便在训练网络模型时使用。In order to accurately detect the faces of students in the classroom images collected by the image acquisition module in real time, we build a small classroom face detection dataset. We use the image acquisition module to take long-term classroom images in multiple classrooms, mark the pictures, give the position of the face in the picture, and do simple processing on the picture to adjust it to a fixed size suitable for model input. It is convenient to use when training the network model.
学生上课表情数据集Student facial expressions dataset
为方便上课教师更加准确实时地了解课堂上每个学生的听课状态，满足学生上课表情识别需求，我们针对学生课堂听课这一场景，构建一学生上课表情数据集。从采集到的课堂图像中，截取出学生面部表情图片块，给出对应听课认真程度相关表情标签，如专注、兴致盎然、思考、困惑、发呆、厌烦等等。方便授课教师更加方便细致的掌握每位学生的听课状态和课程掌握情况与态度，做出实时的处理与调整。In order to make it easier for teachers to understand the status of each student in the classroom more accurately and in real time, and to meet the needs of students' facial expression recognition in class, we have constructed a student facial expression dataset for the scene of students listening in class. From the collected classroom images, the student’s facial expression picture block is intercepted, and the relevant expression labels corresponding to the seriousness of the class are given, such as concentration, interest, thinking, confusion, daze, boredom, etc. It is convenient for teachers to grasp each student's lecture status and course mastery situation and attitude more conveniently and meticulously, and make real-time processing and adjustments.
q2.训练所述深度张量列网络模型。q2. Train the deep tensor column network model.
在具体实施中，行为检测和表情识别的训练可以分开进行。其区别仅在于采用不同的数据集进行训练。In a specific implementation, the training of behavior detection and expression recognition can be performed separately. The only difference is that different datasets are used for training.
具体的，基于”深度张量列网络”的课堂学生异常行为识别神经网络模型。首先通过多层卷积层自动学习提取课堂图片中学生的行为特征，在使用学习到的课堂行为特征信息经TT分解(张量列分解)后的全连接层对学生课堂行为进行识别，检测出有异常课堂行为的学生。Specifically, a neural network model for identifying abnormal behaviors of classroom students based on the "Deep Tensor Network". Firstly, the behavior characteristics of students in classroom pictures are automatically learned and extracted through multi-layer convolutional layers, and the fully connected layer after TT decomposition (tensor column decomposition) is used to identify students' classroom behaviors by using the learned classroom behavior feature information, and detects the students' behavior characteristics. Students with unusual classroom behavior.
如附图7所示，张量列分解(tensor train decomposition， TT-decomposition)是一种张量分解模型，将张量的每一个元素都用若干个矩阵的乘积表示。假设存在d阶张量(Ik表示第k阶的维数)，张量的张量列分解为：As shown in Figure 7, tensor train decomposition (TT-decomposition) is a tensor decomposition model, and each element of the tensor is represented by the product of several matrices. Suppose there is a tensor of order d (Ik represents the dimension of the kth order), tensor The tensor columns decompose into:
其中是张量第k阶级对应的核矩阵，规模为rk-1′rk，k＝1,2,...d， r0＝rd＝1；(r0，r1，…rd)是d阶张量进行张量列时对应的TT-rank，实际上是规模为rk-1 Ikrk三阶张量，所以又叫核张量。in is a tensor The kernel matrix corresponding to the kth class has a scale of rk-1 ′rk , k=1,2,...d, r0 =rd =1; (r0 , r1 ,...rd ) is d rank tensor The corresponding TT-rank when performing tensor columns, in fact is a third-order tensor of size rk-1 Ik rk , so Also called nuclear tensor.
如附图8所示，矩阵的张量列分解，假设矩阵A∈RM×N，选择重构方案，如重构方案：选定重构方案后，矩阵的张量列分解首先将矩阵映射到d阶张量再对张量进行张量列分解，即As shown in Figure 8, the tensor column decomposition of the matrix, assuming the matrix A∈RM×N , select the reconstruction scheme, such as the reconstruction scheme: After the reconstruction scheme is selected, the tensor column decomposition of the matrix first maps the matrix to a tensor of order d Again for tensor Perform tensor column decomposition, i.e.
如附图2所示，所述步骤q1包括以下步骤：As shown in accompanying drawing 2, described step q1 comprises the following steps:
q11.所述采集模块在教室长时间拍摄所述课堂图像并存储。q11. The acquisition module takes and stores the classroom image for a long time in the classroom.
q12.选取存在异常的学生图片进行标注。q12. Select the pictures of students with abnormalities for labeling.
由于含有异常行为图像数据可能相对较少，为避免模型过拟合，并增强模型对光照变化等因素的抗干扰能力，我们对采集标注的课堂学生异常行为图片数据做数据增强。分别对图片进行改变对比度、RGB通道强度、加噪声等处理，增加图片数据样本量和种类。Since the image data containing abnormal behavior may be relatively small, in order to avoid over-fitting of the model and enhance the anti-interference ability of the model to factors such as illumination changes, we perform data enhancement on the collected and marked abnormal behavior image data of classroom students. Change the contrast, RGB channel intensity, add noise, etc. to the picture separately to increase the sample size and type of picture data.
在本实施例中，所述步骤q2包括以下步骤：In this embodiment, the step q2 includes the following steps:
q21.通过神经网络模型的多层卷积层提取已标注的所述学生图片中的异常特征,异常特征与分解后的全连接层权重矩阵运算得到输出预测值。q21. The abnormal features in the marked student pictures are extracted through the multi-layer convolution layer of the neural network model, and the abnormal features and the decomposed fully connected layer weight matrix are operated to obtain the output prediction value.
如附图10所示的深度张量列网络的模型结构(此处仅以3层卷积为例说明)；构建该深度张量列网络模型的步骤为：The model structure of the depth tensor column network as shown in accompanying drawing 10 (here only take 3 layers of convolutions as an example); the steps of constructing this depth tensor column network model are:
1.初始化网络模型参数。1. Initialize network model parameters.
2.将构建的课堂学生异常行为数据集中的图片输入到该模型进行训练。2. Input the pictures in the abnormal behavior data set of classroom students into the model for training.
3.图片经过不断卷积池化，在最后一层卷积层输出S x S x m的张量A，即把原图片划分出了一个S x S的网格，每个网格单元对应着原课堂图片的一部分，每个网格中的图片特征对应着该张量中的一个m维向量。3. After continuous convolution and pooling of the picture, the tensor A of S x S x m is output in the last layer of convolution layer, that is, the original picture is divided into a grid of S x S, and each grid unit corresponds to the original Part of the classroom picture, the picture features in each grid correspond to an m-dimensional vector in the tensor.
4.经过改进后的全链接，输出一个S x S x(5a)的张量，即每个网格单元对应的a个异常行为学生检测边界框坐标(x,y,w,h)与识别框中检测为异常行为学生的置信度。其中x和y为异常行为学生识别框中心点坐标，w和h分别为异常行为学生识别框的宽和高，并将坐标进行归一化,使其介于0到1之间。4. After the improved full link, output a tensor of S x S x (5a), that is, a abnormal behavior corresponding to each grid cell. The coordinates (x, y, w, h) and recognition of the student detection bounding box The confidence level of the student detected as anomalous behavior in the box. Among them, x and y are the coordinates of the center point of the abnormal behavior student identification frame, w and h are the width and height of the abnormal behavior student identification frame respectively, and the coordinates are normalized to make it between 0 and 1.
其中改进后的全连接为对传统全连接层矩阵做张量列(TT)分解，从而极大压缩全连接层参数量，提高算法效率，降低对硬件的要求，使得能在嵌入式设备上使用。为本教学辅助系统提高了实时检测课堂异常行为学生的速度，并方便系统以嵌入式设备形式部署，更加方便简单且能够降低成本，利于本课堂异常行为学生识别教学辅助系统的大规模推广。Among them, the improved full connection is the tensor column (TT) decomposition of the traditional fully connected layer matrix, thereby greatly compressing the parameters of the fully connected layer, improving the efficiency of the algorithm, reducing the requirements for hardware, and enabling it to be used on embedded devices. . This teaching assistant system improves the speed of real-time detection of students with abnormal behavior in the classroom, and facilitates the deployment of the system in the form of embedded devices, which is more convenient and simple and can reduce costs, which is conducive to the large-scale promotion of the teaching assistant system for identifying students with abnormal behavior in the classroom.
张量列分解(TT分解)是一种张量分解模型，将张量的每一个元素都用若干个矩阵的乘积表示。矩阵的张量列分解需先选择重构方案，首先将矩阵映射到d阶张量，再对张量进行张量列分解。此处即是对全连接层权重矩阵做张量列分解，以下为对该过程的详细解释(为了方便说明，我们代入一些参数举例，但具体实现不局限于具体参数)Tensor Column Decomposition (TT Decomposition) is a tensor decomposition model that represents each element of a tensor by the product of several matrices. The tensor column decomposition of the matrix needs to choose the reconstruction scheme first, first map the matrix to the d-order tensor, and then perform the tensor column decomposition on the tensor. Here is the tensor column decomposition of the weight matrix of the fully connected layer. The following is a detailed explanation of the process (for the convenience of explanation, we substitute some parameters as examples, but the specific implementation is not limited to specific parameters)
在本实施例中，全连接层权重矩阵张量列分解步骤为：In this embodiment, the decomposing steps of the weight matrix tensor column of the fully connected layer are as follows:
1.如附图5所示，将全连接权值矩阵的行和列均折叠为d个虚拟的维度；此处假设网络模型中S＝4，m＝50,n＝49,即图像采集模块采集到的课堂图像经逐层卷积池化后提取到4x4x50＝800个特征，下一隐层有4x4x49＝784个隐节点，则该全连接层权重参数为800x784的矩阵。为方便表示，取d＝3,将全连接权值矩阵的行和列均折叠为3个虚拟的维度，如附图所示。1. As shown in Figure 5, the rows and columns of the fully connected weight matrix are folded into d virtual dimensions; here it is assumed that S=4, m=50, n=49 in the network model, that is, the image acquisition module The collected classroom images are pooled layer by layer to extract 4x4x50=800 features, and the next hidden layer has 4x4x49=784 hidden nodes, so the weight parameter of the fully connected layer is a matrix of 800x784. For the convenience of representation, d=3 is taken, and the rows and columns of the fully connected weight matrix are folded into 3 virtual dimensions, as shown in the attached figure.
2.如附图5所示，将行列对应的虚拟的维度进行融合，即将全连接权值矩阵重塑为d阶张量；按上述实例方法则原800x784的权重矩阵被重塑为了 700x32x28的3阶张量。2. As shown in Figure 5, the virtual dimensions corresponding to the rows and columns are fused, that is, the fully connected weight matrix is reshaped into a d-order tensor; according to the above example method, the original 800x784 weight matrix is reshaped into a 700x32x28 3 rank tensor.
3.如附图6所示，定义所述d阶张量的张量列秩r，其中rk表示原始张量除去前(k-1)阶效应后沿张量第k阶展开的矩阵的秩，其中r0＝rd＝1是约束条件；本文定义的张量列秩为3。3. As shown in accompanying drawing 6, define the tensor row rank r of described d order tensor, wherein rk represents the original tensor removes the front (k-1) order effect of the matrix along tensor k order expansion Rank, where r0=rd=1 is the constraint condition; the tensor column rank defined in this paper is 3.
4.将所述d阶张量进行张量列分解得到全连接层权值矩阵的张量列分解表示，即其中是规模为rk-1 Ik rk三阶张量，Ik表示高阶张量第k阶的维数。在本实例中，即原700x32x28的3阶张量被分解为了 1x700x3、3x32x3、3x28x1的3个核张量。全连接层权重由原来的627200个参数下降到了2472个参数。4. Decompose the tensor column of the d-order tensor to obtain the tensor column decomposition representation of the weight matrix of the fully connected layer, namely in Is a third-order tensor whose size is rk-1 Ik rk , and Ik represents the k-th order dimension of the high-order tensor. In this example, the original 700x32x28 rank 3 tensor is decomposed into 3 core tensors of 1x700x3, 3x32x3, and 3x28x1. The weight of the fully connected layer is reduced from the original 627200 parameters to 2472 parameters.
为比较直观的表示TT分解(张量列分解)对全连接层权重参数量的压缩效果，现将几种重塑方案下张量列分解前后参数规模计算如下表。由表中计算结果可看出，全连接层权重参数经张量列分解后参数量下降了成百上千倍，能提高算法效率，降低对硬件的要求，方便本教学辅助系统在嵌入式设备上的实现，提高检测课堂上学生异常行为的实时性。In order to more intuitively express the compression effect of TT decomposition (tensor column decomposition) on the weight parameters of the fully connected layer, the parameter scale before and after tensor column decomposition under several reshaping schemes is calculated in the following table. It can be seen from the calculation results in the table that the weight parameters of the fully connected layer are reduced by hundreds or thousands of times after being decomposed by tensor columns, which can improve the efficiency of the algorithm and reduce the requirements for hardware. The realization of the above can improve the real-time performance of detecting abnormal behaviors of students in the classroom.
q22.输出预测值与所述学生图片中的异常行为学生真实标注值的误差构成的损失函数；q22. Output the loss function formed by the error between the predicted value and the real labeled value of the abnormal behavior student in the student picture;
q23.根据损失函数调整网络参数，得到训练好的深度张量列网络模型。q23. Adjust the network parameters according to the loss function to obtain the trained deep tensor column network model.
5.运用反向传播算法，根据输出的预测值与原图中的异常行为学生真实标注值间误差构成的损失函数L(此处损失函数采用平方和误差损失函数，在下文中具体介绍)，调整网络参数，至指定精度。然后保存网络参数。5. Using the backpropagation algorithm, according to the loss function L formed by the error between the output predicted value and the student's real label value of the abnormal behavior in the original image (the loss function here uses the square sum error loss function, which will be described in detail below), adjust Network parameters, to the specified precision. Then save the network parameters.
损失函数使用平方和误差损失函数其中包括3部分，坐标预测函数，包含异常行为学生的识别框的置信度预测函数和不包含异常行为学生的识别框的置信度预测函数。The loss function uses the sum of square error loss function, which includes 3 parts, coordinate prediction function, confidence prediction function of the recognition frame containing abnormal behavior students, and confidence prediction function of recognition frame not containing abnormal behavior students.
其中，x,y是异常行为学生识别框的中心位置坐标,w,h是异常行为学生识别框的宽和高，为判断第i个网格中的第j个识别框是否负责检测，为判断是否有异常行为学生中心落入在网格i中，lcoord为坐标预测权重，lnoobj为不包含异常行为学生的识别框的置信度权重。Among them, x, y are the coordinates of the center position of the abnormal behavior student identification frame, w, h are the width and height of the abnormal behavior student identification frame, In order to judge whether the j-th recognition frame in the i-th grid is responsible for detection, In order to judge whether there is an abnormal behavior student center falling into the grid i, lcoord is the coordinate prediction weight, lnoo bj is the confidence weight of the recognition frame that does not contain abnormal behavior students.
在优选的实施例中，如附图10、附图11所示，为表情识别训练时深度张量列网络模型时，首先训练人脸检测网络模型，人脸检测与行为检测子模块的课堂异常行为检测模型类似，将其中课堂异常行为数据集换成人脸检测数据集，使用人脸检测数据集中的图片输入模型训练，重复上述行为检测子模块中1-5的训练过程即可，使得模型能自动学习人脸特征，从课堂图像中自动检测出学生脸部位置。In a preferred embodiment, as shown in accompanying drawing 10, accompanying drawing 11, when being expression recognition training depth tensor series network model, at first train face detection network model, the class of human face detection and behavior detection submodule is abnormal The behavior detection model is similar. Replace the abnormal behavior data set in the classroom with the face detection data set, use the pictures in the face detection data set to input the model for training, and repeat the training process of 1-5 in the above behavior detection sub-module, so that the model It can automatically learn facial features and automatically detect the position of students' faces from classroom images.
其次训练课堂人脸面部表情识别时采用的卷积神经网络(CNN)分类器。将前述构建的学生上课表情数据集中带表情标签的学生脸部图片块输入表情识别分类器，对表情识别网络模型进行训练。表情识别网络模型如附图11。Secondly, train the convolutional neural network (CNN) classifier used in classroom facial expression recognition. Input the expression recognition classifier to the expression recognition network model by inputting the student face picture blocks with expression labels in the student expression data set constructed above. The facial expression recognition network model is shown in Figure 11.
1.初始化表情识别网络模型参数。1. Initialize the facial expression recognition network model parameters.
2.将构建的学生上课表情数据集中带表情标签的学生脸部图片块输入到该模型进行训练。2. Input the student face picture blocks with expression labels in the constructed student expression data set to the model for training.
3.学生脸部图片块经过不断卷积池化，提取面部表情特征。3. The student's face picture block is continuously convolution pooled to extract facial expression features.
4.经过改进后的全链接，输出预测的学生脸部图片块表情标签。此处也对全连接层权重矩阵做TT分解。具体过程在行为检测子模块中(4)中有详细介绍，此处不再赘述。4. After the improved full link, output the predicted student face picture block expression label. Here, TT decomposition is also performed on the weight matrix of the fully connected layer. The specific process is introduced in detail in (4) in the behavior detection sub-module, and will not be repeated here.
5.运用反向传播算法，根据输出的预测值与真实标注表情标签间误差构成的损失函数L，调整网络参数，至指定精度，然后保存网络参数。5. Use the backpropagation algorithm to adjust the network parameters to the specified accuracy according to the loss function L formed by the error between the output prediction value and the real label expression label, and then save the network parameters.
在其他实施例中，还为检索模型学习的准确性，还包括模型测试的步骤。In other embodiments, to retrieve the accuracy of model learning, a step of model testing is also included.
在为行为检测进行测试时，将上述训练好的网络模型参数导入识别模块中行为检测子模块的深度张量列网络，输入由图像采集模块实时采集的课堂图片，实时检测图片中是否有异常行为学生，如果有则标出并将识别结果由提示模块通知授课教师，并由存储模块存档，以便后续对数据做进一步分析挖掘。是否为异常行为根据网络模型给出的异常行为概率是否大于给定的概率阈值确定，默认概率阈值通过多次测试给出一个合理的符合大众的能较好平衡灵敏度与准确度的值，老师后续可根据个人情况做适当调整，以使得本教学辅助系统更为人性化。测试期间可根据存在的问题，在细节上做适当调整，以便使系统达到最佳状态，然后投入实际使用。When testing for behavior detection, import the above-mentioned trained network model parameters into the deep tensor network of the behavior detection sub-module in the recognition module, input the classroom pictures collected in real time by the image acquisition module, and detect whether there are abnormal behaviors in the pictures in real time Students, if there is, mark it and notify the teacher of the recognition result from the prompt module, and archive it by the storage module, so that the data can be further analyzed and mined later. Whether it is an abnormal behavior is determined according to whether the probability of abnormal behavior given by the network model is greater than the given probability threshold. The default probability threshold is given a reasonable value that is in line with the public and can better balance sensitivity and accuracy through multiple tests. The teacher will follow up Appropriate adjustments can be made according to individual conditions to make the teaching assistance system more humanized. During the test period, according to the existing problems, the details can be adjusted appropriately so that the system can reach the best state, and then it can be put into actual use.
在为表情识别进行测试时，将上述训练好的网络模型参数导入识别模块中表情识别子模块，输入由图像采集模块实时采集的课堂图片，首先由人脸检测网络模型检测出图片中的所有人脸位置，再将检测到的人脸图片块简单处理后，调整成固定大小输入表情识别网络模型识别学生上课表情。使得模型自动检测人脸并识别其表情特征，以便模型能投入实际使用，实时检测分析课堂上学生的表情信息，结合行为检测模块结果，方便上课教师更加准确实时地了解课堂上每个学生的上课状态，让授课教师更能有的放矢，提高教学质量和效率。When testing for expression recognition, import the above-mentioned trained network model parameters into the expression recognition sub-module in the recognition module, input the classroom pictures collected in real time by the image acquisition module, and first detect all the people in the pictures by the face detection network model Face position, and then simply process the detected face picture block, adjust it to a fixed size input expression recognition network model to recognize students' expressions in class. Make the model automatically detect human faces and identify their expression characteristics, so that the model can be put into practical use, real-time detection and analysis of the expression information of students in the classroom, combined with the results of the behavior detection module, it is convenient for teachers to understand the class of each student in the classroom more accurately and in real time State, so that teachers can be more targeted, improve teaching quality and efficiency.
为了解决上述技术问题，本发明还提供一种教学辅助系统，设置有：采集模块、与所述采集模块连接的识别模块、与所述识别模块连接的提示模块。In order to solve the above technical problems, the present invention also provides a teaching assistance system, which is provided with: an acquisition module, an identification module connected to the acquisition module, and a prompt module connected to the identification module.
所述采集模块，用于实时采集现场的课堂图像并传输给识别模块。The collection module is used to collect on-site classroom images in real time and transmit them to the identification module.
采集模块，如附图9所示，图像采集模块采集目标为班级所有学生上半身图片。采集方式是通过在教室前方墙壁的左、中、右顶端分别安装图像采集装置，调整好拍摄角度，以防止遮挡并综合多个视角，设置图像采集装置每次拍摄的时间间隔，把采集到的图片处理成识别模块所需大小后传输到识别模块，为进行课堂行为识别提供数据；Acquisition module, as shown in Figure 9, the acquisition target of the image acquisition module is the upper body pictures of all students in the class. The acquisition method is to install the image acquisition device on the left, middle and right top of the front wall of the classroom, adjust the shooting angle to prevent occlusion and integrate multiple viewing angles, set the time interval of each shooting of the image acquisition device, and collect the collected The image is processed into the size required by the recognition module and then transmitted to the recognition module to provide data for classroom behavior recognition;
识别模块，用于对所述课堂图像进行分析，并判断所述课堂图像中行为异常的学生；其具体包括以下单元：The identification module is used to analyze the classroom image and judge the students with abnormal behavior in the classroom image; it specifically includes the following units:
行为检测单元，用于使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行行为检测。The behavior detection unit is used to use the trained deep tensor column network model to detect the behavior of the students in the classroom image.
具体的，识别模块的目的是识别图像采集模块中上课学生的具体课堂行为与表情来判断学生是否在认真听课，了解学生对授课内容的接受程度。行为检测方法是先采集课堂中上课学生图片数据，对图片做人工标注，标出其中异常学生，即未认真听课学生，具体包括睡觉、说话、做小动作、发呆等。再使用构建的课堂行为图片数据集训练深度张量列网络模型，使得识别模块能自动学习图片特征，检测出图片中的学生异常行为。最后将训练好的模型投入实际使用，实时获取图像采集模块中传输来的3张图像(本专利以三幅图像为例进行说明，硬件设备许可的条件下，可以实时采集多幅图像)，分别检测图片中的学生异常行为，并根据给定的概率阈值框出行为异常学生。Specifically, the purpose of the identification module is to identify the specific classroom behaviors and expressions of the students in the image acquisition module to determine whether the students are listening to the class seriously, and to understand the students' acceptance of the teaching content. The behavior detection method is to first collect the picture data of the students in the classroom, manually mark the pictures, and mark the abnormal students, that is, the students who are not paying attention to the class, including sleeping, talking, doing small movements, dazed, etc. Then use the constructed classroom behavior picture dataset to train the deep tensor network model, so that the recognition module can automatically learn the picture features and detect the abnormal behavior of students in the picture. Finally, the trained model is put into practical use, and the 3 images transmitted from the image acquisition module are obtained in real time (this patent uses three images as an example, and under the condition of hardware equipment permission, multiple images can be collected in real time), respectively. Detect the abnormal behavior of students in the picture, and frame the students with abnormal behavior according to the given probability threshold.
所述提示模块，用于将所述识别模块的识别结果通知授课教师。The prompting module is used to notify the teacher of the recognition result of the recognition module.
提示模块，提示模块实时的将识别结果综合以某种方式通知授课教师，若3个角度的图像都无异常则不通知，教师可通过调节概率阈值以调节识别灵敏度。教师在接收到提示后可以实时了解课堂学生的听课状态和对其所教授的内容的接受程度，可以以此为基础对其中接受程度不是很好的同学重点提问或采取相应的对策。Prompt module. The prompt module notifies the teacher of the recognition results in a certain way in real time. If the images from the three angles are normal, no notification will be made. The teacher can adjust the recognition sensitivity by adjusting the probability threshold. After receiving the reminder, the teacher can know the status of the students in the classroom and their acceptance of the content taught in real time. Based on this, they can focus on asking questions or taking corresponding countermeasures for students who do not accept the content very well.
优选的，还设置有：与所述识别模块连接的存储模块；所述存储模块，用于同步存档所述识别结果并进行编辑分类。Preferably, a storage module connected to the recognition module is also provided; the storage module is used for synchronously archiving the recognition results and performing editing and classification.
存储模块，存储模块是将该系统的所有识别的最终结果以班级为目，学生为类，以学生个人档案的形式进行存储，学校可以充分利用这些电子档案，从中挖掘出有用信息，一方面可以根据学生整体接受情况来分析和评估教学中的不足，另一方面可以分析学生的学习曲线，找到学生成绩不好的真正原因，可以有针对性地进行查缺补漏。The storage module, the storage module is to store all the identified final results of the system in the form of student personal files with the class as the purpose and students as the class. The school can make full use of these electronic files to dig out useful information. On the one hand, it can Analyze and evaluate the deficiencies in teaching according to the overall acceptance of students. On the other hand, it can analyze the learning curve of students, find out the real reasons for students' poor grades, and make targeted checks and make up for omissions.
所述识别模块还包括：The identification module also includes:
表情识别单元，用于使用训练好的深度张量列网络模型来对所述课堂图像中的学生进行表情识别。The facial expression recognition unit is used to use the trained deep tensor network model to recognize the facial expressions of the students in the classroom image.
所述表情识别单元包括人脸检测子单元和卷积神经网络分类器。The expression recognition unit includes a face detection subunit and a convolutional neural network classifier.
表情识别子模块方法与行为检测类似，不同之处在于标注异常表情进行算法训练。本专利中以两子模块以相同算法模型分开并行识别方式进行描述，但也可以通过改变损失函数，通过多任务损失函数将两项任务融合到同一模型中识别，此处不做具体阐述，但亦在本专利保护范围之内。The expression recognition sub-module method is similar to behavior detection, the difference is that abnormal expressions are marked for algorithm training. In this patent, the two sub-modules are described in parallel recognition with the same algorithm model, but the loss function can also be changed to integrate the two tasks into the same model through the multi-task loss function for recognition. No specific elaboration is made here, but Also within the protection scope of this patent.
本发明所要求保护的方案很好的解决了通过对课堂图像进行分析处理以辅助老师进行教学活动的技术问题，避免了现有的教学设备过于依赖外部的图像识别装置导致硬件要求高且识别不准确的缺陷，提升了老师教学工作的效率。The solution claimed in the present invention solves the technical problem of assisting teachers to carry out teaching activities by analyzing and processing classroom images, and avoids the existing teaching equipment that relies too much on external image recognition devices, resulting in high hardware requirements and poor recognition. Accurate defects improve the efficiency of teachers' teaching work.
上述内容，仅为本发明的较佳实施例，并非用于限制本发明的实施方案，本领域普通技术人员根据本发明的主要构思和精神，可以十分方便地进行相应的变通或修改，故本发明的保护范围应以权利要求书所要求的保护范围为准。The above content is only a preferred embodiment of the present invention, and is not intended to limit the implementation of the present invention. Those of ordinary skill in the art can easily make corresponding modifications or modifications according to the main idea and spirit of the present invention. Therefore, this The protection scope of the invention shall be determined by the protection scope required by the claims.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710667590.7ACN108073888A (en) | 2017-08-07 | 2017-08-07 | A kind of teaching auxiliary and the teaching auxiliary system using this method |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710667590.7ACN108073888A (en) | 2017-08-07 | 2017-08-07 | A kind of teaching auxiliary and the teaching auxiliary system using this method |
| Publication Number | Publication Date |
|---|---|
| CN108073888Atrue CN108073888A (en) | 2018-05-25 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710667590.7APendingCN108073888A (en) | 2017-08-07 | 2017-08-07 | A kind of teaching auxiliary and the teaching auxiliary system using this method |
| Country | Link |
|---|---|
| CN (1) | CN108073888A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108830207A (en)* | 2018-06-06 | 2018-11-16 | 成都邑教云信息技术有限公司 | A kind of Internet education warning system |
| CN108921748A (en)* | 2018-07-17 | 2018-11-30 | 郑州大学体育学院 | Didactic code method and computer-readable medium based on big data analysis |
| CN108960101A (en)* | 2018-06-22 | 2018-12-07 | 张小勇 | Data processing method and system |
| CN109165633A (en)* | 2018-09-21 | 2019-01-08 | 上海健坤教育科技有限公司 | A kind of intelligent interactive learning system based on camera perception |
| CN109359606A (en)* | 2018-10-24 | 2019-02-19 | 江苏君英天达人工智能研究院有限公司 | A kind of classroom real-time monitoring and assessment system and its working method, creation method |
| CN109377432A (en)* | 2018-12-21 | 2019-02-22 | 广东粤众互联信息技术有限公司 | A kind of tutoring system based on big data acquisition |
| CN109522883A (en)* | 2018-12-28 | 2019-03-26 | 广州海昇计算机科技有限公司 | A kind of method for detecting human face, system, device and storage medium |
| CN109977989A (en)* | 2019-01-17 | 2019-07-05 | 北京工业大学 | A kind of processing method of image tensor data |
| CN110009539A (en)* | 2019-04-12 | 2019-07-12 | 烟台工程职业技术学院(烟台市技师学院) | A kind of student is in school learning state smart profile system and application method |
| CN110175534A (en)* | 2019-05-08 | 2019-08-27 | 长春师范大学 | Teaching assisting system based on multitask concatenated convolutional neural network |
| CN110175501A (en)* | 2019-03-28 | 2019-08-27 | 重庆电政信息科技有限公司 | More people's scene focus recognition methods based on recognition of face |
| CN110363245A (en)* | 2019-07-17 | 2019-10-22 | 上海掌学教育科技有限公司 | Wonderful picture screening method, device and system for online classroom |
| CN110414415A (en)* | 2019-07-24 | 2019-11-05 | 北京理工大学 | Human behavior recognition method for classroom scenes |
| CN110827595A (en)* | 2019-12-12 | 2020-02-21 | 广州三人行壹佰教育科技有限公司 | Interaction method and device in virtual teaching and computer storage medium |
| CN110827491A (en)* | 2019-09-26 | 2020-02-21 | 天津市华软创新科技有限公司 | School student behavior big data analysis system |
| CN111339809A (en)* | 2018-12-20 | 2020-06-26 | 深圳市鸿合创新信息技术有限责任公司 | Classroom behavior analysis method and device, and electronic equipment |
| CN111832595A (en)* | 2019-04-23 | 2020-10-27 | 北京新唐思创教育科技有限公司 | Method for determining teacher style and computer storage medium |
| WO2020216286A1 (en)* | 2019-04-23 | 2020-10-29 | 北京新唐思创教育科技有限公司 | Method for training teaching style prediction model, and computer storage medium |
| CN112116181A (en)* | 2019-06-20 | 2020-12-22 | 北京新唐思创教育科技有限公司 | Training method of classroom quality model, classroom quality evaluation method and device |
| CN112201116A (en)* | 2020-09-29 | 2021-01-08 | 深圳市优必选科技股份有限公司 | Logic board identification method and device and terminal equipment |
| WO2021047185A1 (en)* | 2019-09-12 | 2021-03-18 | 深圳壹账通智能科技有限公司 | Monitoring method and apparatus based on facial recognition, and storage medium and computer device |
| CN112597977A (en)* | 2021-03-02 | 2021-04-02 | 南京泛在实境科技有限公司 | HSV-YOLOv 3-based online class student behavior identification method |
| CN114897647A (en)* | 2022-04-27 | 2022-08-12 | 合创智能家具(广东)有限公司 | Teaching auxiliary system |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110169932A1 (en)* | 2010-01-06 | 2011-07-14 | Clear View Technologies Inc. | Wireless Facial Recognition |
| CN103258204A (en)* | 2012-02-21 | 2013-08-21 | 中国科学院心理研究所 | Automatic micro-expression recognition method based on Gabor features and edge orientation histogram (EOH) features |
| CN106097790A (en)* | 2016-08-31 | 2016-11-09 | 王翠丽 | A kind of teaching auxiliary device |
| CN106778745A (en)* | 2016-12-23 | 2017-05-31 | 深圳先进技术研究院 | A kind of licence plate recognition method and device, user equipment |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110169932A1 (en)* | 2010-01-06 | 2011-07-14 | Clear View Technologies Inc. | Wireless Facial Recognition |
| CN103258204A (en)* | 2012-02-21 | 2013-08-21 | 中国科学院心理研究所 | Automatic micro-expression recognition method based on Gabor features and edge orientation histogram (EOH) features |
| CN106097790A (en)* | 2016-08-31 | 2016-11-09 | 王翠丽 | A kind of teaching auxiliary device |
| CN106778745A (en)* | 2016-12-23 | 2017-05-31 | 深圳先进技术研究院 | A kind of licence plate recognition method and device, user equipment |
| Title |
|---|
| 胡勇等: "复杂掩模图数据处理与转换的研究", 《计算机技术与应用》* |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108830207A (en)* | 2018-06-06 | 2018-11-16 | 成都邑教云信息技术有限公司 | A kind of Internet education warning system |
| CN108960101B (en)* | 2018-06-22 | 2022-06-14 | 张小勇 | Data processing method and system |
| CN108960101A (en)* | 2018-06-22 | 2018-12-07 | 张小勇 | Data processing method and system |
| CN108921748A (en)* | 2018-07-17 | 2018-11-30 | 郑州大学体育学院 | Didactic code method and computer-readable medium based on big data analysis |
| CN108921748B (en)* | 2018-07-17 | 2022-02-01 | 郑州大学体育学院 | Teaching planning method based on big data analysis and computer readable medium |
| CN109165633A (en)* | 2018-09-21 | 2019-01-08 | 上海健坤教育科技有限公司 | A kind of intelligent interactive learning system based on camera perception |
| CN109359606A (en)* | 2018-10-24 | 2019-02-19 | 江苏君英天达人工智能研究院有限公司 | A kind of classroom real-time monitoring and assessment system and its working method, creation method |
| WO2020082971A1 (en)* | 2018-10-24 | 2020-04-30 | 江苏君英天达人工智能研究院有限公司 | Real-time classroom monitoring and evaluation system and operation and creation method thereof |
| CN111339809A (en)* | 2018-12-20 | 2020-06-26 | 深圳市鸿合创新信息技术有限责任公司 | Classroom behavior analysis method and device, and electronic equipment |
| CN109377432A (en)* | 2018-12-21 | 2019-02-22 | 广东粤众互联信息技术有限公司 | A kind of tutoring system based on big data acquisition |
| CN109522883A (en)* | 2018-12-28 | 2019-03-26 | 广州海昇计算机科技有限公司 | A kind of method for detecting human face, system, device and storage medium |
| CN109977989A (en)* | 2019-01-17 | 2019-07-05 | 北京工业大学 | A kind of processing method of image tensor data |
| CN109977989B (en)* | 2019-01-17 | 2021-04-20 | 北京工业大学 | Image tensor data processing method |
| CN110175501A (en)* | 2019-03-28 | 2019-08-27 | 重庆电政信息科技有限公司 | More people's scene focus recognition methods based on recognition of face |
| CN110009539A (en)* | 2019-04-12 | 2019-07-12 | 烟台工程职业技术学院(烟台市技师学院) | A kind of student is in school learning state smart profile system and application method |
| CN111832595A (en)* | 2019-04-23 | 2020-10-27 | 北京新唐思创教育科技有限公司 | Method for determining teacher style and computer storage medium |
| WO2020216286A1 (en)* | 2019-04-23 | 2020-10-29 | 北京新唐思创教育科技有限公司 | Method for training teaching style prediction model, and computer storage medium |
| CN110175534A (en)* | 2019-05-08 | 2019-08-27 | 长春师范大学 | Teaching assisting system based on multitask concatenated convolutional neural network |
| CN112116181A (en)* | 2019-06-20 | 2020-12-22 | 北京新唐思创教育科技有限公司 | Training method of classroom quality model, classroom quality evaluation method and device |
| CN110363245A (en)* | 2019-07-17 | 2019-10-22 | 上海掌学教育科技有限公司 | Wonderful picture screening method, device and system for online classroom |
| CN110363245B (en)* | 2019-07-17 | 2023-05-12 | 上海掌学教育科技有限公司 | Screening method, device and system for wonderful pictures in online classroom |
| CN110414415A (en)* | 2019-07-24 | 2019-11-05 | 北京理工大学 | Human behavior recognition method for classroom scenes |
| WO2021047185A1 (en)* | 2019-09-12 | 2021-03-18 | 深圳壹账通智能科技有限公司 | Monitoring method and apparatus based on facial recognition, and storage medium and computer device |
| CN110827491A (en)* | 2019-09-26 | 2020-02-21 | 天津市华软创新科技有限公司 | School student behavior big data analysis system |
| CN110827595A (en)* | 2019-12-12 | 2020-02-21 | 广州三人行壹佰教育科技有限公司 | Interaction method and device in virtual teaching and computer storage medium |
| CN112201116A (en)* | 2020-09-29 | 2021-01-08 | 深圳市优必选科技股份有限公司 | Logic board identification method and device and terminal equipment |
| CN112597977A (en)* | 2021-03-02 | 2021-04-02 | 南京泛在实境科技有限公司 | HSV-YOLOv 3-based online class student behavior identification method |
| CN114897647A (en)* | 2022-04-27 | 2022-08-12 | 合创智能家具(广东)有限公司 | Teaching auxiliary system |
| Publication | Publication Date | Title |
|---|---|---|
| CN108073888A (en) | A kind of teaching auxiliary and the teaching auxiliary system using this method | |
| WO2019028592A1 (en) | Teaching assistance method and teaching assistance system using said method | |
| CN108875606A (en) | A kind of classroom teaching appraisal method and system based on Expression Recognition | |
| CN108399376A (en) | Student classroom learning interest intelligent analysis method and system | |
| WO2023131361A2 (en) | Big-data-based teaching planning method and system | |
| CN115984956B (en) | Multi-mode visual analysis system for class investment of students through man-machine cooperation | |
| CN112069970B (en) | Classroom teaching event analysis method and device | |
| CN110175534A (en) | Teaching assisting system based on multitask concatenated convolutional neural network | |
| CN113139481B (en) | Classroom people counting method based on yolov3 | |
| CN115907507B (en) | Student class behavior detection and learning analysis method combined with class scene | |
| CN117523627A (en) | A video-based classroom cognitive emotion recognition and evolution analysis method and system | |
| CN110414415A (en) | Human behavior recognition method for classroom scenes | |
| CN117933787A (en) | Teacher teaching ability evaluation method and system aiming at course heavy difficulty knowledge | |
| CN111611854B (en) | A Classroom Situation Evaluation Method Based on Pattern Recognition | |
| CN118781660A (en) | A teacher behavior recognition method based on improved spatiotemporal graph convolutional network | |
| CN119963382A (en) | Real-time dynamic learning data monitoring and visualization platform and method | |
| Yang et al. | Student in-class behaviors detection and analysis system based on CBAM-YOLOv5 | |
| CN118247841A (en) | Classroom concentration assessment method and device integrating behavior analysis and emotion analysis | |
| Liu | AI proctoring for offline examinations with 2-Longitudinal-Stream Convolutional Neural Networks | |
| Wang et al. | SLBDetection-Net: Towards closed-set and open-set student learning behavior detection in smart classroom of K-12 education | |
| CN120278604A (en) | Classroom content analysis method, system, equipment and storage medium based on AI | |
| He et al. | Analysis of concentration in English education learning based on CNN model | |
| Shen et al. | Behavior Recognition of Teachers and Students in the Smart Classroom Based on Deep Learning | |
| Su et al. | Recognition and identification of college students\classroom behaviors through deep learning | |
| Trabelsi et al. | Behavioral-based real-time cheating detection in academic exams using deep learning techniques |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication | Application publication date:20180525 |