Disclosure of Invention
Aiming at the problems, the invention provides a learning and guiding system based on remote on-screen communication, which aims at realizing the problem of providing real-time guidance for students in a learning scene.
In a first aspect, the invention provides a learning and guiding system based on remote on-screen communication, which comprises a learning end and a guiding end, wherein the learning end comprises a learning platform, a camera and a first communication terminal.
The learning table is a table top for learning and is used for placing learning articles and the like;
the camera is used for shooting a photograph or video of the learning platform to generate first content, and sending the first content to a first communication terminal of the learning end or a second communication terminal of the coaching end; optionally, the camera starts shooting in response to a learning coaching start event triggered by a student or teacher;
the first communication terminal is, for example, a computer, a PAD and the like placed on the learning table, and is used for displaying the first content, carrying out remote on-screen communication with the second communication terminal of the coaching end, initiating an on-screen communication request, receiving the on-screen communication request sent by the second communication terminal, and executing consent or rejection processing on the request.
The tutoring terminal comprises at least a second communication terminal for performing learning tutoring, wherein the second communication terminal, such as a computer, a PAD, a mobile phone, etc., is used for displaying the first content and performing remote on-screen communication with the first communication terminal.
In the system, the remote on-screen communication comprises, but is not limited to, synchronous display operation communication content, synchronous mark and graffiti, audio and video communication and the like, and the first communication terminal and the second communication terminal comprise an on-screen module, an audio and video communication module and a synchronous graffiti module, wherein the on-screen module is used for realizing display and operation synchronization between the first terminal and the second terminal; the synchronous graffiti module is used for realizing synchronous graffiti and marking between the first alternating current terminal and the second alternating current terminal, and comprises the steps of collecting graffiti data on one side, sending the graffiti data to the other side, receiving and displaying the graffiti data on the other side; the audio-video module is used for realizing audio-video dialogue communication between students and teachers.
According to the system, learning and coaching can be performed by using the system, learning content of students can be obtained, and based on the content, the teacher and the students are provided with same-screen communication, and audio and video communication, display pictures and operation of the two parties are kept synchronous, so that the system has a coaching effect of 'face to face'. The application can produce the following technical effects: remote guidance for offline learning content is realized, and the method has a face-to-face guidance effect.
Optionally, the system of the present application further includes a service end, where the service end runs a co-screen communication service, and the service provides remote co-screen communication service for the tutor end and the learning end, and the functions of the system include: receiving first content sent by a learning end; providing the first content for a first communication terminal and a second communication terminal; the method comprises the steps of providing remote on-screen communication services for a first communication terminal and a second communication terminal, wherein the remote on-screen communication services comprise, but are not limited to, the functions of creating a communication room, generating and processing on-screen communication content, receiving and sending synchronous data, remote communication data communication, remote communication process control, remote communication content recording, providing security measures, providing an API interface and the like. In this implementation, the server is responsible for managing and controlling client connections, ensuring synchronization and coordination between clients, and is also responsible for handling all data transmission and processing tasks, including receiving and sending data, parsing data, executing service logic, etc., and in addition, the server may provide some auxiliary functions, such as security authentication, data backup and recovery, etc. In a word, the server is a core component of the same-screen system and is responsible for coordinating and managing the operation of the whole system.
Optionally, the learning terminal further includes a tablet device, where the tablet device generally includes a tablet and an electromagnetic pen, and when the learning terminal is implemented, the tablet is placed on the learning table and below the learning material, so that the student uses the electromagnetic pen to perform operations such as drawing, labeling, handwriting, and the like on the learning material, and the tablet device can collect input data, and use the input data as first synchronous doodling data, and the first synchronous doodling data and the second synchronous doodling data are synchronously displayed in the first communication terminal and the second communication terminal for communication between a teacher and the student. The implementation mode provides a mode for the student to communicate with the teacher based on the actual scene, for example, the student only needs to carry out position marking on the test paper, and the teacher can acquire the student marked information from the second communication terminal.
According to the above optional implementation manner, the student performs a labeling operation in an actual learning scene, displays learning scene pictures in the first communication terminal and the second terminal, and simultaneously draws labeling information in the scene pictures, and preferably, the first communication terminal further comprises a graffiti position correction module for correcting consistency of an input position of the electromagnetic pen on the learning material and a graffiti position in the first content displayed in the first communication terminal, so as to ensure correct display of the label. Optionally, the graffiti position calibration module receives a calibration request to start, displays a graffiti calibration reference picture on the first ac terminal, wherein the graffiti calibration reference picture comprises the first content and a calibration input requirement, a student triggers an electromagnetic pen input event according to the first content and the calibration input requirement at a position appointed by the input requirement in an actual learning scene, and the graffiti position calibration module acquires input data of the digital panel to generate calibration data and sends the calibration data; the server side calculates and generates a proofreading result based on the proofreading data, wherein the proofreading result comprises adjustment parameters, and the adjustment parameters are used for adjusting the display of the first synchronous graffiti data so as to keep the drawing of the first synchronous graffiti data on the first content consistent with the drawing of the first synchronous graffiti data in an actual scene.
Optionally, the learning end further includes an update key, where the update key is used to trigger an update event, where the update event is used to update the first content, that is, call the camera to re-shoot the picture or video of the learning platform to generate a new first content, and send the new first content to the learning end and the coaching end, so as to update the content that the first communication terminal and the second communication terminal are exchanging with the screen.
Optionally, the learning terminal further includes an image monitoring module, the module monitors changes of the learning platform pictures or video contents shot by the camera, when the changes are monitored, the updating event of the first content is triggered, the camera is called to shoot the learning platform pictures or video again to generate the first content, and the first content is sent to the first communication terminal and the second communication terminal to be updated. The implementation mode realizes the automatic update of the on-screen communication content so as to enable the target content of the on-screen communication to be automatically consistent with the learning desk content.
Optionally, the learning end further includes a first sensing device, where the first sensing device may obtain an online identifier of the student, and send the online identifier to the second communication terminal, so that a teacher may learn about online information of the student.
Optionally, the second communication terminal further includes a content management module, which provides a function of replacing communication content for the teacher, for example, replacing the first content with the second content or switching the second content with the first content, where the second content is content specified by the teacher, for example, teaching materials prepared by the teacher, learning materials from the internet, or the like.
Optionally, the server further includes a first recording module, where the first recording module records on-screen communication process data, and the recorded data is used for playing back the on-screen communication process. Optionally, the first communication terminal or the second communication terminal includes a second recording module, where the second recording module records the display content of the first communication terminal or the second communication terminal to generate a remote on-screen communication process video, or the second recording module records one or more of the first content, the synchronous data, the synchronous graffiti data, and the audio-video communication data, and records the corresponding time information to generate remote on-screen communication process record data.
Optionally, the first communication terminal further includes a playback module, where the playback module is configured to obtain communication process record data, and perform playback of a remote on-screen communication process, play a video when the communication process data is a video, load content data of the record on-screen communication when the communication process data is the communication process record data, and correspondingly restore the communication process operation according to time information of occurrence of the recorded operation data, so as to obtain a playback effect consistent with the record.
Optionally, the system further includes a projection module, and the learning end further includes a projection device, where the projection device is configured to project, on the learning platform, graffiti data or specified image data, where the graffiti data is graffiti data generated at the first ac terminal or the second ac terminal, and the projection module processes the graffiti data into a first image that displays a graffiti track with a bright color, and sends the first image to the projection device for projection. Through the implementation mode, a teacher in the second communication terminal can directly project the mark information of the first content in an actual scene at the learning end, so that the mind of students in the communication process can be concentrated on learning materials.
Optionally, the projection module starts a projection calibration process based on a projection calibration request, a projection calibration reference picture is displayed on the first ac terminal, the projection calibration reference picture includes the first content and the calibration projection requirement, and the student adjusts the projection device according to the projection effect specified by the projection requirement in the actual learning scene according to the first content and the calibration projection requirement, so that the actual projection effect is consistent with the projection calibration reference picture.
By the proposal provided by the embodiment, the remote guidance of the teacher to the students is realized, and the guidance content is not limited, the student can conduct coaching on offline content of study, and has face-to-face coaching effect.
In a second aspect, the present application also provides a learning desk with a learning and coaching function for remote on-screen communication, wherein the learning desk includes:
a learning table for placing learning materials;
the camera is used for shooting a learning material image on the learning table, and the image photo or video generates first content;
the first communication terminal is used for displaying the first content and carrying out remote on-screen communication with the coaching terminal;
optionally, the learning desk further comprises:
the tablet device comprises a tablet and an electromagnetic pen, wherein the tablet is arranged on the learning table and is arranged below the learning data, the electromagnetic pen is used for executing drawing, marking and handwriting operations on the learning data, the tablet device is used for collecting tablet input data and is used as first synchronous graffiti data, and the first synchronous graffiti data is used for being synchronously displayed in the first communication terminal and the second communication terminal for communication between a teacher and students.
Optionally, the learning desk further comprises:
the projection device is used for projecting the graffiti data generated at the first communication terminal or the second communication terminal and projecting the graffiti data on the learning platform.
In summary, the learning and coaching system based on remote on-screen communication provided by the application realizes remote coaching of students in learning content, has a face-to-face coaching effect, does not need excessive preparation work input by students in coaching process, is quick to start, is simple and convenient to operate, has strong instantaneity and high efficiency, is interesting, and supports a training mode of combining on-line data with off-line data.
Detailed Description
The description of the embodiments of the application is intended to be illustrative of the application and is not intended to be limiting.
Embodiments of the present application will now be described with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all embodiments of the present application. As one of ordinary skill in the art can know, with the development of technology and the appearance of new scenes, the technical scheme provided by the embodiment of the application is also applicable to similar technical problems.
Referring to fig. 1, a schematic diagram of a learning and guiding system based on remote on-screen communication is provided in an embodiment of the present application, where the system includes a learning end 10 and a guiding end 20, and the learning end 10 is used by students and the guiding end 20 is used by teachers. In the learning terminal 10, the learning station 101, the camera 102, and the first communication terminal 103 are included, wherein the learning station 101 provides a smooth working surface for students, so that the students can place books, notebook computers, other learning articles, etc. thereon, provide enough space to organize and store the learning articles, such as books, folders, and stationery, or it can be understood that the learning station 101 provides a special learning area for students. The camera 102 has a main function of shooting a learning scene picture on a learning table, such as a textbook, a test paper, learning materials and the like which are learned by students, and the shot generated content comprises pictures, videos and the like. The first communication terminal 103 provides a remote on-screen communication function for students, namely, performs remote on-screen communication with the second communication terminal 201 of the coaching terminal, and specifically includes the following aspects:
A. Audio and video communication with a teacher is carried out, audio or video information of students is collected from the first communication terminal 103 and sent to the second communication terminal 201, and the audio or video information from the second communication terminal 201 is received and played, so that the face-to-face communication effect of voice conversations and video conversations of both parties is achieved;
B. the remote on-screen communication with the teacher is carried out, wherein the on-screen communication is synchronous communication, namely, in the communication process, the display pictures and the contents, the operation and the like of the first communication terminal 103 and the second communication terminal 201 are kept consistent, and the on-screen communication effect is achieved;
C. the remote on-screen communication with the teacher also comprises a synchronous graffiti and marking function, namely, the first communication terminal 103 collects the graffiti data of the student, sends the graffiti data to the second communication terminal 201, receives the graffiti data from the second communication terminal 201 and draws the graffiti data so as to realize the synchronous graffiti effect.
It should be noted that, implementing the synchronization between the first ac terminal 103 and the second ac terminal 201 includes various implementations, such as video streaming, application synchronization, and web page synchronization. The simple introduction is as follows:
the remote synchronization is realized through video push, which generally refers to collecting an operation picture at one end of the first ac terminal 103, the second ac terminal 201, etc. to generate a video, and then pushing the video to the other end in real time through the internet, so that the other end can watch the same operation picture in real time, thereby realizing the effect of remote synchronization.
The method adopts a video push flow mode to realize remote same-screen communication, and comprises the following specific steps:
collecting video signals: and recording or collecting a running picture at one side to generate a video signal.
Video coding: the acquired video signals are encoded and compressed into smaller data packets for transmission over a network.
Video push stream: pushing the encoded video data to a remote server through a network. The push flow can be performed by adopting RTMP, HLS and other protocols.
Video decoding: after receiving the push stream, the receiving end needs to decode the video and restore the video to the original video signal.
Video display: and outputting the decoded video signal on the other side, so that a user can watch a real-time running picture.
Through the steps, the remote synchronization function can be realized, so that users on two sides can watch the same picture in real time. It should be noted that this approach requires a high network bandwidth and a stable network connection to ensure smoothness and real-time video push and receive.
The implementation of obtaining a remote synchronization effect by using the synchronous running of a program generally refers to performing certain operations by the program in one place, and then synchronizing the operations to another place in real time, so that the program in the other place also performs the same operations to achieve the remote synchronization effect. The specific implementation steps can be divided into the following parts:
Programming: the same program is written both locally and remotely for performing the operations that require synchronization.
Network communication: and establishing connection through a network to realize communication between the local program and the remote program. The communication may be performed using protocols such as TCP, UDP, etc.
Operation synchronization: after the local program executes certain operations, the operation information is sent to the remote program through the network, and the remote program also executes the same operations after receiving the operation information.
Synchronous feedback: the local program and the remote program need to feed back the synchronous result in real time so as to find and process the synchronous problem in time.
Through the steps, the remote synchronous system function can be realized, the remote user can synchronize to the local operation in real time, and the synchronous problem can be timely processed through a feedback mechanism. It should be noted that such a system requires low network bandwidth, but requires a highly stable network connection to ensure real-time and accuracy of the operational synchronization.
In a specific implementation scenario, preferably, the above remote on-screen communication is implemented through a web page synchronization technology, specifically:
operating a target webpage on the first communication terminal and the second communication terminal, wherein the target webpage is a webpage with a webpage synchronization function;
The target webpage comprises content to be communicated with a screen;
operating a target webpage on the first communication terminal and the second communication terminal, and establishing communication connection between the two terminals, wherein a WebSocket communication mode is generally adopted;
and monitoring a user operation event, generating a webpage operation instruction by the user operation after monitoring the user operation on one side, and sending the webpage operation instruction to the other side, wherein the other side receives the webpage operation instruction and executes the webpage operation instruction.
Optionally, in the target web page, a monitoring event of the related web page content change is established, for example, monitoring by the following events:
domkubtreemodified: triggering when the subtrees of the DOM tree change;
the MutationObserver: the change of the DOM tree can be observed, and a callback function is executed when the change occurs;
domnodeInerted: triggering when a new child node is inserted into an element;
DOMNodeRemoved: triggering when a child node is removed from an element;
domnodeInertedIntodocument: triggering when a new element is inserted into the document;
domnoderemovedbrom document: triggering when an element is removed from the document.
DOMAttrModified: triggering when an element's properties are modified.
The events can be monitored through JavaScript, after the change of the webpage is monitored, the changed webpage data are collected and sent to the other side, so that the webpage data on the two sides are kept synchronous, the same display effect can be obtained according to the display principle of the webpage, and the remote synchronous effect is achieved.
In summary, the above implementation manner can implement the system function of remote synchronization, and the embodiment of the present application includes, but is not limited to, the implementation manner of the above remote synchronization system, which is characterized in that the learning scene of the student is obtained as the content of the remote on-screen communication.
In this embodiment, the content of the teacher and the student in the on-screen communication is the content obtained by the real-time shooting by the camera 102, so that the two parties can conduct remote communication coaching on the content. It can be seen that the embodiment of the application has the obvious technical characteristics that: the learning content of the students is obtained and converted into remote on-screen communication content. The remote tutoring method has the obvious beneficial effect that remote tutoring of the student offline in learning content is realized.
In one possible implementation manner, the learning and coaching system based on remote on-screen communication as shown in fig. 2 is a schematic diagram, the system includes a service end 30, the service end 30 includes a remote on-screen communication service 301, the coaching end 20 and the learning end 10 are respectively connected with the service end 30 as clients, and functions of the service end 30 include one or more of the following:
processing the connection requests of the coaching terminal 20 and the learning terminal 10: the server is in charge of receiving a connection request of the client, establishing connection and maintaining a connection state;
management session: the server 30 is responsible for establishing a session between clients and managing session states so as to ensure that on-screen communication between clients is performed smoothly;
forwarding data: the server 30 is responsible for receiving data sent by a client and forwarding the data to other clients. In the on-screen communication system, the server 30 needs to ensure the real-time performance and stability of the data, so that the client can acquire the content on the same screen in time.
Control authority: the server 30 needs to control the rights of the coaching terminal 20 and the learning terminal 10, so as to ensure that only authorized users can perform on-screen communication, and prevent the damage to the system caused by malicious users.
Monitoring the state of the system: the server 30 also includes a status of the monitoring system, including a load of the server, a connection status of the client, etc., so as to discover and solve problems in the system in time.
In light of the above, the server 30 plays a vital role in the remote on-screen communication system, which is the core of the whole system, responsible for coordinating and managing interactions between clients, and ensuring the stability and security of the system.
Optionally, the function of the server 30 further includes receiving the first content sent by the camera 102, generating on-screen content for remote on-screen communication at the server, for example, after receiving a picture taken from a learning desk, taking the picture as the first content, adding the first content to a target web page for on-screen communication, and so on. Optionally, the server 30 further includes a processing function for the first content, for example, after receiving the first content picture, performing transformation, cropping, contrast, saturation processing on the picture, and performing compression processing on the picture or video. For example, the camera is not located right above the learning table, and the learning scene picture shot by the camera is generally a trapezoidal image, but a rectangular image is required to be displayed in the ac terminal, so that the first content picture needs to be subjected to conversion processing, and optionally, the camera is set to realize trapezoidal correction.
In one possible implementation, the learning terminal 10 further includes a digitizer 104, which may be understood as an electronic device, also called a digitizer device, that allows a user to write, draw, mark, etc. on its surface using a dedicated digitizer pen. The digitizer tablet works by sensing the pen position and pressure to capture the user's input, then converting it into a digital signal, and transmitting it to a computer or other device. The tablet 104 generally includes a tablet 1041 and an electromagnetic pen 1042, in a specific implementation scenario, the tablet 1041 is disposed on the learning table 101 and disposed below learning materials, the student operates the electromagnetic pen 1042 to perform drawing, labeling, and handwriting operations on the learning materials, the tablet 104 collects such input data and sends the same as first synchronous graffiti data, and the first synchronous graffiti data is used for being synchronously displayed in the first communication terminal and the second communication terminal for a teacher to communicate with the student. Through the implementation mode, the students can directly conduct graffiti and marking in own learning scenes, conduct remote communication with the coaching teacher, the graffiti content is displayed in communication terminals of the two parties, for example, after the students mark a question in a test paper, the marked content can be directly seen in the Pad of the first communication terminal. Optionally, the tablet device 104 is connected to the first ac terminal 103 device, and a program for receiving the first synchronous graffiti data is installed on the first ac terminal 103 device, and after the first synchronous graffiti data is received, the first synchronous graffiti data is sent to the server 30, where in this implementation manner, the data collected by the tablet is usually two-dimensional coordinate data, that is, an abscissa and an ordinate, and when a digital pen is used for drawing, writing or other operations on the tablet, the tablet detects the position of the digital pen through an inductor and converts the position of the digital pen into two-dimensional coordinate data, so that the transmitted first synchronous graffiti data has the advantages of small data and high speed.
Optionally, the first ac terminal 103 displays the cursor information of the electromagnetic pen 1042, and the second ac terminal 201 displays the cursor information of the electromagnetic pen 1042.
It should be noted that, according to the first synchronous graffiti data collected by the tablet device 104, which is two-dimensional coordinate data collected based on an actual learning scene, optionally, the student directly performs the graffiti operation on the first ac terminal 103, and generates second synchronous graffiti data for sending, where the second synchronous graffiti data can also implement the synchronous graffiti function.
According to the possible implementation manner, the student performs the graffiti drawing operation on the tablet 104, and the final graffiti result is presented in the first communication terminal and the second communication terminal and is presented on the content photographed by the camera 102, so that it is seen that the first content is drawn according to the two-dimensional coordinate data included in the first synchronization data, and the dislocation phenomenon is unavoidable. In order to obtain a position consistent with that in the actual scene, the position of the graffiti display must be checked to obtain a corresponding transformation parameter, so, optionally, as shown in fig. 4, a graffiti position checking module 1031 is further included in the first ac terminal 103, where the graffiti position checking module 1031 checks the position of the graffiti display, obtains a corresponding transformation parameter after checking, and transforms the coordinate data received subsequently to make correct display according to the transformation parameter, so as to obtain a graffiti effect consistent with the actual scene. Alternatively, the graffiti location collation module 1031 may be activated manually or passively to perform collation.
In one possible implementation manner, as shown in fig. 5, the learning terminal 10 further includes a learning scene update module 105, which is configured to update content of the on-screen communication in real time during the remote tutoring process, where the learning scene update module 105 functions include: receiving a learning scene update event triggered by a user, calculating and starting the learning scene update according to a preset judgment rule, calling the camera 102 to acquire new first content, processing the first content, and transmitting to update the first content.
According to the above possible implementation manner, optionally, as shown in fig. 6, the learning terminal 10 further includes an image monitoring module 106, where the image monitoring module 106 is configured to monitor a change occurring in the learning scene, for example, monitor that a student changes learning materials, perform a page turning action, write text, etc. on existing learning materials, and call the learning scene updating module 105 to acquire new first content and send to update on-screen communication content after monitoring the change. Optionally, the image monitoring module 106 uses computer vision technology to identify that the learning scene has changed. Optionally, the video listening module 106 uses image processing algorithms to analyze the images to identify that the learning scene has changed.
In a possible implementation manner, as shown in fig. 7, the learning terminal 10 further includes a first sensing device 107, where the first sensing device 107 is configured to obtain an online identifier of a student, and further configured to send the online identifier to the second ac terminal, so that a teacher may learn online information of the student, and in a specific implementation scenario, optionally, the first sensing device 107 is a pressure sensor, for example, a pressure sensor may be installed on a seat, where when the student sits on the seat, body weight may be applied to the sensor, so that the sensor outputs a signal. When the student leaves the seat, the sensor is no longer under pressure and the signal changes. By detecting a change in the sensor output signal, it can be determined whether the student is sitting or leaving. Alternatively, the first sensing device 107 is an infrared sensor, which may be mounted on a table, and the infrared sensor may emit an infrared beam and receive the reflected beam. When a student sits on the seat, the body blocks the transmission of the infrared light beam, resulting in a change in the intensity of light received by the sensor. When the student leaves the seat, the light beam can directly reach the sensor, and the light intensity can not change obviously. By detecting the change in the intensity of the light received by the sensor, it can be determined whether the student is sitting or leaving. Alternatively, the first sensing device 107 may be connected to the first ac terminal or other control system for reading and transmitting signals from the sensor to monitor the seating status of the student in real time. The first sensing device 107 may provide online or offline student information to a teacher remotely, provide support for enabling a remote tutoring connection, and may not be perceived by the student to avoid taking up learning time.
In one possible implementation, as shown in fig. 8, the second communication terminal 201 further includes a content management module 2011, where the content management module 2011 provides a teacher with a function related to managing on-screen communication content, for example, switches a first content in on-screen communication to a second content, where the first content is a scene that a student is currently learning, and the second content is other content specified by the teacher, except for the first content. Optionally, the content management module 2011 further includes a function of generating the second content by performing web searching or AI computing based on the first content, for example, when the content management module 2011 recognizes that a certain poem of a certain poem is included in the first content, it automatically performs internet searching and obtains a translation, an explanation, etc. of the relevant poem, and profile information of an author, or generates a web page for automatically reading the ancient poem. Preferably, the content management module 2011 has functions of remote on-screen communication content management and switching, and the first content and the second content can be converted into content for remote on-screen communication.
In one possible implementation, as shown in fig. 9, the server 30 further includes a first recording module 302, where the first recording module 302 may record the communication process data in a plurality of manners. Alternatively, the first recording module 302 may record the communication process by recording the screen content and audio of the participant. This may be accomplished by using specialized recording software or recording functionality that is self-contained in the video conferencing system. Optionally, the first recording module 302 may record events and operation logs during the communication, including participants joining/leaving meetings, sending messages, sharing screens, etc., which may be used for subsequent playback, statistics, and analysis. Optionally, the first recording module 302 may capture network packets during the communication process, which may include content of the communication between the participants, and may be used to analyze and reproduce the communication process. Alternatively, the first recording module 302 may store the communication process data in a database. Thus, the data management and inquiry can be conveniently carried out, and meanwhile, the long-term storage and backup of the data can be ensured. It should be noted that the first recording module 302 should comply with related privacy and data protection regulations when recording the communication process data, so as to ensure that personal information and sensitive data of the participants are reasonably protected.
Optionally, the first communication terminal or the second communication terminal includes a second recording module, where the second recording module records display content of the first communication terminal or the second communication terminal to generate remote on-screen communication process video, or the second recording module records one or more of the first content, the synchronous data, the synchronous graffiti data, and the audio/video communication data, and records corresponding time information to generate remote on-screen communication process record data.
In a possible implementation manner, as shown in fig. 10, the first communication terminal 103 further includes a playback module 1031, where the playback module 1031 functions include, receiving a playback request triggered by a user, obtaining the on-screen communication process record data, parsing a plurality of on-screen communication process record data and performing playback according to a type of the data, for example, the on-screen communication process record data is video, and the playback module 1031 includes related functions of a video player; the on-screen communication process record data is event and log data, and the playback module 1031 restores the communication process according to the time sequence according to the event information of the event and log contained in the on-screen communication process record data.
In one possible implementation, as shown in fig. 11, the learning terminal 10 further includes a projection module 108, where the projection module 108 includes at least a projection device, and the projection device is configured to project a graffiti image on the learning platform, and optionally, the projection device is connected to the first ac terminal 103, and the first ac terminal 103 receives graffiti data from the second ac terminal 201, and sends the graffiti data to the projection device to project the graffiti data on the learning platform 101, so that a student can directly obtain tuitous information from a teacher in a learning scenario. Alternatively, the projection device may be used to project images other than graffiti on a learning station. Optionally, the projection module 108 further includes a calibration function, preferably, the projection device projects the reference point image, and displays a reference point requirement projection position indication in the first ac terminal 103, and the student adjusts the direction, focal length, zoom size, and the like of the projection device so as to keep the projected target consistent with the requirement to complete calibration.
In combination with the learning and guidance system based on remote on-screen communication provided in the above embodiment, the embodiment of the present application further provides a learning and guidance method based on remote on-screen communication, please refer to fig. 12, which includes the following steps:
In the learning terminal 10:
s1201, receiving a user-triggered start-up coaching event, starting up a camera to take pictures, and generating first content;
s1202, sending first content;
in the server 30:
s1203, receiving first content, processing and generating content and data for communication with a screen;
s1204, starting the on-screen communication service;
s1205, S1206, transmitting synchronization data;
in the learning terminal 10:
s1206, entering the same-screen communication, receiving synchronous data, and displaying first content on a first communication terminal;
in the coaching terminal 20:
s1207, entering the same screen communication, receiving the synchronous data and displaying the first content on the second communication terminal;
in the above step, S1201 receives a user-triggered start tutoring event, optionally, a student-triggered start tutoring event, optionally, a teacher-triggered start tutoring event.
Optionally, at the learning end, the first communication terminal 103 device is a PAD, a notebook computer, or the like. Optionally, at the learning end, the camera is connected to the first ac terminal 103 device, and the first ac terminal 103 device is provided with a computer program, where the program includes a program for connecting and controlling the camera, and further includes a program for providing for students to perform on-screen communication, and the program is connected to the server 30.
Optionally, before step S1201, communication connection is established between the first ac terminal 103 of the learning end and the second ac terminal 201 of the coaching end and the service end 30, respectively, and the first ac terminal 103 and the second ac terminal 201 are connected to the remote ac service 301 and perform data communication, preferably, the data communication adopts a WebSocket long-connection communication mode. Optionally, according to this implementation manner, after the first ac terminal 103 and the second ac terminal 201 enter the remote on-screen communication, the teacher triggers the start tutoring event through the second ac terminal 201, and then step S1201 is executed, and the camera is started to take a picture through the first ac terminal 103.
In the above steps, the execution time of the step S1204 is not particularly limited, and for example, the on-screen communication may be started first, the first content may be acquired on the basis of establishing communication for the two terminals, or the first content may be photographed first and then the remote on-screen communication may be started based on the first content, which is executed according to the flow shown in fig. 12.
In the above steps, the first content includes a picture, a video, etc., and the picture is taken as an example, and the content is a learning scene picture shot by the camera 102, or can be understood as a learning material picture on the learning table. In the present application, the first content is defined to include a learning material screen content that a student is learning.
In the above step, S1203 receives the first content, processes and generates the content and data for on-screen communication, i.e. adds the received content to the on-screen communication content, for example, adds the first content to the on-screen communication web page. Optionally, after receiving the first content, S1203 processes the content, for example, processes rotation, scaling, contrast adjustment, color adjustment, and the like on the received picture. Optionally, in the first communication terminal 103 or the second communication terminal 201, a user program for manually adjusting the first content is provided, and further, after the user performs such adjustment, the step of recording adjustment is a batch instruction set, so that the first content shot later can be automatically processed according to the batch instruction set.
According to the above implementation manner, optionally, the tutor sends a learning content obtaining request, the camera shoots the learning platform picture or video to generate the first content, and the second communication terminal 201 obtains and displays the first content, so that the teacher knows the learning content of the student.
In one possible implementation manner, the tutor initiates a tutoring request, after the learner accepts the request, the learner performs S1201 and subsequent steps, or the learner initiates a tutoring request, after the tutor accepts the request, the tutor performs S1201 and subsequent steps. Optionally, the first ac terminal 103 and the second ac terminal 201 perform related operations that one party requests and the other party grants before entering a remote on-screen communication. Preferably, the first communication terminal 103 or the second communication terminal 201 generates on-screen communication confirmation information to confirm the learning guidance request sent by the other side, and enters on-screen communication after confirmation, otherwise, returns rejection information to the initiating terminal.
In a possible implementation manner, after the step S1204 is performed by the server 30 and the on-screen communication service is started, optionally, a remote on-screen communication process recording is started based on the configuration information of the on-screen communication service, where recording and communicating display content of the first communication terminal 103 or the second communication terminal 201 generates a remote on-screen communication process video, or recording one or more of the first content, the synchronization data, the synchronous graffiti data, and the audio/video communication data, and recording corresponding time information thereof to generate remote on-screen communication process record data. Optionally, the server 30 starts a remote on-screen communication process recording based on a recording request sent by the first communication terminal 103 or the second communication terminal 201. Optionally, the client recording is performed at the first ac terminal 103 or the second ac terminal 201 client, for example, the ac client is a web page program, and the steps of implementing the on-screen ac process recording may be summarized as the following steps:
1. monitoring user operation: the client needs to monitor the operation of the user on the webpage through front-end technologies such as JavaScript and the like, including clicking, inputting, scrolling and the like.
2. Capturing operation data: when the user performs an operation, the client captures relevant operation data such as a clicked element, an inputted content, a scroll position, and the like. These data may be obtained through JavaScript code and saved into a variable or data structure.
3. Storing data: the client may store the captured operational data via a browser-provided local storage mechanism, such as localStorage or indeeddb. This ensures that the data persists locally on the client and that the previous record can be restored even if the user opens again after closing the web page.
4. Transmitting data to a server: the client can send the captured operation data to the server for storage or further processing through a network request. The data may be sent using Ajax, fetch API, or the like techniques.
5. Data processing and analysis: after the server receives the data sent by the client, further data processing and analysis can be performed. This includes storing data, analyzing user behavior, generating reports, and the like.
It should be noted that recording web page data and the operation process requires explicit authorization of the user, so privacy protection and compliance issues need to be considered in the implementation. In addition, in order to improve user experience and reduce data transmission amount, operation data can be compressed, filtered or aggregated through reasonable strategies.
In one possible implementation, the remote on-screen communication process video or remote on-screen communication process record data is acquired and played back in response to an on-screen communication process playback event triggered by a student on the first communication terminal 103 or in response to an on-screen communication process playback event triggered by the teacher on the first communication terminal 103. And executing playback based on the remote on-screen communication process record data, and calling a corresponding playing program to execute playback according to the data type in the remote on-screen communication process record data, for example, playing video recorded in the remote on-screen communication process, preferably, the remote on-screen communication process record is webpage operation data, and playing back the webpage operation process by using the webpage operation data as follows:
playback of web page operation data: when a playback operation of a web page is required, a previously recorded sequence of operations is retrieved from the stored data. The operation data may be sequentially read using a loop or recursive manner, etc., and the user's operation may be simulated at an appropriate timing.
Simulating user operation: according to the recorded operation data, the operation behaviors of the user, such as clicking specific elements, scrolling pages, inputting texts and the like, are simulated. JavaScript can be used to dynamically modify the state of the web page element, triggering the corresponding event or operation.
Monitoring and controlling: during playback, the playback effect can be monitored, and whether the state of the webpage during playback is consistent with the state of the recorded webpage is compared, so that the playback accuracy is ensured. At the same time, control functions such as pause, fast forward, rewind, etc. may also be provided so that the user can freely control the playback process.
Through the steps, the webpage operation process can be played back by using the webpage operation data, and the operation behaviors of the user on the webpage in the remote communication process can be restored.
In one possible implementation, the system includes a digitizer 104 in the learning end 10, the digitizer 104 including a tablet and an electromagnetic pen, the tablet being placed on the learning table and under the learning material, as shown in fig. 3. Referring to the flow chart shown in fig. 13, the main steps are as follows:
in the learning terminal 10:
s1301, monitoring drawing, labeling and handwriting operations executed by an electromagnetic pen on the learning materials, collecting input data of a digital panel and generating first synchronous doodle data;
s1302, sending first synchronous graffiti data;
in the server 30:
s1303, receiving first synchronous doodling data, and generating data synchronous to a client;
S1304A, transmitting synchronization data;
S1305B, transmitting synchronous data;
in the learning terminal 10:
s1306, receiving synchronous data, and displaying first synchronous graffiti data on a first alternating current terminal;
in the coaching terminal 20:
and S1307, receiving the synchronous data, and displaying the first synchronous graffiti data on the first alternating-current terminal.
According to the above steps, the student uses the electromagnetic pen to operate in an actual learning scene, for example, the student uses the electromagnetic pen to make a mark on a test paper, and the teacher can see the mark information of the student in the second communication terminal 201, and at the same time, the mark is synchronously displayed in the first communication terminal 103. Generally, in the embodiment of the present application, if the learning end is not configured with a tablet device, the student may perform a graffiti operation through the first ac terminal 103, collect the graffiti operation data of the student in the first ac terminal 103, generate a second synchronous graffiti data to send, and according to the same principle, the server 30 sends the second synchronous graffiti data to the second ac terminal 201 after receiving the second synchronous graffiti data to achieve synchronization. Optionally, the second ac terminal 201 collects the graffiti operation data of the teacher, generates second synchronous graffiti data, and the server 30 sends the second synchronous graffiti data to the first ac terminal 103 after receiving the second synchronous graffiti data to achieve synchronization. Optionally, in step S1301, the method further includes monitoring a movement operation of the electromagnetic pen on the learning material, collecting movement coordinate data to generate first synchronous graffiti data, in step S1306, displaying cursor information of the electromagnetic pen on the first ac terminal 103, and in step S1307, displaying cursor information of the electromagnetic pen on the second ac terminal 201.
In a possible implementation manner, input information of a student is directly collected through a digital board in a learning scene, when the input position and the position displayed in the first content picture are inevitably misplaced when the input position is displayed in an ac terminal, therefore, after the first ac terminal 103 or the second ac terminal 201 finds the misplaced phenomenon, a correction request is sent to a server 30, the server 30 processes the request and notifies the first ac terminal 103 to start a correction program, please refer to a flow chart shown in fig. 14, and main steps include S1401, displaying the first content, i.e. displaying a learning scene picture, such as a test paper on a learning table, which requires the student to perform correction input, at the same time displaying a reference point of the relevant prompt information, S1402, the student performs input in an actual learning scene with reference to the required input reference point on the first content according to the prompt information, and sends the acquired input information to the server 30, wherein the server 30 generates an actual mapping relation between the first content according to the received input information and the actual mapping. For example, in the above step S1401, the picture of the test paper is displayed, and two reference points a and B are displayed thereon, so that the student is required to click on the corresponding positions of A, B points on the actual test paper in sequence, and we count the positions of the two points on the test paper as a 'and B', because the actual test paper and the picture of the test paper are in a similar relationship, according to the corresponding relationship between a 'and a, and between B' and B, the transformation parameters between the two can be obtained, and according to the transformation parameters, the corresponding positions of any position on the test paper on the picture of the test paper can be calculated and obtained.
In one possible implementation manner, as shown in the flow chart of fig. 15, specific steps include: s1501, monitoring the change of the first content, that is, monitoring the change of the content captured by the camera, may be implemented using a computer vision technology and an image processing algorithm, firstly, acquiring a captured image or acquiring a frame of image from a captured video, and applying an image processing and a computer vision algorithm to detect and track an object of interest, identify an action or change, etc., which may include technologies of object detection, motion detection, background modeling, etc., detecting the change therein by comparing two continuous images or two frames of images, and may be implemented by comparing pixel level differences, color changes, motion trajectories, etc. S1502, after confirming that the change meets a preset judging rule, collecting new first content and sending, wherein the preset judging rule is used for judging that the first content is changed and needs to be updated, and sending the first content for updating the content communicated by the two parties on the same screen. In S1503, the first ac terminal 103 and the second ac terminal 201 update the first content to be exchanged. According to the implementation purpose of the embodiment of the application, a remote coaching method for learning the learning content is provided for students and teachers, and the learning content of the students is certainly changed in a specific learning scene.
Optionally, the embodiment of the present application further provides a method for manually updating on-screen communication content by a student, in response to the first content update event triggered by the student in the first communication terminal 103, or in response to a designated key or a custom key triggered by the student on the electromagnetic pen or the digitizer, starting the first content update, including collecting a new first content transmission, where the first communication terminal 103 and the second communication terminal 201 update the first content in communication.
Optionally, the embodiment of the present application further provides a method for manually updating content in a same screen by a teacher, and the method for starting the first content update in response to the first content update event triggered by the teacher in the second communication terminal 201 includes sending a first content update instruction to a learning end, where the camera or the first communication terminal 103 receives the update instruction, where the camera collects the first content, sends the first content, and where the first communication terminal 103 and the second communication terminal 201 update the first content in communication.
Optionally, the embodiment of the present application further provides a method for a teacher to manually switch the content of the same screen communication, in response to a content switching request triggered by the teacher in the second communication terminal 201, to switch the first content of the communication between the first communication terminal 103 and the second communication terminal 201 to a designated second content, or to switch the second content of the communication between the first content and the second content, where the second content is a content designated by the teacher, for example, a Word file, a PPT file, a PDF file, a picture or a group of pictures, etc. uploaded by the teacher, or learning data from the internet, etc.
In one possible implementation manner, the learning terminal further includes a projection device, where the projection device is connected to the first ac terminal 103, the first ac terminal 103 sends synchronous graffiti data to the projection device, the projection device receives the synchronous graffiti data, and the projection device projects the synchronous graffiti data on the learning table, so that the teacher can directly present the mark information of the learning material in a learning scene of the student. Optionally, a dedicated projection device control program is installed at the learning end, the projection device control program is connected with the projection device to send projection content, and the projection device control program is connected with the first terminal or the server end to receive the synchronous graffiti data. Alternatively, the projection device may project the teacher-specified content. Alternatively, the projection device may be implemented instead of the display function of the first ac terminal 103.
According to the above implementation manner, the projection device projects the graffiti content sent by the teacher on the learning table, and the position of the projected graffiti information on the actual learning table may be misplaced with the mark position on the first content when the teacher performs graffiti, so in one possible implementation manner, the embodiment of the present application further includes a projection calibration module, which is used for calibrating to solve the above graffiti misplacement problem, and the method includes sending calibration data to the projection device, where the projection device receives the calibration data and projects one or more reference points on the learning table, sending a calibration reference picture to the first ac terminal 103, where the calibration reference picture includes the current first content and the required projection effect, where the projection effect requires that the specified reference point be projected at the specified position, and where the student refers to the calibration reference picture, adjusting the projection device, including but not limited to focal length adjustment, projection calibration, and the like, and simultaneously comparing the reference picture to obtain a consistent effect.
In addition, an embodiment of the present application further provides a learning table, as shown in fig. 16, where the learning table includes:
a learning table 101 for placing learning materials;
a camera 102 for photographing a learning material image on the learning table 101, the image photo or video generating a first content;
the first communication terminal 103 is configured to display the first content, and perform remote on-screen communication with the tutor;
optionally, the learning table further includes a tablet device 104, where the tablet device 104 includes a tablet 1041 and an electromagnetic pen 1042, the tablet 1041 is disposed on the learning table 101 and disposed below learning materials, the electromagnetic pen 1042 is configured to perform drawing, labeling, and handwriting operations on the learning materials, and the tablet device 104 is configured to collect tablet input data and serve as first synchronous graffiti data, where the first synchronous graffiti data is configured to be synchronously displayed in the first communication terminal and the second communication terminal for a teacher to communicate with students.
Optionally, the learning desk further includes a projection device 1081, where the projection device 1081 is configured to project the graffiti data generated at the first ac terminal or the second ac terminal, and project the graffiti data on the learning desk 101.
It should be noted that, in the above embodiment of the method of the present application, some steps may be replaced by similar steps, and some steps may be combined, decomposed, and sequentially implemented, so the above embodiment does not limit a learning guidance system based on remote on-screen communication.
In view of the above, the beneficial effects produced by the embodiments of the present application can be summarized as follows: the remote on-screen coaching provided by the embodiment of the application also supports the execution of on-screen coaching for students in learned contents, solves the problems of lack of face-to-face interaction, limited coaching contents by network data and the like in conventional network coaching, simultaneously retains the advantages of network training, such as flexibility and convenience, supports on-line and off-line learning resources, and has the face-to-face effect, supports coaching for off-line learning contents and provides great convenience for coaching learning.
The foregoing is merely a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or substitutions that may be easily contemplated by those skilled in the art within the technical scope of the embodiment of the present application should be covered by the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiments of the present application shall be subject to the protection scope of the claims.