Intelligent dictation system based on dot matrix pen and method thereofTechnical Field
The invention belongs to the technical field of intelligent dictation, and particularly relates to an intelligent dictation system based on a dot matrix pen and a method thereof.
Background
Dictation is an efficient means of memory. Traditional dictation requires at least two roles, one read and one write. The role of the read is also responsible for the examination. This requires a corresponding level of knowledge of the role to enable correct reading and checking. The traditional dictation process has two disadvantages, one is that too many people are occupied, and the other is that the requirement on the character knowledge level is too high. The situation is different for each family, it is difficult to ensure that each parent has time to dictation for the child, and it is more difficult to ensure that each parent has sufficient knowledge level to complete the correct dictation and examination. Although many APPs also have the function of dictating words, the keyboard type dictation mode hardly enables children to form writing memory.
At present, handwriting can be acquired in various ways, for example, infrared dot matrix recognition is to print an invisible dot matrix pattern on ordinary paper, and a high-speed camera at the front end of a digital pen captures the motion trail of the handwriting at any time. Meanwhile, the pressure sensor transmits pressure data back to the data processor, and finally, the information is transmitted to the mobile phone or the tablet computer through the Bluetooth or the USB cable, and the mobile phone or the tablet computer synchronously draws handwriting.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention effectively combines the intelligent writing pen with the image recognition function to obtain more efficient dictation experience, and provides an intelligent dictation system based on a dot matrix pen and a method thereof.
The invention provides an intelligent dictation system based on a dot matrix pen, which comprises the dot matrix pen, a client and a cloud, wherein the dot matrix pen is in communication connection with the client through a built-in Bluetooth communication module, and the client is connected with the cloud through a built-in cloud interface; the dot matrix pen includes the shell and installs the pressure sensor that is close to the nib in the shell, follow the high-speed camera that the sensor is connected and all be connected with pressure sensor and high-speed camera with pressure, the treater is connected with the storage module electricity of dot matrix pen to the information storage after will handling transmits for the customer end by the bluetooth communication module of dot matrix pen in the storage module of dot matrix pen, and the well lower part of shell is equipped with holds the groove, hold the sunken direction in groove and when writing the staff holds the finger when pen and hold a posture phase-match.
Further, the processor is used for processing pressure data and movement speed acquired by the pressure sensor, dot matrix coordinates and handwriting sequence acquired by the high-speed camera, and storing acquired information in the memory.
Further, the client is a client APP or a client applet and comprises a graphical user interface, a Bluetooth module and a cloud interface, the graphical user interface is used for displaying an operation user interface when the dot matrix pen writes, and the Bluetooth module is matched with a Bluetooth communication module of the dot matrix pen.
Further, the cloud comprises a database module for storing handwriting data collected in the client and audio input data in the system and an algorithm module for recognizing and processing notes.
Furthermore, the database module comprises a word data submodule, an audio data submodule, a handwriting data submodule and a dictation historical data submodule, and the algorithm module comprises an internal segmentation algorithm submodule and a Chinese character recognition algorithm submodule.
The invention also discloses an intelligent dictation method based on the dot matrix pen, which comprises the following steps:
(1) the dot matrix pen is connected with an intelligent terminal Bluetooth of the client;
(2) when a user selects a class at a client, after confirmation, sequentially playing audio corresponding to words;
(3) writing on dot matrix paper by using a dot matrix pen, triggering a pressure sensor at a pen point, starting a built-in high-speed camera, collecting pen point information, and transmitting information such as dot matrix coordinates, handwriting sequence, pressure data, movement speed and the like to a built-in processor;
(4) after writing, clicking a confirmation button, transmitting the information stored in the dot matrix pen to a client through a Bluetooth module, processing the information acquired in the step (3) into a bitmap document by the client, and uploading the bitmap document to the cloud;
(5) the algorithm module in the cloud carries out recognition statistics according to an internal separation algorithm and a Chinese character recognition algorithm, the database module stores data, the data are displayed in different colors at the client according to different dictation states of learning contents, and states of all learning contents or knowledge points of a user are presented in a user interface in a graphic mode.
Further, the internal partitioning algorithm in the step (5) is a word or phrase internal partitioning algorithm, and the specific steps of the algorithm are
51. Collecting handwritten letter or Chinese character data set, and counting character width including minimum character width DminMaximum character width DmaxAnd average character width Dmean;
52. Binarizing the bitmap file in the step 51, and removing interference noise;
53. segmenting the character; detecting black pixel points on the picture to obtain a character block, starting to execute a detection process at another black point outside the character block until all the black points on the picture are detected, and determining blocks of a plurality of characters;
54. for sticky character blocks: examining the width L of each block, if L>DmaxThen it is considered as a sticky word block.
Further, the Chinese character recognition algorithm in thestep 5 is a recognition algorithm of handwritten letters or handwritten Chinese characters, and the specific recognition step is that firstly, a handwritten letter or Chinese character data set is collected, the real meaning of each sample is marked, and the data are divided into a training set and a verification set; and then training the convolutional neural network by using the training set, testing the training effect by using the testing set, and continuously adjusting the network until the identification accuracy of the network to the verification set exceeds an acceptable threshold ratio.
Further, the dictation state in the step (5) is provided with: not dictating, dictating correctly, dictating incorrectly.
Further, in the step (5), each learning content or knowledge point is represented by a plurality of colored color blocks, green represents mastered, red represents not mastered, and gray represents not contacted.
Compared with the prior art, the invention has the advantages and the technical effects that:
1. the dictation process is a real writing experience, and a keyboard is not knocked, so that writing memory is more easily formed for a user;
2. automatically identifying foreign words, Chinese characters, idioms, ancient poems and the like written by a user, and rapidly giving dictation result judgment;
3. and memorizing and storing the dictation result of the user to form a memory situation map of the learning content, so as to help the user to manage the learning content and improve the acquisition rate of the learning content for the user. The method and the system can effectively improve the writing and memorizing efficiency of the learning content of the user.
Drawings
Fig. 1 is a flow chart of an implementation of the intelligent dictation method of the present invention.
Fig. 2 is a schematic diagram of an internal structure of the smart dot-matrix pen according to the present invention.
FIG. 3 is a schematic structural diagram of a casing of the dot matrix pen according to the present invention.
FIG. 4 is a diagram of a Chinese character recognition process of the recognition algorithm module of the present invention.
FIG. 5 is a diagram of a word recognition process of the recognition algorithm module of the present invention.
Fig. 6 is an overall framework diagram of the intelligent dictation system of the present invention.
In the figure: 1. a pressure sensor; 2. a high-speed camera; 3. a processor; 4. a battery; 5. a memory; 6. a Bluetooth communication module; 7. a housing; 71. and (4) holding the groove.
Detailed Description
The technical solution of the present invention will be described in further detail with reference to the accompanying drawings and the detailed description.
Example 1
The utility model provides an intelligence dictation system based on dot matrix pen, it includes dot matrix pen, customer end and high in the clouds, the dot matrix pen includesshell 7 and installspressure sensor 1 that is close to the nib in theshell 7, follow the high-speed camera that the sensor is connected and all be connected withpressure sensor 1 and high-speed camera withpressure processor 3, and the well lower part ofshell 7 is equipped with holdsgroove 71, and the finger when holding pen with the people hand when writing holds pen holds a posture phase-match, guides the user to possess the correct pen of holding, writes the posture (fig. 3). Theprocessor 3 is electrically connected with a storage module of the dot matrix pen, the processed information is stored in the storage module of the dot matrix pen and is transmitted to a client by aBluetooth communication module 6 of the dot matrix pen, (fig. 2 and 3) theprocessor 3 is used for processing pressure data and movement speed acquired by thepressure sensor 1, dot matrix coordinates and handwriting sequence acquired by the high-speed camera, and storing the acquired information in thememory 5. The client is a client APP or a client applet and comprises a graphical user interface, a Bluetooth module and a cloud interface, the cloud comprises a database and an algorithm module, the database stores words and corresponding audio, handwritten letter picture data and dictation history data, the algorithm module comprises a word/phrase internal segmentation algorithm and a handwritten letter/Chinese character recognition algorithm, and the frame relationship among the modules is specifically shown in FIG. 6.
The method comprises the following specific steps of acquiring and processing the handwriting information of the dot matrix pen: when the pen point is pressed down, thepressure sensor 1 is triggered, a built-in high-speed camera is started, the dot matrix passed by the pen point is photographed at a high speed (for example, 100 times per second), information such as dot matrix coordinates, handwriting sequence, pressure data and movement speed is transmitted to the built-inprocessor 3, and finally, the information is output outwards through the Bluetooth unit. The dot matrix pen may also temporarily store the data in thememory 5 and output it to the outside when necessary. The information is transmitted to the client and processed into a bitmap document.
The specific steps of the word/phrase internal segmentation algorithm (fig. 4 and 5) in the algorithm module in the cloud are as follows:
51. first, a handwritten letter or Chinese character data set is collected. Counting character widths, including minimum character width DminMaximum character width DmaxAndaverage character width Dmean。
52. And then carrying out binarization on the bitmap file, and removing interference noise points.
53. The character is then segmented. The process is as follows: detecting black pixel points on the picture, after detecting one black pixel point, continuously detecting the black point in eight directions nearby the black pixel point by taking the black pixel point as a center, and continuously detecting the black point in the direction nearby the black pixel point every time one black point is detected, wherein the process is continuously repeated until no new black point can be detected. After this process is completed, a block of characters is ascertained. Starting with another black dot outside the block, the above process is continued until all black dots on the picture are detected, thus determining blocks of several characters.
54. For sticky character blocks: examining the width L of each block, if L>DmaxThen, the character is considered as a sticky character block, and the number of characters, num, is round (L/D)mean) Round means rounding. Then the number of segmentation points is num-1. And (5) vertically projecting the sticky character block, and counting the number of the minimum points of the projection as s. Find num of the s minimum points as segmentation points. For p ═ 0, ypIs the starting point of the tile, for p ═ 1, 2, …, num-1, ypIs the last point to satisfy the following equation. The specific formula is as follows:
the specific steps of the handwritten letter/Chinese character recognition algorithm (fig. 4 and 5) are as follows: firstly, a handwritten letter or Chinese character data set is collected, and the real meaning of each sample is marked. These data are divided into training and validation sets. Then training the convolutional neural network by using the training set, testing the training effect by using the testing set, and continuously adjusting the network until the identification accuracy of the network on the verification set exceeds a certain acceptable threshold value (for example, 95%). And inputting the new initial letter picture into the trained convolutional neural network, and outputting a prediction result by the convolutional neural network.
The specific method for collecting and distinguishing dictation effects by using the dot matrix pen and the client app comprises the following steps:
(1) the lattice pen is connected with an intelligent terminal for installing the client app through Bluetooth;
(2) when a user selects a class at a client, after confirmation, sequentially playing audio corresponding to words;
(3) writing on dot matrix paper by using a dot matrix pen, triggering apressure sensor 1 at a pen point, starting a built-in high-speed camera, collecting pen point information, and transmitting information such as dot matrix coordinates, handwriting sequence, pressure data, movement speed and the like to a built-inprocessor 3;
(4) after writing, clicking a confirmation button, transmitting the information stored in the dot matrix pen to a client through a Bluetooth module, processing the information acquired in the step (3) into a bitmap document by the client, and uploading the bitmap document to the cloud;
(5) the algorithm module in the cloud carries out recognition statistics according to an internal separation algorithm and a Chinese character recognition algorithm, the database module stores data, the data are displayed in different colors at the client according to different dictation states of learning contents, and states of all learning contents or knowledge points of a user are presented in a user interface in a graphic mode.
The whole dictation process is writing in dot matrix paper, so that real writing experience is achieved, and a user can more easily form writing memory instead of knocking a keyboard; and foreign words, Chinese characters, idioms, ancient poems, etc. written by the user are automatically identified, and dictation result judgment is rapidly given; the dictation result of the user is memorized and stored to form a memory situation map of the learning content, so that the user is helped to manage the learning content, the acquisition rate of the learning content by the user is improved, and the writing and memorizing efficiency of the learning content of the user is effectively improved.
Example 2
Different fromembodiment 1, the method for using the intelligent system based on the dot matrix pen in the embodiment is as shown in fig. 1, and includes the following steps:
(1) user installs client APP or client applet on intelligent terminal (such as mobile phone, PAD, etc.)
(2) The Bluetooth module is opened by the dot matrix pen, the Bluetooth module is searched by the mobile phone end or the pad end, and Bluetooth connection is established with the intelligent terminal
(3) When a user selects a class at a client, words and audio corresponding to the class are extracted from a database at the cloud end
(4) And the client sequentially plays the audio corresponding to the words in sequence to wait for the user to write the words.
(5) The user writes words or Chinese characters on the dot matrix paper by using the dot matrix pen, and clicks the 'confirm' button after writing.
(6) And the user program acquires the dot matrix pen handwriting, converts the dot matrix pen handwriting into a graphic format and uploads the graphic format to the cloud.
(7) The written content picture is input into a handwriting recognition model, and the handwriting recognition module is divided into two modules, namely an internal segmentation algorithm module and a handwritten letter/Chinese character recognition algorithm module. The internal segmentation algorithm module firstly segments each letter in the word, and then the handwritten letter/Chinese character recognition algorithm module respectively recognizes each letter.
(8) And comparing the letter recognition result with the letters of the correct word in sequence, and if all the words are the same, feeding back a result of 'dictating correct'. If there is an error, the model feeds back the result of the "dictation error" error. And the results are displayed at the client.
(9) The learning content in the database has different states: not dictating, dictating correctly, dictating incorrectly. Different dictation states are shown in different colors at the client.
(10) The state of the user's entire learning content (knowledge points) is graphically presented in the user interface. Each learning content (knowledge point) is represented by a plurality of colored color blocks, for example, green represents mastered, red represents not mastered, and gray represents not contacted, so that the user can perform visualization and data management on the target learning content.
The above description is only an example of the present invention, and is not intended to limit the present invention in any way, and those skilled in the art can make many variations and modifications of the present invention without departing from the scope of the present invention by using the method disclosed above, and the present invention is covered by the claims.