技术领域technical field
本发明涉及计算机软件技术领域，尤其涉及一种软件界面文本信息的辅助浏览方法。The invention relates to the technical field of computer software, in particular to an auxiliary browsing method for text information on a software interface.
背景技术Background technique
随着计算机软硬件技术的飞速进步，当今社会已经进入了一个信息化的时代，计算机、因特网、移动通讯已经成为人们日常生活、学习和工作的重要组成部分。在信息社会中，通过计算机获取信息和交流信息已经成为人们工作和生活的基本手段之一，因而信息平等已经成为信息社会中人人平等的基础。所谓信息平等就是任何人在任何情况下都能平等、方便、无障碍地获取信息、交流信息和利用信息。With the rapid progress of computer software and hardware technology, today's society has entered an information age, and computers, the Internet, and mobile communications have become an important part of people's daily life, study and work. In the information society, obtaining information and exchanging information through computers has become one of the basic means of people's work and life, so information equality has become the basis for everyone's equality in the information society. The so-called information equality means that everyone can obtain information, exchange information and use information equally, conveniently and without barriers under any circumstances.
信息无障碍化是一项民生工程，是国家和社会体现人文关怀精神的必然要求，是建设全面信息化社会的基本内容，是实现行业全面、协调且可持续发展的必要途径，也是缩小数字鸿沟、建立公平信息社会的一项重要工作。Information accessibility is a livelihood project, an inevitable requirement for the country and society to embody the spirit of humanistic care, the basic content of building a comprehensive information society, a necessary way to achieve comprehensive, coordinated and sustainable development of the industry, and a way to narrow the digital divide , An important task in building a fair information society.
目前，研究人员已开发出多种用于消除盲人使用计算机时存在的各种障碍的技术，MSAA(Microsoft Active Accessibility，微软主动式辅助)技术就是其中之一。对于符合MSAA技术标准的应用程序，当盲人用户通过输入设备(键盘、鼠标等)将屏幕指针定位到应用程序界面的某一位置时，应用程序或者读屏软件可以朗读出该位置所包含的文本。例如，屏幕指针的当前位置为按键时，应用程序可以读出该按键上显示的文字。At present, researchers have developed a variety of technologies for eliminating various obstacles that blind people have when using computers, and MSAA (Microsoft Active Accessibility, Microsoft Active Accessibility) technology is one of them. For applications that comply with MSAA technical standards, when a blind user positions the screen pointer to a certain position on the application interface through an input device (keyboard, mouse, etc.), the application or screen reading software can read the text contained in that position . For example, when the current position of the screen pointer is a key, the application can read the text displayed on the key.
同时，为了满足用户对软件产品的易用性、扩展性和个性化的需求，软件界面的开发越来越强调高效、易于扩展和艺术化，人们常见的标准化界面所占的比重在不断降低。例如，中国的软件开发商腾讯、迅雷等开发的软件客户端产品多采用上述方式来设计软件界面，从而很好地将软件界面与功能逻辑进行分离，同时易于实现各种界面效果，如换肤，透明等。但是，这种软件界面开发技术并不支持MSAA技术标准，导致现有的读屏软件无法获取这些软件中包含的文本信息，盲人用户无法操作这些软件，给盲人使用计算机带来很大的不便。At the same time, in order to meet the needs of users for ease of use, scalability and personalization of software products, the development of software interfaces is increasingly emphasizing efficiency, ease of expansion and artistry, and the proportion of common standardized interfaces is constantly decreasing. For example, the software client products developed by Chinese software developers such as Tencent and Xunlei mostly adopt the above-mentioned method to design the software interface, so as to separate the software interface from the functional logic well, and at the same time, it is easy to realize various interface effects, such as skin changing. , transparent, etc. However, this software interface development technology does not support the MSAA technical standard, resulting in that the existing screen reading software cannot obtain the text information contained in these software, and blind users cannot operate these software, which brings great inconvenience to blind people in using computers.
发明内容Contents of the invention
本发明所要解决的技术问题是，克服现有技术的不足，提供一种在不需要MSAA等技术标准支持的情况下，实现软件界面文本信息辅助浏览的方法。The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide a method for realizing assisted browsing of software interface text information without the support of technical standards such as MSAA.
为了解决上述问题，本发明提供一种软件界面文本信息的辅助浏览方法，该方法包括如下步骤：In order to solve the above-mentioned problem, the present invention provides a kind of auxiliary browsing method of software interface text information, and this method comprises the following steps:
A：生成目标窗口的文本坐标映射表；所述文本坐标映射表中包含所述目标窗口上显示的文本及相应的位置坐标；A: Generate a text coordinate mapping table of the target window; the text coordinate mapping table includes the text displayed on the target window and the corresponding position coordinates;
B：用户使用输入设备移动屏幕指针后，在所述文本坐标映射表中查找与屏幕指针的当前位置最接近的界面元素的文本；B: After the user uses the input device to move the screen pointer, search for the text of the interface element closest to the current position of the screen pointer in the text coordinate mapping table;
C：播放查找到的文本的内容。C: Play the content of the found text.
此外，在所述步骤B之后，还包含如下步骤：In addition, after the step B, the following steps are also included:
如果在所述文本坐标映射表中查找到与屏幕指针的当前位置最接近的界面元素的文本，则根据所述文本坐标映射表中记录的该文本所对应的位置坐标，将屏幕指针定位到相应的位置。If the text of the interface element closest to the current position of the screen pointer is found in the text coordinate mapping table, the screen pointer is positioned at the corresponding s position.
此外，在所述步骤B中，采用如下方式在所述文本坐标映射表中查找与屏幕指针的当前位置最接近的界面元素的文本：In addition, in the step B, the text of the interface element closest to the current position of the screen pointer is searched in the text coordinate mapping table in the following manner:
根据屏幕指针的移动方向，在所述文本坐标映射表中查找该移动方向上与屏幕指针的当前位置最接近的界面元素的文本。According to the moving direction of the screen pointer, the text of the interface element closest to the current position of the screen pointer in the moving direction is searched in the text coordinate mapping table.
此外，在所述步骤B中，采用如下方式实现根据屏幕指针的移动方向，在所述文本坐标映射表中查找该移动方向上与屏幕指针的当前位置最接近的界面元素的文本：In addition, in the step B, according to the moving direction of the screen pointer, the text of the interface element closest to the current position of the screen pointer in the moving direction is searched in the text coordinate mapping table in the following manner:
当屏幕指针的移动方向为向上时，如果屏幕指针当前位置所属的列的上方有文本，则获取位于该列的与屏幕指针当前位置最接近的文本；否则将屏幕指针向左或向右移动一列或多列并在对应列向上查找，直到查找到文本或屏幕指针达到所述目标窗口的左边界或右边界；When the moving direction of the screen pointer is upward, if there is text above the column where the current position of the screen pointer belongs, then get the text in the column that is closest to the current position of the screen pointer; otherwise, move the screen pointer to the left or right by one column or multiple columns and search upwards in the corresponding column until the text is found or the screen pointer reaches the left boundary or the right boundary of the target window;
当屏幕指针的移动方向为向下时，如果屏幕指针当前位置所属的列的下方有文本，则获取位于该列的与屏幕指针当前位置最接近的文本；否则将屏幕指针向右或向左移动一列或多列并在对应列向下查找，直到查找到文本或屏幕指针达到所述目标窗口的右边界或左边界；When the moving direction of the screen pointer is downward, if there is text below the column to which the current position of the screen pointer belongs, obtain the text in the column closest to the current position of the screen pointer; otherwise, move the screen pointer to the right or left one or more columns and search downwards in the corresponding columns until the text is found or the screen pointer reaches the right or left boundary of the target window;
当屏幕指针的移动方向为向左时，如果屏幕指针当前位置所属的行的左方有文本，则获取位于该行的与屏幕指针当前位置最接近的文本；否则将屏幕指针向上或向下移动一行或多行并在对应行向左查找，直到查找到文本或屏幕指针达到所述目标窗口的上边界或下边界；When the moving direction of the screen pointer is to the left, if there is text on the left of the line where the current position of the screen pointer belongs, then get the text in this line that is closest to the current position of the screen pointer; otherwise, move the screen pointer up or down One or more lines and search to the left in the corresponding line until the text is found or the screen pointer reaches the upper or lower boundary of the target window;
当屏幕指针的移动方向为向右时，如果屏幕指针当前位置所属的行的右方有文本，则获取位于该行的与屏幕指针当前位置最接近的文本；否则将屏幕指针向下或向上移动一行或多行并在对应行向右查找，直到查找到文本或屏幕指针达到目标窗口的下边界或上边界。When the moving direction of the screen pointer is to the right, if there is text on the right of the line where the current position of the screen pointer belongs, then get the text in this line that is closest to the current position of the screen pointer; otherwise, move the screen pointer down or up One or more lines and search to the right in the corresponding line until the text is found or the screen pointer reaches the lower or upper boundary of the target window.
此外，在所述步骤B中，用户使用输入设备移动屏幕指针之后、在所述文本坐标映射表中查找与屏幕指针的当前位置最接近的界面元素的文本之前，还包括如下步骤：In addition, in the step B, after the user uses the input device to move the screen pointer and before searching for the text of the interface element closest to the current position of the screen pointer in the text coordinate mapping table, the following steps are also included:
B1：判断屏幕指针的当前位置是否在所述目标窗口内部，如果不在，则将屏幕指针的当前位置定位到所述目标窗口的内部。B1: judging whether the current position of the screen pointer is inside the target window, if not, positioning the current position of the screen pointer inside the target window.
此外，在所述步骤B1中，所述目标窗口的内部为所述目标窗口的边界。In addition, in the step B1, the inside of the target window is the boundary of the target window.
此外，在所述步骤B1中，所述目标窗口的边界为所述目标窗口的左上角。In addition, in the step B1, the boundary of the target window is the upper left corner of the target window.
此外，在所述步骤B1中，如果判定屏幕指针的当前位置不在所述目标窗口内部，则采用如下方式之一定位屏幕指针的当前位置：In addition, in the step B1, if it is determined that the current position of the screen pointer is not inside the target window, one of the following methods is used to locate the current position of the screen pointer:
当用户向右移动屏幕指针时，将屏幕指针的初始位置定位到所述目标窗口的左边界；When the user moves the screen pointer to the right, locate the initial position of the screen pointer to the left border of the target window;
当用户向左移动屏幕指针时，将屏幕指针的初始位置定位到所述目标窗口的右边界；When the user moves the screen pointer to the left, locate the initial position of the screen pointer to the right border of the target window;
当用户向下移动屏幕指针时，将屏幕指针的初始位置定位到所述目标窗口的上边界；When the user moves the screen pointer downwards, positioning the initial position of the screen pointer to the upper boundary of the target window;
当用户向上移动屏幕指针时，将屏幕指针的初始位置定位到所述目标窗口的下边界。When the user moves the screen pointer upwards, the initial position of the screen pointer is positioned at the lower boundary of the target window.
此外，在所述步骤A之后，还包括如下步骤：In addition, after the step A, the following steps are also included:
当接收到前景变换消息时，清空所述文本坐标映射表中的内容，将当前活动窗口作为新的目标窗口，生成该新的目标窗口的文本坐标映射表。When the foreground change message is received, the content in the text coordinate mapping table is cleared, the current active window is used as a new target window, and the text coordinate mapping table of the new target window is generated.
此外，在所述步骤A之后，还包括如下步骤：In addition, after the step A, the following steps are also included:
当接收到所述目标窗口的拖放开始消息时，清空所述文本坐标映射表中的内容；When receiving the drag-and-drop start message of the target window, emptying the contents in the text coordinate mapping table;
当接收到所述目标窗口的拖放完成消息时，重新生成所述目标窗口的文本坐标映射表。When the drag and drop completion message of the target window is received, the text coordinate mapping table of the target window is regenerated.
此外，在所述步骤A之后，还包括如下步骤：In addition, after the step A, the following steps are also included:
以预先设定的时间间隔重复执行如下操作：清空所述文本坐标映射表中的内容，重新生成所述目标窗口的文本坐标映射表、或将当前活动窗口作为新的目标窗口，生成该新的目标窗口的文本坐标映射表。Repeat the following operations at preset time intervals: clear the content in the text coordinate mapping table, regenerate the text coordinate mapping table of the target window, or use the current active window as a new target window to generate the new The text coordinate mapping table of the target window.
综上所述，本发明提供了一种不依赖MSAA技术实现的辅助用户浏览软件界面文本信息的方法，该方法通过获取软件界面上显示的文本及相应的位置坐标来生成文本坐标映射表，并使用文本坐标映射表查找与当前屏幕指针位置最接近的界面元素的文本，进而实现辅助用户浏览软件界面文本信息的功能。该方法可以让用户快捷地在软件界面上的各文本信息间进行跳转、浏览。此外，本发明的方法使用了很少的快捷键，操作起来更加方便和人性化，大大降低了盲人用户操作软件的门槛。本发明的方法具有很好的通用性，可以支持各种基于Windows(视窗操作系统)环境的软件。In summary, the present invention provides a method for assisting users in browsing software interface text information that does not rely on MSAA technology. The method generates a text coordinate mapping table by acquiring text displayed on the software interface and corresponding position coordinates, and Use the text coordinate mapping table to find the text of the interface element closest to the current screen pointer position, and then realize the function of assisting users to browse the text information of the software interface. The method allows the user to quickly jump and browse among various text information on the software interface. In addition, the method of the present invention uses few shortcut keys, which makes the operation more convenient and humanized, and greatly reduces the threshold for blind users to operate the software. The method of the invention has good versatility and can support various software based on Windows (window operating system) environment.
附图说明Description of drawings
图1是本发明实施例软件界面文本信息的辅助浏览方法的流程图。FIG. 1 is a flow chart of an assisted browsing method for text information on a software interface according to an embodiment of the present invention.
具体实施方式Detailed ways
本发明的核心是，生成包含软件界面上显示的文本及相应的位置坐标的文本坐标映射表，当用户使用输入设备移动屏幕指针时，在文本坐标映射表中查找与当前屏幕指针位置最接近的界面元素的文本，并播放查找到的文本的内容。The core of the present invention is to generate a text coordinate mapping table that includes the text displayed on the software interface and the corresponding position coordinates. When the user uses an input device to move the screen pointer, the text coordinate mapping table is searched for the closest position to the current screen pointer. The text of the interface element and plays the content of the found text.
下面将结合附图和实施例对本发明进行详细描述。The present invention will be described in detail below with reference to the drawings and embodiments.
图1是本发明实施例软件界面文本信息的辅助浏览方法的流程图。如图1所示，该方法包含如下步骤：FIG. 1 is a flow chart of an assisted browsing method for text information on a software interface according to an embodiment of the present invention. As shown in Figure 1, the method includes the following steps:
步骤101：获得用户(通常为盲人用户)当前正在使用的软件(可称为目标软件或目标程序)的系统运行信息；Step 101: Obtain the system operation information of the software (which may be called target software or target program) currently being used by the user (usually a blind user);
上述系统运行信息包括：进程ID(Identification，标识符)，窗口句柄。The above system operation information includes: process ID (Identification, identifier) and window handle.
上述进程ID可用于在后续步骤中进行远程线程注入DLL(Dynamic LinkLibrary，动态链接库)的操作；上述窗口句柄可用于获取目标程序窗口的位置信息。Above-mentioned process ID can be used for carrying out the operation of remote thread injection DLL (Dynamic LinkLibrary, dynamic link library) in subsequent steps; Above-mentioned window handle can be used for obtaining the position information of target program window.
步骤102：执行远程线程注入DLL的操作；Step 102: Execute the operation of injecting the remote thread into the DLL;
具体地说，本步骤中可以通过目标程序的进程ID，在对应的进程(可称为目标进程)中创建远程线程，并且在该远程线程中加载并初始化用于拦截文本输出函数的动态链接库(可以称为文本函数拦截链接库)。Specifically, in this step, a remote thread can be created in the corresponding process (may be referred to as the target process) through the process ID of the target program, and the dynamic link library for intercepting the text output function is loaded and initialized in the remote thread (It may be referred to as a text function interception link library).
步骤103：在文本函数拦截链接库中执行以下操作：在目标进程的DLL导入表项中找到名称为GDI32.DLL的项，将其中包含的由系统提供的文本输出函数(可称为系统文本输出函数)的入口地址修改为相应的自定义文本输出函数的入口地址；并记录修改前的系统文本输出函数的入口地址；Step 103: perform the following operations in the text function interception link library: find the item named GDI32.DLL in the DLL import entry of the target process, and the text output function (which can be called system text output) provided by the system contained therein function) to modify the entry address of the corresponding user-defined text output function; and record the entry address of the system text output function before modification;
上述系统文本输出函数包括：TextOutA，TextOutW，ExtTextOutA，ExtTextOutW和ScriptShape等；相应的自定义文本输出函数可以记作：MyTextOutA，MyTextOutW，MyExtTextOutA，MyExtTextOutW和MyScriptShape等。The above-mentioned system text output functions include: TextOutA, TextOutW, ExtTextOutA, ExtTextOutW and ScriptShape, etc.; the corresponding custom text output functions can be recorded as: MyTextOutA, MyTextOutW, MyExtTextOutA, MyExtTextOutW and MyScriptShape, etc.
步骤104：主动刷新目标程序的目标窗口(通常为当前桌面的活动窗口，或称为前景窗口)，或等待目标窗口被刷新(可以称为被动刷新，例如，目标窗口由非活动窗口被切换为活动窗口，目标窗口将被刷新)，以引发系统对文本输出函数的调用。Step 104: actively refresh the target window of the target program (usually the active window of the current desktop, or called the foreground window), or wait for the target window to be refreshed (can be called passive refresh, for example, the target window is switched from an inactive window to active window, the target window will be refreshed) to cause the system to call the text output function.
上述主动刷新和被动刷新的刷新范围可以是整个目标窗口，也可以是目标窗口的一部分。The refreshing range of the above-mentioned active refreshing and passive refreshing may be the whole target window or a part of the target window.
步骤105：当系统调用系统文本输出函数以便对窗口中的包含文本的界面元素进行绘制时，由于在步骤103中已将系统文本输出函数的入口地址修改为自定义文本输出函数的入口地址，因此引发了系统对自定义文本输出函数的调用，自定义文本输出函数根据系统传递的函数调用参数，生成目标窗口的文本坐标映射表；Step 105: When the system invokes the system text output function so that the interface elements containing text in the window are drawn, since the entry address of the system text output function has been modified to the entry address of the user-defined text output function in step 103, The system calls the custom text output function, and the custom text output function generates the text coordinate mapping table of the target window according to the function call parameters passed by the system;
上述包含文本的界面元素可以是：标题栏、菜单、工具栏、地址栏、客户区等。The above-mentioned interface elements containing text may be: title bar, menu, tool bar, address bar, client area and so on.
上述函数调用参数中包含：文本及相应的相对位置坐标。上述相对位置坐标是文本相对于目标窗口的位置坐标(通常为相对于目标窗口左上角的位置坐标)。The parameters of the above function call include: text and corresponding relative position coordinates. The above relative position coordinates are the position coordinates of the text relative to the target window (usually the position coordinates relative to the upper left corner of the target window).
上述文本坐标映射表中包含了目标窗口中文本，以及该文本所对应的位置坐标。在本实施例中，上述位置坐标为相对位置坐标。The above text coordinate mapping table includes the text in the target window and the position coordinates corresponding to the text. In this embodiment, the above-mentioned position coordinates are relative position coordinates.
在本发明的其它实施例中，上述位置坐标可以是绝对位置坐标(即文本相对于桌面的位置坐标)。文本的绝对位置坐标可以根据文本的相对位置坐标和目标窗口的位置信息换算得出。In other embodiments of the present invention, the above-mentioned position coordinates may be absolute position coordinates (that is, the position coordinates of the text relative to the desktop). The absolute position coordinates of the text can be calculated according to the relative position coordinates of the text and the position information of the target window.
本实施例中，在自定义文本输出函数中，将文本以及对应的位置坐标写入文本坐标映射表后，可以调用对应的系统文本输出函数完成后续的文本输出操作(即在目标窗口的特定位置绘制相应的文本)。当然，也可以不调用对应的系统文本输出函数，由自定义文本输出函数完成后续的文本输出操作。In this embodiment, in the custom text output function, after the text and the corresponding position coordinates are written into the text coordinate mapping table, the corresponding system text output function can be called to complete the subsequent text output operation (that is, in the specific position of the target window draw the corresponding text). Of course, the corresponding system text output function may not be called, and the subsequent text output operation may be completed by the user-defined text output function.
需要注意的是，本步骤中系统可能会多次调用一个或多个系统文本输出函数，从而会多次调用一个或多个自定义文本输出函数，每次调用都会在文本坐标映射表中写入一条记录，最终生成完整的目标窗口文本坐标映射表。It should be noted that in this step, the system may call one or more system text output functions multiple times, thus calling one or more custom text output functions multiple times, and each call will write in the text coordinate mapping table A record, and finally generate a complete target window text coordinate mapping table.
步骤106：当用户使用输入设备(例如，键盘)进行移动屏幕指针的操作后(例如，按下某一方向键时)，获取屏幕指针(例如，鼠标指针、光标等)的当前位置，并判断当前的屏幕指针是否在目标窗口的内部，如果不在，则执行步骤107；否则跳转至步骤108；Step 106: After the user uses an input device (for example, a keyboard) to move the screen pointer (for example, when pressing a direction key), obtain the current position of the screen pointer (for example, a mouse pointer, a cursor, etc.), and determine Whether the current screen pointer is inside the target window, if not, then execute step 107; otherwise jump to step 108;
本实施例中，可以通过目标窗口的窗口句柄获得目标窗口的位置信息，并根据屏幕指针的当前位置和目标窗口的位置信息来判断屏幕指针是否在目标窗口的内部。In this embodiment, the location information of the target window can be obtained through the window handle of the target window, and whether the screen pointer is inside the target window is determined according to the current location of the screen pointer and the location information of the target window.
上述窗口的位置信息通常为窗口矩形左上角的坐标，以及窗口的长宽值；当然，根据系统的不同，也可以用其它方式来表示窗口的位置信息，例如，用矩形左上角和右下角的坐标来表示矩形窗口的位置信息。The position information of the above window is usually the coordinates of the upper left corner of the window rectangle, and the length and width of the window; of course, depending on the system, other ways can also be used to represent the position information of the window, for example, using the upper left corner and the lower right corner of the rectangle Coordinates to represent the position information of the rectangular window.
步骤107：将屏幕指针的初始位置(当前位置)定位到目标窗口的内部(包括窗口的边界)；Step 107: Position the initial position (current position) of the screen pointer to the inside of the target window (including the boundary of the window);
为了便于用户的后续操作，本步骤中通常将屏幕指针的初始位置定位到目标窗口的边界(例如，上边界、或下边界、或左边界、或右边界)。In order to facilitate the user's subsequent operations, in this step, the initial position of the screen pointer is usually positioned at the boundary of the target window (for example, the upper boundary, or the lower boundary, or the left boundary, or the right boundary).
本实施例中，如果当前的屏幕指针不在目标窗口的内部，则将屏幕指针的初始位置定位到目标窗口的左上角。In this embodiment, if the current screen pointer is not inside the target window, the initial position of the screen pointer is positioned at the upper left corner of the target window.
在本发明的其它实施例中，还可以根据用户的操作方向(即屏幕指针的移动方向)确定屏幕指针的初始位置。例如：In other embodiments of the present invention, the initial position of the screen pointer may also be determined according to the user's operation direction (ie, the moving direction of the screen pointer). For example:
当用户按下向右的方向键，且屏幕指针的当前位置不在目标窗口的内部时，将屏幕指针的初始位置定位到目标窗口的左边界(包括左上角和左下角)，优选方案为定位到左上角；When the user presses the right arrow key and the current position of the screen pointer is not inside the target window, position the initial position of the screen pointer to the left border of the target window (including the upper left corner and the lower left corner), preferably at upper left corner;
当用户按下向左的方向键，且屏幕指针的当前位置不在目标窗口的内部时，将屏幕指针的初始位置定位到目标窗口的右边界(包括右上角和右下角)，优选方案为定位到右上角；When the user presses the left arrow key and the current position of the screen pointer is not inside the target window, position the initial position of the screen pointer to the right border (including the upper right corner and the lower right corner) of the target window. upper right corner;
当用户按下向下的方向键，且屏幕指针的当前位置不在目标窗口的内部时，将屏幕指针的初始位置定位到目标窗口的上边界(包括左上角和右上角)，优选方案为定位到左上角；When the user presses the down arrow key and the current position of the screen pointer is not inside the target window, locate the initial position of the screen pointer to the upper boundary of the target window (including the upper left corner and the upper right corner), preferably at upper left corner;
当用户按下向上的方向键，且屏幕指针的当前位置不在目标窗口的内部时，将屏幕指针的初始位置定位到目标窗口的下边界(包括左下角和右下角)，优选方案为定位到左下角。When the user presses the up arrow key and the current position of the screen pointer is not inside the target window, position the initial position of the screen pointer to the lower boundary of the target window (including the lower left and right corners), preferably to the lower left horn.
步骤108：在文本坐标映射表中查找与当前屏幕指针位置最接近的界面元素的文本；本步骤可以通过将当前屏幕指针的坐标与文本坐标映射表中各文本的位置坐标做对比来实现；Step 108: Find the text of the interface element closest to the current screen pointer position in the text coordinate mapping table; this step can be realized by comparing the coordinates of the current screen pointer with the position coordinates of each text in the text coordinate mapping table;
此外，为了便于用户进行后续的操作，本步骤中在查找到与当前屏幕指针位置最接近的界面元素的文本后，还需要将屏幕指针定位到该文本对应的位置。In addition, in order to facilitate the user to perform subsequent operations, in this step, after finding the text of the interface element closest to the current screen pointer position, the screen pointer needs to be positioned at the position corresponding to the text.
作为优选的方案，本实施例中，可以根据用户的操作方向(即屏幕指针的移动方向)在文本坐标映射表中查找在该方向上与当前屏幕指针位置最接近的界面元素的文本；例如：As a preferred solution, in this embodiment, the text of the interface element closest to the current screen pointer position in this direction can be searched in the text coordinate mapping table according to the user's operation direction (ie, the moving direction of the screen pointer); for example:
当用户按下向上的方向键时，如果屏幕指针当前位置所属的列的上方有文本，则获取位于该列的与屏幕指针当前位置最接近的文本；否则将屏幕指针向左(或向右)移动一列或多列并在对应列向上查找，直到查找到文本或屏幕指针达到目标窗口的左边界(或右边界)。When the user presses the up arrow key, if there is text above the column where the screen pointer's current position belongs, then get the text in that column that is closest to the current position of the screen pointer; otherwise, move the screen pointer to the left (or right) Move one or more columns and search up the corresponding column until the text is found or the screen pointer reaches the left border (or right border) of the target window.
当用户按下向下的方向键时，如果屏幕指针当前位置所属的列的下方有文本，则获取位于该列的与屏幕指针当前位置最接近的文本；否则将屏幕指针向右(或向左)移动一列或多列并在对应列向下查找，直到查找到文本或屏幕指针达到目标窗口的右边界(或左边界)。When the user presses the down arrow key, if there is text below the column where the current position of the screen pointer belongs, then get the text in the column that is closest to the current position of the screen pointer; otherwise, move the screen pointer to the right (or to the left) ) to move one or more columns and search down the corresponding column until the text is found or the screen pointer reaches the right border (or left border) of the target window.
当用户按下向左的方向键时，如果屏幕指针当前位置所属的行的左方有文本，则获取位于该行的与屏幕指针当前位置最接近的文本；否则将屏幕指针向上(或向下)移动一行或多行并在对应行向左查找，直到查找到文本或屏幕指针达到目标窗口的上边界(或下边界)。When the user presses the left arrow key, if there is text to the left of the line where the current position of the screen pointer belongs, then get the text closest to the current position of the screen pointer in this line; otherwise, move the screen pointer up (or down) ) to move one or more lines and search to the left in the corresponding line until the text is found or the screen pointer reaches the upper boundary (or lower boundary) of the target window.
当用户按下向右的方向键时，如果屏幕指针当前位置所属的行的右方有文本，则获取位于该行的与屏幕指针当前位置最接近的文本；否则将屏幕指针向下(或向上)移动一行或多行并在对应行向右查找，直到查找到文本或屏幕指针达到目标窗口的下边界(或上边界)。When the user presses the right arrow key, if there is text on the right of the line where the screen pointer's current position belongs, then get the text that is closest to the current position of the screen pointer in this line; otherwise, move the screen pointer down (or up) ) to move one or more lines and search to the right in the corresponding line until the text is found or the screen pointer reaches the lower boundary (or upper boundary) of the target window.
步骤109：播放(通常采用朗读的方式)查找到的文本的内容。Step 109: Play (usually read aloud) the content of the found text.
步骤110：当用户关闭目标程序时，清空文本坐标映射表；Step 110: clear the text coordinate mapping table when the user closes the target program;
当用户关闭目标程序时，通常会触发文本函数拦截链接库的动态链接库卸载事件(DLL_PROCESS_DETACH事件)，而对该事件通常在文本函数拦截链接库的入口函数(通常为DllMain函数)中进行处理，因此可以在文本函数拦截链接库的入口函数中判别是否发生动态链接库卸载事件，当发生该事件时清空文本坐标映射表，并将GDI32.DLL中包含的文本输出函数的入口地址修改(恢复)为修改前的系统文本输出函数的入口地址。When the user closes the target program, the dynamic link library unloading event (DLL_PROCESS_DETACH event) of the text function intercepting link library is usually triggered, and the event is usually processed in the entry function (usually DllMain function) of the text function intercepting link library, Therefore, it is possible to judge whether a dynamic link library unloading event occurs in the entry function of the text function interception link library. When the event occurs, the text coordinate mapping table is cleared, and the entry address of the text output function contained in GDI32.DLL is modified (restored) Output the entry address of the function for the system text before modification.
图1所对应的流程详细地描述了目标窗口文本坐标映射表的生成(步骤101～105)、屏幕指针初始位置的定位(步骤106～107)以及对与屏幕指针当前位置最接近的界面元素的文本进行浏览(步骤108～109)的整个过程。在用户没有进行应用程序/窗口的切换，且没有进行窗口移动和缩放等操作的情况下，采用上述方法可以帮助用户快捷、准确地对软件界面(窗口)的文本信息进行浏览。The process corresponding to Figure 1 describes in detail the generation of the target window text coordinate mapping table (steps 101-105), the positioning of the initial position of the screen pointer (steps 106-107), and the positioning of the interface element closest to the current position of the screen pointer. The whole process of text browsing (steps 108-109). When the user does not switch applications/windows, and does not perform operations such as window movement and zooming, the above method can help the user browse the text information of the software interface (window) quickly and accurately.
为了节约系统资源，通常仅生成和使用一个文本坐标映射表，该文本坐标映射表对应当前桌面的活动窗口。在这种情况下，当用户进行应用程序/窗口切换、或进行窗口移动和缩放等操作时，文本坐标映射表中的文本、和/或相应的位置坐标会与当前桌面的活动窗口所对应的信息不一致，需要对文本坐标映射表进行更新操作，下面将简要介绍相应的方法。In order to save system resources, usually only one text coordinate mapping table is generated and used, and the text coordinate mapping table corresponds to the active window of the current desktop. In this case, when the user switches applications/windows, or performs operations such as window movement and zooming, the text in the text coordinate mapping table and/or the corresponding position coordinates will correspond to the active window of the current desktop The information is inconsistent, and the text coordinate mapping table needs to be updated. The corresponding method will be briefly introduced below.
进行应用程序切换时文本坐标映射表的更新方法Method for updating text coordinate mapping table when switching applications
当用户启动新的应用程序，该新应用程序的窗口自动被切换为当前桌面的活动窗口，或用户手动进行应用程序切换，将其它应用程序的窗口手动切换为当前桌面的活动窗口时，可采用以下方式之一进行文本坐标映射表的更新：When the user starts a new application, the window of the new application is automatically switched to the active window of the current desktop, or the user manually switches the application, and manually switches the windows of other applications to the active window of the current desktop. Update the text coordinate mapping table in one of the following ways:
方式一(手动刷新)：当用户接收到新的应用程序启动完成的通知、或新的应用程序的窗口已切换为当前桌面的活动窗口的通知后，使用预先设置的文本坐标映射表更新接口完成如下操作：清空当前文本坐标映射表中的内容，将当前活动窗口作为目标窗口重新生成文本坐标映射表(可以采用图1步骤101～105的方法)，如果采用图1所示的方法，还需要将原应用程序GDI32.DLL中包含的文本输出函数的入口地址修改(恢复)为修改前的系统文本输出函数的入口地址；Method 1 (manual refresh): After the user receives the notification that the new application has been launched, or the window of the new application has been switched to the active window of the current desktop, the interface is updated using the preset text coordinate mapping table. The following operations: clear the content in the current text coordinate mapping table, use the current active window as the target window to regenerate the text coordinate mapping table (the method of steps 101 to 105 in Figure 1 can be used), if the method shown in Figure 1 is used, it is necessary to Modify (restore) the entry address of the text output function contained in the original application program GDI32.DLL to the entry address of the system text output function before modification;
上述文本坐标映射表更新接口通常由预先设置的快捷键以及相应的按键处理程序实现。The above-mentioned interface for updating the text coordinate mapping table is usually realized by preset shortcut keys and corresponding key processing procedures.
方式二(自动刷新)：当接收到系统发送的前景变换消息(例如，EVENT_SYSTEM_FOREGROUND消息)，表明新的窗口变为当前的活动窗口时，在相应的消息处理程序中完成如下操作：判断当前的活动窗口所对应的应用程序(进程)与文本坐标映射表所对应的应用程序(进程)是否一致，如果不一致(表明发生了应用程序的切换)，则清空当前文本坐标映射表中的内容，将当前活动窗口作为目标窗口重新生成文本坐标映射表(可以采用图1步骤101～105的方法)，如果采用图1所示的方法，还需要将原应用程序GDI32.DLL中包含的文本输出函数的入口地址修改(恢复)为修改前的系统文本输出函数的入口地址；Mode 2 (automatic refresh): When receiving the foreground change message (for example, EVENT_SYSTEM_FOREGROUND message) sent by the system, indicating that the new window becomes the current active window, complete the following operations in the corresponding message processing program: determine the current activity Whether the application program (process) corresponding to the window is consistent with the application program (process) corresponding to the text coordinate mapping table; The active window is used as the target window to regenerate the text coordinate mapping table (the method of steps 101 to 105 in Figure 1 can be used). If the method shown in Figure 1 is adopted, the entry of the text output function contained in the original application program GDI32. Address modification (recovery) is the entry address of the system text output function before modification;
采用这种方式刷新文本坐标映射表时需要记录当前文本坐标映射表所对应的应用程序的进程ID，以便进行上述判断。When refreshing the text coordinate mapping table in this way, it is necessary to record the process ID of the application program corresponding to the current text coordinate mapping table, so as to make the above judgment.
方式三(定时刷新)：以预先设定的时间间隔(例如，10秒)重复执行如下操作：判断当前的活动窗口所对应的应用程序(进程)与文本坐标映射表所对应的应用程序(进程)是否一致，如果不一致(表明发生了应用程序的切换)，则清空当前文本坐标映射表中的内容，将当前活动窗口作为目标窗口重新生成文本坐标映射表(可以采用图1步骤101～105的方法)，如果采用图1所示的方法，还需要将原应用程序GDI32.DLL中包含的文本输出函数的入口地址修改(恢复)为修改前的系统文本输出函数的入口地址；Mode 3 (timing refresh): Repeat the following operations at a preset time interval (for example, 10 seconds): determine the application program (process) corresponding to the current active window and the application program (process) corresponding to the text coordinate mapping table ) are consistent, if inconsistent (indicating that the switching of the application program has taken place), then clear the content in the current text coordinate mapping table, and use the current active window as the target window to regenerate the text coordinate mapping table (you can use steps 101 to 105 in Figure 1 method), if adopt the method shown in Fig. 1, also need to modify (restore) the entry address of the text output function contained in the original application program GDI32.DLL to be the entry address of the system text output function before modification;
采用这种方式刷新文本坐标映射表时需要记录当前文本坐标映射表所对应的应用程序的进程ID，以便进行上述判断。When refreshing the text coordinate mapping table in this way, it is necessary to record the process ID of the application program corresponding to the current text coordinate mapping table, so as to make the above judgment.
进行同一应用程序的不同窗口间切换时文本坐标映射表的更新方法Method for updating text coordinate mapping table when switching between different windows of the same application program
方式四(自动刷新)：当接收到系统发送的前景变换消息(例如，EVENT_SYSTEM_FOREGROUND消息)，表明新的窗口变为当前的活动窗口时，在相应的消息处理程序中完成如下操作：判断当前的活动窗口所对应的应用程序(进程)与文本坐标映射表所对应的应用程序(进程)是否一致，如果一致(表明发生了同一应用程序的窗口切换)，则清空当前文本坐标映射表中的内容，强制刷新窗口界面(可以通过在窗口表面绘制透明窗口等方式实现)，以触发步骤105，完成当前活动窗口(即新的目标窗口)的文本坐标映射表的重新生成；Mode 4 (automatic refresh): When receiving the foreground change message (for example, EVENT_SYSTEM_FOREGROUND message) sent by the system, indicating that the new window becomes the current active window, complete the following operations in the corresponding message processing program: determine the current activity Whether the application program (process) corresponding to the window is consistent with the application program (process) corresponding to the text coordinate mapping table, if consistent (showing that the window switching of the same application program has taken place), then clear the content in the current text coordinate mapping table, Forcibly refresh the window interface (can be realized by drawing a transparent window on the window surface, etc.), to trigger step 105, to complete the regeneration of the text coordinate mapping table of the current active window (i.e. the new target window);
采用这种方式刷新文本坐标映射表时需要记录当前文本坐标映射表所对应的应用程序的进程ID，以便进行上述判断。When refreshing the text coordinate mapping table in this way, it is necessary to record the process ID of the application program corresponding to the current text coordinate mapping table, so as to make the above judgment.
方式五(定时刷新)：以预先设定的时间间隔(例如，10秒)重复执行如下操作：判断当前的活动窗口所对应的应用程序(进程)及窗口与文本坐标映射表所对应的应用程序(进程)及窗口是否一致，如果应用程序一致且窗口不一致(表明发生了同一应用程序的窗口切换)，则清空当前文本坐标映射表中的内容，并强制刷新当前活动窗口(可以通过在窗口表面绘制透明窗口等方式实现)，以触发步骤105，完成当前活动窗口(即新的目标窗口)的文本坐标映射表的重新生成；Mode 5 (timing refresh): Repeat the following operations at a preset time interval (for example, 10 seconds): determine the application program (process) corresponding to the current active window and the application program corresponding to the window and text coordinate mapping table (process) and the window are consistent, if the application is consistent and the window is inconsistent (indicating that the window switching of the same application has occurred), then clear the contents of the current text coordinate mapping table, and force refresh the current active window (can be passed on the window surface draw transparent window etc. to realize), to trigger step 105, complete the regeneration of the text coordinate mapping table of the current active window (i.e. new target window);
采用这种方式刷新文本坐标映射表时需要记录当前文本坐标映射表所对应的应用程序的进程ID，以及对应的窗口句柄，以便进行上述判断。When refreshing the text coordinate mapping table in this way, it is necessary to record the process ID of the application program corresponding to the current text coordinate mapping table and the corresponding window handle, so as to make the above judgment.
进行窗口移动或缩放操作时文本坐标映射表的更新方法The update method of the text coordinate mapping table when the window is moved or zoomed
当用户对目标窗口进行移动或缩放操作时，目标窗口的文本及其对应的位置坐标会发生变化，可采用以下方式之一进行文本坐标映射表的更新：When the user moves or zooms the target window, the text of the target window and its corresponding position coordinates will change, and one of the following methods can be used to update the text coordinate mapping table:
方式六(自动刷新)：当接收到系统发送的目标窗口的拖放开始消息(例如，EVENT_SYSTEM_DRAGDROPSTART消息)，表明用户开始进行窗口移动或缩放操作时，在相应的消息处理程序中完成如下操作：清空当前文本坐标映射表中的内容；当接收到系统发送的目标窗口的拖放完成消息(例如，EVENT_SYSTEM_DRAGDROPEND消息)，表明用户已完成窗口移动或缩放操作时，在相应的消息处理程序中完成如下操作：强制刷新窗口界面(可以通过在窗口表面绘制透明窗口等方式实现)，以触发步骤105，完成文本坐标映射表的重新生成。Method 6 (automatic refresh): When receiving the drag-and-drop start message (for example, EVENT_SYSTEM_DRAGDROPSTART message) of the target window sent by the system, indicating that the user starts to move or zoom the window, complete the following operations in the corresponding message handler: clear The content in the current text coordinate mapping table; when receiving the drag-and-drop completion message (for example, EVENT_SYSTEM_DRAGDROPEND message) of the target window sent by the system, indicating that the user has completed the window moving or zooming operation, complete the following operations in the corresponding message handler : to forcibly refresh the window interface (can be realized by drawing a transparent window on the window surface, etc.), to trigger step 105, to complete the regeneration of the text coordinate mapping table.
方式七(定时刷新)：以预先设定的时间间隔(例如，10秒)重复执行如下操作：判断当前的活动窗口所对应的应用程序(进程)及窗口与文本坐标映射表所对应的应用程序(进程)及窗口是否一致，如果都一致(表明未发生应用程序及窗口的切换)，则清空当前文本坐标映射表中的内容，并强制刷新当前活动窗口(可以通过在窗口表面绘制透明窗口等方式实现)，以触发步骤105，完成文本坐标映射表的重新生成；Method 7 (timing refresh): Repeat the following operations at a preset time interval (for example, 10 seconds): determine the application program (process) corresponding to the current active window and the application program corresponding to the window and text coordinate mapping table Whether the (process) and the window are consistent, if they are consistent (indicating that the switching of the application program and the window does not occur), then clear the content in the current text coordinate mapping table, and force the current active window to be refreshed (you can draw a transparent window on the window surface, etc. way), to trigger step 105, to complete the regeneration of the text coordinate mapping table;
采用这种方式刷新文本坐标映射表时需要记录当前文本坐标映射表所对应的应用程序的进程ID，以及对应的窗口句柄，以便进行上述判断。When refreshing the text coordinate mapping table in this way, it is necessary to record the process ID of the application program corresponding to the current text coordinate mapping table and the corresponding window handle, so as to make the above judgment.
此外，采用方式七进行文本坐标映射表的刷新还可以防止窗口内部的局部刷新所造成的文本坐标映射表信息错误。In addition, using the seventh method to refresh the text coordinate mapping table can also prevent information errors in the text coordinate mapping table caused by partial refreshing inside the window.
需要注意的是，本发明的上述实施例通过拦截API(ApplicationProgramming Interface，应用程序编程接口)调用，即拦截系统文本输出函数调用的方式来获取系统文本输出函数的函数调用参数，并使用系统文本输出函数的函数调用参数(包括文本及相应的位置坐标)来生成文本坐标映射表；在本发明的其它实施例中，还可以采用其它方式来获取系统文本输出函数的函数调用参数，例如：直接改写(替换)系统文本输出函数，在系统文本输出函数中向当前活动窗口的文本坐标映射表中写入该函数的调用参数(包括文本及相应的位置坐标)。It should be noted that the above-described embodiments of the present invention obtain the function call parameters of the system text output function by intercepting the API (Application Programming Interface, application programming interface) call, that is, intercept the system text output function call, and use the system text output The function call parameter (comprising text and corresponding position coordinate) of function generates text coordinate mapping table; In other embodiments of the present invention, can also adopt other ways to obtain the function call parameter of system text output function, for example: directly rewrite (replacement) system text output function, in the system text output function, write the call parameter (comprising text and corresponding position coordinate) of this function in the text coordinate mapping table of current active window.
此外，除了本发明实施例所描述的API拦截方法外，还可以采用设置钩子函数(SetWindowsHookEx)、修改注册表等方式进行API的拦截，具体方法请参阅相关文档。In addition, in addition to the API interception methods described in the embodiments of the present invention, API interception can also be performed by setting a hook function (SetWindowsHookEx), modifying the registry, etc. For specific methods, please refer to related documents.
此外，对于应用程序设计者，也可以采用如下方式生成文本坐标映射表：在应用程序中加入用于生成文本坐标映射表的代码，例如，在进行窗口绘制时(通常是对ON_PAINT消息进行处理的函数中)将待绘制的文本及相应的位置坐标自行写入文本坐标映射表。In addition, for the application program designer, the text coordinate mapping table can also be generated in the following way: add the code for generating the text coordinate mapping table in the application program, for example, when the window is drawn (usually the ON_PAINT message is processed function) write the text to be drawn and the corresponding position coordinates into the text coordinate mapping table by itself.
与现有技术相比，本发明的有益效果体现在如下几个方面：Compared with the prior art, the beneficial effects of the present invention are reflected in the following aspects:
(1)高效性：本发明实现了Windows环境下基于API拦截的辅助用户(通常为盲人用户)对软件界面文本信息进行浏览方法，该方法使用户通过快捷键可以在上下左右四个方向迅速浏览软件界面上文字信息，这种操作方式很好地利用了现有软件布局，让盲人用户的操作习惯与正常用户渐趋一致，更高效地操作使用软件；(1) Efficiency: the present invention realizes the method for browsing the text information of the software interface by an auxiliary user (usually a blind user) based on API interception under the Windows environment. This method enables the user to quickly browse in the four directions of up, down, left, and right through shortcut keys Text information on the software interface, this operation method makes good use of the existing software layout, so that the operating habits of blind users are gradually consistent with those of normal users, and the software can be operated and used more efficiently;
(2)易用性：本发明的方法使用了较少的快捷键，操作起来更加方便和人性化，大大地降低了盲人用户进行软件操作的门槛；(2) Ease of use: the method of the present invention uses fewer shortcut keys, which is more convenient and humanized to operate, and greatly reduces the threshold for blind users to perform software operations;
(3)通用性：本发明的方法无需采用MSAA技术，更广泛地支持各种Windows平台的软件，使得盲人用户在使用软件时可以有更多的选择。(3) Versatility: the method of the present invention does not need to adopt MSAA technology, and supports various Windows platform software more widely, so that blind users can have more choices when using software.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2010105934993ACN102023805B (en) | 2010-12-16 | 2010-12-16 | Method for assisting in browsing text messages on software interface |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2010105934993ACN102023805B (en) | 2010-12-16 | 2010-12-16 | Method for assisting in browsing text messages on software interface |
| Publication Number | Publication Date |
|---|---|
| CN102023805Atrue CN102023805A (en) | 2011-04-20 |
| CN102023805B CN102023805B (en) | 2012-08-22 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2010105934993AExpired - Fee RelatedCN102023805B (en) | 2010-12-16 | 2010-12-16 | Method for assisting in browsing text messages on software interface |
| Country | Link |
|---|---|
| CN (1) | CN102023805B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013143407A1 (en)* | 2012-03-31 | 2013-10-03 | International Business Machines Corporation | Data processing, data collection |
| CN104951188A (en)* | 2015-06-18 | 2015-09-30 | 烟台朱葛软件科技有限公司 | Visual information interactive method and control system |
| CN109597541A (en)* | 2018-10-30 | 2019-04-09 | 北京五岳鑫信息技术股份有限公司 | A kind of character method for transformation and system |
| CN113490025A (en)* | 2021-08-03 | 2021-10-08 | Vidaa美国公司 | Display apparatus and electronic program guide display method |
| CN114816401A (en)* | 2022-04-13 | 2022-07-29 | 上海弘玑信息技术有限公司 | Interface element positioning method, electronic equipment and storage medium |
| CN115061685A (en)* | 2022-06-30 | 2022-09-16 | 上海弘玑信息技术有限公司 | Interface element positioning method of software client, electronic device and storage medium |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5374924A (en)* | 1991-08-19 | 1994-12-20 | International Business Machines Corporation | Computer user interface with stereo and filtered sound effects |
| CN101201718A (en)* | 2006-09-14 | 2008-06-18 | 阿瓦雅技术有限公司 | Audible computer user interface method and apparatus |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5374924A (en)* | 1991-08-19 | 1994-12-20 | International Business Machines Corporation | Computer user interface with stereo and filtered sound effects |
| CN101201718A (en)* | 2006-09-14 | 2008-06-18 | 阿瓦雅技术有限公司 | Audible computer user interface method and apparatus |
| Title |
|---|
| 《中国优秀硕士论文全文数据库》 20100531 郑志 残疾人人机交互辅助系统设计与实现 全文 1-11 , 2* |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013143407A1 (en)* | 2012-03-31 | 2013-10-03 | International Business Machines Corporation | Data processing, data collection |
| CN103365555A (en)* | 2012-03-31 | 2013-10-23 | 国际商业机器公司 | Data processing method and system and data collecting method and system |
| GB2519863A (en)* | 2012-03-31 | 2015-05-06 | Ibm | Data Processing Data Collection |
| US10140377B2 (en) | 2012-03-31 | 2018-11-27 | International Business Machines Corporation | Data processing, data collection |
| CN104951188A (en)* | 2015-06-18 | 2015-09-30 | 烟台朱葛软件科技有限公司 | Visual information interactive method and control system |
| CN109597541A (en)* | 2018-10-30 | 2019-04-09 | 北京五岳鑫信息技术股份有限公司 | A kind of character method for transformation and system |
| CN113490025A (en)* | 2021-08-03 | 2021-10-08 | Vidaa美国公司 | Display apparatus and electronic program guide display method |
| CN114816401A (en)* | 2022-04-13 | 2022-07-29 | 上海弘玑信息技术有限公司 | Interface element positioning method, electronic equipment and storage medium |
| CN114816401B (en)* | 2022-04-13 | 2022-12-23 | 上海弘玑信息技术有限公司 | Interface element positioning method, electronic equipment and storage medium |
| CN115061685A (en)* | 2022-06-30 | 2022-09-16 | 上海弘玑信息技术有限公司 | Interface element positioning method of software client, electronic device and storage medium |
| Publication number | Publication date |
|---|---|
| CN102023805B (en) | 2012-08-22 |
| Publication | Publication Date | Title |
|---|---|---|
| US12050857B2 (en) | Device, method, and graphical user interface for editing screenshot images | |
| CN108491131B (en) | Operation method, device and intelligent interactive tablet for intelligent interactive tablet | |
| CN108958608B (en) | Interface element operation method and device of electronic whiteboard and interactive intelligent equipment | |
| CN111651116B (en) | Split screen interaction method, electronic equipment and computer storage medium | |
| US10775971B2 (en) | Pinch gestures in a tile-based user interface | |
| US20150339050A1 (en) | Ink for Interaction | |
| CN102023805B (en) | Method for assisting in browsing text messages on software interface | |
| US20140089824A1 (en) | Systems And Methods For Dynamically Altering A User Interface Based On User Interface Actions | |
| US20120216141A1 (en) | Touch gestures for text-entry operations | |
| CN108804002B (en) | Text annotation method and device for interactive smart device | |
| US20140145945A1 (en) | Touch-based input control method | |
| KR20140028384A (en) | User terminal apparatus and contol method thereof | |
| CN111813302A (en) | Screen projection display method, device, terminal device and storage medium | |
| TW201525851A (en) | Touch-based reorganization of page element | |
| CN110471609B (en) | Text information editing method, device, computer equipment and storage medium | |
| US20140181737A1 (en) | Method for processing contents and electronic device thereof | |
| CN104216646A (en) | Method and device for creating application program based on gesture | |
| CN103150113A (en) | Method and device for selecting display content of touch screen | |
| CN112462989A (en) | Operation method and device of intelligent interaction panel, intelligent interaction panel and medium | |
| CN108492349B (en) | Processing method, device and equipment for writing strokes and storage medium | |
| TWI485616B (en) | Method for recording trajectory and electronic apparatus | |
| CN103279263B (en) | The method and system that a kind of file quickly creates and edits | |
| WO2012134479A1 (en) | Augmenting user interface elements | |
| CN117270719A (en) | Interface display method, device, electronic equipment and readable storage medium | |
| WO2023217177A1 (en) | Method for processing application window, and storage medium and intelligent interaction tablet computer |
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C17 | Cessation of patent right | ||
| CF01 | Termination of patent right due to non-payment of annual fee | Granted publication date:20120822 Termination date:20121216 |