技术领域Technical field
本申请涉及人工智能和RPA技术领域，尤其涉及一种与任务关联的界面元素的处理方法。This application relates to the technical fields of artificial intelligence and RPA, and in particular, to a method of processing interface elements associated with tasks.
背景技术Background technique
RPA(Robotic Process Automation，机器人流程自动化)是以软件机器人及AI(Artificial Intelligence，人工智能)为基础的业务过程自动化技术，RPA机器人可以通过模仿用户在电脑或移动端的手动操作方式，自动操作UI(User Interface，用户界面)来实现用户手动操作流程自动化。RPA (Robotic Process Automation) is a business process automation technology based on software robots and AI (Artificial Intelligence). RPA robots can automatically operate UI ( User Interface (User Interface) to automate the user's manual operation process.
针对某个待执行的任务，RPA机器人如何实现对与该任务关联的各个界面元素进行自动操作，以自动完成该任务，是非常重要的。For a certain task to be performed, it is very important for the RPA robot to automatically operate various interface elements associated with the task to automatically complete the task.
发明内容Contents of the invention
本申请旨在至少在一定程度上解决相关技术中的技术问题之一。The present application aims to solve, at least to a certain extent, one of the technical problems in the related art.
本申请提出一种与任务关联的界面元素的处理方法，以实现由元素识别方式链中的至少一个元素识别方式节点识别得到与目标任务关联的各目标界面元素的元素位置和元素类别，从而RPA机器人可以根据各目标界面元素的元素位置，对各目标界面元素执行与对应元素类别匹配的操作，以保证目标任务的有效执行，从而利于RPA技术的推广。This application proposes a method for processing interface elements associated with tasks to achieve the element position and element category of each target interface element associated with the target task through identification of at least one element identification method node in the element identification method chain, so that RPA The robot can perform operations on each target interface element that match the corresponding element category based on the element position of each target interface element to ensure the effective execution of the target task, thus facilitating the promotion of RPA technology.
本申请第一方面实施例提出了一种与任务关联的界面元素的处理方法，应用于RPA机器人，包括：The first embodiment of the present application proposes a method for processing interface elements associated with tasks, which is applied to RPA robots, including:
获取与待执行的目标任务关联的任务流程链，其中，所述任务流程链中包括至少一个任务流程节点；Obtain a task process chain associated with the target task to be executed, wherein the task process chain includes at least one task process node;
获取与所述任务流程链中的各所述任务流程节点所指示的目标应用界面中的目标界面元素对应的元素位置和元素类别；其中，所述目标界面元素的元素位置和元素类别是通过调用元素识别方式链中的至少一个元素识别方式节点识别得到的；Obtain the element position and element category corresponding to the target interface element in the target application interface indicated by each task process node in the task process chain; wherein, the element position and element category of the target interface element are obtained by calling Recognized by at least one element recognition mode node in the element recognition mode chain;
依次根据各所述任务流程节点所指示的目标界面元素的元素位置，对相应目标界面元素执行与所述相应目标界面元素的元素类别匹配的目标操作。In turn, according to the element position of the target interface element indicated by each of the task process nodes, a target operation matching the element category of the corresponding target interface element is performed on the corresponding target interface element.
本申请第二方面实施例提出了一种与任务关联的界面元素的处理装置，应用于RPA机器人，包括：The second embodiment of the present application proposes a processing device for interface elements associated with tasks, which is applied to RPA robots, including:
第一获取模块，用于获取与待执行的目标任务关联的任务流程链，其中，所述任务流程链中包括至少一个任务流程节点；A first acquisition module, configured to acquire a task process chain associated with the target task to be executed, where the task process chain includes at least one task process node;
第二获取模块，用于获取与所述任务流程链中的各所述任务流程节点所指示的目标应用界面中的目标界面元素对应的元素位置和元素类别；其中，所述目标界面元素的元素位置和元素类别是通过调用元素识别方式链中的至少一个元素识别方式节点识别得到的；The second acquisition module is used to acquire the element position and element category corresponding to the target interface element in the target application interface indicated by each task process node in the task process chain; wherein, the elements of the target interface element The position and element category are identified by calling at least one element identification method node in the element identification method chain;
执行模块，用于依次根据各所述任务流程节点所指示的目标界面元素的元素位置，对相应目标界面元素执行与所述相应目标界面元素的元素类别匹配的目标操作。An execution module, configured to execute a target operation matching the element category of the corresponding target interface element on the corresponding target interface element according to the element position of the target interface element indicated by each of the task process nodes.
本申请第三方面实施例提出了一种电子设备，包括：存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时，实现如本申请第一方面实施例提出的与任务关联的界面元素的处理方法。The third embodiment of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the application implements The first aspect embodiment provides a method for processing interface elements associated with tasks.
本申请第四方面实施例提出了一种非临时性计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如本申请第一方面实施例提出的与任务关联的界面元素的处理方法。The fourth embodiment of the present application proposes a non-transitory computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the interface associated with the task as proposed in the first embodiment of the present application is implemented. How to handle elements.
本申请第五方面实施例提出了一种计算机程序产品，当所述计算机程序产品中的指令由处理器执行时，执行如本申请第一方面实施例提出的与任务关联的界面元素的处理方法。The fifth embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor, the method for processing interface elements associated with tasks as proposed in the first embodiment of the present application is performed. .
本申请上述实施例提供的技术方案至少带来以下几方面的有益效果：The technical solutions provided by the above embodiments of the present application at least bring beneficial effects in the following aspects:
第一方面，通过链式方式，拾取界面元素，当链中的某个节点无法拾取到界面元素时，调用链中位于该节点之后的节点进行界面元素的拾取，一方面，通过链中的多个节点有序地拾取界面元素，可以提升元素拾取的有效性，从而保证RPA机器人执行任务的有效性，另一方面，当链中的某个节点拾取到界面元素后，停止调用链中位于该节点之后的节点进行界面元素的拾取，可以提升元素的拾取效率，降低计算资源的占用和处理负担。On the one hand, the interface elements are picked up through the chain method. When a node in the chain cannot pick up the interface elements, the node located after the node in the chain is called to pick up the interface elements. On the one hand, through multiple nodes in the chain, Nodes pick up interface elements in an orderly manner, which can improve the effectiveness of element picking, thereby ensuring the effectiveness of the RPA robot's task execution. On the other hand, when a node in the chain picks up an interface element, stop calling the node located in the chain. The nodes after the node are used to pick up interface elements, which can improve the picking efficiency of elements and reduce the occupation of computing resources and processing burden.
第二方面，对于链中的首个节点，可以调用与现有的每种操作系统、软件架构和开发语言匹配的拾取方法，来进行界面元素的拾取，可以提升元素拾取的有效性，以保证各种类型的设备上运行的RPA机器人均能够有效地获取界面元素的元素位置和元素类型，从而提升任务执行的有效性，利于RPA技术的推广。Secondly, for the first node in the chain, the picking method matching each existing operating system, software architecture and development language can be called to pick up interface elements, which can improve the effectiveness of element picking to ensure RPA robots running on various types of equipment can effectively obtain the element positions and element types of interface elements, thereby improving the effectiveness of task execution and conducive to the promotion of RPA technology.
第三方面，对于链中的第二个节点，可以基于图像之间画面内容的相似度、图像分类技术、OCR(Optical Character Recognition，光学字符识别)技术等多种图像识别技术，对应用界面上的界面元素进行识别，可以提升界面元素识别的准确性，从而在任务执行场景中，可以提升RPA机器人执行任务的准确性。Thirdly, for the second node in the chain, the application interface can be modified based on various image recognition technologies such as the similarity of the content between images, image classification technology, OCR (Optical Character Recognition, optical character recognition) technology, etc. Recognizing interface elements can improve the accuracy of interface element recognition, thereby improving the accuracy of RPA robot task execution in task execution scenarios.
第四方面，在界面元素的拾取场景中，基于链式结构和目标检测技术，来进行界面元素的拾取，以提升元素拾取的效率和准确性。The fourth aspect is that in the interface element picking scene, the interface elements are picked based on the chain structure and target detection technology to improve the efficiency and accuracy of element picking.
本申请附加的方面和优点将在下面的描述中部分给出，部分将从下面的描述中变得明显，或通过本申请的实践了解到。Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.
附图说明Description of the drawings
本申请上述的和/或附加的方面和优点从下面结合附图对实施例的描述中将变得明显和容易理解，其中：The above and/or additional aspects and advantages of the present application will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
图1为本申请实施例所提供的一种与任务关联的界面元素的处理方法的流程示意图；Figure 1 is a schematic flowchart of a method for processing interface elements associated with tasks provided by an embodiment of the present application;
图2为本申请实施例所提供的另一种与任务关联的界面元素的处理方法的流程示意图；Figure 2 is a schematic flowchart of another method for processing interface elements associated with tasks provided by an embodiment of the present application;
图3为本申请实施例所提供的另一种与任务关联的界面元素的处理方法的流程示意图；Figure 3 is a schematic flowchart of another method for processing interface elements associated with tasks provided by an embodiment of the present application;
图4为本申请实施例所提供的另一种与任务关联的界面元素的处理方法的流程示意图；Figure 4 is a schematic flowchart of another method for processing interface elements associated with tasks provided by an embodiment of the present application;
图5为本申请实施例所提供的另一种与任务关联的界面元素的处理方法的流程示意图；Figure 5 is a schematic flowchart of another method for processing interface elements associated with tasks provided by an embodiment of the present application;
图6为本申请实施例所提供的另一种与任务关联的界面元素的处理方法的流程示意图；Figure 6 is a schematic flowchart of another method for processing interface elements associated with tasks provided by an embodiment of the present application;
图7(a)为本申请实施例所提供的元素捕获或拾取方式示意图；Figure 7(a) is a schematic diagram of an element capturing or picking method provided by an embodiment of the present application;
图7(b)为本申请实施例所提供的元素解析和操作方式示意图；Figure 7(b) is a schematic diagram of element analysis and operation methods provided by the embodiment of the present application;
图8为本申请实施例所提供的一种与任务关联的界面元素的处理装置的结构示意图；Figure 8 is a schematic structural diagram of a device for processing interface elements associated with tasks provided by an embodiment of the present application;
图9是本申请一示例性实施例所示出的电子设备的结构示意图。FIG. 9 is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present application.
具体实施方式Detailed ways
下面详细描述本申请的实施例，所述实施例的示例在附图中示出，其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的，旨在用于解释本申请，而不能理解为对本申请的限制。The embodiments of the present application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present application, but should not be construed as limiting the present application.
目前，RPA机器人可以监视使用者对应用软件中的GUI所进行的工作，并且直接在GUI上自动重复这些工作。但是，UI不仅涉及传统的B/S(Browser/Server，浏览器/服务器)、C/S(Client/Server，客户端/服务器)等软件架构，还涉及C、C++、Python、java等语言架构的应用，同时还涉及到不同的操作系统(如电脑端的Windows、Linux等操作系统、移动端的安卓Android、IOS等操作系统)下不同架构的应用，RPA机器人的开发者需要花费大量的时间对不同架构下的应用进行适配，且很难做到完全适配。Currently, RPA robots can monitor the work performed by users on the GUI in application software and automatically repeat these tasks directly on the GUI. However, UI not only involves traditional B/S (Browser/Server, browser/server), C/S (Client/Server, client/server) and other software architectures, but also involves language architectures such as C, C++, Python, and Java. The application also involves the application of different architectures under different operating systems (such as Windows, Linux and other operating systems on the computer side, Android, IOS and other operating systems on the mobile side). The developers of RPA robots need to spend a lot of time on different Adapt to the applications under the architecture, and it is difficult to achieve complete adaptation.
并且，目前RPA机器人的厂商无法做到针对不同应用进行不同元素拾取方法的自主切换，这就使RPA机器人的应用存在人员限制，即，必须拥有开发经验的人员才能对RPA机器人进行操作，与RPA机器人能够针对所有人员使用的初衷不一致。Moreover, currently the manufacturers of RPA robots are unable to autonomously switch different element picking methods for different applications, which places personnel restrictions on the application of RPA robots. That is, only personnel with development experience can operate RPA robots, which is different from RPA robots. The original intention of robots being used by all people is inconsistent.
同时，还存在部分应用和界面元素使用传统的元素拾取方法拾取不到的情况，导致RPA流程自动化很难落地，严重阻碍了RPA技术的推广。At the same time, there are still cases where some applications and interface elements cannot be picked up using traditional element picking methods, making it difficult to implement RPA process automation and seriously hindering the promotion of RPA technology.
针对上述存在的至少一项问题，本申请提出一种与任务关联的界面元素的处理方法。To address at least one of the above existing problems, this application proposes a method for processing interface elements associated with tasks.
下面参考附图描述本申请实施例的与任务关联的界面元素的处理方法及其装置。在具体描述本申请实施例之前，为了便于理解，首先对常用技术词进行介绍：The method and device for processing interface elements associated with tasks according to the embodiment of the present application will be described below with reference to the accompanying drawings. Before describing the embodiments of the present application in detail, in order to facilitate understanding, common technical terms are first introduced:
目标检测模型，是指采用目标检测算法生成的模型。其中，目标检测算法包括但不限于：R-CNN(Region with CNN(Convolutional Neural Network，卷积神经网络)，区域卷积神经网络)、Faster R-CNN(快速区域卷积神经网络)、YOLO(You Only Look Once，你只看一眼)、SSD(Single Shot MultiBox Detector，单步多框目标检测)等等。The target detection model refers to the model generated using the target detection algorithm. Among them, target detection algorithms include but are not limited to: R-CNN (Region with CNN (Convolutional Neural Network, convolutional neural network), regional convolutional neural network), Faster R-CNN (fast regional convolutional neural network), YOLO ( You Only Look Once, you only take a look), SSD (Single Shot MultiBox Detector, single-step multi-frame target detection), etc.
目标应用，可以为运行在RPA机器人所在设备中的任意一个软件应用，如浏览器等。The target application can be any software application running on the device where the RPA robot is located, such as a browser.
操作系统信息，用于指示目标应用的软件运行环境，包括但不限于：Windows、Linux、Ubuntu、Pks、Android、IOS等操作系统。Operating system information is used to indicate the software running environment of the target application, including but not limited to: Windows, Linux, Ubuntu, Pks, Android, IOS and other operating systems.
软件架构信息，用于指示目标应用的软件架构，包括但不限于：B/S、C/S等。Software architecture information is used to indicate the software architecture of the target application, including but not limited to: B/S, C/S, etc.
开发语言信息，用于指示目标应用的开发语言，包括但不限于：Java、C、C++、Python等开发语言。Development language information is used to indicate the development language of the target application, including but not limited to: Java, C, C++, Python and other development languages.
图1为本申请实施例所提供的一种与任务关联的界面元素的处理方法的流程示意图。FIG. 1 is a schematic flowchart of a method for processing interface elements associated with tasks provided by an embodiment of the present application.
本申请实施例所提供的与任务关联的界面元素的处理方法，可以由RPA机器人执行。The processing method of interface elements associated with tasks provided by the embodiments of the present application can be executed by an RPA robot.
其中，RPA机器人可以运行在任一具有计算能力的电子设备中，其中，电子设备。例如可以为个人电脑、移动终端、服务器等，移动终端例如可以为车载设备、手机、平板电脑、个人数字助理、穿戴式设备等具有各种操作系统、触摸屏和/或显示屏的硬件设备。Among them, RPA robots can run in any electronic device with computing capabilities, including electronic devices. For example, it can be a personal computer, a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device and other hardware devices with various operating systems, touch screens and/or display screens.
如图1所示，该与任务关联的界面元素的处理方法可以包括以下步骤：As shown in Figure 1, the method for processing the interface elements associated with the task may include the following steps:
步骤S101，获取与待执行的目标任务关联的任务流程链，其中，任务流程链中包括至少一个任务流程节点。Step S101: Obtain a task flow chain associated with the target task to be executed, where the task flow chain includes at least one task flow node.
在本申请实施例中，目标任务为RPA机器人需要执行的任务。In the embodiment of this application, the target task is a task that the RPA robot needs to perform.
在本申请实施例中，目标任务具有关联的任务流程链，该任务流程链中包括至少一个任务流程节点，其中，每个任务流程节点用于对一个界面元素执行与该界面元素的元素类别匹配的操作。In this embodiment of the present application, the target task has an associated task process chain, and the task process chain includes at least one task process node, wherein each task process node is used to perform matching on an interface element with the element category of the interface element. operation.
其中，界面元素或元素类别包括但不限于应用界面上的按钮、文本、文本框(或输入框)、单选框、复选框等等。Among them, interface elements or element categories include but are not limited to buttons, text, text boxes (or input boxes), radio buttons, check boxes, etc. on the application interface.
在本申请实施例中，用户或相关人员可以针对目标任务，预先对与该目标任务关联的各界面元素(本申请中记为目标界面元素)进行人工操作，本申请中，可以记录用户的相关操作，以得到与该目标任务关联的任务流程链。In the embodiment of this application, the user or relevant personnel can perform manual operations on each interface element associated with the target task (referred to as the target interface element in this application) in advance for the target task. In this application, the user's relevant information can be recorded. Operation to get the task process chain associated with the target task.
举例而言，假设完成目标任务，需要依次对三个目标界面元素执行相应操作，比如，首先对目标界面元素1(例如文本框1)执行文本写入操作，之后，对目标界面元素2(例如单选框2)执行单击操作，最后，对目标界面元素3(例如按钮3，如“确认”按钮)执行单击操作，则任务流程链中包括按序排列的三个任务流程节点，第一个任务流程节点用于对目标界面元素1执行操作，第二个任务流程节点用于对目标界面元素2执行操作，第三个任务流程节点用于对目标界面元素1执行操作。For example, assuming that to complete the target task, you need to perform corresponding operations on three target interface elements in sequence. For example, first perform a text writing operation on target interface element 1 (such as text box 1), and then perform a text writing operation on target interface element 2 (such as text box 1). Radio button 2) performs a click operation, and finally, performs a click operation on the target interface element 3 (such as button 3, such as the "Confirm" button), then the task process chain includes three task process nodes arranged in sequence. One task process node is used to perform operations on target interface element 1, the second task process node is used to perform operations on target interface element 2, and the third task process node is used to perform operations on target interface element 1.
步骤S102，获取与任务流程链中的各任务流程节点所指示的目标应用界面中的目标界面元素对应的元素位置和元素类别；其中，目标界面元素的元素位置和元素类别是通过调用元素识别方式链中的至少一个元素识别方式节点识别得到的。Step S102, obtain the element position and element category corresponding to the target interface element in the target application interface indicated by each task process node in the task process chain; wherein, the element position and element category of the target interface element are obtained by calling the element identification method At least one element in the chain is identified by the node identified by it.
其中，元素位置用于指示目标界面元素在目标应用界面中的位置，例如，目标界面元素的捕获路径，即DOM(Document Object Model，文档对象模型)路径，再如，目标界面元素在目标应用界面中的坐标位置等，本申请对此并不做限制。Among them, the element position is used to indicate the position of the target interface element in the target application interface. For example, the capture path of the target interface element, that is, the DOM (Document Object Model, Document Object Model) path. For another example, the target interface element is in the target application interface. The coordinate position, etc. in this application does not limit this.
在本申请实施例中，不同任务流程节点所指示的目标应用界面可以相同，或者也可以不同，本申请对此并不作限制，并且，不同任务流程节点所指示的目标界面元素可以相同，或者也可以不同，本申请对此并不作限制。In the embodiment of the present application, the target application interfaces indicated by different task process nodes may be the same, or may be different. This application does not limit this, and the target interface elements indicated by different task process nodes may be the same, or may be different. It may be different, and this application does not limit this.
在本申请实施例中，元素识别方式链中包括至少一个元素识别方式节点，其中，每个元素识别方式节点用于采用对应的元素识别方式(或称为元素拾取方式、元素识别方法、元素拾取方法)，对应用界面(本申请中记为目标应用界面)中的目标界面元素进行识别或拾取，以得到目标界面元素对应的元素位置和元素类别。In the embodiment of the present application, the element identification method chain includes at least one element identification method node, wherein each element identification method node is used to adopt the corresponding element identification method (also known as element picking method, element identification method, element picking method). Method), identify or pick up the target interface element in the application interface (referred to as the target application interface in this application) to obtain the element position and element category corresponding to the target interface element.
在本申请实施例中，用户或相关人员可以针对目标任务，预先对与该目标任务关联的各目标界面元素进行人工操作，本申请中，可以记录用户的相关操作，并调用元素识别方式链中的至少一个元素识别方式节点对用户触发的各目标界面元素进行识别或拾取，以得到各目标界面元素对应的元素位置和元素类别，并记录或存储各目标界面元素对应的元素位置和元素类别。In the embodiment of this application, the user or relevant personnel can manually operate each target interface element associated with the target task in advance for the target task. In this application, the user's relevant operations can be recorded and the element identification method chain can be called. At least one element identification mode node identifies or picks up each target interface element triggered by the user to obtain the element position and element category corresponding to each target interface element, and records or stores the element position and element category corresponding to each target interface element.
从而本申请中，可以查询记录数据或存储数据，以获取与任务流程链中的各任务流程节点所指示的目标应用界面中的目标界面元素对应的元素位置和元素类别。Therefore, in this application, the recorded data or stored data can be queried to obtain the element position and element category corresponding to the target interface element in the target application interface indicated by each task process node in the task process chain.
步骤S103，依次根据各任务流程节点所指示的目标界面元素的元素位置，对相应目标界面元素执行与相应目标界面元素的元素类别匹配的目标操作。Step S103: According to the element position of the target interface element indicated by each task process node, a target operation matching the element type of the corresponding target interface element is performed on the corresponding target interface element.
其中，目标操作包括但不限于：单击、悬停、文本读取、文本写入、文本删除等操作。Among them, the target operations include but are not limited to: click, hover, text reading, text writing, text deletion and other operations.
在本申请实施例中，RPA机器人可以依次根据任务流程节点中各任务流程节点所指示的目标界面元素的元素位置，定位相应的目标界面元素，并对相应的目标界面元素执行与该相应的目标界面元素的元素类别匹配的目标操作。In the embodiment of this application, the RPA robot can sequentially locate the corresponding target interface element according to the element position of the target interface element indicated by each task process node in the task process node, and execute the corresponding target interface element on the corresponding target interface element. The interface element's element category matches the target operation.
本申请实施例的与任务关联的界面元素的处理方法，通过RPA机器人获取与待执行的目标任务关联的任务流程链，其中，任务流程链中包括至少一个任务流程节点；获取与任务流程链中的各任务流程节点所指示的目标应用界面中的目标界面元素对应的元素位置和元素类别；其中，目标界面元素的元素位置和元素类别是通过调用元素识别方式链中的至少一个元素识别方式节点识别得到的；依次根据各任务流程节点所指示的目标界面元素的元素位置，对相应目标界面元素执行与相应目标界面元素的元素类别匹配的目标操作。由此，可以实现由元素识别方式链中的至少一个元素识别方式节点识别得到与目标任务关联的各目标界面元素的元素位置和元素类别，从而RPA机器人可以根据各目标界面元素的元素位置，对各目标界面元素执行与对应元素类别匹配的操作，以保证目标任务的有效执行，从而利于RPA技术的推广。The method for processing interface elements associated with tasks in the embodiment of the present application uses the RPA robot to obtain the task process chain associated with the target task to be executed, where the task process chain includes at least one task process node; the task process chain is obtained by The element position and element category corresponding to the target interface element in the target application interface indicated by each task process node; wherein, the element position and element category of the target interface element are determined by calling at least one element identification method node in the element identification method chain Recognized; according to the element position of the target interface element indicated by each task process node, perform the target operation matching the element category of the corresponding target interface element on the corresponding target interface element. As a result, the element position and element category of each target interface element associated with the target task can be obtained by identifying at least one element recognition method node in the element recognition method chain, so that the RPA robot can identify the element position and element category of each target interface element based on the element position of each target interface element. Each target interface element performs operations that match the corresponding element category to ensure the effective execution of the target task, thereby facilitating the promotion of RPA technology.
为了清楚说明本申请上述实施例中是如何调用元素识别方式链中的至少一个元素识别方式节点识别各目标界面元素的元素位置和元素类别的，本申请还提出一种与任务关联的界面元素的处理方法。In order to clearly explain how in the above embodiments of the present application, at least one element identification method node in the element identification method chain is called to identify the element position and element category of each target interface element, the present application also proposes an interface element associated with a task. Approach.
图2为本申请实施例所提供的另一种与任务关联的界面元素的处理方法的流程示意图。FIG. 2 is a schematic flowchart of another method for processing interface elements associated with tasks provided by an embodiment of the present application.
如图2所示，在图1所示实施例的基础上，任务流程链中的任意一个任务流程节点所指示的目标应用界面中的目标界面元素的元素位置和元素类别，可以采用以下步骤识别得到：As shown in Figure 2, based on the embodiment shown in Figure 1, the element position and element category of the target interface element in the target application interface indicated by any task process node in the task process chain can be identified using the following steps get:
步骤S201，调用元素识别方式链中的至少一个元素识别方式节点，对任一任务流程节点所指示的目标应用界面中的目标界面元素进行识别，以得到各元素识别方式节点的识别状态和识别结果。Step S201: Call at least one element recognition mode node in the element recognition mode chain to identify the target interface element in the target application interface indicated by any task process node to obtain the recognition status and recognition result of each element recognition mode node. .
其中，识别状态包括但不限于：识别成功状态、识别失败状态、未识别状态等。Among them, the recognition status includes but is not limited to: recognition success status, recognition failure status, unrecognition status, etc.
其中，识别结果至少包括元素位置和元素类别。Among them, the recognition result includes at least element position and element category.
在本申请实施例中，针对任务流程链中的任意一个任务流程节点，可预先采用设计器调用元素识别方式链中的至少一个元素识别方式节点，对该任务流程节点所指示的目标应用界面中的目标界面元素进行识别，以得到各元素识别方式节点的识别状态和识别结果。In the embodiment of this application, for any task process node in the task process chain, the designer can be used to call at least one element recognition mode node in the element recognition mode chain in advance, and the target application interface indicated by the task process node The target interface elements are identified to obtain the recognition status and recognition results of each element recognition mode node.
在本申请实施例的一种可能的实现方式中，元素识别方式链的调用方式，例如为：In a possible implementation manner of the embodiment of this application, the calling method of the element identification method chain is, for example:
1、首先，调用元素识别方式链中的首个元素识别方式节点对该任务流程节点所指示的目标应用界面中的目标界面元素进行识别，如果该首个元素识别方式节点的识别状态为识别成功状态，则停止调用该元素识别方式链中的后续元素识别方式节点，并根据该首个元素识别方式节点的识别结果，确定该任务流程节点所指示的目标界面元素的元素位置和元素类别。1. First, call the first element identification method node in the element identification method chain to identify the target interface element in the target application interface indicated by the task process node. If the identification status of the first element identification method node is identification success status, stop calling subsequent element recognition mode nodes in the element recognition mode chain, and determine the element position and element category of the target interface element indicated by the task process node based on the recognition result of the first element recognition mode node.
2、如果该首个元素识别方式节点的识别状态为识别失败状态，则继续调用元素识别方式链中的第二个元素识别方式节点对该任务流程节点所指示的目标应用界面中的目标界面元素进行识别，如果该第二个元素识别方式的识别状态为识别成功状态，则停止调用该元素识别方式链中的后续元素识别方式节点，并根据该第二个元素识别方式节点的识别结果，确定该任务流程节点所指示的目标界面元素的元素位置和元素类别。2. If the recognition status of the first element recognition method node is a recognition failure state, continue to call the second element recognition method node in the element recognition method chain to the target interface element in the target application interface indicated by the task process node. Recognition is performed. If the recognition status of the second element recognition method is the recognition success status, stop calling subsequent element recognition method nodes in the element recognition method chain, and determine based on the recognition result of the second element recognition method node. The element position and element category of the target interface element indicated by this task process node.
3、如果该第二个元素识别方式节点的识别状态为识别失败状态，则继续调用元素识别方式链中的第三个元素识别方式节点对该任务流程节点所指示的目标应用界面中的目标界面元素进行识别，直到确定某个元素识别方式节点的识别状态为识别成功状态时，停止调用该元素识别方式链，并根据该识别成功的元素识别方式节点的识别结果，确定该任务流程节点所指示的目标界面元素的元素位置和元素类别。3. If the recognition status of the second element recognition method node is a recognition failure state, continue to call the target interface in the target application interface indicated by the third element recognition method node in the element recognition method chain for the task process node. The element is recognized until it is determined that the recognition status of an element recognition method node is the recognition success status, stop calling the element recognition method chain, and determine the task process node indicated by the recognition result of the successfully recognized element recognition method node. The element position and element category of the target interface element.
步骤S202，从各元素识别方式节点中，确定识别状态为识别成功状态的目标元素识别方式节点。Step S202: From each element recognition mode node, determine the target element recognition mode node whose recognition status is the recognition success state.
在本申请实施例中，可以根据各元素识别方式节点的识别状态，从各元素识别方式节点中确定目标识别方式节点，其中，目标识别方式节点的识别状态为识别成功状态。In the embodiment of the present application, the target recognition mode node can be determined from each element recognition mode node according to the recognition status of each element recognition mode node, where the recognition status of the target recognition mode node is a recognition success status.
步骤S203，根据目标元素识别方式节点的识别结果，确定任一任务流程节点所指示的目标界面元素的元素位置和元素类别。Step S203: Determine the element position and element type of the target interface element indicated by any task process node according to the recognition result of the target element recognition mode node.
在本申请实施例中，可以根据目标元素识别方式节点的识别结果，确定上述任务流程节点所指示的目标界面元素的元素位置和元素类别。In this embodiment of the present application, the element position and element category of the target interface element indicated by the above task process node can be determined according to the recognition result of the target element recognition mode node.
本申请实施例的与任务关联的界面元素的处理方法，根据元素识别方式链中识别成功的目标识别方式节点的识别结果，确定任务流程节点所指示的目标界面元素的元素位置和元素类别，可以提升确定结果的准确性。The method for processing interface elements associated with a task in the embodiment of the present application determines the element position and element category of the target interface element indicated by the task process node according to the identification result of the successfully identified target identification method node in the element identification method chain. Improve the accuracy of determined results.
作为本申请实施例的一种可能的实现方式，以元素识别方式链中包括三个节点，分别为拾取匹配节点、图像识别节点和目标检测节点进行示例，为了清楚说明本申请上述实施例中步骤S201是如何调用元素识别方式链中的各元素识别方式节点，对任务流程节点所指示的目标应用界面中的目标界面元素进行识别的，本申请还提出一种与任务关联的界面元素的处理方法。As a possible implementation of the embodiment of the present application, the element recognition method chain includes three nodes, respectively, as an example, a picking and matching node, an image recognition node and a target detection node. In order to clearly explain the steps in the above embodiment of the present application. How does S201 call each element identification mode node in the element identification mode chain to identify the target interface element in the target application interface indicated by the task process node? This application also proposes a method for processing interface elements associated with the task. .
图3为本申请实施例所提供的另一种与任务关联的界面元素的处理方法的流程示意图。FIG. 3 is a schematic flowchart of another method for processing interface elements associated with tasks provided by an embodiment of the present application.
如图3所示，在图2所示实施例的基础上，步骤S201可以包括以下步骤：As shown in Figure 3, based on the embodiment shown in Figure 2, step S201 may include the following steps:
步骤S301，调用拾取匹配节点对任一任务流程节点所指示的目标应用界面中的目标界面元素进行拾取，并获取拾取匹配节点返回的第一反馈信息；其中，第一反馈信息中包括拾取匹配节点的识别状态。Step S301: Call the pick-up matching node to pick up the target interface elements in the target application interface indicated by any task process node, and obtain the first feedback information returned by the pick-up matching node; wherein the first feedback information includes the pick-up matching node identification status.
在本申请实施例中，针对任务流程链中的任意一个任务流程节点，首先可以调用拾取匹配节点对该任务流程节点所指示的目标应用界面中的目标界面元素进行拾取，并获取拾取匹配节点返回的第一反馈信息。In the embodiment of the present application, for any task process node in the task process chain, the picking matching node can first be called to pick up the target interface element in the target application interface indicated by the task process node, and the picking matching node is returned. first feedback information.
在本申请实施例的一种可能的实现方式中，第一反馈信息中还可以包括拾取匹配节点的识别结果，当第一反馈信息中的识别状态为识别失败状态时，第一反馈信息中的识别结果为空。In a possible implementation manner of the embodiment of the present application, the first feedback information may also include the identification result of picking up the matching node. When the identification status in the first feedback information is the identification failure status, the first feedback information may The recognition result is empty.
而当第一反馈信息中的识别状态为识别成功状态时，第一反馈信息中的识别结果至少包括该任务流程节点所指示的目标界面元素的元素位置和元素类别。When the recognition status in the first feedback information is a recognition success status, the recognition result in the first feedback information at least includes the element position and element category of the target interface element indicated by the task process node.
本申请中，在第一反馈信息中的识别状态为识别成功状态时，可以根据第一反馈信息中的识别结果，确定该任务流程节点所指示的目标界面元素的元素位置和元素类别，并且，可以停止调用元素识别方式链中的图像识别节点和目标检测节点，并确定图像识别节点和目标检测节点的识别结果均为空，以及确定图像识别节点和目标检测节点的识别状态均为未识别状态。In this application, when the recognition status in the first feedback information is the recognition success status, the element position and element category of the target interface element indicated by the task process node can be determined based on the recognition result in the first feedback information, and, You can stop calling the image recognition node and target detection node in the element recognition method chain, and determine that the recognition results of the image recognition node and target detection node are both empty, and determine that the recognition status of the image recognition node and target detection node are both unrecognized. .
而在第一反馈信息中的识别状态为识别失败状态时，为了保证目标界面元素的有效拾取，可以继续调用元素识别方式链中的图像识别节点对该任务流程节点所指示的目标应用界面中的目标界面元素进行图像识别。When the recognition status in the first feedback information is a recognition failure status, in order to ensure the effective pickup of the target interface element, you can continue to call the image recognition node in the element recognition method chain in the target application interface indicated by the task process node. Target interface elements for image recognition.
步骤S302，在第一反馈信息中的识别状态为识别失败状态时，调用图像识别节点对任一任务流程节点所指示的目标应用界面中的目标界面元素进行图像识别，并获取图像识别节点返回的第二反馈信息；其中，第二反馈信息中包括图像识别节点的识别状态。Step S302: When the recognition status in the first feedback information is a recognition failure status, call the image recognition node to perform image recognition on the target interface element in the target application interface indicated by any task process node, and obtain the image recognition node returned by the image recognition node. Second feedback information; wherein the second feedback information includes the recognition status of the image recognition node.
在本申请实施例中，在第一反馈信息中的识别状态为识别失败状态时，可以继续调用元素识别方式链中的图像识别节点对上述任务流程节点所指示的目标应用界面中的目标界面元素进行图像识别，并获取图像识别节点返回的第二反馈信息。In this embodiment of the present application, when the recognition status in the first feedback information is a recognition failure status, the image recognition node in the element recognition method chain can be continued to be called to target the target interface element in the target application interface indicated by the above task process node. Perform image recognition and obtain the second feedback information returned by the image recognition node.
在本申请实施例的一种可能的实现方式中，第二反馈信息中还可以包括图像识别节点的识别结果，当第二反馈信息中的识别状态为识别失败状态时，第二反馈信息中的识别结果为空。In a possible implementation manner of the embodiment of this application, the second feedback information may also include the recognition result of the image recognition node. When the recognition status in the second feedback information is the recognition failure status, the second feedback information may also include the recognition result of the image recognition node. The recognition result is empty.
而当第二反馈信息中的识别状态为识别成功状态时，第二反馈信息中的识别结果至少包括该任务流程节点所指示的目标界面元素的元素位置和元素类别。When the recognition status in the second feedback information is a recognition success status, the recognition result in the second feedback information at least includes the element position and element category of the target interface element indicated by the task process node.
本申请中，在第二反馈信息中的识别状态为识别成功状态时，可以根据第二反馈信息中的识别结果，确定该任务流程节点所指示的目标界面元素的元素位置和元素类别，并且，可以停止调用元素识别方式链中的目标检测节点，并确定目标检测节点的识别结果为空，以及确定目标检测节点的识别状态为未识别状态。In this application, when the recognition status in the second feedback information is the recognition success status, the element position and element category of the target interface element indicated by the task process node can be determined based on the recognition result in the second feedback information, and, You can stop calling the target detection node in the element recognition mode chain, and determine that the recognition result of the target detection node is empty, and determine that the recognition status of the target detection node is an unrecognized state.
而在第二反馈信息中的识别状态为识别失败状态时，为了保证目标界面元素的有效拾取，可以继续调用元素识别方式链中的目标检测节点对该任务流程节点所指示的目标应用界面中的目标界面元素进行目标检测。When the recognition status in the second feedback information is a recognition failure status, in order to ensure the effective pickup of the target interface element, the target detection node in the element recognition method chain can continue to be called in the target application interface indicated by the task process node. Target interface elements for target detection.
步骤S303，在第二反馈信息中的识别状态为识别失败状态时，调用目标检测节点，对任一任务流程节点所指示的目标应用界面中的目标界面元素进行目标检测，得到目标检测节点返回的第三反馈信息。Step S303: When the recognition status in the second feedback information is a recognition failure status, call the target detection node to perform target detection on the target interface element in the target application interface indicated by any task process node, and obtain the target interface element returned by the target detection node. Third feedback information.
在本申请实施例中，在第二反馈信息中的识别状态为识别失败状态时，可以继续调用元素识别方式链中的目标检测节点对该任务流程节点所指示的目标应用界面中的目标界面元素进行目标检测，以得到目标检测节点返回的第三反馈信息。其中，第三反馈信息中可以包括目标检测节点的识别结果和识别状态。In this embodiment of the present application, when the recognition status in the second feedback information is a recognition failure status, the target interface element in the target application interface indicated by the task process node by the target detection node in the element recognition method chain can be continued. Target detection is performed to obtain third feedback information returned by the target detection node. The third feedback information may include the recognition result and recognition status of the target detection node.
在本申请实施例的与任务关联的界面元素的处理方法，可以实现调用多个元素识别方式节点对任务流程节点所指示的目标界面元素进行识别，即采用多种元素识别(或拾取、捕获)方式，对任务流程节点所指示的目标界面元素进行识别，可以避免目标界面元素无法识别或捕获的情况发生，提升目标界面元素识别的有效性和可靠性，从而有利于RPA技术的推广。In the processing method of interface elements associated with tasks in the embodiment of the present application, multiple element identification mode nodes can be called to identify the target interface element indicated by the task process node, that is, multiple element identification (or picking, capturing) In this way, identifying the target interface elements indicated by the task process nodes can avoid the situation where the target interface elements cannot be identified or captured, improve the effectiveness and reliability of the target interface element identification, and thus be conducive to the promotion of RPA technology.
为了清楚说明本申请任一实施例中拾取匹配节点是如何对任务流程节点所指示的目标应用界面中的目标界面元素进行拾取的，本申请还提出一种与任务关联的界面元素的处理方法。In order to clearly explain how the picking matching node in any embodiment of the present application picks up the target interface element in the target application interface indicated by the task process node, the present application also proposes a method for processing interface elements associated with the task.
图4为本申请实施例所提供的另一种与任务关联的界面元素的处理方法的流程示意图。FIG. 4 is a schematic flowchart of another method for processing interface elements associated with tasks provided by an embodiment of the present application.
如图4所示，在图3所示实施例的基础上，拾取匹配节点可以采用以下步骤，对任一任务流程节点所指示的目标应用界面中的目标界面元素进行拾取：As shown in Figure 4, based on the embodiment shown in Figure 3, the following steps can be used to pick up the matching node to pick up the target interface element in the target application interface indicated by any task process node:
步骤S401，获取RPA机器人所在设备的操作系统信息，并获取任一任务流程节点所指示的目标应用界面所属的目标应用的软件架构信息和开发语言信息。Step S401: Obtain the operating system information of the device where the RPA robot is located, and obtain the software architecture information and development language information of the target application to which the target application interface indicated by any task process node belongs.
在本申请实施例中，针对任意一个任务流程节点，可以分别获取RPA机器人所在设备的操作系统信息、该任务流程节点所指示的目标应用界面所属的目标应用的软件架构信息、目标应用的开发语言信息。In the embodiment of this application, for any task process node, the operating system information of the device where the RPA robot is located, the software architecture information of the target application to which the target application interface indicated by the task process node belongs, and the development language of the target application can be obtained respectively. information.
步骤S402，查询与操作系统信息、软件架构信息和开发语言信息匹配的元素拾取方法。Step S402: Query the element picking method that matches the operating system information, software architecture information, and development language information.
在本申请实施例中，可以预先针对不同的操作系统、软件架构和开发语言，开发或设计相应的元素拾取(或捕获)方法，并保存不同的操作系统、软件架构、开发语言和元素拾取方法之间的对应关系，从而本申请中，可以根据步骤S401中获取的操作系统信息、软件架构信息和开发语言信息，查询上述对应关系，以获取与该操作系统信息、软件架构信息和开发语言信息对应或匹配的元素拾取方法。In the embodiment of the present application, corresponding element picking (or capturing) methods can be developed or designed in advance for different operating systems, software architectures, and development languages, and different operating systems, software architectures, development languages, and element picking methods can be saved. Therefore, in this application, the above corresponding relationship can be queried based on the operating system information, software architecture information and development language information obtained in step S401 to obtain the operating system information, software architecture information and development language information. The corresponding or matching element picking method.
步骤S403，调用元素拾取方法对任一任务流程节点所指示的目标应用界面中的各界面元素进行拾取，以得到拾取匹配节点的识别状态和识别结果。Step S403: Call the element picking method to pick up each interface element in the target application interface indicated by any task process node, so as to obtain the recognition status and recognition result of the picked matching node.
在本申请实施例中，可以调用上述元素拾取方法，对该任务流程节点所指示的目标应用界面中的各界面元素进行拾取，以得到拾取匹配节点的识别状态和识别结果。In this embodiment of the present application, the above element picking method can be called to pick up each interface element in the target application interface indicated by the task process node, so as to obtain the identification status and identification result of the picked matching node.
作为一种可能的实现方式，可以采用以下步骤，确定拾取匹配节点的识别状态和识别结果：As a possible implementation, the following steps can be used to determine the recognition status and recognition results of the picked matching nodes:
1、调用上述元素拾取方法对该任务流程节点所指示的目标应用界面中的各界面元素进行拾取，并判断元素拾取方法是否拾取到各界面元素的元素定位信息，若是，则执行步骤2至步骤3，若否，则执行步骤4。1. Call the above element picking method to pick up each interface element in the target application interface indicated by the task process node, and determine whether the element picking method has picked up the element positioning information of each interface element. If so, perform steps 2 to 2. 3. If not, go to step 4.
其中，元素定位信息包括但不限于：界面元素的捕获路径(如DOM路径)、属性(包括但不限于元素标识(如ID、名称等)、元素类别等)等定位操作信息。Among them, element positioning information includes but is not limited to: positioning operation information such as the capture path (such as DOM path), attributes (including but not limited to element identification (such as ID, name, etc.), element category, etc.) of the interface element.
2、在拾取到各界面元素的元素定位信息的情况下，可以根据各界面元素的元素定位信息和与该任务流程节点关联的触控点(包括但不限于用户手指触摸的位置点、鼠标点击的位置点、鼠标或光标所在的位置点等)的位置信息，确定该任务流程节点所指示的目标界面元素的元素位置和元素类别。2. When the element positioning information of each interface element is picked up, the element positioning information of each interface element and the touch point associated with the task process node (including but not limited to the position point touched by the user's finger, mouse click position point, the position point of the mouse or cursor, etc.) to determine the element position and element category of the target interface element indicated by the task process node.
举例而言，针对任务流程链中的每个任务流程节点，可以通过人工触发鼠标指定或选择与该任务流程节点关联的目标界面元素，相应的，本申请中，可以根据鼠标位置，从各界面元素中确定与该任务流程节点关联的目标界面元素，并根据目标界面元素的元素定位信息，确定该目标界面元素的元素位置和元素类别。For example, for each task process node in the task process chain, you can manually trigger the mouse to specify or select the target interface element associated with the task process node. Correspondingly, in this application, you can select from each interface based on the mouse position. The target interface element associated with the task process node is determined among the elements, and the element position and element category of the target interface element are determined based on the element positioning information of the target interface element.
3、根据该任务流程节点所指示的目标界面元素的元素位置和元素类别，生成拾取匹配节点的识别结果，并生成拾取匹配节点的识别状态为识别成功状态。3. Based on the element position and element category of the target interface element indicated by the task process node, generate the recognition result of the picked-up matching node, and generate the recognition status of the picked-up matching node as the recognition success status.
进一步地，在拾取匹配节点的识别状态为识别成功状态时，还可以对目标应用界面上拾取匹配节点拾取得到的目标界面元素进行标注(如高亮显示该界面元素、元素类别和元素位置等，并由用户确定拾取匹配节点的识别结果是否正确，在正确的情况下，才记录或保存该拾取匹配节点的识别结果，而在不正确的情况下，可以由人工对拾取匹配节点的识别结果进行修正，或者，可以对上述元素拾取方法进行修正，以使拾取匹配节点的识别结果正确无误。Further, when the recognition status of the picked matching node is the recognition success state, the target interface elements picked up by the picked matching node on the target application interface can also be marked (such as highlighting the interface element, element category and element position, etc., And the user determines whether the identification result of the picked matching node is correct. If it is correct, the identification result of the picked matching node will be recorded or saved. If it is incorrect, the identification result of the picked matching node can be manually processed. Correction, or the above element picking method can be corrected so that the identification result of picking matching nodes is correct.
4、在未拾取到各界面元素的元素定位信息的情况下，确定拾取匹配节点的识别结果为空，并生成拾取匹配节点的识别状态为识别失败状态。4. When the element positioning information of each interface element is not picked up, it is determined that the recognition result of the picked matching node is empty, and the recognition status of the picked matching node is generated as a recognition failure status.
本申请实施例所提供的与任务关联的界面元素的处理方法，同时基于RPA机器人所在设备的操作系统信息、任务流程节点所指示的目标应用界面所属的目标应用的软件架构信息和开发语言信息，对该任务流程节点所指示的目标应用界面中的目标界面元素进行拾取，可以提升拾取结果的准确性和可靠性。The method for processing interface elements associated with tasks provided by the embodiments of this application is based on the operating system information of the device where the RPA robot is located, the software architecture information and development language information of the target application to which the target application interface indicated by the task process node belongs. Picking up the target interface elements in the target application interface indicated by the task process node can improve the accuracy and reliability of the picking results.
为了清楚说明本申请任一实施例中图像识别节点是如何对任务流程节点所指示的目标应用界面中的目标界面元素进行图像识别的，本申请还提出一种与任务关联的界面元素的处理方法。In order to clearly explain how the image recognition node in any embodiment of the present application performs image recognition on the target interface element in the target application interface indicated by the task process node, the present application also proposes a method for processing interface elements associated with the task. .
图5为本申请实施例所提供的另一种与任务关联的界面元素的处理方法的流程示意图。FIG. 5 is a schematic flowchart of another method for processing interface elements associated with tasks provided by an embodiment of the present application.
如图5所示，在图3所示实施例的基础上，图像识别节点可以采用以下步骤，对任一任务流程节点所指示的目标应用界面中的目标界面元素进行图像识别：As shown in Figure 5, based on the embodiment shown in Figure 3, the image recognition node can use the following steps to perform image recognition on the target interface elements in the target application interface indicated by any task process node:
步骤S501，获取参考图像；其中，参考图像是对任一任务流程节点所指示的目标界面元素执行截图操作得到的。Step S501: Obtain a reference image; wherein the reference image is obtained by performing a screenshot operation on the target interface element indicated by any task process node.
在本申请实施例中，针对任意一个任务流程节点，参考图像是响应于用户对该任务流程节点所指示的目标界面元素触发的点击或选中操作，对该目标界面元素进行截图得到的。In this embodiment of the present application, for any task process node, the reference image is obtained by taking a screenshot of the target interface element in response to a click or selection operation triggered by the user on the target interface element indicated by the task process node.
步骤S502，对任一任务流程节点所指示的目标应用界面进行截图，得到待识别的第一目标图像。Step S502: Take a screenshot of the target application interface indicated by any task process node to obtain the first target image to be recognized.
在本申请实施例中，可以对上述任务流程节点所指示的目标应用界面进行截图，得到待识别的第一目标图像。In this embodiment of the present application, a screenshot of the target application interface indicated by the above task process node can be taken to obtain the first target image to be identified.
步骤S503，判断第一目标图像中是否存在目标区域，以得到判断结果，其中，目标区域与参考图像之间相似度高于设定相似度阈值。Step S503: Determine whether there is a target area in the first target image to obtain a judgment result, wherein the similarity between the target area and the reference image is higher than the set similarity threshold.
在本申请实施例中，可以将参考图像与第一目标图像中的各图像区域进行匹配，以确定各图像区域中是否存在目标区域，其中，目标区域与参考图像之间的相似度高于设定相似度阈值。In the embodiment of the present application, the reference image can be matched with each image area in the first target image to determine whether there is a target area in each image area, where the similarity between the target area and the reference image is higher than the set Set the similarity threshold.
需要说明的是，第一目标图像和参考图像可以为RGB(Red Green Blue，红绿蓝)图像，可以直接计算两个RGB图像之间的相似度，或者，也可以将第一目标图像和参考图像转换为灰度图像，并计算两个灰度图像之间的相似度，本申请对此并不做限制。It should be noted that the first target image and the reference image can be RGB (Red Green Blue) images, and the similarity between the two RGB images can be directly calculated, or the first target image and the reference image can also be The image is converted into a grayscale image, and the similarity between the two grayscale images is calculated. This application does not limit this.
步骤S504，根据判断结果，确定图像识别节点的识别状态和识别结果。Step S504: Determine the recognition status and recognition result of the image recognition node based on the judgment result.
在本申请实施例中，可以根据判断结果，确定图像识别节点的识别状态和识别结果。In the embodiment of the present application, the recognition status and recognition result of the image recognition node can be determined based on the judgment result.
作为一种可能的实现方式，可以采用以下步骤，确定图像识别节点的识别状态和识别结果：As a possible implementation, the following steps can be used to determine the recognition status and recognition results of the image recognition node:
1、判断第一目标图像中是否存在目标区域，若是，则执行步骤2至步骤4，若否，则执行步骤5。1. Determine whether there is a target area in the first target image. If so, perform steps 2 to 4. If not, perform step 5.
2、根据目标区域在第一目标图像中的位置信息，确定上述任务流程节点所指示的目标界面元素的元素位置。2. Determine the element position of the target interface element indicated by the above task process node according to the position information of the target area in the first target image.
3、根据参考图像对应的元素类别，确定上述任务流程节点所指示的目标界面元素的元素类别。3. According to the element category corresponding to the reference image, determine the element category of the target interface element indicated by the above task process node.
其中，参考图像的元素类别可以基于图像识别技术(如图像分类技术、OCR(Optical Character Recognition，光学字符识别)技术等)识别得到，或者，也可以由人工指定，本申请对此并不作限制。Among them, the element categories of the reference image can be recognized based on image recognition technology (such as image classification technology, OCR (Optical Character Recognition, optical character recognition) technology, etc.), or can also be manually specified, and this application does not limit this.
4、根据上述任务流程节点所指示的目标界面元素的元素位置和元素类别，生成图像识别节点的识别结果，并生成图像识别节点的识别状态为识别成功状态。4. Based on the element position and element category of the target interface element indicated by the above task process node, generate the recognition result of the image recognition node, and generate the recognition status of the image recognition node as the recognition success status.
进一步地，在图像识别节点的识别状态为识别成功状态时，还可以对目标应用界面上图像识别节点识别得到的目标界面元素进行标注(如高亮显示该界面元素、元素类别和元素位置等，并由用户确定图像识别节点的识别结果是否正确，在正确的情况下，才记录或保存该图像识别节点的识别结果，而在不正确的情况下，可以由人工对图像识别节点的识别结果进行修正，或者，可以对上述图像识别节点所采用的图像识别方法进行修正，或者，对设定相似度阈值进行修正，以使图像识别节点的识别结果正确无误。Further, when the recognition status of the image recognition node is the recognition success state, the target interface elements recognized by the image recognition node on the target application interface can also be marked (such as highlighting the interface element, element category, element position, etc., And it is up to the user to determine whether the recognition result of the image recognition node is correct. If it is correct, the recognition result of the image recognition node will be recorded or saved. If it is incorrect, the recognition result of the image recognition node can be manually performed. Correction, or the image recognition method used by the above image recognition node can be corrected, or the set similarity threshold can be corrected, so that the recognition result of the image recognition node is correct.
5、确定图像识别节点的识别结果为空，并生成图像识别节点的识别状态为识别失败状态。5. Determine that the recognition result of the image recognition node is empty, and generate the recognition status of the image recognition node as a recognition failure state.
本申请实施例所提供的与任务关联的界面元素的处理方法，基于展示有目标界面元素的参考图像与展示有目标应用界面的第一目标图像之间的相似度，有效确定图像识别节点的识别状态和识别结果，可以提升图像识别的有效性。The method for processing interface elements associated with a task provided by embodiments of the present application effectively determines the identification of image recognition nodes based on the similarity between the reference image displaying the target interface element and the first target image displaying the target application interface. Status and recognition results can improve the effectiveness of image recognition.
为了清楚说明本申请任一实施例中目标检测节点是如何对任务流程节点所指示的目标应用界面中的目标界面元素进行目标检测的，本申请还提出一种与任务关联的界面元素的处理方法。In order to clearly explain how the target detection node in any embodiment of the present application performs target detection on the target interface element in the target application interface indicated by the task process node, the present application also proposes a method for processing interface elements associated with the task. .
图6为本申请实施例所提供的另一种与任务关联的界面元素的处理方法的流程示意图。FIG. 6 is a schematic flowchart of another method for processing interface elements associated with tasks provided by an embodiment of the present application.
如图6所示，在图3所示实施例的基础上，目标检测节点可以采用以下步骤，对任一任务流程节点所指示的目标应用界面中的目标界面元素进行目标检测：As shown in Figure 6, based on the embodiment shown in Figure 3, the target detection node can use the following steps to perform target detection on the target interface elements in the target application interface indicated by any task process node:
步骤S601，对任一任务流程节点所指示的目标应用界面进行截图，得到待识别的第二目标图像。Step S601: Take a screenshot of the target application interface indicated by any task process node to obtain the second target image to be recognized.
在本申请实施例中，针对任意一个任务流程节点，可以对该任务流程节点所指示的目标应用界面进行截图，以得到展示有该目标应用界面的第二目标图像。In this embodiment of the present application, for any task process node, a screenshot of the target application interface indicated by the task process node can be taken to obtain a second target image showing the target application interface.
步骤S602，采用目标检测模型对第二目标图像进行目标的回归预测，以得到至少一个检测框的位置信息。Step S602: Use a target detection model to perform regression prediction of the target on the second target image to obtain position information of at least one detection frame.
在本申请实施例中，可以采用目标检测模型对第二目标图像进行目标(即界面元素)的回归预测，以得到至少一个检测框的位置信息。In this embodiment of the present application, a target detection model can be used to perform regression prediction of the target (ie, interface element) on the second target image to obtain the position information of at least one detection frame.
步骤S603，采用目标检测模型对第二目标图像进行目标的类别预测，以得到各检测框内目标所属的元素类别。Step S603: Use the target detection model to predict the target category on the second target image to obtain the element category to which the target in each detection frame belongs.
在本申请实施例中，可以采用目标检测模型对第二目标图像进行目标(即界面元素)的类别预测，以得到各检测框内目标所属的元素类别。In this embodiment of the present application, a target detection model can be used to predict the category of the target (ie, interface element) on the second target image, so as to obtain the element category to which the target in each detection frame belongs.
步骤S604，根据各检测框的位置信息和元素类别，确定目标检测节点的识别状态和识别结果。Step S604: Determine the recognition status and recognition result of the target detection node based on the position information and element category of each detection frame.
在本申请实施例中，可以根据各检测框的位置信息和元素类别，确定目标检测节点的识别状态和识别结果。In this embodiment of the present application, the recognition status and recognition result of the target detection node can be determined based on the position information and element category of each detection frame.
作为一种可能的实现方式，可以采用以下步骤，确定目标检测节点的识别状态和识别结果：As a possible implementation method, the following steps can be used to determine the recognition status and recognition results of the target detection node:
1、根据各检测框的位置信息和与上述任务流程节点关联的触控点(包括但不限于用户手指触摸的位置点、鼠标点击的位置点、鼠标或光标所在的位置点等)的位置信息，从各检测框中确定目标检测框；其中，目标检测框中包括该任务流程节点所指示的目标界面元素。1. Based on the position information of each detection frame and the position information of the touch points associated with the above-mentioned task process nodes (including but not limited to the position point touched by the user's finger, the position point clicked by the mouse, the position point where the mouse or cursor is located, etc.) , determine the target detection frame from each detection frame; wherein, the target detection frame includes the target interface element indicated by the task process node.
举例而言，针对任务流程链中的每个任务流程节点，可以通过人工鼠标指定或选择与该任务流程节点关联的目标界面元素，相应的，本申请中，可以根据鼠标位置，从各检测框中确定目标检测框，其中，鼠标位置位于该目标检测框内，该目标检测框中包括该任务流程节点所指示的目标界面元素。For example, for each task process node in the task process chain, you can use a manual mouse to specify or select the target interface element associated with the task process node. Correspondingly, in this application, you can select from each detection frame based on the mouse position. A target detection frame is determined in the target detection frame, wherein the mouse position is within the target detection frame, and the target detection frame includes the target interface element indicated by the task process node.
2、根据目标检测框的位置信息，确定该任务流程节点所指示的目标界面元素的元素位置。2. Determine the element position of the target interface element indicated by the task process node based on the position information of the target detection frame.
3、根据目标检测框的元素类别，确定该任务流程节点所指示的目标界面元素的元素类别。3. According to the element category of the target detection frame, determine the element category of the target interface element indicated by the task process node.
4、根据该任务流程节点所指示的目标界面元素的元素位置和元素类别，生成目标检测节点的识别结果，并生成目标检测节点的识别状态为识别成功状态。4. Based on the element position and element category of the target interface element indicated by the task process node, generate the recognition result of the target detection node, and generate the recognition status of the target detection node as the recognition success status.
需要说明的是，为了提升目标检测(即界面元素检测)的准确性，目标检测模型可以为经过训练的AI模型，其中，目标检测模型可以采用以下步骤训练得到：It should be noted that in order to improve the accuracy of target detection (ie, interface element detection), the target detection model can be a trained AI model. The target detection model can be trained using the following steps:
1、获取样本图像，其中，样本图像中展示有至少一个样本界面元素。1. Obtain a sample image, where at least one sample interface element is displayed in the sample image.
其中，对样本图像的获取方式不做限制，比如，样本图像可以是在线采集的图像，再比如，样本图像可以是人工合成的图像，再比如，样本图像可以是人工对应用界面中的界面元素(本申请中记为样本界面元素)进行截取得到的图像，等等，本申请对此并不作限制。Among them, there is no restriction on the acquisition method of the sample image. For example, the sample image can be an image collected online. For another example, the sample image can be an artificially synthesized image. For another example, the sample image can be an interface element in the application interface. (recorded as sample interface elements in this application), etc., this application does not limit this.
2、采用初始检测模型对样本图像进行目标检测，以得到至少一个预测框的位置信息和预测框内目标所属的预测元素类别。2. Use the initial detection model to perform target detection on the sample image to obtain the position information of at least one prediction frame and the prediction element category to which the target in the prediction frame belongs.
本申请中，可以采用初始检测模型对样本图像进行目标的回归预测，以得到至少一个预测框的位置信息，并且，可以采用初始检测模型对样本图像进行目标的类别预测，以得到至少一个预测框内目标所属的预测元素类别。In this application, the initial detection model can be used to perform regression prediction of the target on the sample image to obtain the position information of at least one prediction frame, and the initial detection model can be used to perform target category prediction on the sample image to obtain at least one prediction frame. The prediction element category to which the inner target belongs.
3、根据样本图像对应的标注框的位置信息和预测框的位置信息之间的差异，生成回归损失值。3. Generate a regression loss value based on the difference between the position information of the annotation frame corresponding to the sample image and the position information of the prediction frame.
本申请中，可以根据样本图像对应的标注框的位置信息和预测框的位置信息之间的差异，生成回归损失值。其中，回归损失值与上述差异呈正相关关系，即差异越小，回归损失值越小，反之，差异越大，回归损失值越大。In this application, the regression loss value can be generated based on the difference between the position information of the annotation frame and the position information of the prediction frame corresponding to the sample image. Among them, the regression loss value is positively correlated with the above-mentioned difference, that is, the smaller the difference, the smaller the regression loss value, and conversely, the larger the difference, the greater the regression loss value.
4、根据标注框内样本界面元素所属的标注元素类别和预测框的预测元素类别之间的差异，生成分类损失值。4. Generate a classification loss value based on the difference between the annotation element category to which the sample interface element in the annotation frame belongs and the predicted element category of the prediction frame.
在本申请实施例中，可以根据标注框内样本界面元素所属的标注元素类别和预测框的预测元素类别之间的差异，生成分类损失值。其中，分类损失值与该差异呈正相关关系，即差异越小，分类损失值越小，反之，差异越大，分类损失值越大。In the embodiment of the present application, a classification loss value can be generated based on the difference between the annotation element category to which the sample interface element in the annotation frame belongs and the predicted element category of the prediction frame. Among them, the classification loss value is positively correlated with the difference, that is, the smaller the difference, the smaller the classification loss value, and conversely, the larger the difference, the greater the classification loss value.
5、根据回归损失值和分类损失值，对初始检测模型进行训练，以得到目标检测模型。5. Based on the regression loss value and classification loss value, train the initial detection model to obtain the target detection model.
在本申请实施例中，可以根据回归损失值和分类损失值，确定总损失值。In the embodiment of this application, the total loss value can be determined based on the regression loss value and the classification loss value.
作为一种示例，可以将回归损失值和分类损失值之和，作为总损失值。As an example, the sum of the regression loss value and the classification loss value can be used as the total loss value.
作为另一种示例，可以将回归损失值和分类损失值的均值，作为总损失值。As another example, the average of the regression loss value and the classification loss value can be used as the total loss value.
作为又一种示例，可以对回归损失值和分类损失值进行加权求和，以得到总损失值。As yet another example, a weighted sum of the regression loss value and the classification loss value can be performed to obtain the total loss value.
本申请中，可以根据总损失值，对初始检测模型进行训练，以得到目标检测模型。In this application, the initial detection model can be trained according to the total loss value to obtain the target detection model.
例如，可以根据总损失值，对初始检测模型中的模型参数进行调整，以使总损失值最小化，得到目标检测模型。For example, the model parameters in the initial detection model can be adjusted according to the total loss value to minimize the total loss value to obtain the target detection model.
需要说明的是，上述仅以模型训练的终止条件为总损失值最小化进行示例，实际应用时，还可以设置其他的终止条件，比如终止条件还可以为训练次数达到设定次数，或者，终止条件还可以为训练时长达到设定时长，等等，本申请对此并不做限制。It should be noted that the above example only uses the termination condition of model training as the minimization of the total loss value. In actual application, other termination conditions can also be set. For example, the termination condition can also be that the number of training times reaches the set number, or terminate. The condition can also be that the training duration reaches a set duration, etc. This application does not impose restrictions on this.
由此，采用经过训练的目标检测模型，对任务流程节点所指示的目标应用界面中的目标界面元素进行目标检测，可以提升检测结果的准确性和可靠性。Therefore, using the trained target detection model to perform target detection on the target interface elements in the target application interface indicated by the task process node can improve the accuracy and reliability of the detection results.
本申请实施例所提供的与任务关联的界面元素的处理方法，基于目标检测技术，识别目标界面元素的元素位置和元素类别，可以提升识别结果的准确性。The method for processing interface elements associated with a task provided by the embodiments of the present application is based on target detection technology and identifies the element position and element category of the target interface element, which can improve the accuracy of the identification results.
本申请主要解决在RPA机器人开发过程中遇到不同操作系统下不同软件架构的应用界面中的界面元素拾取时，元素拾取方法不能自动切换，或者，部分应用界面中的界面元素使用传统技术拾取不到，导致RPA技术难以落地，无法满足使用者的需求的问题。This application mainly solves the problem that when encountering interface element picking in application interfaces with different software architectures under different operating systems during the development of RPA robots, the element picking method cannot be automatically switched, or the interface elements in some application interfaces cannot be picked up using traditional technologies. Therefore, it is difficult to implement RPA technology and cannot meet the needs of users.
针对上述问题，本申请中，在界面元素拾取时，采用融合拾取方法，实现元素拾取方法的自动切换，如，在元素识别方式链中的拾取匹配节点(采用传统的元素拾取方法)拾取不到界面元素时，自动切换到元素识别方式链中的图像识别节点(采用图像识别方法)，在图像识别节点无法识别界面元素时，自动切换到元素识别方式链中的目标检测节点(采用目标检测方法)。操作时能够根据元素信息自动匹配到相应元素解析方法，定位元素位置和元素类别，实现对界面元素执行针对性的操作，实现全部应用流程自动化。In response to the above problems, in this application, when picking up interface elements, a fusion picking method is used to realize automatic switching of element picking methods. For example, the picking matching node in the element recognition mode chain (using the traditional element picking method) cannot be picked up. When an interface element is detected, it automatically switches to the image recognition node in the element recognition method chain (using the image recognition method). When the image recognition node cannot recognize the interface element, it automatically switches to the target detection node in the element recognition method chain (using the target detection method). ). During operation, it can automatically match the corresponding element analysis method according to the element information, locate the element position and element category, implement targeted operations on interface elements, and realize the automation of all application processes.
其中，RPA流程融合拾取解析方式为：在RPA流程开发时，针对GUI操作，首先拾取界面元素的元素信息，设计器在捕获界面元素的元素信息时，可以根据操作系统信息、应用的软件架构信息和开发语言信息，自动切换到相应的元素拾取方法(或称为元素捕获方法)，当元素拾取方法捕获不到元素信息时，自动切换到图像识别方法，如果仍然无法识别到元素信息，则自动切换到目标检测方法。Among them, the RPA process integration picking and parsing method is: when developing the RPA process, for GUI operations, the element information of the interface elements is first picked up. When the designer captures the element information of the interface elements, it can be based on the operating system information and the application software architecture information. and develop language information, automatically switch to the corresponding element picking method (or element capture method). When the element picking method cannot capture the element information, it automatically switches to the image recognition method. If the element information still cannot be recognized, it automatically switches to the image recognition method. Switch to target detection method.
RPA机器人运行时，根据元素信息自动匹配到对界面元素进行解析的方法，即，可以根据元素信息，确定元素类别和元素位置，根据元素位置，定位界面元素，并对界面元素执行与元素类别匹配的操作。When the RPA robot is running, it automatically matches the method of parsing interface elements based on element information. That is, it can determine the element category and element position based on the element information, locate the interface element based on the element position, and perform matching on the interface element with the element category. operation.
其中，传统的元素拾取方法：主要是针对不同操作系统、不同应用进行的有效适配方法，在捕获界面元素时，会自动判断操作系统类型、软件架构类型、开发语言类型，自动切换到适配方法获取界面元素的元素信息。在对界面元素进行操作时，可以根据获取元素信息的方法，自动匹配相应的解析方法获取界面元素的元素位置和元素类别，然后调用后台方法或模拟鼠标、键盘对元素区域进行模拟人工操作，实现流程自动化。Among them, the traditional element picking method: It is mainly an effective adaptation method for different operating systems and different applications. When capturing interface elements, it will automatically determine the operating system type, software architecture type, and development language type, and automatically switch to the adaptation method. Method to obtain element information of interface elements. When operating interface elements, you can automatically match the corresponding parsing method to obtain the element position and element category of the interface element according to the method of obtaining element information, and then call the background method or simulate the mouse and keyboard to simulate manual operations on the element area to achieve Process automation.
图像识别方法：主要采图像识别技术，获取元素截图、位置、大小等信息，然后采用图像匹配技术对图像进行有效定位，之后，模拟鼠标、键盘操作对图像区域进行单击、悬停、文本读取、文本写入等操作，实现流程自动化。Image recognition method: It mainly uses image recognition technology to obtain information such as element screenshots, positions, sizes, etc., and then uses image matching technology to effectively position the image. After that, it simulates mouse and keyboard operations to click, hover, and read text in the image area. Fetch, text writing and other operations to realize process automation.
目标检测方法：采用深度学习技术，获取界面元素的元素信息。通过大量的样本图像对目标检测模型进行训练，并采用经过训练的目标检测模型来有效识别界面元素的元素类别，如按钮、文本、文本框、单复选框等类别，捕获时返回界面元素的元素类别、区域位置等信息。在对界面元素进行操作时，根据元素类别、区域位置自动匹配到相应元素位置，根据元素类别调用相应方法实现流程自动化。Target detection method: Use deep learning technology to obtain element information of interface elements. The target detection model is trained through a large number of sample images, and the trained target detection model is used to effectively identify the element categories of interface elements, such as buttons, text, text boxes, single check boxes, etc., and the interface elements are returned when capturing. Element category, area location and other information. When operating interface elements, the corresponding element position is automatically matched according to the element category and region position, and the corresponding method is called according to the element category to realize process automation.
即本申请中，可以通过调用不同的元素拾取方法，获取界面元素的元素信息，并根据元素拾取方法自动匹配解析方法进行元素定位，根据元素类别自动匹配界面元素的操作方法。That is, in this application, the element information of the interface elements can be obtained by calling different element picking methods, and the element positioning can be automatically matched according to the element picking method and the analysis method, and the operation method of the interface element can be automatically matched according to the element category.
作为一种示例，在设计阶段，设计器触发元素拾取功能后，实时检测鼠标所在应用及元素位置，并实时根据操作系统类型和应用类型，匹配调用元素识别方式链中的拾取匹配节点(采用传统的元素拾取方法)，并高亮展示当前元素、元素类别、元素位置等，用户点击确定后，返回元素捕获的路径(即DOM路径)、属性等元素定位信息(或称为定位操作信息)。As an example, during the design phase, after the designer triggers the element picking function, it detects the application and element position where the mouse is located in real time, and calls the picking matching node in the element recognition method chain in real time based on the operating system type and application type (using the traditional element picking method), and highlights the current element, element category, element position, etc. After the user clicks OK, the element positioning information (or called positioning operation information) such as the path captured by the element (i.e., DOM path) and attributes are returned.
当调用拾取匹配节点无法识别出元素路径、属性等信息，或者人工确定拾取匹配节点识别错误，无法对界面元素进行定位时，自动切换到图像识别节点，可以自动或手动高亮显示元素区域，对元素区域进行截图，返回元素截图(本申请中记为参考图像)、元素位置、大小等元素定位信息(或称为定位操作信息)，返回后可以对图像识别节点进行像素、特征选择，对设定相似度阈值进行设置，对灰度匹配进行勾选等配置(即，将参考图像和第一目标图像转换为灰度图像后进行相似度计算)，以提高图像识别率。When calling the pick matching node cannot identify the element path, attributes and other information, or manually determines that the pick matching node is recognized incorrectly and cannot locate the interface element, it will automatically switch to the image recognition node, and the element area can be highlighted automatically or manually. Take a screenshot of the element area and return the element screenshot (recorded as a reference image in this application), element position, size and other element positioning information (or called positioning operation information). After returning, you can perform pixel and feature selection on the image recognition node, and then Set a certain similarity threshold, check the grayscale matching and other configurations (that is, convert the reference image and the first target image into grayscale images and then perform similarity calculation) to improve the image recognition rate.
当图像识别节点无法识别出元素路径、属性等信息时，自动切换到目标检测节点，调用目标检测方法识别元素位置和元素类别。其中，目标检测方法能够识别出不同的元素类别，识别时可实时展示元素类别并高亮展示，用户点击确认时，可返回元素类别、元素所处区域的位置、图像信息等元素定位信息(或称为定位操作信息)。When the image recognition node cannot identify the element path, attributes and other information, it automatically switches to the target detection node and calls the target detection method to identify the element position and element category. Among them, the target detection method can identify different element categories. During recognition, the element categories can be displayed and highlighted in real time. When the user clicks to confirm, the element positioning information such as the element category, the location of the element's area, and image information can be returned (or called positioning operation information).
也就是说，拾取匹配节点能够实现根据操作系统、软件架构、开发语言，自动切换已适配方法，图像识别节点能够实现单个界面元素捕获不到时，采用图像识别技术进行元素匹配，目标检测节点能够实现应用无法识别时，采用目标检测技术进行元素的智能匹配。In other words, the picking and matching node can automatically switch the adapted method according to the operating system, software architecture, and development language. The image recognition node can use image recognition technology to match elements when a single interface element cannot be captured. The target detection node It can use target detection technology to intelligently match elements when the application cannot be recognized.
在捕获得到界面元素的元素类别和元素位置后，可以保存界面元素的元素类别和元素位置，从而RPA机器人在执行任务时，可以直接通过查询的方式，获取各界面元素的元素类别和元素位置，并根据元素位置，定位界面元素，并对界面元素执行与元素类别匹配的操作。After capturing the element category and element position of the interface element, the element category and element position of the interface element can be saved, so that when the RPA robot performs a task, it can directly obtain the element category and element position of each interface element through query. And based on the element position, locate the interface element, and perform operations on the interface element that match the element category.
作为一种示例，在捕获每个界面元素时，捕获流程可以如图7(a)所示，可以针对该界面元素所属应用的软件架构信息、开发语言信息和操作系统信息，匹配拾取匹配节点中的元素拾取方法(或称为传统的元素拾取方法)，判断该元素拾取方法是否能够捕获到该界面元素，若是，则获取该界面元素所在应用路径、属性等信息，若否，则进一步判断是无法捕获界面元素还是无法识别应用。As an example, when capturing each interface element, the capture process can be as shown in Figure 7(a). The software architecture information, development language information and operating system information of the application to which the interface element belongs can be matched to pick up the matching node. The element picking method (or called the traditional element picking method) determines whether the element picking method can capture the interface element. If so, obtain the application path, attributes and other information of the interface element. If not, further determine whether it is The interface elements cannot be captured or the application cannot be recognized.
若无法捕获界面元素，则调用图像处理节点，获取图像位置、大小、截图等信息，若无法识别应用，则调用目标检测节点，获取元素类别、生成元素定位操作信息。If the interface element cannot be captured, the image processing node is called to obtain image position, size, screenshot and other information. If the application cannot be identified, the target detection node is called to obtain the element category and generate element positioning operation information.
最后，可以根据元素拾取方法自动匹配解析方法进行元素定位，根据元素类别自动匹配界面元素的操作方法，具体流程可如图7(b)所示。Finally, the element positioning can be automatically matched according to the element picking method and the parsing method, and the operation method of the interface element can be automatically matched according to the element category. The specific process is shown in Figure 7(b).
综上，本申请所提供的方法，至少具有以下优点：To sum up, the method provided by this application has at least the following advantages:
1、能够使普通的业务人员使用RPA开发流程；1. Enable ordinary business personnel to use the RPA development process;
2、能够解决界面元素捕获不到导致RPA技术无法落地的情况；2. It can solve the problem that interface elements cannot be captured, resulting in the failure of RPA technology to be implemented;
3、解决因应用无法识别导致RPA技术无法落地的情况；3. Solve the situation where RPA technology cannot be implemented due to unrecognizable applications;
4、能够提高开发效率，缩短RPA流程的开发时间；4. It can improve development efficiency and shorten the development time of RPA process;
5、快速实现流程，做到业务的敏捷，为响应市场赢得宝贵的时间。5. Quickly implement processes to achieve business agility and gain valuable time to respond to the market.
与上述图1至图6实施例提供的与任务关联的界面元素的处理方法相对应，本申请还提供一种与任务关联的界面元素的处理装置，由于本申请实施例提供的与任务关联的界面元素的处理装置与上述图1至图6实施例提供的与任务关联的界面元素的处理方法相对应，因此在与任务关联的界面元素的处理方法的实施方式也适用于本申请实施例提供的与任务关联的界面元素的处理装置，在本申请实施例中不再详细描述。Corresponding to the processing method for task-related interface elements provided by the above-mentioned embodiments of FIG. 1 to FIG. 6 , the present application also provides a processing device for task-related interface elements. Since the task-related interface elements provided by the embodiments of the present application are The device for processing interface elements corresponds to the method for processing interface elements associated with tasks provided in the above-mentioned embodiments of FIGS. 1 to 6 . Therefore, the implementation of the method for processing interface elements associated with tasks is also applicable to the methods provided in the embodiments of the present application. The device for processing interface elements associated with the task will not be described in detail in the embodiment of this application.
图8为本申请实施例所提供的一种与任务关联的界面元素的处理装置的结构示意图。FIG. 8 is a schematic structural diagram of a device for processing interface elements associated with tasks provided by an embodiment of the present application.
如图8所示，该与任务关联的界面元素的处理装置800可以包括：第一获取模块810、第二获取模块820以及执行模块830。As shown in FIG. 8 , the device 800 for processing interface elements associated with a task may include: a first acquisition module 810 , a second acquisition module 820 and an execution module 830 .
其中，第一获取模块810，用于获取与待执行的目标任务关联的任务流程链，其中，任务流程链中包括至少一个任务流程节点。The first acquisition module 810 is used to acquire a task process chain associated with the target task to be executed, where the task process chain includes at least one task process node.
第二获取模块820，用于获取与任务流程链中的各任务流程节点所指示的目标应用界面中的目标界面元素对应的元素位置和元素类别；其中，目标界面元素的元素位置和元素类别是通过调用元素识别方式链中的至少一个元素识别方式节点识别得到的。The second acquisition module 820 is used to obtain the element position and element category corresponding to the target interface element in the target application interface indicated by each task process node in the task process chain; wherein, the element position and element category of the target interface element are Recognized by calling at least one element recognition mode node in the element recognition mode chain.
执行模块830，用于依次根据各任务流程节点所指示的目标界面元素的元素位置，对相应目标界面元素执行与相应目标界面元素的元素类别匹配的目标操作。The execution module 830 is configured to execute a target operation matching the element category of the corresponding target interface element on the corresponding target interface element according to the element position of the target interface element indicated by each task process node.
在本申请实施例的一种可能的实现方式中，任一任务流程节点所指示的目标界面元素的元素位置和元素类别，采用以下模块预先获取：In a possible implementation manner of the embodiment of this application, the element position and element category of the target interface element indicated by any task process node are pre-obtained using the following module:
调用模块，用于调用元素识别方式链中的至少一个元素识别方式节点，对任一任务流程节点所指示的目标应用界面中的目标界面元素进行识别，以得到各元素识别方式节点的识别状态和识别结果。The calling module is used to call at least one element recognition mode node in the element recognition mode chain to identify the target interface element in the target application interface indicated by any task process node, so as to obtain the recognition status and sum of each element recognition mode node. Recognition results.
第一确定模块，用于从各元素识别方式节点中，确定识别状态为识别成功状态的目标元素识别方式节点。The first determination module is used to determine the target element recognition mode node whose recognition status is the recognition success state from each element recognition mode node.
第二确定模块，用于根据目标元素识别方式节点的识别结果，确定任一任务流程节点所指示的目标界面元素的元素位置和元素类别。The second determination module is used to determine the element position and element category of the target interface element indicated by any task process node according to the recognition result of the target element recognition mode node.
在本申请实施例的一种可能的实现方式中，元素识别方式链中的各元素识别方式节点包括以下节点中的至少一种：拾取匹配节点、图像识别节点和目标检测节点；调用模块，具体用于：调用拾取匹配节点对任一任务流程节点所指示的目标应用界面中的目标界面元素进行拾取，并获取拾取匹配节点返回的第一反馈信息；其中，第一反馈信息中包括拾取匹配节点的识别状态；在第一反馈信息中的识别状态为识别失败状态时，调用图像识别节点对任一任务流程节点所指示的目标应用界面中的目标界面元素进行图像识别，并获取图像识别节点返回的第二反馈信息；其中，第二反馈信息中包括图像识别节点的识别状态；在第二反馈信息中的识别状态为识别失败状态时，调用目标检测节点，对任一任务流程节点所指示的目标应用界面中的目标界面元素进行目标检测，得到目标检测节点返回的第三反馈信息。In a possible implementation manner of the embodiment of the present application, each element identification method node in the element identification method chain includes at least one of the following nodes: a pick-up matching node, an image recognition node, and a target detection node; a calling module, specifically Used for: calling the pick-up matching node to pick up the target interface element in the target application interface indicated by any task process node, and obtaining the first feedback information returned by the pick-up matching node; wherein the first feedback information includes the pick-up matching node The recognition status; when the recognition status in the first feedback information is the recognition failure status, call the image recognition node to perform image recognition on the target interface element in the target application interface indicated by any task process node, and obtain the image recognition node to return The second feedback information; wherein, the second feedback information includes the recognition status of the image recognition node; when the recognition status in the second feedback information is the recognition failure status, the target detection node is called, and any task process node indicates The target interface element in the target application interface performs target detection and obtains the third feedback information returned by the target detection node.
在本申请实施例的一种可能的实现方式中，第一反馈信息中还包括拾取匹配节点的识别结果，在第一反馈信息中的识别状态为识别失败状态时，第一反馈信息中的识别结果为空；在第一反馈信息中的识别状态为识别成功状态时，第一反馈信息中的识别结果包括任一任务流程节点所指示的目标界面元素的元素位置和元素类别。In a possible implementation manner of the embodiment of the present application, the first feedback information also includes the identification result of picking up the matching node. When the identification status in the first feedback information is the identification failure status, the identification result in the first feedback information The result is empty; when the recognition status in the first feedback information is the recognition success status, the recognition result in the first feedback information includes the element position and element category of the target interface element indicated by any task process node.
其中，在第一反馈信息中的识别状态为识别成功状态时，调用模块，还用于：停止调用图像识别节点和目标检测节点；确定图像识别节点和目标检测节点的识别结果均为空；确定图像识别节点和目标检测节点的识别状态均为未识别状态。Wherein, when the recognition status in the first feedback information is the recognition success status, the calling module is also used to: stop calling the image recognition node and the target detection node; determine that the recognition results of the image recognition node and the target detection node are both empty; determine The recognition status of the image recognition node and the target detection node are both unrecognized status.
在本申请实施例的一种可能的实现方式中，第二反馈信息中还包括图像识别节点的识别结果，在第二反馈信息中的识别状态为识别失败状态时，第二反馈信息中的识别结果为空；在第二反馈信息中的识别状态为识别成功状态时，第二反馈信息中的识别结果包括任一任务流程节点所指示的目标界面元素的元素位置和元素类别。In a possible implementation manner of the embodiment of the present application, the second feedback information also includes the recognition result of the image recognition node. When the recognition status in the second feedback information is the recognition failure status, the recognition result in the second feedback information The result is empty; when the recognition status in the second feedback information is the recognition success status, the recognition result in the second feedback information includes the element position and element category of the target interface element indicated by any task process node.
其中，在第二反馈信息中的识别状态为识别成功状态时，调用模块，还用于：停止调用目标检测节点；确定目标检测节点的识别结果为空；确定目标检测节点的识别状态为未识别状态。Wherein, when the recognition status in the second feedback information is the recognition success status, the calling module is also used to: stop calling the target detection node; determine that the recognition result of the target detection node is empty; determine that the recognition status of the target detection node is not recognized. state.
在本申请实施例的一种可能的实现方式中，拾取匹配节点，具体用于：获取RPA机器人所在设备的操作系统信息，并获取任一任务流程节点所指示的目标应用界面所属的目标应用的软件架构信息和开发语言信息；查询与操作系统信息、软件架构信息和开发语言信息匹配的元素拾取方法；调用元素拾取方法对任一任务流程节点所指示的目标应用界面中的各界面元素进行拾取，以得到拾取匹配节点的识别状态和识别结果。In a possible implementation of the embodiment of the present application, picking up the matching node is specifically used to: obtain the operating system information of the device where the RPA robot is located, and obtain the target application to which the target application interface indicated by any task process node belongs. Software architecture information and development language information; query the element picking method that matches the operating system information, software architecture information and development language information; call the element picking method to pick up each interface element in the target application interface indicated by any task process node , to obtain the recognition status and recognition results of the picked matching nodes.
在本申请实施例的一种可能的实现方式中，拾取匹配节点，具体用于：调用元素拾取方法对任一任务流程节点所指示的目标应用界面中的各界面元素进行拾取；判断元素拾取方法是否拾取到各界面元素的元素定位信息；在拾取到各界面元素的元素定位信息的情况下，根据各界面元素的元素定位信息和与任一任务流程节点关联的触控点的位置信息，确定任一任务流程节点所指示的目标界面元素的元素位置和元素类别；根据任一任务流程节点所指示的目标界面元素的元素位置和元素类别，生成拾取匹配节点的识别结果，并生成拾取匹配节点的识别状态为识别成功状态；在未拾取到各界面元素的元素定位信息的情况下，确定拾取匹配节点的识别结果为空，并生成拾取匹配节点的识别状态为识别失败状态。In a possible implementation of the embodiment of the present application, picking up matching nodes is specifically used to: call the element picking method to pick up each interface element in the target application interface indicated by any task process node; determine the element picking method Whether the element positioning information of each interface element is picked up; if the element positioning information of each interface element is picked up, it is determined based on the element positioning information of each interface element and the position information of the touch point associated with any task process node. The element position and element category of the target interface element indicated by any task process node; based on the element position and element category of the target interface element indicated by any task process node, generate the identification result of the pick-up matching node, and generate the pick-up matching node The recognition status of is the recognition success status; when the element positioning information of each interface element is not picked up, it is determined that the recognition result of the picked matching node is empty, and the recognition status of the picked matching node is generated as the recognition failure status.
在本申请实施例的一种可能的实现方式中，图像识别节点，具体用于：获取参考图像；其中，参考图像是对任一任务流程节点所指示的目标界面元素执行截图操作得到的；对任一任务流程节点所指示的目标应用界面进行截图，得到待识别的第一目标图像；判断第一目标图像中是否存在目标区域，以得到判断结果，其中，目标区域与参考图像之间相似度高于设定相似度阈值；根据判断结果，确定图像识别节点的识别状态和识别结果。In a possible implementation of the embodiment of the present application, the image recognition node is specifically used to: obtain a reference image; wherein the reference image is obtained by performing a screenshot operation on the target interface element indicated by any task process node; Take a screenshot of the target application interface indicated by any task process node to obtain the first target image to be identified; determine whether there is a target area in the first target image to obtain the judgment result, where the similarity between the target area and the reference image is is higher than the set similarity threshold; based on the judgment result, the recognition status and recognition result of the image recognition node are determined.
在本申请实施例的一种可能的实现方式中，图像识别节点，具体用于：在判断结果为第一目标图像中存在目标区域的情况下，根据目标区域在第一目标图像中的位置信息，确定任一任务流程节点所指示的目标界面元素的元素位置；根据参考图像对应的元素类别，确定任一任务流程节点所指示的目标界面元素的元素类别；根据任一任务流程节点所指示的目标界面元素的元素位置和元素类别，生成图像识别节点的识别结果，并生成图像识别节点的识别状态为识别成功状态；在判断结果为第一目标图像中不存在目标区域的情况下，确定图像识别节点的识别结果为空，并生成图像识别节点的识别状态为识别失败状态。In a possible implementation of the embodiment of the present application, the image recognition node is specifically used to: when the judgment result is that there is a target area in the first target image, based on the position information of the target area in the first target image , determine the element position of the target interface element indicated by any task process node; determine the element category of the target interface element indicated by any task process node according to the element category corresponding to the reference image; determine the element category of the target interface element indicated by any task process node; according to the element category indicated by any task process node The element position and element category of the target interface element are used to generate the recognition result of the image recognition node, and the recognition status of the image recognition node is generated as the recognition success status; when the judgment result is that the target area does not exist in the first target image, determine the image The recognition result of the recognition node is empty, and the recognition status of the generated image recognition node is recognition failure status.
在本申请实施例的一种可能的实现方式中，目标检测节点，具体用于：对任一任务流程节点所指示的目标应用界面进行截图，得到待识别的第二目标图像；采用目标检测模型对第二目标图像进行目标的回归预测，以得到至少一个检测框的位置信息；采用目标检测模型对第二目标图像进行目标的类别预测，以得到各检测框内目标所属的元素类别；根据各检测框的位置信息和元素类别，确定目标检测节点的识别状态和识别结果。In a possible implementation of the embodiment of the present application, the target detection node is specifically used to: take a screenshot of the target application interface indicated by any task process node to obtain the second target image to be identified; use a target detection model Perform regression prediction of the target on the second target image to obtain the position information of at least one detection frame; use a target detection model to perform target category prediction on the second target image to obtain the element category to which the target in each detection frame belongs; according to each detection frame The position information and element category of the detection frame determine the recognition status and recognition results of the target detection node.
在本申请实施例的一种可能的实现方式中，目标检测节点，具体用于：根据各检测框的位置信息和与任一任务流程节点关联的触控点的位置信息，从各检测框中确定目标检测框；其中，目标检测框中包括任一任务流程节点所指示的目标界面元素；根据目标检测框的位置信息，确定任一任务流程节点所指示的目标界面元素的元素位置；根据目标检测框的元素类别，确定任一任务流程节点所指示的目标界面元素的元素类别；根据任一任务流程节点所指示的目标界面元素的元素位置和元素类别，生成目标检测节点的识别结果，并生成目标检测节点的识别状态为识别成功状态。In a possible implementation manner of the embodiment of the present application, the target detection node is specifically used to: based on the position information of each detection frame and the position information of the touch point associated with any task process node, from each detection frame Determine the target detection frame; wherein, the target detection frame includes the target interface element indicated by any task process node; according to the position information of the target detection frame, determine the element position of the target interface element indicated by any task process node; according to the target The element category of the detection frame determines the element category of the target interface element indicated by any task process node; based on the element position and element category of the target interface element indicated by any task process node, the identification result of the target detection node is generated, and The recognition status of the generated target detection node is the recognition success status.
在本申请实施例的一种可能的实现方式中，目标检测模型采用以下模块获取得到：In a possible implementation manner of the embodiment of this application, the target detection model is obtained by using the following module:
第三获取模块，用于获取样本图像，其中，样本图像中展示有至少一个样本界面元素；The third acquisition module is used to acquire a sample image, where at least one sample interface element is displayed in the sample image;
检测模块，用于采用初始检测模型对样本图像进行目标检测，以得到至少一个预测框的位置信息和预测框内目标所属的预测元素类别；The detection module is used to perform target detection on the sample image using the initial detection model to obtain the position information of at least one prediction frame and the prediction element category to which the target in the prediction frame belongs;
第一生成模块，用于根据样本图像对应的标注框的位置信息和预测框的位置信息之间的差异，生成回归损失值；The first generation module is used to generate a regression loss value based on the difference between the position information of the annotation frame corresponding to the sample image and the position information of the prediction frame;
第二生成模块，用于根据标注框内样本界面元素所属的标注元素类别和预测框的预测元素类别之间的差异，生成分类损失值；The second generation module is used to generate a classification loss value based on the difference between the annotation element category to which the sample interface element in the annotation frame belongs and the predicted element category of the prediction frame;
训练模块，用于根据回归损失值和分类损失值，对初始检测模型进行训练，以得到目标检测模型。The training module is used to train the initial detection model based on the regression loss value and the classification loss value to obtain the target detection model.
本申请实施例的与任务关联的界面元素的处理装置，通过RPA机器人获取与待执行的目标任务关联的任务流程链，其中，任务流程链中包括至少一个任务流程节点；获取与任务流程链中的各任务流程节点所指示的目标应用界面中的目标界面元素对应的元素位置和元素类别；其中，目标界面元素的元素位置和元素类别是通过调用元素识别方式链中的至少一个元素识别方式节点识别得到的；依次根据各任务流程节点所指示的目标界面元素的元素位置，对相应目标界面元素执行与相应目标界面元素的元素类别匹配的目标操作。由此，可以实现由元素识别方式链中的至少一个元素识别方式节点识别得到与目标任务关联的各目标界面元素的元素位置和元素类别，从而RPA机器人可以根据各目标界面元素的元素位置，对各目标界面元素执行与对应元素类别匹配的操作，以保证目标任务的有效执行，从而利于RPA技术的推广。The device for processing interface elements associated with tasks in the embodiment of the present application uses the RPA robot to obtain the task process chain associated with the target task to be executed, where the task process chain includes at least one task process node; the task process chain is obtained by The element position and element category corresponding to the target interface element in the target application interface indicated by each task process node; wherein, the element position and element category of the target interface element are determined by calling at least one element identification method node in the element identification method chain Recognized; according to the element position of the target interface element indicated by each task process node, perform the target operation matching the element category of the corresponding target interface element on the corresponding target interface element. As a result, the element position and element category of each target interface element associated with the target task can be obtained by identifying at least one element recognition method node in the element recognition method chain, so that the RPA robot can identify the element position and element category of each target interface element based on the element position of each target interface element. Each target interface element performs operations that match the corresponding element category to ensure the effective execution of the target task, thereby facilitating the promotion of RPA technology.
为了实现上述实施例，本申请还提出一种电子设备，其中，电子设备可以为任一具有计算能力的设备，该电子设备包括：存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时，实现如本申请前述任一实施例提出的与任务关联的界面元素的处理方法。In order to implement the above embodiments, this application also proposes an electronic device. The electronic device can be any device with computing capabilities. The electronic device includes: a memory, a processor, and a device stored in the memory and capable of running on the processor. A computer program, when the processor executes the program, implements the method for processing interface elements associated with tasks as proposed in any of the foregoing embodiments of this application.
作为一种示例，图9是本申请一示例性实施例所示出的电子设备900的结构示意图，如图9所示，上述电子设备900，还可以包括：As an example, FIG. 9 is a schematic structural diagram of an electronic device 900 shown in an exemplary embodiment of the present application. As shown in FIG. 9, the above-mentioned electronic device 900 may also include:
存储器910及处理器920，连接不同组件(包括存储器910和处理器920)的总线930，存储器910存储有计算机程序，当处理器920执行所述程序时实现本申请实施例所述的与任务关联的界面元素的处理方法。The memory 910 and the processor 920 are connected to the bus 930 of different components (including the memory 910 and the processor 920). The memory 910 stores a computer program. When the processor 920 executes the program, the task association described in the embodiment of the present application is realized. How to handle interface elements.
总线930表示几类总线结构中的一种或多种，包括存储器总线或者存储器控制器，外围总线，图形加速端口，处理器或者使用多种总线结构中的任意总线结构的局域总线。举例来说，这些体系结构包括但不限于工业标准体系结构(ISA)总线，微通道体系结构(MAC)总线，增强型ISA总线、视频电子标准协会(VESA)局域总线以及外围组件互连(PCI)总线。Bus 930 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect ( PCI) bus.
电子设备900典型地包括多种电子设备可读介质。这些介质可以是任何能够被电子设备900访问的可用介质，包括易失性和非易失性介质，可移动的和不可移动的介质。Electronic device 900 typically includes a variety of electronic device-readable media. These media can be any available media that can be accessed by electronic device 900, including volatile and nonvolatile media, removable and non-removable media.
存储器910还可以包括易失性存储器形式的计算机系统可读介质，例如随机存取存储器(RAM)940和/或高速缓存存储器950。服务器900可以进一步包括其它可移动/不可移动的、易失性/非易失性计算机系统存储介质。仅作为举例，存储系统960可以用于读写不可移动的、非易失性磁介质(图9未显示，通常称为“硬盘驱动器”)。尽管图9中未示出，可以提供用于对可移动非易失性磁盘(例如“软盘”)读写的磁盘驱动器，以及对可移动非易失性光盘(例如CD-ROM，DVD-ROM或者其它光介质)读写的光盘驱动器。在这些情况下，每个驱动器可以通过一个或者多个数据介质接口与总线930相连。存储器910可以包括至少一个程序产品，该程序产品具有一组(例如至少一个)程序模块，这些程序模块被配置以执行本申请各实施例的功能。Memory 910 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 940 and/or cache memory 950. Server 900 may further include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only, storage system 960 may be used to read and write to non-removable, non-volatile magnetic media (not shown in Figure 9, commonly referred to as a "hard drive"). Although not shown in FIG. 9, a disk drive may be provided for reading and writing to removable non-volatile disks (e.g., "floppy disks"), and for removable non-volatile optical disks (e.g., CD-ROM, DVD-ROM or other optical media) that can read and write optical disc drives. In these cases, each drive may be connected to bus 930 through one or more data media interfaces. The memory 910 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of various embodiments of the present application.
具有一组(至少一个)程序模块970的程序/实用工具980，可以存储在例如存储器910中，这样的程序模块970包括——但不限于——操作系统、一个或者多个应用程序、其它程序模块以及程序数据，这些示例中的每一个或某种组合中可能包括网络环境的实现。程序模块970通常执行本申请所描述的实施例中的功能和/或方法。A program/utility 980 having a set of (at least one) program modules 970, which may be stored, for example, in memory 910, such program modules 970 including, but not limited to, an operating system, one or more applications, other programs Modules, as well as program data, each of these examples or some combination may include an implementation of a network environment. Program modules 970 generally perform functions and/or methods in the embodiments described herein.
电子设备900也可以与一个或多个外部设备990(例如键盘、指向设备、显示器991等)通信，还可与一个或者多个使得用户能与该电子设备900交互的设备通信，和/或与使得该电子设备900能与一个或多个其它计算设备进行通信的任何设备(例如网卡，调制解调器等等)通信。这种通信可以通过输入/输出(I/O)接口992进行。并且，电子设备900还可以通过网络适配器993与一个或者多个网络(例如局域网(LAN)，广域网(WAN)和/或公共网络，例如因特网)通信。如图所示，网络适配器993通过总线930与电子设备900的其它模块通信。应当明白，尽管图中未示出，可以结合电子设备900使用其它硬件和/或软件模块，包括但不限于：微代码、设备驱动器、冗余处理单元、外部磁盘驱动阵列、RAID系统、磁带驱动器以及数据备份存储系统等。Electronic device 900 may also communicate with one or more external devices 990 (e.g., keyboard, pointing device, display 991, etc.), with one or more devices that enable a user to interact with electronic device 900, and/or with Any device (eg, network card, modem, etc.) that enables the electronic device 900 to communicate with one or more other computing devices. This communication may occur through input/output (I/O) interface 992. Furthermore, the electronic device 900 may also communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN), and/or a public network, such as the Internet) through a network adapter 993. As shown, network adapter 993 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although not shown in the figures, other hardware and/or software modules may be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives And data backup storage system, etc.
处理器920通过运行存储在存储器910中的程序，从而执行各种功能应用以及数据处理。The processor 920 executes various functional applications and data processing by running programs stored in the memory 910 .
需要说明的是，本实施例的电子设备的实施过程和技术原理参见前述对本申请实施例的与任务关联的界面元素的处理方法的解释说明，此处不再赘述。It should be noted that, for the implementation process and technical principles of the electronic device of this embodiment, please refer to the aforementioned explanation of the processing method of task-related interface elements in the embodiment of this application, and will not be described again here.
为了实现上述实施例，本申请还提出一种非临时性计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如本申请前述任一实施例提出的与任务关联的界面元素的处理方法。In order to implement the above embodiments, the present application also proposes a non-transitory computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the task-related tasks as proposed in any of the foregoing embodiments of the present application are implemented. How to handle interface elements.
为了实现上述实施例，本申请还提出一种计算机程序产品，当所述计算机程序产品中的指令由处理器执行时，执行如本申请前述任一实施例提出的与任务关联的界面元素的处理方法。In order to implement the above embodiments, this application also proposes a computer program product. When instructions in the computer program product are executed by a processor, the processing of interface elements associated with tasks as proposed in any of the previous embodiments of this application is performed. method.
在本说明书的描述中，参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本申请的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不必须针对的是相同的实施例或示例。而且，描述的具体特征、结构、材料或者特点可以在任一个或多个实施例或示例中以合适的方式结合。此外，在不相互矛盾的情况下，本领域的技术人员可以将本说明书中描述的不同实施例或示例以及不同实施例或示例的特征进行结合和组合。In the description of this specification, reference to the terms "one embodiment," "some embodiments," "an example," "specific examples," or "some examples" or the like means that specific features are described in connection with the embodiment or example. , structures, materials or features are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine different embodiments or examples and features of different embodiments or examples described in this specification unless they are inconsistent with each other.
此外，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括至少一个该特征。在本申请的描述中，“多个”的含义是至少两个，例如两个，三个等，除非另有明确具体的限定。In addition, the terms “first” and “second” are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Therefore, features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of this application, "plurality" means at least two, such as two, three, etc., unless otherwise expressly and specifically limited.
流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为，表示包括一个或更多个用于实现定制逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分，并且本申请的优选实施方式的范围包括另外的实现，其中可以不按所示出或讨论的顺序，包括根据所涉及的功能按基本同时的方式或按相反的顺序，来执行功能，这应被本申请的实施例所属技术领域的技术人员所理解。Any process or method descriptions in flowcharts or otherwise described herein may be understood to represent modules, segments, or portions of code that include one or more executable instructions for implementing customized logical functions or steps of the process. , and the scope of the preferred embodiments of the present application includes additional implementations in which functions may be performed out of the order shown or discussed, including in a substantially simultaneous manner or in the reverse order, depending on the functionality involved, which shall It should be understood by those skilled in the technical field to which the embodiments of this application belong.
在流程图中表示或在此以其他方式描述的逻辑和/或步骤，例如，可以被认为是用于实现逻辑功能的可执行指令的定序列表，可以具体实现在任何计算机可读介质中，以供指令执行系统、装置或设备(如基于计算机的系统、包括处理器的系统或其他可以从指令执行系统、装置或设备取指令并执行指令的系统)使用，或结合这些指令执行系统、装置或设备而使用。就本说明书而言，"计算机可读介质"可以是任何可以包含、存储、通信、传播或传输程序以供指令执行系统、装置或设备或结合这些指令执行系统、装置或设备而使用的装置。计算机可读介质的更具体的示例(非穷尽性列表)包括以下：具有一个或多个布线的电连接部(电子装置)，便携式计算机盘盒(磁装置)，随机存取存储器(RAM)，只读存储器(ROM)，可擦除可编辑只读存储器(EPROM或闪速存储器)，光纤装置，以及便携式光盘只读存储器(CDROM)。另外，计算机可读介质甚至可以是可在其上打印所述程序的纸或其他合适的介质，因为可以例如通过对纸或其他介质进行光学扫描，接着进行编辑、解译或必要时以其他合适方式进行处理来以电子方式获得所述程序，然后将其存储在计算机存储器中。The logic and/or steps represented in the flowcharts or otherwise described herein, for example, may be considered a sequenced list of executable instructions for implementing the logical functions, and may be embodied in any computer-readable medium, For use by, or in combination with, instruction execution systems, devices or devices (such as computer-based systems, systems including processors or other systems that can fetch instructions from and execute instructions from the instruction execution system, device or device) or equipment. For the purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections with one or more wires (electronic device), portable computer disk cartridges (magnetic device), random access memory (RAM), Read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium may even be paper or other suitable medium on which the program may be printed, as the paper or other medium may be optically scanned, for example, and subsequently edited, interpreted, or otherwise suitable as necessary. process to obtain the program electronically and then store it in computer memory.
应当理解，本申请的各部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中，多个步骤或方法可以用存储在存储器中且由合适的指令执行系统执行的软件或固件来实现。如，如果用硬件来实现和在另一实施方式中一样，可用本领域公知的下列技术中的任一项或他们的组合来实现：具有用于对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路，具有合适的组合逻辑门电路的专用集成电路，可编程门阵列(PGA)，现场可编程门阵列(FPGA)等。It should be understood that various parts of the present application can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if it is implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: discrete logic gate circuits with logic functions for implementing data signals; Logic circuits, application specific integrated circuits with suitable combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.
本技术领域的普通技术人员可以理解实现上述实施例方法携带的全部或部分步骤是可以通过程序来指令相关的硬件完成，所述的程序可以存储于一种计算机可读存储介质中，该程序在执行时，包括方法实施例的步骤之一或其组合。Those of ordinary skill in the art can understand that all or part of the steps involved in implementing the methods of the above embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. The program can be stored in a computer-readable storage medium. When executed, one of the steps of the method embodiment or a combination thereof is included.
此外，在本申请各个实施例中的各功能单元可以集成在一个处理模块中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。所述集成的模块如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。In addition, each functional unit in various embodiments of the present application can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software function modules. If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
上述提到的存储介质可以是只读存储器，磁盘或光盘等。尽管上面已经示出和描述了本申请的实施例，可以理解的是，上述实施例是示例性的，不能理解为对本申请的限制，本领域的普通技术人员在本申请的范围内可以对上述实施例进行变化、修改、替换和变型。The storage media mentioned above can be read-only memory, magnetic disks or optical disks, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above-mentioned embodiments are illustrative and cannot be understood as limitations of the present application. Those of ordinary skill in the art can make modifications to the above-mentioned embodiments within the scope of the present application. The embodiments are subject to changes, modifications, substitutions and variations.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310854675.1ACN116932111A (en) | 2023-07-12 | 2023-07-12 | Processing method of interface element associated with task |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310854675.1ACN116932111A (en) | 2023-07-12 | 2023-07-12 | Processing method of interface element associated with task |
| Publication Number | Publication Date |
|---|---|
| CN116932111Atrue CN116932111A (en) | 2023-10-24 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310854675.1APendingCN116932111A (en) | 2023-07-12 | 2023-07-12 | Processing method of interface element associated with task |
| Country | Link |
|---|---|
| CN (1) | CN116932111A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112231033A (en)* | 2019-12-23 | 2021-01-15 | 北京来也网络科技有限公司 | Software interface element matching method and device combining RPA and AI |
| CN113015956A (en)* | 2019-10-14 | 2021-06-22 | 尤帕斯公司 | System and method for active target selection for robotic process automation |
| CN113391871A (en)* | 2021-08-17 | 2021-09-14 | 杭州实在智能科技有限公司 | RPA element intelligent fusion picking method and system |
| CN114926139A (en)* | 2022-03-23 | 2022-08-19 | 上海来也伯特网络科技有限公司 | Online business handling method and device combining RPA and AI and electronic equipment |
| CN116010066A (en)* | 2023-03-21 | 2023-04-25 | 山东小数点信息技术有限公司 | RPA robot and implementation method |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113015956A (en)* | 2019-10-14 | 2021-06-22 | 尤帕斯公司 | System and method for active target selection for robotic process automation |
| CN112231033A (en)* | 2019-12-23 | 2021-01-15 | 北京来也网络科技有限公司 | Software interface element matching method and device combining RPA and AI |
| CN113391871A (en)* | 2021-08-17 | 2021-09-14 | 杭州实在智能科技有限公司 | RPA element intelligent fusion picking method and system |
| CN114926139A (en)* | 2022-03-23 | 2022-08-19 | 上海来也伯特网络科技有限公司 | Online business handling method and device combining RPA and AI and electronic equipment |
| CN116010066A (en)* | 2023-03-21 | 2023-04-25 | 山东小数点信息技术有限公司 | RPA robot and implementation method |
| Publication | Publication Date | Title |
|---|---|---|
| US12051236B2 (en) | Method for recognizing video action, and device and storage medium thereof | |
| CN109948542B (en) | Gesture recognition method, device, electronic device and storage medium | |
| CN110610510B (en) | Target tracking method, device, electronic device and storage medium | |
| CN110837403B (en) | Robot process automation | |
| US20190095758A1 (en) | Method and system for obtaining picture annotation data | |
| WO2020011014A1 (en) | Method and system for detecting and recognizing object in real-time video, storage medium and device | |
| CN117893845A (en) | Methods, AI platforms, computing devices, and storage media for providing AI models | |
| CN110851299A (en) | Abnormal elimination method, device, equipment and storage medium for automated process | |
| CN107666987A (en) | Robotic process automates | |
| CN113609028A (en) | Page testing method, computing device and readable storage medium | |
| CN108829371A (en) | Interface control method, device, storage medium and electronic equipment | |
| CN111191590A (en) | Model training method and device, storage medium and electronic equipment | |
| WO2022193632A1 (en) | Gesture recognition method and apparatus, and electronic device | |
| WO2024012367A1 (en) | Visual-target tracking method and apparatus, and device and storage medium | |
| CN113641933A (en) | Abnormal webpage identification method, abnormal site identification method and device | |
| CN112612469A (en) | Interface element processing method and device and electronic equipment | |
| CN114495103B (en) | Text recognition method and device, electronic equipment and medium | |
| CN112579587B (en) | Data cleaning method and device, equipment and storage medium | |
| CN115917446A (en) | Systems and methods for robotic process automation | |
| CN115658523A (en) | Automatic control and test method for human-computer interaction interface and computer equipment | |
| CN114842261A (en) | Image processing method, device, electronic device and storage medium | |
| CN110443191A (en) | The method and apparatus of article for identification | |
| CN113139095A (en) | Video retrieval method and device, computer equipment and medium | |
| CN116932111A (en) | Processing method of interface element associated with task | |
| CN108052506A (en) | Natural language processing method, apparatus, storage medium and electronic equipment |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |