技术领域Technical Field
本发明涉及游戏自动化领域，具体的说是一种基于OCR识别的游戏自动化系统及方法。The present invention relates to the field of game automation, and in particular to a game automation system and method based on OCR recognition.
背景技术Background Art
游戏自动化是一个复杂的过程，涉及到多个领域的技术挑战，当前游戏自动化面临以下难点：1、游戏环境具有多样性，不同的游戏有不同的规则、界面和交互方式，自动化系统需要能够适应这些差异;2、自动化系统需要能够识别和解析游戏中的视觉信息，包括角色、物品、环境等，这在图形复杂或像素化的游戏上尤为困难;3、游戏自动化需要实时响应游戏环境的变化，这对算法的效率和响应速度提出了高要求；4、游戏开发者可能会更新游戏规则或修复漏洞，这要求自动化系统需要能够适应这些变化。环顾目前现有的游戏自动化系统，专业性强，但受限于元素定位、图像匹配，目前没有解决包含游戏界面多样性、图形复杂性等诸多难点组合而成的技术解决方案或者系统。Game automation is a complex process involving technical challenges in multiple fields. Currently, game automation faces the following difficulties: 1. The game environment is diverse. Different games have different rules, interfaces, and interaction methods. The automation system needs to be able to adapt to these differences; 2. The automation system needs to be able to recognize and parse visual information in the game, including characters, items, environments, etc., which is particularly difficult in games with complex or pixelated graphics; 3. Game automation needs to respond to changes in the game environment in real time, which places high demands on the efficiency and response speed of the algorithm; 4. Game developers may update game rules or fix vulnerabilities, which requires the automation system to be able to adapt to these changes. Looking around the existing game automation systems, they are highly professional, but are limited by element positioning and image matching. There is currently no technical solution or system that solves the combination of many difficulties, including the diversity of game interfaces and the complexity of graphics.
例如PyAutoGame（基于Python的游戏自动化框架），它的核心是屏幕识别和事件模拟，它利用OpenCV (跨平台计算机视觉库)进行图像处理，通过对屏幕截图进行模板匹配或颜色识别，定位游戏中的元素，然后使用PyAutoGUI模块来模拟鼠标和键盘操作，精确控制游戏中的行为。但是随着技术的进步，真实的业务场景的复杂度超越了以往，单纯通过模板匹配和颜色识别已经无法稳定定位游戏中的元素。For example, PyAutoGame (a Python-based game automation framework) is centered around screen recognition and event simulation. It uses OpenCV (a cross-platform computer vision library) for image processing, locates game elements by performing template matching or color recognition on screenshots, and then uses the PyAutoGUI module to simulate mouse and keyboard operations to precisely control game behaviors. However, with the advancement of technology, the complexity of real business scenarios has surpassed that of the past, and simply using template matching and color recognition can no longer stably locate game elements.
再例如公开号为CN115858358A的中国专利提供了一种多端自动化游戏测试系统，其中自动测试模块对当前处于活动状态的客户端中的屏幕区域进行识别和点击操作，系统将游戏分成若干段同时进行测试，从而提高测试的效率，降低常规测试的时间成本，该技术方案能够解决游戏自动化中需要多次重复测试和多次修复的问题，但其缺少对游戏场景多变，元素多变的技术能力支持。For another example, a Chinese patent with publication number CN115858358A provides a multi-terminal automated game testing system, in which the automatic testing module identifies and clicks on the screen area in the currently active client, and the system divides the game into several segments for simultaneous testing, thereby improving the efficiency of the test and reducing the time cost of conventional testing. This technical solution can solve the problem of multiple repeated tests and multiple repairs required in game automation, but it lacks technical support for changing game scenes and elements.
发明内容Summary of the invention
鉴于现有自动化系统存在的缺陷，本发明所要解决的技术问题是：提供一种基于OCR识别的游戏自动化系统及方法，其能够解决因为复杂的游戏环境、实时的页面变化带来的游戏元素难识别、游戏场景难定位的问题。In view of the defects of the existing automation system, the technical problem to be solved by the present invention is to provide a game automation system and method based on OCR recognition, which can solve the problems of difficult identification of game elements and difficult positioning of game scenes caused by complex game environment and real-time page changes.
为了实现上述目的，根据本发明的一个方面，本发明是通过如下技术措施实现的：一种基于OCR识别的游戏自动化系统，包括业务模块、自动化模块、OCR识别模块、调度模块；To achieve the above object, according to one aspect of the present invention, the present invention is implemented by the following technical measures: a game automation system based on OCR recognition, including a business module, an automation module, an OCR recognition module, and a scheduling module;
所述业务模块用于自动化配置信息管理、任务管理，执行设备管理；The business module is used for automated configuration information management, task management, and execution equipment management;
所述自动化模块用于匹配自动化脚本，控制任务队列，分布式调度任务，规范化自动化执行结果；The automation module is used to match automation scripts, control task queues, distribute scheduling tasks, and standardize automation execution results;
所述OCR识别模块用于接收自动化执行过程中的截图样本，判断当前自动化执行的游戏状态，提取和回调自动化场景中的关键参数；The OCR recognition module is used to receive screenshot samples during the automation execution process, determine the current game status of the automation execution, and extract and call back key parameters in the automation scene;
所述调度模块用于分配任务执行的设备节点，控制任务执行的队列，监控系统下设备状态。The scheduling module is used to allocate device nodes for task execution, control the queue for task execution, and monitor the device status under the system.
进一步地，所述业务模块通过接口配置自动化任务中解析参数、下载代码等场景的密钥信息，修改设备报警阈值，将任务和执行设备集群关联。Furthermore, the business module configures key information for scenarios such as parsing parameters and downloading codes in the automation task through the interface, modifies the device alarm threshold, and associates the task with the execution device cluster.
进一步地，所述自动化模块接收校验通过的有效请求参数，根据参数匹配用于执行对应游戏自动化任务的设备节点集群，通过查询其中的可用设备节点，把自动化任务下发到匹配的可用设备节点上，根据数据模块配置的工程地址从云端仓库上下载游戏自动化代码到设备节点上，执行游戏自动化代码并监控执行过程，执行过程通过自动化截图并通过request库请求OCR识别模块的服务来判断当前游戏状态，通过OCR识别模块的返回结果来决定下一步的自动化操作策略，达到精准识别游戏元素和状态的目的。Furthermore, the automation module receives valid request parameters that have passed verification, matches the device node cluster used to execute the corresponding game automation task according to the parameters, queries the available device nodes therein, sends the automation task to the matching available device nodes, downloads the game automation code from the cloud warehouse to the device node according to the project address configured by the data module, executes the game automation code and monitors the execution process. During the execution process, the current game status is determined by automatically taking screenshots and requesting the service of the OCR recognition module through the request library, and the next automation operation strategy is determined by the return result of the OCR recognition module, so as to achieve the purpose of accurately identifying game elements and status.
进一步地，所述OCR识别模块接收自动化脚本在游戏的关键点自动截取屏幕图像，并利用图像处理技术对截图进行预处理，以提高OCR识别的准确性，然后使用图像识别技术来定位截图中包含关键信息的区域，再使用OCR技术对定位区域内的文本进行识别，将识别出的文本转换为结构化数据，例如如果识别到的是血量，将其转换为数值；如果是任务描述，提取关键词或任务ID；然后根据提取的信息，通过自动化脚本判断当前游戏的状态，再将识别和提取的关键参数回调到自动化脚本中，以便脚本根据这些参数做出决策，例如如果血量低于某个阈值，判断需要执行补血操作时会触发补血机制，如果识别到特定任务关键词，判断需要切换到相应的任务执行流程使会触发任务导航；最后自动化脚本在执行操作后，再次进行截图和识别，以监控操作的效果，并根据需要调整策略。Furthermore, the OCR recognition module receives an automated script that automatically captures screen images at key points in the game, and uses image processing technology to pre-process the screenshots to improve the accuracy of OCR recognition, then uses image recognition technology to locate areas in the screenshots that contain key information, and then uses OCR technology to recognize text in the located area, and converts the recognized text into structured data. For example, if the health volume is recognized, it is converted into a numerical value; if it is a task description, keywords or task IDs are extracted; then based on the extracted information, the current state of the game is determined through the automated script, and the identified and extracted key parameters are called back to the automated script so that the script can make decisions based on these parameters. For example, if the health volume is lower than a certain threshold, it is determined that a blood replenishment operation needs to be performed, which will trigger a blood replenishment mechanism. If a specific task keyword is recognized, it is determined that it is necessary to switch to the corresponding task execution process, which will trigger task navigation; finally, after executing the operation, the automated script takes a screenshot and recognizes again to monitor the effect of the operation and adjust the strategy as needed.
进一步地，所述关键点包括游戏的菜单界面、战斗场景、任务提示等，所述图像处理技术包括灰度化、二值化、去噪、边缘检测等。Furthermore, the key points include the game's menu interface, battle scenes, task prompts, etc., and the image processing technology includes grayscale, binarization, denoising, edge detection, etc.
进一步地，所述调度模块在接收到自动化任务的情况下，分配任务执行的设备节点，控制任务执行的队列，监控设备节点集群的连接状态，在设备连接断开的情况下进行特定项目群报警；同时接收自动化模块的请求，在自动化执行过程中遇到游戏更新、异常断网等异常情况下，执行禁用、重启、重连设备节点的指令策略。Furthermore, when the scheduling module receives an automation task, it allocates device nodes for task execution, controls the queue for task execution, monitors the connection status of the device node cluster, and issues an alarm for a specific project group when the device connection is disconnected; at the same time, it receives requests from the automation module, and executes instruction strategies for disabling, restarting, and reconnecting device nodes when encountering abnormal situations such as game updates and abnormal network disconnections during the automation execution process.
本发明的另一方面，提供了一种基于前述的基于OCR识别的游戏自动化系统的游戏自动化方法，所述方法包括如下步骤：Another aspect of the present invention provides a game automation method based on the aforementioned game automation system based on OCR recognition, the method comprising the following steps:
S1、设计游戏自动化脚本，其中游戏界面识别环节借助OCR模块的识别返回来制定自动化执行策略；S1. Design game automation scripts, in which the game interface recognition link uses the recognition return of the OCR module to formulate an automation execution strategy;
S2、本地自动化脚本调试通过后，将代码上传到GitLab，通过配置管理模块新增该游戏自动化场景配置，通过任务配置模块设置脚本需要接收的参数和执行的机器节点；S2. After the local automation script is debugged, upload the code to GitLab, add the game automation scene configuration through the configuration management module, and set the parameters that the script needs to receive and the machine node to execute through the task configuration module;
S3、通过自动化模块提供的接口发起游戏自动化任务，根据接口返回的自动化任务唯一id可以查询任务状态，自动化模块在接受到请求后会对参数进行解析和校验，如果校验通过则通知调度层查询可执行机器并把任务分配空闲的可执行机器上，如果可执行机器不足则执行报警策略；S3. Initiate a game automation task through the interface provided by the automation module. The task status can be queried based on the unique ID of the automation task returned by the interface. After receiving the request, the automation module will parse and verify the parameters. If the verification passes, the scheduling layer will be notified to query the executable machine and assign the task to an idle executable machine. If the executable machine is insufficient, the alarm strategy will be executed.
S4、自动化任务在执行机器上启动，调度层会清理工作空间，然后把S1中的自动化脚本下载到执行机器上并启动，游戏脚本驱动对应的游戏程序，在执行过程中进行截图上传，然后请求OCR接口，根据OCR接口返回的信息来判断当前游戏状态，以此来执行脚本中下一步的策略，在整个游戏自动化的过程中重复以上步骤；S4, the automation task is started on the execution machine. The scheduling layer will clean up the workspace, then download the automation script in S1 to the execution machine and start it. The game script drives the corresponding game program, takes screenshots and uploads them during the execution process, and then requests the OCR interface. The current game status is determined based on the information returned by the OCR interface, so as to execute the next strategy in the script. The above steps are repeated throughout the entire game automation process.
S5、游戏自动化正常结束或者出现异常后，自动化脚本会把执行的结果，状态回调给自动化模块，自动化模块将参数格式化后存储入库，如果出现异常，则根据配置管理中预设的报警配置，进行告警推送，引导开发人员定位问题；S5. After the game automation ends normally or an exception occurs, the automation script will call back the execution result and status to the automation module. The automation module will format the parameters and store them in the library. If an exception occurs, an alarm will be pushed according to the alarm configuration preset in the configuration management to guide the developer to locate the problem.
S6、开发人员可以根据任务id、任务类型、任务名称等字段通过自动化模块查询任务日志、任务状态，根据游戏场景、时间拉取数据统计报表。S6. Developers can query task logs and task status through the automation module based on fields such as task ID, task type, and task name, and pull data statistical reports based on game scenarios and time.
与现有技术相比，本发明的优点在于：本一种基于OCR识别的游戏自动化系统及方法，针对游戏自动化领域元素变化大、难定位等业务特点，提供了一种高稳定性、高容错率的游戏全自动化解决方案。Compared with the prior art, the advantages of the present invention are: this game automation system and method based on OCR recognition provides a highly stable and high fault-tolerant game automation solution for business characteristics such as large element changes and difficult positioning in the field of game automation.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
构成本发明一部分的附图用来提供对本发明的进一步理解，本发明的示意性实施例及其说明用于解释本发明，并不构成对本发明的不当限定。在附图中：The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:
图1为本发明所述的一种基于OCR识别的游戏自动化系统的系统架构示意图；FIG1 is a schematic diagram of the system architecture of a game automation system based on OCR recognition according to the present invention;
图2为本发明所述的一种游戏自动化方法的执行流程图；FIG2 is a flowchart of an execution of a game automation method according to the present invention;
图3为本发明所述的一种游戏自动化方法的报警策略流程图。FIG. 3 is a flow chart of an alarm strategy of a game automation method according to the present invention.
具体实施方式DETAILED DESCRIPTION
下文中将参考附图并结合实施例来详细说明本发明。需要说明的是，在不冲突的情况下，本申请中的实施例及实施例中的特征可以相互组合。The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.
在本发明的描述中，需要理解的是，术语“中心”、“上”、“下”、“左”、“右”、“竖直”、“水平”、“内”、“外”、“底部”、“顶部”等指示的方位或位置关系为基于附图所示的方位或位置关系，仅是为了便于描述本发明和简化描述，而不是指示或暗示所指的装置或元件必须具有特定的方位、以特定的方位构造和操作，因此不能理解为对本发明的限制。In the description of the present invention, it is necessary to understand that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", "bottom", "top", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.
请参照图1，本实施例提供的一种基于OCR识别的游戏自动化系统，包括业务模块、自动化模块、OCR识别模块、调度模块；Please refer to FIG. 1 , a game automation system based on OCR recognition provided in this embodiment includes a business module, an automation module, an OCR recognition module, and a scheduling module;
业务模块用于自动化配置信息管理、任务管理，执行设备管理，业务模块通过接口配置自动化任务中解析参数、下载代码等场景的密钥信息，修改设备报警阈值，将任务和执行设备集群关联；The business module is used for automated configuration information management, task management, and execution device management. The business module uses the interface to configure the key information for scenarios such as parsing parameters and downloading codes in automated tasks, modify device alarm thresholds, and associate tasks with execution device clusters.
自动化模块用于匹配自动化脚本，控制任务队列，分布式调度任务，规范化自动化执行结果，自动化模块接收校验通过的有效请求参数，根据参数匹配用于执行对应游戏自动化任务的设备节点集群，通过查询其中的可用设备节点，把自动化任务下发到匹配的可用设备节点上，根据数据模块配置的工程地址从云端仓库上下载游戏自动化代码到设备节点上，执行游戏自动化代码并监控执行过程，执行过程通过自动化截图并通过request库请求OCR识别模块的服务来判断当前游戏状态，通过OCR识别模块的返回结果来决定下一步的自动化操作策略，达到精准识别游戏元素和状态的目的；The automation module is used to match automation scripts, control task queues, distribute scheduling tasks, and standardize automation execution results. The automation module receives valid request parameters that have passed verification, matches the device node cluster used to execute the corresponding game automation task according to the parameters, queries the available device nodes, sends the automation task to the matching available device nodes, downloads the game automation code from the cloud warehouse to the device node according to the project address configured by the data module, executes the game automation code and monitors the execution process. The execution process is judged by automatic screenshots and requesting the service of the OCR recognition module through the request library, and determines the next automation operation strategy based on the return result of the OCR recognition module, so as to achieve the purpose of accurately identifying game elements and status;
OCR识别模块用于接收自动化执行过程中的截图样本，判断当前自动化执行的游戏状态，提取和回调自动化场景中的关键参数，OCR识别模块接收自动化脚本在游戏的关键点自动截取屏幕图像，并对截图进行预处理，以提高OCR识别的准确性，关键点包括游戏的菜单界面、战斗场景、任务提示等，图像处理技术包括灰度化、二值化、去噪、边缘检测等；然后使用图像识别技术（如模板匹配、特征点匹配等）来定位截图中包含关键信息的区域（例如需要识别并定位到角色的血量条、经验值、任务描述等），再使用OCR技术对定位区域内的文本进行识别（针对特定游戏的字体可能需要进行单独训练），将识别出的文本转换为结构化数据（例如如果识别到的是血量，可以将其转换为数值；如果是任务描述，可以提取关键词或任务ID），然后根据提取的信息，自动化脚本判断当前游戏的状态（例如如果血量低于某个阈值，判断可能需要执行补血操作；如果识别到特定任务关键词，判断可能需要切换到相应的任务执行流程），然后将识别和提取的关键参数回调到自动化脚本中，以便脚本根据这些参数做出决策（例如血量信息可能触发补血机制，任务信息可能触发任务导航），最后自动化脚本在执行操作后，再次进行截图和识别，以监控操作的效果，并根据需要调整策略；The OCR recognition module is used to receive screenshot samples during the automation execution process, determine the current game status of the automation execution, extract and call back key parameters in the automation scene, and receive the automation script to automatically capture the screen image at the key points of the game, and pre-process the screenshot to improve the accuracy of OCR recognition. The key points include the game's menu interface, battle scenes, task prompts, etc. The image processing techniques include grayscale, binarization, denoising, edge detection, etc.; then use image recognition technology (such as template matching, feature point matching, etc.) to locate the area containing key information in the screenshot (for example, the character's health bar, experience value, task description, etc. that need to be identified and located), and then use OCR technology to recognize the text in the located area (for specific game fonts, it may be necessary to Conduct separate training), convert the recognized text into structured data (for example, if the health is recognized, it can be converted into a numerical value; if it is a task description, keywords or task IDs can be extracted), and then the automated script determines the current game status based on the extracted information (for example, if the health is below a certain threshold, it is judged that a blood replenishment operation may need to be performed; if a specific task keyword is recognized, it is judged that it may be necessary to switch to the corresponding task execution process), and then the key parameters identified and extracted are called back to the automated script so that the script can make decisions based on these parameters (for example, health information may trigger a blood replenishment mechanism, and task information may trigger task navigation). Finally, after executing the operation, the automated script takes screenshots and recognizes again to monitor the effect of the operation and adjust the strategy as needed;
调度模块用于分配任务执行的设备节点，控制任务执行的队列，监控系统下设备状态，调度模块在接收到自动化任务的情况下，分配任务执行的设备节点，控制任务执行的队列，监控设备节点集群的连接状态，在设备连接断开的情况下进行特定项目群报警；同时接收自动化模块的请求，在自动化执行过程中遇到游戏更新、异常断网等异常情况下，执行禁用、重启、重连设备节点的指令策略。The scheduling module is used to assign device nodes for task execution, control the queue for task execution, and monitor the device status under the system. When receiving an automated task, the scheduling module assigns device nodes for task execution, controls the queue for task execution, monitors the connection status of the device node cluster, and issues an alarm for a specific project group when the device connection is disconnected. At the same time, it receives requests from the automation module, and executes instruction strategies to disable, restart, and reconnect device nodes in the event of abnormal situations such as game updates and abnormal network disconnections during the automated execution process.
请参照图1，本实施例提供的一种基于OCR识别的游戏自动化系统的系统架构包括如下内容：Referring to FIG. 1 , the system architecture of a game automation system based on OCR recognition provided in this embodiment includes the following contents:
1、业务层，包括任务管理、配置管理、脚本管理、设备管理；1. Business layer, including task management, configuration management, script management, and device management;
1.1任务管理：以接口的形式提供任务列表的动态参数配置，任务列表以字典形式存储在数据库中，其中的key代表具体的游戏自动化场景，value代表游戏场景和自动化任务的关联识别码，当用户在页面上发起自动化游戏任务请求时，自动化模块会在权限校验通过之后，解析任务参数并和任务列表的动态参数配置做匹配，从而关联到唯一的一个自动化任务，再通过自动化任务去调度空余的节点机执行对应的自动化游戏脚本，任务管理提供了任务列表的新增、修改、删除等接口，用于对任务列表进行管理；1.1 Task management: Dynamic parameter configuration of the task list is provided in the form of an interface. The task list is stored in the database in the form of a dictionary, where the key represents a specific game automation scene, and the value represents the associated identification code between the game scene and the automation task. When a user initiates an automated game task request on the page, the automation module will parse the task parameters and match them with the dynamic parameter configuration of the task list after the permission check is passed, thereby associating it with a unique automation task. The automation task is then used to schedule the idle node machines to execute the corresponding automated game script. Task management provides interfaces for adding, modifying, and deleting task lists to manage the task list;
1.2配置管理：以接口的形式提供配置信息的管理途径，其中包括推送报警信息的项目群和邮件组，包括任务参数进行哈希处理的密钥，包括GitLab拉取代码仓库的密钥，包括任务发起检测的执行机器阈值（在任务发起时先检测空余的执行机器，空余机器数目少于特定参数时，会有特定的报警设置）；1.2 Configuration management: Provides management channels for configuration information in the form of interfaces, including project groups and email groups for pushing alarm information, keys for hashing task parameters, keys for GitLab to pull code repositories, and execution machine thresholds for task initiation detection (free execution machines are detected first when a task is initiated. When the number of free machines is less than a specific parameter, a specific alarm setting will be set);
1.3脚本管理：以代码仓库的形式统一在GitLab上管理，接口层面提供检测和远程调度能力，支持在执行节点上一键更新代码，一键拉取代码；1.3 Script management: managed uniformly on GitLab in the form of code repositories, providing detection and remote scheduling capabilities at the interface level, supporting one-click code updates and one-click code pulls on execution nodes;
1.4设备管理：以接口的形式提供设备管理能力，在自动化模块中集成了Jenkins的组件库，所有的自动化运行设备都以节点的形式连接到Jenkins组件中，通过Jenkins提供的API设备的连接、禁用、启动、查询、日志导出等等的能力；同时这批设备也接入了JumpServer堡垒机，支持通过链接分享形式的可视化远程连接和调试。1.4 Device management: Provides device management capabilities in the form of interfaces. The Jenkins component library is integrated into the automation module. All automated running devices are connected to the Jenkins component in the form of nodes, and the API provided by Jenkins provides the ability to connect, disable, start, query, export logs, etc. At the same time, these devices are also connected to the JumpServer bastion host, which supports visual remote connection and debugging through link sharing.
2、服务层，包括自动化模块和OCR模块；2. Service layer, including automation module and OCR module;
2.1自动化模块：以接口的形式提供自动化服务，提供包括但不限于以下能力：请求权限和参数的校验，校验通过后进行游戏自动化任务的发起，在任务进行中通过和OCR模块互相调用保持对游戏自动化状态的监控，在自动化任务结束后接收脚本的执行结果回调，把回调的交过结构化之后存储到数据库中，支持对游戏自动化任务的查询，在游戏自动化任务出现异常时支持对任务的重拾，支持接收各种游戏截图的上传和下载，支持在脚本运行过程中进行验证码等动态参数的接收和查询；2.1 Automation module: Provides automation services in the form of interfaces, including but not limited to the following capabilities: request permission and parameter verification, initiate game automation tasks after verification, monitor the game automation status by calling the OCR module during the task, receive the script execution result callback after the automation task is completed, store the callback in the database after structuring, support query of game automation tasks, support task resumption when an exception occurs in the game automation task, support upload and download of various game screenshots, and support receiving and querying dynamic parameters such as verification codes during script execution;
2.2 OCR模块：以接口的形式提供OCR服务，提供包括但不限于以下能力：对截图进行预处理，以提高OCR识别的准确性，这可能包括灰度化、二值化、去噪、边缘检测等图像处理技术；使用图像识别技术（如模板匹配、特征点匹配等）来定位截图中包含关键信息的区域，例如需要识别并定位到角色的血量条、经验值、任务描述等；用OCR技术对定位区域内的文本进行识别，将识别出的文本转换为结构化数据，例如如果识别到的是血量，可以将其转换为数值；如果是任务描述，可以提取关键词或任务ID；将识别和提取的关键参数回调到自动化脚本中，以便脚本根据这些参数做出决策，例如血量信息可能触发补血机制，任务信息可能触发任务导航；设计错误处理机制，以应对OCR识别错误或自动化操作失败的情况，这可能包括重试机制、异常捕获和日志记录。2.2 OCR module: Provides OCR service in the form of an interface, including but not limited to the following capabilities: Pre-processing screenshots to improve the accuracy of OCR recognition, which may include image processing technologies such as grayscale, binarization, denoising, and edge detection; using image recognition technology (such as template matching, feature point matching, etc.) to locate areas in the screenshot that contain key information, such as the character's health bar, experience value, task description, etc. that need to be identified and located; using OCR technology to recognize text in the located area and convert the recognized text into structured data. For example, if the recognized text is health, it can be converted into a numerical value; if it is a task description, keywords or task IDs can be extracted; calling back the identified and extracted key parameters to the automation script so that the script can make decisions based on these parameters, for example, health information may trigger a blood replenishment mechanism, and task information may trigger task navigation; designing an error handling mechanism to deal with OCR recognition errors or automated operation failures, which may include a retry mechanism, exception capture, and logging.
3、调度层：3. Scheduling layer:
3.1调度模块：主要由Jenkins和JumpServer组成，前者Jenkins负责在收到游戏自动化任务时，根据任务关联的节点机器信息分配空闲的节点机器执行脚本，执行的过程中先进行代码本地空间的清理，然后根据任务的配置信息和密钥，从GitLab上拉取游戏脚本代码执行，同时跟踪任务的执行情况；同时提供了API供自动化模块对任务，节点进行管理；后者JumpServer提供了远程可视化连接节点机器的能力，可以让开发人员实时跟踪到游戏自动化任务的执行状态。3.1 Scheduling module: It is mainly composed of Jenkins and JumpServer. The former Jenkins is responsible for allocating idle node machines to execute scripts according to the node machine information associated with the task when receiving the game automation task. During the execution process, the local code space is first cleaned up, and then the game script code is pulled from GitLab for execution according to the configuration information and key of the task, while tracking the execution of the task; at the same time, an API is provided for the automation module to manage tasks and nodes; the latter JumpServer provides the ability to remotely visualize the connection to the node machine, allowing developers to track the execution status of the game automation task in real time.
4、数据层：4. Data layer:
4.1 OSS：提供了截图管理的能力，自动化脚本在执行过程中的截图会通过自动化模块提供的接口上传到OSS上，同时回调OCR模块对应的截图OSS外链，OCR模块会通过OSS外链下载截图进行识别和信息提取，并回调结果给游戏自动化脚本;4.1 OSS: Provides screenshot management capabilities. The screenshots taken during the execution of the automation script will be uploaded to OSS through the interface provided by the automation module, and the screenshot OSS external link corresponding to the OCR module will be called back. The OCR module will download the screenshots through the OSS external link for recognition and information extraction, and call back the results to the game automation script;
4.2 Redis：提供任务信息，配置信息等数据的缓存服务;4.2 Redis: Provides cache services for task information, configuration information and other data;
4.3 MySQL：提供自动化任务参数，自动化执行结果，动态参数配置，节点机列表等信息的存储服务。4.3 MySQL: Provides storage services for information such as automated task parameters, automated execution results, dynamic parameter configuration, and node machine lists.
5、运行环境：5. Operating environment:
5.1电脑节点：提供了阿里云电脑主机，实体电脑主机用于端游的自动化硬件支撑，这些电脑上需要安装JDK，Git，python,其中JDK用于Jenkins的接入，Git用于从Gitlab上拉取游戏自动化脚本，python用于游戏自动化的运行环境依赖，这些电脑主机都作为Jenkins的节点接入到自动化模块中，接受自动化模块的统一调度和管理；5.1 Computer nodes: Alibaba Cloud computer hosts are provided. Physical computer hosts are used for hardware support of automation of end-games. JDK, Git, and Python need to be installed on these computers. JDK is used for access to Jenkins, Git is used to pull game automation scripts from Gitlab, and Python is used for the running environment dependency of game automation. These computer hosts are connected to the automation module as Jenkins nodes and accept unified scheduling and management of the automation module.
5.2移动节点：提供了移动云真机和移动模拟器，移动端的执行机通过usb连接到某一台或者多台电脑节点上，这些电脑节点在安装JDK，Git，python的同时，还需要安装移动自动化环境，例如为Android机器安装adb环境，为iOS机器安装WebDriverAgent工具。5.2 Mobile nodes: Mobile cloud real machines and mobile simulators are provided. The mobile execution machines are connected to one or more computer nodes via USB. These computer nodes need to install JDK, Git, and Python as well as the mobile automation environment, such as installing the adb environment for Android machines and the WebDriverAgent tool for iOS machines.
请参照图2，本实施例还提供了一种基于OCR识别的游戏自动化方法，包括如下步骤：Referring to FIG. 2 , this embodiment further provides a game automation method based on OCR recognition, comprising the following steps:
S1、设计游戏自动化脚本，其中游戏界面识别环节借助OCR模块的识别返回来制定自动化执行策略；S1. Design game automation scripts, in which the game interface recognition link uses the recognition return of the OCR module to formulate an automation execution strategy;
S2、本地自动化脚本调试通过后，将代码上传到GitLab，通过配置管理模块新增该游戏自动化场景配置，通过任务配置模块设置脚本需要接收的参数和执行的机器节点；S2. After the local automation script is debugged, upload the code to GitLab, add the game automation scene configuration through the configuration management module, and set the parameters that the script needs to receive and the machine node to execute through the task configuration module;
S3、通过自动化模块提供的接口发起游戏自动化任务，根据接口返回的自动化任务唯一id可以查询任务状态，自动化模块在接受到请求后会对参数进行解析和校验，如果校验通过则通知调度层查询可执行机器并把任务分配空闲的可执行机器上，如果可执行机器不足则根据图3执行报警策略；S3, initiate the game automation task through the interface provided by the automation module, and query the task status according to the unique ID of the automation task returned by the interface. After receiving the request, the automation module will parse and verify the parameters. If the verification passes, the scheduling layer will be notified to query the executable machine and assign the task to the idle executable machine. If the executable machine is insufficient, the alarm strategy will be executed according to Figure 3;
S4、自动化任务在执行机器上启动，调度层会清理工作空间，然后把S1中的自动化脚本下载到执行机器上并启动，游戏脚本驱动对应的游戏程序，在执行过程中进行截图上传，然后请求OCR接口，根据OCR接口返回的信息来判断当前游戏状态，以此来执行脚本中下一步的策略，在整个游戏自动化的过程中重复以上步骤；S4, the automation task is started on the execution machine. The scheduling layer will clean up the workspace, then download the automation script in S1 to the execution machine and start it. The game script drives the corresponding game program, takes screenshots and uploads them during the execution process, and then requests the OCR interface. The current game status is determined based on the information returned by the OCR interface, so as to execute the next strategy in the script. The above steps are repeated throughout the entire game automation process.
S5、游戏自动化正常结束或者出现异常后，自动化脚本会把执行的结果，状态回调给自动化模块，自动化模块将参数格式化后存储入库，如果出现异常，则根据配置管理中预设的报警配置，进行告警推送，引导开发人员定位问题；S5. After the game automation ends normally or an exception occurs, the automation script will call back the execution result and status to the automation module. The automation module will format the parameters and store them in the library. If an exception occurs, an alarm will be pushed according to the alarm configuration preset in the configuration management to guide the developer to locate the problem.
S6、开发人员可以根据任务id、任务类型、任务名称等字段通过自动化模块查询任务日志、任务状态，根据游戏场景、时间拉取数据统计报表。S6. Developers can query task logs and task status through the automation module based on fields such as task ID, task type, and task name, and pull data statistical reports based on game scenarios and time.
以上所述仅为本发明的优选实施例而已，并不用于限制本发明，对于本领域的技术人员来说，本发明可以有各种改变和变化。凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410876120.1ACN118557959A (en) | 2024-07-02 | 2024-07-02 | Game automation system and method based on OCR (optical character recognition) |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410876120.1ACN118557959A (en) | 2024-07-02 | 2024-07-02 | Game automation system and method based on OCR (optical character recognition) |
| Publication Number | Publication Date |
|---|---|
| CN118557959Atrue CN118557959A (en) | 2024-08-30 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410876120.1APendingCN118557959A (en) | 2024-07-02 | 2024-07-02 | Game automation system and method based on OCR (optical character recognition) |
| Country | Link |
|---|---|
| CN (1) | CN118557959A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118890347A (en)* | 2024-09-24 | 2024-11-01 | 深圳依时货拉拉科技有限公司 | Multi-machine synchronous control method, device, equipment and medium based on cloud real machine |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130263090A1 (en)* | 2012-03-30 | 2013-10-03 | Sony Online Entertainment Llc | System and method for automated testing |
| KR20190019009A (en)* | 2018-07-19 | 2019-02-26 | 넷마블 주식회사 | Method and apparatus for automating game test |
| US20200192788A1 (en)* | 2017-08-16 | 2020-06-18 | Netmarble Corporation | Game Test Automation Device and Method |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130263090A1 (en)* | 2012-03-30 | 2013-10-03 | Sony Online Entertainment Llc | System and method for automated testing |
| CN103365773A (en)* | 2012-03-30 | 2013-10-23 | 索尼在线娱乐有限责任公司 | System and method for automated testing |
| US20200192788A1 (en)* | 2017-08-16 | 2020-06-18 | Netmarble Corporation | Game Test Automation Device and Method |
| KR20190019009A (en)* | 2018-07-19 | 2019-02-26 | 넷마블 주식회사 | Method and apparatus for automating game test |
| Title |
|---|
| 知乎用户5QGYJK: "《Python开发游戏自动化脚本（六）数字识别》", pages 1 - 3, Retrieved from the Internet <URL:https://zhuanlan.zhihu.com/p/365202405>* |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118890347A (en)* | 2024-09-24 | 2024-11-01 | 深圳依时货拉拉科技有限公司 | Multi-machine synchronous control method, device, equipment and medium based on cloud real machine |
| Publication | Publication Date | Title |
|---|---|---|
| EP3842948B1 (en) | Method and apparatus for testing edge computing, device, and readable storage medium | |
| EP3859533A2 (en) | Method and apparatus for testing map service, electronic device, storage medium and computer program product | |
| CN108108297B (en) | Method and device for automatic testing | |
| US9069903B2 (en) | Multi-platform test automation enhancement | |
| JP5684946B2 (en) | Method and system for supporting analysis of root cause of event | |
| CN112306880A (en) | Test method, apparatus, electronic device, and computer-readable storage medium | |
| WO2017165071A1 (en) | Visual regression analysis | |
| CN111462811A (en) | Automatic testing method and device, storage medium and electronic equipment | |
| CN107608902A (en) | Routine interface method of testing and device | |
| CN118557959A (en) | Game automation system and method based on OCR (optical character recognition) | |
| CN114064510A (en) | Function testing method and device, electronic equipment and storage medium | |
| CN107807869A (en) | A kind of test system and method for testing | |
| US20220147437A1 (en) | Automated testing of mobile devices using visual analysis | |
| CN111654495B (en) | Method, apparatus, device and storage medium for determining traffic generation source | |
| CN112612469A (en) | Interface element processing method and device and electronic equipment | |
| CN105955838A (en) | System halt reason check method and device | |
| US9158641B2 (en) | Cloud auto-test system, method and non-transitory computer readable storage medium of the same | |
| CN119341899A (en) | Fault detection method, device, storage medium and electronic device | |
| CN110489256B (en) | Downtime positioning and repairing method and system | |
| CN118503100A (en) | Cross-platform test system, method and program product based on image recognition technology | |
| CN117914737A (en) | Mirror image resource testing method and device for network target range | |
| KR102160951B1 (en) | Mobile app auto-test device | |
| CN114564326B (en) | A method and system for performing abnormal scanning on applications of a kubernetes cluster | |
| CN113656291B (en) | Dynamic calling method for software script engine | |
| CN111158989B (en) | Method, system, equipment and medium for automatically testing load balancing cluster performance |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |