技术领域Technical Field
本发明涉及供操作者通过网络对远程地点的机器人进行远程操作的系统中的数据生成装置、数据生成方法、数据生成程序以及远程操作系统。The present invention relates to a data generating device, a data generating method, a data generating program and a remote operating system in a system in which an operator remotely operates a robot at a remote location via a network.
背景技术Background technique
以往，公知有供操作者通过网络来远程操作远程地点的机器人的系统。在这种系统中，存在在操作者操作的操作装置与机器人之间产生通信延迟的担忧。例如在专利文献1中公开有能够掌握从装置与主装置之间的通信延迟的远程操作系统。In the past, there is a known system for an operator to remotely operate a robot at a remote location via a network. In such a system, there is a concern that a communication delay may occur between an operating device operated by the operator and the robot. For example, Patent Document 1 discloses a remote operating system that can grasp the communication delay between a slave device and a master device.
在专利文献1中公开的远程操作系统中，具备主装置和从装置。从装置具备依据从主装置发送来的与操作者的操作对应的操作信息进行动作的从动机器人。从动机器人具有拍摄自身的作业环境的拍摄设备，从装置依次拍摄作业环境，并将拍摄到的实际图像向主装置发送。主装置具备基于向从动机器人发送的操作信息来进行动作模拟的模拟器，并通过将从从装置发送来的实际图像与利用动作模拟获得的模拟图像合成来生成显示图像。显示图像上的实际图像与模拟图像的合成的比例与主装置和从装置之间的通信延迟相应地改变。具体而言，在通信延迟较大的情况下，增大模拟图像的比例来生成显示图像。实际图像是展示从动机器人的状况并且展示作业环境的背景等的图像，相对于此，模拟图像是仅显示有从动机器人的状况的图像。因此，操作者能够根据显示图像中的例如作业环境的背景是否以浓重颜色被显示等来容易地掌握通信延迟产生到何种程度。In the remote control system disclosed in Patent Document 1, a master device and a slave device are provided. The slave device is provided with a slave robot that operates according to operation information corresponding to the operation of the operator sent from the master device. The slave robot has a photographing device for photographing its own working environment, and the slave device photographs the working environment in sequence and sends the photographed actual image to the master device. The master device is provided with a simulator for performing action simulation based on the operation information sent to the slave robot, and generates a display image by synthesizing the actual image sent from the slave device with the simulated image obtained by the action simulation. The ratio of the synthesis of the actual image and the simulated image on the display image changes in accordance with the communication delay between the master device and the slave device. Specifically, in the case of a large communication delay, the ratio of the simulated image is increased to generate the display image. The actual image is an image that shows the status of the slave robot and the background of the working environment, etc., while the simulated image is an image that only shows the status of the slave robot. Therefore, the operator can easily grasp the extent of the communication delay by, for example, whether the background of the working environment in the display image is displayed in a dark color.
专利文献1：日本特开2015-47666号公报Patent Document 1: Japanese Patent Application Publication No. 2015-47666
然而，在专利文献1的系统中，操作者需要根据实际图像与模拟图像的合成比例来判断通信延迟的程度，另外，在判断为通信延迟较大的情况下，需要适当采取减少对主装置施加的操作量等对策。因此，在上述的系统中，操作者除机器人的操作以外还需要进行伴随通信延迟的状况判断，存在操作者无法集中在机器人的操作的担忧。However, in the system of Patent Document 1, the operator needs to judge the degree of communication delay based on the synthesis ratio of the actual image and the simulated image, and in addition, if it is judged that the communication delay is large, it is necessary to appropriately take measures such as reducing the amount of operation applied to the main device. Therefore, in the above-mentioned system, the operator needs to judge the situation accompanied by communication delay in addition to operating the robot, and there is a concern that the operator cannot concentrate on operating the robot.
发明内容Summary of the invention
因此本发明的目的在于提供能够抑制操作者操作的操作终端与机器人之间的通信延迟对操作者的操作施加的影响的数据生成装置、数据生成方法、数据生成程序以及远程操作系统。Therefore, an object of the present invention is to provide a data generating device, a data generating method, a data generating program, and a remote operating system that can suppress the influence of a communication delay between an operation terminal operated by an operator and a robot on the operator's operation.
为了解决上述的课题，本发明的一方式所涉及的数据生成装置用于在具备具有接受操作者的操作的操作器和上述操作者能够视觉辨认的显示器的操作终端、和设置于实际作业空间内且经由能够进行数据通信的网络与上述操作终端连接的实际机器人的远程操作系统中，生成在显示于上述显示器的图像的生成中使用的至少一部分的数据，上述数据生成装置的特征在于，在上述显示器中，作为动态图像(time-varying image：时变图像)而显示将上述实际作业空间模型化而得的作业空间模型，上述作业空间模型包括将上述实际机器人模型化而得的机器人模型、和将位于上述实际机器人的周边的实际周边物模型化而得的周边物模型，上述机器人模型被制成为根据上述操作者对上述操作器的操作而进行动作，上述数据生成装置具备：状态信息取得部，其取得表示上述实际周边物的状态的状态信息；和预测部，其基于上述状态信息，对自当前时刻开始经过规定时间后的上述实际周边物的状态进行预测，并且将预测出的结果生成为在显示于上述显示器的上述周边物模型的制作中使用的周边物模型数据。In order to solve the above-mentioned problems, a data generating device involved in one embodiment of the present invention is used to generate at least a part of data used in generating an image displayed on the above-mentioned display in a remote operation system having an operating device that accepts an operator's operation and a display that can be visually recognized by the above-mentioned operator, and an actual robot set in an actual work space and connected to the above-mentioned operating terminal via a network capable of data communication. The above-mentioned data generating device is characterized in that, in the above-mentioned display, a work space model obtained by modeling the above-mentioned actual work space is displayed as a dynamic image (time-varying image), and the above-mentioned work space model includes a robot model obtained by modeling the above-mentioned actual robot, and a surrounding object model obtained by modeling actual surrounding objects located around the above-mentioned actual robot, and the above-mentioned robot model is made to move according to the operation of the above-mentioned operating device by the above-mentioned operator. The above-mentioned data generating device includes: a state information acquisition unit, which acquires state information representing the state of the above-mentioned actual surrounding objects; and a prediction unit, which predicts the state of the above-mentioned actual surrounding objects after a specified time has passed since the current time based on the above-mentioned state information, and generates the predicted result as surrounding object model data used in the production of the above-mentioned surrounding object model displayed on the above-mentioned display.
在操作终端与实际机器人之间存在通信延迟的情况下，由于从操作者操作操作器到实际机器人进行与该操作对应的动作之间的时滞，同一时刻下的机器人模型的动作与实际机器人的动作之间可能产生偏差。根据上述结构，通过预测部来预测自当前时刻开始经过规定时间的实际周边物的状态，并将该预测结果生成为在显示于显示器的周边物模型的制作中使用的周边物模型数据。因此，能够使周边物模型的状态和实际周边物的状态之间产生与机器人模型和实际机器人之间相同的时间偏差。由此，能够使机器人模型与周边物模型的时间轴一致，因此能够抑制通信延迟对操作者的操作施加的影响。In the case where there is a communication delay between the operation terminal and the actual robot, due to the time lag between the operator operating the operator and the actual robot performing the action corresponding to the operation, a deviation may occur between the action of the robot model and the action of the actual robot at the same time. According to the above structure, the state of the actual surrounding objects after a specified time from the current time is predicted by the prediction unit, and the prediction result is generated as the surrounding object model data used in the creation of the surrounding object model displayed on the display. Therefore, the same time deviation as that between the robot model and the actual robot can be generated between the state of the surrounding object model and the state of the actual surrounding objects. As a result, the time axis of the robot model and the surrounding object model can be consistent, so the influence of the communication delay on the operator's operation can be suppressed.
也可以构成为，上述实际周边物包括作为上述实际机器人的作业对象的工件、搬运上述工件的搬运装置、以及使上述实际机器人移动的移动装置中的至少一个。The actual surrounding object may include at least one of a workpiece to be worked on by the actual robot, a transport device for transporting the workpiece, and a moving device for moving the actual robot.
也可以构成为，上述状态信息包含设置于上述作业空间的拍摄装置拍摄上述实际周边物从而生成的拍摄信息。The state information may include photographing information generated by photographing the actual surrounding objects using a photographing device provided in the work space.
也可以构成为，上述状态信息包含对作为上述实际周边物的周边设备设定的设定信息。The state information may include setting information set for a peripheral device serving as the actual surrounding object.
也可以构成为，上述的数据生成装置还具备偏差检测部，其对在规定的时刻显示于上述显示器的上述作业空间模型的状况与自上述规定的时刻开始经过上述规定时间后的上述实际作业空间的状况之间的偏差的程度进行检测。例如，在通过偏差检测部检测的偏差大于规定的值的情况下，能够进行使实际机器人的作业停止、或者修正显示于显示器的模型等对策。The data generating device may also be configured to further include a deviation detection unit that detects the degree of deviation between the state of the workspace model displayed on the display at a specified time and the state of the actual workspace after the specified time has passed since the specified time. For example, when the deviation detected by the deviation detection unit is greater than a specified value, countermeasures such as stopping the operation of the actual robot or correcting the model displayed on the display can be taken.
也可以构成为，上述的数据生成装置还具备模型修正部，其在由上述偏差检测部检测出的偏差超出预先设定的范围的情况下，以消除上述偏差的方式修正上述作业空间模型。The data generating device may further include a model correction unit configured to correct the workspace model so as to eliminate the deviation when the deviation detected by the deviation detecting unit exceeds a preset range.
也可以构成为，上述操作终端是包括作为上述操作器的控制器的游戏装置。The operation terminal may be a game device including a controller as the operation unit.
也可以构成为，上述操作终端是个人信息终端(PDA(Personal Data Assistant：个人数据助手))、智能电话、个人计算机、平板电脑、以及机器人专用远程操作器中的至少一个。The operation terminal may be at least one of a personal information terminal (PDA (Personal Data Assistant)), a smart phone, a personal computer, a tablet computer, and a robot-dedicated remote operator.
另外，本发明的一方式所涉及的数据生成方法用于在具备具有接受操作者的操作的操作器和上述操作者能够视觉辨认的显示器的操作终端、和设置于作业空间内，且经由能够数据通信的网络与上述操作终端连接的实际机器人的远程操作系统中，生成在显示于上述显示器的图像的生成中使用的至少一部分的数据，上述数据生成方法的特征在于，在上述显示器中，作为动态图像而显示将上述实际作业空间模型化而得的作业空间模型，上述作业空间模型包括将上述实际机器人模型化而得的机器人模型、和将位于上述实际机器人的周边的周边物模型化而得的周边物模型，上述机器人模型被制成为根据上述操作者对上述操作器的操作而进行动作，上述数据生成方法包含：状态信息取得步骤，其取得表示位于上述实际机器人的周边的周边物的状态的状态信息；和预测步骤，其基于上述状态信息，对当前时刻之后的上述周边物的状态进行预测，并且将预测出的结果生成为在显示于上述显示器的上述周边物模型的制作中使用的周边物模型数据。In addition, a data generation method involved in one embodiment of the present invention is used to generate at least a part of data used in generating an image displayed on the above-mentioned display in a remote operation system of an operation terminal having an operator that accepts an operator's operation and a display that can be visually recognized by the above-mentioned operator, and an actual robot arranged in a work space and connected to the above-mentioned operation terminal via a network capable of data communication. The above-mentioned data generation method is characterized in that, in the above-mentioned display, a work space model obtained by modeling the above-mentioned actual work space is displayed as a dynamic image, and the above-mentioned work space model includes a robot model obtained by modeling the above-mentioned actual robot, and a surrounding object model obtained by modeling surrounding objects located around the above-mentioned actual robot, and the above-mentioned robot model is made to act according to the above-mentioned operator's operation on the above-mentioned operator. The above-mentioned data generation method includes: a state information acquisition step, which acquires state information representing the state of surrounding objects located around the above-mentioned actual robot; and a prediction step, which predicts the state of the above-mentioned surrounding objects after the current moment based on the above-mentioned state information, and generates the predicted result as surrounding object model data used in the production of the above-mentioned surrounding object model displayed on the above-mentioned display.
另外，本发明的一方式所涉及的数据生成程序在具备具有接受操作者的操作的操作器和上述操作者能够视觉辨认的显示器的操作终端、和设置于作业空间内，且经由能够进行数据通信的网络与上述操作终端连接的实际机器人的远程操作系统中，被计算机执行，由此生成在显示于上述显示器的图像的生成中使用的至少一部分的数据，上述数据生成程序的特征在于，在上述显示器中，作为动态图像而显示将上述实际作业空间模型化而得的作业空间模型，上述作业空间模型包括将上述实际机器人模型化而得的机器人模型、和将位于上述实际机器人的周边的周边物模型化而得的周边物模型，上述机器人模型通过上述操作者对上述操作器的操作来生成，上述数据生成程序使上述计算机执行：状态信息取得步骤，其取得表示位于上述实际机器人的周边的周边物的状态的状态信息的状态信息；和预测步骤，其基于上述状态信息，对当前时刻之后的上述周边物的状态进行预测，并且将预测出的结果生成为在显示于上述显示器的上述周边物模型的制作中使用的周边物模型数据。In addition, a data generation program involved in one embodiment of the present invention is executed by a computer in a remote operation system of an operating terminal having an operating device that accepts operations of an operator and a display that can be visually recognized by the above-mentioned operator, and an actual robot arranged in a working space and connected to the above-mentioned operating terminal via a network capable of data communication, thereby generating at least a part of data used in generating an image displayed on the above-mentioned display. The above-mentioned data generation program is characterized in that, in the above-mentioned display, a working space model obtained by modeling the above-mentioned actual working space is displayed as a dynamic image, and the above-mentioned working space model includes a robot model obtained by modeling the above-mentioned actual robot, and a surrounding object model obtained by modeling surrounding objects located around the above-mentioned actual robot, and the above-mentioned robot model is generated by the above-mentioned operator operating the above-mentioned operating device. The above-mentioned data generation program causes the above-mentioned computer to execute: a state information acquisition step, which acquires state information representing the state of surrounding objects located around the above-mentioned actual robot; and a prediction step, which predicts the state of the above-mentioned surrounding objects after the current moment based on the above-mentioned state information, and generates the predicted result as surrounding object model data used in the production of the above-mentioned surrounding object model displayed on the above-mentioned display.
此外，上述数据生成程序存储于存储装置。上述存储装置是内置或者外装于计算机的能够读写或者能够读取的装置，例如能够使用硬盘、闪存、光盘等。存储于上述存储装置的程序可以在供上述存储装置直接连接的计算机中执行，也可以在经由网络(例如，因特网)与上述存储装置连接的计算机中下载来执行。In addition, the data generation program is stored in a storage device. The storage device is a device that is built into or external to a computer and can read, write or read, such as a hard disk, a flash memory, an optical disk, etc. The program stored in the storage device can be executed in a computer directly connected to the storage device, or can be downloaded and executed in a computer connected to the storage device via a network (e.g., the Internet).
另外，本发明的一方式所涉及的远程操作系统具备：操作终端，其具有接受操作者的操作的操作器和上述操作者能够视觉辨认的显示器；和实际机器人，其设置于实际作业空间内，且经由能够进行数据通信的网络与上述操作终端连接，上述远程操作系统的特征在于，在上述显示器中，作为动态图像而显示将上述实际作业空间模型化而得的作业空间模型，上述作业空间模型包括将上述实际机器人模型化而得的机器人模型、和将位于上述实际机器人的周边的实际周边物模型化而得的周边物模型，上述机器人模型被制成为根据上述操作者对上述操作器的操作而进行动作，上述远程操作系统具备数据生成装置，该数据生成装置具备：状态信息取得部，其取得表示上述实际周边物的状态的状态信息；和预测部，其基于上述状态信息，对自当前时刻开始经过规定时间后的上述实际周边物的状态进行预测，并且将预测出的结果生成为在显示于上述显示器的上述周边物模型的制作中使用的周边物模型数据。In addition, a remote operation system involved in one embodiment of the present invention comprises: an operation terminal, which has an operator for accepting operations of an operator and a display that can be visually recognized by the above-mentioned operator; and an actual robot, which is arranged in an actual work space and is connected to the above-mentioned operation terminal via a network capable of data communication. The above-mentioned remote operation system is characterized in that, in the above-mentioned display, a work space model obtained by modeling the above-mentioned actual work space is displayed as a dynamic image, and the above-mentioned work space model includes a robot model obtained by modeling the above-mentioned actual robot, and a surrounding object model obtained by modeling actual surrounding objects located around the above-mentioned actual robot, and the above-mentioned robot model is made to act according to the operation of the above-mentioned operator by the above-mentioned operator. The above-mentioned remote operation system has a data generating device, which comprises: a state information acquisition unit, which acquires state information representing the state of the above-mentioned actual surrounding objects; and a prediction unit, which predicts the state of the above-mentioned actual surrounding objects after a specified time has passed since the current time based on the above-mentioned state information, and generates the predicted result as surrounding object model data used in the production of the above-mentioned surrounding object model displayed on the above-mentioned display.
根据本发明，能够抑制操作者操作的操作终端与机器人之间的通信延迟对操作者的操作施加的影响。According to the present invention, it is possible to suppress the influence of a communication delay between an operation terminal operated by an operator and a robot on the operation of the operator.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
图1是表示本发明的一实施方式所涉及的远程操作系统的整体结构的框图。FIG. 1 is a block diagram showing the overall configuration of a remote operation system according to an embodiment of the present invention.
图2是表示图1的游戏装置、中转装置以及中介装置的硬件构成的一个例子的框图。FIG. 2 is a block diagram showing an example of the hardware configuration of the game device, the transfer device, and the mediation device of FIG. 1 .
图3是示意性地表示图1的机器人系统的一个例子的图。FIG. 3 is a diagram schematically showing an example of the robot system of FIG. 1 .
图4是表示图1的游戏装置的控制部的功能性结构的框图。FIG. 4 is a block diagram showing a functional structure of a control unit of the game device of FIG. 1 .
图5是表示图1的中转装置的控制部的功能性结构的框图。FIG. 5 is a block diagram showing a functional structure of a control unit of the relay device of FIG. 1 .
图6是表示机器人的作业开始前的游戏装置和中转装置各自中的处理流程的图。FIG. 6 is a diagram showing the processing flow in each of the game device and the relay device before the robot's operation starts.
图7是表示机器人的作业开始后的游戏装置和中转装置各自中的处理流程的图。FIG. 7 is a diagram showing the processing flow in each of the game device and the relay device after the robot's operation starts.
图8是用于对分别在游戏装置和中转装置中伴随着时间的经过而执行的处理的一个例子进行说明的图。FIG. 8 is a diagram for explaining an example of processing executed in each of the game device and the transfer device as time passes.
具体实施方式Detailed ways
以下，针对本发明的一实施方式，参照附图进行说明。Hereinafter, one embodiment of the present invention will be described with reference to the drawings.
(系统的概要)(System Overview)
首先参照图1对本实施方式所涉及的远程操作系统1的概要进行说明。图1是表示远程操作系统1的整体结构的框图。本实施方式的远程操作系统1将公知的各种游戏装置2(操作终端)与在和操作者所居住的场所不同的远程地点的作业现场(作业空间。以下，也称为“实际作业空间”。)设置的机器人51经由通信网络4连接。而且，使操作者使用游戏装置2远程操作机器人51，而使该机器人51进行规定的作业。First, the outline of the remote operation system 1 involved in the present embodiment is described with reference to FIG1. FIG1 is a block diagram showing the overall structure of the remote operation system 1. The remote operation system 1 of the present embodiment connects various well-known game devices 2 (operation terminals) to a robot 51 set up at a work site (work space. Hereinafter, also referred to as "actual work space") at a remote location different from the place where the operator lives via a communication network 4. Then, the operator uses the game device 2 to remotely operate the robot 51, and the robot 51 performs a prescribed operation.
远程操作系统1包含多个游戏装置2、一个中介装置6、以及多个机器人系统5，它们例如能够经由因特网等通信网络4相互通信。游戏装置2例如是放置于操作者的自家住宅等的不动型的游戏装置或者操作者携带的便携式游戏装置。The remote operation system 1 includes a plurality of game devices 2, an intermediary device 6, and a plurality of robot systems 5, which can communicate with each other via a communication network 4 such as the Internet. The game device 2 is, for example, a stationary game device placed in the operator's home or a portable game device carried by the operator.
机器人系统5具备被操作者远程操作的对象亦即机器人51、设置于机器人51的周边的1个以上的周边设备52、以及中转装置53。机器人51、周边设备52以及中转装置53均设置于远离对游戏装置2进行操作的操作者所居住的场所的远程地点的作业现场。在作业现场存在1个以上的机器人系统5。The robot system 5 includes a robot 51 which is an object to be remotely operated by an operator, one or more peripheral devices 52 provided around the robot 51, and a transfer device 53. The robot 51, the peripheral devices 52, and the transfer device 53 are all provided at a work site at a remote location away from the place where the operator who operates the game device 2 lives. There are one or more robot systems 5 at the work site.
此外，远程操作系统1所具有的多个机器人系统5的一部分或者全部可以相互设置于相同的作业现场，也可以设置于不同的作业现场。另外，设置于相同的作业现场的多个机器人系统5也可以具有相互共享的周边设备52。此外，多个机器人系统5能够包含相互种类相同或者不同的多个周边设备52，但在图1中，为了将图简化，针对1个机器人系统5仅示出一个表示周边设备52的模块。In addition, part or all of the multiple robot systems 5 of the remote operation system 1 may be installed at the same work site or at different work sites. In addition, the multiple robot systems 5 installed at the same work site may also have shared peripheral devices 52. In addition, the multiple robot systems 5 can include multiple peripheral devices 52 of the same or different types, but in FIG1, in order to simplify the figure, only one module representing the peripheral device 52 is shown for one robot system 5.
中转装置53与包含该中转装置53的相同的机器人系统5的机器人51和周边设备52分别以能够通信的方式连接。中转装置53将经由通信网络4从游戏装置2或者中介装置6发送的信息向机器人51或者周边设备52发送、将机器人51或者周边设备52的信息向游戏装置2或者中介装置6发送。The transfer device 53 is connected to the robot 51 and the peripheral device 52 of the same robot system 5 including the transfer device 53 in a communicative manner. The transfer device 53 sends information sent from the game device 2 or the intermediary device 6 to the robot 51 or the peripheral device 52 via the communication network 4, and sends information of the robot 51 or the peripheral device 52 to the game device 2 or the intermediary device 6.
中介装置6针对1个操作者(1个游戏装置2)分配1个机器人系统5。更具体而言，操作者从游戏装置2访问中介装置6而预先进行用户登录，通过用户登录来给予操作者用户ID。若操作者在游戏装置2中输入自己的用户ID并对中介装置6发送操作请求，则接收了操作请求的中介装置6将游戏装置2与其中一个机器人系统5建立对应关系，并且使游戏装置2与建立了对应关系的机器人系统5的中转装置53经由通信网络4相互通信连接。The intermediary device 6 allocates one robot system 5 to one operator (one game device 2). More specifically, the operator accesses the intermediary device 6 from the game device 2 and performs user login in advance, and the operator is given a user ID through user login. If the operator enters his or her user ID in the game device 2 and sends an operation request to the intermediary device 6, the intermediary device 6 that receives the operation request establishes a correspondence between the game device 2 and one of the robot systems 5, and enables the game device 2 and the transfer device 53 of the robot system 5 with which the correspondence is established to communicate with each other via the communication network 4.
例如中介装置6若从游戏装置2接受到操作请求，则将表示作业内容等的作业列表信息向该游戏装置2发送。操作请求中包含操作者输入的希望条件信息。希望条件信息中包含机器人的种类、机器人的作业内容、对象工件、作业量、作业时间的一部分或者全部。中介装置6若接收到操作请求，则基于该操作请求中包含的希望条件信息，筛选符合操作者所希望的条件的内容，并向游戏装置2发送筛选出的作业列表。若操作者从显示于游戏装置2的显示装置25的作业列表之中指定一个希望的作业，则从游戏装置2向中介装置6发送与操作者的指定对应的指定信息。中介装置6将该游戏装置2同与指定信息对应的机器人系统5的中转装置53连接。For example, if the intermediary device 6 receives an operation request from the game device 2, it sends the job list information indicating the job content, etc. to the game device 2. The operation request includes the desired condition information input by the operator. The desired condition information includes part or all of the type of robot, the robot's job content, the target workpiece, the amount of work, and the job time. If the intermediary device 6 receives an operation request, it filters the content that meets the operator's desired conditions based on the desired condition information included in the operation request, and sends the filtered job list to the game device 2. If the operator specifies a desired job from the job list displayed on the display device 25 of the game device 2, the game device 2 sends the designated information corresponding to the operator's designation to the intermediary device 6. The intermediary device 6 connects the game device 2 to the transfer device 53 of the robot system 5 corresponding to the designated information.
这样，在远程操作系统1中，操作者能够使用游戏装置2远程操作物理上较远地远离的各种作业现场的机器人51。例如，通过远程操作系统1，操作者能够边居于自家住宅，边操作位于地球的背面侧的作业现场的机器人51。在相互连接的游戏装置2与机器人系统5(更具体而言为机器人51)之间，可能产生因游戏装置2和机器人系统5各自的通信环境、游戏装置2和机器人系统5相互的物理上的距离等引起的通信延迟。在本实施方式中，像后述那样实现抑制在相互连接的游戏装置2与机器人51之间产生的通信延迟对操作者的操作施加的影响的远程操作系统1。Thus, in the remote operation system 1, the operator can remotely operate the robot 51 at various work sites that are physically far away using the game device 2. For example, through the remote operation system 1, the operator can operate the robot 51 at a work site located on the back side of the earth while staying at his own home. Between the game device 2 and the robot system 5 (more specifically, the robot 51) that are connected to each other, communication delay may occur due to the communication environment of each of the game device 2 and the robot system 5, the physical distance between the game device 2 and the robot system 5, etc. In this embodiment, as described later, a remote operation system 1 is implemented that suppresses the influence of the communication delay generated between the game device 2 and the robot 51 that is connected to each other on the operator's operation.
(硬件构成)(Hardware Configuration)
在图2中示出游戏装置2、中转装置53以及中介装置6的硬件构成的一个例子。此外，在图2中，仅示出多个游戏装置2中的1个游戏装置2，且仅示出多个机器人系统5中的1个机器人系统5。游戏装置2具备游戏装置主体2a、与其连接的显示装置25(显示器)、扬声器27、以及控制器28(操作器)。Fig. 2 shows an example of the hardware configuration of the game device 2, the transfer device 53, and the intermediary device 6. In addition, Fig. 2 shows only one game device 2 among the multiple game devices 2, and only one robot system 5 among the multiple robot systems 5. The game device 2 includes a game device body 2a, a display device 25 (display) connected thereto, a speaker 27, and a controller 28 (operator).
如图2所示，游戏装置主体2a在总线20上具备控制部21、通信部22、以及硬盘或者存储卡等存储部23。控制部21基于控制器28的操作，生成经由通信网络4向机器人51发送的操作信息。此外，机器人51基于该操作信息来进行动作。另外，控制部21基于控制器28的操作，生成显示于显示装置25的图像。这样的控制部21具备CPU210、ROM(闪存)211、RAM212、图像处理器213、声音处理器214、以及操作部215。As shown in FIG2 , the game device body 2a includes a control unit 21, a communication unit 22, and a storage unit 23 such as a hard disk or a memory card on a bus 20. The control unit 21 generates operation information to be sent to the robot 51 via the communication network 4 based on the operation of the controller 28. In addition, the robot 51 performs actions based on the operation information. In addition, the control unit 21 generates an image displayed on the display device 25 based on the operation of the controller 28. Such a control unit 21 includes a CPU 210, a ROM (flash memory) 211, a RAM 212, an image processor 213, a sound processor 214, and an operation unit 215.
CPU210控制游戏装置2的各部的动作。在ROM211中存储有游戏装置2的基本程序等。在存储部23中存储有用于通过远程操作来使机器人51进行动作的远程操作程序、用于执行各种游戏的游戏程序等。在RAM212中设定CPU210执行游戏程序时使用的工作区域。此外，在本实施方式中，存储部23中的上述远程操作程序的存储是必须的，但上述游戏程序的存储并非是必须的。此外，在以下的说明中，将存储了各种程序、数据的控制部21的ROM211、RAM212以及存储部23等统称为游戏装置2的存储装置。The CPU 210 controls the actions of each part of the game device 2. The basic program of the game device 2 is stored in the ROM 211. The remote operation program for causing the robot 51 to move by remote operation, the game program for executing various games, etc. are stored in the storage unit 23. The working area used by the CPU 210 when executing the game program is set in the RAM 212. In addition, in the present embodiment, the storage of the above-mentioned remote operation program in the storage unit 23 is necessary, but the storage of the above-mentioned game program is not necessary. In addition, in the following description, the ROM 211, RAM 212 and storage unit 23 of the control unit 21 storing various programs and data are collectively referred to as the storage device of the game device 2.
图像处理器213具备能够生成游戏画面的GPU(图形处理单元)。对图像处理器213连接影像RAM(VRAM)24。在该VRAM24连接有显示装置25。The image processor 213 includes a GPU (Graphics Processing Unit) capable of generating a game screen. A video RAM (VRAM) 24 is connected to the image processor 213. A display device 25 is connected to the VRAM 24.
声音处理器214具备生成游戏声音的DSP(数字信号处理器)。声音处理器214将生成的游戏声音向包含D/A转换器的放大器26发送。放大器26将该声音信号放大并向扬声器27发送。The sound processor 214 includes a DSP (digital signal processor) for generating game sounds. The sound processor 214 sends the generated game sounds to the amplifier 26 including a D/A converter. The amplifier 26 amplifies the sound signal and sends it to the speaker 27.
在操作部215借助无线或者有线通信连接有控制器28。控制器28包含十字按钮、按键开关、操纵杆、鼠标、键盘以及触摸面板等。另外，操作部215对用户发出的经由控制器28的操作信号进行检测，并将该操作信号向CPU210发送。The operation unit 215 is connected to the controller 28 by wireless or wired communication. The controller 28 includes a cross button, a key switch, a joystick, a mouse, a keyboard, a touch panel, etc. In addition, the operation unit 215 detects an operation signal sent by the user via the controller 28 and sends the operation signal to the CPU 210.
通信部22是经由通信网络4与中介装置6和中转装置53通信的通信设备。The communication unit 22 is a communication device that communicates with the mediation device 6 and the relay device 53 via the communication network 4 .
中介装置6具备控制部61、通信部62以及存储部63。控制部61例如由具有处理器和存储器的运算器构成。具体而言，该运算器例如由微型控制器、MPU、FPGA(现场可编程门阵列)、PLC(可编程逻辑控制器)、计算机、个人计算机等构成。控制部61可以由进行集中控制的单独的运算器构成，也可以由进行分散控制的多个运算器构成。通信部62是经由通信网络4与游戏装置2以及中转装置53通信的通信设备。存储部63是能够进行读写的或者能够进行读取的存储装置，例如是硬盘、闪存、光盘等。控制部61控制中介装置6的各部的动作。在控制部61的存储器、存储部63中存储有将游戏装置2与机器人系统5建立对应关系的程序等控制中介装置6的动作的各种程序、数据。The intermediary device 6 has a control unit 61, a communication unit 62 and a storage unit 63. The control unit 61 is composed of, for example, an arithmetic unit having a processor and a memory. Specifically, the arithmetic unit is composed of, for example, a microcontroller, an MPU, an FPGA (field programmable gate array), a PLC (programmable logic controller), a computer, a personal computer, etc. The control unit 61 can be composed of a single arithmetic unit for centralized control, or can be composed of a plurality of arithmetic units for decentralized control. The communication unit 62 is a communication device that communicates with the game device 2 and the transfer device 53 via the communication network 4. The storage unit 63 is a storage device that can be read and written or can be read, such as a hard disk, a flash memory, an optical disk, etc. The control unit 61 controls the actions of each part of the intermediary device 6. Various programs and data for controlling the actions of the intermediary device 6, such as a program for establishing a correspondence between the game device 2 and the robot system 5, are stored in the memory of the control unit 61 and the storage unit 63.
中转装置53具备控制部55、通信部56以及存储部57。控制部55例如由具有处理器和存储器的运算器构成。具体而言，该运算器例如由微型控制器、MPU、FPGA(现场可编程门阵列)、PLC(可编程逻辑控制器)、计算机、个人计算机等构成。控制部55可以由进行集中控制的单独的运算器构成，也可以由进行分散控制的多个运算器构成。通信部56是经由通信网络4与游戏装置2、中介装置6、机器人51、以及周边设备52通信的通信设备。存储部57是能够进行读写的或者能够进行读取的存储装置，例如是硬盘、闪存、光盘等。控制部55控制中转装置53的动作。在控制部55的存储器、存储部57中存储有控制中转装置53的动作的各种程序、数据。The transfer device 53 has a control unit 55, a communication unit 56 and a storage unit 57. The control unit 55 is composed of, for example, an arithmetic unit having a processor and a memory. Specifically, the arithmetic unit is composed of, for example, a microcontroller, an MPU, an FPGA (field programmable gate array), a PLC (programmable logic controller), a computer, a personal computer, etc. The control unit 55 can be composed of a single arithmetic unit for centralized control, or can be composed of multiple arithmetic units for decentralized control. The communication unit 56 is a communication device that communicates with the game device 2, the intermediary device 6, the robot 51, and the peripheral device 52 via the communication network 4. The storage unit 57 is a storage device that can be read and written or can be read, such as a hard disk, a flash memory, an optical disk, etc. The control unit 55 controls the action of the transfer device 53. Various programs and data for controlling the action of the transfer device 53 are stored in the memory of the control unit 55 and the storage unit 57.
在图3中示意性地示出机器人系统5的一个例子。在该机器人系统5中，机器人51进行分拣利用发送带52a搬运的工件W的作业。在图3所示的机器人系统5中，机器人51是工业用机器人。机器人51包含被操作者远程操作的对象亦即机器人主体51a、和控制机器人主体51a的动作的机器人控制器51b。图3所示的机器人主体51a是在前端部安装有工具的垂直多关节型的机械臂，在本例中，作为工具，在垂直多关节型的机械臂的前端部安装有能够把持工件W的把持手部。机器人控制器51b具备处理器，利用处理器进行储存着的程序、从外部输入的各种信号的解读、运算处理等，且负责机器人主体51a的动作控制、从各种输出端口的信号输出等。另外，图3的机器人系统5作为周边设备52而具备用于搬运作业对象亦即工件的发送带52a、拍摄机器人51的作业状况的1个以上(在本例中为2个)的拍摄装置52b、以及检测工件W的位置的传感器52c等。FIG3 schematically shows an example of a robot system 5. In the robot system 5, a robot 51 performs the operation of sorting a workpiece W transported by a conveyor belt 52a. In the robot system 5 shown in FIG3, the robot 51 is an industrial robot. The robot 51 includes a robot body 51a, which is an object to be remotely operated by an operator, and a robot controller 51b that controls the movement of the robot body 51a. The robot body 51a shown in FIG3 is a vertical multi-joint type robot arm with a tool installed at the front end. In this example, as a tool, a gripping hand capable of gripping the workpiece W is installed at the front end of the vertical multi-joint type robot arm. The robot controller 51b has a processor, which uses the processor to perform stored programs, interpret various signals input from the outside, perform calculations, etc., and is responsible for the movement control of the robot body 51a, the signal output from various output ports, etc. In addition, the robot system 5 of Figure 3 has a conveyor belt 52a for transporting the work object, i.e., the workpiece, one or more (two in this example) imaging devices 52b for imaging the working status of the robot 51, and a sensor 52c for detecting the position of the workpiece W, etc. as peripheral equipment 52.
此外，图3所示的机器人系统5的结构是一个例子，机器人51、周边设备52的种类等为与机器人51的作业内容相应的结构。例如，机器人51的作业除可以是分拣作业以外，例如也可以是涂装作业、便当盛放作业、焊接作业等。另外，机器人51也可以并非垂直多关节型机器人，例如也可以是水平多关节型机器人、平行连杆型机器人、极坐标机器人、圆柱型机器人、直角坐标型机器人等工业用机器人。另外，作为搬运作业对象亦即工件W的周边设备52的搬运装置也可以是发送带以外的搬运装置。周边设备52也可以包含使机器人主体51a移动的移动装置。作为周边设备52的1个以上的传感器也可以替代检测工件W的位置的传感器是检测机器人51的位置、姿势的传感器等，或者除检测工件W的位置的传感器以外，是检测机器人51的位置、姿势的传感器等。在作为周边设备52的1个以上的传感器中包含对检查对象物的位置、朝向或者朝向进行检测的传感器。另外，机器人系统5也可以作为周边设备52而具备多个拍摄装置52b。如图3所示，拍摄装置52b可以安装于机器人主体51a，也可以设置于作业空间中固定的位置。In addition, the structure of the robot system 5 shown in FIG3 is an example, and the types of the robot 51 and the peripheral device 52 are structures corresponding to the work content of the robot 51. For example, the work of the robot 51 may be a painting work, a lunch box placing work, a welding work, etc. in addition to the sorting work. In addition, the robot 51 may not be a vertical multi-jointed robot, for example, it may be an industrial robot such as a horizontal multi-jointed robot, a parallel link robot, a polar coordinate robot, a cylindrical robot, a rectangular coordinate robot, etc. In addition, the conveying device of the peripheral device 52 that is the object of the conveying work, that is, the workpiece W, may also be a conveying device other than a conveying belt. The peripheral device 52 may also include a moving device that moves the robot body 51a. One or more sensors as the peripheral device 52 may also be a sensor that detects the position and posture of the robot 51 instead of the sensor that detects the position of the workpiece W, or a sensor that detects the position and posture of the robot 51 in addition to the sensor that detects the position of the workpiece W. The one or more sensors as the peripheral device 52 include a sensor that detects the position, orientation or orientation of the inspection object. In addition, the robot system 5 may include a plurality of imaging devices 52b as the peripheral equipment 52. As shown in Fig. 3 , the imaging device 52b may be attached to the robot body 51a, or may be installed at a fixed position in the work space.
(功能性结构)(Functional structure)
图4是表示游戏装置2的控制部21的功能性结构的框图。游戏装置2的控制部21作为功能性结构而具有通信控制部31、操作侧时间管理部32、状态信息取得部33、模拟部(预测部)34、图像显示部35、通信延迟测量部36、偏差检测部37、以及模型修正部38。对于这些功能部而言，通过其与存储于游戏装置2的存储装置的规定的程序的配合来功能性地实现。此外，在存储于游戏装置2的存储装置的规定的程序中包含本发明的“数据生成程序”。4 is a block diagram showing the functional structure of the control unit 21 of the game device 2. The control unit 21 of the game device 2 includes a communication control unit 31, an operation side time management unit 32, a state information acquisition unit 33, a simulation unit (prediction unit) 34, an image display unit 35, a communication delay measurement unit 36, a deviation detection unit 37, and a model correction unit 38 as a functional structure. These functional units are functionally realized by cooperating with a predetermined program stored in the storage device of the game device 2. In addition, the predetermined program stored in the storage device of the game device 2 includes the "data generation program" of the present invention.
通信控制部31控制通信部22，向中介装置6发送上述的操作请求、指定信息、或从中介装置6接收列表信息。另外，通信控制部31控制通信部22，从中介装置6接收用于供中介装置6同与游戏装置2建立了对应关系的机器人系统5进行通信连接的信息。另外，通信控制部31控制通信部22，将操作者操作控制器28从而生成的操作信息向对应的机器人系统5的中转装置53发送。The communication control unit 31 controls the communication unit 22 to send the above-mentioned operation request, designated information to the intermediary device 6, or receive list information from the intermediary device 6. In addition, the communication control unit 31 controls the communication unit 22 to receive information from the intermediary device 6 for the intermediary device 6 to communicate with the robot system 5 that has established a corresponding relationship with the game device 2. In addition, the communication control unit 31 controls the communication unit 22 to send the operation information generated by the operator operating the controller 28 to the transfer device 53 of the corresponding robot system 5.
操作侧时间管理部32管理游戏装置2侧的时间，以使得从操作者对控制器28进行操作开始至机器人51基于该操作进行动作为止的时间保持为恒定。The operation-side time management unit 32 manages the time on the game device 2 side so that the time from when the operator operates the controller 28 to when the robot 51 moves based on the operation is kept constant.
状态信息取得部33和模拟部34生成在显示于显示装置25的图像的生成中使用的至少一部分数据。具体而言，在显示装置25，将对机器人(以下，称为“实际机器人”)51实际存在的作业空间(以下，称为“实际作业空间”)进行了模型化而得的作业空间模型作为动态图像来显示。作业空间模型包含配置于假想的作业空间的机器人模型和周边物模型。机器人模型是将实际机器人51模型化而得的模型。周边物模型是将位于实际机器人51的周边的规定的周边物(以下，称为“实际周边物”)模型化而得的模型。实际周边物中包含位于机器人51的周边的周边设备52和工件W，周边物模型中包含与这些对应的周边设备模型和工件模型。在显示于显示装置25的图像的生成中使用机器人模型数据、周边物模型数据。The state information acquisition unit 33 and the simulation unit 34 generate at least a portion of data used in the generation of an image displayed on the display device 25. Specifically, on the display device 25, a workspace model obtained by modeling the workspace (hereinafter referred to as the "actual workspace") where the robot (hereinafter referred to as the "actual robot") 51 actually exists is displayed as a dynamic image. The workspace model includes a robot model and a surrounding object model configured in the virtual workspace. The robot model is a model obtained by modeling the actual robot 51. The surrounding object model is a model obtained by modeling the specified surrounding objects (hereinafter referred to as the "actual surrounding objects") located around the actual robot 51. The actual surrounding objects include peripheral equipment 52 and a workpiece W located around the robot 51, and the surrounding object model includes peripheral equipment models and workpiece models corresponding to these. The robot model data and the surrounding object model data are used in the generation of the image displayed on the display device 25.
机器人模型数据中包含实际机器人51的静态信息。机器人模型的静态信息中例如包含表示实际机器人51的构造的构造信息(机器人主体51a的关节数、连杆长度、工具的构造等)、表示作业开始前的位置以及/或者姿势的信息(例如机器人主体51a所具有的伺服马达的旋转角度信息)等。另外，机器人模型数据中包含机器人模型的动态信息、即操作者对控制器28的操作信息(指令)。为了使机器人模型进行动作而使用该操作信息，并且将其经由通信网络4从游戏装置2向实际机器人51发送，也在实际机器人51的动作中使用。即，若操作者一边观看显示装置25的显示画面一边操作控制器28，使显示画面上的机器人模型进行动作，则作业现场的实际机器人51也相同地进行动作。但是，如后所述，实际机器人51相比机器人模型的动作延迟恒定时间来进行动作。The robot model data includes static information of the actual robot 51. The static information of the robot model includes, for example, structural information indicating the structure of the actual robot 51 (the number of joints of the robot body 51a, the length of the connecting rod, the structure of the tool, etc.), information indicating the position and/or posture before the start of the operation (for example, the rotation angle information of the servo motor possessed by the robot body 51a), etc. In addition, the robot model data includes dynamic information of the robot model, that is, the operation information (instructions) of the operator to the controller 28. This operation information is used to make the robot model move, and it is sent from the game device 2 to the actual robot 51 via the communication network 4, and is also used in the movement of the actual robot 51. That is, if the operator operates the controller 28 while watching the display screen of the display device 25, so that the robot model on the display screen moves, the actual robot 51 at the work site also moves in the same way. However, as described later, the actual robot 51 moves with a constant delay compared to the movement of the robot model.
周边物模型数据中包含实际周边物的静态信息。实际周边物的静态信息中包含周边设备52的构造信息、表示周边设备52的作业开始前的位置以及/或者姿势的信息、实际机器人51的作业对象亦即工件W的形状数据、构造信息、表示工件W的作业开始前的位置以及/或者姿势的信息。另外，周边物模型数据中包含预测实际周边物的未来规定时间的位置、姿势而得的信息。状态信息取得部33和模拟部34进行该预测。在本实施方式中，具备状态信息取得部33和模拟部34的游戏装置2与本发明的“数据生成装置”对应。The surrounding object model data includes static information of the actual surrounding objects. The static information of the actual surrounding objects includes the structural information of the peripheral device 52, the information indicating the position and/or posture of the peripheral device 52 before the start of the operation, the shape data of the workpiece W, which is the work object of the actual robot 51, the structural information, and the information indicating the position and/or posture of the workpiece W before the start of the operation. In addition, the surrounding object model data includes information obtained by predicting the position and posture of the actual surrounding objects at a specified time in the future. The state information acquisition unit 33 and the simulation unit 34 perform this prediction. In the present embodiment, the game device 2 equipped with the state information acquisition unit 33 and the simulation unit 34 corresponds to the "data generation device" of the present invention.
具体而言，状态信息取得部33取得表示实际机器人51的周围的周边设备52、工件W等实际周边物的状态的状态信息。而且，模拟部34基于状态信息，模拟实际周边物的伴随时间经过的位置、姿势的变化。例如，在状态信息取得部33将表示对处于某个时刻的作为周边设备52的搬运装置(在本例中，为发送带51a)设定的搬运速度的信息和表示被该搬运装置搬运的工件W的位置的信息作为状态信息而取得的情况下，模拟部34能够根据该搬运速度信息和工件位置信息，容易地计算未来一定时间的工件W的位置、姿势。这样，模拟部34通过模拟来预测自当前时刻开始经过规定时间后的实际周边物的状态。而且，模拟部34将预测的结果生成为在周边物模型的制成中使用的周边物模型数据。Specifically, the state information acquisition unit 33 acquires state information representing the state of actual surrounding objects such as the peripheral equipment 52 and the workpiece W around the actual robot 51. Moreover, the simulation unit 34 simulates the change of the position and posture of the actual surrounding objects over time based on the state information. For example, when the state information acquisition unit 33 acquires information representing the transport speed set for the transport device (in this example, the transmission belt 51a) as the peripheral equipment 52 at a certain moment and information representing the position of the workpiece W transported by the transport device as state information, the simulation unit 34 can easily calculate the position and posture of the workpiece W at a certain time in the future based on the transport speed information and the workpiece position information. In this way, the simulation unit 34 predicts the state of the actual surrounding objects after a specified time has passed since the current moment through simulation. Moreover, the simulation unit 34 generates the predicted result as the surrounding object model data used in the creation of the surrounding object model.
图像显示部35将基于机器人模型数据和周边物模型数据制成的作业空间模型显示于显示装置25。例如，图像显示部35在基于机器人模型数据和周边物模型数据制成的机器人模型和周边物模型所配置的假想的作业空间中配置假想的照相机。将通过该假想的照相机拍摄的图像显示于显示装置25。假想照相机的位置、朝向、变焦镜头等可以预先决定，例如也可以能够与操作者对控制器28的操作相应地进行变更。假想作业空间中的假想照相机的位置、朝向也可以与实际作业空间中的拍摄装置52b的位置、朝向分别对应。The image display unit 35 displays the workspace model made based on the robot model data and the surrounding object model data on the display device 25. For example, the image display unit 35 configures a virtual camera in the virtual workspace configured by the robot model and the surrounding object model made based on the robot model data and the surrounding object model data. The image captured by the virtual camera is displayed on the display device 25. The position, orientation, zoom lens, etc. of the virtual camera can be predetermined, and can be changed according to the operator's operation on the controller 28, for example. The position and orientation of the virtual camera in the virtual workspace can also correspond to the position and orientation of the imaging device 52b in the actual workspace.
针对通信延迟测量部36、偏差检测部37以及模型修正部38，在后面叙述它们的详细内容。The communication delay measurement unit 36 , the deviation detection unit 37 , and the model correction unit 38 will be described in detail later.
图5是表示中转装置53的控制部55的功能性结构的框图。中转装置53的控制部55作为功能性结构而具有通信控制部71、和机器人侧时间管理部72。对于这些功能部而言，通过其与存储于中转装置53的控制部55以及/或者存储部57的规定的程序的配合来功能性地实现。Fig. 5 is a block diagram showing the functional structure of the control unit 55 of the transfer device 53. The control unit 55 of the transfer device 53 has a communication control unit 71 and a robot-side time management unit 72 as a functional structure. These functional units are functionally implemented by cooperating with the control unit 55 and/or the storage unit 57 stored in the transfer device 53.
通信控制部71控制通信部56，从游戏装置2接收操作者操作控制器28从而生成的操作信息。The communication control unit 71 controls the communication unit 56 to receive operation information generated by an operator operating the controller 28 from the game device 2 .
机器人侧时间管理部72管理机器人系统5侧的时间，以使得从操作者对控制器28进行操作开始至机器人51基于该操作进行动作为止的时间保持为恒定。The robot-side time management unit 72 manages the time on the robot system 5 side so that the time from when the operator operates the controller 28 to when the robot 51 operates based on the operation is kept constant.
(处理流程)(Processing Flow)
接下来，针对游戏装置2和中转装置53各自中的执行处理，参照图6～8进行说明。Next, the execution processes in the game device 2 and the transfer device 53 are described with reference to FIGS. 6 to 8 .
图6是表示机器人51的作业开始前的游戏装置2和中转装置53各自中的处理流程的图。在作业开始前，制成作业空间模型中的机器人模型、周边设备模型以及工件模型以使得它们与实际机器人51、周边设备52以及工件W分别为相互相同的状态。Fig. 6 is a diagram showing the processing flow in the game device 2 and the transfer device 53 before the operation of the robot 51 starts. Before the operation starts, the robot model, peripheral equipment model and workpiece model in the operation space model are made so that they are in the same state as the actual robot 51, peripheral equipment 52 and workpiece W.
具体而言，若通过中介装置6将游戏装置2与中转装置53连接，则首先中转装置53的通信控制部71将用于制成作业开始前的作业空间模型的信息向游戏装置2发送(步骤S201)。用于制成作业开始前的作业空间模型的信息中包含表示位于实际机器人51的周边的实际周边物的状态的状态信息。Specifically, when the game device 2 is connected to the transfer device 53 via the intermediary device 6, the communication control unit 71 of the transfer device 53 first sends information for creating a workspace model before the operation starts to the game device 2 (step S201). The information for creating a workspace model before the operation starts includes state information indicating the state of the actual surrounding objects located around the actual robot 51.
在本实施方式中，状态信息中包含通过设置于实际作业空间的拍摄装置52b拍摄周边设备52和工件W从而生成的拍摄信息。另外，状态信息中包含作为周边设备52的传感器的检测信息。传感器的检测信息中例如包含表示工件W是否位于作业空间的规定的位置的信息、表示工件W的位置或者姿势的信息等。另外，状态信息中包含对周边设备52设定的设定信息。例如，在机器人系统5作为周边设备52而包含搬运装置的情况下，设定信息中也可以包含针对搬运装置而设定的搬运速度、搬运间隔。搬运间隔可以是被搬运的工件W间的距离，也可以是从某个工件W被搬运到了机器人51前的规定的位置的时刻开始至其下一个工件W被搬运到规定的位置为止的时间间隔等。In the present embodiment, the status information includes shooting information generated by shooting the peripheral equipment 52 and the workpiece W by the shooting device 52b set in the actual working space. In addition, the status information includes detection information of the sensor as the peripheral equipment 52. The detection information of the sensor includes, for example, information indicating whether the workpiece W is located at a specified position in the working space, information indicating the position or posture of the workpiece W, etc. In addition, the status information includes setting information set for the peripheral equipment 52. For example, in the case where the robot system 5 includes a conveying device as the peripheral equipment 52, the setting information may also include the conveying speed and conveying interval set for the conveying device. The conveying interval may be the distance between the workpieces W being conveyed, or it may be the time interval from the moment when a workpiece W is conveyed to a specified position in front of the robot 51 to the moment when the next workpiece W is conveyed to a specified position, etc.
另外，在步骤S201中从中转装置53向游戏装置2发送的状态信息中也包含表示实际机器人51的状态的信息。表示实际机器人51的状态的信息中例如也可以包含机器人控制器51b所存储的实际机器人51的姿势信息、位置信息。另外，表示实际机器人51的状态的信息中也可以包含由作为周边设备52的拍摄装置52b获得的拍摄信息、传感器52c的检测信息。In addition, the state information sent from the transfer device 53 to the game device 2 in step S201 also includes information indicating the state of the actual robot 51. The information indicating the state of the actual robot 51 may also include, for example, posture information and position information of the actual robot 51 stored by the robot controller 51b. In addition, the information indicating the state of the actual robot 51 may also include photographic information obtained by the photographing device 52b as the peripheral device 52 and detection information of the sensor 52c.
在游戏装置2侧，状态信息取得部33取得从中转装置53接收到的状态信息(步骤S101)。而且，模拟部34基于由状态信息取得部33取得的状态信息，制作与作业开始前的实际作业空间相同的状态的作业空间模型(步骤S102)。状态信息中例如包含有工件W的形状数据、构造信息、工件W的位置、朝向、周边设备52的位置信息、构造信息、设定信息等。On the game device 2 side, the state information acquisition unit 33 acquires the state information received from the transfer device 53 (step S101). Furthermore, the simulation unit 34 creates a work space model in the same state as the actual work space before the work starts based on the state information acquired by the state information acquisition unit 33 (step S102). The state information includes, for example, shape data of the workpiece W, structural information, position and orientation of the workpiece W, position information, structural information, and setting information of the peripheral equipment 52.
具体而言，模拟部34基于作业开始前的实际机器人51的位置信息(在相对于实际作业空间设定的坐标系中的位置坐标)和姿势信息(机器人主体51a所具有的伺服马达的旋转角度信息等)，以使机器人模型的状态(位置、姿势等)为与实际机器人51相同的状态的方式，制成机器人模型。另外，模拟部34基于作业开始前的实际周边物的状态信息，以使周边物模型为与实际周边物相同的状态的方式，制成周边物模型。Specifically, the simulation unit 34 creates a robot model based on the position information (position coordinates in a coordinate system set relative to the actual work space) and posture information (rotation angle information of a servo motor of the robot body 51a, etc.) of the actual robot 51 before the start of the operation so that the state (position, posture, etc.) of the robot model is the same as that of the actual robot 51. In addition, the simulation unit 34 creates a surrounding object model based on the state information of the actual surrounding objects before the start of the operation so that the surrounding object model is the same as that of the actual surrounding objects.
图像显示部35生成在步骤S102中制成的作业空间模型的图像，并显示于显示装置25(步骤S103)。这样，作业开始的准备齐全。The image display unit 35 generates an image of the work space model created in step S102, and displays it on the display device 25 (step S103). In this way, preparations for starting the work are complete.
图7是表示机器人51的作业开始后的游戏装置2和中转装置53各自的处理流程。另外，图8是用于对分别在游戏装置2和中转装置53中伴随着时间的经过而执行的处理的一个例子进行说明的图。Fig. 7 is a diagram showing the processing flow of each of the game device 2 and the relay device 53 after the operation of the robot 51 starts. Fig. 8 is a diagram for explaining an example of the processing executed in each of the game device 2 and the relay device 53 as time passes.
步骤S103之后，游戏装置2的控制部21对是否进行了指示作业开始的操作进行判定(步骤S105)。在没有作业开始指示的情况(步骤S105：否)下，直到有作业开始指示为止为待机的状态。若通过操作者对控制器28进行指示作业开始的操作(步骤S105：是)，则通信控制部31向中转装置53发送作业开始指示(步骤S106)。另外，操作侧时间管理部32将作业开始指示的发送时刻作为操作侧基准时刻t1存储于游戏装置2的存储装置(RAM212或者存储部23等)(步骤S107，也参照图8)。After step S103, the control unit 21 of the game device 2 determines whether an operation to instruct the start of the operation has been performed (step S105). In the case where there is no operation start instruction (step S105: No), the game device 2 is in a standby state until there is an operation start instruction. If the operator performs an operation to instruct the start of the operation on the controller 28 (step S105: Yes), the communication control unit 31 sends the operation start instruction to the relay device 53 (step S106). In addition, the operation side time management unit 32 stores the sending time of the operation start instruction as the operation side reference time t1 in the storage device (RAM212 or storage unit 23, etc.) of the game device 2 (step S107, also refer to Figure 8).
在机器人系统5侧，若中转装置53的通信控制部71接收到作业开始指示(步骤S202)，则通信控制部71将从作业开始指示的接收时刻开始待机了规定时间Δt后的时刻作为作业开始时刻，向机器人51和周边设备52发送作业开始指示(步骤S203)。另外，机器人侧时间管理部72将从作业开始指示的接收时刻开始经过规定时间Δt后的时刻作为机器人侧基准时刻t2来存储(步骤S204，也参照图8)。On the robot system 5 side, if the communication control unit 71 of the relay device 53 receives the operation start instruction (step S202), the communication control unit 71 takes the time after the predetermined time Δt from the time of receiving the operation start instruction as the operation start time, and sends the operation start instruction to the robot 51 and the peripheral device 52 (step S203). In addition, the robot-side time management unit 72 stores the time after the predetermined time Δt from the time of receiving the operation start instruction as the robot-side reference time t2 (step S204, also refer to FIG. 8).
这样，像以下说明的那样，在显示于显示装置25的作业空间模型中，从操作侧基准时刻t1起开始机器人模型的作业，另一方面，在实际作业空间中，从机器人侧基准时刻t2起开始实际机器人的作业。换而言之，实际作业空间比作业空间模型延迟基准时刻的差值(t2-t1)来进行作业。例如在从游戏装置2向机器人系统5发送的作业开始指示中也可以包含使周边设备52从停止状态移至运转状态的指令。例如，也可以构成为作业空间模型的工件模型被搬运装置模型从操作侧基准时刻t1开始搬运，实际作业空间的实际工件W被搬运装置从机器人侧基准时刻t2开始搬运。Thus, as described below, in the workspace model displayed on the display device 25, the operation of the robot model starts from the reference time t1 on the operating side, and on the other hand, in the actual workspace, the operation of the actual robot starts from the reference time t2 on the robot side. In other words, the actual workspace performs the operation by the difference (t2-t1) of the reference time delayed from the workspace model. For example, the operation start instruction sent from the game device 2 to the robot system 5 may also include an instruction to move the peripheral device 52 from a stopped state to an operating state. For example, the workpiece model of the workspace model may also be constructed to be transported by a transport device model from the reference time t1 on the operating side, and the actual workpiece W in the actual workspace may be transported by the transport device from the reference time t2 on the robot side.
在游戏装置2中，在步骤S107中设定了基准时刻t1后，能够进行机器人51的远程操作。即，在对控制器28进行了用于操作机器人51的操作的情况下(步骤S108：是)，生成操作信息。模拟部34基于操作信息，模拟机器人模型的动作(步骤S109)。另外，通信控制部31将表示直到进行与该操作信息对应的操作为止所经过的时间的从基准时刻t1开始的经过时间T和操作信息一同向中转装置53发送(步骤S110)。In the game device 2, after the reference time t1 is set in step S107, the remote operation of the robot 51 can be performed. That is, when the controller 28 is operated to operate the robot 51 (step S108: Yes), the operation information is generated. The simulation unit 34 simulates the action of the robot model based on the operation information (step S109). In addition, the communication control unit 31 sends the elapsed time T from the reference time t1, which indicates the time until the operation corresponding to the operation information is performed, to the relay device 53 together with the operation information (step S110).
在机器人系统5侧，中转装置53的通信控制部71若以组合的形式从游戏装置2接收到操作信息和时间信息，则向机器人51发送动作指令，以使得其在从基准时刻t2开始的经过时间T后，执行基于接收到的操作信息的操作(步骤S205)。此外，在步骤S110中，替代经过时间T，发送显示表示进行了操作的时刻的时刻信息等、表示进行了与操作信息对应的操作的时间的时间信息即可。On the robot system 5 side, if the communication control unit 71 of the relay device 53 receives the operation information and time information from the game device 2 in a combined form, it sends an action instruction to the robot 51 so that it performs an operation based on the received operation information after the time T has passed since the reference time t2 (step S205). In addition, in step S110, instead of the elapsed time T, time information indicating the time when the operation corresponding to the operation information was performed, such as time information indicating the time when the operation was performed, may be sent.
另外，无论是否判定为在步骤S108中进行了操作，模拟部34均模拟实际周边物的状态(步骤S111)。具体而言，模拟部34基于状态信息，对自当前时刻开始经过规定时间后的实际周边物的状态进行预测，并将预测而得的结果生成为在周边物模型的制作中使用的周边物模型数据。此外，状态信息也可以并非是状态信息取得部33在作业开始前在步骤S101中取得的状态信息，也可以是在作业开始后取得的信息。即，也可以从机器人系统5向游戏装置2依次发送最新的状态信息，状态信息取得部33也可以依次取得最新的状态信息。模拟部34也可以基于最新的状态信息来预测实际周边物的状态。In addition, regardless of whether it is determined that an operation is performed in step S108, the simulation unit 34 simulates the state of the actual surrounding objects (step S111). Specifically, the simulation unit 34 predicts the state of the actual surrounding objects after a specified time from the current moment based on the state information, and generates the predicted result as the surrounding object model data used in the production of the surrounding object model. In addition, the state information may not be the state information obtained by the state information acquisition unit 33 in step S101 before the operation starts, but may be information obtained after the operation starts. That is, the latest state information may also be sent from the robot system 5 to the game device 2 in sequence, and the state information acquisition unit 33 may also obtain the latest state information in sequence. The simulation unit 34 may also predict the state of the actual surrounding objects based on the latest state information.
图像显示部35根据基于步骤S109和S111中的模拟结果而生成的数据，将表示作业空间模型的图像显示于显示装置25(步骤S112)。图像显示部35显示相对于实际的作业空间的状况早机器人侧基准时刻t2与操作侧基准时刻t1的差值(t2-t1)前的状况的作业空间模型。换而言之，如图8所示，实际的作业空间相对于显示于显示装置25的作业空间模型延迟机器人侧基准时刻t2与操作侧基准时刻t1的差值(t2-t1)来进行作业。The image display unit 35 displays an image representing the workspace model on the display device 25 based on the data generated based on the simulation results in steps S109 and S111 (step S112). The image display unit 35 displays the workspace model of the condition earlier than the difference (t2-t1) between the robot side reference time t2 and the operation side reference time t1 relative to the actual workspace condition. In other words, as shown in FIG8 , the actual workspace is operated by delaying the difference (t2-t1) between the robot side reference time t2 and the operation side reference time t1 relative to the workspace model displayed on the display device 25.
直到针对控制器28进行用于作业结束的操作为止，或者直到一定的作业结束为止(步骤S113：否)，重复步骤S108～S112。Steps S108 to S112 are repeated until an operation for completing the work is performed on the controller 28 or until a certain work is completed (step S113 : No).
在上述的步骤S203中，机器人侧时间管理部72不将作业开始指示的接收时刻作为机器人侧基准时刻t2，而将从作业开始指示的接收时刻开始待机了规定时间Δt后的时刻作为机器人侧基准时刻t2来设定。这样，在操作侧基准时刻t1与机器人侧基准时刻t2之间设置恒定的间隔，从而吸收游戏装置2与机器人系统5的通信延迟(参照图8的Δd1、Δd2)的变动。In the above step S203, the robot-side time management unit 72 does not use the reception time of the work start instruction as the robot-side reference time t2, but sets the time after waiting for a predetermined time Δt from the reception time of the work start instruction as the robot-side reference time t2. In this way, a constant interval is set between the operation-side reference time t1 and the robot-side reference time t2, thereby absorbing the variation of the communication delay (refer to Δd1 and Δd2 in FIG8 ) between the game device 2 and the robot system 5.
此外，待机时间Δt也可以基于游戏装置2与机器人系统5(详细而言为中转装置53)之间的实际通信延迟来设定。通信延迟测量部36对游戏装置2与和其建立了对应关系的机器人系统5(更详细而言为中转装置53)之间的通信延迟进行测量。通信延迟的测量利用公知的方法来实施。机器人侧时间管理部72也可以与在步骤S105之前由通信延迟测量部36测量出的通信延迟的变动程度相应地，设定有待机时间Δt的长度、换而言之机器人侧基准时刻t2与操作侧基准时刻t1的差值(t2-t1)。In addition, the standby time Δt can also be set based on the actual communication delay between the game device 2 and the robot system 5 (more specifically, the transfer device 53). The communication delay measurement unit 36 measures the communication delay between the game device 2 and the robot system 5 (more specifically, the transfer device 53) with which the corresponding relationship is established. The measurement of the communication delay is implemented using a known method. The robot-side time management unit 72 can also set the length of the standby time Δt, in other words, the difference (t2-t1) between the robot-side reference time t2 and the operation-side reference time t1, according to the degree of variation of the communication delay measured by the communication delay measurement unit 36 before step S105.
另外，在本实施方式中，在重复步骤S108～S112时，定期地修正作业空间模型。偏差检测部37对在规定的时刻显示于显示装置25的作业空间模型的状况与自规定的时刻开始经过规定的时间后的实际作业空间的状况之间的偏差的程度进行检测。更具体而言，偏差检测部37对规定的时刻的作业空间模型的状况与自该规定的时刻开始经过机器人侧基准时刻t2和操作侧基准时刻t1的差值(t2-t1)后的实际作业空间的状况之间的偏差的程度进行检测。In addition, in the present embodiment, when steps S108 to S112 are repeated, the workspace model is corrected periodically. The deviation detection unit 37 detects the degree of deviation between the state of the workspace model displayed on the display device 25 at a specified time and the state of the actual workspace after a specified time has passed since the specified time. More specifically, the deviation detection unit 37 detects the degree of deviation between the state of the workspace model at a specified time and the state of the actual workspace after the difference (t2-t1) between the robot side reference time t2 and the operator side reference time t1 has passed since the specified time.
例如，偏差检测部37也可以对表示某个时刻ta的实际作业空间的状态的状态信息、与由模拟部34基于在此时刻ta之前取得的状态信息而预测的此时刻ta的作业空间模型的状态进行比较来检测偏差的程度。表示偏差检测部37所比较的实际作业空间的状态的状态信息中也可以包含用于模拟部34预测实际周边物的状态而使用的状态信息。偏差检测部37例如也可以从机器人系统5接收时刻ta的实际作业空间的拍摄信息，并根据该拍摄信息利用图像识别来判断实际周边物(周边设备52、工件W)的位置、姿势。偏差检测部37对根据时刻ta的拍摄信息而判断出的实际周边物的位置、姿势与和时刻ta的实际作业空间对应的时刻(即，时刻(ta-(t2-t1)))的作业空间模型中的周边物模型的位置、姿势进行比较。即，偏差检测部37对根据时刻ta的拍摄信息而判断出的实际周边物的状态(周边设备52、工件W的位置、姿势等)与比时刻ta早时间(t2-t1)前的周边物模型的状态进行比较。For example, the deviation detection unit 37 may detect the degree of deviation by comparing the state information indicating the state of the actual work space at a certain time ta with the state of the work space model at this time ta predicted by the simulation unit 34 based on the state information obtained before this time ta. The state information indicating the state of the actual work space compared by the deviation detection unit 37 may also include state information used by the simulation unit 34 to predict the state of the actual surrounding objects. For example, the deviation detection unit 37 may receive the photographing information of the actual work space at time ta from the robot system 5, and determine the position and posture of the actual surrounding objects (peripheral equipment 52, workpiece W) by image recognition based on the photographing information. The deviation detection unit 37 compares the position and posture of the actual surrounding objects determined based on the photographing information at time ta with the position and posture of the surrounding object model in the work space model at the time corresponding to the actual work space at time ta (that is, time (ta-(t2-t1))). That is, the deviation detection unit 37 compares the state of the actual surrounding objects (the position and posture of the peripheral equipment 52, workpiece W, etc.) determined based on the photographing information at time ta with the state of the surrounding object model before time (t2-t1) before time ta.
而且，模型修正部38在由偏差检测部37检测出的偏差超出预先设定的范围的情况下以消除偏差的方式修正作业空间模型。例如，模型修正部38也可以将模拟部34中使用的状态信息调整由偏差检测部37检测出的偏差的量。或者，模型修正部38也可以将由模拟部34生成的周边物模型数据调整由偏差检测部37检测出的偏差的量。此外，在模型修正部38进行了作业空间模型的修正后，模拟部34模拟实际周边物的状态的情况下，将作业空间模型的修正考虑在内，来模拟实际周边物的状态。Furthermore, the model correction unit 38 corrects the workspace model in a manner to eliminate the deviation when the deviation detected by the deviation detection unit 37 exceeds a preset range. For example, the model correction unit 38 may adjust the state information used in the simulation unit 34 by the amount of the deviation detected by the deviation detection unit 37. Alternatively, the model correction unit 38 may adjust the surrounding object model data generated by the simulation unit 34 by the amount of the deviation detected by the deviation detection unit 37. Furthermore, after the model correction unit 38 corrects the workspace model, when the simulation unit 34 simulates the state of the actual surrounding objects, the correction of the workspace model is taken into account to simulate the state of the actual surrounding objects.
像以上说明的那样，根据作为本实施方式的数据生成装置的游戏装置2和远程操作系统1，通过模拟部34来预测自当前时刻开始经过规定时间后的实际周边物的状态，并将该预测结果生成为在显示于显示装置25的周边物模型的制成中使用的周边物模型数据。因此，能够使周边物模型的状态与实际周边物的状态之间也产生同机器人模型与实际机器人之间相同的时间偏差。由此，能够使机器人模型与周边物模型的时间轴一致，因此能够抑制通信延迟对操作者的操作施加的影响。As described above, according to the game device 2 and the remote operation system 1 as the data generation device of the present embodiment, the state of the actual surrounding objects after a predetermined time has passed from the current moment is predicted by the simulation unit 34, and the prediction result is generated as the surrounding object model data used in the creation of the surrounding object model displayed on the display device 25. Therefore, the state of the surrounding object model and the state of the actual surrounding objects can also produce the same time deviation as that between the robot model and the actual robot. As a result, the time axis of the robot model and the surrounding object model can be made consistent, so that the influence of communication delay on the operator's operation can be suppressed.
另外，在本实施方式中，偏差检测部37对在规定的时刻显示于显示装置25的作业空间模型的状况与自规定的时刻开始经过规定时间后的实际作业空间的状况之间的偏差的程度进行检测，模型修正部38在由偏差检测部37检测出的偏差超出预先设定的范围的情况下以消除偏差的方式修正作业空间模型。因此，能够抑制显示于显示装置25的作业空间模型的状况与实际作业空间的状况之间的偏差。In addition, in the present embodiment, the deviation detection unit 37 detects the degree of deviation between the state of the workspace model displayed on the display device 25 at a predetermined time and the state of the actual workspace after a predetermined time has passed from the predetermined time, and the model correction unit 38 corrects the workspace model in a manner to eliminate the deviation when the deviation detected by the deviation detection unit 37 exceeds a preset range. Therefore, the deviation between the state of the workspace model displayed on the display device 25 and the state of the actual workspace can be suppressed.
(其他的实施方式)(Other Embodiments)
本发明并不限定于上述的实施方式，能够在不脱离本发明的主旨的范围内进行各种变形。The present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit and scope of the present invention.
例如，在上述实施方式中，作为操作终端而例示了游戏装置2，但本发明中的操作终端也可以不是游戏装置2。操作终端只要具有接受操作者的操作的操作器和操作者能够视觉辨认的显示器即可。例如游戏装置2除是公知的各种游戏装置以外，例如也可以是个人信息终端(PDA(Personal Data Assistant：个人数据助手))、智能电话、个人计算机、平板电脑、以及机器人专用远程操作器中的任一个。For example, in the above-mentioned embodiment, the game device 2 is illustrated as an operation terminal, but the operation terminal in the present invention may not be the game device 2. The operation terminal only needs to have an operator that accepts the operation of the operator and a display that the operator can visually recognize. For example, the game device 2 may be any one of a personal information terminal (PDA (Personal Data Assistant)), a smart phone, a personal computer, a tablet computer, and a robot-specific remote operator in addition to various well-known game devices.
另外，例如，中介装置6或者其他的服务装置的控制部也可以通过执行规定的程序来作为状态信息取得部和预测部而发挥功能。即，本发明的“数据生成装置”也可以不是供操作者操作的操作终端，也可以是与操作终端通信的装置。例如，本发明的数据生成装置也可以是机器人控制器51b、中转装置53、中介装置6、或者与中介装置6不同的服务装置。本发明的数据生成装置也可以不具备通信延迟测量部36、偏差检测部37以及模型修正部38的一部分或者全部。In addition, for example, the control unit of the intermediary device 6 or other service device may also function as a state information acquisition unit and a prediction unit by executing a prescribed program. That is, the "data generation device" of the present invention may not be an operation terminal for the operator to operate, but may be a device that communicates with the operation terminal. For example, the data generation device of the present invention may also be a robot controller 51b, a transfer device 53, an intermediary device 6, or a service device different from the intermediary device 6. The data generation device of the present invention may not have a part or all of the communication delay measurement unit 36, the deviation detection unit 37, and the model correction unit 38.
另外，本发明的“数据生成程序”也可以替代存储于作为操作终端的游戏装置2的存储装置或者在其基础上，存储于机器人控制器51b、中转装置53、中介装置6、以及与中介装置6不同的服务装置中的至少一个所具备的存储装置。另外，本发明的“数据生成程序”被机器人控制器51b、中转装置53、中介装置6、以及与中介装置6不同的服务装置中的至少一个所分别内置的计算机执行，由此使该计算机作为状态信息取得部和预测部而发挥功能。沿着图6和图7说明的游戏装置2和中转装置53的处理流程也并非限制本发明。In addition, the "data generation program" of the present invention can also be stored in a storage device of at least one of the robot controller 51b, the transfer device 53, the intermediary device 6, and a service device different from the intermediary device 6, instead of or on the basis of the storage device stored in the game device 2 as the operation terminal. In addition, the "data generation program" of the present invention is executed by a computer respectively built into at least one of the robot controller 51b, the transfer device 53, the intermediary device 6, and a service device different from the intermediary device 6, thereby making the computer function as a state information acquisition unit and a prediction unit. The processing flow of the game device 2 and the transfer device 53 described along Figures 6 and 7 does not limit the present invention.
另外，远程操作系统也可以只具备一个游戏装置2。另外，远程操作系统1也可以只具备一个机器人系统5。另外，远程操作系统1也可以不具备中介装置6。另外，机器人系统5中的中转装置53与机器人控制器51b也可以一体构成。即，机器人系统5也可以具备兼具中转装置53和机器人控制器51b各自的功能的1台控制装置。In addition, the remote operation system may also include only one game device 2. In addition, the remote operation system 1 may also include only one robot system 5. In addition, the remote operation system 1 may not include the intermediary device 6. In addition, the transfer device 53 and the robot controller 51b in the robot system 5 may also be integrally formed. That is, the robot system 5 may also include a control device that has both the functions of the transfer device 53 and the robot controller 51b.
另外，本发明中的实际机器人也可以不是工业用机器人，只要是与操作终端上的操作者的操作相应地进行动作的机器人即可。例如本发明的实际机器人也可以是提供护理、医疗、运搬、清洁、烹饪等服务的服务用机器人。另外，本发明的实际机器人也可以是类人物。实际机器人基于通过操作者对操作器的操作而生成的操作信息来进行动作，但实际机器人也可以不仅基于该操作信息也基于预先设定的任务程序而进行动作。In addition, the actual robot in the present invention may not be an industrial robot, as long as it is a robot that moves in accordance with the operation of the operator on the operation terminal. For example, the actual robot of the present invention may also be a service robot that provides nursing, medical, transportation, cleaning, cooking and other services. In addition, the actual robot of the present invention may also be a person-like robot. The actual robot moves based on the operation information generated by the operator's operation of the manipulator, but the actual robot may also move based not only on the operation information but also on a pre-set task program.
在上述实施方式中，数据生成程序是使作为操作终端的游戏装置2的操纵器21执行：取得表示实际周边物的状态的状态信息的状态信息取得步骤、和基于上述状态信息，对当前时刻之后的上述实际周边物的状态进行预测，并且将预测的结果生成为在显示于上述显示器的上述周边物模型的制作中使用的周边物模型数据的预测步骤，但本发明的数据生成程序也可以使其他的计算机执行上述的状态信息取得步骤和预测步骤。例如，本发明的数据生成程序也可以使机器人控制器51b、中转装置53、中介装置6、以及与中介装置6不同的服务装置中的至少一个所具备的操纵器(计算机)执行上述的状态信息取得步骤和预测步骤。数据生成程序也可以分散存储于多个存储装置。例如，也可以构成为数据生成程序的一部分存储于操作终端的存储装置，除此以外存储于其他的存储装置(例如中转装置53等)。另外，本发明的数据生成程序也可以使计算机执行：对在规定的时刻显示于上述显示器的上述作业空间模型的状况与自上述规定的时刻开始经过上述规定时间后的上述实际作业空间的状况之间的偏差的程度进行检测的偏差检测步骤。另外，本发明的数据生成程序也可以使计算机执行：在通过上述偏差检测步骤检测出的偏差超出预先设定的范围的情况下，以消除上述偏差的方式修正上述作业空间模型的模型修正步骤。In the above embodiment, the data generation program is to make the manipulator 21 of the game device 2 as the operation terminal execute: a state information acquisition step of acquiring state information representing the state of the actual surrounding objects, and a prediction step of predicting the state of the actual surrounding objects after the current moment based on the state information, and generating the predicted result as the surrounding object model data used in the production of the surrounding object model displayed on the above display, but the data generation program of the present invention can also make other computers execute the above state information acquisition step and prediction step. For example, the data generation program of the present invention can also make the manipulator (computer) possessed by at least one of the robot controller 51b, the transfer device 53, the intermediary device 6, and the service device different from the intermediary device 6 execute the above state information acquisition step and prediction step. The data generation program can also be stored in a plurality of storage devices in a distributed manner. For example, it can also be configured so that a part of the data generation program is stored in the storage device of the operation terminal, and the rest is stored in other storage devices (such as the transfer device 53, etc.). In addition, the data generation program of the present invention may also cause the computer to execute: a deviation detection step for detecting the degree of deviation between the condition of the workspace model displayed on the display at a specified time and the condition of the actual workspace after the specified time has passed since the specified time. In addition, the data generation program of the present invention may also cause the computer to execute: a model correction step for correcting the workspace model in a manner to eliminate the deviation when the deviation detected by the deviation detection step exceeds a preset range.
附图标记说明Description of Reference Numerals
1…远程操作系统；2…游戏装置(操作终端)；4…通信网络；5…机器人系统；6…中介装置；25…显示装置(显示器)；28…控制器(操作器)；33…状态信息取得部；34…模拟部(预测部)；35…图像显示部；36…通信延迟测量部；37…偏差检测部；38…模型修正部；51…机器人；51a…机器人主体；51b…机器人控制器；52…周边设备；52a…发送带；52b…拍摄装置；52c…传感器；53…中转装置；55…控制部。1…remote operation system; 2…game device (operation terminal); 4…communication network; 5…robot system; 6…intermediary device; 25…display device (display); 28…controller (operator); 33…status information acquisition unit; 34…simulation unit (prediction unit); 35…image display unit; 36…communication delay measurement unit; 37…deviation detection unit; 38…model correction unit; 51…robot; 51a…robot body; 51b…robot controller; 52…peripheral equipment; 52a…transmission belt; 52b…shooting device; 52c…sensor; 53…transfer device; 55…control unit.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2018-151917 | 2018-08-10 | ||
| JP2018151917 | 2018-08-10 | ||
| JP2019-105754 | 2019-06-05 | ||
| JP2019105754AJP7281349B2 (en) | 2018-08-10 | 2019-06-05 | remote control system |
| PCT/JP2019/031495WO2020032211A1 (en) | 2018-08-10 | 2019-08-08 | Data generating device, data generating method, data generating program, and remote operation system |
| Publication Number | Publication Date |
|---|---|
| CN112469538A CN112469538A (en) | 2021-03-09 |
| CN112469538Btrue CN112469538B (en) | 2024-04-19 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201980048635.XAActiveCN112469538B (en) | 2018-08-10 | 2019-08-08 | Data generation device and method, data generation program, and remote operation system |
| Country | Link |
|---|---|
| KR (1) | KR102518766B1 (en) |
| CN (1) | CN112469538B (en) |
| TW (1) | TW202014278A (en) |
| WO (1) | WO2020032211A1 (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022003871A1 (en)* | 2020-07-01 | 2022-01-06 | 東芝三菱電機産業システム株式会社 | Diagnosis assistance device for manufacturing equipment |
| JPWO2023053520A1 (en)* | 2021-09-30 | 2023-04-06 | ||
| TWI822406B (en)* | 2022-10-20 | 2023-11-11 | 國立中正大學 | Universal translation control system for remote control of robots with joysticks |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7312766B1 (en)* | 2000-09-22 | 2007-12-25 | Canadian Space Agency | Method and system for time/motion compensation for head mounted displays |
| JP2014229157A (en)* | 2013-05-24 | 2014-12-08 | 日本電信電話株式会社 | Delay compensation device, method, program, and recording medium |
| KR20160002329A (en)* | 2014-06-30 | 2016-01-07 | 가부시키가이샤 야스카와덴키 | Robot simulator and file generation method for robot simulator |
| JP2017056529A (en)* | 2015-09-17 | 2017-03-23 | 株式会社安川電機 | Transfer system and transfer method |
| CN107921640A (en)* | 2015-08-25 | 2018-04-17 | 川崎重工业株式会社 | Teleoperation robot system and method of operation thereof |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH01209505A (en)* | 1988-02-17 | 1989-08-23 | Toshiba Corp | Teaching device for remote control robot |
| JPH0719818A (en)* | 1993-06-30 | 1995-01-20 | Kajima Corp | 3D motion prediction device |
| JP2015047666A (en) | 2013-09-02 | 2015-03-16 | トヨタ自動車株式会社 | Remote operation device and operation image display method |
| US9579799B2 (en)* | 2014-04-30 | 2017-02-28 | Coleman P. Parker | Robotic control system using virtual reality input |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7312766B1 (en)* | 2000-09-22 | 2007-12-25 | Canadian Space Agency | Method and system for time/motion compensation for head mounted displays |
| JP2014229157A (en)* | 2013-05-24 | 2014-12-08 | 日本電信電話株式会社 | Delay compensation device, method, program, and recording medium |
| KR20160002329A (en)* | 2014-06-30 | 2016-01-07 | 가부시키가이샤 야스카와덴키 | Robot simulator and file generation method for robot simulator |
| CN107921640A (en)* | 2015-08-25 | 2018-04-17 | 川崎重工业株式会社 | Teleoperation robot system and method of operation thereof |
| JP2017056529A (en)* | 2015-09-17 | 2017-03-23 | 株式会社安川電機 | Transfer system and transfer method |
| Publication number | Publication date |
|---|---|
| KR102518766B1 (en) | 2023-04-06 |
| WO2020032211A1 (en) | 2020-02-13 |
| TW202014278A (en) | 2020-04-16 |
| CN112469538A (en) | 2021-03-09 |
| KR20210041048A (en) | 2021-04-14 |
| Publication | Publication Date | Title |
|---|---|---|
| JP7281349B2 (en) | remote control system | |
| CN110573308B (en) | Computer-based method and system for spatial programming of robotic devices | |
| US11192249B2 (en) | Simulation device for robot | |
| CN112469538B (en) | Data generation device and method, data generation program, and remote operation system | |
| EP2963513B1 (en) | Teaching apparatus and robot system | |
| US20080013825A1 (en) | Simulation device of robot system | |
| CN112543960A (en) | Information processing apparatus, mediation apparatus, simulation system, and information processing method | |
| CN112847336A (en) | Action learning method, action learning device, storage medium and electronic equipment | |
| CN115997182A (en) | Simulation device and simulation system | |
| TW202122225A (en) | System and method for robotic bin picking using advanced scanning techniques | |
| JP7681756B2 (en) | Robot operation system and information processing method | |
| CN114502337B (en) | Robot system and method for forming three-dimensional model of workpiece | |
| JP2018122376A (en) | Image processing device, robot control device, and robot | |
| US20230098696A1 (en) | Work Program Production System and Work Program Production Method | |
| JP2021142596A (en) | Simulation device and program | |
| CN115952389A (en) | Real-time compensation method, device, equipment and storage medium for automatic machinery | |
| JP2020163499A (en) | Object detection method, object detection device and robot system | |
| JP2022132506A (en) | Simulation method, simulation system, and data structure of module data | |
| JP7401250B2 (en) | Image processing device, control method and program | |
| CN114115288A (en) | Robot gait adjustment method, device, equipment and storage medium | |
| CN105997247A (en) | A control terminal, an operation apparatus terminal, a method and a system for remote operations | |
| JP2020160615A (en) | Object posture control program and object posture control system | |
| TW202448646A (en) | Teaching program generating apparatus | |
| JP2021154426A (en) | Image processing device, control method of image processing device and program | |
| CN120693577A (en) | Simulation system, method, field equipment, electronic equipment and medium for device under test |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |