TWI859580B - Assembly instruction system and method thereof - Google Patents

Abstract

An assembly instruction system including a camera, an augmented reality glasses, and a computing device. The camera is used to take pictures at the working area to generate a reality image, wherein an object that needs to be assembled is set in the working area and there is an identification code on the object that needs to be assembled. The augmented reality glasses is used to be controlled to show a reality and virtual overlap image. The computing device is used to obtain an assembly process that is corresponding to the object that needs to be assembled from a plurality of pre-saved assembly process according to the identification code in the reality image, and overlaps the instruction graphs on the reality image to generate the reality and virtual overlap image. After the reality and virtual overlap image is shown, it determines whether a sub-image that satisfies the object assembly process exists, and shows a warning signal on the augmented reality glasses when the sub-image does not exist in the reality image.

Description

Translated fromChinese

組裝指引系統及其方法Assembly guidance system and method thereof

本發明係關於一種組裝指引系統及方法，特別是關於擴增實境的組裝指引系統。The present invention relates to an assembly guidance system and method, and in particular to an augmented reality assembly guidance system.

擴增實境是指透過攝影機影像的位置及角度精算並加上圖像分析技術，讓螢幕上的虛擬世界能夠與現實世界場景進行結合與互動的技術。這種技術於1990年提出。隨著隨身電子產品運算能力的提升，擴增實境的用途也越來越廣。Augmented reality refers to a technology that uses the position and angle of camera images and image analysis technology to allow the virtual world on the screen to combine and interact with the real world scene. This technology was proposed in 1990. With the improvement of the computing power of portable electronic products, the use of augmented reality is becoming more and more extensive.

現行作業員上產線作業之前會進行組裝作業的教學，但即使再熟練的作業員於組裝時，仍可能一時疏忽而造成組裝錯誤而需要重工。Currently, operators are taught how to assemble products before they start working on the production line. However, even the most experienced operators may still make mistakes due to negligence and need to re-do the work.

鑒於上述，本發明提供一種組裝指引系統及方法。In view of the above, the present invention provides an assembly guidance system and method.

依據本發明一實施例的組裝指引系統包含相機、擴增實境眼鏡，以及運算裝置。相機用於拍攝作業區域以產生實境影像，其中作業區域設置有待組裝物，且待組裝物上標示有識別碼。擴增實境眼鏡，用於受控以顯示虛實疊合影像。運算裝置，用於根據實境影像中對應於識別碼的圖案，從多個預存組裝流程中取得該待組裝物的目標組裝流程，根據目標組裝流程於實境影像疊合指引圖像以產生虛實疊合影像，在虛實疊合影像顯示後，判斷實境影像中是否存在符合目標組裝流程的目標子影像，且在實境影像中不存在目標子影像時，通知擴增實境眼鏡於虛實疊合影像上呈現警示訊息。According to an embodiment of the present invention, an assembly guidance system includes a camera, augmented reality glasses, and a computing device. The camera is used to photograph a work area to generate a real image, wherein the work area is provided with objects to be assembled, and the objects to be assembled are marked with identification codes. The augmented reality glasses are controlled to display virtual and real superimposed images. The computing device is used to obtain the target assembly process of the object to be assembled from multiple pre-stored assembly processes according to the pattern corresponding to the identification code in the real image, and to superimpose the guide image on the real image according to the target assembly process to generate a virtual superimposed image. After the virtual superimposed image is displayed, it is determined whether there is a target sub-image that meets the target assembly process in the real image, and when there is no target sub-image in the real image, the augmented reality glasses are notified to present a warning message on the virtual superimposed image.

依據本發明一實施例的組裝指引方法包含以相機拍攝作業區域以產生實境影像，其中作業區域設置有待組裝物，且待組裝物上標示有識別碼、以運算裝置根據實境影像中對應於識別碼的圖案，從多個預存組裝流程中取得待組裝物的目標組裝流程，根據目標組裝流程中的一步驟於實境影像疊合指引圖像以產生虛實疊合影像、以擴增實境眼鏡顯示虛實疊合影像、以運算裝置判斷實境影像中是否存在符合目標組裝流程的目標子影像、以運算裝置通知擴增實境眼鏡於虛實疊合影像上呈現警示訊息、以運算裝置判斷目標組裝流程中的步驟是否已執行完，以及移除虛實疊合影像中的指引圖像，並根據目標組裝流程中的下個步驟疊合另一指引圖像以更新虛實疊合影像。According to an embodiment of the present invention, an assembly guidance method includes photographing a work area with a camera to generate a real image, wherein the work area is provided with an object to be assembled and an identification code is marked on the object to be assembled, and a computing device obtains a target assembly process of the object to be assembled from a plurality of pre-stored assembly processes according to a pattern corresponding to the identification code in the real image, and superimposes a guidance image on the real image according to a step in the target assembly process to generate a virtual superimposed image, and augments the real image with a virtual superimposed image. The augmented reality glasses display a virtual superimposed image, use a computing device to determine whether there is a target sub-image that meets the target assembly process in the real image, use the computing device to notify the augmented reality glasses to present a warning message on the virtual superimposed image, use the computing device to determine whether the steps in the target assembly process have been completed, and remove the guide image in the virtual superimposed image, and superimpose another guide image according to the next step in the target assembly process to update the virtual superimposed image.

綜上所述，本發明一或多個實施例的組裝指引系統及其方法可以藉由有效的指引作業員使其節省學習時間以及提高作業正確率。In summary, the assembly guidance system and method of one or more embodiments of the present invention can save learning time and improve the accuracy of operations by effectively guiding operators.

以上之關於本揭露內容之說明及以下之實施方式之說明係用以示範與解釋本發明之精神與原理，並且提供本發明之專利申請範圍更進一步之解釋。The above description of the disclosed content and the following description of the implementation method are used to demonstrate and explain the spirit and principle of the present invention, and provide a further explanation of the scope of the patent application of the present invention.

1’,1:組裝指引系統1’,1:Assembly guidance system

11:相機(第一相機)11: Camera (first camera)

12:擴增實境眼鏡12: Augmented reality glasses

13:運算裝置13: Computing device

14:第二相機14: Second camera

I1:第一指引圖像I1: First guidance image

I2:第二指引圖像I2: Second guidance image

I3:第三指引圖像I3: The third guiding image

R1:實境影像R1: Realistic image

S1~S7:方法步驟流程S1~S7: Method steps flow

圖1係依據本發明一實施例所繪示的組裝指引系統的功能方塊圖。FIG1 is a functional block diagram of an assembly guidance system according to an embodiment of the present invention.

圖2係依據本發明一實施例所繪示的組裝指引方法的方法流程圖。Figure 2 is a flow chart of an assembly guidance method according to an embodiment of the present invention.

圖3A係依據本發明一實施例在安裝主機板時所示的虛實疊合影像的示意圖。FIG3A is a schematic diagram of a virtual and real superimposed image when installing a motherboard according to an embodiment of the present invention.

圖3B係依據本發明一實施例在安裝CPU時所示的虛實疊合影像的示意圖。FIG3B is a schematic diagram of a virtual-real superimposed image when installing a CPU according to an embodiment of the present invention.

圖3C係依據本發明一實施例在安裝主機板螺絲時所示的虛實疊合影像的示意圖。FIG3C is a schematic diagram of a virtual and real superimposed image when installing motherboard screws according to an embodiment of the present invention.

圖4係依據本發明另一實施例的包含多個相機的組裝指引系統。FIG4 is an assembly guidance system including multiple cameras according to another embodiment of the present invention.

以下在實施方式中詳細敘述本發明之詳細特徵以及優點，其內容足以使任何熟習相關技藝者了解本發明之技術內容並據以實施，且根據本說明書所揭露之內容、申請專利範圍及圖式，任何熟習相關技藝者可輕易地理解本發明相關之目的及優點。以下之實施例係進一步詳細說明本發明之觀點，但非以任何觀點限制本發明之範疇。The detailed features and advantages of the present invention are described in detail in the following implementation method. The content is sufficient for anyone familiar with the relevant technology to understand the technical content of the present invention and implement it accordingly. According to the content disclosed in this specification, the scope of the patent application and the drawings, anyone familiar with the relevant technology can easily understand the relevant purposes and advantages of the present invention. The following embodiments are to further illustrate the viewpoints of the present invention, but do not limit the scope of the present invention by any viewpoint.

請參考圖1，圖1為依據本發明一實施例所繪示的組裝指引系統的功能方塊圖。圖1所示的組裝指引系統1包含相機11、擴增實境(Augmented Reality，簡稱AR)眼鏡12以及運算裝置13。其中相機11連接於運算裝置13，且運算裝置13連接於擴增實境眼鏡12。特別來說，相機11、擴增實境眼鏡12以及運算裝置13可以透過5G通訊技術連接。Please refer to FIG. 1, which is a functional block diagram of an assembly guidance system according to an embodiment of the present invention. Theassembly guidance system 1 shown in FIG. 1 includes acamera 11, an augmented reality (AR)glasses 12, and acomputing device 13. Thecamera 11 is connected to thecomputing device 13, and thecomputing device 13 is connected to the augmentedreality glasses 12. In particular, thecamera 11, the augmentedreality glasses 12, and thecomputing device 13 can be connected via 5G communication technology.

相機11，例如數位相機、廣角相機、深度相機或其他相機，用於拍攝作業區域以產生實境影像，其中作業區域設置有待組裝物，例如為雙列直插式記憶體模組(Dual In-line Memory Module，DIMM)或其他經組裝而成的電子裝置。特別來說，實境影像可以為持續拍攝而產生，因此其會根據操作過程不斷地變動。擴增實境眼鏡12用於受控顯示虛實疊合影像。以上所述相機11以及擴增實境眼鏡12具體地說可以整合為具有相機設備的擴增實境眼鏡，但是本發明並不受限於前述。Thecamera 11, such as a digital camera, a wide-angle camera, a depth camera or other camera, is used to shoot a working area to generate a real image, wherein the working area is provided with an object to be assembled, such as a dual in-line memory module (DIMM) or other assembled electronic devices. In particular, the real image can be generated by continuous shooting, so it will change continuously according to the operation process. The augmentedreality glasses 12 are used for controlled display of virtual and real superimposed images. The above-mentionedcamera 11 and augmentedreality glasses 12 can be specifically integrated into augmented reality glasses with camera equipment, but the present invention is not limited to the above.

運算裝置13例如為伺服器或工業電腦。運算裝置13用於根據實境影像中對應於識別碼的圖案，從多個預存組裝流程中取得待組裝物的目標組裝流程，並根據目標組裝流程於實境影像疊合指引圖像以產生虛實疊合影像，在虛實疊合影像顯示後，判斷實境影像中是否存在符合目標組裝流程的目標子影像，並且在實境影像不存在符合的目標子影像時，通知擴增實境眼鏡12於虛實疊合影像上呈現警示訊息。其中指引圖像更以為文字或圖案。除了指引圖像外，運算裝置13更可以根據目標組裝流程產生指引語音訊息，並透過擴增實境眼鏡12輸出指引語音訊息。Thecomputing device 13 is, for example, a server or an industrial computer. Thecomputing device 13 is used to obtain the target assembly process of the object to be assembled from multiple pre-stored assembly processes according to the pattern corresponding to the identification code in the real image, and to superimpose the guide image on the real image according to the target assembly process to generate a virtual superimposed image. After the virtual superimposed image is displayed, it is determined whether there is a target sub-image that meets the target assembly process in the real image, and when there is no matching target sub-image in the real image, the augmentedreality glasses 12 are notified to present a warning message on the virtual superimposed image. The guide image is more preferably text or a pattern. In addition to the guide image, thecomputing device 13 can also generate a guide voice message according to the target assembly process, and output the guide voice message through the augmentedreality glasses 12.

進一步說明，運算裝置13可以包含標準作業程序(SOP)資料庫、零件資料庫、影像處理模組及人工智慧模組。標準作業程序(SOP)資料庫可以儲存多種類型或/及型號的待組裝物各自的組裝流程(即前述預存組裝流程)、所述多種待組裝物具有的識別碼以及組裝流程與識別碼之間的對應關係。組裝流程可以包含一或多個組裝步驟，每個步驟可以包含需組裝之零件資訊(名稱、編號等)、零件組裝位置及組裝動作中的一或多個。零件資料庫可以儲存多種類型或/及型號的零件的圖像(平面或/及立體)、所述多種零件的名稱、編號等零件資訊以及圖像與零件資訊之間的對應關係。To further explain, thecomputing device 13 may include a standard operating procedure (SOP) database, a parts database, an image processing module, and an artificial intelligence module. The standard operating procedure (SOP) database may store the assembly processes of various types and/or models of objects to be assembled (i.e., the aforementioned pre-stored assembly processes), the identification codes of the various objects to be assembled, and the correspondence between the assembly processes and the identification codes. The assembly process may include one or more assembly steps, each of which may include information about the parts to be assembled (name, number, etc.), the assembly position of the parts, and one or more of the assembly actions. The parts database may store images (plane or/and three-dimensional) of various types and/or models of parts, part information such as the names and numbers of the various parts, and the correspondence between the images and the part information.

影像處理模組可以藉由影像辨識影像中待組裝物的識別碼，根據識別碼從SOP資料庫取得對應的目標組裝流程，並根據目標組裝流程於實境影像疊合指引圖像以產生虛實疊合影像。於一實施態樣中，影像處理模組藉由影像辨識實境影像中對應於待組裝物的區域，以及根據目標組裝流程所包含的零件組裝位置以指引圖像標記區域中對應於組裝位置的子區域來產生虛實疊合影像，所述指引圖像例如為方形或圓形線條。於另一實施態樣中，影像處理模組可對實境影像進行辨識，並根據目標組裝流程中的零件資訊從零件資料庫中選擇相對應元件的虛擬圖像，並根據目標組裝流程所包含的零件組裝位置，將元件的虛擬圖像作為指引圖像疊合於實境影像以產生虛實疊合影像。於又一實施態樣中，影像處理模組根據目標組裝流程中的組裝動作產生或從其他資料庫取得手部形狀的外框圖像，將手形外框圖像作為指引圖像疊合於實境影像以產生虛實疊合影像。The image processing module can obtain the corresponding target assembly process from the SOP database according to the identification code by image recognition of the identification code of the object to be assembled in the image, and superimpose the guide image on the real image according to the target assembly process to generate a virtual superimposed image. In one embodiment, the image processing module generates a virtual superimposed image by image recognition of the area corresponding to the object to be assembled in the real image, and by marking the sub-area corresponding to the assembly position in the guide image according to the assembly position of the parts included in the target assembly process, wherein the guide image is, for example, a square or circular line. In another embodiment, the image processing module can identify the real image, select the virtual image of the corresponding component from the parts database according to the part information in the target assembly process, and superimpose the virtual image of the component on the real image as a guide image according to the assembly position of the parts included in the target assembly process to generate a virtual superimposed image. In another embodiment, the image processing module generates a hand-shaped frame image according to the assembly action in the target assembly process or obtains it from other databases, and superimposes the hand-shaped frame image as a guide image on the real image to generate a virtual superimposed image.

人工智慧模組可以包含圖像識別的卷積神經網絡模型或/及姿勢偵測模型例如媒體管(Mediapipe)模型。運算裝置13藉由人工智慧模組中的多種模型可以判斷實際影像是否包含符合指引圖像的對應圖像來判斷實境影像中是否存在目標組裝流程的目標子影像。舉例來說，當指引圖像為框選元件組裝區或位於組裝區的虛擬元件影像時，指引圖像的對應圖像為實際元件被放置在正確區域的實境影像；而當指引圖像為雙手圖像時，指引圖像的對應圖像便是操作人員雙手擺出對應指引圖像的位置的實境影像。然而目標組裝過程中的子影像並不限於上述，而更可以是利用動作偵測的演算法判斷實境影像中是否具有目標組裝流程所需的動作影像。The artificial intelligence module may include a convolutional neural network model for image recognition and/or a posture detection model such as a Mediapipe model. Thecomputing device 13 can determine whether the actual image includes a corresponding image that matches the guide image by using the various models in the artificial intelligence module to determine whether there is a target sub-image of the target assembly process in the actual image. For example, when the guide image is a framed component assembly area or a virtual component image located in the assembly area, the corresponding image of the guide image is a real image of the actual component being placed in the correct area; and when the guide image is a two-hand image, the corresponding image of the guide image is a real image of the operator's hands placed in a position corresponding to the guide image. However, the sub-images in the target assembly process are not limited to the above, but can also use motion detection algorithms to determine whether the real-world image contains the motion images required for the target assembly process.

於目標組裝流程包含多個步驟例如第一步驟以及第二步驟的實施例中，第一步驟對應於一指引圖像，而第二步驟對應於另一指引圖像。於此實施例中，運算裝置13更在實境影像存在目標子影像時，移除虛實疊合影像中對應於第一步驟的指引圖像，並根據第二步驟於實境影像疊合對應於第二步驟的指引圖像。具體而言，所述指引圖像對應於組裝流程的一個步驟而另一指引圖像對應於組裝流程的下一步驟。當操作者執行到對應所述指引圖像的步驟時，所述指引圖像將於擴增實境眼鏡12上顯示，使操作者根據指引圖像執行步驟。一旦操作者執行完步驟，運算裝置13便可透過相機11確認實境影像是否存在目標子影像，若是符合便會移除所述指引圖像移除，並將對應於下一步驟的另一指引圖像顯示於擴增實境眼鏡12上，反之則會在擴增實境眼鏡12上顯示警示訊息。In an embodiment where the target assembly process includes multiple steps, such as a first step and a second step, the first step corresponds to a guide image, and the second step corresponds to another guide image. In this embodiment, when the target sub-image exists in the real image, thecomputing device 13 removes the guide image corresponding to the first step in the virtual-real superimposed image, and superimposes the guide image corresponding to the second step on the real image according to the second step. Specifically, the guide image corresponds to a step of the assembly process, and the other guide image corresponds to the next step of the assembly process. When the operator executes the step corresponding to the guide image, the guide image will be displayed on theaugmented reality glasses 12, so that the operator executes the step according to the guide image. Once the operator completes the steps, thecomputing device 13 can confirm whether the target sub-image exists in the real image through thecamera 11. If it does, the guidance image will be removed and another guidance image corresponding to the next step will be displayed on theaugmented reality glasses 12. Otherwise, a warning message will be displayed on theaugmented reality glasses 12.

請參考圖1及圖2，圖2係依據本發明一實施例所繪示的組裝指引方法的方法流程圖。如圖2所示，組裝指引方法包含步驟S1到步驟S7，所述方法適用於圖1所示的組裝指引系統1，以下示例性地以圖1所示的組裝指引系統1之運作來說明組裝指引方法。Please refer to Figures 1 and 2. Figure 2 is a method flow chart of an assembly guidance method according to an embodiment of the present invention. As shown in Figure 2, the assembly guidance method includes steps S1 to S7. The method is applicable to theassembly guidance system 1 shown in Figure 1. The following is an example of the operation of theassembly guidance system 1 shown in Figure 1 to illustrate the assembly guidance method.

於步驟S1中，相機11拍攝作業區域以產生實境影像，其中作業區域設置有待組裝物，且待組裝物上標示有識別碼。於步驟S2中，運算裝置13根據實境影像中對應於識別碼的圖案，從多個預存組裝流程中取得待組裝物的目標組裝流程，根據目標組裝流程中的步驟於實境影像疊合指引圖像以產生虛實疊合影像。於步驟S3中，擴增實境眼鏡12顯示虛實疊合影像。於步驟S4中，運算裝置13判斷實境影像中是否存在符合目標組裝流程的目標子影像。進一步來說，步驟S4的實施方式可包含判斷實境影像是否包含符合指引圖像的對應圖像。舉例來說，當指引圖像為零件圖像時，運算裝置13藉由影像辨識判斷實境影像中是否存在形狀及位置皆同於零件圖像的目標子影像。當步驟S4判斷結果為否時，運算裝置13通知擴增實境眼鏡12於虛實疊合影像上呈現警示訊息，即步驟S5。當步驟S4的判斷結果為是時，運算裝置13判斷目標組裝流程中的步驟是否已執行完，即步驟S6。當步驟S6的判斷結果為否時，運算裝置13移除虛實疊合影像中的指引圖像，並根據目標組裝流程中的下個步驟疊合另一指引圖像以更新虛實疊合影像，接著再次執行步驟S4所述之內容。當步驟S6的判斷結果為是時，方法結束。In step S1, thecamera 11 photographs the working area to generate a real image, wherein the working area is provided with an object to be assembled and an identification code is marked on the object to be assembled. In step S2, thecomputing device 13 obtains the target assembly process of the object to be assembled from multiple pre-stored assembly processes according to the pattern corresponding to the identification code in the real image, and superimposes the guide image on the real image according to the steps in the target assembly process to generate a virtual superimposed image. In step S3, theaugmented reality glasses 12 display the virtual superimposed image. In step S4, thecomputing device 13 determines whether there is a target sub-image that meets the target assembly process in the real image. Furthermore, the implementation method of step S4 may include determining whether the real image contains a corresponding image that matches the guide image. For example, when the guide image is a part image, thecomputing device 13 determines whether there is a target sub-image in the real image that is the same in shape and position as the part image by image recognition. When the determination result of step S4 is no, thecomputing device 13 notifies the augmentedreality glasses 12 to present a warning message on the virtual-real superimposed image, i.e., step S5. When the determination result of step S4 is yes, thecomputing device 13 determines whether the steps in the target assembly process have been completed, i.e., step S6. When the judgment result of step S6 is no, thecomputing device 13 removes the guide image in the virtual-real superimposed image, and superimposes another guide image according to the next step in the target assembly process to update the virtual-real superimposed image, and then executes the content described in step S4 again. When the judgment result of step S6 is yes, the method ends.

如前所述，目標組裝流程可以包含一或多個步驟。於目標組裝流程僅包含一個步驟的實施例中，組裝指引方法可僅包含上述步驟S1、S2、S3、S4及S5。As mentioned above, the target assembly process may include one or more steps. In an embodiment where the target assembly process includes only one step, the assembly guidance method may only include the above steps S1, S2, S3, S4 and S5.

請一併參考圖1、圖2以及圖3A到圖3C，圖3A到圖3C係依據本發明一實施例所示的虛實疊合影像。Please refer to Figures 1, 2, and 3A to 3C. Figures 3A to 3C are virtual and real superimposed images shown in an embodiment of the present invention.

圖3A中所示影像為安裝主機板的虛實疊合影像，可以是經由相機11透過拍攝待組裝物的實境影像R1傳送到運算裝置13(步驟S1)、運算裝置13根據實境影像中的識別碼取得目標組裝流程並根據目標組裝流程中的第一個步驟(例如主機板安裝)，將第一指引圖像I1與實境影像R1進行疊合(步驟S2)而產生，其中第一指引圖像I1標示主機板的組裝位置。The image shown in FIG. 3A is a virtual superimposed image of the motherboard installation, which can be generated by thecamera 11 shooting the real image R1 of the object to be assembled and transmitting it to the computing device 13 (step S1), and thecomputing device 13 obtains the target assembly process according to the identification code in the real image and superimposes the first guide image I1 with the real image R1 according to the first step in the target assembly process (such as motherboard installation) (step S2), wherein the first guide image I1 indicates the assembly position of the motherboard.

當圖2中所示步驟S4的判斷結果為是，而步驟S6的判斷結果為否時，組裝指引系統1的運算裝置13便會執行移除虛實疊合影像中的第一指引圖像I1，並根據目標組裝流程中的下個步驟(例如中央處理器安裝)疊合第二指引圖像I2以更新虛實疊合影(步驟S7)，即將第二指引圖像I2疊加於實境影像R1，得到圖3B。When the judgment result of step S4 shown in FIG. 2 is yes and the judgment result of step S6 is no, thecomputing device 13 of theassembly guidance system 1 will remove the first guidance image I1 in the virtual superimposed image, and superimpose the second guidance image I2 according to the next step in the target assembly process (such as central processing unit installation) to update the virtual superimposed image (step S7), that is, superimpose the second guidance image I2 on the real image R1 to obtain FIG. 3B.

重複上述步驟S4、S6及S7，可由圖3B前往下一步得到對應的虛實疊合影像圖3C，然而圖3C的第三指引圖像I3相較於圖3A的第一指引圖像I1以及圖3B的第二指引圖像I2，其可以同時框選多個需要被安裝的相同零件組裝位置。圖3C所對應之組裝步驟例如為主機板螺絲安裝。Repeat the above steps S4, S6 and S7, and go to the next step from Figure 3B to obtain the corresponding virtual-real superimposed image Figure 3C. However, compared with the first guide image I1 of Figure 3A and the second guide image I2 of Figure 3B, the third guide image I3 of Figure 3C can simultaneously select multiple assembly positions of the same parts that need to be installed. The assembly step corresponding to Figure 3C is, for example, the motherboard screw installation.

於此要特別說明的是，圖3A~3C示例性地呈現俯視角，然於相機設置於擴增實境眼鏡的實施例中，虛實疊合影像可依配戴擴增實境眼鏡的使用者的視角呈現。另外，圖3A~3C示例性地呈現指引圖像為框選特定的零件組裝位置，然於其他實施例中，指引圖像可為手勢圖形或/及虛擬零件圖像等多種不同形式。It should be noted that FIGS. 3A to 3C exemplarily present a top-down view, but in an embodiment where the camera is disposed in an augmented reality glasses, the virtual and real superimposed images can be presented according to the perspective of the user wearing the augmented reality glasses. In addition, FIGS. 3A to 3C exemplarily present a guide image as a frame to select a specific part assembly position, but in other embodiments, the guide image can be a hand gesture image or/and a virtual part image and other different forms.

請參考，圖4為包含多個相機的組裝指引系統。除了圖1所示的組裝指引系統1所包含的裝置(相機11後稱為第一相機11)，圖4所示的組裝指引系統1’更包含第二相機14，其餘裝置的連接關係及功能大致如圖1實施例所述。Please refer to FIG. 4 for an assembly guidance system including multiple cameras. In addition to the devices included in theassembly guidance system 1 shown in FIG. 1 (camera 11 is hereinafter referred to as the first camera 11), the assembly guidance system 1' shown in FIG. 4 further includes asecond camera 14, and the connection relationship and functions of the remaining devices are roughly as described in the embodiment of FIG. 1.

第二相機14，例如數位相機、廣角相機、深度相機或其他相機，連接於運算裝置13。第二相機14用於以異於第一相機11的拍攝角度拍攝作業區域產生另一實境影像。於此實施例中，運算裝置13除了對第一相機11所取得的實境影像進行如前所述的判斷，更會判斷第二相機14所取得的實境影像是否存在符合目標組裝流程的另一目標子影像，且當第二相機14所取得的實境影像中不存在所述另一目標子影像時，通知擴增實境眼鏡12於虛實疊合影像上呈現警示訊息，此警示訊息可相同或不相同於前述警示訊息。運算裝置13對於第二相機14取得的實境影像所進行的判斷同理於前述對第一相機11所取得的實境影像所進行的判斷，於此不予贅述。Thesecond camera 14, such as a digital camera, a wide-angle camera, a depth camera or other cameras, is connected to thecomputing device 13. Thesecond camera 14 is used to shoot the working area at a shooting angle different from that of thefirst camera 11 to generate another real image. In this embodiment, in addition to performing the above-mentioned judgment on the real image obtained by thefirst camera 11, thecomputing device 13 further judges whether there is another target sub-image that meets the target assembly process in the real image obtained by thesecond camera 14, and when the other target sub-image does not exist in the real image obtained by thesecond camera 14, theaugmented reality glasses 12 are notified to present a warning message on the virtual-real superimposed image, and this warning message may be the same as or different from the above-mentioned warning message. The judgment made by thecomputing device 13 on the real-life image obtained by thesecond camera 14 is similar to the judgment made on the real-life image obtained by thefirst camera 11, and will not be elaborated here.

在本實施例中，本發明之伺服器係可為人工智慧(Artificial Intelligence，簡稱AI)運算、邊緣運算(edge computing)、雲端伺服器。In this embodiment, the server of the present invention can be artificial intelligence (AI) computing, edge computing, or cloud server.

綜上所述，本發明一或多個實施例的組裝指引系統及其方法可以藉由有效的指引作業員使其節省學習時間以及提高作業正確率。In summary, the assembly guidance system and method of one or more embodiments of the present invention can save learning time and improve the accuracy of operations by effectively guiding operators.

雖然本發明以前述之實施例揭露如上，然其並非用以限定本發明。在不脫離本發明之精神和範圍內，所為之更動與潤飾，均屬本發明之專利保護範圍。關於本發明所界定之保護範圍請參考所附之申請專利範圍。Although the present invention is disclosed as above by the aforementioned embodiments, it is not intended to limit the present invention. Any changes and modifications made within the spirit and scope of the present invention are within the scope of patent protection of the present invention. Please refer to the attached patent application for the scope of protection defined by the present invention.

S1~S7:方法步驟流程S1~S7: Method steps flow

Claims (10)

Translated fromChinese

一種組裝指引系統，包含： 一相機，用於拍攝一作業區域以產生一實境影像，其中該作業區域設置有一待組裝物，且該待組裝物上標示有一識別碼； 一擴增實境眼鏡，用於受控以顯示一虛實疊合影像；以及 一運算裝置，連接於該相機及該擴增實境眼鏡，且用於根據該實境影像中對應於該識別碼的圖案，從多個預存組裝流程中取得該待組裝物的一目標組裝流程，根據該目標組裝流程於該實境影像疊合一指引圖像以產生該虛實疊合影像，在該虛實疊合影像顯示後，判斷該實境影像中是否存在符合該目標組裝流程的一目標子影像，且在該實境影像中不存在該目標子影像時，通知該擴增實境眼鏡於該虛實疊合影像上呈現一警示訊息。An assembly guidance system includes:a camera for photographing a work area to generate a real image, wherein the work area is provided with an object to be assembled and an identification code is marked on the object to be assembled;an augmented reality glasses for being controlled to display a virtual-real superimposed image; andA computing device is connected to the camera and the augmented reality glasses, and is used to obtain a target assembly process of the object to be assembled from a plurality of pre-stored assembly processes according to the pattern corresponding to the identification code in the real image, and to superimpose a guide image on the real image according to the target assembly process to generate the virtual superimposed image. After the virtual superimposed image is displayed, it is determined whether there is a target sub-image in the real image that meets the target assembly process, and when the target sub-image does not exist in the real image, the augmented reality glasses are notified to present a warning message on the virtual superimposed image.如請求項1所述的組裝指引系統，其中該目標組裝流程包含一第一步驟及一第二步驟，該指引圖像對應於該第一步驟，該運算裝置更用於在該實境影像中存在該目標子影像時，移除該虛實疊合影像中的該指引圖像，並根據該第二步驟於該實境影像疊合另一指引圖像以更新該虛實疊合影像。An assembly guidance system as described in claim 1, wherein the target assembly process includes a first step and a second step, the guidance image corresponds to the first step, and the computing device is further used to remove the guidance image in the virtual overlay image when the target sub-image exists in the real image, and to overlay another guidance image on the real image according to the second step to update the virtual overlay image.如請求項1所述的組裝指引系統，其中該運算裝置係藉由判斷該實境影像是否包含符合該指引圖像的對應圖像來判斷該實境影像中是否存在符合該目標組裝流程的該目標子影像。An assembly guidance system as described in claim 1, wherein the computing device determines whether the target sub-image that meets the target assembly process exists in the real image by determining whether the real image includes a corresponding image that meets the guidance image.如請求項1所述的組裝指引系統，其中該目標組裝流程包含一零件及該零件對應於該待組裝物的一組裝位置，且該運算裝置係藉由辨識該實境影像中對應於該待組裝物的區域，以及以該指引圖像標記該區域中對應於該組裝位置的子區域來產生該虛實疊合影像。An assembly guidance system as described in claim 1, wherein the target assembly process includes a part and an assembly position of the part corresponding to the object to be assembled, and the computing device generates the virtual-real overlay image by identifying an area in the real image corresponding to the object to be assembled, and marking a sub-area of the area corresponding to the assembly position with the guidance image.如請求項1所述的組裝指引系統，其中該相機係一第一相機且設置於該擴增實境眼鏡，該組裝指引系統更包含： 一第二相機，連接於該運算裝置，用於以異於該第一相機的拍攝角度拍攝該作業區域以產生另一實境影像； 其中該運算裝置更用於判斷該另一實境影像是否存在符合該目標組裝流程的另一目標子影像，且在該另一實境影像中不存在該另一目標子影像時，通知該擴增實境眼鏡於該虛實疊合影像上呈現另一警示訊息。As described in claim 1, the camera is a first camera and is disposed in the augmented reality glasses, and the assembly guidance system further comprises:A second camera, connected to the computing device, for photographing the working area at a shooting angle different from that of the first camera to generate another real image;The computing device is further used to determine whether the other real image contains another target sub-image that meets the target assembly process, and when the other target sub-image does not exist in the other real image, notify the augmented reality glasses to present another warning message on the virtual superimposed image.如請求項1所述的組裝指引系統，其中該擴增實境眼鏡及該運算裝置透過5G通訊技術連接。An assembly guidance system as described in claim 1, wherein the augmented reality glasses and the computing device are connected via 5G communication technology.一種組裝指引方法，適用於一相機、一擴增實境眼鏡以及一運算裝置，其中該運算裝置連接於該相機及該擴增實境眼鏡，且該組裝指引方法包含： 以該相機拍攝一作業區域以產生一實境影像，其中該作業區域設置有一待組裝物，且該待組裝物上標示有一識別碼； 以該運算裝置根據該實境影像中對應於該識別碼的圖案，從多個預存組裝流程中取得該待組裝物的一目標組裝流程，根據該目標組裝流程於該實境影像疊合一指引圖像以產生一虛實疊合影像； 以該擴增實境眼鏡顯示該虛實疊合影像； 在該擴增實境眼鏡顯示該虛實疊合影像之後，以該運算裝置判斷該實境影像中是否存在符合該目標組裝流程的一目標子影像；以及 當該實境影像中不存在該目標子影像時，以該運算裝置通知該擴增實境眼鏡於該虛實疊合影像上呈現一警示訊息。An assembly guidance method is applicable to a camera, an augmented reality glasses and a computing device, wherein the computing device is connected to the camera and the augmented reality glasses, and the assembly guidance method comprises: Using the camera to shoot a work area to generate a real image, wherein the work area is provided with an object to be assembled, and an identification code is marked on the object to be assembled; Using the computing device to obtain a target assembly process of the object to be assembled from a plurality of pre-stored assembly processes according to a pattern corresponding to the identification code in the real image, and superimposing a guidance image on the real image according to the target assembly process to generate a virtual superimposed image; Using the augmented reality glasses to display the virtual superimposed image; After the augmented reality glasses display the virtual superimposed image, the computing device determines whether there is a target sub-image in the real image that meets the target assembly process; andWhen the target sub-image does not exist in the real image, the computing device notifies the augmented reality glasses to present a warning message on the virtual superimposed image.如請求項7所述的組裝指引方法，其中該目標組裝流程包含一第一步驟及一第二步驟，該指引圖像對應於該第一步驟，且該組裝指引方法更包含當該實境影像中存在該目標子影像時，以該運算裝置移除該虛實疊合影像中的該指引圖像，並根據該第二步驟於該實境影像疊合另一指引圖像以更新該虛實疊合影像。An assembly guidance method as described in claim 7, wherein the target assembly process includes a first step and a second step, the guidance image corresponds to the first step, and the assembly guidance method further includes removing the guidance image in the virtual overlay image by the computing device when the target sub-image exists in the real image, and superimposing another guidance image on the real image according to the second step to update the virtual overlay image.如請求項7所述的組裝指引方法，其中判斷該實境影像中是否存在符合該目標組裝流程的該目標子影像包含判斷該實境影像是否包含符合該指引圖像的對應圖像。In the assembly guidance method as described in claim 7, determining whether the target sub-image that meets the target assembly process exists in the real image includes determining whether the real image includes a corresponding image that meets the guidance image.如請求項7所述的組裝指引方法，其中該目標組裝流程包含一零件及該零件對應於該待組裝物的一組裝位置，且根據該目標組裝流程於該實境影像疊合該指引圖像以產生該虛實疊合影像包含： 辨識該實境影像中對應於該待組裝物的區域；以及 以該指引圖像標記該區域中對應於該組裝位置的子區域。The assembly guidance method as described in claim 7, wherein the target assembly process includes a part and an assembly position of the part corresponding to the object to be assembled, and superimposing the guidance image on the real image according to the target assembly process to generate the virtual superimposed image includes: Identifying the area corresponding to the object to be assembled in the real image; and Marking the sub-area of the area corresponding to the assembly position with the guidance image.

Images