CN114329221A - Commodity searching method, equipment and storage medium - Google Patents

Abstract

Translated fromChinese

本发明公开了一种商品搜索方法、设备及存储介质，方法包括：当处于拍摄界面时，在取景区域显示前景框，以便用户操作使得目标商品对应的图像全部位于所述前景框内；在未检测到对前景框之内的屏幕的触摸信号，并且检测到分割触发信号时，根据所述前景框确定前景提取算法所需的种子像素；根据所述种子像素和前景提取算法从当前取景区域对应的图像中提取得到目标商品对应的前景图像；将所述前景图像发送至服务器，以便服务器根据所述前景图像匹配相应的商品信息并将所述商品信息发送给终端。本发明中的前景提取(即图像分割)直接根据前景框来确定grabcut算法所需的背景种子，无需用户再额外操作，简化了分割前景的流程，同时提升了商品搜索结果的准确性。

The invention discloses a commodity search method, device and storage medium. The method includes: when in a shooting interface, displaying a foreground frame in a viewfinder area, so that the images corresponding to the target commodity are all located in the foreground frame by user operation; When a touch signal to the screen within the foreground frame is detected, and a segmentation trigger signal is detected, the seed pixels required by the foreground extraction algorithm are determined according to the foreground frame; the corresponding seed pixels from the current viewing area are determined according to the seed pixels and the foreground extraction algorithm. The foreground image corresponding to the target product is extracted from the image of the target product; the foreground image is sent to the server, so that the server matches the corresponding product information according to the foreground image and sends the product information to the terminal. The foreground extraction (ie image segmentation) in the present invention directly determines the background seeds required by the grabcut algorithm according to the foreground frame, without any additional operation by the user, which simplifies the foreground segmentation process and improves the accuracy of commodity search results.

Description

Translated fromChinese

一种商品搜索方法、设备及存储介质A kind of commodity search method, device and storage medium

技术领域technical field

本发明涉及电子商务技术领域，特别涉及一种商品搜索方法、设备及存储介质。The invention relates to the technical field of electronic commerce, in particular to a commodity searching method, device and storage medium.

背景技术Background technique

随着移动互联网商城的兴起，怎样让消费者在最短时间内搜索到自己喜欢的商品，商家各显所能。早期的移动互联网商城的商品搜索引擎一般采用文字搜索的方式实现。如公开号为”CN101206672A”，发明名称为”商品搜索无结果智能处理系统及方法”的中国专利(下称为文献1)，文献1公开了一种对用户输入的文字进行分词处理后返回检索结果的技术方案。With the rise of mobile Internet shopping malls, how to let consumers search for their favorite products in the shortest time, merchants show their abilities. The commodity search engine of the early mobile Internet mall was generally implemented by text search. For example, the publication number is "CN101206672A", and the invention name is the Chinese patent (hereinafter referred to as Document 1) with the title of "Intelligent Processing System and Method for No Results in Commodity Search". Document 1 discloses a method that performs word segmentation on the text input by the user and returns to search. The resulting technical solution.

随着技术发展，很多电商也提供了通过图像进行搜索的方式，如阿里巴巴、京东、拼多多等APP均能够实现图像搜索，图像搜索相对于文字搜索更高效便捷，深受用户喜欢。如公开号为”CN101216841A”，发明名称为”交互式图像搜索系统和方法”的中国专利(下称为文献2)，文献2公开了一种能够根据用户输入的照片返回匹配相关商品的技术方案。With the development of technology, many e-commerce companies also provide a way to search through images. For example, Alibaba, JD.com, Pinduoduo and other apps can realize image search. Compared with text search, image search is more efficient and convenient, and is very popular among users. For example, the publication number is "CN101216841A", and the invention name is "interactive image search system and method" Chinese patent (hereinafter referred to as document 2), document 2 discloses a technical solution that can return matching related products according to photos input by users .

相关技术中，在将上述文献2的图像搜索方法应用于商品搜索时，大致的商品搜索流程如下：电商APP在实时调用摄像头进行商品拍摄搜索时，需要先拍摄照片，然后根据用户对屏幕的触摸操作来确定拍摄得到的照片上的感兴趣区域和不感兴趣区域(即种子)，再根据这些种子和相应的图像分割算法确定前景区域，最后根据前景区域的图像特征进行相应的商品匹配，向用户返回对应的商品搜索结果。如1-A所示，当用户需要同时搜索多个商品时，可以将多个商品摆放在同一区域内，电商APP可以拍摄得到一张整体照片，可以在拍摄好的整体照片中通过多次触摸操作选定与各商品对应多组感兴趣区域和不感兴趣区域，从而得到多个商品的搜索结果。In the related art, when the image search method of the above-mentioned document 2 is applied to commodity search, the general commodity search process is as follows: when the e-commerce APP calls the camera in real time for commodity photography and search, it needs to take a photo first, and then according to the user's display of the screen. Touch operation to determine the area of interest and the area of non-interest (that is, the seeds) on the photographed photos, and then determine the foreground area according to these seeds and the corresponding image segmentation algorithm, and finally carry out the corresponding product matching according to the image characteristics of the foreground area. The user returns the corresponding product search result. As shown in 1-A, when the user needs to search for multiple products at the same time, they can place multiple products in the same area, and the e-commerce APP can take an overall photo, which can be used in the overall photo. The second touch operation selects a plurality of groups of regions of interest and regions of no interest corresponding to each commodity, so as to obtain search results of multiple commodities.

然而，上述的相关技术存在以下不足：电商APP在调用摄像头实时拍摄整体照片时，一般只能够实现对部分区域的对焦，对于不在对焦区域的商品，后续用户从整体照片中通过触摸操作方式选定其对应的种子像素用于前景提取算法进行图像分割时，由于该商品对应的图像不够清晰，导致图像分割的效果差，造成商品搜索结果的不准确(如图1-B所示，商品实物为无线耳机盒，而返回的搜索结果却是暗合页)。However, the above-mentioned related technologies have the following deficiencies: when an e-commerce APP calls the camera to take an overall photo in real time, it can generally only focus on a part of the area. For products that are not in the focus area, subsequent users can select from the overall photo by touch operation. When the corresponding seed pixels are used for image segmentation by the foreground extraction algorithm, the image corresponding to the product is not clear enough, resulting in poor image segmentation and inaccurate product search results (as shown in Figure 1-B, the actual product is a wireless headphone box, and the search results returned are dark hinges).

发明内容SUMMARY OF THE INVENTION

本发明的目的在于至少解决现有技术中存在的技术问题之一，提供一种商品搜索方法、设备及存储介质，能够在电商APP进行拍摄搜索商品时，得到用户关注商品的清晰图像，提升图像分割的效果，从而提升商品搜索结果的准确率。The purpose of the present invention is to solve at least one of the technical problems existing in the prior art, and to provide a product search method, device and storage medium, which can obtain a clear image of the product that the user pays attention to when the e-commerce APP is photographing and searching for the product, thereby improving the The effect of image segmentation, thereby improving the accuracy of product search results.

为了实现上述发明目的，提供如下的技术方案：In order to achieve the above-mentioned purpose of the invention, the following technical solutions are provided:

第一方面，本发明提供一种商品搜索方法，所述方法包括：In a first aspect, the present invention provides a commodity search method, the method comprising:

当处于拍摄界面时，在取景区域显示前景框，以便用户操作使得目标商品对应的图像全部位于所述前景框内；When in the shooting interface, a foreground frame is displayed in the viewing area, so that the user operates so that the images corresponding to the target product are all located in the foreground frame;

在未检测到对前景框之内的屏幕的触摸信号，并且检测到分割触发信号时，根据所述前景框确定前景提取算法所需的种子像素；When the touch signal to the screen within the foreground frame is not detected and the segmentation trigger signal is detected, determining the seed pixels required by the foreground extraction algorithm according to the foreground frame;

根据所述种子像素和前景提取算法从当前取景区域对应的图像中提取得到目标商品对应的前景图像；The foreground image corresponding to the target commodity is extracted from the image corresponding to the current viewing area according to the seed pixel and the foreground extraction algorithm;

将所述前景图像发送至服务器，以便服务器根据所述前景图像匹配相应的商品信息并将所述商品信息发送给终端。The foreground image is sent to the server, so that the server matches the corresponding commodity information according to the foreground image and sends the commodity information to the terminal.

第二方面，本发明提供一种电子设备，包括：存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现如本发明第一方面实施例中任一项所述的商品搜索方法。In a second aspect, the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and running on the processor, the processor implements the first aspect of the present invention when the processor executes the program The commodity search method described in any one of the embodiments.

第三方面，本发明提供一种计算机可读存储介质，所述计算机可读存储介质存储有计算机可执行指令，所述计算机可执行指令用于使计算机执行如本发明第一方面中任一项所述的商品搜索方法。In a third aspect, the present invention provides a computer-readable storage medium, where the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to cause a computer to perform any one of the first aspects of the present invention The described commodity search method.

相较于现有技术，本发明至少具有以下的有益效果：Compared with the prior art, the present invention has at least the following beneficial effects:

1.本发明中的前景提取(即图像分割)直接根据前景框来确定grabcut算法所需的背景种子，无需用户再额外操作，简化了分割前景的流程。1. The foreground extraction (ie, image segmentation) in the present invention directly determines the background seed required by the grabcut algorithm according to the foreground frame, without the need for additional operations by the user, which simplifies the process of segmenting the foreground.

2.在提取完当前目标商品对应的前景图像之后，由于终端还处于拍摄界面，因此，可以继续调整拍照角度的操作和对焦操作进而对其他的目标商品继续进行前景图像的提取，保证每个目标商品都能够拍摄得到清晰的图像，提高目标商品对应图像的分割效果，从而提升商品搜索结果的准确性。2. After extracting the foreground image corresponding to the current target product, since the terminal is still in the shooting interface, you can continue to adjust the camera angle and focus operation, and then continue to extract the foreground image for other target products to ensure that each target All products can be photographed to obtain clear images, which improves the segmentation effect of the corresponding images of the target products, thereby improving the accuracy of product search results.

本发明的附加方面和优点将在下面的描述中部分给出，部分将从下面的描述中变得明显，或通过本发明的实践了解到。Additional aspects and advantages of the present invention will be set forth, in part, from the following description, and in part will be apparent from the following description, or may be learned by practice of the invention.

附图说明Description of drawings

下面结合附图和实施例对本发明进一步地说明；Below in conjunction with accompanying drawing and embodiment, the present invention is further described;

图1-A为现有技术中电商APP进行拍摄搜索商品时的界面示意图。FIG. 1-A is a schematic diagram of an interface of an e-commerce APP in the prior art when photographing and searching for commodities.

图1-B为现有技术中电商APP进行拍摄搜索商品时的界面示意图。FIG. 1-B is a schematic diagram of an interface of an e-commerce APP in the prior art when photographing and searching for commodities.

图2为一个实施例中商品搜索方法的应用环境图。FIG. 2 is an application environment diagram of the commodity search method in one embodiment.

图3-A为一个实施例中切换到拍摄界面时初始状态的示意图。FIG. 3-A is a schematic diagram of an initial state when switching to a shooting interface in an embodiment.

图3-B为一个实施例中在拍摄界面中调节好焦距时的示意图。FIG. 3-B is a schematic diagram of adjusting the focal length in the shooting interface in one embodiment.

图4-A为一个实施例中用户在拍摄界面通过拇指和食指调节焦距时初始状态的示意图。FIG. 4-A is a schematic diagram of an initial state when the user adjusts the focus through the thumb and forefinger on the shooting interface in one embodiment.

图4-B为一个实施例中用户在拍摄界面通过拇指和食指调节焦距时最终状态的示意图。FIG. 4-B is a schematic diagram of the final state when the user adjusts the focus through the thumb and the index finger on the shooting interface in one embodiment.

图5为一个实施例中在拍摄界面通过面积调整控件来调整前景框面积的示意图。FIG. 5 is a schematic diagram of adjusting the area of the foreground frame through the area adjustment control on the photographing interface in one embodiment.

图6为一个实施例中在拍摄界面通过形状调整控件来调整前景框形状的示意图。FIG. 6 is a schematic diagram of adjusting the shape of the foreground frame through the shape adjustment control on the shooting interface in one embodiment.

图7为一个实施例中商品搜索方法的流程示意图。FIG. 7 is a schematic flowchart of a method for searching for commodities in one embodiment.

图8为一个实施例中商品搜索方法的流程示意图。FIG. 8 is a schematic flowchart of a method for searching for commodities in one embodiment.

图9为一个实施例中商品搜索方法的流程示意图。FIG. 9 is a schematic flowchart of a method for searching for commodities in one embodiment.

图10为一个实施例中商品搜索方法的流程示意图。FIG. 10 is a schematic flowchart of a method for searching for commodities in one embodiment.

图11为一个实施例中计算机设备的结构框图。FIG. 11 is a structural block diagram of a computer device in one embodiment.

附图标记：Reference number:

100、服务器；200、终端；201、拍摄界面；210、取景区域；211、前景框；220、按钮区域；221、形状调整控件；222、尺寸调整控件；223、分割控件。100, server; 200, terminal; 201, shooting interface; 210, viewfinder area; 211, foreground frame; 220, button area; 221, shape adjustment control; 222, size adjustment control; 223, segmentation control.

具体实施方式Detailed ways

本部分将详细描述本发明的具体实施例，本发明之较佳实施例在附图中示出，附图的作用在于用图形补充说明书文字部分的描述，使人能够直观地、形象地理解本发明的每个技术特征和整体技术方案，但其不能理解为对本发明保护范围的限制。This part will describe the specific embodiments of the present invention in detail, and the preferred embodiments of the present invention are shown in the accompanying drawings. Each technical feature and overall technical solution of the invention should not be construed as limiting the protection scope of the invention.

图2为一个实施例中商品搜索方法的应用环境图。参照图1，该商品搜索方法应用于商品搜索系统。该商品搜索系统包括终端200和服务器100。终端200和服务器100通过网络连接。终端200具体可以是台式终端200或移动终端200，移动终端200具体可以手机、平板电脑、笔记本电脑等中的至少一种。服务器100可以用独立的服务器100或者是多个服务器100组成的服务器集群来实现。FIG. 2 is an application environment diagram of the commodity search method in one embodiment. Referring to FIG. 1, the commodity search method is applied to a commodity search system. The commodity search system includes aterminal 200 and aserver 100 . Theterminal 200 and theserver 100 are connected through a network. Theterminal 200 may specifically be adesktop terminal 200 or amobile terminal 200, and themobile terminal 200 may specifically be at least one of a mobile phone, a tablet computer, a notebook computer, and the like. Theserver 100 may be implemented by anindependent server 100 or a server cluster composed ofmultiple servers 100 .

在某些场景中，用户需要通过电商APP同时拍摄搜索多个商品并且希望这些商品的搜索结果出现在同一页面内方便查看，或者通过电商APP同时拍摄搜索多个同类别不同款式的商品并且希望这些商品的搜索结果出现在同一页面内方便比价。图1-B为现有技术中电商APP进行拍摄搜索商品时的界面示意图。在一个场景中，用户需要通过电商APP对图1-B中的三个目标商品进行搜索，以便于查看，从左至右三个目标商品分别是口香糖盒、无线耳机盒、一次性杯子。可以看到，电商APP在调用摄像头实时拍摄整体照片时，一般只能够实现对部分区域的对焦，对于不在对焦区域的商品，后续用户从整体照片中通过触摸操作方式选定其对应的种子像素用于前景提取算法进行图像分割时，由于该商品对应的图像不够清晰，导致图像分割的效果差，造成商品搜索结果的不准确(如图1-B所示，商品实物为无线耳机盒，而返回的搜索结果却是暗合页)。In some scenarios, users need to shoot and search for multiple products through the e-commerce APP at the same time and hope that the search results of these products will appear on the same page for easy viewing, or shoot and search for multiple products of the same category and different styles through the e-commerce APP at the same time. Hope that the search results of these products appear on the same page to facilitate price comparison. FIG. 1-B is a schematic diagram of an interface of an e-commerce APP in the prior art when photographing and searching for commodities. In one scenario, the user needs to search the three target products in Figure 1-B through the e-commerce APP for easy viewing. The three target products from left to right are chewing gum boxes, wireless earphone boxes, and disposable cups. It can be seen that when the e-commerce APP calls the camera to take the overall photo in real time, it can generally only focus on part of the area. For products that are not in the focus area, the subsequent user selects the corresponding seed pixel from the overall photo by touch operation. When using the foreground extraction algorithm for image segmentation, because the image corresponding to the product is not clear enough, the effect of image segmentation is poor, resulting in inaccurate product search results (as shown in Figure 1-B, the actual product is a wireless earphone box, and The search results returned are dark hinges).

而本发明人发现了存在上述问题的原因：现有技术中，之所以不在拍摄界面201分别对多个目标商品进行分割，是因为分割算法(即前景提取算法)需要通过交互操作(用户触摸屏幕)的方式提供种子像素，而在拍摄界面201中进行交互操作时又会触发相机程序对应的功能(例如在相机程序的拍摄界面201用手指滑动屏幕时会调整相机的焦距等)，即用于确定种子像素的交互动作会与拍照过程的功能存在干扰，因此现有技术都是先拍摄完整体照片，才允许用户在照片上交互操作确定种子像素。本发明人的贡献之一在于发现了这个问题存在的原因，并进一步通过在拍摄界面201上显示前景框211，提醒用户将目标商品对应的图像全部置于前景框211内实现了确定种子像素的交互操作，从而能够在拍摄界面201中通过调整拍照角度的操作和对焦操作将各目标商品的图像调整至最清晰的情况下再分别确定多个商品对应的种子像素，解决了现有技术中存在的技术问题。The inventors have found the reason for the above problem: in the prior art, the reason why theshooting interface 201 is not to segment multiple target commodities is because the segmentation algorithm (ie, the foreground extraction algorithm) needs to be performed through interactive operations (the user touches the screen ) method to provide seed pixels, and when performing interactive operations in theshooting interface 201, the corresponding function of the camera program will be triggered (for example, when theshooting interface 201 of the camera program slides the screen with a finger, the focal length of the camera will be adjusted, etc.), which is used for The interactive action of determining the seed pixels will interfere with the function of the photographing process. Therefore, in the prior art, a complete body photo is taken first, and then the user is allowed to interactively determine the seed pixels on the photo. One of the contributions of the present inventor is to discover the reason for this problem, and further by displaying theforeground frame 211 on theshooting interface 201 to remind the user to place all the images corresponding to the target product in theforeground frame 211 to realize the determination of seed pixels. Interactive operation, so that in theshooting interface 201, the image of each target product can be adjusted to the clearest through the operation of adjusting the photographing angle and the focusing operation, and then the seed pixels corresponding to multiple products can be determined respectively, which solves the problem of existing in the prior art. technical issues.

下面，将通过几个具体的实施例对本发明实施例提供的商品搜索方法进行详细介绍和说明。Hereinafter, the commodity search method provided by the embodiment of the present invention will be introduced and described in detail through several specific embodiments.

如图7所示，在一个实施例中，提供了一种商品搜索方法。本实施例主要以该方法应用于计算机设备来举例说明。该计算机设备具体可以是上述图1中的终端200。As shown in FIG. 7, in one embodiment, a method for searching for commodities is provided. This embodiment is mainly described by taking the method applied to a computer device as an example. Specifically, the computer device may be the terminal 200 in FIG. 1 above.

参照图7，该商品搜索方法具体包括如下步骤：7, the commodity search method specifically includes the following steps:

步骤S100，当终端200处于拍摄界面201时，终端200在取景区域210显示前景框211，以便用户操作使得目标商品对应的图像全部位于所述前景框211内。In step S100, when the terminal 200 is in theshooting interface 201, the terminal 200 displays aforeground frame 211 in theviewing area 210, so that the user operates so that all images corresponding to the target product are located in theforeground frame 211.

如图3-A所示，在一个场景中，用户在电商APP触发图像搜索功能，终端200切换至拍摄界面201。拍摄界面201包括两个区域，左边为取景区域210，用于显示相机的摄像头实时采集到的图像画面，右侧为按钮区域220，布置有多个用于触发相应功能的按钮，从上至下依次包括形状调整控件221、尺寸调整控件222、分割控件223。其中，在取景区域210显示有矩形的前景框211，可以理解的是，该前景框211在取景区域210中的位置是固定的，为了让用户在屏幕上容易看到前景框211的位置，可以将前景框211的四条边采用高亮或者显眼的颜色(例如红色、黄色等)的形式进行显示，以便用户在拍摄的时候通过调整相机摄像头的角度、位置或者调解相机的焦距来使得目标商品对应的图像全部位于所述前景框211内。如图3-A所示，此时最左侧为口香糖盒、中间为无线耳机盒二者对应的图像均全部位于前景框211之内，而一次性杯子对应的图像则只有左半部分位于前景框211之内。As shown in FIG. 3-A , in one scenario, the user triggers the image search function in the e-commerce APP, and the terminal 200 switches to theshooting interface 201 . Theshooting interface 201 includes two areas, the left side is theviewfinder area 210, which is used to display the real-time image captured by the camera of the camera, and the right side is thebutton area 220, which is arranged with a plurality of buttons for triggering corresponding functions, from top to bottom It includes ashape adjustment control 221 , asize adjustment control 222 , and adivision control 223 in sequence. Wherein, arectangular foreground frame 211 is displayed in theviewing area 210. It can be understood that the position of theforeground frame 211 in theviewing area 210 is fixed. The four sides of theforeground frame 211 are displayed in the form of highlighted or conspicuous colors (such as red, yellow, etc.), so that the user can adjust the angle and position of the camera or adjust the focal length of the camera to make the target product correspond to the camera when shooting. The images of all are located within theforeground frame 211 . As shown in Figure 3-A, the images corresponding to the chewing gum box on the far left and the wireless earphone box in the middle are all located in theforeground frame 211, while the image corresponding to the disposable cup only has the left half located in the foreground withinbox 211.

步骤S102，在未检测到对前景框211之内的屏幕的触摸信号，并且检测到分割触发信号时，终端200根据所述前景框211确定前景提取算法所需的种子像素。Step S102 , when no touch signal to the screen within theforeground frame 211 is detected and a segmentation trigger signal is detected, the terminal 200 determines the seed pixels required by the foreground extraction algorithm according to theforeground frame 211 .

在图3-A所示的情况下，相机对应的焦距很小，有可能导致无线耳机盒对应的图像边缘不清晰，不满足分割算法的要求，因此有必要调大焦距，使得无线耳机盒对应的图像不仅全部位于前景框211内，而且无线耳机盒对应的图像占前景框211面积的比值应该尽量大，即如图3-B所示。如图3-B所示，此时的前景框211位于目标商品对应的图像之外，即前景框211之外的图像均为所需分割对象的背景，此时可以很好的适用于grab cut算法。用户此时通过触摸分割控件223，触发生成分割触发信号，进而触发执行提取种子像素的步骤。具体的，如图3-B所示，检测到用户对所述分割控件223的触摸操作时，生成分割触发信号。In the case shown in Figure 3-A, the focal length corresponding to the camera is very small, which may cause the edge of the image corresponding to the wireless earphone box to be unclear, which does not meet the requirements of the segmentation algorithm. Therefore, it is necessary to increase the focal length so that the wireless earphone box corresponds to the Not only are all the images located in theforeground frame 211, but the ratio of the image corresponding to the wireless earphone box to the area of theforeground frame 211 should be as large as possible, as shown in Figure 3-B. As shown in Figure 3-B, theforeground frame 211 at this time is located outside the image corresponding to the target product, that is, the images outside theforeground frame 211 are the background of the object to be segmented, which can be well applied to grab cut at this time. algorithm. At this time, the user touches thesegmentation control 223 to trigger the generation of a segmentation trigger signal, thereby triggering the execution of the step of extracting the seed pixels. Specifically, as shown in FIG. 3-B, when a user's touch operation on thesegmentation control 223 is detected, a segmentation trigger signal is generated.

在一个实施例中，所述根据所述前景框211确定前景提取算法所需的种子像素，具体包括：In one embodiment, determining the seed pixels required by the foreground extraction algorithm according to theforeground frame 211 specifically includes:

步骤S1021，终端200获取当前取景区域210对应的图像；Step S1021, the terminal 200 acquires an image corresponding to thecurrent viewing area 210;

步骤S1022，终端200将当前取景区域210对应的图像上位于所述前景框211之外的部分作为前景提取算法所需的种子像素；其中，所述种子像素为背景种子像素，所述前景提取算法为grab cut算法。Step S1022, the terminal 200 uses the part outside theforeground frame 211 on the image corresponding to thecurrent viewing area 210 as the seed pixels required by the foreground extraction algorithm; wherein, the seed pixels are background seed pixels, and the foreground extraction algorithm For the grab cut algorithm.

可以理解的是，所述获取当前取景区域210对应的图像，具体包括：It can be understood that the obtaining of the image corresponding to thecurrent viewing area 210 specifically includes:

终端200从拍摄的视频中提取出当前时刻对应的视频帧，将所述视频帧作为当前取景区域210对应的图像。或者The terminal 200 extracts a video frame corresponding to the current moment from the captured video, and uses the video frame as an image corresponding to thecurrent viewing area 210 . or

终端200将当前取景区域210上的前景框211隐藏，并获取隐藏前景框211后的取景区域210的截图，将所述截图作为当前取景区域210对应的图像。The terminal 200 hides theforeground frame 211 on thecurrent viewing area 210 , obtains a screenshot of theviewing area 210 after theforeground frame 211 is hidden, and uses the screenshot as an image corresponding to thecurrent viewing area 210 .

本实施例中，获取当前取景区域210对应的图像的方式并非是拍照，所以可以同时分别对多个目标商品进行相应的对焦操作，以获得各自清晰的图像。In this embodiment, the method of acquiring the image corresponding to thecurrent framing area 210 is not to take a photo, so the corresponding focusing operations can be performed on a plurality of target commodities at the same time, so as to obtain respective clear images.

步骤S104，终端200根据所述种子像素和前景提取算法从当前取景区域210对应的图像中提取得到目标商品对应的前景图像。Step S104, the terminal 200 extracts a foreground image corresponding to the target commodity from the image corresponding to thecurrent viewing area 210 according to the seed pixel and the foreground extraction algorithm.

利用grab cut算法提取前景图像属于现有技术，此处不再赘述。Extracting the foreground image by using the grab cut algorithm belongs to the prior art, and details are not described herein again.

步骤S106，终端200将所述前景图像发送至服务器100，以便服务器100根据所述前景图像匹配相应的商品信息并将所述商品信息发送给终端200。Step S106 , the terminal 200 sends the foreground image to theserver 100 , so that theserver 100 matches the corresponding commodity information according to the foreground image and sends the commodity information to the terminal 200 .

相较于现有技术中需要先拍摄多个商品的一张整体照片，再对照片中各商品分别进行触摸操作选取对应商品的前景种子和背景种子用于分割前景的复杂操作，本发明提供的商品搜索方法通过在拍摄界面201的取景区域210显示前景框211，用户在拍摄界面201就可以通过调整拍照角度的操作和对焦操作使得目标商品对应的图像全部位于所述前景框211内，并且在拍摄界面201中就可以触发终端200对当前取景区域210的图像进行前景的提取，并且此时的前景提取(即图像分割)直接根据前景框211来确定grabcut算法所需的背景种子，无需用户再额外操作，简化了分割前景的流程。并且在提取完当前目标商品对应的前景图像之后，由于终端200还处于拍摄界面201，因此，可以继续调整拍照角度的操作和对焦操作进而对其他的目标商品继续进行前景图像的提取，保证每个目标商品都能够拍摄得到清晰的图像，提高目标商品对应图像的分割效果，从而提升商品搜索结果的准确性。可以理解的是，现有技术中，之所以不在拍摄界面201分别对多个目标商品进行分割，是因为分割算法(即前景提取算法)需要通过交互操作(用户触摸屏幕)的方式提供种子像素，而在拍摄界面201中进行交互操作时又会触发相机程序对应的功能(例如在相机程序的拍摄界面201用手指滑动屏幕时会调整相机的焦距等)，即用于确定种子像素的交互动作会与拍照过程的功能存在干扰，因此现有技术都是先拍摄完整体照片，才允许用户在照片上交互操作确定种子像素。本发明人的贡献之一在于发现了这个问题存在的原因，并进一步通过在拍摄界面201上显示前景框211，提醒用户将目标商品对应的图像全部置于前景框211内实现了确定种子像素的交互操作，从而能够在拍摄界面201中通过调整拍照角度的操作和对焦操作将各目标商品的图像调整至最清晰的情况下再分别确定多个商品对应的种子像素，解决了现有技术中存在的技术问题。Compared with the complex operation in the prior art, which needs to take an overall photo of multiple commodities, and then perform a touch operation on each commodity in the photo to select the foreground seeds and background seeds of the corresponding commodities for dividing the foreground. In the commodity search method, theforeground frame 211 is displayed in theframing area 210 of theshooting interface 201, and the user can adjust the camera angle and focus operation on theshooting interface 201 so that the images corresponding to the target commodity are all located in theforeground frame 211, and in theshooting interface 201. In theshooting interface 201, the terminal 200 can be triggered to extract the foreground of the image of thecurrent viewing area 210, and the foreground extraction (ie image segmentation) at this time can directly determine the background seed required by the grabcut algorithm according to theforeground frame 211, without the need for the user to further. Additional operations to simplify the process of segmenting foregrounds. And after the foreground image corresponding to the current target commodity is extracted, since the terminal 200 is still in theshooting interface 201, the operation of adjusting the camera angle and the focusing operation can be continued, and the foreground image extraction of other target commodities can be continued to ensure that each The target product can be captured with clear images, which can improve the segmentation effect of the corresponding image of the target product, thereby improving the accuracy of product search results. It can be understood that, in the prior art, the reason why theshooting interface 201 is not to separately segment multiple target commodities is because the segmentation algorithm (that is, the foreground extraction algorithm) needs to provide seed pixels through an interactive operation (the user touches the screen). When performing an interactive operation in theshooting interface 201, the corresponding function of the camera program will be triggered (for example, when sliding the screen with a finger in theshooting interface 201 of the camera program, the focal length of the camera will be adjusted, etc.), that is, the interactive action used to determine the seed pixel will There is interference with the function of the photographing process, so in the prior art, the entire body photograph is photographed first, and then the user is allowed to interactively operate on the photograph to determine the seed pixels. One of the contributions of the present inventor is to discover the reason for this problem, and further by displaying theforeground frame 211 on theshooting interface 201 to remind the user to place all the images corresponding to the target product in theforeground frame 211 to realize the determination of seed pixels. Interactive operation, so that in theshooting interface 201, the image of each target product can be adjusted to the clearest through the operation of adjusting the photographing angle and the focusing operation, and then the seed pixels corresponding to multiple products can be determined respectively, which solves the problem of existing in the prior art. technical issues.

如图8所示，在一个实施例中，所述方法还包括：As shown in Figure 8, in one embodiment, the method further includes:

步骤S200，终端200在所述按钮区域220显示形状调整控件221。Step S200 , the terminal 200 displays theshape adjustment control 221 in thebutton area 220 .

步骤S202，终端200检测到用户对所述形状调整控件221的触摸操作时，在按钮区域220显示多个第一对象；其中，所述多个第一对象中的每个第一对象分别用来标识不同形状的前景框211。Step S202, when the terminal 200 detects the user's touch operation on theshape adjustment control 221, a plurality of first objects are displayed in thebutton area 220; wherein, each first object in the plurality of first objects is used for Different shapes offoreground boxes 211 are identified.

步骤S204，终端200检测到用户对多个第一对象中的任一第一对象的触摸操作时，将取景区域210中前景框211的形状调整为与所述任一第一对象对应的形状。Step S204 , when the terminal 200 detects a user's touch operation on any one of the multiple first objects, the terminal 200 adjusts the shape of theforeground frame 211 in theviewfinder area 210 to a shape corresponding to the one of the first objects.

如图6所示，第一对象可以是图6中图标为不同几何形状的控件，该实施例中，从上至下依次包括圆形控件、矩形控件、三角形控件、五边形控件等。当用户通过手指触摸相应几何形状的控件时，取景区域210中的前景框211的形状就会调整为相应形状。如图6所示，用户触摸了圆形控件，则前景框211的形状调整为圆形。As shown in FIG. 6 , the first object may be a control whose icons are of different geometric shapes in FIG. 6 . In this embodiment, it includes a circular control, a rectangular control, a triangle control, a pentagon control, and the like from top to bottom. When a user touches a control of a corresponding geometric shape with a finger, the shape of theforeground frame 211 in theviewing area 210 will be adjusted to a corresponding shape. As shown in FIG. 6 , when the user touches the circular control, the shape of theforeground frame 211 is adjusted to be circular.

可以理解的，当前景框211中的前景图像越多，背景图像越少，则分割算法分割精度越高，效果越好。为了实现这一目的，本发明中提供了调整前景框211形状的功能，允许用户根据目标商品对应的图像的形状来选取对应形状的前景框211，从而提高分割的精度。It can be understood that when there are more foreground images in theforeground frame 211 and fewer background images, the segmentation algorithm has higher segmentation accuracy and better effect. To achieve this purpose, the present invention provides a function of adjusting the shape of theforeground frame 211, allowing the user to select theforeground frame 211 of the corresponding shape according to the shape of the image corresponding to the target product, thereby improving the accuracy of segmentation.

如图9所示，在一个实施例中，在检测到对前景框211之内的屏幕的触摸信号时，所述方法还包括：As shown in FIG. 9 , in one embodiment, when a touch signal to the screen within theforeground frame 211 is detected, the method further includes:

步骤S300，终端200接收用户对前景框211之内的屏幕的第一触摸信号和第二触摸信号。Step S300 , the terminal 200 receives a first touch signal and a second touch signal of the user on the screen within theforeground frame 211 .

步骤S302，在所述第一触摸信号和第二触摸信号为连续的滑动信号并且第一触摸信号从第一位置移动到第二位置、第二触摸信号从第三位置移动到第四位置的情况下，终端200判断第一位置和第三位置的第一距离和第三位置和第四位置的第二距离的关系；其中，第一位置、第二位置、第三位置和第四位置均分别位于前景框211之内。Step S302, when the first touch signal and the second touch signal are continuous sliding signals and the first touch signal moves from the first position to the second position, and the second touch signal moves from the third position to the fourth position Next, the terminal 200 judges the relationship between the first distance between the first position and the third position and the second distance between the third position and the fourth position; wherein the first position, the second position, the third position and the fourth position are respectively within theforeground frame 211.

步骤S304，若第一距离小于第二距离，则触发相机调大焦距，并在检测到分割触发信号时，终端200根据第一触摸信号和第二触摸信号移动的轨迹确定前景提取算法所需的前景种子像素，将当前取景区域210对应的图像上位于所述前景框211之外的部分作为前景提取算法所需的背景种子像素；所述前景提取算法为graph cut算法。Step S304, if the first distance is less than the second distance, trigger the camera to increase the focal length, and when the segmentation trigger signal is detected, the terminal 200 determines the foreground extraction algorithm according to the trajectory of the movement of the first touch signal and the second touch signal. For the foreground seed pixels, the part outside theforeground frame 211 on the image corresponding to thecurrent viewing area 210 is used as the background seed pixels required by the foreground extraction algorithm; the foreground extraction algorithm is a graph cut algorithm.

而本发明中，利用用户在使用两个手指在拍摄界面201调节焦距时的触摸信号对应的轨迹来作为获取前景种子像素。其依据在于，如图4-A所示，用户先调整摄像头视角和角度使得无线耳机盒对应的图像弯曲位于前景框211之内，然后为了使得无线耳机盒对应的图像占据前景框211更多的面积，用户通过食指和拇指同时触摸无线耳机盒对应的图像并食指和拇指分别沿图4-A中两个带线箭头的方向向外滑动，触发相机调大焦距。调大焦距的过程中，无线耳机盒对应的图像在前景框211中逐渐变大，用户时刻观察使得无线耳机盒对应的图像不会超出前景框211之外，调大焦距的结果如图4-B所示。In the present invention, however, the track corresponding to the touch signal when the user uses two fingers to adjust the focus on the photographinginterface 201 is used as the acquisition of the foreground seed pixel. The basis is that, as shown in FIG. 4-A, the user first adjusts the viewing angle and angle of the camera so that the image corresponding to the wireless earphone box is curved and located within theforeground frame 211, and then in order to make the image corresponding to the wireless earphone box occupy more of theforeground frame 211. The user touches the corresponding image of the wireless earphone box with the index finger and thumb at the same time, and slides the index finger and thumb outward in the direction of the two line arrows in Figure 4-A, triggering the camera to increase the focal length. During the process of increasing the focal length, the image corresponding to the wireless earphone box gradually becomes larger in theforeground frame 211, and the user always observes so that the image corresponding to the wireless earphone box does not exceed theforeground frame 211. The result of increasing the focal length is shown in Figure 4- shown in B.

在一个示例中，第一位置为A点，第二位置为A’点，第三位置为B点，第四位置为B’点，四个点均分别位于前景框211之内，且均位于无线耳机盒对应的图像范围上。此时第一触摸信号移动的轨迹为AA’，第二触摸信号移动的轨迹为BB’。需要说明的是，A点、A’点、B点、B’点等四个点的坐标均是屏幕坐标系下的坐标，在调节焦距的过程中，A点和B点的坐标都是不变的。在检测到分割触发信号时，获取当前时刻的取景区域210中的拍摄得到的图像，并计算该图像在屏幕坐标系下各像素对应的坐标值，利用AA’、BB’两个轨迹对应的屏幕坐标系确定当前时刻的取景区域210中的拍摄得到的图像中对应的像素点，将这些像素点作为前景种子像素。In one example, the first position is point A, the second position is point A', the third position is point B, and the fourth position is point B', and the four points are respectively located within theforeground frame 211, and all are located within theforeground frame 211. On the image range corresponding to the wireless earphone box. At this time, the track moved by the first touch signal is AA', and the track moved by the second touch signal is BB'. It should be noted that the coordinates of point A, point A', point B, point B' and other four points are the coordinates of the screen coordinate system. During the process of adjusting the focus, the coordinates of point A and point B are different. changing. When the segmentation trigger signal is detected, the captured image in theviewing area 210 at the current moment is acquired, and the coordinate value corresponding to each pixel of the image in the screen coordinate system is calculated, and the screen corresponding to the two trajectories AA' and BB' is used. The coordinate system determines the corresponding pixels in the captured image in theframing area 210 at the current moment, and uses these pixels as foreground seed pixels.

为了进一步提高分割的效果，还可以增加更多的前景种子像素，本实施例中，还可以将第一位置和第三位置连线经过的像素点作为前景种子像素，以提高分割的效果。In order to further improve the effect of segmentation, more foreground seed pixels may be added. In this embodiment, the pixels passing through the line connecting the first position and the third position may be used as foreground seed pixels to improve the effect of segmentation.

步骤S306，终端200根据前景种子像素、背景种子像素和前景提取算法从当前取景区域210对应的图像中提取得到目标商品对应的前景图像。Step S306, the terminal 200 extracts a foreground image corresponding to the target commodity from the image corresponding to thecurrent viewing area 210 according to the foreground seed pixels, the background seed pixels and the foreground extraction algorithm.

步骤S308，终端200将所述前景图像发送至服务器100，以便服务器100根据所述前景图像匹配相应的商品信息并将所述商品信息发送给终端200。Step S308 , the terminal 200 sends the foreground image to theserver 100 , so that theserver 100 matches the corresponding commodity information according to the foreground image and sends the commodity information to the terminal 200 .

grab cut与graph cut的对比如下：The comparison between grab cut and graph cut is as follows:

(1)graph cut的前景与背景的模型是灰度直方图，而grab cut取代为RGB三通道的高斯混合模型；(1) The model of the foreground and background of graph cut is a grayscale histogram, and grab cut is replaced by a Gaussian mixture model of RGB three-channel;

(2)graph cut是能量最小化是一次性达到的，而grab cut取代为一个不断进行分割估计和模型参数学习的交互迭代过程；(2) Graph cut is that the energy minimization is achieved at one time, while grab cut is replaced by an interactive iterative process that continuously performs segmentation estimation and model parameter learning;

(3)graph cut需要用户提供前景和背景的一些种子，但是grab cut只需要提供背景区域的像素集就行。(3) graph cut requires the user to provide some seeds for the foreground and background, but grab cut only needs to provide the pixel set of the background area.

优点：grab cut只需要用户简单的交互，操作简单，效果也不错；Advantages: grab cut only requires simple user interaction, the operation is simple, and the effect is good;

缺点：grab cut计算量大，分割速度慢。Disadvantages: grab cut has a large amount of calculation and slow segmentation.

可以理解的是，grab cut算法由于只需要提供背景的种子像素，其相较于graphcut算法具有更方便的交互效果。但是grab cut为一个不断进行分割估计和模型参数学习的交互迭代过程,其存在计算量大，分割速度慢的缺点。由于本实施例中，是利用终端200来执行分割算法，为了加快分割算法的运行速度，需要提供一种分割运算更快的方案，即使用graph cut算法来进行前景分割。已知利用graph cut算法来进行前景分割的困难在于，需要同时提供图像中前景和背景对应的种子像素，因此在交互上要求比较麻烦，这也是背景技术中的拍照搜索方案为何需要先拍照再在照片上选定不同的目标商品对应的种子的原因。即因为分割算法(即前景提取算法)需要通过交互操作(用户触摸屏幕)的方式提供种子像素，而在拍摄界面201中进行交互操作时又会触发相机程序对应的功能(例如在相机程序的拍摄界面201用手指滑动屏幕时会调整相机的焦距等)，即用于确定种子像素的交互动作会与拍照过程的功能存在干扰，因此现有技术都是先拍摄完整体照片，才允许用户在照片上交互操作确定种子像素。而本实施例中，利用用户在使用两个手指在拍摄界面201调节焦距时的触摸信号对应的轨迹来作为获取前景种子像素，可以利用用户拍摄过程必然进行的调焦操作来实现graph cut算法所需的确定前景种子像素的交互动作，并且在取景区域210设置前景框211让用户在调整视角进行拍摄的过程将目标商品对应的图像放置在该前景框211内，省去了获取背景种子像素的交互动作，极大的简化了分割算法的交互流程。It is understandable that the grab cut algorithm has a more convenient interactive effect than the graphcut algorithm because it only needs to provide the seed pixels of the background. However, grab cut is an interactive iterative process of continuous segmentation estimation and model parameter learning, which has the disadvantages of large amount of computation and slow segmentation. Since in this embodiment, the terminal 200 is used to execute the segmentation algorithm, in order to speed up the running speed of the segmentation algorithm, it is necessary to provide a solution with a faster segmentation operation, that is, use the graph cut algorithm to perform foreground segmentation. It is known that the difficulty of using the graph cut algorithm to perform foreground segmentation is that the seed pixels corresponding to the foreground and background in the image need to be provided at the same time, so the interaction requirements are more troublesome, which is why the photo search scheme in the background technology needs to take a photo first and then The reason why the seeds corresponding to different target products are selected on the photo. That is, because the segmentation algorithm (that is, the foreground extraction algorithm) needs to provide seed pixels through an interactive operation (the user touches the screen), and the interactive operation in theshooting interface 201 will trigger the corresponding function of the camera program (for example, in the shooting of the camera program). Theinterface 201 will adjust the focal length of the camera when sliding the screen with a finger, etc.), that is, the interaction for determining the seed pixel will interfere with the function of the photographing process, so the prior art is to take a complete photo first, and then allow the user to use the photo in the photo. The above interactive operation determines the seed pixel. In this embodiment, the trajectory corresponding to the touch signal when the user uses two fingers to adjust the focus on theshooting interface 201 is used as the acquisition of the foreground seed pixel, and the focus adjustment operation that the user must perform during the shooting process can be used to realize the graph cut algorithm. It is necessary to determine the interactive action of the foreground seed pixels, and set theforeground frame 211 in theviewing area 210 to allow the user to place the image corresponding to the target product in theforeground frame 211 in the process of adjusting the viewing angle to shoot, eliminating the need to obtain the background seed pixels. The interactive action greatly simplifies the interactive process of the segmentation algorithm.

在一个实施例中，在检测到对前景框211之内的屏幕的触摸信号时，所述方法还包括：In one embodiment, when a touch signal to the screen within theforeground frame 211 is detected, the method further includes:

步骤S500，终端200接收用户对前景框211之内的屏幕的第一触摸信号和第二触摸信号。Step S500 , the terminal 200 receives a first touch signal and a second touch signal of the user on the screen within theforeground frame 211 .

步骤S502，在所述第一触摸信号和第二触摸信号为连续的滑动信号并且第一触摸信号从第一位置移动到第二位置、第二触摸信号从第三位置移动到第四位置的情况下，终端200判断第一位置和第三位置的第一距离和第三位置和第四位置的第二距离的关系；其中，第一位置、第二位置、第三位置和第四位置均分别位于前景框211之内。Step S502, when the first touch signal and the second touch signal are continuous sliding signals and the first touch signal moves from the first position to the second position, and the second touch signal moves from the third position to the fourth position Next, the terminal 200 judges the relationship between the first distance between the first position and the third position and the second distance between the third position and the fourth position; wherein the first position, the second position, the third position and the fourth position are respectively within theforeground frame 211.

步骤S504，若第一距离大于第二距离，则触发相机调大焦距，并在检测到分割触发信号时，终端200根据第一触摸信号和第二触摸信号移动的轨迹确定前景提取算法所需的前景种子像素，将当前取景区域210对应的图像上位于所述前景框211之外的部分作为前景提取算法所需的背景种子像素；所述前景提取算法为graph cut算法。Step S504, if the first distance is greater than the second distance, trigger the camera to increase the focal length, and when the segmentation trigger signal is detected, the terminal 200 determines the foreground extraction algorithm according to the moving trajectory of the first touch signal and the second touch signal. For the foreground seed pixels, the part outside theforeground frame 211 on the image corresponding to thecurrent viewing area 210 is used as the background seed pixels required by the foreground extraction algorithm; the foreground extraction algorithm is a graph cut algorithm.

步骤S506，终端200根据前景种子像素、背景种子像素和前景提取算法从当前取景区域210对应的图像中提取得到目标商品对应的前景图像。Step S506, the terminal 200 extracts a foreground image corresponding to the target commodity from the image corresponding to thecurrent viewing area 210 according to the foreground seed pixels, the background seed pixels and the foreground extraction algorithm.

步骤S508，终端200将所述前景图像发送至服务器100，以便服务器100根据所述前景图像匹配相应的商品信息并将所述商品信息发送给终端200。Step S508 , the terminal 200 sends the foreground image to theserver 100 , so that theserver 100 matches the corresponding commodity information according to the foreground image and sends the commodity information to the terminal 200 .

可以理解的是，相较于上述实施例而言，本实施例的拇指和食指动作用于触发相机调小焦距，便于用户在目标商品对应的图像超出前景框211范围之外时，及时调整回来。具体的细节详见上述调大焦距的实施例，此处不再赘述。It can be understood that, compared with the above-mentioned embodiments, the thumb and index finger movements in this embodiment are used to trigger the camera to adjust the focal length, which is convenient for the user to adjust back in time when the image corresponding to the target product exceeds the range of theforeground frame 211. . For details, please refer to the above-mentioned embodiment of increasing the focal length, which will not be repeated here.

如图10所示，在一个实施例中，所述方法还包括：As shown in Figure 10, in one embodiment, the method further includes:

步骤S400，终端200在所述按钮区域220显示尺寸调整控件222。Step S400 , the terminal 200 displays thesize adjustment control 222 in thebutton area 220 .

步骤S402，终端200检测到用户对所述尺寸调整控件222的触摸操作时，在按钮区域220显示滚动条；其中，所述滚动条的不同位置用来标识前景框211的不同尺寸。Step S402 , when the terminal 200 detects the user's touch operation on thesize adjustment control 222 , a scroll bar is displayed in thebutton area 220 , wherein different positions of the scroll bar are used to identify different sizes of theforeground frame 211 .

步骤S404，终端200检测到用户对滚动条的触摸操作时，将取景区域210中前景框211的尺寸调整为与所述滚动条对应的尺寸。Step S404 , when the terminal 200 detects the user's touch operation on the scroll bar, the terminal 200 adjusts the size of theforeground frame 211 in theviewfinder area 210 to a size corresponding to the scroll bar.

如图5所示，当用户通过手指触摸滚动条时，取景区域210中的前景框211的尺寸形状就会调整为相应尺寸。具体的，前景框211调整尺寸时，以前景框211的几何中心作为调整的基点，调整的结果表现为前景框211面积的变化。如图5所示，用户触摸拉动滚动条向上移动时，前景框211以几何中心为基点，其面积变大；用户触摸拉动滚动条向下移动时，前景框211以几何中心为基点，其面积变小。具体的，如果前景框211的几何形状为矩形，则以矩形的两条对角线交点为基点，与基点相交的两条对角线被分割为四段等长的射线，当用户拉动滚动条向上移动时，将每段射线增加相同长度，从而增加矩形前景框211的面积。As shown in FIG. 5 , when the user touches the scroll bar with a finger, the size and shape of theforeground frame 211 in theviewing area 210 will be adjusted to the corresponding size. Specifically, when the size of theforeground frame 211 is adjusted, the geometric center of theforeground frame 211 is used as the base point of adjustment, and the result of the adjustment is shown as a change in the area of theforeground frame 211 . As shown in FIG. 5 , when the user touches and pulls the scroll bar to move up, theforeground frame 211 takes the geometric center as the base point, and its area becomes larger; when the user touches and pulls the scroll bar to move down, theforeground frame 211 takes the geometric center as the base point, and its area becomes larger. become smaller. Specifically, if the geometric shape of theforeground frame 211 is a rectangle, the intersection of the two diagonal lines of the rectangle is used as the base point, and the two diagonal lines intersecting with the base point are divided into four rays of equal length. When the user pulls the scroll bar When moving up, each ray segment is increased by the same length, thereby increasing the area of therectangular foreground frame 211 .

可以理解的，不同用户的手指尺寸是不一样的，本发明由于要求用户在前景框211范围内利用手指触摸操作来调节拍照焦距，所以提供了前景框211的尺寸调节功能，以便适用于不同的用户群体。进一步的，由于本发明要求将目标商品对应的图像全部置于前景框211中，并且为了满足分割算法分割精度要求，一般要求目标商品对应的图像要差不多填充满前景框211，而由于不同终端200相机的光学元件性能不尽相同，拍照环境的光照条件也有很大差异，因此用户针对不同的光照条件和不同光学性能的相机，通过调节前景框211的尺寸可以获得更清晰的目标商品的图像。例如在环境光照十分充足的情况下，在焦距不需要很大的时候就可以获得边缘清晰的目标商品的图像，反而在进一步调大焦距(一般调大焦距时，目标商品在取景区域210中对应的图像范围会相应的变大)时，会造成曝光过度，反而得不到边缘清晰的目标商品的图像。It can be understood that the finger sizes of different users are different. Since the present invention requires the user to adjust the focal length of the photo by touching the finger within the range of theforeground frame 211, the size adjustment function of theforeground frame 211 is provided so as to be suitable for different users. Client. Further, because the present invention requires all images corresponding to the target product to be placed in theforeground frame 211, and in order to meet the segmentation accuracy requirements of the segmentation algorithm, the image corresponding to the target product is generally required to almost fill theforeground frame 211. The performance of the optical elements of the cameras is not the same, and the lighting conditions of the photographing environment are also very different. Therefore, the user can obtain a clearer image of the target product by adjusting the size of theforeground frame 211 for cameras with different lighting conditions and different optical performances. For example, in the case of sufficient ambient light, the image of the target product with clear edges can be obtained when the focal length does not need to be large, but when the focal length is further increased (generally when the focal length is increased, the target product corresponds to the viewfinder area 210 ). The image range will be correspondingly larger), it will cause overexposure, and instead, the image of the target product with clear edges will not be obtained.

图11示出了一个实施例中计算机设备的内部结构图。该计算机设备具体可以是图1中的终端200(或服务器100)。如图11所示，该计算机设备包括通过系统总线连接的处理器、存储器、网络接口、输入装置和显示屏。其中，存储器包括非易失性存储介质和内存储器。该计算机设备的非易失性存储介质存储有操作系统，还可存储有计算机程序，该计算机程序被处理器执行时，可使得处理器实现商品搜索方法。该内存储器中也可储存有计算机程序，该计算机程序被处理器执行时，可使得处理器执行商品搜索方法。本领域技术人员可以理解，图11中示出的结构，仅仅是与本发明方案相关的部分结构的框图，并不构成对本发明方案所应用于其上的计算机设备的限定，具体的计算机设备可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。Figure 11 shows an internal structure diagram of a computer device in one embodiment. The computer device may specifically be the terminal 200 (or the server 100 ) in FIG. 1 . As shown in FIG. 11, the computer equipment includes a processor, memory, network interface, input device and display screen connected by a system bus. Wherein, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system, and also stores a computer program, which, when executed by the processor, enables the processor to implement the commodity search method. A computer program may also be stored in the internal memory, and when the computer program is executed by the processor, the processor may execute the commodity search method. Those skilled in the art can understand that the structure shown in FIG. 11 is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the computer equipment to which the solution of the present invention is applied. Include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

在一个实施例中，本申请提供的商品搜索装置可以实现为一种计算机程序的形式，计算机程序可在如图11所示的计算机设备上运行。计算机设备的存储器中可存储组成该商品搜索装置的各个程序模块，比如，显示模块、种子提取模块、前景图像提取模块和发送模块。各个程序模块构成的计算机程序使得处理器执行本说明书中描述的本申请各个实施例的商品搜索方法中的步骤。In one embodiment, the commodity searching apparatus provided by the present application can be implemented in the form of a computer program, and the computer program can be executed on the computer device as shown in FIG. 11 . The memory of the computer device can store various program modules constituting the commodity searching apparatus, such as a display module, a seed extraction module, a foreground image extraction module and a transmission module. The computer program constituted by each program module enables the processor to execute the steps in the commodity search method of each embodiment of the present application described in this specification.

例如，图11所示的计算机设备可以通过商品搜索装置中的显示模块执行当处于拍摄界面201时，在取景区域210显示前景框211的步骤，以便用户操作使得目标商品对应的图像全部位于所述前景框211内。通过种子提取模块执行在未检测到对前景框211之内的屏幕的触摸信号，并且检测到分割触发信号时，根据所述前景框211确定前景提取算法所需的种子像素的步骤。通过前景图像提取模块执行根据所述种子像素和前景提取算法从当前取景区域210对应的图像中提取得到目标商品对应的前景图像的步骤。通过发送模块执行将所述前景图像发送至服务器100的步骤，以便服务器100根据所述前景图像匹配相应的商品信息并将所述商品信息发送给终端200。For example, the computer device shown in FIG. 11 can perform the step of displaying theforeground frame 211 in theviewfinder area 210 through the display module in the product search device when it is in theshooting interface 201, so that the user operates so that the images corresponding to the target product are all located in the In theforeground box 211 . The step of determining the seed pixels required by the foreground extraction algorithm according to theforeground frame 211 is performed by the seed extraction module when no touch signal to the screen within theforeground frame 211 is detected and a segmentation trigger signal is detected. The step of extracting the foreground image corresponding to the target commodity from the image corresponding to thecurrent viewing area 210 according to the seed pixel and the foreground extraction algorithm is performed by the foreground image extraction module. The step of sending the foreground image to theserver 100 is performed by the sending module, so that theserver 100 matches the corresponding commodity information according to the foreground image and sends the commodity information to the terminal 200 .

在一个实施例中，提供了一种电子设备，包括：存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时执行上述商品搜索方法的步骤。此处商品搜索方法的步骤可以是上述各个实施例的商品搜索方法中的步骤。In one embodiment, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the above-mentioned method for searching for goods when the program is executed. step. The steps of the commodity search method here may be the steps in the commodity search methods of the above-mentioned embodiments.

在一个实施例中，提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机可执行指令，所述计算机可执行指令用于使计算机执行上述商品搜索方法的步骤。此处商品搜索方法的步骤可以是上述各个实施例的商品搜索方法中的步骤。In one embodiment, a computer-readable storage medium is provided, and the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to cause a computer to perform the steps of the above commodity search method. The steps of the commodity search method here may be the steps in the commodity search methods of the above-mentioned embodiments.

本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的程序可存储于一非易失性计算机可读取存储介质中，该程序在执行时，可包括如上述各方法的实施例的流程。其中，本申请所提供的各实施例中所使用的对存储器、存储、数据库或其它介质的任何引用，均可包括非易失性和/或易失性存储器。非易失性存储器可包括只读存储器(ROM)、可编程ROM(PROM)、电可编程ROM(EPROM)、电可擦除可编程ROM(EEPROM)或闪存。易失性存储器可包括随机存取存储器(RAM)或者外部高速缓冲存储器。作为说明而非局限，RAM以多种形式可得，诸如静态RAM(SRAM)、动态RAM(DRAM)、同步DRAM(SDRAM)、双数据率SDRAM(DDRSDRAM)、增强型SDRAM(ESDRAM)、同步链路(Synchlink)DRAM(SLDRA)、存储器总线(Rambus)直接RAM(RDRA)、直接存储器总线动态RAM(DRDRAM)、以及存储器总线动态RAM(RDRAM)等。Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium , when the program is executed, it may include the flow of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database or other medium used in the various embodiments provided in this application may include non-volatile and/or volatile memory. Nonvolatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Road (Synchlink) DRAM (SLDRA), memory bus (Rambus) direct RAM (RDRA), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

以上实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features It is considered to be the range described in this specification.

Claims (10)

1. A method for searching for an item, the method comprising:

when the target commodity is in a shooting interface, displaying a foreground frame in a view finding area so that the user can operate to enable all images corresponding to the target commodity to be located in the foreground frame;

when a touch signal to a screen in a foreground frame is not detected and a segmentation trigger signal is detected, determining seed pixels required by a foreground extraction algorithm according to the foreground frame;

extracting a foreground image corresponding to the target commodity from the image corresponding to the current viewing area according to the seed pixel and the foreground extraction algorithm;

and sending the foreground image to a server so that the server matches corresponding commodity information according to the foreground image and sends the commodity information to a terminal.

2. The item search method of claim 1, wherein the camera interface further comprises a button area, and the method further comprises:

displaying a segmentation control in the button area;

and when the touch operation of the user on the segmentation control is detected, generating a segmentation trigger signal.

3. The method of claim 1, further comprising:

displaying a shape adjustment control in the button area;

when the touch operation of the user on the shape adjusting control is detected, displaying a plurality of first objects in a button area; wherein each of the plurality of first objects is used to identify foreground frames of different shapes, respectively;

when the touch operation of the user on any one of the plurality of first objects is detected, the shape of the foreground frame in the view area is adjusted to be the shape corresponding to the any one of the first objects.

4. The method for searching for a commodity according to claim 1, wherein the determining of the seed pixels required by the foreground extraction algorithm according to the foreground frame specifically comprises:

acquiring an image corresponding to a current viewing area;

taking the part of the image corresponding to the current view area, which is positioned outside the foreground frame, as a seed pixel required by a foreground extraction algorithm; the seed pixels are background seed pixels, and the foreground extraction algorithm is a grab cut algorithm.

5. The method for searching for a commodity according to claim 4, wherein the acquiring of the image corresponding to the current viewing area specifically includes:

extracting a video frame corresponding to the current moment from a video shot by a terminal, and taking the video frame as an image corresponding to a current view area; or

And hiding the foreground frame on the current viewing area, acquiring a screenshot of the viewing area after the foreground frame is hidden, and taking the screenshot as an image corresponding to the current viewing area.

6. The commodity search method according to any one of claims 1 to 5, wherein upon detection of a touch signal to a screen within a foreground frame, the method further comprises:

receiving a first touch signal and a second touch signal of a user on a screen in a foreground frame;

judging the relationship between the first distance between the first position and the third position and the second distance between the third position and the fourth position when the first touch signal and the second touch signal are continuous sliding signals, the first touch signal moves from the first position to the second position, and the second touch signal moves from the third position to the fourth position; the first position, the second position, the third position and the fourth position are respectively positioned in the foreground frame;

if the first distance is smaller than the second distance, the camera is triggered to adjust the focal length to be larger, when a segmentation trigger signal is detected, foreground seed pixels required by a foreground extraction algorithm are determined according to moving tracks of the first touch signal and the second touch signal, and the part, located outside the foreground frame, of the image corresponding to the current view-finding area is used as background seed pixels required by the foreground extraction algorithm; the foreground extraction algorithm is a graph cut algorithm;

extracting a foreground image corresponding to the target commodity from the image corresponding to the current viewing area according to the foreground seed pixel, the background seed pixel and a foreground extraction algorithm;

and sending the foreground image to a server so that the server matches corresponding commodity information according to the foreground image and sends the commodity information to a terminal.

7. The commodity search method according to any one of claims 1 to 5, wherein upon detection of a touch signal to a screen within a foreground frame, the method further comprises:

receiving a first touch signal and a second touch signal of a user on a screen in a foreground frame;

judging the relationship between the first distance between the first position and the third position and the second distance between the third position and the fourth position when the first touch signal and the second touch signal are continuous sliding signals, the first touch signal moves from the first position to the second position, and the second touch signal moves from the third position to the fourth position; the first position, the second position, the third position and the fourth position are respectively positioned in the foreground frame;

if the first distance is greater than the second distance, triggering the camera to adjust the focal length to be larger, determining foreground seed pixels required by a foreground extraction algorithm according to moving tracks of the first touch signal and the second touch signal when a segmentation trigger signal is detected, and taking the part, which is positioned outside the foreground frame, of the image corresponding to the current view-finding area as background seed pixels required by the foreground extraction algorithm; the foreground extraction algorithm is a graph cut algorithm;

extracting a foreground image corresponding to the target commodity from the image corresponding to the current viewing area according to the foreground seed pixel, the background seed pixel and a foreground extraction algorithm;

and sending the foreground image to a server so that the server matches corresponding commodity information according to the foreground image and sends the commodity information to a terminal.

8. The item search method according to claim 6, further comprising:

displaying a sizing control in the button area;

when the touch operation of the user on the size adjusting control is detected, displaying a scroll bar in a button area; wherein different positions of the scrollbar are used to identify different sizes of foreground frames;

and when the touch operation of the user on the scroll bar is detected, adjusting the size of the foreground frame in the view area to the size corresponding to the scroll bar.

9. An electronic device, comprising: memory, processor and computer program stored on the memory and executable on the processor, characterized in that the processor implements the item search method according to any one of claims 1 to 8 when executing the program.

10. A computer-readable storage medium storing computer-executable instructions for causing a computer to execute the item search method according to any one of claims 1 to 8.

Images