Authentification

Before loadingimgrec you first need to initiate yourauthentification credentials. You need an API key from a Google Projectwith access permission for the Google Vision API. For this, you canfirst create a project using the Google Cloud platform. The setupprocess is explained in the APIdocumentation.You will probably need to enable billing, but depending on your featureselection up to 1000 requests per month are free (seepricing). Nextfollowing theinstructionsfor creating an API key. Finally, the API key needs to be set asenvironment variable before using the initialization functiongvision_init():

Sys.setenv(gvision_key ="Your Google Vision API key")

library(imgrec)gvision_init()

In order to avoid callingSys.setenv, you canpermanently store the API key in your.Renviron. Irecommendusethis::edit_r_environ() to find and edit yourenvironment file.

Image annotations

Google Vision accepts common file types such as JPG, PNG, or BMP.Images can be passed to severalget_annotations, either asurl strings or file paths to local images. In the following example,get_annotations is used to retrieve annotations for aposter of the Star Wars movieTheForce Awakens.

sw_image<-'https://upload.wikimedia.org/wikipedia/en/a/a2/Star_Wars_The_Force_Awakens_Theatrical_Poster.jpg'results<-get_annotations(images = sw_image,# image character vectorfeatures ='all',# request all available featuresmax_res =5,# maximum number of results per featuremode ='url')# determine image type

The function returns a response object from the Google Vision API. Italso recognizes if a user passes a character vector with multipleimages. In this case, request batches are created automatically toreduce the number of required calls to the API.

After retrieving annotations, raw data can be stored in an UTF-8encodedJSON file:

temp_file_path<-tempfile(fileext ='.json')save_json(results, temp_file_path)

While some users might prefer to work with raw.jsondata, which includes every single detail returned by the API, thestructure is quite complex and deeply nested. To simplify the data,parse_annotations converts most of the features to dataframes. For each feature, the original identifier of each image isincluded asimg_id.

img_data<-parse_annotations(results)# returns list of data framesnames(img_data)# all available features

Once the features are converted to data frames, other R packages canbe used to analyze the data. For instance, thelabels dataframe contains annotations about image content:

img_labels<- img_data$labelshead(img_labels)

imgrec also extracts bounding polygons for logos, objects,faces and landmarks. We can for instance visualize all recognized logosof the Star Wars movie poster withmagick andggplot2:

[!!] There is currently a bug when usingmagick andggplot2 which leads to upside down annotations. A temporarywork around is to subtract image height (y) values (see codebelow).

library(magick)library(ggplot2)img<-image_read(sw_image)image_ggplot(img)+geom_rect(data = img_data$logos,aes(xmin = poly_x_min,xmax = poly_x_max,ymin =322- poly_y_min,ymax =322- poly_y_max    ),inherit.aes =FALSE,color ='yellow',fill =NA,linetype ='dashed',size =2  )+geom_text(data = img_data$logos,aes(x = poly_x_max,y =322- poly_y_max,label = description),size =4,color ="yellow",vjust =1  )+theme(legend.position ="none")

Please note that forobject recognition data, boundingpolygons are relative to image dimensions. Therefore, you need tomultiply them with image width (x) and height (y). These attributes arenot returned by Google Vision, but can for instance be identified withmagick::image_info():

img_info<-image_info(img)img_info

Additional functions for feature analysis are currently indevelopment.