Detect objects in images with an AutoML-trained model on Apple platforms

After youtrain your own model using AutoML Vision Edge,you can use it in your app to detect objects in images.

Firebase ML's AutoML Vision Edge features are deprecated. Consider usingVertex AI to automatically train ML models, which you can eitherexport as TensorFlow Lite models for on-device use ordeploy for cloud-based inference.

There are two ways to integrate models trained from AutoML Vision Edge. You canbundle the model by copying the model's files into your Xcode project, or youcan dynamically download it from Firebase.

Model bundling options
Bundled in your app
  • The model is part of the bundle
  • The model is available immediately, even when the Apple device is offline
  • No need for a Firebase project
Hosted with Firebase

Before you begin

  1. If you want to download a model, make sure youadd Firebase to your Apple project,if you have not already done so. This is not required when you bundle themodel.

  2. Include the TensorFlow and Firebase libraries in your Podfile:

    For bundling a model with your app:

    Swift

    pod'TensorFlowLiteSwift'

    Objective-C

    pod'TensorFlowLiteObjC'

    For dynamically downloading a model from Firebase, add theFirebase/MLModelInterpreter dependency:

    Swift

    pod'TensorFlowLiteSwift'pod'Firebase/MLModelInterpreter'

    Objective-C

    pod'TensorFlowLiteObjC'pod'Firebase/MLModelInterpreter'

  3. After you install or update your project's Pods, open your Xcode projectusing its.xcworkspace.

1. Load the model

Configure a local model source

To bundle the model with your app,copy the model and labels file to your Xcode project, taking care to selectCreate folder references when you do so. The model file and labelswill be included in the app bundle.

Also, look at thetflite_metadata.json file that was created alongside themodel. You need two values:

  • The model's input dimensions. This is 320x320 by default.
  • The model's maximum detections. This is 40 by default.

Configure a Firebase-hosted model source

To use the remotely-hosted model, create aCustomRemoteModelobject, specifying the name you assigned the model when you published it:

Swift

letremoteModel=CustomRemoteModel(name:"your_remote_model"// The name you assigned in theGoogle Cloud console.)

Objective-C

FIRCustomRemoteModel*remoteModel=[[FIRCustomRemoteModelalloc]initWithName:@"your_remote_model"];

Then, start the model download task, specifying the conditions under which youwant to allow downloading. If the model isn't on the device, or if a newerversion of the model is available, the task will asynchronously download themodel from Firebase:

Swift

letdownloadProgress=ModelManager.modelManager().download(remoteModel,conditions:ModelDownloadConditions(allowsCellularAccess:true,allowsBackgroundDownloading:true))

Objective-C

FIRModelDownloadConditions*conditions=[[FIRModelDownloadConditionsalloc]initWithAllowsCellularAccess:YESallowsBackgroundDownloading:YES];NSProgress*progress=[[FIRModelManagermodelManager]downloadModel:remoteModelconditions:conditions];

Many apps start the download task in their initialization code, but youcan do so at any point before you need to use the model.

Create an object detector from your model

After you configure your model sources, create a TensorFlow LiteInterpreterobject from one of them.

If you only have a locally-bundled model, just create an interpreter from themodel file:

Swift

guardletmodelPath=Bundle.main.path(forResource:"model",ofType:"tflite")else{print("Failed to load the model file.")returntrue}letinterpreter=tryInterpreter(modelPath:modelPath)tryinterpreter.allocateTensors()

Objective-C

NSString*modelPath=[[NSBundlemainBundle]pathForResource:@"model"ofType:@"tflite"];NSError*error;TFLInterpreter*interpreter=[[TFLInterpreteralloc]initWithModelPath:modelPatherror:&error];if(error!=NULL){return;}[interpreterallocateTensorsWithError:&error];if(error!=NULL){return;}

If you have a remotely-hosted model, you will have to check that it has beendownloaded before you run it. You can check the status of the model downloadtask using the model manager'sisModelDownloaded(remoteModel:) method.

Although you only have to confirm this before running the interpreter, if youhave both a remotely-hosted model and a locally-bundled model, it might makesense to perform this check when instantiating theInterpreter: create aninterpreter from the remote model if it's been downloaded, and from the local modelotherwise.

Swift

varmodelPath:String?ifModelManager.modelManager().isModelDownloaded(remoteModel){ModelManager.modelManager().getLatestModelFilePath(remoteModel){path,erroringuarderror==nilelse{return}guardletpath=pathelse{return}modelPath=path}}else{modelPath=Bundle.main.path(forResource:"model",ofType:"tflite")}guardmodelPath!=nilelse{return}letinterpreter=tryInterpreter(modelPath:modelPath)tryinterpreter.allocateTensors()

Objective-C

__blockNSString*modelPath;if([[FIRModelManagermodelManager]isModelDownloaded:remoteModel]){[[FIRModelManagermodelManager]getLatestModelFilePath:remoteModelcompletion:^(NSString*_NullablefilePath,NSError*_Nullableerror){if(error!=NULL){return;}if(filePath==NULL){return;}modelPath=filePath;}];}else{modelPath=[[NSBundlemainBundle]pathForResource:@"model"ofType:@"tflite"];}NSError*error;TFLInterpreter*interpreter=[[TFLInterpreteralloc]initWithModelPath:modelPatherror:&error];if(error!=NULL){return;}[interpreterallocateTensorsWithError:&error];if(error!=NULL){return;}

If you only have a remotely-hosted model, you should disable model-relatedfunctionality—for example, gray-out or hide part of your UI—untilyou confirm the model has been downloaded.

You can get the model download status by attaching observers to the defaultNotification Center. Be sure to use a weak reference toself in the observerblock, since downloads can take some time, and the originating object can befreed by the time the download finishes. For example:

Swift

NotificationCenter.default.addObserver(forName:.firebaseMLModelDownloadDidSucceed,object:nil,queue:nil){[weakself]notificationinguardletstrongSelf=self,letuserInfo=notification.userInfo,letmodel=userInfo[ModelDownloadUserInfoKey.remoteModel.rawValue]as?RemoteModel,model.name=="your_remote_model"else{return}// The model was downloaded and is available on the device}NotificationCenter.default.addObserver(forName:.firebaseMLModelDownloadDidFail,object:nil,queue:nil){[weakself]notificationinguardletstrongSelf=self,letuserInfo=notification.userInfo,letmodel=userInfo[ModelDownloadUserInfoKey.remoteModel.rawValue]as?RemoteModelelse{return}leterror=userInfo[ModelDownloadUserInfoKey.error.rawValue]// ...}

Objective-C

__weaktypeof(self)weakSelf=self;[NSNotificationCenter.defaultCenteraddObserverForName:FIRModelDownloadDidSucceedNotificationobject:nilqueue:nilusingBlock:^(NSNotification*_Nonnullnote){if(weakSelf==nil|note.userInfo==nil){return;}__strongtypeof(self)strongSelf=weakSelf;FIRRemoteModel*model=note.userInfo[FIRModelDownloadUserInfoKeyRemoteModel];if([model.nameisEqualToString:@"your_remote_model"]){// The model was downloaded and is available on the device}}];[NSNotificationCenter.defaultCenteraddObserverForName:FIRModelDownloadDidFailNotificationobject:nilqueue:nilusingBlock:^(NSNotification*_Nonnullnote){if(weakSelf==nil|note.userInfo==nil){return;}__strongtypeof(self)strongSelf=weakSelf;NSError*error=note.userInfo[FIRModelDownloadUserInfoKeyError];}];

2. Prepare the input image

Next, you need to prepare your images for the TensorFlow Lite interpreter.

  1. Crop and scale the image to the model's input dimensions, as specified inthetflite_metadata.json file (320x320 pixels by default). You can do thiswith Core Image or a third-party library

  2. Copy the image data into aData (NSData object):

    Swift

    guardletimage:CGImage=// Your input imageguardletcontext=CGContext(data:nil,width:image.width,height:image.height,bitsPerComponent:8,bytesPerRow:image.width*4,space:CGColorSpaceCreateDeviceRGB(),bitmapInfo:CGImageAlphaInfo.noneSkipFirst.rawValue)else{returnnil}context.draw(image,in:CGRect(x:0,y:0,width:image.width,height:image.height))guardletimageData=context.dataelse{returnnil}varinputData=Data()forrowin0..<320{// Model takes 320x320 pixel images as inputforcolin0..<320{letoffset=4*(col*context.width+row)// (Ignore offset 0, the unused alpha channel)varred=imageData.load(fromByteOffset:offset+1,as:UInt8.self)vargreen=imageData.load(fromByteOffset:offset+2,as:UInt8.self)varblue=imageData.load(fromByteOffset:offset+3,as:UInt8.self)inputData.append(&red,count:1)inputData.append(&green,count:1)inputData.append(&blue,count:1)}}

    Objective-C

    CGImageRefimage=// Your input imagelongimageWidth=CGImageGetWidth(image);longimageHeight=CGImageGetHeight(image);CGContextRefcontext=CGBitmapContextCreate(nil,imageWidth,imageHeight,8,imageWidth*4,CGColorSpaceCreateDeviceRGB(),kCGImageAlphaNoneSkipFirst);CGContextDrawImage(context,CGRectMake(0,0,imageWidth,imageHeight),image);UInt8*imageData=CGBitmapContextGetData(context);NSMutableData*inputData=[[NSMutableDataalloc]initWithCapacity:0];for(introw=0;row <300;row++){for(intcol=0;col <300;col++){longoffset=4*(row*imageWidth+col);// (Ignore offset 0, the unused alpha channel)UInt8red=imageData[offset+1];UInt8green=imageData[offset+2];UInt8blue=imageData[offset+3];[inputDataappendBytes:&redlength:1];[inputDataappendBytes:&greenlength:1];[inputDataappendBytes:&bluelength:1];}}

3. Run the object detector

Next, pass the prepared input to the interpreter:

Swift

tryinterpreter.copy(inputData,toInputAt:0)tryinterpreter.invoke()

Objective-C

TFLTensor*input=[interpreterinputTensorAtIndex:0error:&error];if(error!=nil){return;}[inputcopyData:inputDataerror:&error];if(error!=nil){return;}[interpreterinvokeWithError:&error];if(error!=nil){return;}

4. Get information about detected objects

If object detection succeeds, the model produces as output three arrays of 40elements (or whatever was specified in thetflite_metadata.json file) each.Each element corresponds to one potential object. The first arrayis an array of bounding boxes; the second, an array of labels; and the third, anarray of confidence values. To get the model outputs:

Swift

varoutput=tryinterpreter.output(at:0)letboundingBoxes=UnsafeMutableBufferPointer<Float32>.allocate(capacity:4*40)output.data.copyBytes(to:boundingBoxes)output=tryinterpreter.output(at:1)letlabels=UnsafeMutableBufferPointer<Float32>.allocate(capacity:40)output.data.copyBytes(to:labels)output=tryinterpreter.output(at:2)letprobabilities=UnsafeMutableBufferPointer<Float32>.allocate(capacity:40)output.data.copyBytes(to:probabilities)

Objective-C

TFLTensor*output=[interpreteroutputTensorAtIndex:0error:&error];if(error!=nil){return;}NSData*boundingBoxes=[outputdataWithError:&error];if(error!=nil){return;}output=[interpreteroutputTensorAtIndex:1error:&error];if(error!=nil){return;}NSData*labels=[outputdataWithError:&error];if(error!=nil){return;}output=[interpreteroutputTensorAtIndex:2error:&error];if(error!=nil){return;}NSData*probabilities=[outputdataWithError:&error];if(error!=nil){return;}

Then, you can combine the label outputs with your label dictionary:

Swift

guardletlabelPath=Bundle.main.path(forResource:"dict",ofType:"txt")else{returntrue}letfileContents=try?String(contentsOfFile:labelPath)guardletlabelText=fileContents?.components(separatedBy:"\n")else{returntrue}foriin0..<40{lettop=boundingBoxes[0*i]letleft=boundingBoxes[1*i]letbottom=boundingBoxes[2*i]letright=boundingBoxes[3*i]letlabelIdx=Int(labels[i])letlabel=labelText[labelIdx]letconfidence=probabilities[i]ifconfidence >0.66{print("Object found:\(label) (confidence:\(confidence))")print("  Top-left: (\(left),\(top))")print("  Bottom-right: (\(right),\(bottom))")}}

Objective-C

NSString*labelPath=[NSBundle.mainBundlepathForResource:@"dict"ofType:@"txt"];NSString*fileContents=[NSStringstringWithContentsOfFile:labelPathencoding:NSUTF8StringEncodingerror:&error];if(error!=nil||fileContents==NULL){return;}NSArray<NSString*>*labelText=[fileContentscomponentsSeparatedByString:@"\n"];for(inti=0;i <40;i++){Float32top,right,bottom,left;Float32labelIdx;Float32confidence;[boundingBoxesgetBytes:&toprange:NSMakeRange(16*i+0,4)];[boundingBoxesgetBytes:&leftrange:NSMakeRange(16*i+4,4)];[boundingBoxesgetBytes:&bottomrange:NSMakeRange(16*i+8,4)];[boundingBoxesgetBytes:&rightrange:NSMakeRange(16*i+12,4)];[labelsgetBytes:&labelIdxrange:NSMakeRange(4*i,4)];[probabilitiesgetBytes:&confidencerange:NSMakeRange(4*i,4)];if(confidence >0.5f){NSString*label=labelText[(int)labelIdx];NSLog(@"Object detected: %@",label);NSLog(@"  Confidence: %f",confidence);NSLog(@"  Top-left: (%f,%f)",left,top);NSLog(@"  Bottom-right: (%f,%f)",right,bottom);}}

Tips to improve real-time performance

If you want to label images in a real-time application, follow these guidelines to achieve the best framerates:

  • Throttle calls to the detector. If a new video frame becomes available while the detector is running, drop the frame.
  • If you are using the output of the detector to overlay graphics on the input image, first get the result, then render the image and overlay in a single step. By doing so, you render to the display surface only once for each input frame. See thepreviewOverlayView andFIRDetectionOverlayView classes in the showcase sample app for an example.

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-12-17 UTC.