Background

My fiancée and I were on the lookout for a piece of land to build our future home. We had some criteria in mind, such as the distance to certain places and family members, a minimum and a maximum size, and a few other things.

How we started the search

Each country has their own real estate platforms. In the US, the listing’s metadata is usually public and well-structured, allowing for more advanced searches, and more transparency in general.

In Austria, we mainly have willhaben.at, immowelt.at, immobilienscout24.at, all of which have no publicly available API.

The first step was to setup email alerts on each platform, with our search criteria. Each day, we got emails with all the new listings

The problems

Using the above approach we quickly got overwhelmed with keeping track of the listings, and finding the relevant information. Below are the main problems we encountered:

Remembering which listings we’ve already seen

Many listings were posted on multiple platforms as duplicates, and we had to remember which ones we’ve already looked at. Once we investigated a listing, there was no good way to add notes.

Marketing fluff from real estate agents

Most listings had a lot of unnecessary text, and it took a lot of time to find the relevant information.

[DE] Eine Kindheit wie im Bilderbuch. Am Wochenende aufs Radl schwingen und direkt von zu Hause die Natur entdecken, alte Donau, Lobau, Donauinsel, alles ums Eck. Blumig auch die Straßennamen: Zinienweg, Fuchsienweg, Palargonienweg, Oleanderweg, Azaleengasse, Ginsterweg und AGAVENWEG …. duftiger geht’s wohl nicht.

Which loosely translates to:

[EN] Experience a picture-perfect childhood! Imagine weekends spent effortlessly hopping on your bike to explore nature’s wonders right from your doorstep. With the enchanting Old Danube just a stone’s throw away, adventure is always within reach. Even the street names are a floral delight: Zinienweg, Fuchsienweg, Oleanderweg, Azaleengasse, Ginsterweg, and the exquisite AGAVENWEG… can you imagine a more fragrant and idyllic setting

Although the real estate agent’s poetic flair is impressive, we’re more interested in practical details such as building regulations, noise level and how steep the lot is.

Calculating the distances to POIs

In Austria, the listings usually show you the distances like this:

Children / Schools

• Kindergarten <500 m
• School <1,500 m

Local Amenities

• Supermarket <1,000 m
• Bakery <2,500 m

However I personally get very limited information from this. Instead, we have our own list of POIs that we care about, for example the distance to relatives and to our workplace. Also, just showing the air distance is not helpful, as it’s really about how long it takes to get somewhere by car, bike, public transit, or by foot.

Finding the address

99% of the listings in Austria don’t have any address information available, not even the street name. You can imagine, within a village, being on the main street will be a huge difference when it comes to noise levels and traffic compared to being on a side street. Based on the full listing, it’s impossible to find that information.

The reason for this is that the real estate agents want you to first sign a contract with them, before they give you the address. This is a common practice in Austria, and it’s a way for them to make sure they get their commission.

Visiting the lots

Living in Vienna but searching for plots about 45 minutes away made scheduling viewings a challenge. Even if we clustered appointments, the process was still time-intensive and stressful. In many cases, just seeing the village was often enough to eliminate a lot: highway noise, noticeable power lines, or a steep slope could instantly rule it out.

Additionally, real estate agents tend to have limited information on empty lots—especially compared to houses or condos—so arranging and driving to each appointment wasn’t efficient. We needed a way to explore and filter potential locations before committing to in-person visits.

The solution

It became clear that we needed a way to properly automate and manage this process

A structured way to manage the listings

I wanted a system that allows me to keep track of all the listings we’re interested in a flexible manner:

• Support different views: Excel View, Kanban View, Map View
• Have structured data to filter and sort by, and to do basic calculations on
• Be able to attach images and PDFs
• Be able to add notes to each listing
• Be able to manage the status of each listing (Seen, Interested, Visited, etc.)
• Have it be shareable with my fiancée
• Have it be accessible on the go (for the passenger seat)

We quickly foundAirtable to check all the boxes (Map View is a paid feature):

A simple Telegram bot

Whenever we received new listings per email, we manually went through each one and do a first check on overall vibe, price and village location. Only if we were genuinely interested, we wanted to add it to our Airtable.

So I wrote a simple Telegram bot to which we could send a link to a listing and it’d process it for us.

A way to store a copy of the listings and its images

The simplest and most straightforward way to keep a copy of the listings was to use a headless browser to access the listing’s description and its images.. For that, I simply used theferrum Ruby gem, but any similar tech would work. First, we open the page and prepare the website for a screenshot:

browser=Ferrum::Browser.newbrowser.goto("https://immowelt.at/expose/123456789")# Open the listing# Prepare the website: Depending on the page, you might want to remove some elements to see the full contentifbrowser.current_url.include?("immowelt.at")browser.execute("document.getElementById('usercentrics-root').remove()")rescuenilbrowser.execute("document.querySelectorAll('.link--read-more').forEach(function(el) { el.click() })")# Press all links with the class ".link--read-more", trigger via js as it doesn't work with the driverelsifbrowser.current_url.include?("immobilienscout24.at")browser.execute("document.querySelectorAll('button').forEach(function(el) { if (el.innerText.includes('Beschreibung lesen')) { el.click() } })")end

Once the website is ready, we just took a screenshot of the full page, and save the HTML to have access to it later:

# Take a screenshot of the full pagebrowser.screenshot(path:screenshot_path,full:true)# Save the HTML to have access laterFile.write("listing.html",browser.body)# Find all images referenced on the pageall_images=image_links=browser.css("img").mapdo|img|{name:img["alt"],src:img["src"]}end# The above `all_images` will contain a lot of non-relevant images, such as logos, etc.# Below some messy code to get rid of the majorityimage_links=image_links.selectdo|node|node[:src].start_with?("http")&&!node[:src].include?(".svg")&&!node[:src].include?("facebook.com")end

Important Note: All data processing was done manually on a case-by-case basis for listings we were genuinely interested in. We processed a total of 55 listings over several months across 3 different websites, never engaging in automated scraping or violating any platforms’ terms of service.

A way to extract the relevant info from a listing

One of the main problems with the listings was the amount of irrelevant text, and being able to find the information you care about, like noise levels, building regulations, etc.

Hence, we simply prepared a list of questions we’ll ask AI to answer for us, based on the listing’s description:

generic_context="You are helping a customer search for a property. The customer has shown you a listing for a property they want to buy. You want to help them find the most important information about this property. For each bullet point, please use the specified JSON key. Please answer the following questions:"prompts=["title: The title of the listing","price: How much does this property cost? Please only provide the number, without any currency or other symbols.","size: The total plot area (Gesamtgrundfläche) of the property in m². If multiple areas are provided, please specify '-1'.","building_size: The buildable area or developable area—or the building site—in m². If percentages for buildability are mentioned, please provide those. If no information is available, please provide '-1'.","address: The address, or the street + locality. Please format it in the customary Austrian way. If no exact street or street number is available, please only provide the locality.","other_fees: Any additional fees or costs (excluding broker’s fees) that arise either upon purchase or afterward. Please answer in text form. If no information is available, please respond with an empty string ''.","connected: Is the property already connected (for example, electricity, water, road)? If no information is available, please respond with an empty string ''.","noise: Please describe how quiet or how loud the property is. Additionally, please mention if the property is located on a cul-de-sac. If no details are provided, please use an empty string ''. Please use the exact wording from the advertisement.","accessible: Please reproduce, word-for-word, how the listing describes the accessibility of the property. Include information on how well public facilities can be reached, whether by public transport, by car, or on foot. If available, please include the distance to the nearest bus or train station.","nature: Please describe whether the property is near nature—whether there is a forest or green space nearby, or if it is located in a development, etc. If no information is available, respond with an empty string ''.","orientation: Please describe the orientation of the property. Is it facing south, north, east, west, or a combination? If no information is available, respond with an empty string ''.","slope: Please describe whether the property is situated on a slope or is flat. If it is on a slope, please include details on how steep it is. If no information is available, respond with an empty string ''.","existingBuilding: Please describe whether there is an existing old building on the property. If there is, please include details. If no information is available, respond with an empty string ''.","summary: A summary of this property’s advertisement in bullet points. Please include all important and relevant information that would help a buyer make a decision, specifically regarding price, other costs, zoning, building restrictions, any old building, a location description, public transport accessibility, proximity to Vienna, neighborhood information, advantages or special features, and other standout aspects. Do not mention any brokerage commission or broker’s fee. Provide the information as a bullet-point list. If there is no information about a specific topic, please omit that bullet point entirely. Never say 'not specified' or 'not mentioned' or anything similar. Please do not use Markdown."]

Now we need the full text of the listing. Theferrum gem does a good amount of magic to easily access the text without the need to parse the HTML yourself.

full_text=browser.at_css("body").text

All that’s left is to actually access the OpenAI API (or similar) to get the answers to the questions:

ai_responses=ai.ask(prompts:prompts,context:full_text)

To upload the resulting listing to Airtable I used theairrecord gem.

create_hash={"Title"=>ai_responses["title"],"Price"=>ai_responses["price"].to_i,"Noise"=>ai_responses["noise"],"URL"=>browser.url,"Summary"=>("- "+Array(ai_responses["summary"]).join("\n- "))}new_entry=MyEntry.create(create_hash)

For the screenshots, you’ll need some additionaly boilerplate code to first download, and then upload the images to a temporary S3 bucket, and then to Airtable using the Airtable API.

Below you can see the beautifully structured data in Airtable (in German), already including the public transit times:

A way to find the address

The real estate agents usually actively blur any street names or other indicators if there is a map in the listing. There is likely no good automated way to do this. Since this project was aimed at only actually parsing the listings I was already interested in, I only had a total of 55 listings to manually find the address for.

Turns out, for around 80% for the listings I was able to find the exact address using one of the following approaches:

Variant A: Usinggeoland.at

This is approach is Austria specific, but I could imagine other countries will have similar systems in place. I noticed many listings had a map that looks like this:

There are no street names, street numbers or river names. But you can see some numbers printed on each lot. Turns out, those are the “Grundstücksnummern” (lot numbers). The number tied together with the village name is unique, so you’ll be able to find that area of the village within a minute.

Variant B: By analysing the angles of the roads and rivers

The above map was a tricky one: It’s zoomed in so much that you can’t really see any surroundings. Also, the real estate agent hides the lot numbers, and switched to a terrain view.

The only orientation I had was the river. This village had a few rivers, but only 2 of them went in roughly the direction shown. So I went through those rivers manually to see where the form of the river matches the map, together with the light green background in the center, and the gray outsides. After around 30mins, I was able to find the exact spot (left: listing, right: my map)

Variant C: Requesting the address from the real estate agent

As the last resort, we contacted the real estate agent and ask for the address.

I want to emphasize: this system isn’t about avoiding real estate agents, but optimizing our search efficiency (like getting critical details same-day, and not having to jump on a call). For any property that passed our vetting, we contacted the agent and went through the purchase process as usual.

A way to calculate the distances to POIs

Once the address was manually entered, the Ruby script would pick up that info, and calculate the commute times to a pre-defined list of places using the Google Maps API. This part of the code is mostly boilerplate to interact with the API, and parse its responses.

For each destination we were interested in, we calculated the commute time by car, bike, public transit, and by foot.

One key aspect that I was able to solve was the “getting to the train station” part. In most cases, we want to be able to take public transit, but with Google Maps it’s an “all or nothing”, as in, you either use public transit for the whole route, or you don’t.

More realistically, we wanted to drive to the train station (either by bike or car), and then take the train from there.

The code below shows a simple way I was able to achieve this. I’m well aware that this may not work for all the cases, but it worked well for all the 55 places I used it for.

ifmode=="transit"# For all routes calculated for public transit, first extract the "walking to the train station" part# In the above screenshot, this would be 30mins and 2.3kmres[:walking_to_closest_station_time_seconds]=data["routes"][0]["legs"][0]["steps"][0]["duration"]["value"]res[:walking_to_closest_station_distance_meters]=data["routes"][0]["legs"][0]["steps"][0]["distance"]["value"]# Get the start and end location of the walking partstart_location=data["routes"][0]["legs"][0]["steps"][0]["start_location"]end_location=data["routes"][0]["legs"][0]["steps"][0]["end_location"]# Now calculate the driving distance to the nearest stationres[:drive_to_nearest_station_duration_seconds]=self.calculate_commute_duration(from:"#{start_location["lat"]},#{start_location["lng"]}",to:"#{end_location["lat"]},#{end_location["lng"]}",mode:"driving")[:total_duration_seconds]end

A way to visit the lots without an appointment

Once we had a list of around 15 lots we were interested in, we planned a day to visit them all. Because we have the exact address, there was no need for an appointment.

To find the most efficient route I used theRouteXL. You can upload a list of addresses you need to visit, and define precise rules, and it will calculate the most (fuel & time) efficient route, which you can directly import to Google Maps for navigation.

While driving to the next stop, my fiancée read the summary notes from the Airtable app, so we already knew the price, description, size and other characteristics of the lot by the time we arrive.

This approach was a huge time saver for us. Around 75% of the lots we could immediately rule out as we arrived. Sometimes there was a loud road, a steep slope, a power line, a noisy factory nearby, or most importantly: it just didn’t feel right. There were huge differences invibes when you stand in front of a lot.

We always respected property boundaries - it was completely sufficient to stand in front of the lot, and walk around the area a bit to get a very clear picture.

Conclusion

After viewing 42 lots in-person on 3 full-day driving trips, we found the perfect one for us and contacted the real estate agent to do a proper viewing. We immediately knew it was the right one, met the owner, and signed the contract a few weeks later.

The system we built was a huge time saver for us, and allowed us to smoothly integrate the search process into our daily lives. I loved being able to easily access all the information we needed, and take notes on the go, while exploring the different villages of the Austrian countryside.

If you’re interested in getting access to the code, please reach out to me. I’m happy to share more info, but I want to make sure it’s used responsibly and in a way that doesn’t violate any terms of service of the platforms we used. Also, it’s quite specific to our use case, so it may need some adjustments to work for you.

Note: This is a cross-post of the original publication oncontextsdk.com.

This is the third post of our machine learning (ML) for iOS apps series. Be sure to readpart 1 andpart 2 first. So far we’ve received incredible positive feedback. We always read about the latest advancements in the space of Machine Learning and Artificial Intelligence, but at the same time, we mostly use external APIs that abstract out the ML aspect, without us knowing what’s happening under the hood. This blog post series helps us fully understand the basic concepts of how a model comes to be, how it’s maintained and improved, and how to leverage it in real-life applications

Introduction

One critical aspect of machine learning is to constantly improve and iterate your model. There are many reasons for that, from ongoing changes in user-behavior, other changes in your app, all the way to simply getting more data that allows your model to be more precise.

In this article we will cover:

• How to prevent data blindness
• How to remotely, continuously calibrate thresholds, and encode additional model metadata
• How to group your user-base into separate buckets, allowing you to evaluate real-life performance
• How to monitor and evaluate performance of your models

What we’ve built so far in the first 2 blog posts

Our iOS app sends non-PII real-world context data to our API server, which will store the collected data in our database (full details here).

Our API servers respond with the latest model details so the client can decide if it needs to download an update or not.

Model Metadata Management

It’s important for you to be able to remotely calibrate & fine-tune your models and their metadata, with the random upsell chance being one of those values. Since our SDK already communicates with our API server to get the download info for the most recent ML model, we can provide those details to the client together with the download URL.

privatestructSaveableCustomModelInfo:Codable{letmodelVersion:StringletupsellThreshold:DoubleletrandomUpsellChance:DoubleletcontextSDKSpecificMetadataExample:Int}

• modelVersion: At ContextSDK, we use a UUID as a model version. For our architecture, there is no need for the client to “understand” which model version is higher than the previous one. Our API servers will handle those tasks
• upsellThreshold: Our CoreML model returns a score between 0 - 1 on how likely the user is to convert in the current moment. Depending on our customer’s preference and project goals, we can use this value to decide on the “Prompt Intensity Level”
• randomUpsellChance: That’s the value we described above to help us prevent data blind spots. As we monitor our incoming data, we can remotely change this value to fit our current requirements
• Other model metadata: We use this to include more details on what exact type of data the model requires as inputs

Model Inputs

At ContextSDK, we generate and use more than 180 on-device signals to evaluate how good a moment is to show a certain type of content. With machine learning for this use-case, you don’t want a model to have 180 inputs, as training such a model would require enormous amounts of data, as the training classifier wouldn’t know which columns to start with. Without going into too much Data Science details, you’d want the ratio between columns (inputs) and rows (data entries) to meet certain requirements.

Hence, we have multiple levels of data processing and preparations when training our Machine Learning model. One step is responsible for finding the context signals that contribute the highest amount of weight in the model, and focus on those. The signals used vary heavily depending on the app.

It was easy to dynamically pass in the signals that are used by a given model in our architecture. We’vepublished a blog post on how our stack enforces matching signals across all our components.

For simple models, you can use the pre-generated Swift classes for your model. Apple recommends using theMLFeatureProvider for more complicated cases, like when your data is collected asynchronously, to reduce the amounts of data you’d need to copy, or for other more complicated data sources.

funcfeatureValue(forfeatureName:String)->MLFeatureValue?{// Fetch your value here based on the `featureName`stringValue=self.signalsManager.signal(byString:featureName)// Simplified examplereturnMLFeatureValue(string:stringValue.string())}

We won’t go into full detail on how we implemented the mapping of the various different types. We’ve created a subclass ofMLFeatureProvider and implemented thefeatureValue method to dynamically get the right values for each input.

As part of the MLFeatureProvider subclass, you need to provide a list of all featureNames. You can easily query the required parameters for a given CoreML file using the following code:

featureNames=Set(mlModel.modelDescription.inputDescriptionsByName.map({$0.value.name}))

Grouping your user-base

Most of us have used AB tests with different cohorts, so you’re most likely already familiar with this concept. We wanted something basic, with little complexity, that works on-device, and doesn’t rely on any external infrastructure to assign the cohort.

For that, we createdControlGrouper, a class that takes in any type of identifier that we only use locally to assign a control group:

importCommonCryptoclassControlGrouper{/***        The groups are defined as ranges between the upperBoundInclusive of groups.        The first group will go from 0 to upperBoundInclusive[0]        The next group from upperBoundInclusive[0] to upperBoundInclusive[1]        The last group will be returned if no other group matches, though for clarity the upperBoundInclusive should be set to 0.        If there is only 1 group regardless of specified bounds it is always used. Any upperBoundInclusive higher than 1 acts just like 1.        Groups will be automatically sorted so do not need to be passed in in the correct order.        An arbitrary number of groups can be supplied and given the same userIdentifier and modelName the same assignment will always be made.     */classfuncgetGroupAssignment<T>(userIdentifier:String,modelName:String,groups:[ControlGroup<T>])->T{if(groups.count<=1){returngroups[0].value}// We create a string we can hash using all components that should affect the result the group assignment.letassignmentString="\(userIdentifier)\(modelName)".data(using:String.Encoding.utf8)// Using SHA256 we can map the arbitrary assignment string on to a 256bit space and due to the nature of hashing:// The distribution of input string will be even across this space.// Any tiny change in the assignment string will be massive difference in the output.vardigest=[UInt8](repeating:0,count:Int(CC_SHA256_DIGEST_LENGTH))ifletvalue=(assignmentStringas?NSData){CC_SHA256(value.bytes,CC_LONG(value.count),&digest)}// We slice off the first few bytes and map them to an integer, then we can check from 0-1 where this integer lies in the range of all possible buckets.ifletbucket=UInt32(data:Data(digest).subdata(in:0..<4)){letposition=Double(bucket)/Double(UInt32.max)// Finally knowing the position of the installation in our distribution we can assign a group based on the requested groups by the caller.// We sort here in case the caller does not provide the groups from lowest to higest.letsortedGroups=groups.sorted(by:{$0.upperBoundInclusive<$1.upperBoundInclusive})forgroupinsortedGroups{if(position<=group.upperBoundInclusive){returngroup.value}}}// If no group matches, we use the last one as we can just imagine its upperBoundInclusive extending to the end.returngroups[groups.count-1].value}}structControlGroup<T>{letvalue:TletupperBoundInclusive:Double}

For example, this allows us to split the user-base into 3 equally sized groups, one of which being the control group.

What’s data blindness?

Depending on what you use the model for, it is easy to end up in some type of data blindness once you start using your model.

For example, let’s say your model decides it’s a really bad time to show a certain type of prompt if the battery is below 7%. While this may be statistically correct based on real-data, this would mean you’re not showing any prompts for those cases (< 7% battery level) any more.

However, what if there are certain exceptions for those cases, that you’ll only learn about once you’ve collected more data? For example, maybe that <7% battery level rule doesn’t apply, if the phone is currently plugged in?

This is an important issue to consider when working with machine learning: Once you start making decisions based on your model, you’ll create blind-spots in your learning data.

How to solve data blindness?

The only way to get additional, real-world data for those blind spots is to still sometimes decide to show a certain prompt even if the ML model deems it to be a bad moment to do so. This should be optimized to a small enough percentage that it doesn’t meaningfully reduce your conversion rates, but at the same time enough that you’ll get meaningful, real-world data to train and improve your machine learning model over time. Once we train the initial ML model, we look into the absolute numbers of prompts & sales, and determine an individual value for what the percentage should be.

Additionally, by introducing this concept of still randomly showing a prompt even if the model deems it to be a bad moment, it can help to prevent situations where a user may never see a prompt, due to the rules of the model. For example, a model may learn that there are hardly any sales in a certain region, and therefore decide to always skip showing prompts.

This is something we prevent on multiple levels for ContextSDK, and this one is the very last resort (on-device) to be sure this won’t happen. We continuously analyze, and evaluate our final model weights, as well as the incoming upsell data, to ensure our models leverage enough different types of signals.

lethasInvalidResult=upsellProbability==-1letcoreMLUpsellResult=(upsellProbability>=executionInformation.upsellThreshold||hasInvalidResult)// In order to prevent cases where users never see an upsell this allows us to still show an upsell even if the model thinks it's a bad time.letrandomUpsellResult=Double.random(in:0...1)<executionInformation.randomUpsellChanceletupsellResult=(coreMLUpsellResult||randomUpsellResult)?UpsellResult.shouldUpsell:.shouldSkip// We track if this prompt was shown as part of our random upsells, this way we can track performance.modelBasedSignals.append(SignalBool(id:.wasRandomUpsell,value:randomUpsellResult&&!coreMLUpsellResult))

As an additional layer, we also have a control group (with varying sizes) that we generate and use locally.

How to compare your model’s performance with the baseline

We’re working with a customer who’s currently aggressively pushing prompts onto users. They learned that those prompts lead to churn in their user-base, so their number one goal was to reduce the number of prompts, while keeping as much of the sales as possible.

We decided for a 50/50 split for their user-base to have two large enough buckets to evaluate the model’s performance

Depending on the goal of your model, you may want to target other key metrics to evaluate the performance of your model. In the table above, the main metric we looked for was the conversion rate, which in this case has a performance of +81%.

Above is an example of a model with poor performance: the conversion rate went down by 6% and the total number of sales dropped in half. Again, in our case we were looking for an increase in conversion rate, where in this case this goal is clearly not achieved.

Our systems continuously monitor whatever key metric we want to push (usually sales or conversion rate, depending on the client’s preference). As soon as a meaningful number of sales were made for both buckets, the performance is compared, and if it doesn’t meet our desired outcomes, the rollout will immediately be stopped, and rolled back, thanks to the over-the-air update system described in this article

Conclusion

In this article we’ve learned about the complexity of deploying machine learning models, and measuring and comparing their performance. It’s imperative to continuously monitor how well a model is working, and have automatic safeguards and corrections in place.

Overall, Apple has built excellent machine learning tools around CoreML, which have been built into iOS for many years, making it easy to build intelligent, offline-first mobile apps that nicely blend into the user’s real-world environment.

Note: This is a cross-post of the original publication oncontextsdk.com.

This is the second blog post covering various machine learning (ML)concepts of iOS apps, be sure to readpart 1 first. Initially thiswas supposed to be a 2-piece series, but thanks to the incrediblefeedback of the first one, we’ve decided to cover even more on thistopic, and go into more detail.

Introduction

For some apps it may be sufficient to train a ML (machine learning)model once, and ship it with the app itself. However, most mobile appsare way more dynamic than that, constantly changing and evolving. It istherefore important to be able to quickly adapt and improve your machinelearning models, without having to do a full app release, and go throughthe whole App Store release & review process.

In this series, we will explore how to operate machine learning modelsdirectly on your device instead of relying on external servers vianetwork requests. Running models on-device enables immediatedecision-making, eliminates the need for an active internet connection,and can significantly lower infrastructure expenses.

In the example of this series, we’re using a model to make a decision onwhen to prompt the user to upgrade to the paid plan based on a set ofdevice-signals, to reduce user annoyances, while increasing our paidsubscribers.

Step 1: Shipping a base-model with your app’s binary

We believe in the craft of beautiful, reliable and fast mobile apps.Running machine-learning devices on-device makes your app responsive,snappy and reliable. One aspect to consider is the first app launch,which is critical to prevent churn and get the user hooked to your app.

To ensure your app works out of the box right after its installation, werecommend shipping your pre-trained CoreML file with your app. Ourpart 1 covers how to easily achieve this with Xcode

Step 2: Check for new CoreML updates

Your iOS app needs to know when a new version of the machine learningfile is available. This is as simple as regularly sending an emptynetwork request to your server. Your server doesn’t need to besophisticated, we initially started with a static file host (like S3, oralike) that we update whenever we have a new model ready.

The response could use whatever versioning you prefer:

• A version number of your most recent model
• The timestamp your most recent model was trained
• A checksum
• A randomly generated UUID

Whereas the iOS client would compare the version number of most recentlydownloaded model with whatever the server responds with. Which approachyou choose, is up to you, and your strategy on how you want to rollout,monitor and version your machine learning models.

Over time, you most likely want to optimize the number of networkrequests. Our approach combines a smart mechanism where we’d combine theoutcome collection we use to train our machine learning models with themodel update checks, while also leveraging a flushing technique to batchmany events together to minimize overhead and increase efficiency.

Ideally, the server’s response already contains the download URL of thelatest model, here is an example response:

{"url":"https://krausefx.github.io/CoreMLDemo/models/80a2-82d1-bcf8-4ab5-9d35-d7f257c4c31e.mlmodel"}

The above example is a little simplified, and we’re using the model’sfile name as our version to identify each model.

You’ll also need to consider which app version is supported. In ourcase, a new ContextSDK version may implement additional signals that areused as part of our model. Therefore we provide the SDK version as partof our initial polling request, and our server responds with the latestmodel version that’s supported.

First, we’re doing some basic scaffolding, creating a newModelDownloadManager class:

importFoundationimportCoreMLclassModelDownloadManager{privateletfileManager:FileManagerprivateletmodelsFolder:URLprivateletmodelUpdateCheckURL="https://krausefx.github.io/CoreMLDemo/latest_model_details.json"init(fileManager:FileManager=.default){self.fileManager=fileManagerifletfolder=fileManager.urls(for:.applicationSupportDirectory,in:.userDomainMask).first?.appendingPathComponent("context_sdk_models"){self.modelsFolder=foldertry?fileManager.createDirectory(at:folder,withIntermediateDirectories:true)}else{fatalError("Unable to find or create models folder.")// Handle this more gracefully}}}

And now to the actual code: Downloading the model details to check if anew model is available:

internalfunccheckForModelUpdates()asyncthrows{guardleturl=URL(string:modelUpdateCheckURL)else{throwURLError(.badURL)}let(data,_)=tryawaitURLSession.shared.data(from:url)guardletjsonObject=tryJSONSerialization.jsonObject(with:data)as?[String:Any],letmodelDownloadURLString=jsonObject["url"]as?String,letmodelDownloadURL=URL(string:modelDownloadURLString)else{throwURLError(.cannotParseResponse)}tryawaitdownloadIfNeeded(from:modelDownloadURL)}

Step 3: Download the latest CoreML file

If a new CoreML model is available, your iOS app now needs to downloadthe latest version. You can use any method of downloading the staticfile from your server:

// It's important to immediately move the downloaded CoreML file into a permanent locationprivatefuncdownloadCoreMLFile(fromurl:URL)asyncthrows->URL{let(tempLocalURL,_)=tryawaitURLSession.shared.download(for:URLRequest(url:url))letdestinationURL=modelsFolder.appendingPathComponent(tempLocalURL.lastPathComponent)tryfileManager.moveItem(at:tempLocalURL,to:destinationURL)returndestinationURL}

Considering Costs

Depending on your user-base, infrastructure costs will be a big factoron how you’re gonna implement the on-the-fly update mechanism.

For example, an app with 5 Million active users, and a CoreML file sizeof 1 Megabyte, would generate a total data transfer of 5 Terabyte. Ifyou were to use a simple AWS S3 bucket directly with $0.09 per GBegress costs, this would yield costs of about $450 for each modelrollout (not including the free tier).

As part of this series, we will talk about constantly rolling out new,improved challenger models, running various models in parallel, anditerating quickly, paying this amount isn’t a feasible solution.

One easy fix for us was to leverageCloudFlare R2, whichis faster and significantly cheaper. The same numbers as above costs usless than $2, and would be completely free if we include the free tier.

Step 4: Compile the CoreML file on-device

After successfully downloading the CoreML file, you need to compile iton-device. While this sounds scary, Apple made it a seamless, easy andsafe experience. Compiling the CoreML file on-device is a requirement,and ensures that the file is optimized for the specific hardware it runson.

privatefunccompileCoreMLFile(atlocalFilePath:URL)throws->URL{letcompiledModelURL=tryMLModel.compileModel(at:localFilePath)letdestinationCompiledURL=modelsFolder.appendingPathComponent(compiledModelURL.lastPathComponent)tryfileManager.moveItem(at:compiledModelURL,to:destinationCompiledURL)tryfileManager.removeItem(at:localFilePath)returndestinationCompiledURL}

You are responsible for the file management, including that you storethe resulting ML file into a permanent location. In general, filemanagement on iOS can be a little tedious, covering all the various edgecases.

You can also find the official Apple Docs onDownloading andCompiling a Model on the User’s Device.

Step 5: Additional checks and clean-ups

We don’t yet have a logic on how we decide if we want to download thenew model. In this example, we’ll do something very basic: each model’sfile-name is a unique UUID. All we need to do is to check if a modelunder the exact file name is available locally:

privatefuncdownloadIfNeeded(fromurl:URL)asyncthrows{letlastPathComponent=url.lastPathComponent// Check if the model file already exists (for this sample project we use the unique file name as identifier)ifletlocalFiles=try?fileManager.contentsOfDirectory(at:modelsFolder,includingPropertiesForKeys:nil),localFiles.contains(where:{$0.lastPathComponent==lastPathComponent}){// File exists, you could add a version check here if versions are part of the file name or metadataprint("Model already exists locally. No need to download.")}else{letdownloadedURL=tryawaitdownloadCoreMLFile(from:url)// File does not exist, download itletcompiledURL=trycompileCoreMLFile(at:downloadedURL)trydeleteAllOutdatedModels(keeping:compiledURL.lastPathComponent)print("Model downloaded, compiled, and old models cleaned up successfully.")}}

Of course we want to be a good citizen, and delete all older models fromthe local storage. Also, for this sample project, this is required, aswe’re using UUIDs for versioning, meaning the iOS client actuallydoesn’t know about which version is higher. For sophisticated systemsit’s quite common to not have this transparency to the client, as thebackend may be running multiple experiments and challenger models inparallel across all clients.

privatefuncdeleteAllOutdatedModels(keepingrecentModelFileName:String)throws{leturlContent=tryfileManager.contentsOfDirectory(at:modelsFolder,includingPropertiesForKeys:nil,options:.skipsHiddenFiles)forfileURLinurlContentwherefileURL.lastPathComponent!=recentModelFileName{tryfileManager.removeItem(at:fileURL)}}

Step 6: Execute the newly downloaded CoreML file instead of the bundled version

Now all that’s left is to automatically switch between the CoreML filethat we bundled within our app, and the file we downloaded from ourservers, whereas we’d always want to prefer the one we downloadedremotely.

In our ModelDownloadManager, we want an additional function that exposesthe model we want to use. This can either be the bundled CoreML model,or the CoreML model downloaded most recently over-the-air

internalfunclatestModel()->MyFirstCustomModel?{letfileManagerContents=(try?fileManager.contentsOfDirectory(at:modelsFolder,includingPropertiesForKeys:nil))??[]ifletlatestFileURL=fileManagerContents.sorted(by:{$0.lastPathComponent>$1.lastPathComponent}).first,letotaModel=try?MyFirstCustomModel(contentsOf:latestFileURL){returnotaModel}elseifletbundledModel=try?MyFirstCustomModel(configuration:MLModelConfiguration()){returnbundledModel// Fallback to the bundled model if no downloaded model exists}returnnil}

There are almost no changes needed to our code base frompart 1.

Instead of using the MyFirstCustomModel initializer directly, we nowneed to use the newly created .latestModel() method.

letbatteryLevel=UIDevice.current.batteryLevelletbatteryCharging=UIDevice.current.batteryState==.charging||UIDevice.current.batteryState==.fulldo{letmodelInput=MyFirstCustomModelInput(input:[Double(batteryLevel),Double(batteryCharging?1.0:0.0)])ifletcurrentModel=modelDownloadManager.latestModel(),letmodelMetadata=currentModel.model.modelDescription.metadata[.description]{letresult=trycurrentModel.prediction(input:modelInput)letclassProbabilities=result.featureValue(for:"classProbability")?.dictionaryValueletupsellProbability=classProbabilities?["Purchased"]?.doubleValue??-1showAlertDialog(message:("Chances of Upsell:\(upsellProbability), executed through model\(modelMetadata)"))}else{showAlertDialog(message:("Could not run CoreML model"))}}catch{showAlertDialog(message:("Error running CoreML file:\(error)"))}

Step 7: Decide when you want to trigger the update check

The only remaining code that’s left: triggering the update check. Whenyou do that will highly depend on your app, and the urgency in which youwant to update your models.

Task{do{tryawaitmodelDownloadManager.checkForModelUpdates()showAlertDialog(message:("Model update completed successfully."))}catch{// Handle possible errors hereshowAlertDialog(message:("Failed to update model:\(error.localizedDescription)"))}}

Demo App

As part of this series, we’ve built out a demo app that shows all ofthis end-to-end in action. You can find it available here on GitHub:https://github.com/KrauseFx/CoreMLDemo:

• ModelDownloadManager.swift
• ViewController.swift

What’s next?

Today we’ve covered how you can roll out new machine learning modelsdirectly to your users’ iPhones, running them directly on theirML-optimized hardware. Using this approach you can make decisions onwhat type of content, or prompts you show based on the user’s context,powered by on-device machine learning execution. Updating CoreML filesquickly, on-the-fly without going through the full App Store releasecycle is critical, to quickly react to changing user-behaviors, whenintroducing new offers in your app, and to constantly improve your app,be it increasing your conversion rates, reducing annoyances and churn,or optimizing other parts of your app.

This is just the beginning: Next up, we will talk about how to managethe rollout of new ML models, in particular:

• How to safely rollout new models: monitor, pause or rollback faulty models
• How to monitor performance of deployed models
• How to reliably compare performance between models, and the baseline performance

Excited to share more on what we’ve learned when building ContextSDK topower hundreds of machine learning models distributed across more than25 Million devices.

Update: Head over to thethird post of the ML series

Note: This is a cross-post of the original publication oncontextsdk.com.

Introduction

Machine Learning (ML) in the context of mobile apps is a wide topic,with different types of implementations and requirements. On the highestlevels, you can distinguish between:

1. Running ML models on server infrastructure and accessing it fromyour app through API requests
2. Running ML models on-device within your app (we will focus on this)
3. Fine-tuning pre-trained ML models on-device based on user behavior
4. Training new ML models on-device

As part of this blog series, we will be talking about variant 2: Westart out by training a new ML model on your server infrastructure basedon real-life data, and then distributing and using that model withinyour app. Thanks to Apple’s CoreML technology, this process has becomeextremely efficient & streamlined.

We wrote this guide for all developers, even if you don’t have any priordata science or backend experience.

Step 1: Collecting the data to train your first ML model

To train your first machine learning model, you’ll need some data youwant to train the model on. In our example, we want to optimize when toshow certain prompts or messages in iOS apps.

Let’s assume we have your data in the following format:

• Outcome describes the result of the user interaction, in thiscase, if they purchased an optional premium upgrade
• Battery Level is the user’s current battery level as a float
• Phone Charging defines if the phone is currently plugged in as aboolean

In the above example, the “label” of the dataset is theoutcome. Inmachine learning, a label for training data refers to the output oranswer for a specific instance in a dataset. The label is used to traina supervised model, guiding it to understand how to classify new, unseenexamples or predict outcomes.

How you get the data to train your model is up to you. In our case, we’dcollect non-PII data just like the above example, to train models basedon real-life user behavior. For that we’ve built out our own backendinfrastructure, which we’ve already covered in our Blog:

• Building the Infrastructure to Ingest 40m Context Events perDay
• Unifying Data Models Across a HeterogeneousStack

Step 2: Load and prepare your data

There are different technologies available to train your ML model. Inour case, we chose Python, together with pandas and sklearn.

Load the recorded data into a pandas DataFrame:

importpandasaspdrows=[['Dismissed',0.90,False],['Dismissed',0.10,False],['Purchased',0.24,True],['Dismissed',0.13,True]]data=pd.DataFrame(rows,columns=['Outcome','Battery Level','Phone Charging?'])print(data)

Instead of hard-coded data like above, you’d access your database withthe real-world data you’ve already collected.

Step 3: Split the data between training and test data

To train a machine learning model, you need to split your data into atraining set and a test set. We won’t go into detail about why that’sneeded, since there are many great resources out there that explain thereasoning, like this excellentCGP Video.

fromsklearn.model_selectionimporttrain_test_splitX=data.drop("Outcome",axis=1)Y=data["Outcome"]X_train,X_test,Y_train,Y_test=train_test_split(X,Y,test_size=0.2,shuffle=True)

The code above splits your data by a ratio of 0.2 (⅕) and separates theX and the Y axis, which means separating the label (“Outcome”) from thedata (all remaining columns).

Step 4: Start Model Training

As part of this step, you’ll need to decide on what classifier you wantto use. In our example, we will go with a basic RandomForest classifier:

fromsklearn.ensembleimportRandomForestClassifierfromsklearn.metricsimportclassification_reportclassifier=RandomForestClassifier()classifier.fit(X_train,Y_train)Y_pred=classifier.predict(X_test)print(classification_report(Y_test,Y_pred,zero_division=1))

The output of the above training will give you a classification report. In simplified words, it will tell you more of how accurate the trainedmodel is.

In the screenshot above, we’re only using test data as part of this blogseries. If you’re interested in how to interpret and evaluate theclassification report, checkout this guide.

Step 5: Export your model into a CoreML file

Apple’s officialCoreMLToolsmake it extremely easy to export the classifier (in this case, ourRandom Forest) into a .mlmodel (CoreML) file, which we can run onApple’s native ML chips. CoreMLTools support a variety of classifiers,however not all of them, so be sure to verify its support first.

importcoremltoolscoreml_model=coremltools.converters.sklearn.convert(classifier,input_features="input")coreml_model.short_description="My first model"coreml_model.save("MyFirstCustomModel.mlmodel")

Step 6: Bundle the CoreML file with your app

For now, we will simply drag & drop the CoreML file into our Xcodeproject. In a future blog post we will go into detail on how to deploynew ML models over-the-air.

Once added to your project, you can inspect the inputs, labels, andother model information right within Xcode.

Step 7: Executing your Machine Learning model on-device

Xcode will automatically generate a new Swift class based on yourmlmodel file, including the details about the inputs, and outputs.

letbatteryLevel=UIDevice.current.batteryLevelletbatteryCharging=UIDevice.current.batteryState==.charging||UIDevice.current.batteryState==.fulldo{letmodelInput=MyFirstCustomModelInput(input:[Double(batteryLevel),Double(batteryCharging?1.0:0.0)])letresult=tryMyFirstCustomModel(configuration:MLModelConfiguration()).prediction(input:modelInput)letclassProbabilities=result.featureValue(for:"classProbability")?.dictionaryValueletupsellProbability=classProbabilities?["Purchased"]?.doubleValue??-1print("Chances of Upsell:\(upsellProbability)")}catch{print("Error running CoreML file:\(error)")}

In the above code you can see that we pass in the parameters of thebattery level, and charging status, using an array of inputs, onlyidentified by the index. This has the downside of not being mapped by anexact string, but the advantage of faster performance if you havehundreds of inputs.

Alternatively, during model training and export, you can switch to usinga String-based input for your CoreML file if preferred.

We will talk more about how to best set up your iOS app to get the bestof both worlds, while also supporting over-the-air updates, dynamicinputs based on new models, and how to properly handle errors, processthe response, manage complex AB tests, safe rollouts, and more.

Conclusion

In this guide we went from collecting the data to feed into your MachineLearning model, to training the model, to running it on-device to makedecisions within your app. As you can see, Python and its libraries,including Apple’s CoreMLTools, make it very easy to get started withyour first ML model. Thanks to native support of CoreML files in Xcode,and executing them on-device, we have all the advantages of the Appledevelopment platform, like inspecting model details within Xcode, strongtypes and safe error handling.

In your organization, you’ll likely have a Data Scientist who will be incharge of training, fine-tuning and providing the model. The above guideshows a simple example - with ContextSDK we take more than 180 differentsignals into account, of different types, patterns, and sources,allowing us to achieve the best results, while keeping the resultingmodels small and efficient.

Within the next few weeks, we will be publishing a second post on thattopic, showcasing how you can deploy new CoreML files to Millions of iOSdevices over-the-air within seconds, in a safe & cost-efficient manner,managing complicated AB tests, dynamic input parameters, and more.

Update: Head over to thesecond post of the ML series