Repository files navigation

Note: This code pattern is part of a series that explores telecom call-drop predictions using IBM Cloud Pak for Data, Data Virtualization, Watson OpenScale, and Cognos Analyics. Other assets included in the series include:

• Tutorial -Query across distributed data sources as one: Data virtualization for data analytics
• Code pattern -Predict, Manage and Monitor the call drops of cell towers using Cloud Pak for Data
• Tutorial -Build dashboards in Cognos Analytics on IBM Cloud Pak for Data

Businesses today are increasingly certain that AI will be a driving force in the evolution of their industries over the next few years. To successful infuse AI into your product or solution, there are many factors that challenges its widespread adoption in the business and to achieving their expected outcomes. A few are listed below:

1. Building trust - Organizations and businesses tend to be skeptical about AI because of its "black box" nature, resulting in many promising models no going into production.
2. Algorithm bias - Another inherent problem with AI systems is that they are only as good – or as bad – as the data they are trained on. If the input data is filled with racial, gender, communal or ethnic biases, your model's accuracy is going to eventually drift away.
3. Making decisions explainable - How can the model prove the reasoning behind its decision-making? It is critical that AI outcomes are fully explainable by keeping a complete track of the inputs and outputs of any AI-powered application.

What if there is one console that makes it easier for business users to track and measure AI outcomes?

In this code pattern we demonstrate a way to monitor your AI models in an application using Watson OpenScale. This will be demonstrated with an example of a Telecomm Call Drop Prediction Model. After the user has completed the code pattern, they will learn:

• How to store custom models using open source technology on Watson Machine Learning.
• How to deploy a model and connect the model deployment to Watson OpenScale on Cloud Pak for Data and on IBM Cloud.
• How to setup Model Fairness and Model Quality montiors with Watson OpenScale on Cloud Pak for Data and on IBM Cloud, using a python notebook.
• How to create a project and setup a python notebook on Cloud Pak for Data.

1. Data stored into Cloud Pak for Data internal Db.
2. The joined data is stored back to the internal Db of Cloud Pak for Data and assigned to the current working project.
3. Create machine learning models using Jupyter Python notebooks to predict call drop, one cell tower at a time.
4. Model trained and/or stored in Watson Machine Learning, which is also connected to Watson OpenScale.
5. Configure Fairness, Quality and Explainability Montiors for each cell tower's model, and present within Cloud Pak for Data, or on other external Clouds (Multi-Cloud Architecture).

• Jupyter Notebooks: An open-source web application that allows you to create and share documents that contain live code, equations, visualizations, and explanatory text.
• Pandas: An open source library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.

• AnIBM Cloud Account.
• IBM Cloud Pak for Data
• Watson OpenScale Add On for Cloud Pak for Data

Note: This code pattern includes steps to use WML on IBM Cloud as well as WML within your Cloud Pak for Data instance.

1. Create a Watson Machine Learning instance
2. Create a new project in Cloud Pak for Data
3. Import notebook to Cloud Pak for Data
4. Follow the steps in the notebook
5. Display deployment in Watson OpenScale
6. Additional use-case for Watson OpenScale

Note: This step is only forFor WML on IBM CLOUD users

• Create a newWatson Machine Learning instance on IBM Cloud. Log in to IBM Cloud or sign up for IBM Cloud if you don't have an account by following the on-screen instructions.

• Select the location toDallas region and hit create.

  

• Once the instance is created. Click onService Credentials. Click onNew Credentials and then click onView Credentials. Copy using the icon.

  

NOTE: Save the credentials. It will be required when running the notebook.

• Once you login to your Cloud Pak for Data instance. Click on the (☰)menu icon in the top left corner of your screen and clickProjects.

  

• When you reach the Project list, click onCreate and Empty Project. You will be navigated to a new page where you can enter the desired name(orTelco_CallDrop). Once you click onOk you will go to a new screen. Click onCreate to complete your project creation.

  

• In the created project page, click on theAdd to Project button. Then click onNotebook.

  

• You will be navigated to a new page. Click on theFrom URL tab.

  • For WML on Cloud Users Use this URL -https://github.com/IBM/icp4d-telco-monitor-models-with-wml-openscale/blob/master/notebooks/OpenScale_WML_Cloud.ipynb
  • For WML on Cloud Pak for Data Users Use this URL -https://github.com/IBM/icp4d-telco-monitor-models-with-wml-openscale/blob/master/notebooks/OpenScale_WML_CPD_3.0.ipynb

Enter the nameTowerA_Call_Drop and click on theCreate button.

Note: Choose the Python 3.6 environment.

You will run cells individually by highlighting each cell, then either click theRun button at the top of the notebook. While the cell is running, an asterisk ([*]) will show up to the left of the cell. When that cell has finished executing a sequential number will show up (i.e.[17]).

NOTE: For reference, we have included a completed notebook in the/examples directory of this repo. This version of the notebook includes all the executed steps and outputs. Seehttps://github.com/IBM/icp4d-telco-monitor-models-with-wml-openscale/blob/master/examples/Setup_your_AIOS_Dashboard-Example.jupyter-py36.ipynb

• Click on theRun icon and install the necessary packages described in the first 3 cells.
• Next, restart your kernel by either clicking the restart icon orKernel > Restart.

• Download the dataset using this link-https://github.com/IBM/icp4d-telco-monitor-models-with-wml-openscale/blob/master/data/call_drop_data_train.csv . Right click onDownload and click onSave link as. Save the file with .csv extension.

• In section2.2 Add Dataset, highlight the blank cell by clicking on it. Click on the10/01 button and thebrowse to select the downloaded data set.

  

• Select thecall_drop_data_train.csv file that you added to the project. Under theInsert to code option, clickInsert Pandas DataFrame.

  

IMPORTANT: Ensure the variable name is set todf_data_1.

• Continue running the notebook up to section4.5 Create a new Subscription.

• Before runnning the remaining cells (again, please keep in mind it is better run cell by cell), update the variablesubscription_id in section4.6 Perform Inital Payload Logging, with the ID you created in the previous cell.

  

• You will find this id in the table above the cell, as instructed in the notebook.

• Now, run the rest of the notebook to completion.

• Open the Watson Openscale add-on on Cloud Pak for Data. Click theadd-ons icon located in the top right corner of the panel, which will show the list of add-ons. Use the option menu in the Watson OpenScale add-on tile to open it.

  

• You will see the initial setup looking like this:

  

Included in the data directory of this repo is a file namedTelco_training_final.csv). This contains call drop data for multiple cell towers. If you filter the data on the fieldoutgoing_site_id, you can create a file for each cell tower you would like to monitor. Using the same notebook, you can create a new deployment for each model.

In the following description, we show a dashboard that consists of call drop prediction models for 3 different towers - A, B and C.

OpenScale is configured so that it can monitor how your models are performing over time. The following screen shot gives one such snapshot:

As we can see, the model for Tower C demonstrates a fairness bias warning of 92%.

Data in this day and age comes from a wide variety of sources. Being able to give the right input to a machine learning model is crucial to generating a succesful machine learning implementation in an application. However, if the underlying data is biased in any form, there is a risk that structural inequalities and unfair biases are not just replicated, but also amplified.

Hence, Watson OpenScale is useful when you need to identify the bias, in real-time (i.e., when payload flows into your machine learning system).

Now, that we have identified a bias, let us see the consequence of the bias and what exactly is causing the bias in this example.

Select the desired deployment usingoptions > View Details. You will see a full dashboard with various charts telling you how your model has been performing over various time intervals.

You may not receive the same values as the below screenshots. Since we have only done an initial setup, you will have only see one line in your charts.

Now select_conds in theFairness menu on the left side. If you hover over the graph, you can selectClick to view details. Select a timeline according to your current date.

To interpret the details, have a look at the screenshot below:

Our dataset contains the fields:Clear,Fog,Haze,Partly Cloudy, andRain. The graph above suggests the fieldFog contains a bias.

As we can see with the legend, we have theFavorable Outcome andUnfavorable Outcome. With this bias value we see that with the field valueFog compared to the values has a lesser % of favorable outcomes. Why does this affect my model? It creates a disparity and thereby allows your model to be biased towards the unfavorable outcome (in the case offog).

The best way to explain the consequence of the disparity is by taking this example - suppose we allow the disparity to creep in. Our model will slowly start believing that the moment the weather condition is foggy, it needs to predict the unfavorable outcome(1), and in this case the call will drop. Now, this may not be the case in reality. There maybe other conditions along with the fog that is causing call drops. Hence, it is important to keep monitoring your machine learning model on a real-time basis.

Now click on theView Transactions button shown in the previous screen shot. You will be presented with a list of each transaction. A transaction is basically the recordsof all the data that flows into your model. It gives you the logs and if there is a bias identified, it will also be specified.

Come back to the dashboard and explore theArea under ROC section. Again, hover over the graph and selectClick to view details.

In this example, we have setup only theArea under ROC as a quality measure. As shown in the above image.

You will receive estimates and the configured quality montor values, as shown below. The image shows the overallTrue Positive andFalse Positive ratio for the monitored model.

This code pattern is licensed under the Apache License, Version 2. Separate third-party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to theDeveloper Certificate of Origin, Version 1.1 and theApache License, Version 2.

Apache License FAQ